| *** cshen has quit IRC | 00:09 | |
| *** ianychoi__ has joined #openstack-ansible | 00:14 | |
| *** tosky has quit IRC | 00:15 | |
| *** ianychoi_ has quit IRC | 00:17 | |
| *** cp-- has quit IRC | 00:50 | |
| *** cp- has joined #openstack-ansible | 00:51 | |
| spatel | jrosser i found it why journalbeat overriding origin hostname :) | 00:56 |
|---|---|---|
| spatel | processor: - add_observer_metadata: ~. fixed my issue | 00:57 |
| spatel | by default journalbeat think i am the origin of all logs but when you process external journal file then you need to preserve information using that setting | 00:58 |
| *** cshen has joined #openstack-ansible | 01:02 | |
| *** cshen has quit IRC | 01:07 | |
| *** klamath_atx has joined #openstack-ansible | 02:05 | |
| *** cshen has joined #openstack-ansible | 03:02 | |
| *** klamath_atx has quit IRC | 03:05 | |
| *** cshen has quit IRC | 03:07 | |
| *** macz_ has quit IRC | 03:12 | |
| *** d34dh0r53 has quit IRC | 03:47 | |
| *** d34dh0r53 has joined #openstack-ansible | 03:48 | |
| *** d34dh0r53 has quit IRC | 03:48 | |
| *** d34dh0r53 has joined #openstack-ansible | 03:49 | |
| *** d34dh0r53 has quit IRC | 03:49 | |
| *** d34dh0r53 has joined #openstack-ansible | 03:53 | |
| *** d34dh0r53 has quit IRC | 03:55 | |
| *** d34dh0r53 has joined #openstack-ansible | 03:56 | |
| *** d34dh0r53 has quit IRC | 03:56 | |
| *** d34dh0r53 has joined #openstack-ansible | 03:57 | |
| *** d34dh0r53 has quit IRC | 03:57 | |
| *** d34dh0r53 has joined #openstack-ansible | 03:58 | |
| *** d34dh0r53 has quit IRC | 03:58 | |
| *** d34dh0r53 has joined #openstack-ansible | 03:59 | |
| *** spatel has quit IRC | 04:13 | |
| *** spatel has joined #openstack-ansible | 04:14 | |
| *** zul has quit IRC | 04:16 | |
| *** spatel has quit IRC | 04:24 | |
| *** spatel has joined #openstack-ansible | 04:24 | |
| *** lemko2 has joined #openstack-ansible | 04:28 | |
| *** lemko has quit IRC | 04:29 | |
| *** lemko2 is now known as lemko | 04:29 | |
| *** spatel has quit IRC | 04:34 | |
| *** cshen has joined #openstack-ansible | 05:03 | |
| *** hamzaachi has joined #openstack-ansible | 05:05 | |
| *** cshen has quit IRC | 05:07 | |
| *** macz_ has joined #openstack-ansible | 05:15 | |
| *** macz_ has quit IRC | 05:20 | |
| *** evrardjp has quit IRC | 05:33 | |
| *** evrardjp has joined #openstack-ansible | 05:33 | |
| *** lemko has quit IRC | 05:46 | |
| *** lemko7 has joined #openstack-ansible | 05:46 | |
| *** hamzaachi has quit IRC | 05:54 | |
| *** cshen has joined #openstack-ansible | 06:00 | |
| *** yasemind has joined #openstack-ansible | 06:01 | |
| yasemind | hi, i want to install trove (database as a service ) with OSA, which version I use it ? Which one is stable ? any idea ? | 06:04 |
| *** cshen has quit IRC | 06:04 | |
| *** klamath_atx has joined #openstack-ansible | 06:11 | |
| *** cshen has joined #openstack-ansible | 06:13 | |
| *** cshen has quit IRC | 06:17 | |
| *** dasp has joined #openstack-ansible | 07:00 | |
| *** dasp_ has quit IRC | 07:01 | |
| *** shyamb has joined #openstack-ansible | 07:07 | |
| *** cshen has joined #openstack-ansible | 07:07 | |
| *** hamzaachi has joined #openstack-ansible | 07:24 | |
| *** shyamb has quit IRC | 07:25 | |
| *** hamzaachi has quit IRC | 07:41 | |
| *** shyamb has joined #openstack-ansible | 07:50 | |
| noonedeadpunk | I think trove is in a bad shape atm but not sure - you can try it out | 08:00 |
| noonedeadpunk | I should have try out trove in January but more important stuff raised | 08:01 |
| *** miloa has joined #openstack-ansible | 08:01 | |
| *** jbadiapa has joined #openstack-ansible | 08:05 | |
| noonedeadpunk | I think there are 2 tricky things - you need a dedicated network and a bridge the same way is described for octavia, and access for instances to rabbit. So you will need either to attach this network to rabbit containers or deploy some independant rabbit for interaction with trove worker instances only | 08:05 |
| *** yasemind has quit IRC | 08:10 | |
| *** yasemind has joined #openstack-ansible | 08:13 | |
| *** andrewbonney has joined #openstack-ansible | 08:19 | |
| *** macz_ has joined #openstack-ansible | 08:23 | |
| *** macz_ has quit IRC | 08:27 | |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/victoria: Return centos-8 jobs to voting https://review.opendev.org/c/openstack/openstack-ansible/+/773393 | 08:33 |
| yasemind | noonedeadpunk Do I use master branch ? I think trove is challenging 😰 | 08:36 |
| noonedeadpunk | we haven't changed it in a while anyway | 08:37 |
| yasemind | maybe ussuri version is working | 08:38 |
| noonedeadpunk | well role is passing. but what it configures is really question | 08:38 |
| *** shyamb has quit IRC | 08:39 | |
| jrosser | morning | 08:39 |
| *** tosky has joined #openstack-ansible | 08:40 | |
| *** rpittau|afk is now known as rpittau | 08:41 | |
| CeeMac | o/ | 08:45 |
| *** maharg101 has joined #openstack-ansible | 08:45 | |
| noonedeadpunk | \o/ | 08:45 |
| *** ierdem has joined #openstack-ansible | 08:56 | |
| *** maharg101 has quit IRC | 08:57 | |
| ierdem | Hi everyone, I am encountering this error when I try to launch instance http://paste.openstack.org/show/802272/ . I checked NFS mounts permissions and owner of files is nova. I am using OSA-Stable/Ussuri on Ubuntu 18.04 | 08:58 |
| *** shyamb has joined #openstack-ansible | 09:01 | |
| jrosser | noonedeadpunk: your ansible-hardening patches failing on bionic is quite odd | 09:07 |
| jrosser | seems different things are enabled between bionic/focal | 09:08 |
| jrosser | these tasks run on bionic but are skipped on focal https://codesearch.opendev.org/?q=security_rhel7_remove_shosts_files | 09:08 |
| * jrosser looks suspiciously at the bool filter again..... | 09:09 | |
| jrosser | filter/test | 09:09 |
| jrosser | hrrm it's not that actually, its run multiple times somehow | 09:13 |
| *** yasemind has quit IRC | 09:15 | |
| openstackgerrit | Jonathan Rosser proposed openstack/ansible-hardening master: Use doc8 for documentation generation https://review.opendev.org/c/openstack/ansible-hardening/+/710284 | 09:20 |
| *** shyam89 has joined #openstack-ansible | 09:23 | |
| *** shyamb has quit IRC | 09:25 | |
| *** shyam89 has quit IRC | 09:27 | |
| *** yasemind has joined #openstack-ansible | 09:28 | |
| *** yasemind has quit IRC | 09:28 | |
| openstackgerrit | Merged openstack/ansible-hardening master: Fix linter errors https://review.opendev.org/c/openstack/ansible-hardening/+/771481 | 09:35 |
| openstackgerrit | Jonathan Rosser proposed openstack/ansible-hardening stable/victoria: Fix linter errors https://review.opendev.org/c/openstack/ansible-hardening/+/773672 | 09:37 |
| openstackgerrit | Jonathan Rosser proposed openstack/ansible-hardening stable/victoria: Update .gitreview for stable/victoria https://review.opendev.org/c/openstack/ansible-hardening/+/767766 | 09:38 |
| openstackgerrit | Jonathan Rosser proposed openstack/ansible-hardening stable/victoria: Update TOX_CONSTRAINTS_FILE for stable/victoria https://review.opendev.org/c/openstack/ansible-hardening/+/767767 | 09:38 |
| *** shyamb has joined #openstack-ansible | 09:41 | |
| noonedeadpunk | jrosser: I saw this happening on buster as well | 09:46 |
| ierdem | I am encountering this error when I try to launch instance http://paste.openstack.org/show/802272/ . I checked NFS mounts permissions and owner of files is nova. I am using OSA-Stable/Ussuri on Ubuntu 18.04. Can you help me please? | 09:46 |
| noonedeadpunk | I guess it just depends on fs of the CI region since find just timeouts there I guess.. | 09:47 |
| jrosser | what i was not really understanding is the same tasks appear to be run more that once in the test | 09:47 |
| jrosser | one skipped and one not | 09:47 |
| noonedeadpunk | as we technically should fail here https://zuul.opendev.org/t/openstack/build/52d64a59e7704613bb10401e23a6fa74/log/job-output.txt#3580 | 09:47 |
| noonedeadpunk | because it's run with check first time | 09:48 |
| jrosser | oooooh ok | 09:48 |
| noonedeadpunk | https://zuul.opendev.org/t/openstack/build/52d64a59e7704613bb10401e23a6fa74/log/job-output.txt#2199 | 09:48 |
| noonedeadpunk | I say it's run even 3 times - third one is idempotency | 09:48 |
| jrosser | the find is really broad https://opendev.org/openstack/ansible-hardening/src/branch/master/tasks/rhel7stig/async_tasks.yml#L38 | 09:50 |
| noonedeadpunk | yeah.... | 09:50 |
| jrosser | it's maybe not legitimate to be scanning the entire fs? | 09:50 |
| jrosser | stopping it going in /lib and /usr/lib or something may make it much quicker | 09:51 |
| noonedeadpunk | well yes | 09:51 |
| noonedeadpunk | agree | 09:51 |
| noonedeadpunk | as well as proc and sys | 09:51 |
| noonedeadpunk | where I think it might get hunged | 09:52 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-tests master: Run Centos 8 and Focal in gates https://review.opendev.org/c/openstack/openstack-ansible-tests/+/773862 | 09:54 |
| noonedeadpunk | I'm wondering how we've missed that ^ | 09:54 |
| jrosser | i'll see if theres something simple which reduces the runtime of that find | 09:54 |
| openstackgerrit | Merged openstack/openstack-ansible-lxc_container_create stable/victoria: Fix formatting of LXC container config files https://review.opendev.org/c/openstack/openstack-ansible-lxc_container_create/+/773416 | 09:57 |
| *** shyamb has quit IRC | 10:00 | |
| *** shyamb has joined #openstack-ansible | 10:02 | |
| ierdem | I solved http://paste.openstack.org/show/802272/ this problem. On nfs-server I mount nova path with this parameters (rw,sync,no_subtree_check) but it did not work. After I changed the parameters to (rw,sync,no_root_squash,fsid=0,subtree_check) these, it works now | 10:15 |
| CeeMac | is anyone else having problems with docs.openstack.org not loading? | 10:17 |
| CeeMac | or is it just me | 10:18 |
| andrewbonney | Same here. releases.openstack.org too | 10:19 |
| andrewbonney | Some chat in #opendev | 10:20 |
| CeeMac | hmm, i'm not in that channel, i'll add it to my idling list | 10:21 |
| frickler | I just restarted the apache2 there, should hopefully behave better now | 10:21 |
| andrewbonney | Thanks! | 10:27 |
| openstackgerrit | Andrew Bonney proposed openstack/openstack-ansible-openstack_hosts master: Add hostname resolution to deploy host https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/773868 | 10:30 |
| *** sshnaidm|afk is now known as sshnaidm|ruck | 10:35 | |
| noonedeadpunk | regarding venv_rebuild, I'm wondering if we can just drop this item https://opendev.org/openstack/ansible-role-python_venv_build/src/branch/master/tasks/python_venv_wheel_build.yml#L48 | 10:40 |
| noonedeadpunk | because condition for rebuild is if any of these files are changed https://opendev.org/openstack/ansible-role-python_venv_build/src/branch/master/tasks/python_venv_wheel_build.yml#L128 | 10:41 |
| * noonedeadpunk testing | 10:42 | |
| openstackgerrit | Jonathan Rosser proposed openstack/ansible-hardening master: Exclude system directories (/sys, /proc, /dev) from the shosts file search https://review.opendev.org/c/openstack/ansible-hardening/+/773869 | 10:43 |
| jrosser | noonedeadpunk: yes the wheels will always get rebuilt anyway as it's done in /tmp | 10:45 |
| jrosser | the results are copied to that dir afterwards | 10:45 |
| noonedeadpunk | and the second result is that venv will be dropped https://opendev.org/openstack/ansible-role-python_venv_build/src/branch/master/tasks/python_venv_install.yml#L35-L40 | 10:46 |
| noonedeadpunk | Am I missing smth again? | 10:46 |
| jrosser | we did talk about splitting the var into two | 10:47 |
| jrosser | because venv_rebuild != flush_wheel_cache != wheel_rebuild | 10:47 |
| noonedeadpunk | yeah, ok | 10:47 |
| noonedeadpunk | because you don't need to drop venv as wheels got updated | 10:48 |
| jrosser | if everything is working properly, yes | 10:48 |
| noonedeadpunk | another question - how appropriate that would be for backporting? | 10:48 |
| jrosser | right now it's hugely broken | 10:49 |
| noonedeadpunk | I mean dropping out just this item seems fine to backport | 10:49 |
| noonedeadpunk | and we can split vars for master? | 10:49 |
| noonedeadpunk | or just we should introduce new vars in train/ussuri? | 10:49 |
| jrosser | either is good i guess | 10:49 |
| jrosser | as it basically splits that task into two | 10:50 |
| jrosser | result both ways is that directory is no longer deleted | 10:50 |
| * noonedeadpunk goes to re-read bug report | 10:51 | |
| *** miloa has quit IRC | 11:31 | |
| *** shyamb has quit IRC | 11:33 | |
| *** shyamb has joined #openstack-ansible | 11:33 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_zun stable/victoria: Prevent zun-wsproxy binding to all IP addresses https://review.opendev.org/c/openstack/openstack-ansible-os_zun/+/772806 | 11:45 |
| *** pcaruana has quit IRC | 11:53 | |
| openstackgerrit | Jonathan Rosser proposed openstack/ansible-hardening master: Exclude system directories (/sys, /proc, /dev) from the shosts file search https://review.opendev.org/c/openstack/ansible-hardening/+/773869 | 11:55 |
| *** pcaruana has joined #openstack-ansible | 11:58 | |
| *** shyamb has quit IRC | 11:59 | |
| *** maharg101 has joined #openstack-ansible | 12:08 | |
| *** macz_ has joined #openstack-ansible | 12:20 | |
| *** macz_ has quit IRC | 12:24 | |
| *** MickyMan77 has joined #openstack-ansible | 12:48 | |
| *** SmearedBeard has joined #openstack-ansible | 12:55 | |
| *** MickyMan77 has quit IRC | 13:13 | |
| *** klamath_atx has quit IRC | 13:33 | |
| *** klamath_atx has joined #openstack-ansible | 13:41 | |
| *** zul has joined #openstack-ansible | 13:50 | |
| *** klamath_atx has quit IRC | 13:51 | |
| *** klamath_atx has joined #openstack-ansible | 13:58 | |
| *** klamath_atx has quit IRC | 14:03 | |
| *** klamath_atx has joined #openstack-ansible | 14:06 | |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_neutron master: Fix neutron_keepalived_no_track default logic https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/773916 | 14:23 |
| *** klamath_atx has quit IRC | 14:23 | |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_neutron master: Remove neutron_keepalived_no_track variable https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/770808 | 14:25 |
| *** akahat|rover is now known as akahat | 14:34 | |
| *** arxcruz is now known as arxcruz|ruck | 14:34 | |
| noonedeadpunk | this one we've catched on V with bionic so be aware in case you're not upgrading to 20.04 https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/773916 | 14:35 |
| admin0 | is there an automatic way to build and test an aio ? | 14:37 |
| openstackgerrit | Merged openstack/ansible-role-uwsgi master: [reno] Stop publishing release notes https://review.opendev.org/c/openstack/ansible-role-uwsgi/+/772059 | 14:37 |
| noonedeadpunk | ./scripts/gate-check-commit.sh $SCENARIO deploy source | 14:37 |
| admin0 | also in bootstrap (22.0.0) this fails: Clone git repos (parallel)] ... is that something just in my case or others see it also ? | 14:37 |
| noonedeadpunk | I saw it as well on bionic lately | 14:38 |
| noonedeadpunk | dunno it worked previousy and I know kind of the reason, but I think git wasn't updated though.... | 14:38 |
| admin0 | https://gist.github.com/a1git/99029ee964ae5f437206daf16b967142 -- this is what I see . i am on tag 22.0.0 , ubuntu 20.04 | 14:39 |
| noonedeadpunk | it's because of that https://opendev.org/openstack/openstack-ansible/src/branch/master/scripts/get-ansible-role-requirements.yml#L133 | 14:39 |
| noonedeadpunk | since force on newer git cli is not a supported option | 14:39 |
| noonedeadpunk | ah, it's different | 14:40 |
| noonedeadpunk | you need this https://review.opendev.org/c/openstack/openstack-ansible/+/773352 | 14:40 |
| admin0 | this is another one I see .. during bootstrap: https://gist.github.com/a1git/399b781599be7c490e8a5ab5450004ff | 14:41 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Bootstrap lxc_net mtu for gate https://review.opendev.org/c/openstack/openstack-ansible/+/557484 | 14:42 |
| noonedeadpunk | the same actually | 14:42 |
| Underknowledge | quick and dirty fix for the wrong commit SHA ' sed -i 's/54f7983be281f88714cbc75d393efa637e37ce17/bc1cc666e4b404a6baf04884b10cd3f79c4b9aea/g' /opt/openstack-ansible/ansible-role-requirements.yml' | 14:52 |
| *** d34dh0r53 has quit IRC | 15:01 | |
| *** d34dh0r53 has joined #openstack-ansible | 15:01 | |
| jrosser | it's not the wrong SHA :) | 15:03 |
| jrosser | Underknowledge: the way we make a point release is to 'bump' all those sha to the current tip of the relevant branch every two weeks and run a CI job, if that passes the new SHA become the ones in the openstack-ansible repo and are 'known good' | 15:07 |
| admin0 | how come it passes the CI job and not my setup ? | 15:08 |
| jrosser | over the last two weeks ceph-ansible have merged >10 patches to their stable branch which broke our shallow clone with depth=10 when fetching the roles | 15:08 |
| admin0 | or am i doing something wrong | 15:08 |
| jrosser | because on the day of the CI job that patch was the tip of the branch | 15:08 |
| admin0 | but aren't the commits ( hashes) like unchangable (past) items and anything added on top of it should not affect it ? | 15:09 |
| jrosser | read the commit message on the patch that noonedeadpunk pasted https://review.opendev.org/c/openstack/openstack-ansible/+/773352 | 15:09 |
| jrosser | we do shallow clones | 15:09 |
| admin0 | oh .. | 15:09 |
| admin0 | can't we just do specific clones ? | 15:10 |
| admin0 | like of the exact commit and not a depth | 15:10 |
| openstackgerrit | Merged openstack/openstack-ansible stable/train: Bump SHAs for stable/train https://review.opendev.org/c/openstack/openstack-ansible/+/773265 | 15:12 |
| * noonedeadpunk just finished first T->V upgrade | 15:13 | |
| jrosser | cool :) | 15:13 |
| *** klamath_atx has joined #openstack-ansible | 15:14 | |
| Underknowledge | Okok **currently non working SHA :D | 15:15 |
| jrosser | Underknowledge: did you manage to get further than before with your installation? | 15:16 |
| Underknowledge | a little, a lot of trail and error. the SCENARIO's are a little confusing, ended at the last try with aio_nspawn_barbican_ceph_designate_magnum wanted from me brtfs. | 15:19 |
| jrosser | hmmm nspawn | 15:21 |
| * jrosser needs to make that fail with a "don't use this" message | 15:21 | |
| Underknowledge | next try was with lxc ceph, where the nfs container apperantly didnt had a foler. (wich it had) didnt bortherd further and currently trying (probably) lxc_barbican_designate_magnum, lost the overview wich scenarios I took now 😄 | 15:22 |
| jrosser | just FYI the SCENARIO env var is primarily a CI construct that gets auto populated with the single service being tested | 15:22 |
| jrosser | if you want anything fancier than that then boostrap-aio.sh is probably not for you and generating config by hand is better | 15:23 |
| Underknowledge | reimage! here I go again 😛 | 15:25 |
| jrosser | aio with the default scenario is ideal for figuring out how everything is put together | 15:26 |
| jrosser | nspawn is basically not really usable and needs removing from the code | 15:26 |
| admin0 | jrosser, is there a variable to increase the depth ? | 15:41 |
| admin0 | Underknowledge, barbican works fine . i have it with octavia | 15:41 |
| jrosser | the way it is controlled is shown in that patch | 15:41 |
| admin0 | i see it . default_depth: 20 | 15:42 |
| openstackgerrit | Merged openstack/ansible-hardening master: Exclude system directories (/sys, /proc, /dev) from the shosts file search https://review.opendev.org/c/openstack/ansible-hardening/+/773869 | 15:52 |
| mgariepy | noonedeadpunk, T > V in one shot or you had to upgrade to U ? | 15:54 |
| openstackgerrit | Merged openstack/ansible-hardening stable/victoria: Fix linter errors https://review.opendev.org/c/openstack/ansible-hardening/+/773672 | 15:56 |
| openstackgerrit | Merged openstack/ansible-hardening stable/victoria: Update TOX_CONSTRAINTS_FILE for stable/victoria https://review.opendev.org/c/openstack/ansible-hardening/+/767767 | 15:56 |
| openstackgerrit | Merged openstack/ansible-hardening stable/victoria: Update .gitreview for stable/victoria https://review.opendev.org/c/openstack/ansible-hardening/+/767766 | 15:56 |
| noonedeadpunk | mgariepy: in one shot | 15:58 |
| mgariepy | on which os ? | 15:58 |
| noonedeadpunk | bionic | 15:58 |
| mgariepy | nice | 15:58 |
| mgariepy | any unexpected issue ? | 15:59 |
| noonedeadpunk | several bugs I've already pushed patches for but nothing severe | 16:00 |
| mgariepy | cool | 16:01 |
| *** macz_ has joined #openstack-ansible | 16:12 | |
| *** klamath_atx has quit IRC | 16:14 | |
| *** d34dh0r53 has quit IRC | 16:18 | |
| *** d34dh0r53 has joined #openstack-ansible | 16:19 | |
| *** SmearedBeard has quit IRC | 16:33 | |
| macz_ | Hi all, i'm running into an issue upgrading Pike > Queens | 16:37 |
| macz_ | running setup-hosts is changing the /etc/hosts file for servers, and it's causing services to re-connect with a new hostname UUID. primarily the nova-compute agents, which is causing a lot of issues. is there any way around the hosts name changes? | 16:38 |
| *** spatel has joined #openstack-ansible | 16:39 | |
| *** klamath_atx has joined #openstack-ansible | 16:40 | |
| *** sshnaidm|ruck is now known as sshnaidm | 16:52 | |
| noonedeadpunk | I can hardly recall tbh... things changed so dramatically since Q and P... | 17:03 |
| jrosser | i think the best place to start is the releasenotes because something that significant ought to be documented | 17:04 |
| noonedeadpunk | eventually I thought that it was same script that managed hosts file | 17:04 |
| jrosser | i have some memory of the hostnames previously not being valid dns names | 17:04 |
| jrosser | rfc<something-something-something> | 17:05 |
| noonedeadpunk | https://opendev.org/openstack/openstack-ansible-openstack_hosts/src/branch/stable/queens/templates/openstack-host-hostfile-setup.sh.j2 | 17:05 |
| jrosser | oh that does change subtly P->Q | 17:06 |
| noonedeadpunk | there were 2 changes https://opendev.org/openstack/openstack-ansible-openstack_hosts/commit/32e44dc7bc8b977049ca887237c136b3ca98d757 and https://opendev.org/openstack/openstack-ansible-openstack_hosts/commit/62e039bd0d43e31d5f0aaba0ec6c99f8f3d4310a | 17:07 |
| noonedeadpunk | so second one seems like related | 17:08 |
| *** ierdem has quit IRC | 17:13 | |
| *** spatel has quit IRC | 17:14 | |
| macz_ | thanks, yeah i found that change too. I didn't see anything in the queens release notes about the hosts change though | 17:15 |
| noonedeadpunk | it was backported to queens:( | 17:16 |
| noonedeadpunk | I guess that's why there're no notes | 17:16 |
| *** spatel has joined #openstack-ansible | 17:16 | |
| macz_ | oof :( | 17:18 |
| noonedeadpunk | but I don't have any valid solution I guess | 17:18 |
| noonedeadpunk | I'm wondering why this changed so badly since there're conditions defined, when "${ANSHOSTNAME}" != "${RFCHOSTNAME}" | 17:20 |
| openstackgerrit | Merged openstack/openstack-ansible-lxc_hosts master: Fix lxc_hosts_container_image_url condition https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/773781 | 17:24 |
| macz_ | dang. any hacky workaround? like set the commit to openstack-host-hostfile-setup.sh.j2 to one earlier? | 17:28 |
| noonedeadpunk | it will be adjusted later | 17:28 |
| noonedeadpunk | can you please paste error or how does openstack compute service list look like? | 17:29 |
| noonedeadpunk | because it's hard to understand what exactly went wrong | 17:29 |
| noonedeadpunk | ofc you can just manually place required lines in hosts files at the end of the day... | 17:29 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible-lxc_hosts stable/victoria: Fix lxc_hosts_container_image_url condition https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/773914 | 17:31 |
| *** klamath_atx has quit IRC | 17:37 | |
| Underknowledge | @admin0 you mean the AiO scenario? | 17:39 |
| macz_ | i've done a rebuild so i dont have it live to grab a service list, but for instance nova agent hostnames changed from compute1.openstack.local to cmpt-1 after the run, and nova logs showed: | 17:39 |
| macz_ | "Error updating resources for node compute1.openstack.local.: ResourceProviderCreationFailed: Failed to create resource provider compute1.openstack.local" | 17:39 |
| noonedeadpunk | well I think how ansible see hostnames depends on the order of names in /etc/hosts for 127.0.1.1 | 17:41 |
| noonedeadpunk | on compute nodes | 17:41 |
| noonedeadpunk | no maybe changing that will preserver behaviour in hosts files | 17:41 |
| spatel | noonedeadpunk i have noticed very strange issue not all container mounting their journal to host /var/log/journal/ | 17:42 |
| spatel | in my case only senlin and repo has /var/log/journal mounted but all other doesn't | 17:42 |
| spatel | inside container i found they don't have /var/log/journal directory | 17:43 |
| spatel | so default they are using /run/log/journal | 17:43 |
| spatel | are you aware of that issue? | 17:43 |
| noonedeadpunk | you can verify how ansible will see hostnames with python cli, like python -c "import socket; print(socket.getfqdn)" or sdocket.gethostname | 17:43 |
| noonedeadpunk | spatel: I think andrewbonney has made a patch for that | 17:44 |
| spatel | is that patch part of victoria or under review? | 17:44 |
| noonedeadpunk | or maybe it was other issue | 17:44 |
| spatel | i think we have different issue, i tried my best way to fix it in lab but it doesn't working. I have created /var/log/journal dir on container but still its not working | 17:46 |
| spatel | can you see logs in your deployment on host ? | 17:46 |
| jrosser | spatel: do you see the dirtectories on the host with the machine-id? | 17:46 |
| spatel | http://paste.openstack.org/show/802287/ | 17:46 |
| jrosser | but inside those directories the content is stale or frozen? | 17:47 |
| spatel | look at that each 8.1M is container not able to mount | 17:47 |
| openstackgerrit | Merged openstack/openstack-ansible-os_nova master: Fix nova_libvirt_images_rbd_pool check https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/773732 | 17:47 |
| openstackgerrit | Merged openstack/openstack-ansible-os_senlin master: Remove unused variables from defaults https://review.opendev.org/c/openstack/openstack-ansible-os_senlin/+/771258 | 17:47 |
| noonedeadpunk | I just checked my current sandbox and it has /var/log/journal everywhere... | 17:47 |
| jrosser | the failure case which andrewbonney patched is that all the directories are present and so are the journal files | 17:47 |
| jrosser | but the files are no longer being written to from inside the container | 17:48 |
| spatel | journalctl --file /var/log/journal/eb6f151cd5ee4fa1b1bf5c9801668efc/system.journal -f (this is showing all host journal content, they don't have application logs) | 17:48 |
| jrosser | so you get a different view inside/outside the container | 17:48 |
| spatel | jrosser let me show you something.. hang on | 17:48 |
| noonedeadpunk | well I don't see this happening for me right now... | 17:50 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible-os_nova stable/victoria: Fix nova_libvirt_images_rbd_pool check https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/773915 | 17:51 |
| noonedeadpunk | neither for my upgraded deployment | 17:51 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible-os_senlin stable/victoria: Remove unused variables from defaults https://review.opendev.org/c/openstack/openstack-ansible-os_senlin/+/773956 | 17:51 |
| noonedeadpunk | so I'd say no, I'm not aware of this happening :( | 17:51 |
| spatel | jrosser check this out.. this is what going on - http://paste.openstack.org/show/802288/ | 17:53 |
| spatel | Majority of container i don't have /var/log/journal directory so how does it mount. | 17:55 |
| jrosser | someone else was here the other day asking about this | 17:55 |
| spatel | I tried to create that directory and stop/start container but still didn't mount that | 17:56 |
| jrosser | in particular the case of /run for where the journal is | 17:56 |
| spatel | systemd-journal default use /var/log/journal but if that directory doesn't exist then it use /run/log/journal/ | 17:56 |
| noonedeadpunk | http://paste.openstack.org/show/802289/ | 17:57 |
| spatel | as soon as i created /var/log/journal directory on container it automatically started putting logs in /var/log/journal (but not problem is i can't see that systemd.journal on host) | 17:57 |
| noonedeadpunk | I don't have /openstack/log/ mount though | 17:58 |
| jrosser | spatel: so on the host your journal stops at dec 18th? | 17:58 |
| *** maharg101 has quit IRC | 17:58 | |
| spatel | yes kind of | 17:59 |
| spatel | i am not seeing any new logs or update on that file | 18:00 |
| noonedeadpunk | spatel: how dous mount look like inside container? | 18:00 |
| jrosser | right, so on december 18th something happend, likley you restart the containers for some reason | 18:00 |
| jrosser | and at that point seems like two different things maybe in play | 18:00 |
| jrosser | possibly the aliasing of the journal directory mount | 18:00 |
| spatel | If i create foo file inside this directory then it should be visible on host also file? /var/log/journal/71b7cc49bbf342749eca8faa2ffc81b0/. | 18:00 |
| jrosser | and then the new journal file being in /run rather than /var | 18:00 |
| spatel | but that is not the case.. look like it didn't mount that directory | 18:00 |
| jrosser | if the dir is correctly bind mounted yes | 18:01 |
| spatel | look at this error mesg - http://paste.openstack.org/show/802290/ | 18:01 |
| admin0 | Underknowledge, i meant on production | 18:02 |
| spatel | something else going on.. | 18:02 |
| jrosser | you may be the first trying this for real on a focal host | 18:03 |
| noonedeadpunk | spatel: well order of mount seems wrong, but that is an issue for a while | 18:03 |
| noonedeadpunk | (which should be covered in V though) | 18:03 |
| spatel | hmm | 18:03 |
| spatel | for testing i did this - lxc.mount.entry = /var/log/foo var/log/foo none bind,create=dir 0 0 | 18:03 |
| spatel | it works, i can create file and they are visible on host | 18:04 |
| spatel | but not working for /var/log/journal case | 18:04 |
| noonedeadpunk | so aio having it set like this lxc.mount.entry = /home/zuul/src openstack/src none bind,create=dir 0 0 | 18:04 |
| noonedeadpunk | * http://paste.openstack.org/show/802291/ | 18:04 |
| Underknowledge | Would have wonderd otherwise, I think I tried that already. Barbican is also enabled on our production server. But no one uses it basically :D | 18:05 |
| noonedeadpunk | that means you don't have octavia yet :p | 18:05 |
| spatel | noonedeadpunk you are saying i should move var/log to second position? | 18:07 |
| Underknowledge | Sherlock! True | 18:07 |
| spatel | let me try and see | 18:07 |
| noonedeadpunk | spatel: I guess that shoud mount at least these two correctly https://review.opendev.org/c/openstack/openstack-ansible/+/771205 | 18:08 |
| *** rpittau is now known as rpittau|afk | 18:08 | |
| noonedeadpunk | but it's not in 22.0.0 afaik | 18:08 |
| spatel | noonedeadpunk that works!! | 18:10 |
| spatel | when we will have that patch on victoria :) | 18:11 |
| spatel | look like very important piece of code | 18:11 |
| noonedeadpunk | it's in stable/victoria | 18:11 |
| spatel | I am running 22.0.0.0rc1 | 18:12 |
| noonedeadpunk | will be included in 22.0.1 as well as more pretty nasty bug fixes | 18:12 |
| spatel | why its hard to backport single line code? | 18:13 |
| spatel | This is just order issue right not something impact functionality | 18:14 |
| jrosser | it already is backported | 18:14 |
| jrosser | look in the link, the branch is stable/victoria | 18:14 |
| spatel | I can see it :) - https://review.opendev.org/q/project:openstack%252Fopenstack-ansible+branch:stable%252Fvictoria+status:merged | 18:15 |
| spatel | Thank you! | 18:16 |
| noonedeadpunk | can I summon somebody for another vote on https://review.opendev.org/c/openstack/openstack-ansible/+/773377 ? | 18:16 |
| spatel | jrosser i found my issue related journalbeat, not sure you have noticed my message yesterday or not | 18:17 |
| noonedeadpunk | mgariepy: if you're around :) | 18:17 |
| spatel | it was add_observer_metadata in journalbeat.yml file | 18:17 |
| spatel | after that setting i can see original hostname of journal logs | 18:17 |
| jrosser | spatel: do you think we should add that as a default to the journalbeat.yml template in the ops repo? | 18:17 |
| spatel | that would be great | 18:18 |
| jrosser | is that what you've used to deploy it? | 18:18 |
| spatel | default journal use localhost name but with that setting it preserve remote hostname | 18:18 |
| spatel | Yes i used that option with my deployment | 18:18 |
| spatel | here - http://paste.openstack.org/show/802292/ | 18:19 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_neutron master: Implement uWSGI for neutron-api https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/486156 | 18:20 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible-os_keystone master: Allow OIDCClaimDelimiter to be set in the apache config file https://review.opendev.org/c/openstack/openstack-ansible-os_keystone/+/773978 | 18:21 |
| *** d34dh0r53 has quit IRC | 18:22 | |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_neutron master: Move neutron pip packages from constraints to requirements https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/770276 | 18:22 |
| *** d34dh0r53 has joined #openstack-ansible | 18:24 | |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_neutron master: Switch neutron functional jobs to focal https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/773979 | 18:27 |
| jrosser | noonedeadpunk: ^ feels like we are really really close to dropping those | 18:27 |
| noonedeadpunk | in favor of integrated? | 18:28 |
| jrosser | the OVN integrated repo job already deploys OVS, so it's like a cut down version of that to do an OVS job | 18:28 |
| noonedeadpunk | yeah, might be worth it... | 18:28 |
| jrosser | then we can lose the rest of those | 18:28 |
| *** MickyMan77 has joined #openstack-ansible | 18:28 | |
| noonedeadpunk | I eventually can recall them failing dramatically | 18:29 |
| noonedeadpunk | so decided to see the current state | 18:29 |
| jrosser | yeah, i was asking jamesdenton the other day if he had time to look at this | 18:30 |
| noonedeadpunk | would be great if he could also take a look at https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/762729 | 18:31 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/ansible-role-python_venv_build master: Do not drop all wheels with venv_rebuild https://review.opendev.org/c/openstack/ansible-role-python_venv_build/+/773981 | 18:38 |
| *** MickyMan77 has quit IRC | 18:40 | |
| spatel | noonedeadpunk are we going to continue support of centos-8? | 18:48 |
| spatel | if not then we don't need that patch for Add centos-8 support for ovs-dpdk | 18:48 |
| noonedeadpunk | dunno atm | 18:49 |
| spatel | centos-8 will be dead after few month so we may start looking at RHEL or CentOS-Stream | 18:50 |
| noonedeadpunk | maybe it will make sense to use cloudlinux fork of it or dunno | 18:50 |
| noonedeadpunk | or centos stream... | 18:51 |
| spatel | Now we can add RHEL in our CI job because now Redhat giving 15 server license free of cost | 18:51 |
| noonedeadpunk | I don;t think we can | 18:51 |
| noonedeadpunk | it means that somebody should issue the license and resource amount will be pretty limited | 18:51 |
| noonedeadpunk | it would depend on provider and so on so force | 18:52 |
| spatel | 15 server lic is free for RHEL but anyway we can use CentOS Stream or may be rockylinux | 18:52 |
| spatel | noonedeadpunk we have one more bug, i think few days ago i had issue where nova or neutron was failing when i was trying to add fresh new compute node. me and jrosser found something to wanted to verify with you | 18:57 |
| spatel | when i run openstack-ansible os-nova-install.yml -e venv_rebuild=yes then it whipping out /var/www/repo/os-releases/22.0.0.0b2.dev56/ubuntu-20.04-x86_64/ directory | 18:57 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/ansible-role-python_venv_build master: Split venv_rebuild functionality https://review.opendev.org/c/openstack/ansible-role-python_venv_build/+/773984 | 18:57 |
| noonedeadpunk | jrosser: does it correlates with what you was thinking about? ^ | 18:58 |
| noonedeadpunk | spatel: just posted https://review.opendev.org/c/openstack/ansible-role-python_venv_build/+/773981 that should cover this I guess | 18:58 |
| noonedeadpunk | if my assimptions are correct | 18:58 |
| spatel | :) | 18:58 |
| spatel | damn you are ahead of me | 18:58 |
| noonedeadpunk | at least in aio that helped me solving the issue | 18:58 |
| spatel | let me understand.. this patch won't whip out whole directory ? /var/www/repo/os-releases/22.0.0.0b2.dev56/ubuntu-20.04-x86_64/ | 19:00 |
| spatel | problem is when i use os-nova-install.yml then its removing neutron wheel files and when next time i run neutron it failed because wheel file is missing | 19:01 |
| spatel | but anyway i can test this patch on my production where i am having that behavior | 19:02 |
| jrosser | noonedeadpunk: i think if we delete the wheels directory we have to delete all of the .txt files, not just for the current role | 19:03 |
| noonedeadpunk | I'd say we shouldn't delete it at all? | 19:05 |
| jrosser | there perhaps is no good reason to have a flag to remove the wheel cache | 19:05 |
| noonedeadpunk | yeah I have no idea why this would be needed in any scenario | 19:06 |
| jrosser | odyssey4me: don't know if you are around but we are having issues with the behaviour of venv_rebuild=true | 19:06 |
| noonedeadpunk | but what is might be needed to trigger wheels re-build for specific service | 19:06 |
| noonedeadpunk | and then removal of requirements and constraints files should be more then enough | 19:07 |
| jrosser | indeed | 19:07 |
| noonedeadpunk | to build new wheels and place them isntead of already present ones | 19:07 |
| *** MickyMan77 has joined #openstack-ansible | 19:07 | |
| jrosser | they will get built in /tmp and copied to /var/www/blah | 19:07 |
| noonedeadpunk | yeah so the fix shoudl be as much as https://review.opendev.org/c/openstack/ansible-role-python_venv_build/+/773981/1/tasks/python_venv_wheel_build.yml imo | 19:08 |
| noonedeadpunk | and if we want to split behaviour of the venv and wheel rebuild then we need to introduce another var | 19:08 |
| spatel | why are we putting all wheels role in single directory? can't we create sub-directory based on roles? | 19:10 |
| jrosser | there is a whole ton of common stuff | 19:10 |
| noonedeadpunk | another question why do we use shell here https://opendev.org/openstack/ansible-role-python_venv_build/src/branch/master/tasks/python_venv_wheel_build.yml#L158 | 19:10 |
| noonedeadpunk | I'd say we could use syncronize as well... | 19:12 |
| noonedeadpunk | but whatever | 19:12 |
| jrosser | i expect it moves many files efficiently | 19:12 |
| jrosser | synchronise was always not good with delegate iirc | 19:12 |
| noonedeadpunk | ah, indeed | 19:13 |
| noonedeadpunk | with delegated host it changes soure | 19:13 |
| noonedeadpunk | or not... | 19:14 |
| noonedeadpunk | what's the difference? https://github.com/ansible-collections/ansible.posix/blob/main/plugins/modules/synchronize.py#L268-L278 | 19:14 |
| *** andrewbonney has quit IRC | 19:17 | |
| jrosser | this could always be historical in older ansible | 19:17 |
| spatel | jrosser in openstack-ansible-ops we have {{ elk_macros.beat_processors(processors) }} | 19:19 |
| spatel | that means we can override that option | 19:19 |
| jrosser | ahha | 19:19 |
| *** MickyMan77 has quit IRC | 19:20 | |
| odyssey4me | jrosser o/ can you summarise the issue? | 19:20 |
| noonedeadpunk | I think it's pretty much https://bugs.launchpad.net/openstack-ansible/+bug/1914301 | 19:21 |
| openstack | Launchpad bug 1914301 in openstack-ansible "passing venv_rebuild=true leaves repo server in unusable state" [Undecided,Confirmed] | 19:21 |
| noonedeadpunk | o/ | 19:21 |
| jrosser | o/ | 19:21 |
| spatel | now my logs showing correct source :) I like journalbeat - https://ibb.co/TRWsPYG | 19:21 |
| odyssey4me | haha, oh lol - that subject is great | 19:21 |
| noonedeadpunk | and my suggestion was like that https://review.opendev.org/c/openstack/ansible-role-python_venv_build/+/773981 | 19:21 |
| odyssey4me | yeah, that's legit IIRC | 19:22 |
| odyssey4me | hmm, let me think about that solution | 19:23 |
| jrosser | spatel: if you look at the logstash pipeline stuff you can make those req_id be first class fields in elasticsearch | 19:23 |
| jrosser | then you can query for them across all the services trivially | 19:24 |
| *** mmercer has joined #openstack-ansible | 19:24 | |
| odyssey4me | noonedeadpunk jrosser https://review.opendev.org/c/openstack/ansible-role-python_venv_build/+/773981 seems sensible, the only caveat is that if you find you need to rebuild a specific wheel you'll have to go in there and remove it manually... this is typically only important if you're building something from a local fork | 19:25 |
| spatel | totally possible but in my case i don't have logstash (its graylog but look like we know what is the workaround here) | 19:25 |
| noonedeadpunk | and won't wheels just be moved there with overwriting destination? https://opendev.org/openstack/ansible-role-python_venv_build/src/branch/master/tasks/python_venv_wheel_build.yml#L158 | 19:26 |
| odyssey4me | noonedeadpunk ah, fair point - I'd forgotten about that | 19:26 |
| noonedeadpunk | as we build wheels based on changed txt files? https://opendev.org/openstack/ansible-role-python_venv_build/src/branch/master/tasks/python_venv_wheel_build.yml#L128 | 19:27 |
| noonedeadpunk | I'm pretty much afraid that I'm missing some purpose tbh... | 19:27 |
| noonedeadpunk | or usecase... | 19:27 |
| *** gshippey has joined #openstack-ansible | 19:27 | |
| odyssey4me | noonedeadpunk no you're right, I think... it's been a long time since I worked with this - sorry | 19:27 |
| noonedeadpunk | yeah, I really forget about stuff I made half a year ago... | 19:28 |
| noonedeadpunk | thanks for chiming in! | 19:29 |
| spatel | we have workaround and everyone agreed right? that patch looks good so far | 19:35 |
| spatel | I can give it a try in my lab which is having same issue when i add new compute node :) | 19:35 |
| spatel | yesterday someone was asking about brbican-ui for horizon and i don't think we have that support. any reason we don't have that support or we never thought we need it ? | 19:47 |
| *** maharg101 has joined #openstack-ansible | 19:55 | |
| *** maharg101 has quit IRC | 20:00 | |
| *** cshen has quit IRC | 20:31 | |
| *** cshen has joined #openstack-ansible | 20:36 | |
| *** sshnaidm is now known as sshnaidm|afk | 20:52 | |
| spotz | spatel: My guess is it didn't exist when we added it. Maybe ask redrobot on the Barbican channel? I think he might have done the original work | 21:37 |
| *** MickyMan77 has joined #openstack-ansible | 21:44 | |
| *** MickyMan77 has quit IRC | 21:48 | |
| *** maharg101 has joined #openstack-ansible | 21:56 | |
| *** maharg101 has quit IRC | 22:01 | |
| *** openstackgerrit has quit IRC | 22:11 | |
| *** gshippey has quit IRC | 22:27 | |
| *** cshen has quit IRC | 22:58 | |
| spatel | spotz i will check with them, thank you | 23:06 |
| *** spatel has quit IRC | 23:06 | |
| *** jbadiapa has quit IRC | 23:20 | |
| *** tosky has quit IRC | 23:54 | |
| *** maharg101 has joined #openstack-ansible | 23:57 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!