| ianw | that's not true *either*. the seed pip in virtualenv in centos7 doesn't have the LE ISRG key. *but*, if you upgrade that pip, it brings in 20.3.4 which *does* have that bundled in certifi -- but centos7's 1.0.2 openssl doesn't understand the expiry | 00:18 |
|---|---|---|
| ianw | so it's still broken, just in a different way | 00:18 |
| opendevreview | Ian Wienand proposed openstack/openstack-ansible stable/stein: Workaround broken centos 7 pip https://review.opendev.org/c/openstack/openstack-ansible/+/814506 | 00:57 |
| opendevreview | Ian Wienand proposed openstack/openstack-ansible-os_masakari stable/stein: Remove debian-stable job https://review.opendev.org/c/openstack/openstack-ansible-os_masakari/+/814345 | 00:59 |
| opendevreview | Ian Wienand proposed openstack/openstack-ansible stable/stein: Workaround broken centos 7 pip https://review.opendev.org/c/openstack/openstack-ansible/+/814506 | 01:26 |
| ianw | jrosser / johnsom : well, i tried, but it's getting a bit in-depth for me now. it's now failing somewhere in https://codesearch.opendev.org/?q=python_venv_build. what's your feeling on me force-merging the debian-stable removal changes so we can remove that node type? | 01:50 |
| ianw | it's only on stable/stein | 01:51 |
| ianw | i'm not sure how much excitement there is for fixing centos7 on that branch ... | 01:51 |
| ianw | noonedeadpunk: ^ sorry, not johnsom :) | 01:53 |
| johnsom | Ianw I was wondering what I had forgot. Lol | 01:57 |
| opendevreview | Ian Wienand proposed openstack/openstack-ansible stable/stein: Workaround broken centos 7 pip https://review.opendev.org/c/openstack/openstack-ansible/+/814506 | 02:19 |
| opendevreview | Ian Wienand proposed openstack/openstack-ansible stable/stein: Workaround broken centos 7 pip https://review.opendev.org/c/openstack/openstack-ansible/+/814506 | 04:06 |
| opendevreview | Ian Wienand proposed openstack/openstack-ansible-os_barbican stable/stein: Remove debian-stable job https://review.opendev.org/c/openstack/openstack-ansible-os_barbican/+/814337 | 04:57 |
| opendevreview | Ian Wienand proposed openstack/openstack-ansible-os_blazar stable/stein: Remove debian-stable job https://review.opendev.org/c/openstack/openstack-ansible-os_blazar/+/814339 | 04:58 |
| opendevreview | Ian Wienand proposed openstack/openstack-ansible-os_designate stable/stein: Remove debian-stable job https://review.opendev.org/c/openstack/openstack-ansible-os_designate/+/814341 | 04:59 |
| opendevreview | Ian Wienand proposed openstack/openstack-ansible-os_heat stable/stein: Remove debian-stable job https://review.opendev.org/c/openstack/openstack-ansible-os_heat/+/814342 | 05:00 |
| opendevreview | Ian Wienand proposed openstack/openstack-ansible-os_ironic stable/stein: Remove debian-stable job https://review.opendev.org/c/openstack/openstack-ansible-os_ironic/+/814343 | 05:00 |
| opendevreview | Ian Wienand proposed openstack/openstack-ansible-os_magnum stable/stein: Remove debian-stable job https://review.opendev.org/c/openstack/openstack-ansible-os_magnum/+/814344 | 05:00 |
| opendevreview | Ian Wienand proposed openstack/openstack-ansible-os_masakari stable/stein: Remove debian-stable job https://review.opendev.org/c/openstack/openstack-ansible-os_masakari/+/814345 | 05:00 |
| opendevreview | Ian Wienand proposed openstack/openstack-ansible-os_mistral stable/stein: Remove debian-stable job https://review.opendev.org/c/openstack/openstack-ansible-os_mistral/+/814346 | 05:01 |
| opendevreview | Ian Wienand proposed openstack/openstack-ansible-os_octavia stable/stein: Remove debian-stable job https://review.opendev.org/c/openstack/openstack-ansible-os_octavia/+/814347 | 05:01 |
| opendevreview | Ian Wienand proposed openstack/openstack-ansible-os_rally stable/stein: Remove debian-stable job https://review.opendev.org/c/openstack/openstack-ansible-os_rally/+/814348 | 05:02 |
| opendevreview | Ian Wienand proposed openstack/openstack-ansible-os_sahara stable/stein: Remove debian-stable jobs https://review.opendev.org/c/openstack/openstack-ansible-os_sahara/+/814349 | 05:02 |
| opendevreview | Ian Wienand proposed openstack/openstack-ansible-os_swift stable/stein: Remove debian-stable job https://review.opendev.org/c/openstack/openstack-ansible-os_swift/+/814350 | 05:02 |
| opendevreview | Ian Wienand proposed openstack/openstack-ansible-os_tacker stable/stein: Remove debian-stable job https://review.opendev.org/c/openstack/openstack-ansible-os_tacker/+/814351 | 05:02 |
| opendevreview | Ian Wienand proposed openstack/openstack-ansible-os_trove stable/stein: Remove debian-stable job https://review.opendev.org/c/openstack/openstack-ansible-os_trove/+/814338 | 05:03 |
| ianw | noonedeadpunk: ^ that got a few more passing. the remaining ones i still don't know. e.g. https://review.opendev.org/c/openstack/openstack-ansible-os_barbican/+/814337 is trying to install tempest-barbican from master which has dropped py2 support | 06:55 |
| ianw | i don't think it's worth chasing stuff like that imo | 06:55 |
| opendevreview | Ian Wienand proposed openstack/openstack-ansible-os_barbican stable/stein: Remove debian-stable job https://review.opendev.org/c/openstack/openstack-ansible-os_barbican/+/814337 | 07:50 |
| opendevreview | Ian Wienand proposed openstack/openstack-ansible-os_octavia stable/stein: Remove debian-stable job https://review.opendev.org/c/openstack/openstack-ansible-os_octavia/+/814347 | 07:52 |
| opendevreview | Ian Wienand proposed openstack/openstack-ansible-os_designate stable/stein: Remove debian-stable job https://review.opendev.org/c/openstack/openstack-ansible-os_designate/+/814341 | 07:55 |
| opendevreview | Ian Wienand proposed openstack/openstack-ansible-os_sahara stable/stein: Remove debian-stable jobs https://review.opendev.org/c/openstack/openstack-ansible-os_sahara/+/814349 | 07:57 |
| opendevreview | Ian Wienand proposed openstack/openstack-ansible-os_barbican stable/stein: Remove debian-stable job https://review.opendev.org/c/openstack/openstack-ansible-os_barbican/+/814337 | 07:58 |
| opendevreview | Ian Wienand proposed openstack/openstack-ansible-os_magnum stable/stein: Remove debian-stable job https://review.opendev.org/c/openstack/openstack-ansible-os_magnum/+/814344 | 08:00 |
| jrosser | i'll recheck this to see if its also an issue on ussuri https://review.opendev.org/c/openstack/openstack-ansible/+/803926 | 08:02 |
| jrosser | it looks like we were still python2 in the ansible virtualenv for U https://github.com/openstack/openstack-ansible/blob/stable/ussuri/scripts/bootstrap-ansible.sh#L77 | 08:04 |
| opendevreview | chandan kumar proposed openstack/openstack-ansible-os_tempest master: Use single var file for rpm based distros https://review.opendev.org/c/openstack/openstack-ansible-os_tempest/+/814526 | 08:05 |
| ianw | jrosser: i would say though that ubuntu ships a much later pip in the default virtualenv though | 08:11 |
| ianw | centos7 has pip 9, so the cacert.pem it vendors doesn't have the right certificates for LE now | 08:11 |
| jrosser | yeah, this is all a centos mess | 08:11 |
| ianw | ++ | 08:12 |
| jrosser | need to think about it a bit,as theres fixing CI and then making it do the right thing on a real multinode production deployment | 08:12 |
| ianw | yep :) all i *really* want to do is get rid of debian-stable though | 08:13 |
| jrosser | i know :) | 08:13 |
| ianw | a typical yak-shaving adventure | 08:13 |
| jrosser | i'll have a mess with it today | 08:14 |
| ianw | thanks | 08:14 |
| * ianw day is over now :) | 08:14 | |
| jrosser | thanks for digging into this - appreciated | 08:14 |
| noonedeadpunk | actually - I saw the same hapenning in my ubuntu 18.04 deployments on V | 08:46 |
| noonedeadpunk | but what helped us was upgrading of certifi iirc | 08:46 |
| jrosser | via pip? | 08:46 |
| noonedeadpunk | yep | 08:46 |
| noonedeadpunk | but the thing is that it is constrainted | 08:46 |
| jrosser | yeah so thats in an existing deployment | 08:47 |
| noonedeadpunk | I believe this is smth my team looking into how to fix better now... | 08:47 |
| jrosser | ah ok i was just going to spin a centos7 aio and look specifically at the centos part | 08:47 |
| noonedeadpunk | well - we were doing upgrade from T->V in one of the regions | 08:47 |
| jrosser | but yes existing deployments will be in trouble | 08:48 |
| noonedeadpunk | I think even new ones are... | 08:48 |
| noonedeadpunk | because ce-certificates were latest version but had no effect, until certifi got updated | 08:48 |
| jrosser | is is that u-c holds it back? | 08:49 |
| noonedeadpunk | and it was constrainted to the version that does not have new LE root | 08:49 |
| noonedeadpunk | yeah | 08:49 |
| jrosser | thats uncool - as a result of a pin on requests or smt? | 08:49 |
| noonedeadpunk | It's a separate line https://opendev.org/openstack/requirements/src/branch/stable/victoria/upper-constraints.txt#L39 | 08:50 |
| noonedeadpunk | I guess we should just suggest patch to bump it | 08:50 |
| jrosser | oh wow, if theres one thing you don't want to pin, it's that | 08:50 |
| noonedeadpunk | yeah... | 08:50 |
| noonedeadpunk | prometheanfire: your opinion on pinning certifi?:) | 08:51 |
| opendevreview | chandan kumar proposed openstack/openstack-ansible-os_tempest master: Use single var file for rpm based distros https://review.opendev.org/c/openstack/openstack-ansible-os_tempest/+/814526 | 09:03 |
| opendevreview | chandan kumar proposed openstack/openstack-ansible-os_tempest master: Added var file for EL9 https://review.opendev.org/c/openstack/openstack-ansible-os_tempest/+/814526 | 09:08 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_tempest stable/stein: Fix tempest plugin versions https://review.opendev.org/c/openstack/openstack-ansible-os_tempest/+/814535 | 09:23 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_barbican stable/stein: Remove debian-stable job https://review.opendev.org/c/openstack/openstack-ansible-os_barbican/+/814337 | 09:24 |
| jrosser | noonedeadpunk: this will fix the ansible-runtime venv build for CI and real world https://paste.opendev.org/show/810071/ | 09:58 |
| noonedeadpunk | Um, there should be some filtering as well? | 10:00 |
| noonedeadpunk | u-c feels to me like some burden atm.... | 10:00 |
| noonedeadpunk | ah | 10:02 |
| noonedeadpunk | I got it now... | 10:02 |
| noonedeadpunk | hm, now I start wondering, why we saw issues only during bootstrap... | 10:03 |
| jrosser | read back overnight | 10:03 |
| jrosser | tldr is old pip has bundled certs pem which is not up to date | 10:03 |
| jrosser | so it fails to get the u-c url content with pip internals/requests, which is a precursor to updating pip/setuptools/wheel | 10:04 |
| noonedeadpunk | um, yeah, I got that | 10:05 |
| noonedeadpunk | I was more thinking about what I've seen.... | 10:06 |
| noonedeadpunk | I mean - we saw issues during bootstrap on ubuntu - actually same issue as here with centos | 10:06 |
| noonedeadpunk | but we never saw things failing with same constraints in python_venv_bild | 10:06 |
| jrosser | well i think there are multiple things here | 10:07 |
| noonedeadpunk | and ansible not failing with that certifi version... | 10:07 |
| jrosser | i have a feeling that centos7 is going to break on python_venv_build similarly | 10:08 |
| noonedeadpunk | yeah, at least that's what ianw said... | 10:09 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_tempest stable/stein: Fix tempest plugin versions https://review.opendev.org/c/openstack/openstack-ansible-os_tempest/+/814535 | 10:24 |
| ierdem | Hi everyone, we are trying to deploy OSA Victoria. we use proxy on both infrastructure and compute hosts. While running setup-hosts.yaml, it stuckes on TASK [lxc_hosts : Wait for base image download] task. We set the proxy settings in user_variables but problem still exist. Do u have any ideas? Thanks | 11:02 |
| opendevreview | Merged openstack/openstack-ansible-os_tempest stable/victoria: Pin neutron-tempest-plugin to v1.6.0 https://review.opendev.org/c/openstack/openstack-ansible-os_tempest/+/814258 | 11:13 |
| jrosser | ierdem: can you see the url it is trying to fetch? putting some logs at paste.opendev.org is helpful if you have the | 11:22 |
| jrosser | m | 11:22 |
| ierdem | jrosser, thanks for your answer, i disabled/enabled proxy and after that it works :) | 11:22 |
| ierdem | jrosser, new error occured :/ in setup-infrastruce.yml https://paste.opendev.org/show/810074/ cannot update apt-cache on galera | 11:27 |
| noonedeadpunk | Hm, I do see release file here... https://downloads.mariadb.com/MariaDB/mariadb-10.5.8/repo/ubuntu/dists/focal/ | 11:34 |
| noonedeadpunk | maybe gpg has changed... | 11:34 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/stein: Remove tempest plugins CI overrides https://review.opendev.org/c/openstack/openstack-ansible/+/814558 | 11:35 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_barbican stable/stein: Remove debian-stable job https://review.opendev.org/c/openstack/openstack-ansible-os_barbican/+/814337 | 11:35 |
| ierdem | I manually updated cache, it skipped that step but after https://paste.opendev.org/show/810075/ | 11:36 |
| noonedeadpunk | um, but might be that repo jsut got disabled? | 11:37 |
| opendevreview | Jonathan Rosser proposed openstack/ansible-role-python_venv_build stable/stein: Workaround distro provided pip having old CA certs on centos-7 https://review.opendev.org/c/openstack/ansible-role-python_venv_build/+/814559 | 11:38 |
| ierdem | noonedeadpunk, so should I install mariadb manually ? | 11:39 |
| noonedeadpunk | um, no | 11:39 |
| noonedeadpunk | you should check why repo is not being used | 11:39 |
| noonedeadpunk | or why it got disabled | 11:40 |
| ierdem | may apt-secure cause this problem? | 11:41 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible stable/stein: Fetch upper constraints file with curl rather than allow pip to download it https://review.opendev.org/c/openstack/openstack-ansible/+/814560 | 11:43 |
| noonedeadpunk | it should not on it's own | 11:44 |
| noonedeadpunk | honestly it's hard to say without being able to check some things | 11:45 |
| noonedeadpunk | like if installed gpg is valid and matching | 11:45 |
| noonedeadpunk | if gpg is present at all | 11:45 |
| MrClayPole | Hi, I'm attempting to use LetsEncrypt certificate for haproxy on OSA Train. It appears that the install method via https://dl.eff.org/certbot-auto has been deprecated/not available and this causes "TASK [haproxy_server : Download certbot]" to fail. I can see this has been fixed in Ussuri but is there a way to get this fixed/workaround in Train as I won't be able to upgrade for a while? | 11:52 |
| noonedeadpunk | MrClayPole: Looking at patch, I'm not sure we can backport it easily https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/722421 | 12:11 |
| noonedeadpunk | what you can probably try doing - use ussuri version of the role | 12:12 |
| noonedeadpunk | because there we a lot of changes to it that are tighten together... | 12:12 |
| MrClayPole | Yeah, I was looking myself. Looks like a pretty big change as it adds the option to use the disto package from what I can tell. We have a cert covering us until May 2022 so as long as we can upgrade before then I'll be good. I was just checking it there was an easy way round this issue. | 12:13 |
| MrClayPole | As it looks like there isn't then I'll park this until I can upgrade to at least Ussuri | 12:14 |
| MrClayPole | Thanks for you help noonedeadpunk | 12:18 |
| noonedeadpunk | I think you can just checkout haproxy role to ussuri | 12:20 |
| noonedeadpunk | I don't expect there to be some incompatabilities with ansible version or anything | 12:20 |
| noonedeadpunk | and then upgrade would be easier as well, since you will have all variables set in "new way: | 12:21 |
| mgariepy | anyone tried ceph multi-attach for rbd volumes ? | 12:21 |
| noonedeadpunk | I did and it was broken | 12:22 |
| noonedeadpunk | nova said just - don't use that :) | 12:22 |
| noonedeadpunk | (or it was cinder folks..) | 12:22 |
| noonedeadpunk | but it was broken in multiple pretty obvious places... | 12:22 |
| noonedeadpunk | like manila is what you actually need anyway | 12:23 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-memcached_server master: [doc] Fix haproxy_extra_services layout https://review.opendev.org/c/openstack/openstack-ansible-memcached_server/+/805996 | 12:24 |
| mgariepy | what i need is the user to rewrite his software.. but that's another issue. | 12:29 |
| mgariepy | For manilla i guess you do 2 sepearate ceph cluster on for rbd / s3 and one for manilla ? | 12:31 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_tempest stable/stein: Fix tempest plugin versions https://review.opendev.org/c/openstack/openstack-ansible-os_tempest/+/814535 | 12:36 |
| noonedeadpunk | um, I don't think ita a requirement | 12:37 |
| noonedeadpunk | as for manila you generally need just mds | 12:37 |
| noonedeadpunk | (and I guess ganesha-nfs depending on setup) | 12:38 |
| noonedeadpunk | there're challenges though regarding ganesha to work in HA setup | 12:38 |
| mgariepy | yeah i guess it it could be. | 12:38 |
| mgariepy | have you setup manilla ? | 12:39 |
| noonedeadpunk | that what you want is manila - I was told on the question why not to fix multiattach | 12:39 |
| mgariepy | do you have some number on what type of hw and perf you sqeeze out of it ? | 12:39 |
| noonedeadpunk | well, for test only and have some non-production setup with generic driver as well | 12:39 |
| noonedeadpunk | always wanted to add it to production but not yet( | 12:40 |
| mgariepy | ok | 12:41 |
| mgariepy | Guests need IP connectivity to Ceph cluster.. | 12:43 |
| noonedeadpunk | they don't if you use ganesha) | 12:43 |
| mgariepy | i do not like that.. | 12:43 |
| noonedeadpunk | ganesha acts like proxy | 12:44 |
| noonedeadpunk | so it provides nfs interface to cephfs | 12:44 |
| mgariepy | ganasha will probably endup to be a single point of failure and bottleneck for perfs | 12:44 |
| noonedeadpunk | it does, yes... | 12:44 |
| jrosser | maybe Y release will have new enough libvirt for virtiofs stuff to become real | 12:45 |
| noonedeadpunk | well, there're workarounds I was pointed to | 12:45 |
| jrosser | then it all become much more sane | 12:45 |
| noonedeadpunk | that would be awesome... But I haven't checked if it has ever landed to libvirt at all? | 12:45 |
| jrosser | i think you need 6.6 or later | 12:45 |
| noonedeadpunk | ah | 12:45 |
| jrosser | we were looking at it for something else and were sad to see it not there in 6.0 | 12:46 |
| noonedeadpunk | that would be awesome indeed | 12:46 |
| jrosser | we wanted to give guests a view of their energy consumption inside their VM https://hubblo-org.github.io/scaphandre-documentation/how-to_guides/propagate-metrics-hypervisor-to-vm_qemu-kvm.html | 12:47 |
| noonedeadpunk | But there's a lot of work on nova/manila side as well... | 12:47 |
| noonedeadpunk | I guess nova-compute won't like these changes... | 12:49 |
| noonedeadpunk | And would just drop them? | 12:49 |
| noonedeadpunk | Or there's a way to make them persistant and respected? | 12:49 |
| jrosser | not sure really, didnt look very hard | 12:50 |
| tbarron | most of the work will be in nova: https://review.opendev.org/c/openstack/nova-specs/+/813180 | 12:50 |
| tbarron | gonna take 2-3 cycles though | 12:50 |
| noonedeadpunk | and manila will act like cyborg does kind of? | 12:50 |
| noonedeadpunk | but yeah, I guess spec if pretty self-explasnatory | 12:51 |
| tbarron | umm, compute nodes will just stage remote mounts using manila and then supply them to guests via virtiofs | 12:51 |
| noonedeadpunk | yeah, there're tons of work... | 12:52 |
| tbarron | on the ganesha front, ceph cluster will run ganesha itself (ceph orchestrator backed by cephadm), do its own nfs ha, and put an "ingress" in front of a set of ganeshas (a ganesha cluster) | 12:53 |
| noonedeadpunk | is it already in stable? or it's wip? | 12:54 |
| tbarron | instead of openstack being responsible for running haproxy/pacemaker/etc. and controlling lifecycle of ganesha daemoons | 12:54 |
| tbarron | wip | 12:55 |
| noonedeadpunk | aha | 12:55 |
| tbarron | well, ceph pacific has the basics | 12:55 |
| tbarron | but manila driver isn't using this yet | 12:55 |
| noonedeadpunk | mgariepy: I guess we can balance it with haproxy/keepalived as we run them anyway today | 12:55 |
| jrosser | never seen anyone brave enough to use the ceph orchestrator | 12:55 |
| noonedeadpunk | +1 | 12:55 |
| tbarron | hopefully Y cycle we'llget it wired up upstream | 12:55 |
| mgariepy | will there be an upgrade path to this if a deployment currently do some stuff over manilla then when upgrading to the new release with all those feature implemented ? | 12:56 |
| noonedeadpunk | but I guess modern ceph-ansible about to leverage it? | 12:56 |
| tbarron | jrosser: well, we're going to try to do upstream manila devstack with it real soon now; that's of course different than cutting over a production cluster | 12:56 |
| mgariepy | for my ceph cluster i do use ceph-ansible but not via osa.. | 12:56 |
| noonedeadpunk | https://github.com/ceph/ceph-ansible/blob/master/infrastructure-playbooks/cephadm.yml | 12:57 |
| tbarron | but from upstream dev perspective we have two promising paths for cephfs where you don't put guests directly on the ceph public network: virtiofs, and cephadm/orchestrator | 12:58 |
| tbarron | managed nfs | 12:58 |
| jrosser | virtiofs is super attractive becasue the compute nodes already have all the required connectivity to the ceph cluster today | 12:58 |
| jrosser | and we can keep our really total decoupling between the ceph and openstack deployments, which are really seperate entities | 12:59 |
| noonedeadpunk | yeah, that;s true | 13:00 |
| jrosser | i remeber we have OSA users where the openstack guys don't even get ssh onto the ceph mon hosts | 13:00 |
| jrosser | so supporting those very split responsibilities is important | 13:01 |
| tbarron | mgariepy: upgrade path is not fully solved. It may be easier for virtiofs. For nfs, I don't see a nondisruptive upgrade since the ganesha export ip | 13:03 |
| tbarron | moves from living in openstack to living in the ceph cluster | 13:03 |
| mgariepy | well yep i guess. it will be disruptive for the nfs :D | 13:04 |
| tbarron | fwiw tripleo is discussing the nfs deployment stuff with cephadm/orch in 6 minutes | 13:05 |
| noonedeadpunk | while there're lot of ppl here I want to use moment and ask for some reviews on https://review.opendev.org/c/openstack/openstack-ansible/+/813885 | 13:06 |
| noonedeadpunk | we have ganesha deployment for quite a while but I didn't manage to write haproxy balancing for it :( | 13:08 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: [doc] Fix netplan sample https://review.opendev.org/c/openstack/openstack-ansible/+/814579 | 13:26 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: [doc] Fix netplan sample https://review.opendev.org/c/openstack/openstack-ansible/+/814579 | 13:26 |
| * jrosser curses centos (again) | 13:44 | |
| mgariepy | how comes jrosser ? | 13:57 |
| mgariepy | it's enterprizy.. | 13:57 |
| jrosser | https://review.opendev.org/c/openstack/ansible-role-python_venv_build/+/814559 | 13:58 |
| jrosser | the letsecrypt root CA changes have caught up with python2 on centos-7 | 13:58 |
| mgariepy | arf. | 13:59 |
| mgariepy | stein is still tested? | 13:59 |
| noonedeadpunk | well, it's in EM, but I believe it's pretty widely used | 14:00 |
| noonedeadpunk | and we got bunch of patches there | 14:00 |
| jrosser | just starting my AIO from fresh as it's all gone wrong in CI | 14:00 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-ceph_client master: Ensure role not fail when mon_host is not part of ceph_extra_confs https://review.opendev.org/c/openstack/openstack-ansible-ceph_client/+/814586 | 14:08 |
| noonedeadpunk | and in aio it's fine? | 14:09 |
| noonedeadpunk | btw we also have tests repo failing on master | 14:09 |
| jrosser | it was fine in AIO | 14:10 |
| jrosser | but i did some hacking there to make it work, so just want to double check it with everything clean | 14:10 |
| noonedeadpunk | doh | 14:10 |
| jrosser | thankfully it's quick as it works/breaks for the utility container which is early | 14:11 |
| opendevreview | Merged openstack/openstack-ansible-haproxy_server master: Add option to use alernative CA server for certbot https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/814364 | 14:12 |
| mgariepy | i did an upgrade to stein last week and on python 2.7 it was breaking also. on ubuntu 18.04. | 14:13 |
| noonedeadpunk | yeah, root certs rotation is always tough thing... | 14:14 |
| mgariepy | it failed on creating the ansible-runtime.. | 14:15 |
| noonedeadpunk | same for me | 14:15 |
| noonedeadpunk | at it was because of certifi... but actually making pip to use system trust is maybe better solution | 14:15 |
| mgariepy | i did overwrote https>http.. | 14:18 |
| mgariepy | lol | 14:18 |
| ierdem | hi, my OSA victoria installation stucked at TASK [python_venv_build : Install python packages into the venv] step, setup-infrastrure playbook is running now. No errors, no warnings and no logs. Any ideas? | 14:27 |
| jrosser | threre will be a log file in /var/log/python_venv_build.log which is usually very helpful | 14:28 |
| prometheanfire | noonedeadpunk: pinning certifi? | 14:33 |
| noonedeadpunk | yeah, like https://opendev.org/openstack/requirements/src/branch/stable/victoria/upper-constraints.txt#L39 | 14:34 |
| prometheanfire | as a general matter of policy I'm against pinning (capping) libs, you always forget to uncap/pin and it prevents you from getting updates (security and otherwise) | 14:34 |
| noonedeadpunk | which does not contain correct root ca for LE | 14:34 |
| odyssey4me | hey folks - my OSA is a little rusty... I recall there was something we configured to stop the haproxy checks from spamming the log files... can someone help me find where that was? | 14:34 |
| noonedeadpunk | afaik | 14:34 |
| prometheanfire | odyssey4me: ltns :D | 14:35 |
| odyssey4me | prometheanfire o/ indeed... hope you and yours are doing well! | 14:36 |
| prometheanfire | noonedeadpunk: I'm not sure the reason for pinning, is it because old versions have LE root certs and new versions removed it? | 14:36 |
| prometheanfire | odyssey4me: likewise to you as well | 14:36 |
| noonedeadpunk | I'd say vice versa. So as you might now LE jsut rotated their root CA. And eventually, your systems must contain valid CAs to be able communicate | 14:37 |
| noonedeadpunk | and opendev is also using LE | 14:37 |
| noonedeadpunk | odyssey4me: I can recall smth like that, but I'm not sure if it wasn't rsyslog only? | 14:38 |
| noonedeadpunk | So I guess we added specific user-agent to haproxy checks, and I guess we should have been filtering based on that | 14:38 |
| noonedeadpunk | But I dunno how this could be done with journald | 14:39 |
| prometheanfire | noonedeadpunk: so, what exactly are you proposing? updating the pin on a stable branch? I can see that being needed with the rotation | 14:39 |
| noonedeadpunk | except on before forwarding logs | 14:39 |
| odyssey4me | noonedeadpunk I really thought this was something configured in middleware or something in the op[penstack services... but maybe my memory has become faulty in my old age :p | 14:39 |
| noonedeadpunk | prometheanfire: yeah, that is one thing to do. Or eventually validate if we ever need this to be pinned at all | 14:39 |
| jrosser | like this https://github.com/openstack/openstack-ansible/blob/master/inventory/group_vars/haproxy/haproxy.yml#L49 | 14:39 |
| odyssey4me | jrosser yes! | 14:40 |
| noonedeadpunk | but obviously this alone does not prevent from spamming logs... And I believe we don't have anything prior to that | 14:41 |
| odyssey4me | jrosser and https://opendev.org/openstack/ansible-role-uwsgi/src/branch/master/templates/uwsgi.ini.j2#L37 | 14:41 |
| noonedeadpunk | ah | 14:41 |
| ierdem | jrosser, I've checked the logs in /var/log/python_venv_build.log and saw that utility container can not establish connection with the repo container. My OSA configuration has 1 infra host and 2 computes. Logs --> https://paste.opendev.org/show/810078/ | 14:41 |
| ierdem | What internal_lb_vip_address should be ? | 14:42 |
| ierdem | i set it the same address with br-mgmt | 14:42 |
| jrosser | i think thats fine | 14:43 |
| noonedeadpunk | it should be set on the interface with keepalived | 14:43 |
| jrosser | you should be able to test this all out with curl | 14:43 |
| noonedeadpunk | but looks fine indeed | 14:43 |
| jrosser | not keepalived with just one infra node | 14:43 |
| noonedeadpunk | prometheanfire: because eventually certifi is just set of root CAs and it's always good to have latest I guess? | 14:44 |
| noonedeadpunk | or we really need to keep an eye on it at all times | 14:44 |
| jrosser | ierdem: try to curl the URL in the venv build log, duplicate the problem | 14:46 |
| ierdem | jrosser, I've tried, cannot curl | 14:46 |
| jrosser | then swap out the IP for the one of the repo server, and see if that works | 14:46 |
| jrosser | if the repo server is broken, and haproxy has marked it as bad | 14:46 |
| ierdem | repo server is working | 14:46 |
| jrosser | then you wont be able to connect at the VIP | 14:46 |
| jrosser | right, so then need to debug from the perspective of haproxy | 14:46 |
| jrosser | does it think the repo server is up or down..... | 14:47 |
| jrosser | noonedeadpunk: rather annoyingly https://review.opendev.org/c/openstack/openstack-ansible/+/814560 is working locally for me in a fresh centos-7 vm :( | 14:48 |
| jrosser | i even applied the patches with the gerrit cherry pick to make sure i've got exactly the same changes | 14:50 |
| ierdem | jrosser, haproxy logs --> https://paste.opendev.org/show/810079/ . 192.168.137.10 is the external_lb_vip_address. We use this 192.168.137.X network as external, also our hosts has ip from this subnet | 14:52 |
| ierdem | I can reach hosts from these ips but there is no binding for 192.168.137.10 | 14:52 |
| jrosser | is this an lxc or metal deployment | 14:52 |
| ierdem | lxc | 14:53 |
| jrosser | you should be able to use netstat -plant to see whats already bound to that port/ip | 14:53 |
| jrosser | which should be nothing | 14:54 |
| ierdem | yes, there is nothing | 14:55 |
| mgariepy | ierdem, is keepalived adding the ip to your host? | 14:57 |
| jrosser | if its just one controller then i'm not sure we setup keepalived | 14:57 |
| ierdem | I have 1 infra hosts, so there is no keepalived | 14:57 |
| mgariepy | the ip is configured then ? | 14:58 |
| jrosser | ahh good question :) | 14:58 |
| jrosser | "could not bind to" is either something already is bound, or the IP is not there to use | 14:58 |
| mgariepy | no keepalived> not config to bind to non-existing address. | 14:59 |
| ierdem | I set br-mgmt as internal_lb_vip_address, and this ip has binded to infra1 | 14:59 |
| jrosser | for a single infra host you need to manually set up the external IP on a suitable interface | 15:00 |
| jrosser | as keepalived isnt there to do that for you | 15:00 |
| noonedeadpunk | Just in case - we have PTG now in https://www.openstack.org/ptg/rooms/havana | 15:00 |
| jrosser | oh! | 15:00 |
| prometheanfire | noonedeadpunk: well, if you don't want to pin within OSA I think that's fine (for certifi) | 15:00 |
| prometheanfire | we pin in gate to have a single test surface | 15:01 |
| ierdem | jrosser, how can I set external ip manually? | 15:01 |
| noonedeadpunk | prometheanfire: well, we either use or not use u-c? | 15:03 |
| noonedeadpunk | and I guess we're running circles here... | 15:03 |
| noonedeadpunk | Either I don't get why u-c exists or they're widely misused | 15:04 |
| noonedeadpunk | *what u-c are for | 15:04 |
| prometheanfire | main reason for UC to exist is for gate to have a single test surface | 15:05 |
| prometheanfire | a product of that is that UC is useful for downstream, because all those libs are known to work (for packaging or deployment consumers) | 15:05 |
| prometheanfire | I thought OSA had a way of removing a pin for individual libraries | 15:06 |
| jrosser | certifi isnt a proper python package | 15:06 |
| jrosser | its a .pem of CA certificates | 15:06 |
| prometheanfire | ya, certifi is more special | 15:06 |
| prometheanfire | well, odd lol | 15:06 |
| jrosser | and is now bogusly pinned as the contents are expired | 15:06 |
| jrosser | so this is not the normal "what is u-c for anyway" question | 15:07 |
| ierdem | jrosser, problem resolved. We add 'net.ipv4.ip_nonlocal_bind=1' top /etc/sysctl.conf and ran sysctl -p. After these restarted haproxy and setup-infrastructure.yml. It seems fine now | 15:08 |
| prometheanfire | we do have a procedure for updating UC for stable branches, it's probably a lib that should not be pinned (we have a few of those, this is another) | 15:08 |
| mgariepy | ierdem, how the ip gets on the host then ? | 15:15 |
| mgariepy | i am confused a bit.. | 15:15 |
| ierdem | internal or external ? | 15:15 |
| mgariepy | ha. well the external will need to work at some point. | 15:15 |
| mgariepy | i geuss the deployment will/should work if only internal is reachable. | 15:16 |
| prometheanfire | jrosser, noonedeadpunk: if you want to not pin certifi that's fine, but to do so openstack wide will require a ML discussion | 15:17 |
| prometheanfire | that said, I still remember discussion years ago about OSA being able to remove or change individual lines for constraints | 15:17 |
| jrosser | yes we can do that but we have to patch every instance of that in every role, it's very unwieldy | 15:18 |
| prometheanfire | ah | 15:18 |
| prometheanfire | well, in that case, like I said, I can support removing the pin openstack wide | 15:18 |
| prometheanfire | I don't think we can remove it retroactively though | 15:18 |
| prometheanfire | which sucks, given the cert rotation | 15:19 |
| opendevreview | Merged openstack/openstack-ansible-os_tempest master: python-tempestconf moved from osf/ to openinfra https://review.opendev.org/c/openstack/openstack-ansible-os_tempest/+/814404 | 15:36 |
| opendevreview | Merged openstack/openstack-ansible master: Fix manila haproxy manage https://review.opendev.org/c/openstack/openstack-ansible/+/813885 | 16:06 |
| noonedeadpunk | prometheanfire: but we can update pin on stable branches at least? | 16:59 |
| -opendevstatus- NOTICE: Both Gerrit and Zuul services are being restarted briefly for minor updates, and should return to service momentarily; all previously running builds will be reenqueued once Zuul is fully started again | 17:00 | |
| prometheanfire | noonedeadpunk: yep | 17:00 |
| jrosser | ianw: annoyingly this (and its dependancy) seem to work for me here in a centos-7 vm...... https://review.opendev.org/c/openstack/openstack-ansible/+/814560 | 17:21 |
| jrosser | that addresses both the in-CI and out-of-CI cases | 17:22 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/wallaby: Fix manila haproxy manage https://review.opendev.org/c/openstack/openstack-ansible/+/814650 | 19:02 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/victoria: Fix manila haproxy manage https://review.opendev.org/c/openstack/openstack-ansible/+/814651 | 19:03 |
| ianw | jrosser: that failure is the second one in python_venv_build | 19:08 |
| ianw | it might be from urls in the constraint? | 19:08 |
| jrosser | the depends-on is supposed to fix that | 19:08 |
| jrosser | and appears to locally | 19:09 |
| ianw | jrosser: oh, i see. yeah, guess what -- that overrides pip.conf by default :) | 19:10 |
| jrosser | oh, have I missed something obvious? | 19:11 |
| ianw | jrosser: https://review.opendev.org/c/openstack/openstack-ansible/+/814506/4/tests/roles/bootstrap-host/templates/user_variables.aio.yml.j2 | 19:12 |
| ianw | it is not obvious, it took me quite some time to figure out :) | 19:12 |
| jrosser | doh! | 19:13 |
| ianw | jrosser: i'm sort of ambivalent on how that gets fixed; although i think directing pip at the system cacert is probably a more generic solution (even if we move it into the job, rather than a zuul pre step) | 19:23 |
| ianw | but there's a lot of other issues with stein that follow after that; it seems like thing like barbican tempest plugins all try and install v3 only which fails | 19:24 |
| jrosser | I think noonedeadpunk has patches for that already | 19:24 |
| ianw | oh https://review.opendev.org/c/openstack/openstack-ansible/+/814558 ? | 19:29 |
| ianw | does that mean it just falls back to pip for them? | 19:29 |
| jrosser | it’s maybe just disabling those tests entirely | 19:32 |
| ianw | jrosser: is this something that could be done as a follow-up to the debian-stable removal? | 19:37 |
| ianw | i'm just wondering the most efficient path to getting rid of that | 19:38 |
| jrosser | if you want to force merge some stuff to remove those jobs then sure | 19:38 |
| jrosser | feels like we start a new rabbit hole of fixing centos which is tangential to that | 19:39 |
| ianw | yep :) although i think it's been good to get a clear idea of what is going on | 19:39 |
| ianw | if you're happy then, i might re-propose changes to just remove debian-stable and force merge those. the other changes can be used for testing these other fixes | 19:41 |
| jrosser | I would be fine with that | 19:45 |
| ianw | thank you! | 19:47 |
| spatel | how do i restart dhcp namespace ? i had ip conflict and look like that creating issue so wanted to restart namespace | 20:35 |
| spatel | I did systemctl restart neutron-dhcp-agent | 20:35 |
| spatel | does that restart namespace? | 20:36 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!