| opendevreview | Tushar Trambak Gite proposed openstack/openstack-ansible-os_cinder master: setup.cfg: Replace dashes with underscores https://review.opendev.org/c/openstack/openstack-ansible-os_cinder/+/800418 | 06:14 |
|---|---|---|
| *** rpittau|afk is now known as rpittau | 07:35 | |
| MrClayPole | Hi, we have repo_build.yml playbook failures upgrading from OSA Rocky 18.1.3 to 18.1.20. We are seeing the error http://paste.openstack.org/show/807374/ and the error http://paste.openstack.org/show/807375/ is the wheel build logs for gnocchi. This was with a single repo container. We've then added a repo container to the two other infrastructure nodes to give us three repo nodes. The two new nodes run a repo_install. | 09:07 |
| MrClayPole | yml without error. Are we OK to remove and delete the first repo container and rebuild it? Are we ok running multiple repo containers? | 09:07 |
| noonedeadpunk | MrClayPole: eventually it's designed to run several repo containers, but, real repo build is performed only on single one | 10:02 |
| noonedeadpunk | and then build result is synced with lsyncd to other containers | 10:03 |
| noonedeadpunk | so I guess, that it won't resolve an issue. | 10:03 |
| noonedeadpunk | Instead, you may try to just drop gnocchi from built requirements if you don't need it | 10:03 |
| MrClayPole | noonedeadpunk: Ah I was hoping the new repos were good. We are using gnocchi for billing so can't exclude it. | 10:07 |
| noonedeadpunk | aha, gotcha. | 10:07 |
| noonedeadpunk | I think then you need to see why it fails the build. | 10:07 |
| noonedeadpunk | iirc there were 2 potential issues | 10:08 |
| noonedeadpunk | one of them were ujson version that needed to be 2.0+, while in contraints it was 1.* smth | 10:09 |
| noonedeadpunk | second can't actually recall, maybe related to some ceph libraries or smth like that... | 10:16 |
| CeeMac | noonedeadpunk: hi, do you recall any issues with the 'invalid syntax' error relating to "print(*k)" | 10:28 |
| CeeMac | it seems to trigger as part of a decompressed setuptools run | 10:28 |
| noonedeadpunk | Um, I think yes, but it was smth really trivial | 10:29 |
| CeeMac | any pointers? that seems to be what we're tripping over at the moment | 10:30 |
| CeeMac | we first ran into an issue in 18.1.3 and the issues isn't resolved by bumping to 18.1.20 | 10:30 |
| CeeMac | the whole repo-build process is still very much in the "dark arts" category for me | 10:31 |
| noonedeadpunk | oh, I think it's just trying to install newer setuptools or some package, that is not going to work against py2 | 10:31 |
| noonedeadpunk | I'm glad we actually dropped it in STein | 10:31 |
| CeeMac | so do we need to pin setuptools to a lower version or something? or there is something in the gnocchi role that is trying to pull down an incompatible element fir py2? | 10:32 |
| noonedeadpunk | I think it's not setuptools on their own, since they're bumped by osa in https://opendev.org/openstack/openstack-ansible/src/branch/stable/rocky/global-requirement-pins.txt#L22 | 10:33 |
| CeeMac | i'm trying to see if anything is pinpointed in the traceback http://paste.openstack.org/show/807375/ but its not tracking in my brain very well, probably due to lack of context | 10:33 |
| noonedeadpunk | but it's some package that is not present in upper-constraints, but pulled in by gnocchi or some dependencies of it | 10:34 |
| CeeMac | is there a good/easy way of tracking which dependencies its trying to compile? | 10:35 |
| CeeMac | everything seems to be running in a /tmp/ area which is cleaned up so I can't interogate the file content | 10:35 |
| noonedeadpunk | https://github.com/gnocchixyz/gnocchi/blob/8cbc9ee4c3ff35786fd2d091756e9469abd49e1d/setup.cfg#L27-L109 | 10:35 |
| CeeMac | but the last line that is triggering the error is "File "/tmp/easy_install-Gd0ZKP/setuptools_scm-6.0.1/src/setuptools_scm/utils.py"" | 10:35 |
| noonedeadpunk | so there's really lot's of stuff that are in requirements | 10:36 |
| CeeMac | yowzer | 10:36 |
| CeeMac | and any one of those could have triggered the error? | 10:37 |
| noonedeadpunk | or even their dependencies | 10:37 |
| noonedeadpunk | BUT you can limit a bit scope, because extras_require is installed only from [keystone,mysql] and maybe ceph or ceph_alternative | 10:38 |
| CeeMac | and we dont itemise each one in the build process i guess? so no easy way of seeing which one is tripping us up? | 10:38 |
| noonedeadpunk | well, looking at trace I'm not really sure if it's not setuptools on their own ideed... | 10:39 |
| noonedeadpunk | I would try to figure out what command ansible tries to run | 10:39 |
| noonedeadpunk | I can hardly recall how to debug repo_build, it was 2 years since we really dropped it :( | 10:40 |
| noonedeadpunk | just in case - you do minor upgrade to do major one afterwards? | 10:40 |
| CeeMac | yeah, its a bit frustrating trying to learn something that we're then upgrading away from | 10:41 |
| noonedeadpunk | if so - I think you can jsut do major upgrade at once, and you won't need to mess up with repo_build anymore at all | 10:41 |
| cyberseb | Hello! I have a question regarding openstack deployment with ansible. Is it possible to deploy the haproxy on the control host or do I need extra hardware for that? | 10:41 |
| CeeMac | yeah, we wanted to get to the top of rocky to validate out the rolling upgrade process, then bump up to stein | 10:41 |
| CeeMac | then we need to do a couple of bumps to train and ussuri probably | 10:41 |
| noonedeadpunk | cyberseb: by default we usually deploy haproxy on control hosts | 10:41 |
| noonedeadpunk | CeeMac: eventually I also did R->T direct upgrades back than (considering you're on ubuntu 18.04 or centos 7) | 10:42 |
| noonedeadpunk | but it's smth worth testing in sandbox as well | 10:42 |
| CeeMac | noonedeadpunk: yeah, we might need to consider that if this is going to take a lot of effort. We'll have a poke around | 10:42 |
| CeeMac | noonedeadpunk: oh you got R > T working? We're on Ubuntu 18.04 | 10:43 |
| noonedeadpunk | as eventually we did placement separation during S->T, but it's covered with T upgrade scripts | 10:43 |
| CeeMac | if i could avoid doing the placement hokey-kokey that would be great | 10:43 |
| noonedeadpunk | so yeah, R>T did work for me | 10:43 |
| noonedeadpunk | As well as T>V | 10:43 |
| CeeMac | same upgrade process as noted in R > S regaring mariadb and rabbit individual upgrade to get around those early issues? | 10:44 |
| CeeMac | do you happen to have any notes, or could we just follow the documented ones for stein/train | 10:44 |
| noonedeadpunk | I think you can jsut follow S>T notes. But eventually if you go through what run_upgrade.sh does on T, you will find all actions we did | 10:45 |
| noonedeadpunk | mostly it was covered with commit https://opendev.org/openstack/openstack-ansible/commit/cd32d15cc0cdae0a1e35b6ad81922163bd2ec07c | 10:45 |
| noonedeadpunk | so I'd probably would set `placement-infra_hosts` jsut in case in openstack_user_config | 10:46 |
| CeeMac | thanks noonedeadpunk | 10:47 |
| cyberseb | I already tried to deploy it on the control host with ssl, but haproxy tries to bind to the same IP:Port combinations as the original services. As a result haproxy service fails to start. To avoid this problem I tried to provide different IP addresses for internal and external lb_vip. Can I use the same interface (br-mgmt) for both IPs? | 10:49 |
| noonedeadpunk | I think there was some nit, that we were disabling old placement endpoint too early in run_upgrade.sh, and it could be done more efficiently when ran manually (in case you care that placement was available as much as possible during switch) | 10:50 |
| noonedeadpunk | cyberseb: and you're trying to deploy on bare metal (without LXC containers)? | 10:51 |
| noonedeadpunk | and what release is that? | 10:51 |
| CeeMac | i'm not so fussed about placement tb, we'd be operating a risk window and operations freeze during upgrade and minimising what events are taking place | 10:51 |
| noonedeadpunk | CeeMac: then following run_upgrade.sh logic should just work | 10:52 |
| noonedeadpunk | and you don't _really_ need that minor upgrade on R | 10:52 |
| cyberseb | Releas: Victoria 22.1.4. Yes, I tried to deploy every service on bare metal. That is a requirement for our setup. | 10:53 |
| noonedeadpunk | ok. on V you totally can deploy bare metal and haproxy on the same host. | 10:53 |
| cyberseb | but internal and external lb_vip should be different, right? | 10:54 |
| noonedeadpunk | but you need to have differnt ip addres for VIP (internal/external) and openstack_service_bind_address | 10:54 |
| noonedeadpunk | they can be on the same interface, but must be different | 10:54 |
| CeeMac | thanks noonedeadpunk will take a look at run_upgrade.sh logic | 10:54 |
| cyberseb | ok, thanks noonedeadpunk! | 10:55 |
| noonedeadpunk | we don't currently have logic to support only internal (or only public) VIP | 10:55 |
| noonedeadpunk | as it was never a problem to just have .100 and .101 for example | 10:55 |
| noonedeadpunk | and we also bind all services to just node mgmt address (as defined in openstack_user_config) https://opendev.org/openstack/openstack-ansible/src/branch/master/inventory/group_vars/all/all.yml#L34-L35 | 10:57 |
| CeeMac | thanks noonedeadpunk will take a look at run_upgrade.sh logic | 10:58 |
| noonedeadpunk | sure, np) | 10:58 |
| cyberseb | Hm, ok. I had the problem, that haproxy was configured to bind the repo_all-front-1 to the same IP:PORT as the already running service. | 11:00 |
| cyberseb | As a result haproxy service didn't start. | 11:01 |
| cyberseb | For all the other services the external_lb_vip was set. | 11:02 |
| noonedeadpunk | oh.... | 11:05 |
| noonedeadpunk | I actually wonder if that might be a bug, since in CI we probablt don't have real repo_server (as we run aio there) | 11:05 |
| cyberseb | I will try to reproduce it and write a bug report if I encounter the same behaviour again. k? | 11:08 |
| noonedeadpunk | yes, sure! | 11:50 |
| noonedeadpunk | From what I see repo_server should bind to the same IP as all other services https://opendev.org/openstack/openstack-ansible-repo_server/src/branch/master/defaults/main.yml#L51 | 11:52 |
| jrosser | i think for this case internal vip must != externa vip and also != br-mgmt address | 12:06 |
| jrosser | and be one of used_ips | 12:07 |
| noonedeadpunk | since there;re no containers, I'm not sure if used_ips are still required... | 12:25 |
| noonedeadpunk | in terms of dynamic_inventory... | 12:25 |
| opendevreview | Merged openstack/openstack-ansible-os_keystone stable/victoria: Fix typo in keystone-httpd template https://review.opendev.org/c/openstack/openstack-ansible-os_keystone/+/799587 | 14:55 |
| opendevreview | Merged openstack/openstack-ansible-os_keystone stable/ussuri: Fix typo in keystone-httpd template https://review.opendev.org/c/openstack/openstack-ansible-os_keystone/+/799588 | 14:55 |
| opendevreview | Georgina Shippey proposed openstack/openstack-ansible-os_keystone master: Updates to federation documentation https://review.opendev.org/c/openstack/openstack-ansible-os_keystone/+/800504 | 15:33 |
| *** jgwentworth is now known as melwitt | 16:07 | |
| *** rpittau is now known as rpittau|afk | 16:21 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!