| nurdie | https://docs.openstack.org/kolla-ansible/train/user/centos8.html | 00:27 |
|---|---|---|
| nurdie | Is the kolla-ansible guide OK to follow for a C7 to C8 upgrade? | 00:27 |
| nurdie | SHould I upgrade deployhost first? | 00:27 |
| jrosser | nurdie: which release are you on? | 07:02 |
| jrosser | nurdie: the OS of the deploy host does not matter | 07:03 |
| jrosser | to upgrade/change OS you start with the controllers and then do the computes, that should be possible | 07:04 |
| jrosser | centos7 support was not removed from OSA for old releases but it might be that repos you need are now gone | 07:04 |
| jrosser | *other repos | 07:05 |
| jrosser | but for example I just look at our CI jobs for stable/train and they are still passing for centos7..... | 07:06 |
| opendevreview | Andrew Bonney proposed openstack/openstack-ansible-os_keystone stable/wallaby: Fix shibboleth compatibility for ubuntu 18.04 https://review.opendev.org/c/openstack/openstack-ansible-os_keystone/+/803552 | 07:23 |
| *** rpittau|afk is now known as rpittau | 07:47 | |
| anskiy | jrosser: I've reproduced a bug with OVN service name on Stream on AIO. What should I do next? :) | 10:44 |
| jrosser | anskiy: check it’s something we’ve not already fixed in master... raise a bug on launchpad.... talk to spatel here later when he’s around..... | 10:47 |
| jrosser | I’m not really around this week but spatel and mgariepy have the most practical experience with OVN | 10:47 |
| anskiy | it's already on master, so I'll try to reach spatel, as I have another concern about OVN clustering (looks like it doesn't work) | 10:50 |
| anskiy | thank you | 10:50 |
| jrosser | there is an outstanding patch to h age how the clustering works | 10:50 |
| jrosser | *change | 10:50 |
| jrosser | from haproxy to native cluster | 10:51 |
| anskiy | yeah, I saw that one, but my concern is on the OVN side more, not the neutron <-> OVN | 10:51 |
| anskiy | that one, you're talking about is actually how it should be done: there is no routing in clustered OVN, so the client would just hang if he connects to non-leader node :) | 10:52 |
| spatel | mgariepy hey! morning | 13:06 |
| spatel | how to merge these jobs ? https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/802134 | 13:06 |
| spatel | i meant start the gate process | 13:07 |
| anskiy | spatel: hey! so, I've managed to reproduce a bug with OVN installation on Stream in AIO on current master. It uses wrong service name (ovn-central instead of ovn-northd). | 13:08 |
| spatel | yes, central start ovn-northd behind it | 13:09 |
| nurdie | jrosser: My deployment is Train 20.2.1 on CentOS7. Some of it is now on 20.2.6, but I couldn't finish it so I brought up services as I could. I have no idea how that's passing your CI o.o | 13:09 |
| spatel | nurdie we don't have CentOS7 CI job.. did i miss something? | 13:10 |
| nurdie | One example is that OSA asks pip2 to install keystone==16.0.2.dev19, but that doesn't exist | 13:10 |
| anskiy | spatel: there is no such service on Stream, only ovn-northd | 13:10 |
| nurdie | spatel: Yes, jrosser had asked me a question about 5hours ago when I was drooling and snoring | 13:11 |
| jrosser | spatel: Train has centos7 | 13:11 |
| spatel | ah! so just for train, i wasn't aware of that sorry | 13:11 |
| jrosser | nurdie: the install from source code can totally install that version of keystone | 13:12 |
| spatel | anskiy this is what i am seeing - https://paste.opendev.org/show/807909/ | 13:12 |
| nurdie | From source code? Is there a different way of doing it? I was adding a task to service roles to sed that dev branch out for pip before the installs | 13:13 |
| jrosser | nurdie: check out the centos7 jobs here https://review.opendev.org/c/openstack/openstack-ansible/+/803405 they ran in the last 24 hours | 13:13 |
| anskiy | spatel: wait, is this inside container? | 13:14 |
| jrosser | OSA does not install openstack from pip packages | 13:14 |
| spatel | anskiy behind that file it start these services - https://paste.opendev.org/show/807910/ | 13:14 |
| nurdie | jrosser: I ultimately failed with C7 and pip2 (probably because of pip2?) not liking python-systemd | 13:14 |
| spatel | yes inside LXC container | 13:14 |
| jrosser | nurdie: well there is centos7 and some specific version of centos7.x | 13:15 |
| anskiy | spatel: ah, so, apparantly it's some flavor of debian inside that. I'm talking about full metal deployment. | 13:15 |
| jrosser | anskiy: Debian? | 13:16 |
| anskiy | I mean, OS in this LXC is Debian or Ubuntu, because that's how OVN starts on those | 13:18 |
| nurdie | jrosser: plain ol' 7.9. Example of what I couldn't get passed: https://pastebin.com/WeA4vZE6 | 13:19 |
| anskiy | this is how it looks on Stream with SCENARIO='aio_metal': https://paste.opendev.org/show/807911/ | 13:19 |
| nurdie | That was a nova-api container install. I want to move to ubuntu if it has less package isssues :) | 13:20 |
| spatel | anskiy you are saying you can't see ovn-central service? | 13:21 |
| mgariepy | morning. | 13:21 |
| jrosser | nurdie: 404 Client Error: Not Found for url: http://10.250.0.210:8181/os-releases/20.2.6/centos-7.6-x86_64/ - skipping | 13:22 |
| jrosser | 2021-08-04T18:42:02,414 Skipping link: unsupported archive format: .6-x86_64: | 13:22 |
| mgariepy | after some digging on the sphinx stuff. the issue is the rst parser not doing fallback on ""unknown languages"" | 13:22 |
| jrosser | that is your root cause I think | 13:22 |
| anskiy | spatel: yes, there is no such service, when you deploy openstack on Stream on metal. There are a couple more issues, which are going after this one too. | 13:23 |
| spatel | how the heck CI job passing then? | 13:24 |
| spatel | i did install on metal and it worked for me.. i was able to create vm etc... | 13:25 |
| jrosser | nurdie: I can’t check it for you but you need to look through later code if we ever updated the repo path not to include a ‘.’ | 13:25 |
| jrosser | like I say it’s passing centos-7 jobs currently and all the logs are there on the link I gave you to look through | 13:26 |
| anskiy | spatel: from which package that service came in your case? In my lab ovn-northd.service comes from ovn-2021-central-21.03.0-40.el8s.x86_64 | 13:27 |
| spatel | did you install using distro or source? | 13:27 |
| jrosser | nurdie: it is failing to find the built wheels on the repo server because of that error, falling back to pypi and then not finding the specific ‘devN’ versions from the built wheel constraints | 13:28 |
| anskiy | spatel: source is default, right? I haven't changed it. | 13:28 |
| spatel | jrosser could you kick off the gate process? - https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/802134 | 13:29 |
| spatel | anskiy yes source is default | 13:29 |
| nurdie | jrosser: if a '.' is the cause of all of my woes over the last week, I'm going to beat the nearest printer to death | 13:29 |
| anskiy | spatel: btw, are you sure that it was using OVN in the end? Because, if I don't set neutron_provider_networks in user_*.yml it doesn't trigger OVN installation at all. | 13:30 |
| jrosser | spatel: I can’t because the depends-on are not merged | 13:31 |
| spatel | oh!! got it | 13:31 |
| spatel | anskiy hmm! i just noticed we don't have OVN centos-8-stream job in CI job | 13:32 |
| spatel | anskiy i am kicking off my centos-8-stream lab to deploy AIO with OVN and see how it goes.. i think we should setup zuul CI job for OVN C8 stream | 13:33 |
| mgariepy | downgrading to docutils to 0.16 seems to fix the issue. | 13:33 |
| anskiy | spatel: there are a couple more bugs to this (which lead to non-working OVN), some of which I'm not sure how to fix properly. | 13:33 |
| spatel | I am running c8-stream in my lab and no issue at all.. but i would like you to open bug or show some error so we can fix it | 13:34 |
| anskiy | okay :) | 13:36 |
| jrosser | anskiy: are you interested in helping to maintain the centos stream support in OSA? | 13:36 |
| nurdie | Also, jrosser: what do you mean, "from source code"? As opposed to what, a highly customized deployment where an operator pins their own pip packages? I'm not trying to be facetious! If there's a "better" way, I'll do it | 13:36 |
| jrosser | nurdie: if you can do a “distro” install which uses exclusively apt/rpm from Ubuntu cloud archive or RDO, but I would highly discourage that in an OSA context | 13:37 |
| anskiy | jrosser: I don't have much expirience in contributing to the projects this large, but I can try :) | 13:37 |
| jrosser | anskiy: I think it’s only fair that I say that OSA is maintained by its end users for its end users | 13:38 |
| nurdie | Ah, got it. Yeah, I prefer to leave the version pinning to the pros. I do that with the company I work for's HA version of their product so I understand that :) | 13:39 |
| jrosser | and we see less and less people using centos over time, and without active maintainers it may even come to dropping support | 13:39 |
| nurdie | jrosser: thanks yet again. I will hunt for the missing period and submit a bug if I find it | 13:39 |
| spatel | anskiy you will start maintaining and you won't realized that is how it works :) | 13:39 |
| spatel | jrosser look like it doesn't like this solution - https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/803475 | 13:41 |
| spatel | trying to hunt what is going on | 13:41 |
| jrosser | nurdie: I have a very distant memory of needing to fix that period issue but I can’t look at the history this week | 13:41 |
| nurdie | It's cool. At any rate, I'm very appreciative of your knowledge and willingness to help. You, sir, are a scholar and a star | 13:42 |
| nurdie | If you're a sir* or whatever | 13:42 |
| jrosser | spatel: https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_fd8/803475/1/check/openstack-ansible-deploy-hosts_distro_lxc-centos-8-stream/fd8d618/logs/host/lxc-cache-prep-commands.log.txt | 13:44 |
| spatel | hmm very odd | 13:45 |
| spatel | i am deploying in lab to debug.. | 13:46 |
| spatel | jrosser may be this is the solution - https://access.redhat.com/discussions/4222851 but will see when my build completed | 14:00 |
| nurdie | jrosser: Perhaps it's even worse: a missing trailing slash (/)? https://pastebin.com/2VfBjJZY | 14:05 |
| nurdie | >_< | 14:05 |
| nurdie | Could that be an nginx bug? Needing to add try clauses for missing trailing slashes? | 14:06 |
| nurdie | Er, try_files* rather | 14:07 |
| jrosser | nurdie: you could make a symlink to a simpler path on all the repo servers and override this https://github.com/openstack/ansible-role-python_venv_build/blob/stable/train/defaults/main.yml#L143 | 14:16 |
| jelabarre-rh | for the ansible module/task "yum_repository", is seems that's only for adding repos where you have the full baseurl, rather than simply enabling one that is already defined | 15:08 |
| jelabarre-rh | at least that's the impression I'm getting from looking at the examples on the module's documentation page | 15:09 |
| mgariepy | spatel, jrosser https://github.com/openstack/openstack-helm/commit/9c89c32bd3c862f100cb0170909cb4c2312153c5 | 15:41 |
| mgariepy | that's a way to fix a translation issue.. | 15:42 |
| spatel | Nice! | 15:44 |
| mgariepy | it's not super respectful for the translator IMO. | 15:45 |
| spatel | we have to unblock the gate | 15:54 |
| mgariepy | the issue is the rst parser in docutils not falling back to en when encountering unknown lang. | 15:57 |
| spatel | These stuff beyond my knowledge :) | 16:06 |
| mgariepy | i'm not a python dev either but i can read ;p | 16:06 |
| spatel | why don't we go with non-voting so we can release next wallaby and come back on this later when we have more manpower :) | 16:07 |
| mgariepy | +2 on the patch | 16:13 |
| mgariepy | spotz_, ? do you have a minute? https://review.opendev.org/c/openstack/openstack-ansible/+/803371 | 16:13 |
| opendevreview | Satish Patel proposed openstack/openstack-ansible-lxc_hosts master: Add yum vars for centos-8-stream lxc containers https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/803630 | 16:15 |
| spatel | jrosser this is the fix for centos-8-stream lxc cache failure - https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/803630 | 16:16 |
| jrosser | spatel: are they not needed on the host for a metal job? | 16:27 |
| *** sshnaidm is now known as sshnaidm|afk | 16:29 | |
| spotz_ | mgariepy: looking | 16:33 |
| spotz_ | ha jrosser got it | 16:33 |
| mgariepy | jrosser, is on vacation !:P | 16:40 |
| *** rpittau is now known as rpittau|afk | 16:41 | |
| admin1 | [haproxy_server : regen pem] -- this looks like a new addition .. why would this run when i am supplying my own certs ? | 16:44 |
| admin1 | and it fails | 16:44 |
| jrosser | because it has to concatenate the CA, key and certificate | 16:45 |
| admin1 | so this is a new addition ? before i only had to privide a cert pem and a key pem | 16:46 |
| admin1 | now have to provide all 3 of them separately ? | 16:46 |
| admin1 | haproxy_ssl_self_signed_regen: false ; haproxy_user_ssl_cert: /opt/ssl/cert.pem ; haproxy_user_ssl_key: /opt/ssl/key.pem -- these 3 lines worked fine for years | 16:47 |
| jrosser | which branch fails? | 16:47 |
| admin1 | i am trying to deploy 23.0.0 | 16:48 |
| jrosser | have you read the release notes | 16:48 |
| admin1 | strangely , i have not this time :) | 16:48 |
| jrosser | ahha - I cannot help with it this week but there is a total overhaul for ssl in W | 16:49 |
| admin1 | https://docs.openstack.org/releasenotes/openstack-ansible/unreleased.html - i see some pointers there | 16:49 |
| admin1 | so just change to haproxy_ssl_cert_path and thats it it looks like | 16:51 |
| spatel | jrosser for metal its automatically creating those variables, its only not creating for lxc container, i believe because we are building it in chroot and may need some dependency | 16:51 |
| admin1 | jrosser , is it 1 single pem now ? i can't find anything when doing a grep -ri for the _key | 16:52 |
| jrosser | spatel: https://github.com/openstack/openstack-ansible-lxc_hosts/blob/master/vars/centos-8.3.yml#L24 | 16:54 |
| admin1 | it worked for me for the wildcard . i will try to submit a patch to the documentation to add a few lines in the docs to make it much clear | 17:00 |
| spatel | jrosser sweet!! so we should just need to add - /etc/yum/vars | 17:05 |
| spatel | jrosser should be create centos-8.yml file? | 17:12 |
| jrosser | it’s complicated | 17:15 |
| spatel | then lets go with 8.3 for now or create symlink to centos-8.yml | 17:16 |
| jrosser | ? | 17:16 |
| spatel | i meant add here - https://github.com/openstack/openstack-ansible-lxc_hosts/blob/master/vars/centos-8.3.yml#L24 | 17:16 |
| jrosser | stream vs not stream is what makes it complicated | 17:16 |
| jrosser | iirc 8.3 and 8.4 are not-stream | 17:17 |
| spatel | yes | 17:17 |
| jrosser | vars/redhat.yml covers stream? | 17:17 |
| spatel | hmm it should cover.. | 17:17 |
| jrosser | as it’s neither 8.3 nor 8.4 so falls back to that, I think | 17:17 |
| spatel | let me add in redhat.yml and run test | 17:18 |
| jrosser | there’s no proper detection of stream in ansible so this is a mess | 17:18 |
| spatel | hmm | 17:22 |
| spatel | let me add in redhat.yml and rebuild my lab to see if that works then we will stick with redhat.yml | 17:28 |
| opendevreview | Marc Gariépy proposed openstack/openstack-ansible master: skip -W on sphinx-build for translation. https://review.opendev.org/c/openstack/openstack-ansible/+/803635 | 17:48 |
| mgariepy | let's see if that works. | 17:50 |
| mgariepy | the issue seems to be more the sphinx-build command -W that threats warning as error and the bash script that does exit on error. | 17:52 |
| spatel | figures cross | 18:05 |
| mgariepy | it passed.. but well.. https://review.opendev.org/c/openstack/openstackdocstheme/+/802758 | 18:09 |
| opendevreview | Satish Patel proposed openstack/openstack-ansible-lxc_hosts master: Add yum vars for centos-8-stream lxc containers https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/803630 | 18:12 |
| opendevreview | Satish Patel proposed openstack/openstack-ansible-lxc_hosts master: Add yum vars for centos-8-stream lxc containers https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/803630 | 18:13 |
| spatel | jrosser this works in my lab - https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/803630/3/vars/redhat.yml | 18:13 |
| spatel | mgariepy great! lets push it out.. | 18:14 |
| mgariepy | it might soon be dropped .. :/ | 18:16 |
| admin1 | checking if you guys know why this comes up from time to time "galera_server : Fail if galera_cluster_name doesnt match provided value" | 18:39 |
| admin1 | greenfield lab install | 18:39 |
| admin1 | literally the container was just created | 18:39 |
| admin1 | mgariepy spatel jrosser, any of you already using 23.0 in prod ? | 19:23 |
| admin1 | in my case, either the rabbitmq fails to cluster, or the mysql fails to cluster :( | 19:23 |
| admin1 | prod as in 3x controllers + valid ssl cert for horizon+apis | 19:24 |
| mgariepy | not me. | 19:25 |
| spatel | admin1 i don't have and working on to upgrade to 23.0 but there are some pending patch | 19:28 |
| spatel | admin1 open bug if you think its real issue | 19:34 |
| spatel | before we release 23.1 | 19:34 |
| admin1 | yeah .. i am doing more testing now | 19:40 |
| spatel | +1 | 19:48 |
| spatel | In CI job we don't do multi-node so you will see more issue when go out and start deploying on multi-node (specially cluster apps) | 19:48 |
| -opendevstatus- NOTICE: The Gerrit service on review.opendev.org is going down for a quick restart to adjust its database connection configuration, and should return to service momentarily | 20:02 | |
| nurdie | I evacuated a compute node and 3 intances are stuck in "accepted" in "nova migration-list", even though those hosts are booted and running on a different compute node. Do I need to edit galera to clear that up? | 20:05 |
| admin1 | nurdie, never delete an entry .. just update fields :) | 20:48 |
| admin1 | using tcpdump, maybe make sure its 100% in the other server | 20:48 |
| admin1 | wondering what happens if you cancel the migration job ( after they have already migrated | 20:49 |
| nurdie | admin1: Oh it's definitely on the other server lol. The failed compute node is super dead | 21:08 |
| nurdie | Anyone know off the top of your head what db.table that will be in? | 21:08 |
| opendevreview | Satish Patel proposed openstack/openstack-ansible-openstack_hosts master: Add nova dependency repo for distro install https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/803475 | 21:09 |
| admin1 | #openstack-nova guys will know this nurdie | 21:09 |
| nurdie | admin1: thanks! | 21:11 |
| jrosser | admin1: we normally make the “x.1.0” release after a few people have tried multinode labs / upgrades on a new release and things found in those to fix | 21:15 |
| jrosser | would be great if you could take a look (particularly regarding SSL as that’s a huge change for W). the docs certainly need improving and examples adding for the new features there | 21:16 |
| admin1 | i am on it | 21:16 |
| jrosser | awesome | 21:17 |
| admin1 | setting up 2 labs .. one greenfield multinode .. and one installing a new 22.2.0 now so that once the greenfield works, will test from 22.2.0 > 23.0.0 as well | 21:17 |
| admin1 | to see what needs to be changed for existing variable files | 21:17 |
| jrosser | yeah, there is now a full internal CA | 21:18 |
| jrosser | a new ansible role to manage that and a whole set of new files in /etc/openstack_deploy, and new variables to manage it | 21:18 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!