| darman | I ran `setup-everything.yml`; Do you know in which step I'm getting this error: https://pastebin.ubuntu.com/p/H2zKYN6MZy/ (setup-hosts, setup-infra, setup-openstack)? | 04:19 |
|---|---|---|
| darman | In `openstack_user_config.yml`, I have set: cidr_networks: container: 10.0.0.128/26, but it's assigning IPs between 10.0.0.1-128 while it should start from 129! /etc/hosts: https://pastebin.ubuntu.com/p/SYt29FxmhF/ | 04:38 |
| jrosser | darman: "503 Service Unavailable" is again there is a backend down on haproxy | 09:27 |
| jrosser | darman: you can use codesearch to find where those messages come from like this https://codesearch.opendev.org/?q=Get%20list%20of%20repo%20packages | 09:30 |
| jrosser | which takes you here https://opendev.org/openstack/openstack-ansible/src/branch/master/playbooks/utility-install.yml#L111 | 09:31 |
| darman | > it should start from 129 | 09:35 |
| darman | Solved as It's necessary to remove the `inventory.json` file from /etc/openstack_deploy directory | 09:36 |
| darman | It was using from the previous IP assigning list. | 09:37 |
| darman | When you install something by `apt`, it has a progress bar showing you the download speed of the package(s). But, by the openstack-ansible, it does not show anything related to the speed of downloading stuff | 09:39 |
| darman | Any trick to enable the progress bar? It takes too long and I think there's something there related to the internet. We, the storage team and I, checked the disk IOPs and latency, everything is normal | 09:41 |
| jrosser | darman: that is not openstack-ansible, that is just ansible itself | 10:49 |
| jrosser | and it uses the apt python bindings, not the apt command directly | 10:49 |
| jrosser | ansible in general is not able to provide any feedback during the execution of a task, only at completion | 10:49 |
| jrosser | darman: for information about adjusting inventory.json please see https://docs.openstack.org/openstack-ansible/latest/reference/inventory/manage-inventory.html | 10:51 |
| admin1 | boostrap ansible gave me an error .. --- 25.2.0 -> https://paste.openstack.org/show/beicaN0djeZttA0mwcns/ | 13:57 |
| admin1 | its a brand new server .. fresh jammy install .. | 13:57 |
| admin1 | sed: -e expression #1, char 48: unterminated `s' command | 14:00 |
| admin1 | Line 386 in the paste | 14:00 |
| jrosser | admin1: you are missing this patch https://github.com/openstack/openstack-ansible/commit/f933194813de9d18b89040fa8c5b80bcd0dd967c | 14:06 |
| jrosser | i.e that is already fixed on yoga | 14:07 |
| admin1 | i have to upgrade a 24.0.x cluster today .. i was planning to go to 25.2.0 .. what do you recommend ? | 14:08 |
| admin1 | go to 26.0.0 ? | 14:08 |
| jrosser | 26.x is zed? | 14:09 |
| admin1 | can't we add that patch and make it 25.2.1 so that we know at least this works | 14:09 |
| admin1 | coz right now, we have a tagged version that breaks even at run | 14:09 |
| jrosser | right? | 14:09 |
| admin1 | 26 is zed | 14:09 |
| jrosser | so use a git SHA or stable/yoga instead | 14:09 |
| admin1 | last letter of the alphabet :D | 14:09 |
| jrosser | there is nothing at all magical about a tag | 14:10 |
| jrosser | it has no more weight to it than any other commit on the branch | 14:10 |
| admin1 | i tend to do upgrades based on tag .. beacuse then i know exactly what is in which version | 14:10 |
| jrosser | but you know what is in each version for each commit | 14:10 |
| admin1 | will switch to stable/yoga | 14:10 |
| jrosser | a tag is just a shorthand for a git SHA | 14:10 |
| jrosser | like a pointer | 14:10 |
| admin1 | yes, but stable/yoga might have more shas and then then 2 clusters will be in diff shas right ? | 14:11 |
| jrosser | for what? | 14:11 |
| admin1 | vs using a tag and doing test on it, then you guarantee that all clusters are exactly the same with the same patch at any given moment in time .. even if its done 4 months back | 14:11 |
| jrosser | openstack-ansible, or the services? | 14:11 |
| admin1 | i guess for both . | 14:12 |
| jrosser | nothing stops you writing down the commit you used for a deployment, then re-using that again somewhere else | 14:12 |
| jrosser | thats really all a tag is | 14:12 |
| admin1 | can i upgrade from 24.0 to stable/zed directly ? | 14:13 |
| jrosser | i have no idea | 14:13 |
| jrosser | thats X->Z? | 14:13 |
| admin1 | yeah | 14:13 |
| jrosser | well as you know then only thing we test in CI is N-1 -> N currently | 14:14 |
| admin1 | i will stick to stable/yoga | 14:14 |
| jrosser | well, the thing is you can guarantee that stable/yoga *is* a thing that is potentially different every time you use it | 14:14 |
| jrosser | which is what you were concerned about | 14:15 |
| jrosser | as it is "the head of the branch" rather than any specific point | 14:15 |
| admin1 | right .. | 14:15 |
| admin1 | exactly | 14:15 |
| jrosser | which is why i suggest to use a git sha | 14:15 |
| jrosser | you know that when you do `git log` and it shows you the commit hash, you can use the git hash as a thing to check out? | 14:16 |
| jrosser | that will always be the same | 14:16 |
| admin1 | that is one way | 14:16 |
| jrosser | right, and in the absence of a tag that includes the patch you need is the best way to ensure you get repeatability | 14:16 |
| jrosser | or you fork the OSA repo yourself and apply your own tag | 14:17 |
| admin1 | tags does the same, except i don't have to keep track of the sha in my side and the tag itself does that .. tags have not failed us since 2016 :) .. just this one time on 25.2.0 | 14:44 |
| admin1 | seems like even on stable/yoga i have to apply that patch | 14:46 |
| admin1 | this is strange .. 2 days ago i did 25.2.0 and nothing broke .. ( where I mentioned git missing in repo ) .. | 14:49 |
| admin1 | both times, servers are 22.04 | 14:49 |
| admin1 | issue is something else ..same tag 25.2.0 .. same sha, 2 days apart .. one worked fine, one gives error .. ( both are done by the same deployment script , 22.04.1 LTS ) | 14:52 |
| admin1 | jrosser, is current OSA version supposed to return 25.2.0 there ? | 14:56 |
| admin1 | the version is returned fine in both | 14:57 |
| admin1 | hmm.. intresting | 14:57 |
| admin1 | not sure whats different now | 14:58 |
| admin1 | new error: Clone the role ansible-role-requirements => ERROR! Invalid play strategy specified: openstack.osa.linear The error appears to be in '/opt/openstack-ansible/scripts/get-ansible-collection-requirements.yml': line 16, column 3, | 15:03 |
| jrosser | its not that a tag has failed you | 15:03 |
| admin1 | https://paste.openstack.org/show/bh9D66ZSM4DaiUUTpGcu/ | 15:03 |
| jrosser | OSA is not responsible for things like releases of setuptools or pbr | 15:03 |
| admin1 | i understand | 15:03 |
| jrosser | if those change underneath us in a way that breaks the boostrap scripts then lots of tags won't work any more | 15:03 |
| admin1 | something is causing bootstrap-ansible to break in identical setup where it worked before | 15:04 |
| admin1 | and when I meant identical, in a ubuntu 22.04.1 LTS with minimal install with ssh and python | 15:04 |
| admin1 | that too, in the deploy and not even setup-hosts | 15:04 |
| jrosser | like i say it will be a release of setuptools if you run into the error you pasted before | 15:05 |
| admin1 | the 2nd error also related to that ? | 15:06 |
| jrosser | no | 15:06 |
| admin1 | that error came after I made changes to the patch you mentioned | 15:06 |
| jrosser | https://review.opendev.org/c/openstack/openstack-ansible/+/868273 | 15:06 |
| admin1 | now worked | 15:09 |
| admin1 | after those 2 patches | 15:09 |
| admin1 | thanks jrosser . .. i still need to figure out the difference in my side | 15:09 |
| admin1 | on what happened | 15:09 |
| admin1 | aio scenario aio_ceph fails on ceph-config : run 'ceph-volume lvm list' to see how many osds have already been created => atal: [aio1]: FAILED! => {"changed": false, "cmd": "ceph-volume --cluster ceph lvm list --format=json", "msg": "[Errno 2] No such file or directory: b'ceph-volume'" .. is this a known issue ? | 17:24 |
| admin1 | seems like apt-install ceph-volume on the host seems to have fixed it .. | 18:03 |
| jrosser | admin1: you can see ceph-volume installed here https://zuul.opendev.org/t/openstack/build/4edfe61423244221bd5c05fb77d8f0d9/log/logs/host/apt/history.log.txt#51 | 18:46 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!