| jrosser | NeilHanlon: pretty encouraging with cloud-init 22.2 https://paste.opendev.org/show/b6qQu6zRx1MAfiRnaYEZ/ | 09:17 |
|---|---|---|
| Elnaz | Salam | 12:38 |
| Elnaz | What if I manually install mariadb packages from Ubuntu repo not from the MariaDB repo? | 12:40 |
| Elnaz | I read that limited connectivity, but could find a proper way to mirror the MariaDB repository | 12:41 |
| Elnaz | Why you don't use Ubuntu packages for installing mariadb in Galera LXCes? | 12:42 |
| Elnaz | could not* find | 12:42 |
| Elnaz | It s3eems I have connectivity issue with stuff hosted by GCP! | 12:43 |
| jrosser | Elnaz: I use debmirror here but there are lots of other tools https://help.ubuntu.com/community/Debmirror | 12:44 |
| jrosser | Elnaz: and it is kind of risky to use the mariadb package except for the version we actually test. there have been bad bugs in the past and tying to a specific version gives you something known | 12:45 |
| Elnaz | Two days ago I tried to localize its repo, but I think I ran into the problem that this possibility was available for the enterprise version. I will check again now. (I use aptmirror.) | 12:49 |
| Elnaz | And if it becomes local, what will be its variable to tell the deployer to use the local dir? | 12:51 |
| noonedeadpunk | Elnaz: we mirrored mariadb repos jsut a month ago and there were no issues with that | 13:23 |
| noonedeadpunk | I think we've used aptly, but not 100% sure | 13:25 |
| Elnaz | Probably I'm following the wrong link: https://mariadb.com/docs/xpand/deploy/deployment-methods/repo-mirror/ that taks about the enterprise repo! | 13:26 |
| Elnaz | talks* | 13:26 |
| noonedeadpunk | So we used http://downloads.mariadb.com/MariaDB/mariadb-10.6.10/repo/ubuntu | 13:28 |
| noonedeadpunk | Basically it's constructed here https://opendev.org/openstack/openstack-ansible-galera_server/src/branch/master/vars/debian.yml#L76 | 13:28 |
| noonedeadpunk | But I assume you can define galera_repo: {} and override galera_mariadb_server_package - that might be enough to install from system ones | 13:30 |
| noonedeadpunk | well, maybe also galera_distro_package_pins: [] | 13:30 |
| NeilHanlon | jrosser: yay! | 13:31 |
| jrosser | NeilHanlon: seems those packages worked nicely | 13:31 |
| NeilHanlon | ok, great. i will talk with the RHEL maintainer and see if we can rebase to 22.2 | 13:32 |
| jrosser | cool - it looked like there were a good number of RH bugs open with related things | 13:33 |
| jrosser | and some of the "fixes" applied are really not good | 13:33 |
| NeilHanlon | https://bugzilla.redhat.com/show_bug.cgi?id=2163657 | 13:39 |
| NeilHanlon | just re-using this ticket to talk to the maintainer | 13:39 |
| Elnaz | There' lots of mirror here: https://mirmon.mariadb.org | 13:53 |
| Elnaz | What if I set `galera_repo_host: mirror.serverion.com` in var file? (https://mirror.serverion.com/mariadb/) | 13:55 |
| Elnaz | noonedeadpunk: will it work? | 13:56 |
| Elnaz | by moving from downloads.mariadb.org to mirror.serverion.com | 13:57 |
| noonedeadpunk | Elnaz: well, the thing is that next part of URL is /MariaDB | 14:23 |
| noonedeadpunk | So I'd suggest to fully override `galera_repo` | 14:24 |
| noonedeadpunk | or at least galera_repo_url | 14:24 |
| spatel | noonedeadpunk jrosser look like i have some locking issue in ceph causing VM filesystem error | 15:06 |
| noonedeadpunk | You mean the ones that listed with `rbd lock ls`? | 15:08 |
| spatel | I did but not sure if i miss something - https://paste.opendev.org/show/b8V5D73u4cBYMDFPlKbv/ | 15:08 |
| noonedeadpunk | ie https://docs.ceph.com/en/latest/rbd/rbd-exclusive-locks/ | 15:08 |
| noonedeadpunk | oh, I don't think you want to blacklist clients? | 15:09 |
| noonedeadpunk | as I'd assume this is one of computes | 15:09 |
| spatel | but how it will remove lock? | 15:09 |
| noonedeadpunk | rbd lock rm ? | 15:09 |
| spatel | if i shutdown VM then it should release lock automatically correct? | 15:09 |
| spatel | see - https://paste.opendev.org/show/bUJDQhq6gOea6iKkyLNq/ | 15:10 |
| spatel | still exist.. after rm | 15:10 |
| noonedeadpunk | smth like `rbd lock rm vms/ec6044e6-2231-4906-9e30-1e2e72573e64_disk 139643345791728 client.1211875`? | 15:11 |
| noonedeadpunk | what is 192.168.3.12? | 15:11 |
| noonedeadpunk | Before removing lock ensure you don't have any VM with disk attached running | 15:12 |
| spatel | rbd: releasing lock failed: (2) No such file or directory | 15:12 |
| spatel | getting error | 15:12 |
| spatel | 192.168.3.12 is compute node IP | 15:12 |
| noonedeadpunk | maybe auto instead of 139643345791728... | 15:12 |
| spatel | rbd lock rm vms/ec6044e6-2231-4906-9e30-1e2e72573e64_disk auto client.1211875 | 15:13 |
| noonedeadpunk | command format is `rbd lock rm <image-spec> <lock-id> <locker>` | 15:13 |
| spatel | same errr - rbd: releasing lock failed: (2) No such file or directory | 15:13 |
| spatel | let me try that | 15:13 |
| noonedeadpunk | yeah, maybe `-p vms ec6044e6-2231-4906-9e30-1e2e72573e64_disk` | 15:14 |
| noonedeadpunk | instead of vms/ID | 15:14 |
| spatel | rbd lock rm -p vms ec6044e6-2231-4906-9e30-1e2e72573e64_disk ? | 15:15 |
| spatel | none working tried couple of combination | 15:15 |
| spatel | You gotta be kidding me... this is the command | 15:20 |
| spatel | rbd lock rm -p vms ec6044e6-2231-4906-9e30-1e2e72573e64_disk "auto 139643345791728" client.1211875 | 15:20 |
| noonedeadpunk | ah, yes | 15:23 |
| noonedeadpunk | you're right! | 15:23 |
| noonedeadpunk | I can recall it was tricky, but it was quite a while I ran that hehe | 15:23 |
| mgariepy | i seen this once or twice but it was years ago. | 15:23 |
| mgariepy | rbd lock issue | 15:24 |
| spatel | No joke!!! all my vms up without any filesystem error now :) | 15:25 |
| mgariepy | spatel, what version os ceph was the cluster installed ? | 15:25 |
| spatel | This is epic win for me | 15:26 |
| mgariepy | of ceph ** | 15:26 |
| spatel | Quincy (latest version) | 15:26 |
| spatel | Ceph is new to me.. so not sure in past what was the behavior but this is not good | 15:28 |
| mgariepy | i think it was an issue with luminous back in the old days. | 15:29 |
| spatel | I can't remove lock for 100s of VM.. | 15:29 |
| mgariepy | what profile does your client has ? | 15:29 |
| mgariepy | ceph auth get client.[nova|cinder|other?] | 15:30 |
| jrosser | though i think if you have a power loss to your whole cluster it's not surprising that its disaster-recovery situation rather than everything just comig back cleanly | 15:31 |
| spatel | mgariepy - https://paste.opendev.org/show/bsfnNZ3OmrB8t3iWbfIl/ | 15:32 |
| spatel | jrosser i thought ceph should release lock if VM shutdown (i may be wrong) | 15:32 |
| mgariepy | set profile to osd. | 15:32 |
| jrosser | but it depends where the lock-id is stored | 15:33 |
| mgariepy | also do not paste keys next time. | 15:33 |
| jrosser | if it's temporary in the memory of the rbd client on the compute node then it will be difficult to manage | 15:33 |
| spatel | mgariepy those keys are fake.. i copy paste random number | 15:34 |
| mgariepy | haha ok :) | 15:34 |
| spatel | mgariepy what profile i should set? | 15:34 |
| mgariepy | https://paste.opendev.org/show/b88DjHwbbHSDhY1i2FXf/ | 15:36 |
| mgariepy | this is what i have. | 15:37 |
| Elnaz | https://releases.openstack.org/constraints/upper/fc7e2105e81c352602085bd2928a706d0ab8a80d 🤦🤦🤦 | 15:37 |
| spatel | what is the advantage of it? | 15:38 |
| spatel | i meant what are these options for | 15:38 |
| mgariepy | the profile will git the client the caps it needs. | 15:38 |
| Elnaz | That link is redirected to an opendev link where I have isuue to connect to! | 15:39 |
| Elnaz | Err: https://paste.ubuntu.ir/peacv | 15:40 |
| mgariepy | spatel, https://docs.ceph.com/en/latest/rados/operations/user-management/ | 15:40 |
| noonedeadpunk | Elnaz: it's quite easy to override this path IIRC | 15:40 |
| noonedeadpunk | user_requirements_git_url is the variable you need | 15:41 |
| noonedeadpunk | Elnaz: but be careful! As you want specific version always | 15:43 |
| noonedeadpunk | So include requirements_git_install_branch in path | 15:43 |
| Elnaz | infra1-repo-container-22a71eff:~# `wget https://releases.openstack.org/constraints/upper/fc7e2105e81c352602085bd2928a706d0ab8a80d -O /etc/openstack_deploy/upper-constraints/upper_constraints_fc7e2105e81c352602085bd2928a706d0ab8a80d.txt` didn't help! | 15:43 |
| noonedeadpunk | Elnaz: iirc you have github available there? | 15:43 |
| Elnaz | yes, it's fine here | 15:44 |
| noonedeadpunk | Ok, then you can define following in user_variables - `user_requirements_git_url: "https://raw.githubusercontent.com/openstack/requirements/{{ requirements_git_install_branch }}/upper-constraints.txt"` | 15:45 |
| Elnaz | Thank you, I run it again to see what happens including this new address | 15:49 |
| Elnaz | Do you know what email address I should contact to inform about this connectivity problem with opendev.org? | 15:57 |
| Elnaz | Of course, if it is hosted on GCP, there is no solution, because it is Google that is blocking me! | 16:02 |
| jrosser | Elnaz: whoever provides your internet connectivity ultimately | 16:07 |
| jrosser | and provide a reproducible test case and some traceroutes / mtr to both working and broken things | 16:08 |
| Elnaz | Then I'll try to gather some data by comparing opendev vs github. | 16:09 |
| noonedeadpunk | Elnaz: well, opendev is hosted in multiple providers, but I think they're mainly in US | 16:09 |
| jrosser | as an end user theres no point trying to contact cogent or zayo or whichever transit provider we think is problematic | 16:10 |
| noonedeadpunk | I'd assume that these are possibly some regional limitations at where Elnaz at | 16:11 |
| noonedeadpunk | cause by sanctions or stuff like that | 16:11 |
| jrosser | potentially yes | 16:11 |
| Elnaz | 👌 | 16:12 |
| jrosser | it's a shame we don't manage to merge the patches to easily switch * to github | 16:12 |
| noonedeadpunk | I think we need another vote for that? | 16:12 |
| jrosser | well actually for a ton of stuff | 16:12 |
| jrosser | need to remember we are not having andrew here for months so everthings got a bit stuck | 16:13 |
| noonedeadpunk | that was the one at least? https://review.opendev.org/c/openstack/openstack-ansible/+/869748 | 16:14 |
| jrosser | https://review.opendev.org/q/topic:osa-mirrors | 16:15 |
| jrosser | though i wonder actually if we cover the u-c URL that Elnaz would need there | 16:16 |
| jrosser | that seems like something else again | 16:16 |
| noonedeadpunk | No, we don't cover that | 16:16 |
| Elnaz | The weird The strange thing is that I can easily and quickly open opendev web pages in the browser, but Ansible can't fetch things from opendev! | 16:17 |
| jrosser | thats a more interesting patch as the URL is completely different between opendev/github | 16:17 |
| jrosser | i can probably come up with something for that | 16:18 |
| jrosser | thats needing to cover 3 conditions - opendev, github and local mirror | 16:19 |
| noonedeadpunk | jrosser: I think that it's better to just document u-c case | 16:19 |
| noonedeadpunk | as for some gitlab or local fork it will be different again | 16:19 |
| jrosser | well, that too :) | 16:19 |
| jrosser | we can give a github example in the doc | 16:19 |
| noonedeadpunk | Elnaz: well, you can talk to fungi - he knows way more about how infra things work then we are | 16:20 |
| jrosser | Elnaz: is it the same network for your browser and the deploy host though? like not split-tunnel VPN or something? | 16:20 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-haproxy_server master: Update hatop to latest release, 0.8.2 https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/872262 | 16:22 |
| noonedeadpunk | I think that git and web may end up in different locations | 16:22 |
| noonedeadpunk | They're behind LB after all | 16:23 |
| Elnaz | jrosser: yes, they're different, but let me connect to our infrastructure by `sshuttle` where the ansible is running. | 16:23 |
| jrosser | noonedeadpunk: aaahh i mistook opendev web pages for looking at the git repos at opendev.org | 16:25 |
| jrosser | Elnaz: i think you were investigating with `mtr` before - are you able to share the output from that? | 16:28 |
| Elnaz | Oh, passing traffic through our datacenter, it's really slow for opening opendev link | 16:28 |
| Elnaz | The link https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/872262 finally opened, but after about 1 minute! | 16:29 |
| Elnaz | jrosser: for `mtr`; w8 plz. | 16:30 |
| fungi | https git and web browsing from the same ip address will get load-balanced to the same backend | 16:31 |
| fungi | if you're doing it from different source addresses you may end up going to one of 8 (currently) different backend servers | 16:31 |
| fungi | you can tell from the cn in the ssl cert which backend you ended up with | 16:31 |
| fungi | but also we've seen some very high latency and moderate packet loss traversing the zayo backbone through what seems to be both the london and atlanta networks | 16:33 |
| fungi | depending on which backbone providers you end up routed through, you may see dreadful performance as of the past week or two | 16:33 |
| fungi | odds are the performance difference between one place and another when pulling files from opendev.org is really a problem somewhere out on the internet between the client and server | 16:35 |
| fungi | though if you let me know which backend server the client ends up hitting when they see poor performance, i can take a closer look at that backend too | 16:35 |
| noonedeadpunk | Elnaz: ^ | 16:36 |
| fungi | echo|openssl s_client -connect opendev.org:https|openssl x509 -text|grep gitea | 16:36 |
| fungi | the gitea0X is the backend | 16:37 |
| fungi | echo|openssl s_client -connect opendev.org:https|grep ^subject | 16:39 |
| fungi | slightly shorter | 16:39 |
| Elnaz | fungi: that `echo...` output: https://paste.ubuntu.ir/dskto | 16:43 |
| fungi | thanks, so whatever machine you ran that on is being directed to gitea04 currently. i'll check to see if it's having any resource problems | 16:46 |
| Elnaz | jrosser: not possible to copy mtr output, then please see them here: https://mega.nz/folder/VsgwkDxZ#h_aDc0qv4cXl7oH39wDkbA | 16:46 |
| fungi | gitea04 doesn't seem to be under any unusual stress or resource contention: http://cacti.openstack.org/cacti/graph_view.php?action=tree&tree_id=1&leaf_id=882&nodeid=node1_882&host_group_data= | 16:50 |
| Elnaz | Can I manually change it somewhere not to be directed to Gitea04? | 16:59 |
| jrosser | Elnaz: the link breaks now for me but google and github used a cdn and were served for you in EU, opendev needed a transatlantic hop to US, thats the most obvious difference | 17:01 |
| Elnaz | sorry; mtr: https://mega.nz/folder/lxxUBA4D#OE7rhdCd7zOsLvQAsYeHVg | 17:06 |
| fungi | Elnaz: you can try manually connecting to another backend, for example https://gitea01.opendev.org:3000/openstack/nova | 17:07 |
| fungi | but i expect you'll see the same routing performance since they're all on the same network | 17:08 |
| Elnaz | I'll try it too. | 17:15 |
| Elnaz | galera-install.yml: https://paste.ubuntu.ir/hsnlj Could this error that I am getting now have anything to do with the constraints file that I had a problem with before (that problem of getting the constraints/requirements file was solved by setting a variable to get it from github) | 17:18 |
| noonedeadpunk | um, no, that looks like some weirdness with haproxy management | 17:20 |
| noonedeadpunk | specifically on infra1 | 17:20 |
| Elnaz | vvv: https://paste.ubuntu.ir/fxmry | 17:21 |
| Elnaz | I have st two different network for internal and external: `br-mgmt` and `br-end` | 17:22 |
| Elnaz | there's no route between these to network, completely isolated, is it ok? | 17:22 |
| Elnaz | these two* | 17:22 |
| Elnaz | have set* | 17:23 |
| noonedeadpunk | do you have /run/haproxy.stat on infra1? | 17:23 |
| noonedeadpunk | yes, totally | 17:24 |
| noonedeadpunk | also make sure that haproxy runs there | 17:24 |
| Elnaz | `ls: cannot access '/run/haproxy.stat': No such file or directory` | 17:24 |
| Elnaz | The haproxy service is `Active: active (running)` on infra1. | 17:25 |
| noonedeadpunk | huh | 17:36 |
| noonedeadpunk | I think we're enabling stats socket regardlessly... | 17:37 |
| noonedeadpunk | maybe /var/run/haproxy.stat ? | 17:37 |
| noonedeadpunk | also - is it same on all infra nodes? meaning - is it absent everywhere? | 17:39 |
| noonedeadpunk | or jsut infra1? | 17:39 |
| Elnaz | infra1: `ls: cannot access '/var/run/haproxy.stat': No such file or directory` | 17:40 |
| Elnaz | both `/var/run/haproxy.stat` and `/run/haproxy.stat`exist on the infra2, but not infra1! | 17:41 |
| noonedeadpunk | well. I'd suggest restarting haproxy then | 17:42 |
| Elnaz | noonedeadpunk: Also available on infra3 | 17:42 |
| noonedeadpunk | as it sounds not healthy to me | 17:42 |
| noonedeadpunk | /var/run is likely a symlink to /run | 17:42 |
| noonedeadpunk | restarting on infra1 ofc | 17:43 |
| Elnaz | Ok, (none of the are not a symlink, checked by `ls -l`) | 17:44 |
| noonedeadpunk | I meant directories themselves - on modern OS `/var/run: symbolic link to /run` | 17:52 |
| noonedeadpunk | anyway | 17:52 |
| noonedeadpunk | You should figure out why haproxy don't want to create socket | 17:52 |
| noonedeadpunk | as in config it is defined `stats socket /var/run/haproxy.stat level admin mode 600` | 17:53 |
| noonedeadpunk | I'd assume that haproxy is just dead or does not like it's configuration on infra1 | 17:54 |
| cloudnull | 👋 what's good cloud ? | 18:14 |
| mgariepy | hey cloudnull what's up ? | 18:15 |
| cloudnull | just living the dream ,, you ? | 18:15 |
| mgariepy | mostly need sleep :P | 18:16 |
| cloudnull | again updated my home cloud to the head of master, things are running wonderfully :D | 18:16 |
| cloudnull | mgariepy sleep is overrated | 18:16 |
| mgariepy | yeah you can't die for lack of sleep. | 18:16 |
| mgariepy | so it must be useless | 18:16 |
| cloudnull | right! that's just science | 18:16 |
| cloudnull | mgariepy how's life? doing ok? | 18:18 |
| mgariepy | yep my yougest son just got 5. | 18:18 |
| mgariepy | i'm getting old ;) haha | 18:18 |
| cloudnull | dang!!! 5!? time flys for sure | 18:19 |
| mgariepy | yeah | 18:22 |
| noonedeadpunk | \o/ | 18:28 |
| noonedeadpunk | oh, congrats on that! It means they will go to school soon | 18:30 |
| mgariepy | yeah in september | 18:32 |
| cloudnull | 💯 more time to sleep :D | 18:57 |
| fungi | cloudnull: whoa, good to see you again! glad you're still alive ;) | 19:10 |
| fungi | (at our age that's decreasingly guaranteed) | 19:25 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!