| opendevreview | Roman Krček proposed openstack/kolla-ansible master: Add sysctl role https://review.opendev.org/c/openstack/kolla-ansible/+/912351 | 07:02 |
|---|---|---|
| opendevreview | Roman Krček proposed openstack/kolla-ansible master: Update cell0 database connection https://review.opendev.org/c/openstack/kolla-ansible/+/910924 | 07:11 |
| opendevreview | Roman Krček proposed openstack/kolla-ansible master: Add sysctl role https://review.opendev.org/c/openstack/kolla-ansible/+/912351 | 07:23 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible stable/2023.1: Extend init-runonce.sh script - create addiditonal resources https://review.opendev.org/c/openstack/kolla-ansible/+/914860 | 08:00 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible stable/2023.2: Extend init-runonce.sh script - create addiditonal resources https://review.opendev.org/c/openstack/kolla-ansible/+/914881 | 08:01 |
| opendevreview | Roman Krček proposed openstack/kolla-ansible master: Performance: use filters for service dicts https://review.opendev.org/c/openstack/kolla-ansible/+/914997 | 08:37 |
| mnasiadka | morning | 09:13 |
| SvenKieske | o/ | 09:15 |
| SvenKieske | my list of stuff missing core reviews only grows larger :( https://etherpad.opendev.org/p/KollaWhiteBoard#L67 (and this is far from complete). thanks mnasiadka for taking a look at most of those already. | 09:30 |
| mnasiadka | well, it's not going to really be very smaller this week - we have PTG and I'm busy with other things in Washington DC - so rather next week :) | 09:36 |
| tinne | I'm looking for specific settings for NetApp ONTAP integration in globals.yml for cinder and manila and can't find them in 2023.2 (at least) ... there should be variables like "enable_manila_backend_ontap" or "enable_cinder_backend_ontap_nfs", for example. Is NetApp support deprecated in some sort? Or maybe it's just a failure in my understanding: I just started using kolla-ansible a few days ago, so maybe i need to install | 09:38 |
| tinne | something that i don't know? | 09:38 |
| mnasiadka | I don't believe there's out of the box support for netapp ontap backend in kolla-ansible | 09:41 |
| tinne | Hm okay.. i saw some youtube videos about it and the guy did not mentioned he installed something special or changed something... and there is a "lab on demand" based in Zed at the NetApp Partner Portal which also spins up OpenStack using kolla-ansible including these settings. Strange... | 09:46 |
| tinne | this one for example: https://www.openstack.org/videos/summits/vancouver-2018/containers-openstack-and-netapp-a-private-cloud-trifecta | 09:46 |
| mnasiadka | you can use the standard config override method for cinder to get the backend enabled though | 09:47 |
| tinne | okay thanks ... i suppose that is what he mentions at minute 24:01 when he shows "Changes made to Kolla-Ansible" ... i just thought that these ontap-parameters should exist in the default config like the pure-parameters for PureStorage | 09:49 |
| SvenKieske | mnasiadka: how is the weather in washington? :) | 09:58 |
| SvenKieske | tinne: you might want to read up on kolla customization, the way we do it we don't really need to "support" every openstack feature under the sun, but you can still configure everything: https://docs.openstack.org/kolla-ansible/latest/admin/advanced-configuration.html#openstack-service-configuration-in-kolla | 10:00 |
| SvenKieske | tinne: for a high level reasoning read also: https://docs.openstack.org/kolla-ansible/latest/admin/deployment-philosophy.html#kolla-s-solution-to-customization | 10:00 |
| SvenKieske | we of course could add a convenience switch for netapp ontap, code contributions are always welcome. I guess most devs on our side don't use proprietary storage these days. the most common scenario is to use ceph imho. | 10:01 |
| mnasiadka | SvenKieske: nice, not raining, 20-ish degrees | 10:03 |
| SvenKieske | nice, the weekend here was rather hot for april, even cracked the 30° degree mark in parts of the country I believe. | 10:04 |
| tinne | Thank you, Sven! | 10:05 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: WIP: OpenSearch TLS https://review.opendev.org/c/openstack/kolla-ansible/+/915121 | 10:06 |
| SvenKieske | tinne: you're welcome :) | 10:09 |
| opendevreview | Merged openstack/kolla-ansible stable/zed: Fix broken list concatenation in horizon role https://review.opendev.org/c/openstack/kolla-ansible/+/915018 | 10:23 |
| opendevreview | Merged openstack/kolla-ansible master: Revert "podman: install "rich" dependency" https://review.opendev.org/c/openstack/kolla-ansible/+/902230 | 10:57 |
| kevko | mnasiadka: Czech Republic - 31, Slovakia - 28 :D | 11:00 |
| mnasiadka | kevko: it's 30-ish in Wroclaw where I came from, so maybe it's good that I went to a bit colder area :) | 11:01 |
| kevko | mnasiadka: well, fortunately I am near High Tatras so I have 22 degree :) | 11:08 |
| opendevreview | Verification of a change to openstack/kolla-ansible master failed: CI: bump cirros version https://review.opendev.org/c/openstack/kolla-ansible/+/915117 | 11:12 |
| opendevreview | Roman Krček proposed openstack/kolla-ansible master: Add sysctl role https://review.opendev.org/c/openstack/kolla-ansible/+/912351 | 11:15 |
| opendevreview | Roman Krček proposed openstack/kolla-ansible master: Update cell0 database connection https://review.opendev.org/c/openstack/kolla-ansible/+/910924 | 11:19 |
| opendevreview | Merged openstack/kayobe master: Bump KA Ansible versions to match new defaults https://review.opendev.org/c/openstack/kayobe/+/913571 | 11:27 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: WIP: OpenSearch TLS https://review.opendev.org/c/openstack/kolla-ansible/+/915121 | 11:29 |
| kevko | mnasiadka: SvenKieske: i have something regarding user defined configs ... for example policy file ...user place for example his policy.yaml into /etc/kolla/config/nova-api/policy.yaml .... so kolla-ansible will copy that file into destination server into /etc/kolla/nova-api/policy.yaml ...and will render entry to nova-api's config.json ...so | 11:41 |
| kevko | container will start ...execute kolla_set_configs and script will copy that file from /var/lib/kolla/config_files/..... to /etc/nova/customized_file | 11:41 |
| kevko | mnasiadka: problem is that if user will remove that file from /etc/kolla/config/nova-api ..... and config.json will be re-rendered to omit that customized file ....container will be modified from previous attempt | 11:42 |
| kevko | i think there should be again original default config, shouldn't be ? | 11:42 |
| mnasiadka | we basically have this problem for years - we don't remove files from /etc/kolla/$container that are no longer under kolla/config/ | 11:46 |
| kevko | I propose that during the image building process, we copy the resulting /etc/service directory to /etc/service-default-configs, for example... and take this into account in the kolla_set_configs script... so that it first copies all default configs... and only then copies from /var/lib/kolla/config_files... This will ensure that whatever is defined | 11:46 |
| kevko | in config.json is actually overwritten in /etc/service, while the rest naturally reverts to default. | 11:46 |
| opendevreview | Alex Welsh proposed openstack/kolla-ansible master: Automate prometheus blackbox configuration https://review.opendev.org/c/openstack/kolla-ansible/+/912420 | 11:46 |
| mnasiadka | there was an approach to use synchronize: instead of copying, but that requires rsync | 11:46 |
| kevko | mnasiadka: but this is fixable | 11:46 |
| kevko | mnasiadka: and is it problem ? rsync is standard linux cmd programme | 11:47 |
| mnasiadka | it's another dependency | 11:47 |
| mnasiadka | and instead of discussing it here - add it to the PTG | 11:47 |
| kevko | okay | 11:47 |
| kevko | in a meantime i will try to build base images and check if rsync is not there already ... | 11:47 |
| kevko | mnasiadka: but my proposal don't need rsync :) | 11:48 |
| mnasiadka | well, there are two parts in it | 11:48 |
| mnasiadka | one is the part we leave the files we do not use | 11:48 |
| mnasiadka | second is what you're mentioning - you want to tell me that on container restart we leave policy files in /etc/neutron for example, while the user stopped overriding those? | 11:49 |
| kevko | mnasiadka: why ? for example because they changed his mind ? :D | 11:56 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: WIP: OpenSearch TLS https://review.opendev.org/c/openstack/kolla-ansible/+/915121 | 11:57 |
| kevko | their | 11:57 |
| mnasiadka | kevko: I mean that we don't remove that no longer needed file, so we need proper code in kolla_set_configs that would remove those on restart | 11:57 |
| kevko | mnasiadka: yeah , of course ...btw ..i don't think rsync is problem as dependency ... for debian it's only | 11:59 |
| kevko | The following NEW packages will be installed: | 11:59 |
| kevko | libpopt0 (1.19+dfsg-1) | 11:59 |
| kevko | rsync (3.2.7-1) | 11:59 |
| kevko | 1 mb | 11:59 |
| mnasiadka | yeah, as I said - one thing is we don't remove those files from /etc/kolla/neutron-server (for example) on the destination host, but bigger problem is we don't remove those files - kolla_set_configs should support removing those files from /etc/neutron (for example) | 12:01 |
| mnasiadka | unless we want to delete and recreate the container on config.json change | 12:01 |
| kevko | mnasiadka: i will check both issues | 12:03 |
| kevko | and add to PTG | 12:03 |
| SvenKieske | mhhm | 12:21 |
| SvenKieske | this is a fundamental problem of config mgmt. you need a known clean state to add your new state to it. So imho we should try to enforce a clean rebuild somehow, from a known state. and only then add or remove files/items. not sure rsync is the best tool there, but if it works I think it's okay. Not sure if we run into issues with various templating tasks and symlinks et al. | 12:23 |
| SvenKieske | another option would be to add a "reset_default_state" task to every role, which basically nukes all customizations done by the admin with regards to config files. but that is very very tricky. but it's also tricky if you use rsync. | 12:25 |
| SvenKieske | e.g what if I deployed custom certs and want those removed for services foo and bar, but not baz? | 12:25 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: WIP: OpenSearch TLS https://review.opendev.org/c/openstack/kolla-ansible/+/915121 | 12:27 |
| mnasiadka | SvenKieske: well, maybe we should just recreate instead of restarting | 12:33 |
| SvenKieske | well that might be a solution, but has still issues, e.g. just taking longer to rebuild than to reconfigure. | 12:46 |
| mnasiadka | Kolla vPTG starting here: https://meetpad.opendev.org/kolla-dalmatian-ptg | 13:02 |
| kevko | SvenKieske: that's nothing related to mentioned problem | 13:04 |
| kevko | SvenKieske: because even if you change your cert now and don't want to replace for baz ... You need to use --tags to modify only foo and bar | 13:06 |
| SvenKieske | sure, right. I though you where also talking about modifying customized configs e.g. per host nova config | 13:09 |
| kevko | No, main problem is for me and for now ..that if you once use your own config ..it will stay on host and also in container ... | 13:11 |
| opendevreview | Verification of a change to openstack/kolla-ansible master failed: CI: bump cirros version https://review.opendev.org/c/openstack/kolla-ansible/+/915117 | 13:41 |
| spatel | Folks, I have question about how to debug if container crashing? | 14:23 |
| spatel | Like what is the first setup too look and understand what is wrong? | 14:24 |
| spatel | opensearch_dashboard container keep crashing and restarting and no idea what is wrong because logs not telling any story to start debug | 14:24 |
| spatel | I can't get inside container also to look for evidence | 14:24 |
| SvenKieske | spatel: we are currently doing the PTG meeting so you won't get answers here | 14:50 |
| SvenKieske | also you added a topic there but I think it won't be discussed until you show up there to explain the topic? :) | 14:50 |
| SvenKieske | https://meetpad.opendev.org/kolla-dalmatian-ptg in case you still can join | 14:50 |
| spatel | SvenKieske I am in meeting :) | 15:09 |
| spatel | SvenKieske my question was very general so not sure worth asking in this meeting. I am looking for some basic help to debug stuff of crashing container | 15:34 |
| spatel | I wish we have a some doc around for basic step to follow | 15:34 |
| SvenKieske | spatel: no I meant your topic that was added to the PTG? At least someone with your nickname wrote something down there, I believe. | 15:36 |
| spatel | SvenKieske oh! I see (totally forgot ) | 15:36 |
| SvenKieske | spatel: this one: https://etherpad.opendev.org/p/kolla-dalmatian-ptg#L275 | 15:36 |
| SvenKieske | per serivce rmq, we just decided that we need a volunteer for that, so if you want to provide that code we are happy to take it :) more was not possible with someone presenting the case | 15:37 |
| spatel | SvenKieske I will see if I can come up with some idea to make it possible to deploy rabbitMQ per service. | 15:38 |
| SvenKieske | regarding your above question: did you have a look at the docker logs? "docker logs $container" that should give you a hint why it is crashing | 15:38 |
| spatel | SvenKieske I did look at logs and spiting lots of this - https://paste.opendev.org/show/bXVGJfjbYBnb5CADqNtO/ | 15:41 |
| spatel | I didn't find anything which gives me clue... | 15:41 |
| spatel | How do I disable restart policy for this container so I can get inside and run daemon by hand to see | 15:42 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible stable/2023.1: Extend init-runonce.sh script - create addiditonal resources https://review.opendev.org/c/openstack/kolla-ansible/+/914860 | 15:42 |
| spatel | SvenKieske in end of logs file I saw this - https://paste.opendev.org/show/bHn2Lklr0nNAtlgG5Y2a/ | 15:43 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible stable/2023.2: Extend init-runonce.sh script - create addiditonal resources https://review.opendev.org/c/openstack/kolla-ansible/+/914881 | 15:43 |
| SvenKieske | spatel: did you maybe hit this bug? https://bugs.launchpad.net/kolla-ansible/+bug/2038914 do you use kolla master to deploy? | 15:47 |
| dougszu | spatel: Check the container Docker logs, and also the logs in /var/log/kolla/service.. | 15:48 |
| spatel | hmm I tried 2023.1 image also and same issue | 15:48 |
| spatel | In other DC everything works in 2023.1 but this is new place | 15:48 |
| SvenKieske | mhm weird, it sure works in CI and here, so you must have something differing in your setup | 15:48 |
| spatel | dougszu I have posed above all logs | 15:48 |
| SvenKieske | can you double check if there is any difference in the setup (code wise)? | 15:49 |
| dougszu | doh, was lagging behind | 15:49 |
| SvenKieske | ah wait, need also to check your other paste :) | 15:49 |
| kevko | frickler: you never wrote anything to patching mechanism in kolla - https://review.opendev.org/c/openstack/kolla/+/829295 << so maybe you can check now ...and based on your comments i will leave it abandoned or reopen .... | 15:49 |
| SvenKieske | mhm, that's just info level stuff when importing JS for dashboards, that's not an error | 15:49 |
| kevko | spatel: modify your config.json to sleep infinity ...is your container restarting ? | 15:50 |
| spatel | kevko where should I change - https://paste.opendev.org/show/bUhIAElLsiNwX2K5BEST/ | 15:52 |
| spatel | yes container keep restarting | 15:53 |
| spatel | I want to stop restart so I can debug next setup | 15:54 |
| SvenKieske | spatel: well you can just utilize the docker commands for that? | 15:59 |
| SvenKieske | https://docs.docker.com/reference/cli/docker/container/update/#restart | 16:00 |
| SvenKieske | or use kevko's approach, should work as well | 16:00 |
| spatel | docker update --restart=no opensearch_dashboards | 16:03 |
| spatel | done | 16:03 |
| kevko | spatel: sorry, i am back ... | 16:09 |
| spatel | After setting policy that container disappeared | 16:09 |
| SvenKieske | well it crashed and then it's off :) the "sleep" approach from kevko should work better | 16:10 |
| spatel | let me start from docker ps -a | 16:10 |
| SvenKieske | yes or run interactively | 16:10 |
| kevko | spatel: normally when you don't see any problem in docker logs nor logs for a service i am modifiing config.json | 16:10 |
| spatel | sorry I am little new in docker world so trying to find flags | 16:10 |
| spatel | anyway I have started container from ps -a | 16:11 |
| kevko | spatel: modify original command: "glance-api" for example to _command: "glance-api" << so it will not be read (it's just form of backup what you have before ...) ..then add command: sleep infinity | 16:11 |
| spatel | let me check logs | 16:11 |
| spatel | kevko let me check example of glance | 16:12 |
| kevko | spatel: when you restart a container ..kolla_start will read the command and run ... so sleep infinity will be executed ... if container is running ...that means original command is a problem | 16:12 |
| kevko | spatel: if it is restarting even if you changed to sleep infinity ...you have problem probably somewhere between kolla-start and some extend-start script ...then i am doing something like docker restart service; docker cp /usr/local/bin/kolla_start /tmp/ ; edit /tmp/kolla_start and add sleep infinity somewhere on the beginning ; docker restart | 16:14 |
| kevko | service; docker cp /tmp/kolla_start service:/usr/local/bin/kolla_start and then you are in state before run the command and you can debug what is going on | 16:14 |
| spatel | kevko you meant to say this ? - https://paste.opendev.org/show/bZAnCPVS5dMVEsnYqyQm/ | 16:14 |
| kevko | spatel: this one -> https://paste.openstack.org/show/bJLrBmXo8MXBgLmM3jSP/ | 16:15 |
| spatel | oh! so let me try this with opensearch_dashborad container | 16:16 |
| kevko | spatel: yeah ! | 16:16 |
| kevko | spatel: if you restart your opensearch_dashboard container and check watch -n 1 "docker ps | grep opensearch_dashboard" ..you will see if restarting or not | 16:16 |
| spatel | k | 16:17 |
| SvenKieske | kevko: might be worth to add your example as a debug doc to kolla? :) | 16:17 |
| SvenKieske | so we can just point people to a docs URL :D | 16:17 |
| kevko | SvenKieske: Haha, personally I never read a documentation because it's visible on first seen :D | 16:18 |
| spatel | agreed SvenKieske it should be here - https://docs.openstack.org/kolla-ansible/latest/user/troubleshooting.html | 16:18 |
| spatel | how to debug crashing or restarting container :) | 16:18 |
| SvenKieske | maybe spatel might contribute that as a docs patch? :) | 16:18 |
| spatel | +1 | 16:18 |
| SvenKieske | kevko: well yeah, for experienced people the docs are useless, but we have quite some beginners in the audience I think :) | 16:19 |
| kevko | spatel: I am open to answer all debugging questions ...but I really don't have a time most of the time ... | 16:19 |
| kevko | especially when i need to amend some patch for 50 times until i get 2 +2 :D | 16:19 |
| spatel | kevko container is up last 1 minute so look like its not going to restart.. now let me get inside and debug why opensearch doesn't like it | 16:19 |
| SvenKieske | kevko: at the very least docker debug 101 docs save time when people ask questions, you just post the link then ;) | 16:19 |
| kevko | spatel: well, now you can enter the container and run your command manually ! | 16:19 |
| kevko | spatel: from the config.json | 16:20 |
| spatel | kevko doing it | 16:20 |
| kevko | spatel: also, you can see that your actual command which is "sleep infinity" is in /run_command .... so cat /run_command | 16:20 |
| SvenKieske | to add to that: sometimes you need to care for the user being used etc, if you don't enter the container as root you should be fine most of the time. | 16:21 |
| kevko | spatel: entrypoint for all containers is dumbinit which exec kolla_start placed in /usr/local/bin | 16:21 |
| spatel | cat /run_command only showing sleep infinity | 16:21 |
| spatel | but I grab it from /etc/kolla/opensearch-dashboards/config.json | 16:22 |
| SvenKieske | yeah that should be the command you just patched, kevko just tried to explain the general concept, I think | 16:22 |
| kevko | spatel: yeah, it was just a side note ...that command from config.json is then written into this path and those path is read by kolla_start and passed to dumibit as argument | 16:22 |
| kevko | spatel: question is ...what you can see from a manual execution ? | 16:22 |
| spatel | here is the error - https://paste.opendev.org/show/b5NiSpxsE9Udvx7Gpcon/ | 16:23 |
| kevko | spatel: ok, what if you exec into container as root ? | 16:23 |
| kevko | spatel: so docker exec -itu root opensearch_dashboard .... | 16:24 |
| kevko | spatel: i would say that it will work | 16:24 |
| spatel | its saying you can't run it with root user | 16:24 |
| kevko | spatel: did you exec as root ? | 16:24 |
| spatel | trying this option - /usr/share/opensearch-dashboards/bin/opensearch-dashboards --config /etc/opensearch/opensearch_dashboards.yml --allow-root | 16:24 |
| opendevreview | Merged openstack/kolla-ansible master: CI: bump cirros version https://review.opendev.org/c/openstack/kolla-ansible/+/915117 | 16:25 |
| spatel | here is the logs - https://paste.opendev.org/show/bpZZWuUPxVNMB5rkkvnA/ | 16:25 |
| kevko | spatel: opensearch-dashboard is running as user opensearch-dashboards | 16:25 |
| SvenKieske | kevko: I'm pretty sure opensearch checks actively that it's _not_ run as root | 16:25 |
| SvenKieske | that's why I added the tip above to check to not run as root, afaik opensearch only runs under opensearch user | 16:25 |
| kevko | https://github.com/openstack/kolla/blob/3e502dc34dd4bd2b643c8131839d2853d855171d/docker/opensearch/opensearch-dashboards/Dockerfile.j2#L23 | 16:25 |
| kevko | SvenKieske: is it this container image ? opensearch-dashboards | 16:26 |
| spatel | I did su - opensearch-dashboards | 16:26 |
| spatel | and then try to run command and its showing permission error | 16:26 |
| kevko | spatel: try docker exec -itu opensearch-dashboards opensearch_dashboard | 16:26 |
| spatel | ok | 16:26 |
| kevko | spatel: containers are debian family or ugly redhat family ? | 16:27 |
| spatel | ubuntu 22.04 | 16:27 |
| spatel | This is what I did, by default it use opensearch_dashborad user if don't specify (-u) | 16:28 |
| spatel | https://paste.opendev.org/show/bjGzgDywuWYIBHwFLTkI/ | 16:28 |
| spatel | (opensearch-dashboards)[opensearch-dashboards@mgmt1 /]$ | 16:29 |
| SvenKieske | what are the permissions on that file? | 16:29 |
| spatel | that file own by opensearch | 16:30 |
| spatel | I think i should su - opensearch ? | 16:30 |
| SvenKieske | no, don't use su | 16:30 |
| spatel | ok | 16:31 |
| SvenKieske | docker uses user namespaces to separate user between hosts and containers, you need to specify the user when you enter the docker container, if it is not already specified in the container image - which it is | 16:31 |
| kevko | spatel: it should be owned by opensearch-dashboards I would say | 16:31 |
| SvenKieske | yes | 16:31 |
| kevko | SvenKieske: it is | 16:31 |
| SvenKieske | id uid=42492(opensearch-dashboards) gid=42492(opensearch-dashboards) groups=42492(opensearch-dashboards),42400(kolla) | 16:32 |
| kevko | https://github.com/openstack/kolla/blob/3e502dc34dd4bd2b643c8131839d2853d855171d/docker/opensearch/opensearch-dashboards/Dockerfile.j2#L23 | 16:32 |
| spatel | look at this permission - https://paste.opendev.org/show/bmxC2MMDPuYAIGkQeHlP/ | 16:32 |
| SvenKieske | 7077977 4.0K -rw-r----- 1 42492 42492 354 Dec 28 11:15 /etc/opensearch-dashboards/opensearch_dashboards.yml | 16:32 |
| kevko | spatel: bad user | 16:32 |
| SvenKieske | yeah, that's somehow wrong in your setup | 16:32 |
| spatel | hmm? | 16:32 |
| spatel | all I did opensearch_enable: yes and run playbooks | 16:33 |
| SvenKieske | the owner of the yaml file needs to be like I posted above, opensearch-dashboards, not opensearch | 16:33 |
| kevko | +1 | 16:33 |
| spatel | You are right in other DC I can see permission is opensearch-dashboards | 16:34 |
| SvenKieske | which openstack release is this again? But I doubt it's an error in the container build. | 16:34 |
| spatel | so what went wrong? I am running zed release of openstack | 16:34 |
| SvenKieske | so maybe someone did a chmod manually there? | 16:34 |
| spatel | I doubt.. because container is keep restarting and no way anyone get inside and manually type any command | 16:34 |
| spatel | let me change permission and give it a try | 16:35 |
| kevko | spatel: so how it is possible there is another user on another host ? :D | 16:35 |
| spatel | I am clueless :) | 16:35 |
| kevko | spatel: i would say it will start work | 16:35 |
| spatel | hang on.. let me try to switch permission | 16:35 |
| SvenKieske | rather chown | 16:36 |
| kevko | +1 | 16:36 |
| SvenKieske | there was some foo upstream (so in opensearch) about wrong permissions some time ago, but afaik that was really about permissions, not wrong user | 16:40 |
| SvenKieske | afaik that was this issue (https://github.com/opensearch-project/OpenSearch/issues/8821) so not related. but I seem to remember some bug somewhere with regards to a wrong user being used.. but I can't find that atm. | 16:41 |
| SvenKieske | spatel: it would be best to double/triple check if your observed behaviour is really an issue with vanilla/unpatched kolla or if a colleague maybe messed up the file owners, errors can happen :) | 16:42 |
| SvenKieske | grepped our code for chown doesn't at least yield many results that look like they could fiddle with this. | 16:43 |
| spatel | SvenKieske yes I will try to get to bottom of the issue.. how this happened and from where this permission coming | 16:43 |
| SvenKieske | spatel: thanks, at least it has worked some time ago in your other deployment it seems, so maybe you can go from there and check what changed since then. | 16:44 |
| kevko | SvenKieske: replied | 16:44 |
| kevko | spatel: so is it working now ? | 16:45 |
| kevko | SvenKieske: check also the patch in a chain ..that macro is used there | 16:46 |
| spatel | fixed all other mission but stuck here and trying to find this file to change permission - https://paste.opendev.org/show/bCmT8U10kzcpp1B6lREQ/ | 16:46 |
| *** edebeste3 is now known as edebeste | 16:47 | |
| SvenKieske | kevko: sorry, regarding the patch containers patch: I meant I don't know what the history is with our downstream implementation: I'm pretty sure our downstream was not motivated by the dropping of binary images, afaik we never used those. but we always had a need to do patches (sadly) because upstream didn't accept all our patches/improvements. | 16:47 |
| SvenKieske | so I guess that was just a misunderstanding, I a have not given enough context in my answer I guess, to make that clearer. hope it is clearer now. | 16:47 |
| *** mmalchuk_ is now known as mmalchuk | 16:48 | |
| spatel | kevko Thanks for that tips. Now let me debug and see what is going on. I will redeploy container and let you know | 16:48 |
| kevko | SvenKieske: Well, I have exactly the same experience ! | 16:48 |
| kevko | SvenKieske: that's the reason why I am cherry-picking those two commits from release to release ...and working very nice during serveral releases | 16:50 |
| kevko | SvenKieske: in container there is also an entry what was patched | 16:50 |
| kevko | (designate-central)[root@controller0 /]# cat /etc/kolla/patched | 16:50 |
| kevko | [i] Applied /patches/fix-old-db-migrations-because-of-proxysql.patch | 16:50 |
| SvenKieske | that's actually neat | 16:52 |
| SvenKieske | I replied again on the patch with some more detail and also some opinion on the patch itself. | 16:52 |
| SvenKieske | kevko: I also addressed now the old "-1" comment with a direct reply to that, so we have it written down that waiting for tarballs is not really a solution. | 16:55 |
| kevko | thanks | 17:17 |
| SvenKieske | mnasiadka, kevko, frickler: as far as I can see we just install whatever comes from repositories when it comes to redis, so we have no version pinned, but I guess upstream distros will soon(?) remove redis or replace it, so either we need to adapt to something new or we can just wait and see what happens | 17:33 |
| SvenKieske | but I'm not sure I overlooked something, I'll double check tomorrow. | 17:33 |
| kevko | SvenKieske: hold your horses ... i don't think it will be soon | 17:42 |
| SvenKieske | yeah sure, I just wanted to already take a look how large a change we would need. so assuming redis get's removed upstream I guess we could get away with just renaming packages. worst case we need to configure the repository ourselves, as it might be the case that different distros merge different redis contenders | 17:45 |
| SvenKieske | kevko: not sure if you were present during the redis discussion on PTG? it was an action item to check if we need to pin redis version to avoid pulling non-free versions from upstream. | 17:45 |
| SvenKieske | so I just did that (80% of it, need to check other parts of source if we don't have some custom magic somewhere, but I doubt it) | 17:46 |
| kevko | SvenKieske: redis is installed from package ... debuntu will not break licensing | 17:46 |
| SvenKieske | kevko: yes, so that's fine :) | 17:46 |
| *** mmalchuk_ is now known as mmalchuk | 17:47 | |
| kevko | SvenKieske: to be honest ..i don't see alternative for redis currently | 17:48 |
| kevko | it works really good | 17:48 |
| kevko | i don't want to see etcd in production ...and zookeeper ? ble | 17:48 |
| SvenKieske | kevko: yes that seems also to be the preference for now, see: https://etherpad.opendev.org/p/kolla-dalmatian-ptg#L252 | 17:52 |
| SvenKieske | but it's sensible to revert the zookeeper removal if all redis forks are going to explode somehow :D | 17:52 |
| SvenKieske | I have not really experience with zookeeper to comment on it. | 17:53 |
| SvenKieske | but for now the idea is to just use a redis fork and watch which one "wins" there. I guess/hope we have still time to watch what distros will do.. | 17:53 |
| SvenKieske | this reminds me of reenabling my subscription to fedoral-devel and their newer forum, you get quite some insights there about redhat upstream development | 17:54 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!