| jingvar | Is there an instruction on how to restart the control plan after its full stop? | 06:47 |
|---|---|---|
| yoctozepto | jingvar: not really; just power on all the control plane nodes and run mariadb recovery | 06:58 |
| yoctozepto | that works most of the time | 06:58 |
| jingvar | I think mariadb must be recovered. Is it automated? or I have to go into container and manually set bootstrap flag for maria | 06:58 |
| jingvar | a classic solution is - stop ctl3, ctl2, ctl1 and startup in backorder | 07:00 |
| yoctozepto | jingvar: there is a command in kolla ansible that does the recovery, it's named accordingly ;-) | 07:01 |
| yoctozepto | just like you would do a deploy, you do a mariadb_recovery | 07:02 |
| jingvar | if you have an issues with galera, find last shutdowned node, stop others , set bootstarp flag, start ьныйд | 07:02 |
| jingvar | thanks going to read the doc | 07:03 |
| yoctozepto | jingvar: yeah, that's more or less what it does | 07:27 |
| opendevreview | Merged openstack/kolla-ansible master: Trivial fix nova's healthchecks https://review.opendev.org/c/openstack/kolla-ansible/+/804381 | 07:55 |
| opendevreview | Radosław Piliszek proposed openstack/kolla-ansible stable/wallaby: Trivial fix nova's healthchecks https://review.opendev.org/c/openstack/kolla-ansible/+/804433 | 08:13 |
| opendevreview | Radosław Piliszek proposed openstack/kolla-ansible stable/victoria: Trivial fix nova's healthchecks https://review.opendev.org/c/openstack/kolla-ansible/+/804434 | 08:14 |
| rohit02 | yoctozepto: mgoddard: while deploying kolla victoria we are facing this issue https://paste.opendev.org/raw/808053/ | 09:43 |
| opendevreview | Michal Nasiadka proposed openstack/kolla master: haproxy: Use haproxy 2.2.x from nfv-extras repository https://review.opendev.org/c/openstack/kolla/+/804513 | 09:46 |
| pomac | Anyone that has had issues with docker containers where they somehow stop consulting their own /etc/passwd and sudo -E starts failing? | 12:24 |
| pomac | (kolla_toolbox f.ex. fails we have 3 nodes with this issue and redeploying containers doesn't help, even reinstalling the whole docker thing from scratch hasn't helped - this also affects our cinder nodes but kolla_toolbox felt like a easier target) | 12:25 |
| pomac | exact error: | 12:26 |
| pomac | + sudo -E kolla_set_configs | 12:26 |
| pomac | sudo: you do not exist in the passwd database | 12:26 |
| pomac | --- | 12:26 |
| pomac | Doing docker run <image:tag> -it bash and then running the command - it works (except for the missing files) | 12:27 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: Fix neutron upgrade when running with limit https://review.opendev.org/c/openstack/kolla-ansible/+/804384 | 12:28 |
| kevko | pomac: docker exec -itu root container bash << ? | 12:28 |
| pomac | kevko: hard to do when it doesn't start =) | 12:29 |
| opendevreview | Radosław Piliszek proposed openstack/kolla-ansible master: Allow override of rabbitmq config in kolla toolbox https://review.opendev.org/c/openstack/kolla-ansible/+/785595 | 12:29 |
| kevko | hmm | 12:29 |
| pomac | kevko: from what i can see, the passwd file is correct - several similar issues has been reported to docker - no soultion seems to have been found (some say restart docker but that didn't help.. ;)) | 12:30 |
| kevko | yoctozepto: i reworked patch you reviewed for me yesterday, and I realized that fix could be simple. I've just moved few common tasks and add some limiting ..could you check if my ideas are correct ? | 12:31 |
| pomac | kevko: (current experiment is reinstalling machine from scratch and see if that can get it working,... but it's quite cumbersome) | 12:32 |
| kevko | pomac: what happened before ? | 12:33 |
| pomac | kevko: nothing that we've been able to determine... The interesting part is removing docker and rm -rf:ing /var/lib/docker didn't help, it was still broken after redeployment | 12:34 |
| pomac | kevko: there are some "reuse" issues with docker and images/volumes - we hoped to avoid that but... | 12:34 |
| kevko | pomac: container name ? | 12:35 |
| kevko | and btw what distro using ? binary source ? | 12:35 |
| kevko | i mean ..for docker images | 12:35 |
| pomac | kevko: centos, source and kolla_toolbox, cinder-backup and cinder-volume | 12:35 |
| pomac | it worked "before" with the exact same images - just suddenly stoped working... | 12:36 |
| kevko | pomac: what happens if you run same container with different name ..same issue ? | 12:36 |
| kevko | pomac: docker run --rm -i -v /var/run/docker.sock:/var/run/docker.sock nexdrew/rekcod cinder_volume | 12:37 |
| pomac | kevko: need to know the dump-init part | 12:41 |
| kevko | pomac: just edit --entrypoint "dumb-init --single-child --" to --entrypoint "dumb-init" | 12:42 |
| kevko | and change name to avoid conflicts | 12:43 |
| pomac | kevko: doing this with kolla_toolbox - named it kolla_toolbox2 and it goes "no such file or directory"... | 12:43 |
| pomac | kevko: so, as i understand it docker run --rm -i -v <socket> <image> <name> - but $name is used as an argument? | 12:45 |
| kevko | pomac: see https://paste.opendev.org/show/808056/ | 12:46 |
| kevko | pomac: second command ..check --name changed | 12:46 |
| opendevreview | Radosław Piliszek proposed openstack/kolla-ansible master: Fix kolla-toolbox with IPv6 and disabled RabbitMQ https://review.opendev.org/c/openstack/kolla-ansible/+/804522 | 12:48 |
| pomac | + sudo -E kolla_set_configs | 12:49 |
| pomac | sudo: you do not exist in the passwd database | 12:49 |
| pomac | kevko: so, no, same issue ^ | 12:49 |
| kevko | pomac: do you using nscd ? | 12:49 |
| pomac | we have these exact containers running on hundreds of machines so it's really odd that they don't work on these.. | 12:49 |
| pomac | Oh! | 12:49 |
| pomac | I'll check! | 12:49 |
| pomac | kevko: I had no clue it was running, GOOD CATCH! | 12:51 |
| kevko | so ? | 12:51 |
| pomac | kevko: what do you think?! ;) | 12:52 |
| kevko | pomac: i think it is solved :D | 12:52 |
| pomac | kevko: second time i've been burned by nscd but the first time it had crashed and on a comepletely different system =) | 12:52 |
| pomac | kevko: yep | 12:53 |
| kevko | haha ... i knew it :D | 12:53 |
| kevko | but firstly i wanted to check with you if new container from same image will be same | 12:53 |
| yoctozepto | kevko: you would have to point me to the change you are talking about because I can't recall | 12:54 |
| kevko | that rekcod thing | 12:54 |
| pomac | kevko: We have everything scripted - the whole setup so there is also a question of why it's running on these machines but not the others but hey... thanks! | 12:55 |
| kevko | no problem | 12:55 |
| kevko | pomac: well, scripts are written by humans :D | 12:55 |
| pomac | kevko: Yeah, and i didn't do them ... It all seems so obvious in hindsight but =) | 12:57 |
| opendevreview | Radosław Piliszek proposed openstack/kolla-ansible master: Allow override of rabbitmq config in kolla toolbox https://review.opendev.org/c/openstack/kolla-ansible/+/785595 | 12:57 |
| pomac | kevko: cold be that some newer hp sw activated them (for some reason my colleagues has that running on them now) | 12:57 |
| HN | Hi everyone, anyone know how to inject a password to Windows Server Instance in Openstack Horizon ? I've tested but it's not work for administrator, only new user work fine . Here some information I did : https://paste.opendev.org/show/brXVnFRuYMnRLoACnngr/ | 13:03 |
| kevko | no idea | 13:06 |
| kevko | it is currently long time i was playing with cloudbase-init and windows ..and i am glad that it is in the past :D | 13:07 |
| HN | =( | 13:09 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible stable/wallaby: Use Docker healthchecks for rabbitmq services https://review.opendev.org/c/openstack/kolla-ansible/+/804436 | 13:11 |
| opendevreview | Radosław Piliszek proposed openstack/kolla-ansible master: CI: Test Swift upgrades https://review.opendev.org/c/openstack/kolla-ansible/+/722798 | 13:19 |
| opendevreview | Radosław Piliszek proposed openstack/kolla-ansible master: [CI] Test Swift upgrades https://review.opendev.org/c/openstack/kolla-ansible/+/722798 | 13:21 |
| opendevreview | Skylar Tristan Kelty proposed openstack/kolla-ansible master: Add SAML 2 federation support https://review.opendev.org/c/openstack/kolla-ansible/+/804524 | 13:31 |
| opendevreview | Skylar Tristan Kelty proposed openstack/kolla-ansible master: Add SAML 2 federation support https://review.opendev.org/c/openstack/kolla-ansible/+/804524 | 13:37 |
| opendevreview | Radosław Piliszek proposed openstack/kolla-ansible master: Optimize reconfigure action for swift https://review.opendev.org/c/openstack/kolla-ansible/+/696525 | 14:12 |
| opendevreview | Radosław Piliszek proposed openstack/kolla-ansible master: [CI] Test Swift upgrades https://review.opendev.org/c/openstack/kolla-ansible/+/722798 | 15:27 |
| opendevreview | Radosław Piliszek proposed openstack/kolla-ansible master: Modernize the swift role https://review.opendev.org/c/openstack/kolla-ansible/+/797498 | 15:33 |
| opendevreview | Radosław Piliszek proposed openstack/kolla-ansible master: Optimize reconfigure action for swift https://review.opendev.org/c/openstack/kolla-ansible/+/696525 | 15:45 |
| opendevreview | Pierre Riteau proposed openstack/kayobe master: dnf: add metalink argument to custom dnf repository https://review.opendev.org/c/openstack/kayobe/+/802703 | 15:48 |
| opendevreview | Pierre Riteau proposed openstack/kayobe master: Add support for apt proxy setting https://review.opendev.org/c/openstack/kayobe/+/799629 | 15:53 |
| kevko | yoctozepto: here ? | 16:33 |
| opendevreview | Merged openstack/kayobe stable/victoria: Sync Kolla Ansible feature flags https://review.opendev.org/c/openstack/kayobe/+/803358 | 19:45 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: Use Docker healthchecks for keystone-fernet container https://review.opendev.org/c/openstack/kolla-ansible/+/804546 | 20:35 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: Use Docker healthchecks for keystone-fernet container https://review.opendev.org/c/openstack/kolla-ansible/+/804546 | 21:49 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!