| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible stable/2023.1: WIP: Run slurp upgrade job https://review.opendev.org/c/openstack/kolla-ansible/+/904492 | 08:14 |
|---|---|---|
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: CI: add a message for fluentd string match function https://review.opendev.org/c/openstack/kolla-ansible/+/904677 | 08:22 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible stable/2023.1: WIP: Run slurp upgrade job https://review.opendev.org/c/openstack/kolla-ansible/+/904492 | 08:25 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: WIP: fluentd: Fix xinetd log parsing https://review.opendev.org/c/openstack/kolla-ansible/+/904679 | 08:38 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: WIP: fluentd: Fix xinetd log parsing https://review.opendev.org/c/openstack/kolla-ansible/+/904680 | 08:40 |
| kevko | mnasiadka: I have a dilema, CADF auditing is turned on when api-paste.ini file is modified on pipeline keystone = .* authtoken [app:something] ( added audit filter into pipeline)...most of the time api-paste.ini file is not changed by the user and most of the projects has api-paste.ini file in kolla images and there is no option to change it ... | 08:47 |
| kevko | there is only cinder,cyborg,venus,manila,neutron,barbican,masakari and nova ... in kolla-ansible it works like .. if some custom api-paste from user is found ..it's copied to container .. if not ..there is image burned api-paste used .... moreover masakari and cyborg has it in kolla-ansible (i really don't know if anyone is tracking potentional | 08:47 |
| kevko | changes in those projects ... ).... | 08:47 |
| kevko | mnasiadka: question is - move masakari and cyborg to image and add same option for user to replace custom api-paste if some found .... | 08:47 |
| kevko | mnasiadka: but for CADF it is scriptable ... turn on cadf ..add audit word and config ...if turn off ...remove it ... | 08:48 |
| kevko | mnasiadka: and if there is a user defined api-paste ...don't do anything ... but i am not sure if this combination can be accepted ..or just drop it and leave it as it is ... second option is just not comfortable :D ...because i need to check potentional api-paste changes for projects and change it from version to version .... wdyt ? | 08:49 |
| mnasiadka | so, let me get this correct - if we don't supply any api-paste.ini - the service will use the default | 08:50 |
| mnasiadka | one feature is allowing for user-supplied api-paste.ini | 08:50 |
| mnasiadka | second one is enabling CADF | 08:51 |
| kevko | mnasiadka: yes | 08:52 |
| kevko | and this is dilema :) | 08:52 |
| mnasiadka | well, if you want to do both - I'm fine with that :) | 08:53 |
| kevko | mnasiadka: allow user-supplied api-paste.ini is feature we definitely should provide for users ... this feature exist in some roles ..and in some roles not | 08:53 |
| mnasiadka | so that one should be easy to add | 08:53 |
| kevko | mnasiadka: yuchuu, ok ...let me do it ..i will handle it in scripts | 08:53 |
| mnasiadka | but if you want to enable cadf - then user supplied would be a problem - right? | 08:54 |
| kevko | mnasiadka: btw ...this one ... https://docs.openstack.org/keystonemiddleware/latest/audit.html | 08:54 |
| kevko | mnasiadka: well, script for turn on cadf can just check if there is /var/lib/config_files/api-paste.ini and if exist ...that means it is custom api-paste ..and even if option for cadf is turned on ..it can just log a warning that it is not going to be changed ...if it is default ...it will change and automatically turn on | 08:55 |
| kevko | that's it | 08:55 |
| kevko | point is that i think most of users even don't know what api-paste is :D | 08:55 |
| kevko | and second thing is ...it is not changing that much ...actually it is not changed several versions ...most of the time only when some new version of api is proposed ...v2 vs v3 etc ...but from time to time it is ... | 08:56 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible stable/yoga: fluentd: remove deprecated parameter https://review.opendev.org/c/openstack/kolla-ansible/+/904518 | 08:57 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible stable/yoga: fluentd: remove deprecated parameter https://review.opendev.org/c/openstack/kolla-ansible/+/904518 | 08:57 |
| kevko | so - that was the reason i was thinking about some automagic | 08:57 |
| mnasiadka | well, we can go the simple way - if there's user provided and cadf is enabled - we fail that either this or that | 08:58 |
| kevko | mnasiadka: let my try several approaches and i will propose something ... this is feature i am merging in downstream repos for 6 versions or something like that ..and another customer wants cadf also :D | 08:59 |
| kevko | mnasiadka: and i want to rework it and send upstream to just forget on it :D | 09:01 |
| kevko | mnasiadka: btw, i also noticed that we are copying octavia/etc/* to /etc/octavia/* but as you can see here https://github.com/openstack/octavia/tree/master/etc <<< it is different format inside the directory as another projects ... so actually we are not providing any default as it is in another projects ... | 09:04 |
| mnasiadka | well, I'm sure you'll find a lot of differences ;) | 09:04 |
| kevko | mnasiadka: hahaa , i am sure too ... now i have several months to again improve k and k-a because we upgraded some big deployment and need to go further .. | 09:05 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible stable/yoga: [yoga-only] opensearch: remove kolla_container_engine https://review.opendev.org/c/openstack/kolla-ansible/+/904704 | 09:35 |
| opendevreview | Mark Goddard proposed openstack/kolla stable/2023.2: rabbitmq: Use timeout in healthcheck script https://review.opendev.org/c/openstack/kolla/+/904519 | 09:37 |
| opendevreview | Mark Goddard proposed openstack/kolla stable/2023.1: rabbitmq: Use timeout in healthcheck script https://review.opendev.org/c/openstack/kolla/+/904520 | 09:38 |
| opendevreview | Mark Goddard proposed openstack/kolla stable/zed: rabbitmq: Use timeout in healthcheck script https://review.opendev.org/c/openstack/kolla/+/904721 | 09:38 |
| opendevreview | Mark Goddard proposed openstack/kolla stable/yoga: rabbitmq: Use timeout in healthcheck script https://review.opendev.org/c/openstack/kolla/+/904722 | 09:38 |
| opendevreview | Mark Goddard proposed openstack/kolla-ansible stable/2023.2: Make designate bind9 cmdline configurable https://review.opendev.org/c/openstack/kolla-ansible/+/904723 | 09:39 |
| opendevreview | Mark Goddard proposed openstack/kolla-ansible stable/2023.1: Make designate bind9 cmdline configurable https://review.opendev.org/c/openstack/kolla-ansible/+/896336 | 09:40 |
| opendevreview | Mark Goddard proposed openstack/kolla-ansible stable/zed: Make designate bind9 cmdline configurable https://review.opendev.org/c/openstack/kolla-ansible/+/896337 | 09:40 |
| opendevreview | Mark Goddard proposed openstack/kolla-ansible stable/yoga: Make designate bind9 cmdline configurable https://review.opendev.org/c/openstack/kolla-ansible/+/896338 | 09:41 |
| opendevreview | Mark Goddard proposed openstack/kolla-ansible stable/yoga: Make designate bind9 cmdline configurable https://review.opendev.org/c/openstack/kolla-ansible/+/896338 | 09:42 |
| opendevreview | Mark Goddard proposed openstack/kolla-ansible stable/yoga: Make designate bind9 cmdline configurable https://review.opendev.org/c/openstack/kolla-ansible/+/896338 | 09:43 |
| opendevreview | Mark Goddard proposed openstack/kolla-ansible stable/2023.2: post-2023.1: Remove keystone admin endpoint bits https://review.opendev.org/c/openstack/kolla-ansible/+/904724 | 09:47 |
| kevko | mnasiadka: nah, somewhere we are providing default configs ..somewhere no :( | 09:48 |
| kevko | mnasiadka: i mean, oslo generated ...but still | 09:50 |
| opendevreview | Alex Welsh proposed openstack/kolla-ansible stable/2023.2: Remove nova cell sync comment https://review.opendev.org/c/openstack/kolla-ansible/+/904725 | 09:53 |
| opendevreview | Alex Welsh proposed openstack/kolla-ansible stable/2023.1: Remove nova cell sync comment https://review.opendev.org/c/openstack/kolla-ansible/+/904726 | 09:54 |
| opendevreview | Alex Welsh proposed openstack/kolla-ansible stable/zed: Remove nova cell sync comment https://review.opendev.org/c/openstack/kolla-ansible/+/904727 | 09:54 |
| opendevreview | Alex Welsh proposed openstack/kolla-ansible stable/yoga: Remove nova cell sync comment https://review.opendev.org/c/openstack/kolla-ansible/+/904728 | 09:54 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: CI: Use 2023.2 as previous_release https://review.opendev.org/c/openstack/kolla-ansible/+/904706 | 09:56 |
| opendevreview | Michal Arbet proposed openstack/kolla master: Fix openstack CADF audit maps and installation https://review.opendev.org/c/openstack/kolla/+/904576 | 10:04 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: loadbalancer: Run LE bits in haproxy_run only when it's enabled https://review.opendev.org/c/openstack/kolla-ansible/+/904711 | 10:09 |
| opendevreview | Mark Goddard proposed openstack/kolla-ansible stable/2023.2: post-2023.1: Remove keystone admin endpoint bits https://review.opendev.org/c/openstack/kolla-ansible/+/904724 | 10:13 |
| opendevreview | Mark Goddard proposed openstack/kolla-ansible master: ironic: Remove enable_ironic_pxe_uefi bits https://review.opendev.org/c/openstack/kolla-ansible/+/904575 | 10:21 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: loadbalancer: Run LE bits in haproxy_run only when it's enabled https://review.opendev.org/c/openstack/kolla-ansible/+/904711 | 10:36 |
| kevko | mnasiadka: btw, do you have some specific reason why are u trying to modify LE ? | 10:49 |
| kevko | mnasiadka: did you find some bug ? | 10:50 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible stable/2023.1: Revert "CI: retry smoke tests and instance creation" https://review.opendev.org/c/openstack/kolla-ansible/+/904729 | 10:57 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible stable/2023.1: Revert "CI: retry smoke tests and instance creation" https://review.opendev.org/c/openstack/kolla-ansible/+/904729 | 10:58 |
| mnasiadka | kevko: no, trying to use it and looking at the code once again - not saying it doesn't work ;) | 10:59 |
| kevko | mnasiadka: i really remember that was tricky somehow ..but don't remember details .... but i tried all options ... | 11:25 |
| kevko | mnasiadka: maybe switch from user certificates to LE or vice-versa ... | 11:25 |
| kevko | mnasiadka: i am 80 percent sure that this will create some type of bug ... | 11:26 |
| opendevreview | Merged openstack/kolla-ansible master: Use service-images-pull role for letsencrypt and venus https://review.opendev.org/c/openstack/kolla-ansible/+/904587 | 11:30 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible stable/2023.2: Use service-images-pull role for letsencrypt and venus https://review.opendev.org/c/openstack/kolla-ansible/+/904730 | 11:36 |
| opendevreview | Merged openstack/kolla-ansible stable/2023.2: magnum: Disable CAPI driver when kubeconfig missing https://review.opendev.org/c/openstack/kolla-ansible/+/904511 | 11:56 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible stable/yoga: fluentd: remove deprecated parameter https://review.opendev.org/c/openstack/kolla-ansible/+/904518 | 12:15 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible stable/2023.1: CI: Run slurp upgrade job (upgrade Y-A) https://review.opendev.org/c/openstack/kolla-ansible/+/904492 | 12:24 |
| kevko | I was asking in openstack-lbaas , but i am curious if anyone saw this | 12:30 |
| kevko | Hi, we were upgrading openstack with octavia from version Wallaby to version Xena and we were using provider = octavia (which was in wallaby -> octavia.api.drivers.amphora_driver.v1.driver:AmphoraProviderDriver) BUT in xena provider octavia is changed (octavia.api.drivers.amphora_driver.v2.driver:AmphoraProviderDriver) . Problem is that provider | 12:30 |
| kevko | value is saved in Database, so manipulating with LB stopped work as loadbalancers are v1 but octavia project point provider octavia to v2 .... what we've done was that we patched provider octavia in entrypoints from v2 to v1 .... how to deal with it ... I think THIS IS A BUG ..as default is amphorav2 and octavia has to be backward compatible and | 12:30 |
| kevko | should point to v1 .... | 12:30 |
| kevko | Release note is saying that there is an alias amphorav1 ...but i think this can't work ..or it means that I need to fix provider = octavia to provider = amphorav1 in database ... | 12:30 |
| kevko | So question is, is it bug ? How can I deal with it without entrypoints patch we've made | 12:30 |
| kevko | mnasiadka: btw, did you consider to use KIND as management cluster for magnum CAPI ? | 12:41 |
| kevko | mnasiadka: https://kind.sigs.k8s.io/ | 12:41 |
| mnasiadka | I'm not looking at Kubernetes until I come back from my January vacation, that's yet a bigger mess than OpenStack ;) | 12:42 |
| kevko | mnasiadka: agree | 12:42 |
| kevko | mnasiadka: what about question ^^ | 12:42 |
| mnasiadka | lbaasv1 to v2? Haven't been in that situation ;) | 12:43 |
| mnasiadka | bbezak: didn't you use some latin words to describe lbaasv1 to v2 migration? :D | 12:43 |
| kevko | mnasiadka: we just upgraded octavia service ! | 12:43 |
| kevko | mnasiadka: we didn't want to move from v1 provider to v2 provider | 12:44 |
| mnasiadka | isn't amphorav2 the one that requires coordination and does HA? | 12:44 |
| kevko | mnasiadka: problem was that we have hunreds of loadbalancers created with provider = octavia (which is v1 ) and that information is injected into database ...in xena entrypoint for octavia points to v2 | 12:44 |
| kevko | mnasiadka: https://github.com/openstack/octavia/compare/stable/wallaby...stable/xena << check setup.cfg | 12:46 |
| frickler | I agree that this looks like a bug, missing some kind of db migration in octavia | 12:47 |
| kevko | mnasiadka: they created new provider amphorav1 ...but they changed octavia from v1 to v2 ....so aaaalll balancers which has provider octavia is just failing :D | 12:47 |
| mnasiadka | fun | 12:47 |
| mnasiadka | send a mail to ML and raise a bug? | 12:47 |
| kevko | mnasiadka: i am going to | 12:47 |
| kevko | mnasiadka: we fixed it in kolla image ...we just change v2 -> v1 for octavia provider ...so octavia will load the right provider for 'octavia' provider from database | 12:48 |
| kevko | frickler: mnasiadka: another approach is (i think) to exec massive update from octavia to amphorav1 .... | 12:49 |
| mnasiadka | but there should be a db migration or something similar just like frickler mentioned | 12:50 |
| kevko | and third approach is to just delete old octavia provider ....and let live only amphorav1 ...BUT create a DB migration for octavia provider -> amphorav1 | 12:50 |
| kevko | mnasiadka: what is ML :D ? | 12:51 |
| mnasiadka | kevko: MAILING LIST - openstack-discuss | 12:52 |
| mnasiadka | ;-) | 12:52 |
| kevko | ah :D | 12:52 |
| opendevreview | Merged openstack/kolla-ansible stable/yoga: [yoga-only] opensearch: remove kolla_container_engine https://review.opendev.org/c/openstack/kolla-ansible/+/904704 | 13:07 |
| opendevreview | Merged openstack/kolla-ansible stable/yoga: Make designate bind9 cmdline configurable https://review.opendev.org/c/openstack/kolla-ansible/+/896338 | 13:08 |
| opendevreview | Merged openstack/kolla-ansible stable/zed: Make designate bind9 cmdline configurable https://review.opendev.org/c/openstack/kolla-ansible/+/896337 | 13:08 |
| kevko | frickler: mnasiadka: now I realized that i forgot one thing ...there was some openstack cloud provider which hardcoded octavia in his config | 13:08 |
| mnasiadka | kevko: you're going back to kubernetes ;) | 13:11 |
| opendevreview | Merged openstack/kolla-ansible stable/2023.1: Make designate bind9 cmdline configurable https://review.opendev.org/c/openstack/kolla-ansible/+/896336 | 13:16 |
| kevko | mnasiadka: yes, it scares me | 13:18 |
| kevko | worst combination ever ... | 13:18 |
| opendevreview | Michal Nasiadka proposed openstack/kolla stable/2023.2: CI: Use newer podman/buildah on Ubuntu Jammy https://review.opendev.org/c/openstack/kolla/+/904731 | 13:29 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: Add fluentd syslog entry for neutron agents haproxy https://review.opendev.org/c/openstack/kolla-ansible/+/865542 | 13:36 |
| opendevreview | Michal Nasiadka proposed openstack/kolla master: neutron: Allow fluentd to write in neutron log dir https://review.opendev.org/c/openstack/kolla/+/904751 | 13:45 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: Add fluentd syslog entry for neutron agents haproxy https://review.opendev.org/c/openstack/kolla-ansible/+/865542 | 13:46 |
| opendevreview | Merged openstack/kolla-ansible master: ironic: Remove enable_ironic_pxe_uefi bits https://review.opendev.org/c/openstack/kolla-ansible/+/904575 | 14:29 |
| opendevreview | Merged openstack/kolla-ansible stable/yoga: fluentd: remove deprecated parameter https://review.opendev.org/c/openstack/kolla-ansible/+/904518 | 14:57 |
| opendevreview | Merged openstack/kolla-ansible stable/2023.2: Make designate bind9 cmdline configurable https://review.opendev.org/c/openstack/kolla-ansible/+/904723 | 15:38 |
| opendevreview | Merged openstack/kolla-ansible stable/zed: Fix wsrep sync status task while switched to TCP/IP https://review.opendev.org/c/openstack/kolla-ansible/+/904502 | 15:38 |
| opendevreview | Merged openstack/kolla-ansible stable/2023.1: Fix wsrep sync status task while switched to TCP/IP https://review.opendev.org/c/openstack/kolla-ansible/+/904501 | 15:38 |
| opendevreview | Merged openstack/kolla-ansible stable/2023.2: Fix wsrep sync status task while switched to TCP/IP https://review.opendev.org/c/openstack/kolla-ansible/+/904280 | 15:50 |
| opendevreview | Merged openstack/kolla-ansible stable/2023.2: Use service-images-pull role for letsencrypt and venus https://review.opendev.org/c/openstack/kolla-ansible/+/904730 | 16:16 |
| opendevreview | Merged openstack/kolla-ansible master: post-deploy: add public-openrc.sh https://review.opendev.org/c/openstack/kolla-ansible/+/862159 | 17:11 |
| opendevreview | Pierre Riteau proposed openstack/kolla-ansible master: CI: Test Nova server resize functionality https://review.opendev.org/c/openstack/kolla-ansible/+/904249 | 17:16 |
| kevko | mnasiadka: btw, i see long - about 2 minutes restarts in kolla-ansible upgrade ..reconfigure jobs ... | 17:39 |
| kevko | using systemd | 17:39 |
| kevko | it looks like wait for unit is somehow broken | 17:40 |
| opendevreview | Michal Nasiadka proposed openstack/kolla master: neutron: Allow fluentd to write in neutron log dir https://review.opendev.org/c/openstack/kolla/+/904751 | 17:54 |
| mnasiadka | kevko: that would make CI jobs run really long, so maybe that's something on your env? | 17:56 |
| kevko | mnasiadka: cool, are u going also to split rpc and api ? :P ^^ | 17:56 |
| mnasiadka | Once I fix ML2/OVS logging - that's the plan, but wsgi front is a bit... dangerous. | 17:56 |
| kevko | mnasiadka: nope - i have reaaaly standard deployment | 17:56 |
| kevko | mnasiadka: but already debugging ;-) | 17:56 |
| opendevreview | Merged openstack/kolla-ansible stable/2023.1: Remove nova cell sync comment https://review.opendev.org/c/openstack/kolla-ansible/+/904726 | 18:00 |
| opendevreview | Merged openstack/kolla-ansible stable/yoga: Remove nova cell sync comment https://review.opendev.org/c/openstack/kolla-ansible/+/904728 | 18:00 |
| opendevreview | Merged openstack/kolla-ansible stable/zed: Remove nova cell sync comment https://review.opendev.org/c/openstack/kolla-ansible/+/904727 | 18:00 |
| opendevreview | Merged openstack/kolla-ansible stable/zed: Add check_mode: false to Nova upgrade checks https://review.opendev.org/c/openstack/kolla-ansible/+/881327 | 18:00 |
| opendevreview | Michal Nasiadka proposed openstack/kolla stable/yoga: [yoga-only]: Use pyroute2 from distro vendors https://review.opendev.org/c/openstack/kolla/+/904781 | 18:10 |
| opendevreview | Michal Nasiadka proposed openstack/kolla stable/yoga: [yoga-only]: Use pyroute2 from distro vendors https://review.opendev.org/c/openstack/kolla/+/904781 | 18:10 |
| kevko | mnasiadka: this is it ! https://paste.openstack.org/show/bc3sr8r2gJH4KuCnfVq3/ | 18:19 |
| kevko | it pass after 120 sec | 18:20 |
| kevko | Unit kolla-haproxy-container.service state=failed | sleeping for 5. | 18:20 |
| kevko | Unit kolla-haproxy-container.service state=failed | elapsed = 110, bumping + 5. | 18:20 |
| kevko | Unit kolla-haproxy-container.service state=failed | sleeping for 5. | 18:20 |
| kevko | Unit kolla-haproxy-container.service state=failed | elapsed = 115, bumping + 5. | 18:20 |
| kevko | Unit kolla-haproxy-container.service state=failed | sleeping for 5. | 18:20 |
| kevko | Unit kolla-haproxy-container.service state=failed | elapsed = 120, bumping + 5. | 18:20 |
| kevko | Unit kolla-haproxy-container.service failed | - elapsed = 125 > timeout = 120. | 18:20 |
| mnasiadka | Only haproxy or all services? | 18:36 |
| kevko | mnasiadka: not all services | 18:37 |
| kevko | mnasiadka: this is the reason :) | 18:37 |
| kevko | https://mangolassi.it/topic/19455/what-is-the-difference-between-dead-and-failed-for-service-status/2 | 18:37 |
| kevko | failed vs dead | 18:37 |
| kevko | mnasiadka: nah, https://paste.openstack.org/show/b6mQuVXj4fvPE9TPLHlf/ <<< i will propose a patch :) | 18:39 |
| kevko | mnasiadka: i think it's related to how container is started ..where the sigterm is sent ...if it is a binary ..or shell script which will run the service | 18:42 |
| kevko | mnasiadka: yep, definitely needed to fix ... https://paste.openstack.org/show/bITzBEfYfwHKnLqQshP1/ << | 18:51 |
| kevko | https://www.mail-archive.com/haproxy@formilux.org/msg30473.html | 18:54 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: Fix long restarting services while using systemd https://review.opendev.org/c/openstack/kolla-ansible/+/904805 | 21:32 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: Fix long service restarts while using systemd https://review.opendev.org/c/openstack/kolla-ansible/+/904805 | 21:39 |
| opendevreview | Pierre Riteau proposed openstack/kolla-ansible master: CI: Test Nova server resize functionality https://review.opendev.org/c/openstack/kolla-ansible/+/904249 | 22:46 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!