Friday, 2021-01-29

*** jaicaa has quit IRC00:19
*** jaicaa has joined #openstack-kolla00:19
*** rm_work has quit IRC00:37
*** logan- has quit IRC00:38
*** logan- has joined #openstack-kolla00:40
*** hrw has joined #openstack-kolla00:50
*** LinPeiWen has joined #openstack-kolla01:01
*** sri_ has quit IRC01:11
*** fadenb has quit IRC01:11
*** sri_ has joined #openstack-kolla01:12
*** fadenb has joined #openstack-kolla01:12
*** cz3 has quit IRC01:14
*** cz3_ has joined #openstack-kolla01:14
*** rm_work has joined #openstack-kolla01:18
*** rm_work has quit IRC01:18
*** Wasaac has quit IRC01:22
*** Wasaac has joined #openstack-kolla01:23
*** rm_work has joined #openstack-kolla01:29
*** rm_work has quit IRC01:29
*** k_mouza has joined #openstack-kolla01:48
*** k_mouza has quit IRC01:53
openstackgerritwu.chunyang proposed openstack/kolla-ansible master: CI: octavia: create and test a load balancer  https://review.opendev.org/c/openstack/kolla-ansible/+/75428501:53
*** rm_work has joined #openstack-kolla02:01
*** bsanjeewa has joined #openstack-kolla02:31
*** zzzeek has quit IRC03:01
*** zzzeek has joined #openstack-kolla03:02
*** maharg101 has joined #openstack-kolla03:03
*** maharg101 has quit IRC03:08
*** Wasaac_ has joined #openstack-kolla03:31
*** ianw_ has joined #openstack-kolla03:32
*** Wasaac has quit IRC03:38
*** LinPeiWen has quit IRC03:38
*** hrw has quit IRC03:38
*** ebbex has quit IRC03:38
*** ianw has quit IRC03:38
*** ianw_ is now known as ianw03:38
*** Wasaac_ is now known as Wasaac03:38
*** ebbex has joined #openstack-kolla03:41
*** hrw has joined #openstack-kolla03:42
*** gouthamr has quit IRC03:43
*** skramaja has joined #openstack-kolla03:57
*** gouthamr has joined #openstack-kolla04:00
*** gouthamr has quit IRC04:00
*** gouthamr has joined #openstack-kolla04:13
*** gouthamr has quit IRC04:13
*** gouthamr has joined #openstack-kolla04:18
*** gouthamr has quit IRC04:18
*** ricolin_ has joined #openstack-kolla04:18
*** wuchunyang has joined #openstack-kolla04:48
*** gouthamr has joined #openstack-kolla04:49
*** gouthamr has quit IRC04:49
*** wuchunyang has quit IRC04:52
*** ricolin_ has quit IRC05:03
*** vishalmanchanda has joined #openstack-kolla05:03
*** maharg101 has joined #openstack-kolla05:04
*** maharg101 has quit IRC05:09
*** LinPeiWen has joined #openstack-kolla05:16
*** hjensas|afk is now known as hjensas05:18
*** bsanjeewa has quit IRC05:27
*** wuchunyang has joined #openstack-kolla05:31
*** calbers_ has joined #openstack-kolla05:45
*** calbers has quit IRC05:46
*** calbers_ is now known as calbers05:46
*** ianw has quit IRC05:59
*** ianw has joined #openstack-kolla06:01
openstackgerritfudunwei proposed openstack/kolla-ansible master: Negative seqno need to be considered when comparing seqno  https://review.opendev.org/c/openstack/kolla-ansible/+/77297306:03
*** brinzhang0 has joined #openstack-kolla06:12
*** brinzhang_ has quit IRC06:15
*** brinzhang has joined #openstack-kolla06:15
openstackgerritwu.chunyang proposed openstack/kolla-ansible master: CI: octavia: create and test a load balancer  https://review.opendev.org/c/openstack/kolla-ansible/+/75428506:16
*** brinzhang0 has quit IRC06:18
*** gouthamr has joined #openstack-kolla06:22
*** gouthamr has quit IRC06:22
*** cah_link has joined #openstack-kolla06:22
*** bsanjeewa has joined #openstack-kolla06:29
*** xinliang has joined #openstack-kolla06:38
*** gouthamr has joined #openstack-kolla06:56
*** gouthamr has quit IRC06:56
*** maharg101 has joined #openstack-kolla07:05
*** maharg101 has quit IRC07:09
*** bsanjeewa has quit IRC07:19
*** gouthamr has joined #openstack-kolla07:22
*** openstackgerrit has quit IRC07:23
*** _Cyclone_ has quit IRC07:30
*** _Cyclone_ has joined #openstack-kolla07:32
*** benj_ has quit IRC08:22
*** maharg101 has joined #openstack-kolla08:28
*** bengates has joined #openstack-kolla08:33
*** bengates has quit IRC08:46
*** bengates has joined #openstack-kolla08:46
*** rpittau|afk is now known as rpittau08:52
*** zzzeek has quit IRC09:01
*** zzzeek has joined #openstack-kolla09:03
*** bengates_ has joined #openstack-kolla09:06
*** gfidente has joined #openstack-kolla09:09
*** bengates has quit IRC09:10
*** xinliang has quit IRC09:16
*** bengates_ has quit IRC09:29
*** bengates has joined #openstack-kolla09:30
*** bengates has quit IRC09:33
*** bengates has joined #openstack-kolla09:34
*** bengates_ has joined #openstack-kolla09:36
*** openstackgerrit has joined #openstack-kolla09:37
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: centos: clean up repo enable/disable/keys  https://review.opendev.org/c/openstack/kolla/+/76586409:37
hrwrebased09:37
*** bengates has quit IRC09:39
wuchunyanghi, need help09:56
wuchunyanghttps://zuul.opendev.org/t/openstack/build/5d35a968afa2404d8e58c863f9183f6c/log/primary/logs/ansible/test-magnum#45709:57
wuchunyangkeystone seems has problem in centos 8 (CI primary node)09:58
mnasiadkaNo route to host? Looks like keepalived/haproxy switched or something?10:01
mnasiadkayoctozepto: ^^10:01
wuchunyangis it not all in one ?10:02
*** brinzhang has quit IRC10:06
mnasiadkait is - still no errors on haproxy and keystone level, something is wrong with network :)10:09
wuchunyangyes,  it breaks the task.   is there a way to ssh  the node to  debug it ?10:10
*** kevko has joined #openstack-kolla10:10
mnasiadkadoes it fail the same way all the time?10:11
kevkomorning !10:11
wuchunyangyes10:11
kevkoguys, some time ago I filled a bug for oslo cache and flush_on_reconnect (exponencially raising memcached connections)10:11
mnasiadkawuchunyang: first I would be interested why 192.168.100 network is unreachable at all :)10:12
kevko^^ this helps, but there is also one more thing which i had to change to fix a problem with raising connection (especially when using fluentd)10:12
priteaukevko: This is not the neutron-server memcached connection issues, but something else?10:12
kevkoMaybe I will start from beginning10:13
wuchunyangmnasiadka   the vm  should  in staring status . so we can not access it10:14
wuchunyanghttps://zuul.opendev.org/t/openstack/build/5d35a968afa2404d8e58c863f9183f6c/log/primary/logs/system_logs/ip.txt#26910:14
wuchunyangfrom the log, o-hm0(192.168.100.0)could access the gateway.10:16
kevkowe are using fluentd and fluent-logger .. and when fluent was going down and services can't log ..we saw raising connections (bud didn't know why ..), after some time of investigation I filled bug for oslo cache and send a fix (https://review.opendev.org/c/openstack/oslo.cache/+/742193) ... This fixed almost all .. but still raising connections when fluent were go down .. So I started asking myself why ? Fluent-logger is also adding full context to10:18
kevkologs .. so started to wondering about keystone middleware and found also this bug  for neutron https://bugs.launchpad.net/keystonemiddleware/+bug/1883659 ...so I've configured this for all services with keystone middleware and it helped .. can someone test also ?10:18
openstackLaunchpad bug 1883659 in oslo.cache "keystonemiddleware connections to memcached from neutron-server grow beyond configured values" [Undecided,Confirmed]10:18
kevkoSimply said ... I think there is really issue with keystone middleware and his caching globally ->  and this option in middleware really helped : memcache_use_advanced_pool = True10:20
kevkofrom quick look into code it looks like when memcache_use_advanced_pool is True memcached_pool is used ..otherwise no ..and it looks like it causing problems somehow10:21
openstackgerritwu.chunyang proposed openstack/kolla-ansible master: CI: octavia: create and test a load balancer  https://review.opendev.org/c/openstack/kolla-ansible/+/75428510:22
kevkok-a was also helfull , because I created a graph with memcached connections ..and was sending in several bashes some API requests ... while memcached connections were ok when i was sendign token issue or network list , in other cases i had raising connections ... well keystone itself not using keystone_middleware ..so I compared configs for nova and neutron and found memcache_use_advanced_pool = True10:25
kevkopriteau: ah, I just realized, you were the guy who helped me :D10:25
*** e0ne has joined #openstack-kolla10:26
priteauI addede memcache_use_advanced_pool :D10:26
kevkopriteau: yes, and that's it :D10:26
priteauAre you seeing it in another service than neutron-server?10:26
kevkopriteau: yes, most painful was nova, because we had lot of requests to nova because of kubernetes10:28
kevkospamming10:28
kevkopriteau: https://ibb.co/7g2yRGX10:29
*** zzzeek has quit IRC10:30
kevkopriteau: as I said ..we using fluent-logger for logging which is probably pointing his finger at the problem .... anyway ...this is graph of connections with fluent on, off for service for type of request10:30
priteauSorry, I can't see any picture at this link, nothing loads10:31
*** zzzeek has joined #openstack-kolla10:32
*** wuchunyang has quit IRC10:43
*** k_mouza has joined #openstack-kolla10:53
*** zzzeek has quit IRC10:54
*** zzzeek has joined #openstack-kolla10:55
*** e0ne has quit IRC10:58
*** benj_ has joined #openstack-kolla11:00
kevkopriteau: really ?11:02
kevkopriteau: now ? https://pasteboard.co/JLS5rlN.png11:03
*** wuchunyang has joined #openstack-kolla11:03
yoctozeptomnasiadka: weird, it happened very late; perhaps the keepalived stopped?11:03
mnasiadkayoctozepto: single node, haproxy only it seems - but still weird that it's so late.11:04
yoctozepto2021-01-29T07:50:37.930811020Z Fri Jan 29 07:50:37 2021: Track script check_alive is already running, expect idle - skipping run11:04
yoctozepto2021-01-29T07:50:41.960528412Z Fri Jan 29 07:50:41 2021: Track script check_alive is already running, expect idle - skipping run11:04
yoctozepto2021-01-29T07:50:42.000656974Z Fri Jan 29 07:50:41 2021: VRRP_Script(check_alive) timed_out11:04
yoctozepto2021-01-29T07:50:42.084670407Z Fri Jan 29 07:50:41 2021: (kolla_internal_vip_106) Entering FAULT STATE11:04
yoctozeptoprobably due to load11:04
yoctozeptohaproxy failed to respond11:04
yoctozeptowe had this issue from time to time11:04
yoctozeptoI raised it several time :-)11:05
yoctozeptobut there is not better fix than to stop queruing haproxy all the time11:05
yoctozeptoperhaps it is costly or whatever11:05
yoctozeptoit could be related to mysql checks too11:05
yoctozeptoI hope we will see some nice improvement from proxysql though11:05
kevkothump up :)11:06
mnasiadkayoctozepto: well, we could disable the check in CI at least11:07
mnasiadkabut then we're not testing the check :D11:07
priteaukevko: better, thanks. What's non_f?11:08
kevkopriteau: well, as I said we are using fluent and python-fluent-logger for log ... non_f   is without fluentd turned on (simulating when fluent is down)11:10
priteauSo with fluentd, nova connections still grow unchecked, just more slowly?11:11
kevkopriteau: I think the increase is there because fluent is trying to add the context from cache11:11
yoctozeptokevko: what do "cycles" mean?11:12
kevkoyoctozepto: number of api requests after second11:12
kevko1800 -> 30 min11:12
kevkowith flush_on_reconnect (packed to debian package of oslo_cache) built images AND memcache_use_advanced_pool = True in kyestone_auth in services I have no issue11:14
kevkoso far it looks like that ^^11:16
kevkobut I am still doing tests ..so will see11:16
hrwmeh. my cleanup failed on zuul upgrade job ;(11:22
*** k_mouza has quit IRC11:27
*** bengates_ has quit IRC11:28
*** bengates has joined #openstack-kolla11:29
*** k_mouza has joined #openstack-kolla11:30
*** wuchunyang has quit IRC11:36
*** bengates_ has joined #openstack-kolla11:37
*** bengates_ has quit IRC11:39
*** bengates_ has joined #openstack-kolla11:39
*** bengates has quit IRC11:40
*** LinPeiWen has quit IRC11:50
*** bengates_ has quit IRC11:53
*** bengates has joined #openstack-kolla11:53
kevkoyoctozepto: btw, did you have some time to review proxysql again ? (sorry i'm annoying :/)12:20
*** bengates has quit IRC12:37
*** bengates has joined #openstack-kolla12:38
*** rm_work has quit IRC13:00
*** Jeffrey4l has quit IRC13:00
*** yankcrime has quit IRC13:00
*** rm_work has joined #openstack-kolla13:01
*** Jeffrey4l has joined #openstack-kolla13:01
*** yankcrime has joined #openstack-kolla13:01
*** benj_ has quit IRC13:02
*** benj_ has joined #openstack-kolla13:04
*** benj_ has quit IRC13:05
*** benj_ has joined #openstack-kolla13:06
*** zzzeek has quit IRC13:29
*** zzzeek has joined #openstack-kolla13:31
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: Rename infrastructure images by default  https://review.opendev.org/c/openstack/kolla/+/74403213:42
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: Add 'infra' install type.  https://review.opendev.org/c/openstack/kolla/+/73807213:42
hrwrebased and dropped CI parts13:42
*** e0ne has joined #openstack-kolla13:45
*** Underknowledge has quit IRC13:51
*** cz3_ is now known as cz313:51
*** Underknowledge has joined #openstack-kolla13:52
*** zzzeek has quit IRC13:52
*** zzzeek has joined #openstack-kolla13:56
yoctozeptokevko: no, but will look today; mostly to act example with more than one shard and perhaps cells14:02
*** zzzeek has quit IRC14:02
kevkoyoctozepto: thanks14:03
kevkoyoctozepto: \o/14:03
*** zzzeek has joined #openstack-kolla14:03
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible master: [DNM] ProxySQL with two shards  https://review.opendev.org/c/openstack/kolla-ansible/+/77305214:09
yoctozeptokevko: ^14:09
yoctozeptoI think this should do as it is supposed to work with an unused shard as well14:09
kevkoyoctozepto: ok, thank you14:12
*** skramaja has quit IRC14:13
kevkoyoctozepto: quick question ..if i enable grafana in k-a ? Will I have memcached metrics ?14:14
*** bsanjeewa has joined #openstack-kolla14:14
*** bengates_ has joined #openstack-kolla14:15
*** bengates_ has quit IRC14:15
*** bengates has quit IRC14:15
*** bengates has joined #openstack-kolla14:15
*** e0ne has quit IRC14:19
kevkoi see..going to deploy :)14:23
yoctozeptokevko: no idea :-)14:30
*** kevko has quit IRC14:30
*** dabukalam has joined #openstack-kolla14:32
dabukalamHello, I've just followed the Getting Started guide for kolla-ansible, and I'm running into this issue:14:33
dabukalamERROR: kolla_ansible has to be available in the Ansible PYTHONPATH.14:33
dabukalamPlease install both in the same (virtual) environment.14:33
dabukalamI did use apt to install ansible, but looking at some recent launchpad issues it seems like that's not my problem here... something to do with editable installs post-train?14:34
* dabukalam last installed at Rocky14:34
*** jbadiapa has joined #openstack-kolla14:35
* dabukalam just tries to pip install ansible14:37
dabukalamhave all the ansibles!14:37
*** e0ne has joined #openstack-kolla14:51
ozzzoI'm using Keycloak for login, and I need to delete my Keycloak token from keystone so that I can login as a different user14:51
ozzzoI see the "token revoke" command but I can't find the token ID14:52
ozzzoWhen I look in the database, I see tables like14:52
ozzzoacce3ss_token, application_credential, credential, token, etc14:52
ozzzobut they are all empty even though I am logged in as a Keycloak user14:52
ozzzowhere can I find the token ID to revoke it?14:52
*** cah_link has quit IRC15:20
*** bsanjeewa has quit IRC15:25
*** rpittau is now known as rpittau|afk15:27
priteauyoctozepto: do you know if there is a cirros mirror in opendev?16:04
*** e0ne has quit IRC16:04
dabukalamalright jsut resorted to venv in the end...16:19
*** kevko has joined #openstack-kolla16:39
*** bsanjeewa has joined #openstack-kolla16:52
*** kevko has quit IRC16:54
yoctozeptopriteau: not aware; I think not the new one; but the infra os images actually carry a copy of cirros16:55
yoctozeptodevstack uses it16:55
yoctozeptodo we need to optimise it?16:55
priteauJust asking because a kayobe job failed while trying to download a cirros image16:57
priteauBut I think it's a rare issue, don't remember seeing it before16:57
priteauStupid question maybe, but when a change fails during the gate job, is there a way to rerun just the gate and not the check?16:58
*** vishalmanchanda has quit IRC17:32
*** bsanjeewa has quit IRC17:41
*** jbadiapa has quit IRC17:59
yoctozeptono, it is purposefully disabled :-)18:06
yoctozeptoand there are no stupid questions18:06
openstackgerritMark Goddard proposed openstack/kayobe master: CI: Use cached cirros image for seed jobs  https://review.opendev.org/c/openstack/kayobe/+/77316318:07
mgoddardyoctozepto, priteau: ^18:07
*** k_mouza has quit IRC18:07
yoctozeptomgoddard: nice; could you replicate for kolla-ansible too?18:13
*** maharg101 has quit IRC18:16
*** bengates has quit IRC18:25
*** bengates has joined #openstack-kolla18:25
priteauVery nice, thanks mgoddard18:26
priteauIt doesn't work though, because we fetch it from Bifrost which is running inside a container18:27
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible master: [DNM] ProxySQL with two shards + cells  https://review.opendev.org/c/openstack/kolla-ansible/+/77305218:33
*** cah_link has joined #openstack-kolla18:35
yoctozeptomgoddard, priteau: just noticed that we already have this functionality in k-a CI, sorry for noise19:18
*** cah_link has quit IRC19:18
*** bengates has quit IRC19:38
*** kevko has joined #openstack-kolla19:57
openstackgerritRadosław Piliszek proposed openstack/kolla stable/victoria: Fix Mistral source images to respect upper-constraints  https://review.opendev.org/c/openstack/kolla/+/77301220:01
openstackgerritRadosław Piliszek proposed openstack/kolla stable/ussuri: Fix Mistral source images to respect upper-constraints  https://review.opendev.org/c/openstack/kolla/+/77301320:01
openstackgerritRadosław Piliszek proposed openstack/kolla stable/train: Fix Mistral source images to respect upper-constraints  https://review.opendev.org/c/openstack/kolla/+/77301420:02
openstackgerritRadosław Piliszek proposed openstack/kolla stable/stein: Fix Mistral source images to respect upper-constraints  https://review.opendev.org/c/openstack/kolla/+/77301520:02
openstackgerritRadosław Piliszek proposed openstack/kolla stable/victoria: Horizon: gentler `-o nounset` handling  https://review.opendev.org/c/openstack/kolla/+/77319620:03
*** LinPeiWen has joined #openstack-kolla20:13
*** maharg101 has joined #openstack-kolla20:13
*** ivan_lin has quit IRC20:16
*** maharg101 has quit IRC20:17
*** kevko has quit IRC20:20
*** kevko has joined #openstack-kolla20:21
*** k_mouza has joined #openstack-kolla20:31
*** k_mouza has quit IRC20:38
openstackgerritPierre Riteau proposed openstack/kayobe master: Remove extra space characters  https://review.opendev.org/c/openstack/kayobe/+/77318520:40
*** kevko has quit IRC20:45
*** imcsk8 has quit IRC20:56
*** imcsk8 has joined #openstack-kolla20:57
openstackgerritPierre Riteau proposed openstack/kayobe master: [docs] Update main documentation page  https://review.opendev.org/c/openstack/kayobe/+/77176621:00
*** jonaspaulo has quit IRC21:03
ozzzoI can use "openstack quota list --detail" to check usage for network and compute, but volume says "Volume service doesn't provide detailed quota information"21:12
ozzzois it possible to check volume usage on cli?21:12
*** sean-k-mooney has quit IRC21:24
*** sean-k-mooney has joined #openstack-kolla21:26
*** sean-k-mooney has quit IRC21:36
*** sean-k-mooney has joined #openstack-kolla21:36
*** kevko has joined #openstack-kolla22:02
*** stand has quit IRC22:26
*** kevko has quit IRC22:29
*** k_mouza has joined #openstack-kolla22:38
*** k_mouza has quit IRC22:42
*** ug_egon has quit IRC22:51
*** cah_link has joined #openstack-kolla22:54
*** kevko has joined #openstack-kolla22:56
*** kevko has quit IRC23:20
*** cah_link has quit IRC23:31
*** zzzeek has quit IRC23:48
*** zzzeek has joined #openstack-kolla23:50

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!