Tuesday, 2019-12-10

« Monday, 2019-12-09 Index Wednesday, 2019-12-11 »
*** rcernin_ has joined #openstack-lbaas00:04
*** rcernin has quit IRC00:07
*** rcernin_ has quit IRC00:19
*** rcernin has joined #openstack-lbaas00:27
*** yamamoto has quit IRC00:49
openstackgerritMichael Johnson proposed openstack/octavia master: Update ListenersUpdate for lb/listener dicts  https://review.opendev.org/69811701:00
*** yamamoto has joined #openstack-lbaas01:01
openstackgerritMichael Johnson proposed openstack/octavia master: Transition amphora flows to dicts  https://review.opendev.org/66889801:04
johnsomI think that will fix the problem01:07
*** goldyfruit_ has joined #openstack-lbaas01:09
*** goldyfruit_ has quit IRC01:43
*** nicolasbock has quit IRC02:10
*** abaindur has quit IRC02:21
*** dmellado has quit IRC02:45
*** vesper11 has quit IRC02:45
*** tridde has joined #openstack-lbaas02:45
*** trident has quit IRC02:46
*** vesper11 has joined #openstack-lbaas02:46
*** dmellado has joined #openstack-lbaas02:48
*** yamamoto has quit IRC02:55
*** yamamoto has joined #openstack-lbaas03:01
*** psachin has joined #openstack-lbaas03:32
*** hongbin has joined #openstack-lbaas03:50
*** ramishra has joined #openstack-lbaas03:55
*** nicolasbock has joined #openstack-lbaas04:02
*** hongbin has quit IRC04:41
*** nicolasbock has quit IRC04:46
*** gcheresh has joined #openstack-lbaas05:43
*** gcheresh has quit IRC06:02
*** sapd1 has joined #openstack-lbaas06:04
*** gcheresh has joined #openstack-lbaas06:13
*** yamamoto has quit IRC06:14
*** yamamoto has joined #openstack-lbaas06:21
*** psachin has quit IRC06:42
*** psachin has joined #openstack-lbaas06:50
*** sapd1 has quit IRC07:14
*** pcaruana has joined #openstack-lbaas07:39
*** ut2k3 has joined #openstack-lbaas07:46
ut2k3Hi guys, I've got the problem with the amphora creation. The logs shows the following and then kills the newly spawned amphora: Amphora agent returned unexpected result code 404 with response {u'http_code': 404, u'error': u'404 Not Found: The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.'}07:47
ut2k3Any ideas why this happens? Tried already to update the amphora image too07:48
*** yamamoto has quit IRC07:52
*** rcernin has quit IRC07:54
*** tkajinam has quit IRC08:06
ut2k3Ok found it .. seems the OS Ansible updated the image08:06
*** ut2k3 has quit IRC08:06
*** maciejjozefczyk has joined #openstack-lbaas08:08
*** ccamposr has joined #openstack-lbaas08:12
*** tesseract has joined #openstack-lbaas08:27
*** cgoncalves has quit IRC08:40
*** rpittau|afk is now known as rpittau08:41
*** yamamoto has joined #openstack-lbaas09:06
*** yamamoto has quit IRC09:10
*** ignaziocassano1 has joined #openstack-lbaas09:19
ignaziocassano1Hello All, I got some issues on stein octavia09:20
ignaziocassano1The omphorae instance does not have any service on port 944309:20
ignaziocassano1and worker.log reports: 2019-12-10 10:21:32.175 8703 WARNING octavia.amphorae.drivers.haproxy.rest_api_driver [-] Could not connect to instance. Retrying.: ConnectTimeout: HTTPSConnectionPool(host='10.102.143.93', port=9443): Max retries exceeded with url: // (Caused by ConnectTimeoutError(<urllib3.connection.VerifiedHTTPSConnection object at 0x7f92356d4990>, 'Connection to 10.102.143.93 timed out. (connect timeout=10.0)'))09:21
ignaziocassano1Please, help me09:23
openstackgerritMaciej Józefczyk proposed openstack/octavia-tempest-plugin master: Run tests with algorithm supported by provider driver  https://review.opendev.org/67226410:21
*** salmankhan has joined #openstack-lbaas10:29
*** salmankhan has quit IRC10:35
*** salmankhan has joined #openstack-lbaas10:36
*** salmankhan has quit IRC10:36
*** salmankhan has joined #openstack-lbaas10:36
*** isakgicu has joined #openstack-lbaas10:48
isakgicuHi all10:49
isakgicuis it possible to deploy amphorae instances on network node10:49
isakgicuas I understaood, for this I have to deploy nova service on network node as well10:50
isakgicuam I right?10:50
*** gcheresh has quit IRC10:51
isakgicuI'm finding this a bit unlogic, why network job has to be done by compute nodes..10:52
*** ignaziocassano1 has quit IRC11:09
*** gcheresh has joined #openstack-lbaas11:09
rm_workisakgicu: deploying them on the networking nodes would not scale well... why would you want that?11:36
rm_workthe entire purpose of the octavia project was to fix this specific issue -- lbaasv1 only worked on the network nodes, but deploying that way is generally a very bad idea11:37
rm_workso octavia was created so we could avoid deploying LBs on network nodes, and allow them to use compute, which is a much better solution11:37
*** gcheresh has quit IRC11:55
*** nicolasbock has joined #openstack-lbaas12:00
*** gcheresh has joined #openstack-lbaas12:02
*** CobHead has joined #openstack-lbaas12:06
CobHeadHi johnsom :)12:07
CobHeadI just hit this issue https://bugs.launchpad.net/nova/+bug/1827746 today in Octavia, and it was not a very pleasant experience. I see that you commented it last, so I was wondering if you had any information on whether or not this bug is being worked on12:08
openstackLaunchpad bug 1827746 in OpenStack Compute (nova) "Port detach fails when compute host is unreachable" [Medium,Confirmed]12:08
*** cgoncalves has joined #openstack-lbaas12:09
rm_workyes12:18
rm_workCobHead: he's actively working on refactoring the whole failover flow to completely avoid that issue12:18
rm_workI don't think there was an ETA last time I heard from him about it, but it's still being worked on12:18
rm_workthose flows are very complex12:18
*** gcheresh has quit IRC12:28
*** gcheresh has joined #openstack-lbaas12:32
*** servagem has joined #openstack-lbaas12:47
*** cgoncalves has quit IRC12:47
*** StefanNica has joined #openstack-lbaas12:48
CobHeadI understand. As it is right now, HA LB is not viable in a production environment if the node containing the master Amphora goes down, which kinda makes Octavia a no-go for production.12:56
StefanNicaany takers for a review for a small change to get the Octavia oslo.policy enforcer to work correctly with the oslopolicy-policy-generator tool ?12:58
StefanNicahttps://review.opendev.org/#/c/69729612:59
StefanNicathis is basically aligning Octavia to nova in that regard13:00
rm_worki'll look in a bit StefanNica13:05
StefanNicagreat, thanks :)13:06
rm_workCobHead: yeah it's a bit aggrivating... i'm hoping he can finish that work soon, though i am not helping by distracting him constantly with other stuff XD13:06
rm_workCobHead: though i've not had that problem occur nearly as frequently as other people see it, i guess13:06
rm_workbut it definitely does happen13:06
rm_workin active-standby, the other LB should at least continue to work...13:06
rm_workand assuming the other HV does ever come back up, a manual failover on that amp should fix the issue13:07
rm_workand return the LB to active/HA13:07
CobHeadYeah, unfortunately this node bit the dust and will not be returning in its present form ever.13:10
*** cgoncalves has joined #openstack-lbaas13:23
*** goldyfruit_ has joined #openstack-lbaas13:37
rm_workI believe you could do some stuff in the background to force-detach the port, and THEN trigger a failover in octavia? if the LB is very important13:40
rm_workbut yeah, it's not really a viable workaround for day-to-day operation13:41
*** goldyfruit___ has joined #openstack-lbaas13:41
*** goldyfruit_ has quit IRC13:44
openstackgerritMerged openstack/octavia-lib master: Stop testing python 2  https://review.opendev.org/69313213:56
rm_workah cgoncalves is up and about too :D14:00
rm_worki had to spend a couple of days internally again because I realized we removed py2 so quickly and we still only used py2 internally T_T14:01
rm_workso had to take care of that before things exploded14:01
*** tkajinam has joined #openstack-lbaas14:01
rm_workshould be back to testing AZ / jobboard patches shortly14:01
cgoncalveshmm, yes, I'm up. 3:00 PM here :P14:01
rm_workit looks like the AZ one is good to go though, so if others have time to poke at that, it'd be appreciated too :D14:01
rm_workheh yeah, not gonna say what time it is here, it's dumb14:02
rm_workanyways14:02
rm_workon the AZ stuff, if it mostly works, and doesn't break other stuff, I am ok with merging it and fixing some edge cases in followups, if others are ok with that as well14:02
rm_workonce we can deploy it in our internal env (patches merged upstream for the basics) we will be putting it through the wringer14:03
rm_workso will definitely have all that stuff handled before our next release :D14:03
rm_workbr14:03
rm_work*brb14:03
*** pcaruana has quit IRC14:36
*** tkajinam has quit IRC14:52
*** pcaruana has joined #openstack-lbaas15:01
*** TrevorV has joined #openstack-lbaas15:10
*** goldyfruit_ has joined #openstack-lbaas15:57
*** goldyfruit___ has quit IRC16:00
*** isakgicu has quit IRC16:17
*** goldyfruit___ has joined #openstack-lbaas16:41
*** goldyfruit_ has quit IRC16:44
*** tesseract has quit IRC16:58
johnsomLooks like I fixed the issue on the jobboard chain. Both patches passed the v2 jobs.17:05
johnsomI think it is a good day to pivot back to working on the failover flow refactor.17:06
johnsomEspecially given that someone else is hitting it and the nova issue is still present.17:06
*** maciejjozefczyk has quit IRC17:08
rm_workyeah17:14
rm_workok cool17:14
rm_workwe can merge the rest of those patches then, right? :D17:15
rm_worki'm fighting with senlin currently internally T_T17:15
johnsomWell, I think I had some questions about the amphora patch. The reset of the chain will need reviews.17:16
johnsomEnjoy that senlin stuff. grin17:17
cgoncalvesrm_work, any updates on testing centos 8?17:28
johnsomAhh, back down to just two workspaces.... I was up to six for a while working on so many things.17:28
*** rpittau is now known as rpittau|afk17:29
rm_workcgoncalves: seems to work17:31
* rm_work shrugs17:32
rm_workdeploying to prod today17:32
* rm_work shrugs again17:32
cgoncalvesoctavia is 2 bugs away from being bug-free17:32
johnsomJust two more weeks....17:32
rm_workfigured out senlin issues17:32
rm_workfuck eventlet17:32
rm_workeventlet is poison17:32
johnsomyes17:32
rm_workthank FSM we purged ourselves17:33
johnsomhttps://3i1e5d437yd84efcy34dardm-wpengine.netdna-ssl.com/wp-content/uploads/2015/04/money-pit-two-weeks-two-weeks-gif.png17:33
*** armax has quit IRC17:51
openstackgerritMichael Johnson proposed openstack/octavia master: Update policy docs to reflect json format  https://review.opendev.org/69830717:53
johnsomEasiest review in a few months right there....17:53
*** gcheresh has quit IRC17:57
rm_worki dunnooooooo18:00
johnsomI didn't say of-all-time just to hedge my bet18:01
rm_workok i will admit, i thought i was going to have to compare to some json file output .....18:20
rm_work<_<18:20
johnsomlol18:21
johnsomWell, now zuul is borked for some release smoke test, so ....18:21
rm_work<_<18:21
johnsomI miss the days when restarts of the CI infrastructure were scheduled evens18:21
johnsomevents18:21
rm_worknap time18:21
johnsomYeah, it has been down a while18:22
*** goldyfruit_ has joined #openstack-lbaas18:22
*** salmankhan has quit IRC18:23
*** goldyfruit___ has quit IRC18:25
*** gcheresh has joined #openstack-lbaas18:40
openstackgerritGregory Thiemonge proposed openstack/octavia-tempest-plugin master: Fix timezones in certificate revocation list dates  https://review.opendev.org/69831218:49
*** psachin has quit IRC19:02
*** servagem has quit IRC19:33
*** spatel has joined #openstack-lbaas19:33
johnsomAnd everything zuul is starting over again. Time for lunch19:37
*** gmann is now known as gmann_afk19:40
*** TrevorV has quit IRC19:42
*** rcernin has joined #openstack-lbaas20:16
*** pcaruana has quit IRC20:16
*** sorrison has quit IRC20:24
*** sorrison has joined #openstack-lbaas20:26
openstackgerritMerged openstack/octavia master: Update policy docs to reflect json format  https://review.opendev.org/69830720:34
*** gcheresh has quit IRC20:36
johnsomYay, my tech debt contribution of the day!20:40
*** goldyfruit_ has quit IRC21:24
*** goldyfruit_ has joined #openstack-lbaas21:30
*** gcheresh has joined #openstack-lbaas21:31
cgoncalves"[Fatal] ConsoleApp: EPIC FAIL!" in a Mono-based app. we should consider incorporating these type of messages in our log prints21:34
*** abaindur has joined #openstack-lbaas21:36
*** abaindur has quit IRC21:37
johnsomfollowed by: "[not-so-fatal] Never mind - nothing to see here: Octavia worked around your broken cloud."21:37
*** abaindur has joined #openstack-lbaas21:38
cgoncalvesI can only image the amount of queries we'd receive on how to disable that, lol21:38
cgoncalves"Octavia is filling up the log storage with these messages!"21:39
*** gcheresh has quit IRC22:04
*** ramishra has quit IRC22:06
*** spatel has quit IRC22:09
*** nicolasbock has quit IRC22:12
*** abaindur has quit IRC22:20
*** abaindur has joined #openstack-lbaas22:37
*** tkajinam has joined #openstack-lbaas23:05
*** gmann_afk is now known as gmann23:06
*** abaindur has quit IRC23:25
*** abaindur has joined #openstack-lbaas23:25
*** goldyfruit_ has quit IRC23:30
« Monday, 2019-12-09 Index Wednesday, 2019-12-11 »

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!