| *** hongbin has quit IRC | 00:43 | |
| *** hongbin has joined #openstack-lbaas | 00:47 | |
| *** yamamoto has joined #openstack-lbaas | 00:57 | |
| *** wuchunyang has joined #openstack-lbaas | 00:59 | |
| *** armax has quit IRC | 01:07 | |
| *** yamamoto has quit IRC | 01:11 | |
| *** ramishra has joined #openstack-lbaas | 01:33 | |
| *** yamamoto has joined #openstack-lbaas | 01:37 | |
| *** spatel has joined #openstack-lbaas | 01:57 | |
| *** shtepanie has quit IRC | 02:09 | |
| *** mugsie has quit IRC | 02:59 | |
| *** mugsie has joined #openstack-lbaas | 03:02 | |
| *** armax has joined #openstack-lbaas | 03:08 | |
| *** psachin has joined #openstack-lbaas | 03:25 | |
| *** wuchunyang has quit IRC | 03:56 | |
| *** wuchunyang has joined #openstack-lbaas | 04:01 | |
| *** wuchunyang has quit IRC | 04:04 | |
| *** hongbin has quit IRC | 04:09 | |
| *** wuchunyang has joined #openstack-lbaas | 04:45 | |
| *** vishalmanchanda has joined #openstack-lbaas | 04:49 | |
| *** wuchunyang has quit IRC | 05:00 | |
| *** gcheresh_ has joined #openstack-lbaas | 06:01 | |
| *** spatel has quit IRC | 06:30 | |
| *** also_stingrayza is now known as stingrayza | 06:49 | |
| *** maciejjozefczyk has joined #openstack-lbaas | 06:57 | |
| *** wuchunyang has joined #openstack-lbaas | 06:59 | |
| *** ataraday_ has joined #openstack-lbaas | 07:26 | |
| *** born2bake has joined #openstack-lbaas | 07:33 | |
| *** ramishra has quit IRC | 07:36 | |
| *** rcernin has quit IRC | 07:45 | |
| *** luksky has joined #openstack-lbaas | 07:53 | |
| *** ramishra has joined #openstack-lbaas | 08:19 | |
| *** wuchunyang has quit IRC | 08:39 | |
| *** wuchunyang has joined #openstack-lbaas | 08:47 | |
| *** wuchunyang has quit IRC | 09:01 | |
| *** tkajinam has quit IRC | 09:09 | |
| openstackgerrit | Ann Taraday proposed openstack/octavia master: Add experimental amphorav2 jobs https://review.opendev.org/737993 | 09:23 |
|---|---|---|
| openstackgerrit | Ann Taraday proposed openstack/octavia master: Fix amphorav2 bytes error https://review.opendev.org/738407 | 09:23 |
| *** ccamposr__ has quit IRC | 09:32 | |
| *** ccamposr has joined #openstack-lbaas | 09:33 | |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia stable/ussuri: Refactor the failover flows https://review.opendev.org/739002 | 09:36 |
| *** gcheresh_ has quit IRC | 09:44 | |
| *** gcheresh_ has joined #openstack-lbaas | 09:52 | |
| *** wuchunyang has joined #openstack-lbaas | 10:12 | |
| *** wuchunyang has quit IRC | 10:17 | |
| *** wuchunyang has joined #openstack-lbaas | 10:46 | |
| lxkong | Hi rm_work, I just noticed there are lots of lines like the following in octavia-api log, do you happen to know why? I can find it's coming from here https://github.com/openstack/octavia/blob/master/octavia/api/v2/controllers/pool.py#L518 | 10:48 |
| lxkong | > 2020-07-02 03:56:03.070 1 INFO octavia.api.v2.controllers.pool [req-26a5d590-16dd-4971-a2c1-9970e0eac340 - 0ca95d0cd27c4dc89c37c19d87d36578 - default default] Pool df4cf27d-eb0a-438b-a4d6-248ac91728fe not found | 10:48 |
| *** wuchunyang has quit IRC | 10:59 | |
| cgoncalves | ataraday_, hi! I have a question you may be able to help. amphora v1 controllers run at most $max_workers flows in parallel. how does it work in v2? | 11:00 |
| *** yamamoto has quit IRC | 11:02 | |
| openstackgerrit | Ann Taraday proposed openstack/octavia master: Fix amphorav2 bytes error https://review.opendev.org/738407 | 11:07 |
| *** yamamoto has joined #openstack-lbaas | 11:09 | |
| *** osmanlicilegi has quit IRC | 11:50 | |
| *** yamamoto has quit IRC | 12:01 | |
| *** osmanlicilegi has joined #openstack-lbaas | 12:07 | |
| *** wuchunyang has joined #openstack-lbaas | 12:08 | |
| *** yamamoto has joined #openstack-lbaas | 12:12 | |
| *** servagem has joined #openstack-lbaas | 12:13 | |
| ataraday_ | cgoncalves, hi! The same max_workers in parallel | 12:17 |
| ataraday_ | https://github.com/openstack/octavia/blob/master/octavia/common/base_taskflow.py#L169 | 12:20 |
| ataraday_ | So, engine is parallel, but max workers are not set here. Logic is a bit different I guess as we use taskflow conductors. This one particularly https://docs.openstack.org/taskflow/ocata/conductors.html#module-taskflow.conductors.backends.impl_blocking | 12:24 |
| *** wuchunyang has quit IRC | 12:31 | |
| ataraday_ | so it runs in this number of workers https://github.com/openstack/octavia/blob/master/octavia/cmd/octavia_worker.py#L39 | 12:37 |
| *** yamamoto has quit IRC | 12:42 | |
| *** yamamoto has joined #openstack-lbaas | 13:19 | |
| *** yamamoto has quit IRC | 13:22 | |
| *** yamamoto has joined #openstack-lbaas | 13:46 | |
| *** vishalmanchanda has quit IRC | 13:48 | |
| *** yamamoto has quit IRC | 13:53 | |
| *** psachin has quit IRC | 13:54 | |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia master: WIP: Jobboard on/off option https://review.opendev.org/739053 | 14:26 |
| cgoncalves | ataraday_, thank you | 14:29 |
| *** yamamoto has joined #openstack-lbaas | 14:36 | |
| *** yamamoto has quit IRC | 14:42 | |
| *** also_stingrayza has joined #openstack-lbaas | 14:50 | |
| *** stingrayza has quit IRC | 14:53 | |
| openstackgerrit | Ann Taraday proposed openstack/octavia master: Fix amphorav2 bytes error https://review.opendev.org/738407 | 14:54 |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia stable/ussuri: Add missing reload method in amphora noop driver https://review.opendev.org/739064 | 14:56 |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia stable/ussuri: Fix missing params in amphora base and noop driver https://review.opendev.org/739065 | 14:56 |
| ataraday_ | If someone has some time, please review https://review.opendev.org/#/c/738407 and https://review.opendev.org/#/c/738609/ | 14:57 |
| *** TrevorV has joined #openstack-lbaas | 15:03 | |
| *** JasonF has joined #openstack-lbaas | 15:13 | |
| *** vesper has joined #openstack-lbaas | 15:13 | |
| *** vesper11 has quit IRC | 15:14 | |
| *** irclogbot_3 has quit IRC | 15:14 | |
| *** TrevorV has quit IRC | 15:14 | |
| *** etp has quit IRC | 15:14 | |
| *** dosaboy has quit IRC | 15:14 | |
| *** JayF has quit IRC | 15:14 | |
| *** TrevorV has joined #openstack-lbaas | 15:14 | |
| *** etp has joined #openstack-lbaas | 15:14 | |
| *** JasonF is now known as JayF | 15:14 | |
| *** jamesdenton has quit IRC | 15:15 | |
| *** jamesdenton has joined #openstack-lbaas | 15:18 | |
| *** irclogbot_3 has joined #openstack-lbaas | 15:18 | |
| *** dosaboy has joined #openstack-lbaas | 15:19 | |
| *** gcheresh_ has quit IRC | 15:21 | |
| *** ataraday_ has quit IRC | 15:27 | |
| *** yamamoto has joined #openstack-lbaas | 15:40 | |
| *** also_stingrayza has quit IRC | 15:57 | |
| *** dmellado has joined #openstack-lbaas | 16:00 | |
| *** yamamoto has quit IRC | 16:04 | |
| *** gcheresh_ has joined #openstack-lbaas | 16:06 | |
| *** gcheresh_ has quit IRC | 16:11 | |
| *** dmellado has quit IRC | 16:20 | |
| *** dmellado has joined #openstack-lbaas | 16:22 | |
| *** yamamoto has joined #openstack-lbaas | 16:36 | |
| *** yamamoto has quit IRC | 16:45 | |
| *** armax has quit IRC | 16:50 | |
| *** armax has joined #openstack-lbaas | 16:51 | |
| *** stingrayza has joined #openstack-lbaas | 16:55 | |
| openstackgerrit | Michael Johnson proposed openstack/octavia master: DNM: Testing for IPv6 job debug https://review.opendev.org/739119 | 17:16 |
| openstackgerrit | Michael Johnson proposed openstack/octavia-tempest-plugin master: DNM: Debugging the IPv6 job https://review.opendev.org/738715 | 17:17 |
| *** KeithMnemonic has joined #openstack-lbaas | 17:58 | |
| *** gcheresh_ has joined #openstack-lbaas | 18:10 | |
| *** haleyb has quit IRC | 18:36 | |
| *** haleyb has joined #openstack-lbaas | 18:51 | |
| *** gcheresh_ has quit IRC | 19:00 | |
| *** wuchunyang has joined #openstack-lbaas | 19:28 | |
| *** armax has quit IRC | 19:39 | |
| *** wuchunyang has quit IRC | 19:40 | |
| rm_work | lxkong: not sure, it's been a while since I looked at that | 20:07 |
| *** wuchunyang has joined #openstack-lbaas | 20:11 | |
| johnsom | Fun, fun. So the IPv6 jobs are borked because the nova instance isn't getting an IP address all of sudden for some reason. The plot continues..... | 20:14 |
| johnsom | Well, at least the cloud-init block doesn't have a real IP on it. | 20:15 |
| *** TrevorV has quit IRC | 20:21 | |
| johnsom | hmmm, the subnet is setup for slaac in neutron | 20:21 |
| johnsom | So, should just *work* like has been before. | 20:21 |
| johnsom | haleyb I see that IPv6 slaac is broken with OVN, but we are seeing an issue with OVS. Would the OVN issue cross over to the OVS ML2 as well? | 20:30 |
| johnsom | (pinging you as your name was on one of the patches disabling tempest tests in nova for the OVN issue) | 20:31 |
| *** wuchunyang has quit IRC | 20:38 | |
| *** gcheresh_ has joined #openstack-lbaas | 20:38 | |
| *** wuchunyang has joined #openstack-lbaas | 20:39 | |
| *** wuchunyang has quit IRC | 20:43 | |
| *** wuchunyang has joined #openstack-lbaas | 20:45 | |
| haleyb | johnsom: OVN not doing slaac right shouldn't affect ML2, is this with your debug patch? | 20:47 |
| johnsom | Yeah | 20:48 |
| johnsom | I have added extra debug output, tempest log has ##### sections for the o-hm0 side. Then the o-cw log has the nova console log dumping | 20:48 |
| johnsom | The network and port look ok in neutron. I see it assigned an address fd00::42:f816:3eff:fe4f:c2d6 | 20:50 |
| johnsom | Just never made it to the instance | 20:50 |
| johnsom | Pretty sure the port ID is 2ff7db61-baf2-45ed-928d-59213321f736 | 20:51 |
| haleyb | johnsom: which tempest job? | 20:53 |
| johnsom | https://review.opendev.org/#/c/738715/ | 20:53 |
| johnsom | octavia-v2-dsvm-scenario-ipv6-only | 20:54 |
| haleyb | that would explain why https://review.opendev.org/#/c/739119/ didn't have the ipv6 job | 20:55 |
| johnsom | Yeah, sorry. Created that to get the nova console info. | 20:56 |
| *** gcheresh_ has quit IRC | 20:57 | |
| haleyb | johnsom: dnsmasq error | 21:00 |
| haleyb | bad command line options: cannot match tags in --dhcp-host | 21:00 |
| haleyb | FAILED to start up | 21:00 |
| haleyb | https://zuul.opendev.org/t/openstack/build/288a35a5d2dc4404944f9acff9388068/log/controller/logs/screen-q-dhcp.txt | 21:01 |
| haleyb | there were some issues w/dnsmasq recently and ipv6 addresses, let me find the change | 21:02 |
| lxkong | rm_work: hmm... which request will hit _lookup() method in pool.py? | 21:15 |
| johnsom | lxkong Isn't that just someone putting in a pool ID that doesn't exist or they don't have access to? Like terraform going out to lunch? | 21:17 |
| lxkong | johnsom: i checked the project ID, it seems to be one of our internal service tenant which is doing periodical query, i'm still filtering the logs | 21:18 |
| johnsom | lxkong It's coming from an API call attempting to do something with a member, but the pool ID is bogus. | 21:20 |
| johnsom | So API call like GET http://198.51.100.10:9876/v2/lbaas/pools/24a43e68-36de-45f6-89cf-c03df583131d/members/957a1ace-1bd2-449b-8455-820b6e4b63f3 where 24a43e68-36de-45f6-89cf-c03df583131d is a non-existing pool ID | 21:22 |
| lxkong | johnsom: yeah, thanks for the hint, i will dig more | 21:22 |
| johnsom | It could be any of the member calls. | 21:22 |
| johnsom | haleyb Those look like just checks if it supports that tag. I'm not sure that is the root cause. | 21:30 |
| johnsom | I thought neutron was using radvd for ipv6 router advertisements. | 21:33 |
| haleyb | johnsom: yes, just found that too after looking through other reviews. strange it appears more than once but that's a different problem | 21:33 |
| haleyb | i though this was an isolated subnet | 21:33 |
| johnsom | Yeah, I think that singleton, isn't... lol | 21:33 |
| haleyb | dnsmasq can also do RA on an isolated subnet. there is one other error in the dhcp-agent log regarding a tap without a MAC, but maybe not related | 21:35 |
| johnsom | Yeah, that is interesting | 21:35 |
| haleyb | johnsom: i checked a recently merged neutron change, it has all the same dhcp and MAC address things in the dhcp log, so nothing to see there | 21:45 |
| haleyb | johnsom: i never see that fd00::42 prefix in the l3 agent log | 21:49 |
| johnsom | Yeah, I didn't either | 21:49 |
| haleyb | that's why i thought this was an isolated subnet for management, not connected via a router | 21:50 |
| johnsom | FYI: https://zuul.opendev.org/t/openstack/build/288a35a5d2dc4404944f9acff9388068/log/controller/logs/devstacklog.txt#18260 | 21:53 |
| johnsom | The network and subnet created | 21:53 |
| haleyb | johnsom: in that log section i don't see a router created | 21:55 |
| johnsom | Yeah, I don't think we do. It's just a network/subnet | 21:56 |
| johnsom | Is that what changed? For slaac do we need one? | 21:57 |
| johnsom | We have never put a router on that network to my knowledge. | 21:57 |
| haleyb | so in that case dnsmasq should be used for the RA if i'm remembering correctly | 21:59 |
| johnsom | Ok. | 21:59 |
| haleyb | brb | 21:59 |
| haleyb | johnsom: i'll have to look tomorrow, but maybe there's a bug in that code, not sure now much the isolated network w/SLAAC is tested | 22:07 |
| johnsom | haleyb Tomorrow is a holiday, but yeah, thanks for having a look. | 22:08 |
| haleyb | johnsom: oh yeah, right. it should be easy enough to test by hand too. maybe send an email to Rodolfo, he's been in the dhcp agent code recently | 22:09 |
| haleyb | well then, it's the weekend! | 22:09 |
| johnsom | o/ | 22:09 |
| *** armax has joined #openstack-lbaas | 22:42 | |
| *** rcernin has joined #openstack-lbaas | 22:42 | |
| *** tkajinam has joined #openstack-lbaas | 22:55 | |
| *** luksky has quit IRC | 23:00 | |
| *** born2bake has quit IRC | 23:22 | |
| *** armax has quit IRC | 23:25 | |
| *** gthiemonge has quit IRC | 23:40 | |
| *** gthiemonge has joined #openstack-lbaas | 23:40 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!