| *** yamamoto has joined #openstack-kuryr | 00:03 | |
| *** yamamoto has quit IRC | 00:09 | |
| *** hongbin has joined #openstack-kuryr | 00:40 | |
| *** rh-jelabarre has joined #openstack-kuryr | 00:52 | |
| *** yamamoto has joined #openstack-kuryr | 01:05 | |
| *** yamamoto has quit IRC | 01:10 | |
| *** openstack has joined #openstack-kuryr | 01:17 | |
| *** ChanServ sets mode: +o openstack | 01:17 | |
| *** phuoc has quit IRC | 01:18 | |
| *** phuoc has joined #openstack-kuryr | 01:23 | |
| *** yamamoto has joined #openstack-kuryr | 02:07 | |
| *** yamamoto has quit IRC | 02:12 | |
| *** rh-jelabarre has quit IRC | 02:37 | |
| *** yamamoto has joined #openstack-kuryr | 03:09 | |
| *** yamamoto has quit IRC | 03:14 | |
| *** yamamoto has joined #openstack-kuryr | 04:10 | |
| *** kiseok7 has quit IRC | 04:13 | |
| *** yamamoto has quit IRC | 04:15 | |
| *** hongbin has quit IRC | 04:22 | |
| *** janonymous has joined #openstack-kuryr | 04:38 | |
| *** yamamoto has joined #openstack-kuryr | 05:12 | |
| *** janki has joined #openstack-kuryr | 05:15 | |
| *** yamamoto has quit IRC | 05:18 | |
| snapiri | Good morning | 05:29 |
|---|---|---|
| *** yamamoto has joined #openstack-kuryr | 05:30 | |
| *** gcheresh has joined #openstack-kuryr | 06:15 | |
| *** yboaron has quit IRC | 06:36 | |
| *** janonymous has quit IRC | 07:07 | |
| *** pcaruana has joined #openstack-kuryr | 07:07 | |
| *** celebdor1 has joined #openstack-kuryr | 07:27 | |
| dmellado | morning snapiri | 07:58 |
| *** yboaron has joined #openstack-kuryr | 07:58 | |
| *** garyloug has joined #openstack-kuryr | 08:08 | |
| *** snapiri- has joined #openstack-kuryr | 08:11 | |
| *** snapiri has quit IRC | 08:13 | |
| *** jistr|off is now known as jistr | 08:14 | |
| *** celebdor1 has quit IRC | 08:24 | |
| openstackgerrit | Daniel Mellado proposed openstack/kuryr-kubernetes master: Add contrib folder to irrelevant files https://review.openstack.org/577427 | 08:30 |
| *** snapiri- has left #openstack-kuryr | 08:58 | |
| *** snapiri has joined #openstack-kuryr | 08:59 | |
| snapiri | dmellado: any idea why | 09:00 |
| snapiri | I get timeouts on gate tests | 09:00 |
| dmellado | snapiri: why what? :D | 09:00 |
| dmellado | oh, got it | 09:00 |
| snapiri | :) | 09:00 |
| dmellado | snapiri: actually I'm waiting on a run to check it myself | 09:00 |
| snapiri | e.g. https://review.openstack.org/#/c/577656/ | 09:00 |
| dmellado | dulek: was also investigating it but looks like it has been dependant on the infra | 09:00 |
| dmellado | dulek: did you end up putting any patch related to it? | 09:01 |
| snapiri | dmellado: Seems like it began on Friday evening / Saturday | 09:01 |
| dulek | snapiri, dmellado: Lately I've seen issues with Octavia Amphora IP conflicts. | 09:02 |
| snapiri | dmellado, dulek: the ones that time-out are only the ones with daemon... maybe it is related? | 09:03 |
| dulek | snapiri, dmellado: Check out octavia-api logs to be sure, if it's that, you'll see lots of red logs there. | 09:03 |
| * dulek looking. | 09:03 | |
| snapiri | dulek: kuryr-kubernetes-tempest-lbaasv2 and kuryr-kubernetes-tempest-octavia succeed. kuryr-kubernetes-tempest-daemon-octavia and kuryr-kubernetes-tempest-daemon-lbaasv2 fail | 09:04 |
| snapiri | also the openshift and containerized daemon gates fail | 09:05 |
| dmellado | dulek: check this log from that job | 09:05 |
| dmellado | on kuryr-kubernetes | 09:05 |
| dmellado | http://logs.openstack.org/56/577656/1/check/kuryr-kubernetes-tempest-daemon-octavia/bbfea60/controller/logs/screen-kuryr-kubernetes.txt.gz | 09:05 |
| dmellado | there are some traces | 09:05 |
| dulek | snapiri: I'll check that, just a second. | 09:05 |
| dmellado | seems like werkzeug might've killed us | 09:06 |
| dmellado | snapiri: | 09:06 |
| dmellado | dulek: | 09:06 |
| dmellado | cotyledon | 09:07 |
| snapiri | dmellado: I see quite a few of these: "kuryr_kubernetes.handlers.logging ResourceNotReady: Resource not ready: VIFOpenVSwitch(..." | 09:08 |
| snapiri | not sure what it means :-\ | 09:09 |
| dmellado | snapiri: if you check on the daemon logs | 09:09 |
| dmellado | you'll see the cotyledon issues that I'm speaking about | 09:09 |
| dmellado | basically daemon's not ready | 09:10 |
| dmellado | http://logs.openstack.org/56/577656/1/check/kuryr-kubernetes-tempest-daemon-octavia/bbfea60/controller/logs/screen-kuryr-daemon.txt.gz | 09:10 |
| dmellado | snapiri: http://paste.openstack.org/show/724200/ | 09:10 |
| dulek | dmellado: This is the root cause, IMO: Ie0e8984c4c02a426eb1e7cbac8c5117aaaff5c40 | 09:12 |
| dulek | https://review.openstack.org/#/c/577534/ | 09:12 |
| dmellado | dulek: heh, thanks on the url | 09:12 |
| dulek | BTW - flask is used in non-daemonized as well, but only for controller healthchecks. | 09:13 |
| dulek | dmellado: And we don't test containerized without daemon - so it's not happening. | 09:13 |
| dmellado | yep, in this case flask bump has bumped also werkzeug and this probably the issue | 09:13 |
| dmellado | so options here: | 09:15 |
| dmellado | 1) Either pin werkzeug | 09:15 |
| dmellado | 2) Try to fix it up | 09:15 |
| * dmellado spawning a daemonized environment | 09:15 | |
| dmellado | xD | 09:15 |
| snapiri | dulek: Excellent catch! | 09:15 |
| dulek | dmellado: I'm fixing it, getting out of sync with global requirements is not a great idea. | 09:16 |
| * snapiri is impressed | 09:16 | |
| dmellado | dulek: are you on it then? I don't want to duplicate work here | 09:16 |
| dmellado | snapiri: | 09:16 |
| dmellado | hound on openstack is your friend xD | 09:16 |
| snapiri | xD | 09:16 |
| dmellado | http://codesearch.openstack.org/ | 09:16 |
| dmellado | :P | 09:16 |
| dmellado | I guess the werkzeug/flask did the trick | 09:17 |
| dulek | dmellado: Sure, I'll try to handle this, if not I'll ping you. | 09:17 |
| dmellado | dulek: ack, no worries | 09:17 |
| * snapiri gone for lunch, BBL | 09:18 | |
| dulek | Hm, interestingly threaded=False is still the default… | 09:18 |
| dmellado | dulek: hm | 09:19 |
| dulek | Ah, but Flask is overwriting that. Flask, why you're doing that?! | 09:19 |
| dmellado | then if we don't override that it should be ok | 09:19 |
| dmellado | lol | 09:19 |
| dmellado | Flask, flaaask | 09:19 |
| dmellado | https://www.youtube.com/watch?v=LfmrHTdXgK4 | 09:19 |
| * dulek tries git blame, it's too odd. | 09:20 | |
| * dmellado getting a coffe, brb | 09:20 | |
| dulek | "Threaded mode is enabled by default." - and that's it. They're not checking if processes == 1 before enabling it… | 09:22 |
| dmellado | botched job... | 09:23 |
| dulek | Okay, at least fix is pretty easy, give me a few minutes. | 09:23 |
| dmellado | let'd disable it and that'd be it | 09:23 |
| dmellado | sure, thanks dulek | 09:23 |
| openstackgerrit | Michał Dulko proposed openstack/kuryr-kubernetes master: Explicitly set threaded=False when starting Flask https://review.openstack.org/577760 | 09:29 |
| dulek | This should solve it. Other Flask servers we're starting are not multiprocess, so it was required only for the daemon one. | 09:30 |
| dmellado | Awesome, one less thing | 09:32 |
| dmellado | After this, we might need to check out a few more timeouts | 09:35 |
| dmellado | + I was wondering about adding containers logs to the ci logs too | 09:36 |
| *** kiseok7 has joined #openstack-kuryr | 09:39 | |
| dulek | dmellado: Containers logs? They're there, I think. | 09:40 |
| dulek | dmellado: Look in kubernetes/pod_logs folder. | 09:40 |
| dmellado | Did we already add those? | 09:40 |
| dmellado | Ohz yeah xD | 09:40 |
| dulek | dmellado: Hey, ping me once you're fine with multinode patch. I'm not following the Gerrit email stream too closely. | 10:15 |
| dmellado | hey dulek, I'm actually checking some timeouts there myself too | 10:16 |
| dmellado | feel free to take a look once the experimental pipeline ends | 10:16 |
| dmellado | dulek: in the meanwhile you can +A this | 10:17 |
| dmellado | as is quite trivial | 10:17 |
| dmellado | https://review.openstack.org/#/c/577427/ | 10:17 |
| dmellado | dulek: ping | 10:44 |
| dmellado | we're having issues with the timeouts | 10:44 |
| dulek | dmellado: What's up? | 10:44 |
| dmellado | even on trivial patches.... | 10:44 |
| dmellado | https://review.openstack.org/#/c/577427/ | 10:44 |
| dmellado | i.e. | 10:44 |
| dmellado | let's take a look and see if we're overlooking somethign | 10:45 |
| dmellado | something | 10:45 |
| dmellado | or if infra has any issues | 10:45 |
| dulek | dmellado: I don't understand you. This looks exactly as https://bugs.launchpad.net/kuryr-kubernetes/+bug/1778483 | 10:45 |
| openstack | Launchpad bug 1778483 in kuryr-kubernetes "kuryr-daemon broken with Flask update" [Critical,In progress] - Assigned to Michal Dulko (michal-dulko-f) | 10:45 |
| dmellado | oh, damn | 10:46 |
| dmellado | thought it was merged already | 10:46 |
| dulek | dmellado: Why would you expect it not happening on trivial patch that triggers whole gate? | 10:46 |
| dmellado | nvm | 10:46 |
| dmellado | too many things | 10:46 |
| dulek | dmellado: :) | 10:46 |
| dmellado | lol | 10:46 |
| dulek | dmellado: Same here! :) | 10:46 |
| * dmellado needs holidays and beach xD | 10:46 | |
| dulek | dmellado: You know who'll have holidays soon? | 10:47 |
| dulek | dmellado: Polish football team. :D | 10:47 |
| dmellado | lol | 10:48 |
| dmellado | sorry on the 0-3 dude! | 10:49 |
| dmellado | I feel pity about south korea, though | 10:49 |
| dmellado | it seems that there's a player who as they didn't reach 1/8 of the championship | 10:49 |
| dmellado | will be now forced to go to military service | 10:50 |
| dulek | dmellado: Aww, that's bad. Involuntary military service is something that shouldn't exist in civilized world. | 10:51 |
| dmellado | totally agree | 10:51 |
| dulek | dmellado: Anyway with 0:3 result at least we get some nice jokes. | 10:51 |
| dmellado | it used to exist in Spain too, but they dropped it way before it was my turn | 10:51 |
| dulek | My favorite is: | 10:51 |
| dulek | dmellado: Same here. ;) | 10:52 |
| dulek | So Argentina shown us that they're strong candidates to be worst team of world cup. Unfortunately the Polish answer was devastating. :D | 10:52 |
| dmellado | lol | 10:52 |
| dmellado | dulek: just wait for Spain | 10:53 |
| dmellado | I'm pretty sure we'll screw badly at some point xD | 10:53 |
| dulek | dmellado: Seems like everybody does that in this world cup. :D | 10:54 |
| dmellado | well, at least it's being a funny one so far xD | 10:54 |
| dulek | Exactly! | 10:54 |
| dmellado | heh, and see | 10:55 |
| dmellado | google predicts you'll lose against Japan! xD | 10:55 |
| dmellado | 39% japan win | 10:55 |
| dmellado | 30% tie | 10:56 |
| dmellado | 31% poland wins | 10:56 |
| dulek | dmellado: Well, you know, it's a match for honor of Polish team. And you know, Japan is known for putting honor very high in their culture… :P | 10:57 |
| dmellado | xD | 11:01 |
| dmellado | dulek: when I was a kid they used to air a japanese cartoon about soccer, Captain Tsubasa | 11:03 |
| dmellado | have you ever seen that one? | 11:03 |
| dmellado | pretty sure it came to Spain years and years after its original air date as it looks quite old, but... xD | 11:03 |
| dmellado | https://en.wikipedia.org/wiki/Captain_Tsubasa | 11:03 |
| dulek | dmellado: Of course. It was translated to "Captain Hawk" here in Poland IIRC. | 11:04 |
| dmellado | I always wondered how they could reach the goalkeeper if it looked like he was kilometers away | 11:06 |
| *** atoth has joined #openstack-kuryr | 11:15 | |
| *** garyloug has quit IRC | 11:50 | |
| *** phuoc has quit IRC | 11:56 | |
| *** phuoc has joined #openstack-kuryr | 11:57 | |
| *** garyloug has joined #openstack-kuryr | 11:59 | |
| *** rh-jelabarre has joined #openstack-kuryr | 12:15 | |
| *** garyloug has quit IRC | 12:38 | |
| openstackgerrit | Genadi Chereshnya proposed openstack/kuryr-tempest-plugin master: Testing VM to loadBalancer service connectivity https://review.openstack.org/571968 | 12:52 |
| *** yboaron_ has joined #openstack-kuryr | 13:29 | |
| *** yboaron has quit IRC | 13:31 | |
| openstackgerrit | Daniel Mellado proposed openstack/kuryr-tempest-plugin master: Skip service tests https://review.openstack.org/577821 | 13:41 |
| dulek | dmellado: celebdor is on PTO until Wednesday, so it might be better to find someone else to take care of bug 1778516. | 13:46 |
| openstack | bug 1778516 in kuryr-kubernetes "Amphora fails to start with IP address already allocated error" [Critical,New] https://launchpad.net/bugs/1778516 - Assigned to Antoni Segura Puimedon (celebdor) | 13:46 |
| dulek | dmellado: And I'm *not* volunteering. ;) | 13:46 |
| dmellado | hmmm I didn't know, thought he was coming back this afternoon as he asked me for an environment | 13:47 |
| dmellado | ... | 13:47 |
| dmellado | hum | 13:47 |
| dmellado | yboaron_: would you like to take a look at that? | 13:47 |
| dmellado | btw dulek | 13:47 |
| * dulek checks again. | 13:47 | |
| dmellado | in order to start the merge fest and so | 13:47 |
| dmellado | I've submitted a patch to temporarily skip service tests | 13:47 |
| dmellado | then hopefully we'll be able to merge your patch | 13:47 |
| dmellado | and a few after | 13:48 |
| dmellado | then we'll reenable them | 13:48 |
| dulek | dmellado: I'm not strongly against it, but it's super dangerous, services are core functionality. | 13:49 |
| dmellado | dulek: yeah, I know | 13:49 |
| dmellado | so we'll have to take extra care on reviewing | 13:49 |
| dmellado | basically it's only to unblock the gate for a while | 13:49 |
| dmellado | otherwise dependencies won't allow us to merge any patch | 13:50 |
| dulek | dmellado: I have 10 minutes to the meeting. I haven't tried to debug IP conflict, I'll look now. | 13:50 |
| dulek | dmellado: Had you taken look at that? | 13:50 |
| dmellado | dulek: I've tried, but I can't reproduce it locally xD | 13:50 |
| dmellado | so worst case I'll ask for a VM to infra | 13:50 |
| dmellado | but I'd like to start fixing your patch issue on the meanwhile | 13:51 |
| dmellado | otherwise we'll have the same issue | 13:51 |
| dmellado | ltomasbo: yboaron_ do you have some time to take a look at that? | 13:51 |
| ltomasbo | dmellado, look at what? | 13:52 |
| yboaron_ | dmellado, in a meeting , what ? | 13:52 |
| dmellado | https://bugs.launchpad.net/kuryr-kubernetes/+bug/1778516 | 13:52 |
| openstack | Launchpad bug 1778516 in kuryr-kubernetes "Amphora fails to start with IP address already allocated error" [Critical,New] - Assigned to Antoni Segura Puimedon (celebdor) | 13:52 |
| dmellado | ^^ | 13:52 |
| dmellado | that one | 13:52 |
| dmellado | I assigned it to celebdor ass he did some previous fix on that | 13:53 |
| dmellado | but if he's on pto 'til wed I'd like someone to try take a look at it beforehand | 13:53 |
| yboaron_ | dmellado, I'll take a look , altough as far as I remember we merged Toni's patch that define separate pool for octavia and K8s services | 13:54 |
| dmellado | yboaron_: yeah, that's it | 13:54 |
| dmellado | we need to check what was happening | 13:54 |
| dmellado | yboaron_: thanks! I'll assign it to you then | 13:54 |
| yboaron_ | dmellado, I'll take a look later | 13:54 |
| dmellado | yboaron_: ack, all yours now | 13:55 |
| *** yboaron_ has quit IRC | 13:59 | |
| *** salv-orlando has joined #openstack-kuryr | 14:03 | |
| *** salv-orlando has quit IRC | 14:08 | |
| *** maysamacedos has joined #openstack-kuryr | 14:08 | |
| *** yboaron_ has joined #openstack-kuryr | 14:13 | |
| *** janki has quit IRC | 14:22 | |
| *** yamamoto has quit IRC | 14:26 | |
| *** hongbin has joined #openstack-kuryr | 14:30 | |
| openstackgerrit | Shachar Snapiri proposed openstack/kuryr-kubernetes master: Remove redundant line from dragonflow config https://review.openstack.org/577656 | 14:36 |
| *** phuoc has quit IRC | 14:40 | |
| *** phuoc has joined #openstack-kuryr | 14:45 | |
| *** garyloug has joined #openstack-kuryr | 14:47 | |
| *** phuoc has quit IRC | 14:49 | |
| *** phuoc has joined #openstack-kuryr | 14:54 | |
| *** janki has joined #openstack-kuryr | 14:55 | |
| *** phuoc has quit IRC | 14:58 | |
| *** salv-orlando has joined #openstack-kuryr | 15:04 | |
| *** rh-jelabarre has quit IRC | 15:05 | |
| *** salv-orlando has quit IRC | 15:08 | |
| *** pcaruana has quit IRC | 15:13 | |
| *** gcheresh has quit IRC | 15:15 | |
| *** yamamoto has joined #openstack-kuryr | 15:16 | |
| *** rh-jelabarre has joined #openstack-kuryr | 15:18 | |
| *** phuoc has joined #openstack-kuryr | 15:21 | |
| *** phuoc has quit IRC | 15:25 | |
| *** phuoc has joined #openstack-kuryr | 15:40 | |
| *** phuoc has quit IRC | 15:45 | |
| *** phuoc has joined #openstack-kuryr | 15:59 | |
| *** salv-orlando has joined #openstack-kuryr | 16:04 | |
| *** phuoc_ has joined #openstack-kuryr | 16:05 | |
| *** phuoc has quit IRC | 16:06 | |
| *** yboaron_ has quit IRC | 16:06 | |
| *** salv-orlando has quit IRC | 16:09 | |
| *** phuoc_ has quit IRC | 16:15 | |
| *** salv-orlando has joined #openstack-kuryr | 17:05 | |
| *** salv-orlando has quit IRC | 17:10 | |
| *** janki has quit IRC | 17:59 | |
| *** salv-orlando has joined #openstack-kuryr | 18:06 | |
| *** salv-orlando has quit IRC | 18:11 | |
| *** gcheresh has joined #openstack-kuryr | 18:27 | |
| *** atoth has quit IRC | 18:58 | |
| *** salv-orlando has joined #openstack-kuryr | 19:07 | |
| *** salv-orlando has quit IRC | 19:11 | |
| *** pcaruana has joined #openstack-kuryr | 19:27 | |
| *** gcheresh has quit IRC | 19:40 | |
| *** yamamoto has quit IRC | 19:44 | |
| *** yamamoto has joined #openstack-kuryr | 19:46 | |
| *** yamamoto has quit IRC | 19:49 | |
| *** yamamoto has joined #openstack-kuryr | 19:49 | |
| *** yamamoto has quit IRC | 19:49 | |
| *** salv-orlando has joined #openstack-kuryr | 20:08 | |
| *** salv-orlando has quit IRC | 20:13 | |
| *** maysamacedos has quit IRC | 20:40 | |
| *** pcaruana has quit IRC | 20:43 | |
| *** yamamoto has joined #openstack-kuryr | 20:50 | |
| *** yamamoto has quit IRC | 20:58 | |
| *** salv-orlando has joined #openstack-kuryr | 21:08 | |
| *** salv-orlando has quit IRC | 21:13 | |
| *** yamamoto has joined #openstack-kuryr | 21:54 | |
| *** yamamoto has quit IRC | 21:59 | |
| *** salv-orlando has joined #openstack-kuryr | 22:09 | |
| *** salv-orlando has quit IRC | 22:14 | |
| *** maysamacedos has joined #openstack-kuryr | 22:23 | |
| *** yamamoto has joined #openstack-kuryr | 22:56 | |
| *** yamamoto has quit IRC | 23:02 | |
| *** salv-orlando has joined #openstack-kuryr | 23:10 | |
| *** pmannidi has joined #openstack-kuryr | 23:14 | |
| *** salv-orlando has quit IRC | 23:15 | |
| *** hongbin has quit IRC | 23:17 | |
| *** garyloug has quit IRC | 23:32 | |
| *** rh-jelabarre has quit IRC | 23:53 | |
| *** yamamoto has joined #openstack-kuryr | 23:57 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!