| *** gcb has joined #openstack-release | 00:15 | |
| *** csomerville has quit IRC | 00:56 | |
| *** cody-somerville has joined #openstack-release | 01:03 | |
| *** dave-mccowan has joined #openstack-release | 01:06 | |
| *** harlowja has quit IRC | 01:13 | |
| *** zhurong has joined #openstack-release | 01:13 | |
| *** hongbin has joined #openstack-release | 01:14 | |
| *** bobh has quit IRC | 01:21 | |
| *** mriedem1 has joined #openstack-release | 01:22 | |
| *** mriedem has quit IRC | 01:24 | |
| *** yamahata has quit IRC | 01:43 | |
| *** iyamahat has quit IRC | 01:43 | |
| *** bobh has joined #openstack-release | 01:45 | |
| *** mriedem1 is now known as mriedem | 01:59 | |
| *** ricolin has joined #openstack-release | 02:16 | |
| *** mriedem has quit IRC | 03:02 | |
| *** bobh has quit IRC | 03:11 | |
| *** udesale has joined #openstack-release | 03:33 | |
| *** hongbin has quit IRC | 03:36 | |
| *** ekcs has joined #openstack-release | 03:47 | |
| *** dave-mccowan has quit IRC | 04:01 | |
| *** Dinesh_Bhor has joined #openstack-release | 04:11 | |
| *** Dinesh_Bhor has quit IRC | 04:14 | |
| *** Dinesh_Bhor has joined #openstack-release | 04:15 | |
| *** harlowja has joined #openstack-release | 04:31 | |
| *** yamahata has joined #openstack-release | 04:45 | |
| *** harlowja has quit IRC | 04:47 | |
| *** zhurong has quit IRC | 04:55 | |
| *** armax has quit IRC | 05:14 | |
| *** armax has joined #openstack-release | 05:15 | |
| *** armax has quit IRC | 05:15 | |
| *** armax has joined #openstack-release | 05:16 | |
| *** armax has quit IRC | 05:16 | |
| *** armax has joined #openstack-release | 05:17 | |
| *** armax has quit IRC | 05:17 | |
| *** Dinesh_Bhor has quit IRC | 05:17 | |
| *** armax has joined #openstack-release | 05:17 | |
| *** armax has quit IRC | 05:18 | |
| *** armax has joined #openstack-release | 05:18 | |
| *** armax has quit IRC | 05:18 | |
| *** harlowja has joined #openstack-release | 05:46 | |
| *** zhurong has joined #openstack-release | 05:50 | |
| *** Adri2000 has joined #openstack-release | 06:08 | |
| *** ekcs has quit IRC | 06:20 | |
| *** harlowja has quit IRC | 06:42 | |
| *** amoralej|off is now known as amoralej | 06:44 | |
| *** Dinesh_Bhor has joined #openstack-release | 06:50 | |
| *** Dinesh_Bhor has quit IRC | 06:50 | |
| *** udesale has quit IRC | 07:01 | |
| *** udesale has joined #openstack-release | 07:01 | |
| *** shardy has joined #openstack-release | 07:11 | |
| *** apuimedo has joined #openstack-release | 07:28 | |
| *** dtantsur|afk is now known as dtantsur | 08:09 | |
| *** pbandark has joined #openstack-release | 08:12 | |
| *** lucas-afk is now known as lucasagomes | 08:18 | |
| *** shardy is now known as shardy_afk | 08:21 | |
| *** electrofelix has joined #openstack-release | 08:27 | |
| *** cody-somerville has quit IRC | 08:36 | |
| *** shardy_afk is now known as shardy | 08:36 | |
| *** cody-somerville has joined #openstack-release | 08:36 | |
| *** cody-somerville has quit IRC | 08:36 | |
| *** cody-somerville has joined #openstack-release | 08:36 | |
| *** dtantsur is now known as dtantsur|brb | 08:41 | |
| *** openstackgerrit has quit IRC | 08:49 | |
| *** mjblack has quit IRC | 09:10 | |
| *** sdague has joined #openstack-release | 09:37 | |
| *** openstackgerrit has joined #openstack-release | 09:44 | |
| openstackgerrit | Akihiro Motoki proposed openstack/reno master: Clean up rendered HTML with openstackdocstheme https://review.openstack.org/485129 | 09:44 |
|---|---|---|
| openstackgerrit | Merged openstack/reno master: Clean up rendered HTML with openstackdocstheme https://review.openstack.org/485129 | 09:56 |
| *** wolverineav has joined #openstack-release | 10:15 | |
| *** udesale has quit IRC | 10:24 | |
| *** Dinesh_Bhor has joined #openstack-release | 10:27 | |
| openstackgerrit | Merged openstack/releases master: request 4.1.0 release for congress https://review.openstack.org/484161 | 10:28 |
| *** yamahata has quit IRC | 10:30 | |
| *** tellesnobrega has quit IRC | 10:41 | |
| *** zhurong has quit IRC | 10:53 | |
| *** gcb has quit IRC | 10:55 | |
| *** gcb has joined #openstack-release | 10:57 | |
| *** alexchadin has joined #openstack-release | 11:09 | |
| dims | ttx : dhellmann : o/ | 11:15 |
| dhellmann | dims, ttx: o/ | 11:15 |
| dims | dhellmann : only thing i am waiting on is glance_store | 11:25 |
| *** dave-mccowan has joined #openstack-release | 11:26 | |
| dhellmann | dims : the requirements/constraints update? | 11:30 |
| dims | a new release dhellmann - update for your review https://review.openstack.org/#/c/483467/ | 11:30 |
| dhellmann | ah | 11:31 |
| dhellmann | forgot about that one | 11:31 |
| dhellmann | looks like jokke_ and rosmaita had more patches they wanted in, let me see what happens if I update that | 11:32 |
| openstackgerrit | Doug Hellmann proposed openstack/releases master: glance-store 0.21.0 https://review.openstack.org/483467 | 11:33 |
| openstackgerrit | Doug Hellmann proposed openstack/releases master: teach new-release how to step over series boundaries https://review.openstack.org/483468 | 11:33 |
| dhellmann | dims : I rebased so we could land the tool change while we wait for the glance team on the library release | 11:34 |
| dims | sounds good dhellmann. | 11:35 |
| *** wolverineav has quit IRC | 11:35 | |
| gcb | dims, dhellmann : Do we need other releases for oslo except tooz? | 11:39 |
| dhellmann | gcb : I think we had some bug fixes in oslo.config, did those land? | 11:39 |
| gcb | yes, I saw we just merged fix for oslo.config , that's why I'm asking :-) let me check | 11:40 |
| dhellmann | gcb : yes, let's release that | 11:41 |
| dhellmann | gcb : I'll pull a list of other changes quickly, give me a minute | 11:41 |
| gcb | dhellmann: okay, I'm also listing unreleased changes of oslo .... | 11:42 |
| dhellmann | gcb : http://paste.openstack.org/show/615836/ | 11:43 |
| dhellmann | I don't know if we need that oslo.db exception change | 11:44 |
| dhellmann | or the castellan change | 11:45 |
| *** tellesnobrega has joined #openstack-release | 11:45 | |
| *** wolverineav has joined #openstack-release | 11:45 | |
| dhellmann | might as well, though? | 11:45 |
| gcb | I think we don't need new release for oslo.db castellan, the commit seems not bug fix | 11:46 |
| *** wolverineav has quit IRC | 11:46 | |
| dhellmann | that makes sense | 11:47 |
| gcb | I'm little worried about tooz, we still can't figure out a cinder bug, trying to revert some patches in https://review.openstack.org/485035 and https://review.openstack.org/485036 | 11:47 |
| dhellmann | gcb : is the problem with cinder resolved by moving back to an older release of tooz? or are we guessing at the cause? | 11:49 |
| gcb | dhellmann: I will release oslo.config now, with minor change ? | 11:49 |
| dhellmann | gcb : ++ | 11:49 |
| gcb | dhellmann: as I know an older release of tooz works well for Cinder | 11:50 |
| dhellmann | gcb : do we need to block that release in the requirements list? | 11:51 |
| dhellmann | jd_ : are you around? ^^ | 11:51 |
| * jd_ appears in a cape surrounded by smoke | 11:51 | |
| jd_ | who summoned me? | 11:51 |
| gcb | dhellmann: we have done that | 11:52 |
| dhellmann | jd_ : apparently the cinder team is having a problem with tooz that is resolved by moving back to an older release, but you're blocking the reverts | 11:52 |
| dhellmann | so the question is do we block the bad release, approve the reverts, or do something else? | 11:52 |
| jd_ | dhellmann: fix Cinder? | 11:52 |
| dhellmann | do we know what's actually wrong? | 11:52 |
| jd_ | dhellmann: good question! I'm glad you ask. No we don't because nobody in Cinder stepped up to look deeply into it it seems :( | 11:53 |
| dhellmann | smcginnis : are you around? | 11:53 |
| dhellmann | ^^ | 11:53 |
| jd_ | dhellmann: I asked a question about the error they have but I did not have any reply yet | 11:53 |
| jd_ | I don't understand eventlet :( | 11:53 |
| jd_ | the patches in tooz are not cosmetic, they are fixing a race condition | 11:54 |
| jd_ | so maybe they introduce a new bug, but I can't see it on tooz side, so I tend to think it hits a problem in eventlet and/or cinder | 11:54 |
| dhellmann | all of that sounds sane. | 11:55 |
| jd_ | so I'd prefer to have an understanding of the problem rather than blinding reverting a patch that will re-introduce a bug | 11:55 |
| jd_ | :( | 11:55 |
| jd_ | if anyone knows cinder or eventlet and can give a hint it's https://bugs.launchpad.net/cinder/+bug/1694509/ | 11:55 |
| openstack | Launchpad bug 1694509 in tooz "eventlet errors and cannot switch to a different thread" [High,Confirmed] | 11:55 |
| jd_ | I'll be happy to fix anything in tooz that is proved to be broken :) | 11:56 |
| openstackgerrit | ChangBo Guo(gcb) proposed openstack/releases master: Add final release of oslo.config for Pike https://review.openstack.org/485175 | 11:56 |
| dhellmann | it sounds like maybe the fix for the race condition exposed an issue with eventlet's monkeypatching | 11:58 |
| *** wolverineav has joined #openstack-release | 11:58 | |
| jd_ | dhellmann: this is what I'm scared of TBH, but it's out of my expertise (and also interest, you know my disdain for eventlet monkeys) | 11:59 |
| *** wolverineav has quit IRC | 12:01 | |
| dhellmann | jd_, gcb, smcginnis : it sounds like we need the cinder team's help on this, before the requirements freeze comes up next week. | 12:03 |
| dhellmann | it would be weird to block the current tooz release, then branch, because we might never be able to update it on that branch | 12:03 |
| dhellmann | but we need to work on this problem as a whole, not in isolation, to make the whole release work | 12:04 |
| tonyb | dhellmann: Can you take a look at https://review.openstack.org/#/c/482365/ when you have time. I think it's good to merge | 12:04 |
| *** lucasagomes is now known as lucas-hungry | 12:04 | |
| dhellmann | tonyb : +2a | 12:05 |
| openstackgerrit | Merged openstack/releases master: Add Queens election to the schedule https://review.openstack.org/482365 | 12:09 |
| gcb | dhellmann: dims confirmed that the revert/blacklist for 1.57.4 worked see comment #14 in https://bugs.launchpad.net/cinder/+bug/1694509/. I aslo agree with jd_ we need find out the root cause, but what we need mostly is time. If we can't fix the issue before final release, the only choice may be reverting patches :( | 12:09 |
| openstack | Launchpad bug 1694509 in tooz "eventlet errors and cannot switch to a different thread" [High,Confirmed] | 12:09 |
| jd_ | gcb: we're not going to revert patches to re-introduce an old bug just because the old bug is not seen by Cinder | 12:10 |
| tonyb | dhellmann: Thanks | 12:10 |
| jd_ | gcb: I don't think putting problems under the carpet is a good solution for any project | 12:10 |
| gcb | jd_: sileht aslo points that we' ll break api if we merge https://review.openstack.org/#/c/485035/. so we cant' revert this one, we blacklist 1.57.4 in https://github.com/openstack/requirements/blob/master/global-requirements.txt#L313 that means the fix for older bug doesn't work now | 12:15 |
| gcb | jd_: I will abandon https://review.openstack.org/#/c/485035/ now | 12:16 |
| jd_ | gcb: if you want to break others OpenStack projects because Cinder might be broken, it's up to you :) | 12:16 |
| jd_ | s/break/introduce bugs in/ | 12:17 |
| gcb | jd_: we need cinder team help find which patch affect their tests. then decide how to fix it. I really don't want to break any project :) | 12:18 |
| jd_ | gcb: I agree we need Cinder to dig into that; I asked question 2 days ago on the bug report and for now there's 0 reply | 12:18 |
| jd_ | I don't think we can fix that without Cinder help | 12:18 |
| gcb | jd_: let's raise this in cinder's weekly meeting or discuss this in #openstack-cinder | 12:21 |
| jd_ | gcb: when is that? | 12:22 |
| gcb | jd_: http://eavesdrop.openstack.org/#Cinder_Team_Meeting | 12:23 |
| gcb | Weekly on Monday at 2000 UTC in #openstack-meeting-alt | 12:23 |
| gcb | sorry, ignore last message | 12:24 |
| jd_ | yeah it's today | 12:24 |
| jd_ | ok i might be around | 12:24 |
| jd_ | feel free to ping me if needed | 12:24 |
| gcb | jd_: thanks | 12:26 |
| openstackgerrit | John Trowbridge proposed openstack/releases master: Last release for tripleo-quickstart which supports Mitaka https://review.openstack.org/484909 | 12:26 |
| *** mriedem has joined #openstack-release | 12:31 | |
| *** dtantsur|brb is now known as dtantsur | 12:40 | |
| dtantsur | hi folks! is there a quick way for a PTL to see which projects have which release type and cadence? | 12:47 |
| * dtantsur tries to figure out which requests he has to make till tomorrow | 12:47 | |
| dhellmann | dtantsur : we have some tools in the releases repo | 12:52 |
| dhellmann | you probably want the list-deliverables command | 12:53 |
| dhellmann | dtantsur : try checking out openstack/releases and running "tox -e venv -- list-deliverables -h" | 12:53 |
| *** gcb has quit IRC | 12:54 | |
| dhellmann | dtantsur : in particular, the --team option may be useful for you | 12:54 |
| dhellmann | dtantsur : and -v | 12:54 |
| dtantsur | thanks, I'll try | 13:01 |
| *** udesale has joined #openstack-release | 13:01 | |
| dtantsur | that feeling when tox fails with InvalidSyntax >_< | 13:06 |
| dtantsur | dhellmann: I think setting basepython = python3 is a bit optimistic, given that lazr.restfulclient seems not python3-aware | 13:07 |
| *** amoralej is now known as amoralej|lunch | 13:11 | |
| *** wolverineav has joined #openstack-release | 13:14 | |
| *** mriedem is now known as mriedem_meeting | 13:17 | |
| smcginnis | dhellmann: Around now. | 13:18 |
| smcginnis | We asked for help on the toox issue. | 13:18 |
| smcginnis | Something is not getting monkey patched correctly or something from what I was able to find out. | 13:18 |
| smcginnis | But after several days of multiple projects being affected with gate failures, the best we could do is block that release of tooz. | 13:19 |
| smcginnis | Since before that the logstash showed 0 failures. | 13:19 |
| smcginnis | jd_: Which bug are ou referring to? | 13:20 |
| dhellmann | I think https://bugs.launchpad.net/cinder/+bug/1694509/ | 13:20 |
| openstack | Launchpad bug 1694509 in tooz "eventlet errors and cannot switch to a different thread" [High,Confirmed] | 13:20 |
| smcginnis | dhellmann: OK, I don't know what we're expected to comment on there. | 13:21 |
| dhellmann | I'm not sure | 13:21 |
| dhellmann | jd_ : the current release of tooz is already being blocked in global-requirements it seems | 13:24 |
| dhellmann | smcginnis : we have the most recent release of tooz blocked from use right before we're about to branch it, which means we won't be able to use any bug fix releases on that branch. what do we need to do to get a team together to work on the problem? | 13:26 |
| smcginnis | dhellmann: I had hoped the tooz team would do some more testing to see what was changed to cause the failures. | 13:28 |
| dhellmann | smcginnis : according to jd_ the recent changes fix known race conditions in the library itself | 13:29 |
| dhellmann | we need some folks from cinder and tooz to get together and address the problem | 13:29 |
| smcginnis | dhellmann: Not sure what we can do. Maybe put up a devtack patch to try running iwthout etcd? But it was an intermittent failure, so not sure how valuable that would be. | 13:29 |
| dhellmann | the tooz folks don't know cinder code | 13:29 |
| smcginnis | dhellmann: Especially in a limited time window. | 13:29 |
| smcginnis | dhellmann: I don't think knowledge of cinder code is really required here. | 13:30 |
| smcginnis | dhellmann: But it's intermittent nature certainly makes debugging difficult. | 13:30 |
| jd_ | smcginnis: "the tooz team" lol | 13:30 |
| dhellmann | I don't know. I'm just trying to get people to actually fix it instead of handing it back and forth. | 13:30 |
| smcginnis | Or at least verifying that a change actually addresses it. | 13:30 |
| jd_ | smcginnis: I wish there was such a team :) | 13:30 |
| smcginnis | jd_: Should I send a ML post that tooz is unmaintained? ;) | 13:30 |
| jd_ | smcginnis: if you want! | 13:31 |
| jd_ | it's maintained by me only | 13:31 |
| smcginnis | Not really. | 13:31 |
| jd_ | and random people pressing +2A to make me pleased | 13:31 |
| jd_ | :) | 13:31 |
| jd_ | smcginnis: I don't think having Cinder knowledge is needed, thought it seems to be related to eventlet so that's knowledge that might be needed | 13:32 |
| jd_ | at least | 13:32 |
| jd_ | and also time to dig into that with Cinder as a test base | 13:32 |
| jd_ | smcginnis: I asked explanation about what the error meant but I got no reply :( | 13:32 |
| smcginnis | There were plenty of references to that error online, but all of the examples shown for the cause were very contrived. | 13:33 |
| smcginnis | jd_: Explanation from Cinder? | 13:33 |
| jd_ | smcginnis: from anyone that knows Eventlet :) | 13:34 |
| jd_ | smcginnis: yeah I googled a bit but did not find anything obvious :( | 13:34 |
| smcginnis | Not aware of anyone in Cinder that is an eventlet expert either. | 13:34 |
| jd_ | and outside Cinder? | 13:35 |
| smcginnis | I would hope someone knows it. | 13:35 |
| dhellmann | mriedem_meeting posted Ibde3c0fff1320a35a799ab25028d123b48e06b0b to blacklist the latest tooz release, was he involved in troubleshooting? | 13:35 |
| smcginnis | dhellmann: Yes | 13:36 |
| smcginnis | dhellmann: He helped track it down to tooz after we spent some time scrambling and getting nowhere tryingn to figure out why things were suddenly failing. | 13:36 |
| openstackgerrit | Merged openstack/releases master: New heatclient newton release https://review.openstack.org/484686 | 13:37 |
| dhellmann | can we pull him into this conversation? | 13:37 |
| *** bobh has joined #openstack-release | 13:37 | |
| mriedem_meeting | jd_: smcginnis: eventlet people would work in oslo.messaging, | 13:37 |
| mriedem_meeting | like bnemec, | 13:37 |
| dhellmann | mriedem_meeting : that team is also hardly active any more | 13:37 |
| mriedem_meeting | and the other guy's name i always forget | 13:37 |
| mriedem_meeting | sileht | 13:38 |
| mriedem_meeting | was the name i was thinking of | 13:38 |
| smcginnis | I'm really concerned the number of libraries we are relying on with one (or less) maintainers. | 13:38 |
| mriedem_meeting | but, maybe i'm thinking of rabbitmq :) | 13:38 |
| dhellmann | mriedem_meeting : yeah, I think you are | 13:38 |
| dhellmann | smcginnis : I am, too. | 13:38 |
| dhellmann | also with the number of times the root cause traces back to something to do with eventlet | 13:39 |
| *** askb has quit IRC | 13:39 | |
| dhellmann | but what I'm most concerned with right now is the impasse that's going to screw tooz's releases for the stable branch of pike | 13:40 |
| smcginnis | dhellmann: My thought would be to branch from last release, then see if we can get some help in Queens to sort it out. | 13:41 |
| dhellmann | smcginnis : if we branch from something before a patch release, we have no room for new version numbers | 13:42 |
| smcginnis | dhellmann: Ah, and we've released subsequent patch releases, so version would have to change. | 13:42 |
| smcginnis | Is it a lot? Revert changes, do release and branch? | 13:43 |
| dhellmann | so if we can't fix the issue, we will need to revert the changes and do a new release to create the branch, then unrevert and potentially backport other fixes later | 13:43 |
| dhellmann | jd_ : ^^ | 13:43 |
| smcginnis | And maybe unrevert slowly to see where the failures start? | 13:44 |
| * ttx pops in just before going to bed | 13:44 | |
| smcginnis | Haven't looked to see how many changes were talking about here. | 13:44 |
| dhellmann | smcginnis : yeah | 13:44 |
| dhellmann | mriedem_meeting identified 2 changes to try reverting | 13:45 |
| jd_ | dhellmann: so reverting is re-introducing a race condition bug | 13:45 |
| dhellmann | although we've had others on master since those were released so who knows | 13:45 |
| jd_ | dhellmann: I don't see that as a gain. Telemetry uses tooz, we don't want that bug in Pike. | 13:45 |
| dhellmann | jd_ : I understand that. Failing to either fix the problem or revert the changes means you get no bug fix releases from tooz for pike. | 13:45 |
| dhellmann | jd_ : the release with those fixes is already being blocked by the global requirements list, so you're already not getting them. | 13:46 |
| smcginnis | We could revert all but that race fix and see how things look in the last couple weeks. | 13:46 |
| jd_ | dhellmann: we don't use requirements – exactly for that kind of reasons… | 13:46 |
| *** sileht has joined #openstack-release | 13:46 | |
| mriedem_meeting | dhellmann: smcginnis: jd_: i'm in a call but this is clear evidence of the problem that was killing the gate for a few days http://status.openstack.org/elastic-recheck/#1694509 | 13:46 |
| dhellmann | jd_ : you do when a distro delivers your packages | 13:46 |
| smcginnis | Failure was intermittent, but it happened enough that we should be able to tell fairly quick if the race fix is the one that introduced the failures. | 13:46 |
| jd_ | dhellmann: sure, but that's not our fault, that's just requirements doing wrong things :) | 13:47 |
| jd_ | dhellmann: we can't fix everything :( | 13:47 |
| *** udesale has quit IRC | 13:47 | |
| dhellmann | jd_ : I'm only asking you to help smcginnis and mriedem_meeting, not to fix it yourself. | 13:47 |
| jd_ | smcginnis: considering the error message I'd bet on it yeah | 13:47 |
| jd_ | dhellmann: how can I help? | 13:47 |
| dhellmann | work the problem? | 13:48 |
| smcginnis | jd_: Well, then you can have a small window in Telemetry, or cause failures in the majority of deployments. | 13:48 |
| *** hongbin has joined #openstack-release | 13:49 | |
| jd_ | dhellmann: that's what I'm doing! | 13:49 |
| *** jrist has quit IRC | 13:53 | |
| dhellmann | smcginnis, jd_ : can we set up a cinder job that runs with tooz from source so we can test changes in tooz against cinder without doing releases? | 13:53 |
| smcginnis | dhellmann: We could going forward. | 13:54 |
| dhellmann | smcginnis : can we do that today, to see if we can prove that one or both of those tooz patches needs to be reverted? | 13:55 |
| * dhellmann has to step away | 13:55 | |
| smcginnis | dhellmann: So do the reverts, and have tests running to see when they get added back? | 13:55 |
| dhellmann | smcginnis : set up the test job, show that you get a failure on master, revert the patch(es) and see if that helps. | 13:55 |
| jd_ | dhellmann: we already know which patch triggers the bug in Cinder, no? | 13:56 |
| dhellmann | jd_ : do we? I didn't think we did? | 13:56 |
| jd_ | dhellmann: we have 2 candidates according to gcb and i'm pretty sure which one of the 2 it is :) | 13:56 |
| smcginnis | Would be good to verify. | 13:56 |
| dhellmann | if we have a cinder patch in question why are we talking about reverting tooz patches? | 13:56 |
| dhellmann | I thought those were in tooz? | 13:56 |
| dhellmann | I really have to go afk. Please continue to work on this together. | 13:56 |
| smcginnis | And know where to focus efforts.Cindner patch? | 13:57 |
| jd_ | dhellmann: it's a bug fix in tooz that triggered an issue in Cinder | 13:57 |
| *** tellesnobrega has quit IRC | 13:57 | |
| smcginnis | So I would say for now, revert that patch and do another release. If that's good, branch from there. Then once the trigger of the failure is identified it can be backported. | 13:58 |
| jd_ | smcginnis: dhellmann: https://review.openstack.org/#/c/478202/ very likely | 13:58 |
| *** tellesnobrega has joined #openstack-release | 13:58 | |
| jd_ | it adds usage of threading.Lock | 13:59 |
| jd_ | and that's likely what's making eventlet explodes | 13:59 |
| *** armax has joined #openstack-release | 13:59 | |
| *** alexchadin has quit IRC | 14:02 | |
| *** jrist has joined #openstack-release | 14:05 | |
| dhellmann | smcginnis : how about trying to revert ^^ | 14:07 |
| dhellmann | oops, jd_ I mean | 14:07 |
| smcginnis | I mean I could, but... :) | 14:07 |
| dhellmann | smcginnis, jd_ : it would be useful to set up that job to test the tooz revert from source against cinder. Who is going to do that? | 14:12 |
| smcginnis | dhellmann: I can get that going. | 14:13 |
| mriedem_meeting | there is already a revert up for the tooz change, but yeah you need a -src- job to test it | 14:13 |
| dhellmann | thank you | 14:13 |
| dhellmann | mriedem_meeting : right, that's what we need | 14:13 |
| mriedem_meeting | although, can't you just push a devstack change that uses LIBS_FROM_GIT for the tooz change, | 14:13 |
| mriedem_meeting | then you don't need a src job | 14:13 |
| dhellmann | oh, there you go, smcginnis ^^ | 14:13 |
| mriedem_meeting | \o/ | 14:13 |
| dhellmann | I knew there was a way to do that without a new job | 14:13 |
| mriedem_meeting | yeah that's basically what the src jobs do | 14:13 |
| smcginnis | That would make every job run that way, no? | 14:13 |
| smcginnis | Not sure we want that. | 14:13 |
| dhellmann | so the devstack patch would depend on the tooz patch, right? | 14:14 |
| mriedem_meeting | smcginnis: it's a DNM patch | 14:14 |
| mriedem_meeting | yes | 14:14 |
| mriedem_meeting | set the TOOZ_BRANCH or whatever in devstack, this is all possible in a single DNM patch | 14:14 |
| * dhellmann goes back to his meeting | 14:14 | |
| mriedem_meeting | that you then run recheck on a dozen times or something | 14:14 |
| smcginnis | Well, it didn't fail every time, but we could start there. | 14:14 |
| dhellmann | yeah, we can run the jobs against the patch a few times | 14:15 |
| mriedem_meeting | it was ~50% fail | 14:15 |
| mriedem_meeting | so should be easy to know | 14:15 |
| mriedem_meeting | i'll dig for a change to show you how to do this | 14:16 |
| smcginnis | OK, so inec a tooz patch is ready we can do a Depends-on devstck patch. Then recheck several times until we're comfortable. | 14:16 |
| mriedem_meeting | the devstack change depends on the tooz revert | 14:16 |
| smcginnis | Somehow, inec == as soon as | 14:16 |
| jd_ | mriedem_meeting: smcginnis: is anyone able to write a small program that reproduces the issue (even 50% of the time)? | 14:17 |
| *** amoralej|lunch is now known as amoralej | 14:17 | |
| smcginnis | Probably easiest doing the devstack change. Would be great to have something to isolate the failure, but not sure the effort on that. | 14:19 |
| openstackgerrit | Merged openstack/releases master: Add final release of oslo.config for Pike https://review.openstack.org/485175 | 14:31 |
| openstackgerrit | Brian Rosmaita proposed openstack/releases master: glance-store 0.21.0 https://review.openstack.org/483467 | 14:35 |
| *** abadger1999 has quit IRC | 14:39 | |
| mriedem_meeting | smcginnis: i'm failing to find the LIBS_FROM_GIT change i was looking for in devstack, but can poke more after this meeting | 14:42 |
| mriedem_meeting | i know scottda has done this | 14:42 |
| smcginnis | I know I've seen it too. There should be some good examples out there somewhere. | 14:45 |
| *** lucas-hungry is now known as lucasagomes | 14:47 | |
| * dims peeks | 14:55 | |
| *** tellesnobrega has quit IRC | 14:55 | |
| *** iyamahat has joined #openstack-release | 14:59 | |
| sileht | smcginnis, dhellmann dims I found the root cause: https://bugs.launchpad.net/cinder/+bug/1694509/comments/19 | 15:02 |
| openstack | Launchpad bug 1694509 in tooz "eventlet errors and cannot switch to a different thread" [High,Confirmed] | 15:02 |
| sileht | mriedem_meeting, ^ | 15:03 |
| dims | sileht : w00t! | 15:03 |
| dims | smcginnis : can we get that code into cinder please? | 15:03 |
| mriedem_meeting | cool | 15:03 |
| mriedem_meeting | told you sileht was the eventlet guy :) | 15:03 |
| *** mriedem_meeting is now known as mriedem | 15:03 | |
| smcginnis | sileht: Awesome, thank you. | 15:03 |
| smcginnis | dims: Was actually just waitingn on Jenkins to check that out, but I'll push it through. | 15:04 |
| dims | mriedem :) | 15:04 |
| dims | jd_ : thanks for twisting sileht 's arm to look at it :) | 15:05 |
| *** electrofelix has quit IRC | 15:05 | |
| *** electrofelix has joined #openstack-release | 15:06 | |
| smcginnis | Unblock for using tooz 1.57.4: https://review.openstack.org/#/c/485233/ | 15:08 |
| dims | smcginnis : throw a depends on to https://review.openstack.org/#/c/447079/ | 15:09 |
| dhellmann | sileht, smcginnis, mriedem : thank you! | 15:15 |
| *** yamahata has joined #openstack-release | 15:15 | |
| sileht | thx to jd_ that have provided the fix before the issue was encounter :p | 15:17 |
| smcginnis | dims: Good call, added that depends on tag. | 15:17 |
| *** gcb has joined #openstack-release | 15:32 | |
| jd_ | dims: sileht is too kind, I can't even twist anything :p | 15:33 |
| *** rajathagasthya has joined #openstack-release | 15:46 | |
| dhellmann | dims : it would be good to merge https://review.openstack.org/#/c/483468/2 so we don't block that glance-store release | 15:49 |
| *** Rockyg has joined #openstack-release | 15:52 | |
| dims | sileht : jd_ :) | 15:53 |
| dims | dhellmann : on it | 15:53 |
| *** gcb has quit IRC | 15:54 | |
| *** gcb has joined #openstack-release | 15:55 | |
| *** tellesnobrega has joined #openstack-release | 15:58 | |
| openstackgerrit | Merged openstack/releases master: teach new-release how to step over series boundaries https://review.openstack.org/483468 | 16:01 |
| *** dtantsur is now known as dtantsur|afk | 16:03 | |
| *** sridhar_ram has quit IRC | 16:06 | |
| *** wolverineav has quit IRC | 16:17 | |
| *** bobh has quit IRC | 16:24 | |
| *** lucasagomes is now known as lucas-afk | 16:24 | |
| *** bobh has joined #openstack-release | 16:30 | |
| *** gcb has quit IRC | 16:50 | |
| openstackgerrit | Merged openstack/releases master: Last release for tripleo-quickstart which supports Mitaka https://review.openstack.org/484909 | 17:05 |
| *** harlowja has joined #openstack-release | 17:25 | |
| *** harlowja has quit IRC | 17:25 | |
| *** harlowja has joined #openstack-release | 17:26 | |
| *** bobh has quit IRC | 17:29 | |
| *** bobh has joined #openstack-release | 17:31 | |
| openstackgerrit | Erno Kuvaja proposed openstack/releases master: glance-store 0.21.0 https://review.openstack.org/483467 | 17:34 |
| *** ekcs has joined #openstack-release | 17:36 | |
| *** Rockyg has quit IRC | 17:54 | |
| *** yamahata has quit IRC | 17:54 | |
| *** iyamahat has quit IRC | 17:54 | |
| *** amoralej is now known as amoralej|off | 18:01 | |
| *** jd_ has left #openstack-release | 18:02 | |
| *** electrofelix has quit IRC | 18:05 | |
| openstackgerrit | Merged openstack/releases master: glance-store 0.21.0 https://review.openstack.org/483467 | 18:09 |
| *** iyamahat has joined #openstack-release | 18:12 | |
| *** ekcs has quit IRC | 18:20 | |
| *** ekcs has joined #openstack-release | 18:22 | |
| *** bobh has quit IRC | 18:24 | |
| openstackgerrit | sean mooney proposed openstack/releases master: release new os-vif version for pike https://review.openstack.org/485304 | 18:25 |
| *** u_nuSLASHkm8 has joined #openstack-release | 18:26 | |
| *** u_nuSLASHkm8 has left #openstack-release | 18:27 | |
| openstackgerrit | sean mooney proposed openstack/releases master: release new os-vif version for pike https://review.openstack.org/485304 | 18:29 |
| *** yamahata has joined #openstack-release | 18:31 | |
| toabctl | dhellmann, re https://review.openstack.org/#/c/484887/ - Igor (the PTL) might not be here. ttx just merged previous pymod2pkg release. This is only affecting rpm-packagjng... | 18:34 |
| *** bobh has joined #openstack-release | 18:38 | |
| dhellmann | toabctl : ok, the team should really appoint a release liaison to have a backup person for these cases | 18:40 |
| toabctl | dhellmann, where should I put the name of that person? | 18:41 |
| dhellmann | toabctl : https://wiki.openstack.org/wiki/CrossProjectLiaisons | 18:41 |
| toabctl | dhellmann, ok. thanks. we'll discuss it during the next meeting.. | 18:41 |
| dhellmann | toabctl : thanks | 18:42 |
| openstackgerrit | sean mooney proposed openstack/releases master: release new os-vif version for pike https://review.openstack.org/485304 | 18:43 |
| openstackgerrit | Merged openstack/releases master: Update pymod2pkg to 0.8.1 https://review.openstack.org/484887 | 18:45 |
| *** rajathagasthya has quit IRC | 19:03 | |
| *** tinyurl_comSLASH has joined #openstack-release | 19:38 | |
| *** tinyurl_comSLASH has left #openstack-release | 19:41 | |
| *** rajathagasthya has joined #openstack-release | 19:46 | |
| *** dtroyer has quit IRC | 20:05 | |
| *** rajathagasthya has quit IRC | 20:07 | |
| dhellmann | dims, I'm going to take off for the evening. I'll talk to you tomorrow morning. have a good night! | 20:20 |
| dims | dhellmann : sounds good. | 20:20 |
| *** rajathagasthya has joined #openstack-release | 20:28 | |
| mriedem | +1 on the final os-vif release for pike https://review.openstack.org/#/c/485304/ | 20:34 |
| dims | thanks mriedem | 20:35 |
| *** dtroyer has joined #openstack-release | 20:46 | |
| *** tellesnobrega has quit IRC | 21:13 | |
| *** shardy has quit IRC | 21:13 | |
| *** tellesnobrega has joined #openstack-release | 21:21 | |
| *** pbandark has quit IRC | 21:28 | |
| *** tmorin has joined #openstack-release | 21:35 | |
| *** bobh has quit IRC | 21:36 | |
| *** tmorin has left #openstack-release | 21:37 | |
| *** sdague has quit IRC | 22:33 | |
| *** hongbin has quit IRC | 23:30 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!