Wednesday, 2019-06-19

*** diablo_rojo has joined #openstack-tc00:21
*** tosky has quit IRC00:36
*** diablo_rojo has quit IRC00:51
mnaser\o/01:00
mnaserhi all01:01
*** mriedem_away has quit IRC01:01
gmanno/01:02
fungithat time once more01:04
fungiit's also first contact sig meeting time in #openstack-upstream-institute01:05
mnaseri had a fun kinda-technical-but-maybe-not-so-tc topic01:05
mnaserhttp://lists.openstack.org/pipermail/openstack-discuss/2019-May/005956.html was a discussion that was had a while ago regarding upper-constraints01:06
*** diablo_rojo has joined #openstack-tc01:06
mnaserosa (as a deployment tool) relies on that, we have users that are having problems with buggy libraries in our upper constraints (amqp for example for ssl)01:06
mnaserin this case, we actually have a scenario where the upper constraints are an actual test of *non-functional* (if using ssl) dependencies01:07
mnaserfrom an osa perspective, it's nice that we can just slurp a file that we can maintain upstream (and that perhaps others can refer to to get a working set of constraints)01:07
mnaseri dont think it would be productive if we maintained our own..01:07
*** lbragstad has quit IRC01:08
fungimy take is that users should not be relying on the security of pip installing dependencies of our stable branches. we're not a linux distro, we don't get advance warning of pending security advisories, our ability to update them at all for stable branches is iffy at best, and evenif we can the delay in doing so means deployments sit vulnerable for faaar too long01:09
mnaserit does seem like it would be a productive thing for this type of thing to be shared across deployment tools / packagers..01:09
fungithey should get dependencies from distro packages01:09
mnaseryou cant install those in a virtual environment :(01:10
fungisounds like virtualenvs are a security risk then01:10
fungi(but also you can expose them into virtualenvs if system site packages are enabled when the env is built)01:11
mnaseryeah, i figured as much.. but i don't know if that would be easy to accomplish01:11
fungii have no idea01:12
mnaseri.e. install all these python{2,3}-{foo} things and then create a virtualenv and install these packages from system site packges, i see wher you're coming from though01:12
mnaserit does mean if something breaks we are at the mercy of distros01:12
fungithe upper-constraints.txt is an artifact of how we test newer versions of software than our target distros package. we freeze them when we branch so that we reduce unanticipated test regressions without having to switch to completely different test models for releases once they branch01:14
fungii don't think we should be encouraging production deployments of stable branches using packages versions listed in our frozen upper-constraints.txt files since we're never going to be able to make that anywhere close to as secure as a security-supported linux distro01:15
mnaserbut do linux distros ship anything newer than upper constraints?01:16
fungithey backport security fixes to those things yes, and generally have those backported fixes applied to coincide with the corresponding security advisory publication dates01:16
mnaserwhat about things that aren't security issues but actual bugs where the library is simply broken01:17
mnaseri.e. ssl + amqp is just plain broken01:17
fungibroken such that our tests stop working?01:18
mnasercontext: https://bugs.launchpad.net/oslo.messaging/+bug/1800957/01:19
openstackLaunchpad bug 1800957 in oslo.messaging (Ubuntu) "Upgrading to pike version causes rabbit timeouts with ssl" [Undecided,New]01:19
fungiupdating entries in upper-constraints.txt to better stabilize the tests it's designed to stabilize makes some sense, yes01:19
mnaseri assume there was just not a test that caught that01:20
fungiif there's no test then the constraints not providing stability to that nonexistent test seems irrelevant01:20
mnaseri guess the issue here was that there was no test that checked for this (but we promise ssl support in oslo.messaging), a bug was caught, a fix was done, but.. we still don't test what someone might actually be running in prod01:21
mnaseri guess i view upper-constraints differently01:21
fungiagain, the upper-constraints.txt file is there to stabilize our testing. reusing it for other purposes and then asking to make it less useful for the purpose it was designed for seems like a bad idea01:21
mnaserit just seems to be like if OSA starts maintaing its own set of constraints, we're just going to start duplicating efforts across projets01:23
mnaseri think kolla even builds containers using upper-constraints too.. i htink01:23
fungiit's quite good at stabilizing tests, which is why we chose that solution. it's really not that good for managing dependencies of a production deployment (for a variety of reasons, security vulnerabilities chief amongst those). making it less effective at stabilizing tests so that people can use it in other dangerous ways isn't something i can get behind01:24
mnaseri'm just trying to visualize the ideal solution for this which doesn't create a lot of duplicate work across deployment tooling01:25
fungii honestly don't see making your own distro out of a constant churn of packages from pypi as anything other an sisyphean. if you want to talk about duplicating effort, what you're actually describing is kolla and osa duplicating the same work well-funded and staffed linux distros already do, but very, very poorly01:26
fungithere is a reason they freeze package versions and then carefully try to backport critical fixes to those frozen versions rather than just constantly mixing in new releases of random upstream projects01:27
fungiand it's because the latter doesn't work for an lts model, full stop01:28
mnaserthat's fair, i totally agree with you.01:28
fungiit works well enough in a rolling release model01:28
fungiwhich is why i think it's feasible for folks who want to deploy openstack from master01:28
mnaserit's just trying to balance the usability / usefulness of users who might want to make a change or two.  and also, it's tough because distros are generally not good at running multiple versions of a library01:29
mnaseri.e. if i am doing a rolling upgrade and doing keystone first, i cant just have a 'virtualenv' or whatever that runs newer packages01:29
mnaserand openstack doesn't exactly test with those debs/rpms so it's just this inconsistency01:29
fungii get the desire, and have empathy for users who wish that could be less complicated01:30
mnaserto me, the biggest value of virtualenv deploys is having something with all of its dependencies live in a little bucket to help make upgrades painless01:30
mnaserand maybe that isn't openstack's problem to solve.. but its there..01:30
fungii think it's entirely reasonable for us to want to solve it, but the problem is way bigger than openstack yes, and we're not the only people who want to work on solving that same problem01:31
fungiso i think the solution also needs to be bigger than openstack, and a much wider-reaching effort01:32
clarkbfwiw venvs dont require you use our constraints or constraints at all01:33
clarkbseema orthogonal to me tbh01:33
mnaserif you deploy stuff without constraints, you're likely going to end up with a broken system01:33
mnasers/system/service/01:33
fungiright, i gather the desire there is the usual contradictory one: you want something stable but with updates01:33
fungimore updates means less stable. more stable means less updates01:34
mnaseri can't disagree with that01:35
fungithe task of making updates as stable as possible is increasingly complicated the closer you try to drive that margin, and is why lts server distros invest sooooo much effort in that (it's really probably their greatest selling point, and they know it)01:35
mnaseri agree that they know what they're doing, but i'm just trying to figure out a way where we can use all those lts dependencies, but with upstream openstack.01:36
mnaserto me that's the hard thing to juggle01:36
mnaserand a lot of those lts operating systems put openstack in a special spot where it gets mucher newer packages01:37
fungifor our stable branch testing we opted for as stable as possible, which means no updates. production deployments need updates to fix untested bugs (including newly discovered security vulnerabilities)01:37
mnaser(i.e. UCA, RDO)01:37
fungiyeah, we move "too fast" for stable distros01:37
mnaserso the whole lts promise is pretty much not that useful, do a yum install openstack-nova and you'll find msot things are being pulled from rdo rather than base/updates01:38
fungiwell, yes and no. i gather the "newer" packages in, say, the rocky version of rdo will be held basically unchanging for as long as red hat supports rocky01:39
fungiwhich may well be years01:40
fungiand that's attractive to a certain sort of businesses who dislike the idea of upgrading major versions of software ore often than every 3-4 years01:40
fungithe other thing we've said in the past, which deployment projects have more recently not understood, disagreed with, ignored, or something, is that our stable branches are a place for coordinating backports of critical fixes but are not intended as a comprehensive maintained distribution. the idea was that distributions would pick patches from the stable branches to integrate01:44
fungiso the stable branches as branches provide us a way to test all those backports together to make sure they're hopefully coherent and work as a whole01:45
mnaserwell that puts OSA in a weird place i guess01:45
fungibut we weren't expecting stable branch deployments from source outside of using them to test future backports01:46
mnaseri guess it kinda implies we should maintain branches for all services as well01:46
fungii do think that the fact some of our deployment projects have wanted to deploy stable branches from source (or that their users have asked them for that model) suggests that they may be more useful for other purposes than those which they were originally intended01:49
fungibut it's worth noting that's not a purpose they were designed for, and so the way in which we designed them may make them less effective at that as a result01:50
mnaseri mean i think i see a huge value in our stable branches being something you can deploy IMHO01:50
fungiit may be that we should change fundamental aspects of our stable branch maintenance and testing to better suit those use cases, but that means we also need to 1. be realistic about what we can actually promise even if we do, and 2. determine whether making them less effective at the things we originally designed them for is acceptable01:51
mnaseri think a lot of distros do a straight rebuild of stable branches (i.e. rdo afaik?)01:53
fungiit's also possible we could end up with a compromise like the linux kernel has: two kinds of stable series01:53
fungiyes, distros have generally gotten accustomed to just being able to slurp in everything in our stable branches *because* we have a fastidious approach to keeping them stable (not increasing dependencies in them, for example)01:54
fungiso if we choose to destabilize our stable branches to accommodate a source distribution model which demands updated deps, then distros may no longer be able to make the assumption they can directly consume our stable branches01:55
fungiand will have to go back to selectively importing patches from them01:56
fungiand that may be fine, but it's a trade-off we need to take into account01:57
mnaserfungi: thanks for that conversation :>01:59
* mnaser has some thinking to do about how we can make this work for everyone (and be a win-win-win)02:00
fungimy mentioning the linux kernel model was to point out that it's not necessarily wrong to have different branches catering to different stability needs. it may seem like a duplication of effort, but that can still be justifiable given the right mix of needs02:01
fungimy bigger concern is i think we still can't deliver on a promise to provide a security-supported distribution of software we don't write, at least not without a lot more volunteers to make it happen (and basically becoming a small-scale/derivative distro in our own right). that's not a decision to be taken lightly02:03
*** whoami-rajat has joined #openstack-tc02:08
fungithe starlingx community went down that path, and now they're spending massive amounts of effort on something which is not their core competency (and i think are actively working toward *not* having to maintain their own distro in the future)02:08
*** diablo_rojo has quit IRC02:49
*** ricolin has joined #openstack-tc02:55
*** e0ne has joined #openstack-tc05:15
*** e0ne has quit IRC05:20
*** dklyle_ has joined #openstack-tc05:24
*** david-lyle has quit IRC05:27
*** purplerbot has quit IRC05:27
*** purplerbot has joined #openstack-tc05:28
*** irclogbot_2 has quit IRC05:30
*** irclogbot_0 has joined #openstack-tc05:32
*** Luzi has joined #openstack-tc05:50
*** spsurya has joined #openstack-tc06:15
*** ricolin has quit IRC06:18
*** ricolin has joined #openstack-tc06:19
*** tosky has joined #openstack-tc07:15
*** iurygregory has joined #openstack-tc07:24
*** iurygregory has quit IRC07:35
*** iurygregory has joined #openstack-tc07:40
*** dtantsur|afk is now known as dtantsur07:56
*** jpich has joined #openstack-tc07:59
asettleo/08:25
*** e0ne has joined #openstack-tc08:34
openstackgerritAlexandra Settle proposed openstack/project-team-guide master: Reviewing the PTL document for accessibility  https://review.opendev.org/66569909:15
*** tosky has quit IRC09:47
asettlericolin, I've finally got some time to review https://etherpad.openstack.org/p/explain-team-formate-differentiate09:47
asettleWas there anything else I needed to review for this?09:48
*** tosky has joined #openstack-tc09:48
*** lpetrut has joined #openstack-tc09:49
ricolinasettle, not for now, I will update that etherpad today and try to targeting those feedback09:54
asettleOkay thanks :) going through and placing edits and occasional comments09:55
ricolinasettle, that will be super helpful!09:58
asettlenp :) nearly done09:58
*** lpetrut has quit IRC10:13
ricolinasettle, try to explain those unknow meaning part in https://etherpad.openstack.org/p/explain-team-formate-differentiate10:23
ricolinasettle, let me know if it's still confusing10:23
asettleOkay looking :)10:24
asettleGood good!10:24
asettleThank you :)10:24
ricolinasettle, will also add another paragraph to introduce other formats(which you may not create but it's good to know they exists like Foundation level working group)10:26
asettleCool sounds good. Shoot me a ping when that's done and I can look at that oto.10:26
*** lpetrut has joined #openstack-tc10:51
jrollfungi: mnaser: thanks for the good conversation in scrollback. I'm not sure I have anything to add right now, but it's certainly making my brain churn. :)11:09
smcginnisfungi: Since you've been able to articulate it better than anyone else I've seen, would you have a few minutes to add some comments to the top of https://opendev.org/openstack/requirements/src/branch/master/upper-constraints.txt to make it a little more clear what the expectations are for that file and the package/versions listed there?11:46
*** tdasilva has joined #openstack-tc12:05
fungismcginnis: sure, i'll see if i can distil some useful nuggets for that once i'm more thoroughly caffeinated12:10
smcginnisThanks! :)12:10
openstackgerritAdam Spiers proposed openstack/project-team-guide master: Document running tests with stestr  https://review.opendev.org/66627312:13
*** ricolin_phone has joined #openstack-tc12:39
*** ricolin_phone has quit IRC12:42
*** ricolin_phone has joined #openstack-tc12:52
*** ricolin_phone has quit IRC13:09
*** mriedem has joined #openstack-tc13:11
*** ricolin_phone has joined #openstack-tc13:14
*** ijolliffe has joined #openstack-tc13:16
*** ricolin_phone has quit IRC13:17
*** lbragstad has joined #openstack-tc13:20
*** jamesmcarthur has joined #openstack-tc14:05
*** lpetrut has quit IRC14:14
*** zaneb has quit IRC14:30
*** zaneb has joined #openstack-tc14:30
*** Luzi has quit IRC14:36
*** dklyle_ has quit IRC14:42
*** dklyle has joined #openstack-tc14:42
*** iurygregory has quit IRC14:52
*** lpetrut has joined #openstack-tc14:58
*** lpetrut has quit IRC14:59
*** lpetrut has joined #openstack-tc14:59
*** tdasilva_ has joined #openstack-tc15:06
*** tdasilva has quit IRC15:07
*** iurygregory has joined #openstack-tc15:18
*** Sundar has joined #openstack-tc15:24
*** lpetrut has quit IRC15:30
*** thiago__ has joined #openstack-tc15:39
*** tdasilva_ has quit IRC15:42
*** iurygregory_ has joined #openstack-tc15:45
*** iurygregory has quit IRC15:46
*** diablo_rojo has joined #openstack-tc16:00
*** cdent has joined #openstack-tc16:08
*** cdent has left #openstack-tc16:09
*** lpetrut has joined #openstack-tc16:10
*** spsurya has quit IRC16:18
*** thiago__ is now known as tdasilva16:19
*** tellesnobrega has joined #openstack-tc16:27
*** jpich has quit IRC16:27
tellesnobregatonyb, hey, I was the PTL for sahara for last releases but now the new PTL is jeremy freudberg, I would like to know how can I add him to the sahara-stable-maint16:29
tellesnobregaand also update the team, we have a lot of people that are not working on sahara anymore that needs to be cleaned out of that team16:29
mugsietellesnobrega: https://docs.openstack.org/project-team-guide/stable-branches.html#stable-maintenance-teams is how new people are added to the sable-maint team16:31
tellesnobregathanks mugsie16:31
mugsiePTLs are not automatically added to that group - they have to show their knowledge of the stable policy like everyone else16:32
tellesnobregamugsie, thanks16:33
tellesnobregaas a member of that team, I have two names that should be there16:33
tellesnobregajeremyfreudberg (Jeremy Freudberg) and tosky (Luigi Toscano) they both have more than proved their knowledge of review process and stable branches policies16:34
mugsieemailing the list with the [stable] tag, or pinging the team in #openstack-stable is a good place to get them added then16:35
tellesnobregaalso, thanks16:35
smcginnistellesnobrega: Do you know off hand - does Sahara assert the stable:follows-policy tag?16:36
smcginnisI suppose I could look that up.16:36
mugsiesmcginnis: it does16:36
smcginnisOK, then everything above is a good start. ;)16:37
*** tosky has quit IRC16:47
*** ricolin has quit IRC16:48
*** lpetrut has quit IRC16:49
*** diablo_rojo has quit IRC16:56
*** dtantsur is now known as dtantsur|afk16:56
*** e0ne has quit IRC17:20
*** diablo_rojo has joined #openstack-tc17:26
openstackgerritAdam Spiers proposed openstack/project-team-guide master: Document running tests with stestr  https://review.opendev.org/66627317:36
*** brett-soric has joined #openstack-tc17:41
*** tellesnobrega has left #openstack-tc17:49
*** melwitt is now known as jgwentworth17:55
*** diablo_rojo has quit IRC18:05
*** tdasilva has quit IRC18:17
*** brett-soric has left #openstack-tc18:30
*** diablo_rojo has joined #openstack-tc18:31
*** tdasilva has joined #openstack-tc18:35
*** e0ne has joined #openstack-tc18:35
*** e0ne has quit IRC18:36
*** jamesmcarthur has quit IRC18:40
*** jamesmcarthur has joined #openstack-tc18:41
*** jamesmcarthur has quit IRC18:46
*** diablo_rojo has quit IRC18:46
*** diablo_rojo has joined #openstack-tc18:57
*** jamesmcarthur has joined #openstack-tc19:04
*** e0ne has joined #openstack-tc19:05
*** tdasilva has quit IRC19:06
*** jamesmcarthur has quit IRC19:10
*** jamesmcarthur has joined #openstack-tc19:23
*** bsilverman has quit IRC19:47
*** jgwentworth is now known as melwitt19:52
*** ijolliffe has quit IRC19:53
*** e0ne has quit IRC20:11
*** jamesmcarthur has quit IRC20:13
*** jamesmcarthur has joined #openstack-tc20:15
*** jamesmcarthur has quit IRC20:23
openstackgerritAdam Spiers proposed openstack/project-team-guide master: Document running tests with stestr  https://review.opendev.org/66627320:41
*** whoami-rajat has quit IRC20:47
*** e0ne has joined #openstack-tc20:55
*** dirk has quit IRC20:55
*** e0ne has quit IRC20:59
*** e0ne has joined #openstack-tc21:00
*** e0ne has quit IRC21:39
*** jamesmcarthur has joined #openstack-tc21:40
*** jamesmcarthur has quit IRC21:51
*** mriedem has quit IRC21:59
*** diablo_rojo has quit IRC22:05
*** diablo_rojo has joined #openstack-tc22:35
*** Sundar has quit IRC22:44
openstackgerritAdam Spiers proposed openstack/project-team-guide master: Document running tests with stestr  https://review.opendev.org/66627323:36

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!