| *** jamesmcarthur has joined #openstack-infra | 00:08 | |
| *** tosky has quit IRC | 00:15 | |
| *** rlandy has quit IRC | 00:21 | |
| *** hamalq has quit IRC | 01:15 | |
| *** jamesmcarthur has quit IRC | 01:23 | |
| *** jamesmcarthur has joined #openstack-infra | 01:35 | |
| *** dychen has joined #openstack-infra | 01:47 | |
| *** dychen has quit IRC | 01:51 | |
| *** dwilde has quit IRC | 02:04 | |
| *** dwilde has joined #openstack-infra | 02:07 | |
| *** dwilde has joined #openstack-infra | 02:08 | |
| *** jamesmcarthur has quit IRC | 02:24 | |
| *** jamesmcarthur has joined #openstack-infra | 02:37 | |
| *** dwilde has quit IRC | 02:44 | |
| *** rcernin_ has quit IRC | 03:14 | |
| *** armax has quit IRC | 03:17 | |
| *** ricolin has quit IRC | 03:19 | |
| *** psachin has joined #openstack-infra | 03:31 | |
| *** d34dh0r53 has quit IRC | 03:32 | |
| *** d34dh0r53 has joined #openstack-infra | 03:33 | |
| *** d34dh0r53 has quit IRC | 03:35 | |
| *** d34dh0r53 has joined #openstack-infra | 03:35 | |
| *** armax has joined #openstack-infra | 03:36 | |
| *** jamesmcarthur has quit IRC | 03:41 | |
| *** jamesmcarthur has joined #openstack-infra | 03:43 | |
| *** zzzeek has quit IRC | 04:06 | |
| *** zzzeek has joined #openstack-infra | 04:07 | |
| *** ricolin_ has joined #openstack-infra | 04:17 | |
| *** d34dh0r53 has quit IRC | 04:19 | |
| *** d34dh0r53 has joined #openstack-infra | 04:20 | |
| *** d34dh0r53 has quit IRC | 04:22 | |
| *** d34dh0r53 has joined #openstack-infra | 04:22 | |
| *** rcernin_ has joined #openstack-infra | 04:25 | |
| *** rcernin_ has quit IRC | 04:25 | |
| *** rcernin has joined #openstack-infra | 04:25 | |
| *** d34dh0r53 has quit IRC | 04:59 | |
| *** d34dh0r53 has joined #openstack-infra | 04:59 | |
| *** jamesmcarthur has quit IRC | 04:59 | |
| *** jamesmcarthur has joined #openstack-infra | 05:00 | |
| *** d34dh0r53 has quit IRC | 05:01 | |
| *** d34dh0r53 has joined #openstack-infra | 05:01 | |
| *** d34dh0r53 has quit IRC | 05:03 | |
| *** d34dh0r53 has joined #openstack-infra | 05:03 | |
| *** d34dh0r53 has quit IRC | 05:03 | |
| *** d34dh0r53 has joined #openstack-infra | 05:04 | |
| *** jamesmcarthur has quit IRC | 05:05 | |
| *** dchen has quit IRC | 05:20 | |
| *** dchen has joined #openstack-infra | 05:22 | |
| *** ykarel has joined #openstack-infra | 05:23 | |
| *** ykarel_ has joined #openstack-infra | 05:26 | |
| *** ykarel has quit IRC | 05:29 | |
| *** vishalmanchanda has joined #openstack-infra | 05:35 | |
| *** Vivek has joined #openstack-infra | 05:40 | |
| *** jamesmcarthur has joined #openstack-infra | 06:00 | |
| *** ykarel_ has quit IRC | 06:01 | |
| *** jamesmcarthur has quit IRC | 06:05 | |
| *** gyee has quit IRC | 06:17 | |
| *** ykarel has joined #openstack-infra | 06:20 | |
| *** rcernin has quit IRC | 06:26 | |
| *** rcernin has joined #openstack-infra | 06:26 | |
| *** armax has quit IRC | 06:50 | |
| *** dchen has quit IRC | 06:53 | |
| *** dchen has joined #openstack-infra | 06:53 | |
| *** zzzeek has quit IRC | 07:02 | |
| *** zzzeek has joined #openstack-infra | 07:07 | |
| *** rcernin has quit IRC | 07:07 | |
| *** jamesmcarthur has joined #openstack-infra | 07:09 | |
| *** jcapitao has joined #openstack-infra | 07:09 | |
| *** ykarel_ has joined #openstack-infra | 07:11 | |
| *** ykarel has quit IRC | 07:12 | |
| *** jamesmcarthur has quit IRC | 07:15 | |
| *** Vivek has quit IRC | 07:15 | |
| *** Vivek has joined #openstack-infra | 07:24 | |
| *** nightmare_unreal has joined #openstack-infra | 07:27 | |
| *** jamesmcarthur has joined #openstack-infra | 07:30 | |
| *** dave-mccowan has quit IRC | 07:30 | |
| *** jamesmcarthur has quit IRC | 07:34 | |
| *** hashar has joined #openstack-infra | 07:37 | |
| *** eolivare has joined #openstack-infra | 07:38 | |
| *** jpena|off is now known as jpena | 07:46 | |
| *** Vivek has quit IRC | 07:48 | |
| *** ralonsoh has joined #openstack-infra | 07:48 | |
| *** ykarel_ has quit IRC | 07:55 | |
| *** ykarel_ has joined #openstack-infra | 07:56 | |
| *** ykarel__ has joined #openstack-infra | 07:58 | |
| *** ykarel__ is now known as ykarel | 08:01 | |
| *** ykarel_ has quit IRC | 08:01 | |
| *** slaweq has joined #openstack-infra | 08:03 | |
| *** rpittau|afk is now known as rpittau | 08:08 | |
| *** gfidente|afk is now known as gfidente | 08:13 | |
| *** Vivek has joined #openstack-infra | 08:13 | |
| *** andrewbonney has joined #openstack-infra | 08:17 | |
| *** tosky has joined #openstack-infra | 08:22 | |
| *** sboyron has joined #openstack-infra | 08:33 | |
| *** jamesmcarthur has joined #openstack-infra | 08:51 | |
| *** xek has joined #openstack-infra | 08:53 | |
| *** ysandeep is now known as ysandeep|lunch | 08:55 | |
| *** jamesmcarthur has quit IRC | 08:55 | |
| *** lucasagomes has joined #openstack-infra | 09:04 | |
| *** piotrowskim has joined #openstack-infra | 09:04 | |
| *** Vivek has quit IRC | 09:09 | |
| *** Vivek has joined #openstack-infra | 09:11 | |
| *** derekh has joined #openstack-infra | 09:30 | |
| *** mgoddard has joined #openstack-infra | 09:32 | |
| *** ricolin_ has quit IRC | 09:39 | |
| *** dtantsur|afk is now known as dtantsur | 09:42 | |
| *** Vivek has quit IRC | 10:00 | |
| *** dklyle has quit IRC | 10:01 | |
| *** ociuhandu has joined #openstack-infra | 10:14 | |
| *** jamesmcarthur has joined #openstack-infra | 10:15 | |
| *** Vivek has joined #openstack-infra | 10:18 | |
| *** jamesmcarthur has quit IRC | 10:19 | |
| *** ociuhandu has quit IRC | 10:24 | |
| *** ociuhandu has joined #openstack-infra | 10:30 | |
| *** ociuhandu has quit IRC | 10:32 | |
| *** ociuhandu has joined #openstack-infra | 10:32 | |
| *** ociuhandu has quit IRC | 10:42 | |
| *** ysandeep|lunch is now known as ysandeep | 10:43 | |
| *** rpittau has quit IRC | 11:06 | |
| *** ociuhandu has joined #openstack-infra | 11:07 | |
| *** rpittau has joined #openstack-infra | 11:07 | |
| *** jcapitao is now known as jcapitao_lunch | 11:22 | |
| *** jamesmcarthur has joined #openstack-infra | 11:44 | |
| *** ociuhandu has quit IRC | 11:45 | |
| *** Vivek has quit IRC | 11:49 | |
| *** jamesmcarthur has quit IRC | 11:49 | |
| *** ricolin_ has joined #openstack-infra | 12:03 | |
| *** ociuhandu has joined #openstack-infra | 12:15 | |
| *** ociuhandu has quit IRC | 12:19 | |
| *** ociuhandu has joined #openstack-infra | 12:19 | |
| *** ociuhandu has quit IRC | 12:20 | |
| *** ociuhandu has joined #openstack-infra | 12:20 | |
| *** ociuhandu has quit IRC | 12:23 | |
| *** ociuhandu has joined #openstack-infra | 12:24 | |
| *** jpena is now known as jpena|lunch | 12:25 | |
| *** fresta has quit IRC | 12:30 | |
| *** fresta has joined #openstack-infra | 12:30 | |
| *** ociuhandu has quit IRC | 12:31 | |
| *** jcapitao_lunch is now known as jcapitao | 12:41 | |
| *** ociuhandu has joined #openstack-infra | 12:45 | |
| *** ociuhandu has quit IRC | 12:46 | |
| *** ociuhandu has joined #openstack-infra | 12:47 | |
| *** ricolin_ has quit IRC | 12:47 | |
| *** rlandy has joined #openstack-infra | 12:50 | |
| *** ociuhandu has quit IRC | 13:07 | |
| *** jamesmcarthur has joined #openstack-infra | 13:08 | |
| *** jamesmcarthur has quit IRC | 13:12 | |
| *** jamesmcarthur has joined #openstack-infra | 13:14 | |
| *** jamesmcarthur has quit IRC | 13:19 | |
| *** jpena|lunch is now known as jpena | 13:21 | |
| *** ociuhandu has joined #openstack-infra | 13:37 | |
| *** rfolco has joined #openstack-infra | 13:41 | |
| *** ociuhandu has quit IRC | 13:50 | |
| *** ricolin_ has joined #openstack-infra | 13:58 | |
| *** ricolin_ has quit IRC | 13:58 | |
| *** ricolin has joined #openstack-infra | 13:59 | |
| *** ociuhandu has joined #openstack-infra | 14:21 | |
| *** jamesmcarthur has joined #openstack-infra | 14:38 | |
| *** armax has joined #openstack-infra | 15:11 | |
| *** lbragstad has quit IRC | 15:29 | |
| *** lbragstad has joined #openstack-infra | 15:34 | |
| *** lbragstad has quit IRC | 15:35 | |
| *** lbragstad has joined #openstack-infra | 15:35 | |
| *** dklyle has joined #openstack-infra | 15:47 | |
| *** d34dh0r53 has quit IRC | 15:51 | |
| *** d34dh0r53 has joined #openstack-infra | 15:53 | |
| *** psachin has quit IRC | 16:13 | |
| *** d34dh0r53 has quit IRC | 16:19 | |
| *** d34dh0r53 has joined #openstack-infra | 16:19 | |
| *** jcapitao has quit IRC | 16:39 | |
| *** sshnaidm|ruck is now known as sshnaidm|afk | 16:47 | |
| *** lucasagomes has quit IRC | 16:58 | |
| *** jpena is now known as jpena|off | 17:01 | |
| *** ysandeep is now known as ysandeep|away | 17:01 | |
| *** slaweq has quit IRC | 17:10 | |
| *** artom has quit IRC | 17:10 | |
| *** d34dh0r53 has quit IRC | 17:10 | |
| *** tbachman has quit IRC | 17:25 | |
| *** d34dh0r53 has joined #openstack-infra | 17:27 | |
| *** tbachman has joined #openstack-infra | 17:30 | |
| *** rpittau is now known as rpittau|afk | 17:32 | |
| *** piotrowskim has quit IRC | 17:33 | |
| *** eharney has quit IRC | 17:34 | |
| *** ociuhandu has quit IRC | 17:41 | |
| *** sboyron has quit IRC | 17:46 | |
| *** eharney has joined #openstack-infra | 17:46 | |
| *** eolivare has quit IRC | 17:47 | |
| *** sboyron has joined #openstack-infra | 17:48 | |
| *** dtantsur is now known as dtantsur|afk | 17:51 | |
| *** hamalq has joined #openstack-infra | 17:58 | |
| *** ykarel has quit IRC | 18:00 | |
| *** ralonsoh has quit IRC | 18:02 | |
| *** ociuhandu has joined #openstack-infra | 18:06 | |
| *** ociuhandu has quit IRC | 18:06 | |
| *** eharney has quit IRC | 18:09 | |
| *** d34dh0r53 has quit IRC | 18:13 | |
| *** gfidente is now known as gfidente|afk | 18:19 | |
| *** eharney has joined #openstack-infra | 18:21 | |
| *** openstackgerrit has joined #openstack-infra | 18:29 | |
| openstackgerrit | Corey Bryant proposed openstack/project-config master: Create microstack-specs project https://review.opendev.org/c/openstack/project-config/+/770460 | 18:29 |
|---|---|---|
| *** d34dh0r53 has joined #openstack-infra | 18:31 | |
| *** jamesmcarthur has quit IRC | 18:32 | |
| *** jamesmcarthur has joined #openstack-infra | 18:34 | |
| *** jamesmcarthur has quit IRC | 18:39 | |
| *** ianychoi_ has quit IRC | 18:46 | |
| *** nightmare_unreal has quit IRC | 18:46 | |
| openstackgerrit | Merged openstack/project-config master: Remove openstack/monasca-analytics https://review.opendev.org/c/openstack/project-config/+/751993 | 18:55 |
| *** ianw_pto is now known as ianw | 18:58 | |
| *** gyee has joined #openstack-infra | 19:02 | |
| *** andrewbonney has quit IRC | 19:07 | |
| openstackgerrit | Merged openstack/project-config master: Use StoryBoard for sandbox repos https://review.opendev.org/c/openstack/project-config/+/759450 | 19:13 |
| *** jamesmcarthur has joined #openstack-infra | 19:17 | |
| *** jamesmcarthur has quit IRC | 19:22 | |
| *** hashar has quit IRC | 19:30 | |
| *** openstackgerrit has quit IRC | 19:40 | |
| *** slaweq has joined #openstack-infra | 19:44 | |
| *** slaweq has quit IRC | 20:03 | |
| *** openstackgerrit has joined #openstack-infra | 20:10 | |
| openstackgerrit | Merged openstack/project-config master: Use internal mirror for RAX IAD/DFW https://review.opendev.org/c/openstack/project-config/+/760495 | 20:10 |
| *** arxcruz has quit IRC | 20:14 | |
| openstackgerrit | Sorin Sbârnea proposed openstack/project-config master: Update git-review test matrix (drop py27) https://review.opendev.org/c/openstack/project-config/+/763808 | 20:16 |
| *** jamesmcarthur has joined #openstack-infra | 20:36 | |
| *** jamesmcarthur has quit IRC | 20:40 | |
| *** jamesmcarthur has joined #openstack-infra | 20:40 | |
| *** vishalmanchanda has quit IRC | 20:49 | |
| *** derekh has quit IRC | 20:56 | |
| *** derekh has joined #openstack-infra | 20:56 | |
| *** eharney has quit IRC | 21:00 | |
| *** ociuhandu has joined #openstack-infra | 21:04 | |
| *** derekh has quit IRC | 21:05 | |
| *** ociuhandu has quit IRC | 21:07 | |
| *** eharney has joined #openstack-infra | 21:13 | |
| *** rfolco has quit IRC | 21:25 | |
| *** hashar has joined #openstack-infra | 21:31 | |
| *** lbragstad has quit IRC | 21:35 | |
| *** lbragstad has joined #openstack-infra | 21:38 | |
| *** eharney has quit IRC | 21:39 | |
| *** eharney has joined #openstack-infra | 21:42 | |
| *** derekh has joined #openstack-infra | 21:50 | |
| *** derekh has quit IRC | 21:50 | |
| lbragstad | o/ has anyone seen issues with l-c job timeouts on stable branches? | 22:21 |
| clarkb | lbragstad: is the pip dep resolution taking forever beacues it has a hard time in that situation? | 22:22 |
| clarkb | fungi has been following the l-c job situation much more closely than I have though | 22:22 |
| lbragstad | well - it doesn't look like it's even able to create the environment? | 22:23 |
| lbragstad | https://zuul.opendev.org/t/openstack/build/3fc8afd5e449461aa01187a031fd17a0/log/job-output.txt#635 | 22:23 |
| lbragstad | we're seeing this happen on stable/train and stable/ussuri patches | 22:23 |
| fungi | the new dep solver in pip can take a very, very long time if presented with effectively unsolveable conflicts in complex sets of transitive dependencies | 22:24 |
| clarkb | ya thats the install deps step which is when the dep solver would be churning | 22:25 |
| clarkb | so I expect that is the problem | 22:25 |
| fungi | lbragstad: https://zuul.opendev.org/t/openstack/build/3fc8afd5e449461aa01187a031fd17a0/log/tox/lower-constraints-1.log#217 | 22:25 |
| fungi | "INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking | 22:26 |
| lbragstad | oh - dang.. that looks like the smoking gun | 22:28 |
| fungi | i just happened to already know where to look | 22:29 |
| * lbragstad parse the fixes in https://review.opendev.org/q/topic:%22fix-relmgt-pip-doc%22+(status:open%20OR%20status:merged) | 22:32 | |
| lbragstad | it looks like we have a doc/requirements.txt, test-requirements, and requirements.txt on keystone's stable/stein branch | 22:33 |
| lbragstad | is there something else we should be doing that we aren't? | 22:34 |
| clarkb | I think this is related to the fundamental issues with lower constraints jobs (whihc is why there is a push to remove them) | 22:35 |
| * lbragstad nods | 22:35 | |
| clarkb | basically they never quite worked how we expected and the dep solver in pip exposees that now | 22:35 |
| lbragstad | we already did that on master | 22:35 |
| clarkb | pip wants to find the newest versions that meet the requirements but lower constraints is doing the exact opposite so they fight each other | 22:35 |
| fungi | they're going to be even harder to fix on stable branches than they would have been to fix in master | 22:35 |
| lbragstad | =/ | 22:36 |
| lbragstad | i know cmurphy brought up a point earlier when we were talking about it how removing that testing can be problematic for packagers, but it also sounds like we're out of options? | 22:36 |
| fungi | the lower-constraints.txt files used by most projects were: 1. incomplete (missing much of the transitive set which is where the real challenges lie for dep solving), and 2. not necessarily representative of a consistent subset to begin with | 22:37 |
| lbragstad | ack | 22:37 |
| clarkb | lbragstad: I think the alternative is to properly curate the list so that the dep solver doesn't have any work to do | 22:37 |
| fungi | lbragstad: removing testing can be problematic if you're removing working tests. if you're removing tests which weren't testing what you thought they were testing, you've lost nothing | 22:37 |
| clarkb | constraints essentially bypass dep solving except if the list is incomplete then the dep solver has to solve for that missing space | 22:37 |
| lbragstad | clarkb in other words, having a super-opinionated (and arguable fragile) lower-constraints.txt file? | 22:38 |
| lbragstad | arguably* | 22:38 |
| clarkb | ya and then spending the time to curate it over time | 22:38 |
| fungi | also the constraints can be patently incorrect, representing conflicts which are unsolveable. old pip didn't care that some theoretical version conflicts were present and punted (it really just wasn't smart enough to know) | 22:39 |
| lbragstad | which is kind of what we do now, no? but we'd just be doing it more? | 22:39 |
| lbragstad | ahh | 22:39 |
| clarkb | sort of, aiui the lower constraints were just a guess before | 22:39 |
| clarkb | whereas upper constraints are actually generated from passing job runs and what pip's solver produces | 22:39 |
| lbragstad | but now it's spoken word because we have a smarter dependency resolver? | 22:39 |
| clarkb | ya | 22:39 |
| lbragstad | ok - interesting | 22:40 |
| clarkb | upper constraints don't have this problem beacuse they are the result of dep solving | 22:40 |
| fungi | right. now pip has logic to attempt to find the newest version of every package which will satisfy every other package's requirements (previously it just sort of went entry by entry and overwrote in the case of a conflict) | 22:40 |
| clarkb | lower constraints were human guesses and now pip isn't able to process it | 22:40 |
| fungi | and yeah, upper-constraints.txt is generated by saying "hey pip install all the packages mentioned in this requirements list and their dependencies" and then freezing the result. there is no equivalent tool to find the oldest versions described by the transitive set rather than the highest | 22:41 |
| fungi | so we have no way to generate coherent constraints lists from lower bounds, only from upper bounds (by taking advantage of the fact that's what pip knows how to calculate already) | 22:42 |
| fungi | i've suggested many times that a feature could be implemented to invert the selection algorithm in pip, as probably the most straightforward means of generating a coherent lower-constraints.txt | 22:43 |
| lbragstad | ok - and the new functionality/dependency resolver obviously doesn't include that functionality today? | 22:44 |
| fungi | but that's still going to run into the problem that projects are even less thorough about listing their lower bounds than their upper bounds, and assumes that your transitive dependencies are mostly performing some similar testing as well | 22:44 |
| fungi | lbragstad: it does not, no | 22:45 |
| fungi | basically it wants to install the newest version of everything which satisfies the requirements of each package, and then the lower-constraints job is overriding *some* of those forcing a choice of older versions | 22:46 |
| lbragstad | and the new pip functionality doesn't know how to deal with those conflicts, hence the chrun? | 22:46 |
| lbragstad | churn* | 22:46 |
| fungi | if those older versions plus the newer versions of whatever they depend on form a coherent set, then the solver will be happy and go forward with the job | 22:46 |
| fungi | but the only way to really work that out is through painstaking manual trial and error | 22:47 |
| fungi | or to write your own dep solver which solves for lowest version instead of highest | 22:47 |
| lbragstad | ok - i think this is starting to make sense | 22:52 |
| *** ociuhandu has joined #openstack-infra | 22:53 | |
| *** ociuhandu has quit IRC | 22:58 | |
| fungi | yeah, there's two forces at play: solving collective dependencies from ranges, and constraining those selection choices. the constraints file acts as a sort of filter to say "you must select these versions of the listed packages" so the result can be one of: coherent, conflicting, incalculable | 23:01 |
| *** rcernin has joined #openstack-infra | 23:02 | |
| fungi | as i warned folks repeatedly when the original lower-constraints implementation was being formulated, it didn't actually test the versions of things people assumed it would. now that pip is aware enough to actually check coherence of the dependent set, that's become more obvious to folks | 23:04 |
| fungi | previously there was a fair bit of hand-waving with regard to "well it's probably testing some old versions of some dependencies even if it's not exactly right, and maybe that's good enough" but it relied on pip not actually caring that it hadn't installed what it was asked to install | 23:05 |
| *** arxcruz has joined #openstack-infra | 23:09 | |
| fungi | in fact, it was our community which invented the constraints feature in pip to begin with, as a stop-gap to allow us to externally perform dep solving because pip lacked a proper solver (a prototype for which was also proposed initially and got a lukewarm reception, so constraints was actually a concession) | 23:09 |
| *** hashar has quit IRC | 23:15 | |
| *** jamesmcarthur has quit IRC | 23:16 | |
| *** rlandy has quit IRC | 23:19 | |
| *** jamesmcarthur has joined #openstack-infra | 23:31 | |
| *** jamesmcarthur has quit IRC | 23:35 | |
| *** jamesmcarthur has joined #openstack-infra | 23:35 | |
| *** sboyron has quit IRC | 23:45 | |
| *** sboyron has joined #openstack-infra | 23:46 | |
| *** sboyron has quit IRC | 23:54 | |
| *** sboyron has joined #openstack-infra | 23:56 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!