Thursday, 2021-08-12

ianwis there any way for, say, https://zuul.opendev.org/t/openstack/job/puppet-openstack-integration-5-scenario003-tempest-debian-stable to tell where the job was defined?00:25
corvusianw: check the inheritance path in the inventory.yaml file00:59
ianwyeah, this is an odd case where the job definitions were accidentally left in on a branch -> https://review.opendev.org/c/openstack/puppet-openstack-integration/+/80431001:00
*** rpittau|afk is now known as rpittau06:22
*** dpawlik6 is now known as dpawlik07:07
opendevreviewBenjamin Schanzel proposed zuul/zuul master: Add tenant name on NodeRequests for Nodepool  https://review.opendev.org/c/zuul/zuul/+/78868007:32
*** holser is now known as holser_07:32
*** jpena|off is now known as jpena07:33
opendevreviewDong Zhang proposed zuul/zuul master: Disable aliases in inventory.yaml for better readibility  https://review.opendev.org/c/zuul/zuul/+/80267407:42
*** sshnaidm|afk is now known as sshnaidm08:55
opendevreviewBenjamin Schanzel proposed zuul/nodepool master: Add Tenant-Scoped Resource Quota  https://review.opendev.org/c/zuul/nodepool/+/80076511:09
*** jcapitao is now known as jcapitao_lunch11:10
*** dviroel|ruck|out is now known as dviroel|ruck11:24
*** jpena is now known as jpena|lunch11:33
*** jcapitao_lunch is now known as jcapitao11:56
*** jpena|lunch is now known as jpena12:24
*** jpena is now known as jpena|off12:33
*** ysandeep|PTO is now known as ysandeep13:26
*** holser is now known as holser_14:11
opendevreviewJames E. Blair proposed zuul/zuul master: Add delete-state command to delete everything from ZK  https://review.opendev.org/c/zuul/zuul/+/80430414:16
*** dmellado_ is now known as dmellado14:24
*** jpena|off is now known as jpena14:43
*** rpittau is now known as rpittau|afk15:31
*** jpena is now known as jpena|off15:34
pabelanger[m]I'm thinking about creating a new 'manual enqueue' pipeline in zuul. Where the job only triggers via zuul CLI enqueue command or other none git event.  It would run much like periodic timer pipelines do today.  Curious if anyone else is using a pipeline like that?15:40
fungiin opendev we have "manually triggered" pipelines which only enqueue changes where someone leaves a particular review comment, but no we don't have anything which only gets items enqueued via rpc/api15:42
fungiwe're very much about avoiding "manual" anything though, so unlikely to have a use case for it15:43
pabelanger[m]my first thought is create a new timer trigger pipeline, but only triggers in the year 2050 :)15:43
pabelanger[m]then we can still use enqueue command for it15:43
fungiyeah, that's what i expected would be the simplest solution, but seems like longer term a "null" trigger might be easy enough to add15:43
pabelanger[m]yah, in this case we don't have a direct connection to the downstream system to 'trigger' properly via git15:44
pabelanger[m]because of firewalls15:44
pabelanger[m]but would like a way to periodic run something, on our nodepool infrastructure15:44
pabelanger[m]otherwise, we have to duplicate it all15:44
pabelanger[m]long term, we'll move everything back behind the firewall and setup proper integration, but looking to do a minimal POC first to show the jobs working15:45
Clark[m]If your zuul can clone the repo wouldnt the normal git driver work then you can trigger on ref updates15:54
tristanCpabelanger[m]: you mean share a single nodepool with two zuul?15:54
MaheshBarai[m]hello... is this room active for zuul technical discussion16:00
fungiMaheshBarai[m]: yes, absolutely16:01
MaheshBarai[m]Thanks fungi 16:02
pabelanger[m]tristanC: more nodepool to allow for triggers other then zuul16:07
pabelanger[m]basically, we have a complicated image requirements for network appliances, I really don't want to duplicate that16:08
*** marios is now known as marios|out16:08
pabelanger[m]so for now, we can write a zuul jobs that only runs an something outside of git enqueing the change into the pipeline16:08
pabelanger[m]for some manual testing16:08
clarkbpabelanger[m]: but how does the content of the change end up in the node in your manual enqueue scneario?16:22
clarkbis that also completely manual and you just set an ssh key or similar credentials and let $random thing take over?16:26
pabelanger[m]clarkb: it is all self contained in a container16:54
pabelanger[m]that we pull from a registry16:54
clarkbpabelanger[m]: I would probably do a job on a repo that does that then. Then you can modify the file that looks for the container image or whatever16:55
clarkbpabelanger[m]: opendev does similar with the docker images we consume from upstreams16:55
clarkbwhere we can run and test them with an update to system-config but we don't directly trigger on updates to the image or code16:55
pabelanger[m]right, that is a way too.  We mostly don't way github users creating a PR to trigger it right now so it would be a different source16:56
pabelanger[m]why I was thinking of manually enqueue to start16:56
clarkbon openstack-discuss there is an email about sqlalchemy's pending 2.0 release which is expected to cause problems17:26
clarkb"virtually anyone that uses sqlalchemy will need to make changes" to paraphrase17:27
fungiyeah, i wondered what the impact might be for zuul's testing17:28
clarkband for zuul itself17:28
fungido we still test with sqla at all, or only with service-based rdbms17:28
clarkbsqlalchemy is used in testing17:29
clarkbThere is apparently a way to turn on deprecation warnings to point out things that need updating. Also we don't use slqalchmeny migrate so are fine on that end17:29
clarkbhttps://docs.sqlalchemy.org/en/14/changelog/migration_20.html17:31
corvusfungi: you may be thinking of sqlite17:31
fungier, yes sqlite, sorry17:32
fungisqlalchemy is the topic17:32
fungiand zuul definitely uses that17:32
fungifor some reason when i read stephenfin's e-mail i had sqlite in my head the entire time17:32
clarkbI can get a change up that tries to run on the warnings17:33
pabelanger[m]do we have any examples of encrypting binary data (eg: zipfile) as a zuul secret?17:33
pabelanger[m]someone asked me downstream about it, and figured it would work17:33
fungipabelanger[m]: it's designed to work, yes. the data is treated as raw binary17:33
pabelanger[m]cool17:34
corvuspabelanger: afs tokens are an example17:34
pabelanger[m]great17:34
fungiat one point i used it to encrypt gnupg keyrings as well17:34
pabelanger[m]I forgot about afs tokens17:34
clarkbhrm does stestr not expose python flags? that makes this sort of testing much more difficult when you need to set env vars and python -w always::DeprecationWarning17:37
clarkbfungi: ^ I seem to recall you do python testing with warnings as errors. Do you use stestr with that?17:38
fungii set testenv.setenv in tox.ini17:41
fungiadding PYTHONWARNINGS = whatever17:41
fungithere is a fairly extensive language for the PYTHONWARNINGS envvar to be able to express what you do or don't want treated as errors17:42
clarkbya I think that is what always::DeprecationWarning does17:42
clarkbor it exposes that warning always but not as an error17:42
clarkbya ok I can change that toe error::DeprecationWarning then it will error17:43
*** holser_ is now known as holser17:44
fungiyes, error::DeprecationWarning will cause all deprecation warnings to be raised as exceptions17:45
fungifor one project i do something like this:17:46
fungiPYTHONWARNINGS = error, ignore::FutureWarning:Cython.Compiler.Main, ignore::DeprecationWarning:setuptools.config, ignore::DeprecationWarning:distutils.command.install, ignore::ImportWarning:importlib._bootstrap, ignore::DeprecationWarning:pip._vendor.urllib3.connection, ignore::DeprecationWarning:pip._vendor.urllib3.util.ssl_, ignore:The "license_file" option is deprecated. Use "license_files"17:46
fungiinstead.:DeprecationWarning:wheel.bdist_wheel, ignore:The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives:DeprecationWarning, ignore:Creating a LegacyVersion has been deprecated and will be removed in the next major release:DeprecationWarning17:46
fungisorry, should have used a paste service17:46
fungithat's all one line, btw17:46
clarkbthanks17:46
clarkbI'm making all DepreactionWarnings an error to start and will see what breaks17:46
clarkbthen once I've got a config that makes sense I can psuh that up17:47
fungii note that basically all the ignores i have in there are for DeprecationWarning (one lonesome ImportWarning)17:48
clarkboh I guess the issue is that this is likely to catching things outside of sqlalchemy17:49
fungithe newer the python you run it on though, the more you're going to find fundamental bits of the packaging ecosystem, or your dependencies, are using deprecations17:49
clarkbso I need to scope it to sqlalchemy better17:49
clarkbyup immediately hit a deprecation warning in pip trying to install deps. Need to scope this better17:50
fungiand like i said, the python interpreter version you're running on will influence what comes up too17:50
funginot just because of things deprecated in stdlib, but also because of vendored libs in things like ensurepip17:51
fungior in venv's seed modules17:51
clarkbI'm trying error::DeprecationWarning:sqlalchemy17:51
fungiyeah, that may help17:52
fungithough it might need sqlalchemy.* there17:52
fungiif you don't hit anything without the .* try adding17:52
clarkbwill do17:53
clarkbI also worry that the env vars aren't making it to the child processes of stestr. looking /proc for that info now17:55
*** sshnaidm is now known as sshnaidm|afk17:55
clarkbnevermind seems they get passed through properly17:56
fungiit would be odd if they weren't. i mean, they make it through to cython compiler subshells in my projects17:57
fungi(that's for pyyaml, btw)17:57
clarkbya I was worried that stestr would be filtering them17:58
clarkbbut it seems to use multiprocessing and that handles it implicitly17:59
clarkbif even seems to rewrite the PYTHONWARNINGS var to -w on the python args list17:59
fungialso some of the complexity of my example above is from testing with python 3.1017:59
clarkband so far everything is succeeding which makes me suspect that .* is necessary18:00
fungiunsurprisingly, a lot of released tooling hasn't removed stuff which 3.10 deprecates, and also nobody publishes wheels for 3.10 so i get to see any which arise from exercising build from sdist of my deps18:01
fungibut yeah, the module matching is very explicit from what i've seen (also a bit fiddly)18:02
fungipart of why i went with the approach of erroring on everything and then finding ways to filter out the things i can't fix18:03
*** holser is now known as holser_18:04
fungiotherwise i'm afraid it will silently skip things i want it to complain about18:04
clarkbI expect that error on any sqlalchemy deprecation warnings is sufficient based on the doc I linked above18:05
fungiyeah, maybe try introducing something you know should be deprecated and see if it gets caught?18:05
fungii mean, it's possible zuul will work as-is with sqlalchemy v2, but it's more likely that there's something subtly wrong in checking18:06
clarkbgood point18:06
clarkberror:::mymodule[.*] that actually seems to be the format?18:07
clarkbI need the brackets I guess18:07
fungioh, i missed the brackets, maybe that's been part of my problem18:08
fungii thought the [] were indicating "optional"18:08
fungilike in a manpage18:08
clarkbfungi: https://docs.python.org/3/library/warnings.html#describing-warning-filters18:08
clarkbit isn't super clear18:09
fungiyeah18:09
clarkbbtu it does say in module and any submodules18:09
clarkbbut even that doesn't seem to be working. Or we're fine and have no warnings18:15
clarkbya switching to always and forcing a failure in the mysql test_migration test gives me a bunch of warnings18:18
fungiso good news!18:20
fungizuul apparently designed for it without even realizing18:20
clarkbwell we have warnings but I can't make them error for some reason18:21
opendevreviewJames E. Blair proposed zuul/zuul master: Use branchesForRepoState in all cases  https://review.opendev.org/c/zuul/zuul/+/80417818:21
clarkbfungi: wow it is like the module matching just doesn't work at all18:30
fungiahh, yeah that was some of my takeaway. i'm able to get it to match very specific modules, but have been unable to figure out the wildcarding it supposedly supports18:31
clarkbfungi: are you matching the module of the warning class or the module of the location the warning is raise?18:31
fungi"yes" ;)18:32
fungii think my intent was to match the module where it's raised, but that didn't always work and sometimes i had to go back one or more steps in the stacktrace18:32
fungimind you, i've been taking a subtractive approach rather than an additive one, so no clue if that changes the equation either18:33
clarkbin my case I want to match every instance of a specific type of warning18:33
clarkbbut expressing that doesn't seem to be possible?18:33
clarkbalso I don't seem to get a traceback just a string?18:34
fungiis stestr stuffing the traceback into the subunit stream?18:35
fungithat's where i'd expect to find it at least, so presumably where you're looking18:36
fungitherefore the answer is presumably no18:36
clarkbya I'ev forced a failure in the test so that we capture the logging18:37
clarkband that is what I seem to get out when always::DeprecationWarning is used18:37
fungitried error::DeprecationWarning instead?18:37
clarkbya that just breaks beacuse of other errors.  Ineed to filter to sqlalchemy. However, I think I may be making progress it filters where the code triggers the warning based on some testing18:38
fungiright, and yeah "breaking" is the only way i know to get it to provide a traceback, if that makes sense18:39
fungi(only way i know via the warnings feature anyway)18:39
clarkbI see18:40
clarkbalso now that I was able to make it work specifying a specific path I am trying to make the globbing work one level up from that without an luck at all18:40
fungiwhich of course turns it into a game of whack-a-mole since you can only see the first one it hits18:40
clarkbI've tried .* [.*] * etc18:40
clarkbI think the docs must just be wrong18:41
fungii've almost reached the point of digging into the python source code to see what it actually expects for wildcarding18:41
fungii agree it's frustrating, and the envvar has plenty of other shortcomings too (like lack of escaping, so no way to match an exception string which contains a comma because that's the field separator for its language)18:42
fungiapparently the cli option isn't much better18:43
fungiif you diddle the global for it at runtime you get much more flexibility, but that's the hardest to do as a sort of test wrapper18:43
clarkbPYTHONWARNINGS=default::DeprecationWarning:zuul.driver.sql.sqlconnection works but PYTHONWARNINGS=default::DeprecationWarning:zuul.driver.sql.sqlconnectio. does not and the docs say it is a regex18:43
clarkbThat is the most trivial regex I can think of which makes me think they aren't actually using a regex filter18:44
fungiyes, if you look at the python/cpython issues on github you'll find a fair number about the warnings module18:44
clarkbmodule = re.escape(module) + r'\Z'18:46
clarkbI think they may be escaping all the . and *18:46
fungihah18:46
clarkbhttps://github.com/python/cpython/blob/3.9/Lib/warnings.py#L227-L22818:47
fungiso far from accepting a regex, it's treating every regex as a literal18:47
clarkbit is a regex with all the useful bits escaped out18:47
fungior rather, turning everything which might make it into an actual regex into just a string18:47
fungiand main (future 3.10) is no better18:48
fungilooks like that escape for the module string was introduced by the fix for https://bugs.python.org/issue39056 in 3.918:52
fungidoes it work for you under 3.8?18:52
fungifirst appeared in v3.9.0a3 but i guess it may have also been backported18:52
fungiahh, yep helpfully backported to 3.8 and 3.718:53
clarkbya I'm on 3.818:53
clarkbfungi: I'm not completelysure that the code path flows through there for the env var settings but it certainly seems like they do given the function name18:54
clarkbI wonder if we should file a bug18:54
fungii compile my own interpreters from source, so easy enough for me to test18:59
fungii'll revert that line from "module = re.escape(module) + r'\Z'" to "module = module + '$'" and see what happens19:01
fungibasically undoing just that part of https://github.com/python/cpython/commit/41ec17e19:01
clarkbthat sould confirm it19:06
fungii'll know in a little while19:07
clarkbI need to eat lunch but I've made progress fixing deprecation warnings that test_migration trips over. I've been focusing on the mysql side though and need to get postgres running next. But I think this chagne is actually pretty minimal since it seems that it is largely the test tooling that hits problems and not the sql driver in zuul19:12
clarkbThat said, transaction semantics change so would be good to review for that being wrong too19:12
fungii found https://bugs.python.org/issue34624 and https://github.com/python/cpython/pull/9358 mentioned therein19:16
fungimodule = r'\A(' + module + r')\Z'19:16
fungiwhich seems to confirm our observations19:16
fungiin "awaiting core review" since 201819:17
fungi(and people complain about how long it takes to get things reviewed in our projects!)19:18
clarkbmaybe we should push a change to update the docs19:19
clarkbthen at least people won't be so confused19:19
fungii found one of those too19:20
corvusclarkb: thanks!  any time you want to push up a wip change and get another hand, let me know.  otherwise i'll stay out of your way for now.  :)19:20
corvusfungi: thank you too :)19:20
clarkbcorvus: will do shouldn't be long after lunch19:20
clarkbfungi: hahahahaha that just made my day19:20
fungishouldn't be all that surprising, python has the infinite monkeys thing going on... "Pull requests: 1.4k"19:33
fungii didn't even know github started abbreviating counts19:33
fungibut anyway, https://github.com/python/cpython/commit/62ec638 looks like will be in 3.10 when it releases and tries (albeit not all that well) to indicate that you need to manipulate warnings.filterwarnings() if you want to use a regex19:40
fungivstinner also started a python-dev ml thread back in april about it and related topics: https://mail.python.org/archives/list/python-dev@python.org/thread/JMKLA4RUJLTORBPPTM4BWL76VNNMKYVG/19:43
opendevreviewClark Boylan proposed zuul/zuul master: Prepare for SQLAlchemy 2.0  https://review.opendev.org/c/zuul/zuul/+/80445620:00
clarkbcorvus: ^ I don't know that that needs to be WIP. thest db migration tests pass locally for me but I haven't run a full suite20:00
clarkbcorvus: I confirmed that setting future=True on the orm objects seems to also raise errors if we do old style stuff so that should catch a lot of stuff20:01
fungithat's probably a safer and more manageable approach anyway20:02
corvusclarkb: you say sa2.0 will not support executing strings, but you change to using execdriversql -- is execdriversql considered "not sqlalchemy" (ie, just a pass through?)20:04
clarkbcorvus: I wasn't super clear you can't use execute('sql here')20:05
clarkbyou can use execute(sqlalchemy.text('sql here')) or exec_driver_sql20:05
corvusgotcha, thx20:05
clarkbI suspect that if you are doing production work with it that text() is probably best and does safety escaping20:05
clarkbbut for what the tests are doing it was simple to use exec_driver_sql20:05
clarkbcorvus: https://docs.sqlalchemy.org/en/14/core/connections.html#sqlalchemy.engine.Connection.exec_driver_sql there is a "here be dragons" block there and seems that exec_driver_sql doesn't trigger certain events? For testing I think that is fine21:05
clarkbbut if you'd prefer execute(text('sql here')) I can rewrite it21:05
corvusclarkb: those don't seem important for testing21:06
dmsimardlol, I like that there is an actual "here be dragons" block21:08
corvusyeah, almost makes you actually want to read 'em21:09
opendevreviewJames E. Blair proposed zuul/zuul-operator master: Fix wait for scheduler to settle in tests  https://review.opendev.org/c/zuul/zuul-operator/+/80446221:11
opendevreviewJames E. Blair proposed zuul/zuul master: Move sigterm_method to zuul.conf  https://review.opendev.org/c/zuul/zuul/+/80446421:23
corvusfungi, clarkb, tristanC, tobiash: ^ I was about to approve https://review.opendev.org/800284 (since i made the 4.8.0 release) and suddenly had the thought that it felt inconsistent, and I couldn't really think of a good reason to use an env var instead of zuul.conf for that; can you take a look at the followup in https://review.opendev.org/804464 and let me know if you agree?21:25
clarkbthats a good point. The env var would've been read fresh every time so if you could somehow hack it via /proc then you could chnge it while it was running? that said I'm not sure we would want peopel to rely on that21:27
corvusi think we just ended up at env var because of the way the idea started; env vars are sometimes easier to deal with in k8s, but the executor requires a zuul.conf anyway, so it's no big deal.21:27
clarkbthe sqlalchemy 2.0 change passed python 3.8 unittesting but timed out on the 3.6 unittesting. I have rechecked it21:33
corvusclarkb: so you're saying there's a chance? :)21:37
clarkbyes21:37
clarkbI think the core bits in sqlconnection.py were already largely new sqlalchemy clean because they use the context manager stuff21:38
clarkbcorvus: it looks like zuul's release notes didn't update for 4.8.0?21:42
clarkbhttps://zuul-ci.org/docs/zuul/reference/releasenotes.html that page specifically21:43
corvusclarkb: need to merge a change to master for that21:43
clarkbaha21:43
clarkbwe only run the publish when we land changes not on tags. got it21:43
corvusclarkb: https://zuul-ci.org/docs/zuul/4.8.0/reference/releasenotes.html exists though, and that's what the announcement links to21:43
clarkbhttps://review.opendev.org/c/zuul/zuul/+/800121/ should do it then as it is in the gate21:43
fungii'm perfectly fine with putting the toggle in the config, i wondered about it but assumed envvars were a "this is how you container" choice21:49
fungicorvus: related to the release, we said after zuul released with the gear pin, we could revisit tagging new gear. are we to that point yet, or should there be some additional buffer time? to make sure people on releases have upgraded?21:50
corvusfungi: i think it's fine; i think only ppl doing new installs of old releases might be affected (and we don't actually think it's a problem anyway)21:51
*** dviroel|ruck is now known as dviroel|ruck|out21:51
fungiyeah, i don't expect any gear-induced regressions, this was more belt-and-braces21:53
fungii'll stick the gear release on my reminders for next week in that case21:53
fungimy dance card is pretty full up to the weekend21:53
clarkbin the time since about lunch the sky outside has turned brown and the air is less good for you say the internet sensor aggregation map21:54
clarkbkind of amazing how quickly that happens21:54
fungiyeah, the government is trying to convince me that the sun wants to kill me today, and that i should hide indoors. pfft22:00
opendevreviewMerged zuul/zuul master: Remove some unused methods in sql reporter  https://review.opendev.org/c/zuul/zuul/+/80012122:27
ianwa bit of a hail-mary, but does anyone have any insight into "A state mutation was detected inside a dispatch, in the path: `job.jobs`. Take a look at the reducer(s) handling the action {"type":"JOB_FETCH_SUCCESS"23:06
ianwi just wanted to update the job page which i spent quite a lot of time looking at yesterday finding debian-stable jobs23:07
clarkbtristanC: ^ is probably the best person to ask?23:08
clarkbwoot https://review.opendev.org/c/zuul/zuul/+/804456 got a +1 from zuul. Maybe the sqlalchemy 2.0 prep is that simple23:08
corvusianw: that probably means something in store.js didn't update the state dictionary right (you can't mutate it directly, you have to return a copy; the weird functions in that file do that)23:41
corvusthough as someone pointed out on the list recently, the ... operator can actually be used as a simpler version of that23:42
ianwyeah, i'm trying to rewrite it with that now23:42
ianwwe seem to use it in some places, and others use $merge (which i think comes form some libray)23:43
corvusyeah $merge is the "weird function" i was thinking of :)23:43
fungiclarkb: zzzeek just followed up to the openstack-discuss thread with a link to the sqla v2 migration docs too23:58
funginot sure if you found those in your research23:59

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!