Tuesday, 2020-07-21

*** openstack has joined #zuul07:32
*** ChanServ sets mode: +o openstack07:32
*** holser has joined #zuul07:34
*** tosky has joined #zuul07:37
ianwclarkb: might want to squish https://review.opendev.org/#/c/742057/ into your wheel patch to get something that works together07:38
*** wuchunyang has joined #zuul07:45
openstackgerritFabien Boucher proposed zuul/zuul master: gitlab: remove default mutables  https://review.opendev.org/74193807:46
*** jpena|off is now known as jpena07:50
*** piotrowskim has joined #zuul07:58
*** nils has joined #zuul08:13
*** wuchunyang has quit IRC08:46
*** bhavikdbavishi has quit IRC08:57
*** bhavikdbavishi1 has joined #zuul08:57
*** djinni` has joined #zuul08:58
openstackgerritFabien Boucher proposed zuul/zuul master: gitlab: support the MR merged event  https://review.opendev.org/74210708:58
*** bhavikdbavishi has joined #zuul09:00
*** jcapitao has quit IRC09:00
*** bhavikdbavishi1 has quit IRC09:02
*** jcapitao has joined #zuul09:02
*** holser has quit IRC09:26
*** holser has joined #zuul09:27
*** jcapitao has quit IRC09:34
*** bhavikdbavishi has quit IRC10:07
*** bhavikdbavishi has joined #zuul10:07
*** bhavikdbavishi has quit IRC10:18
*** jcapitao has joined #zuul10:30
*** jcapitao is now known as jcapitao_lunch10:30
*** jcapitao_lunch has quit IRC10:39
*** bhavikdbavishi has joined #zuul10:40
*** jcapitao_lunch has joined #zuul11:27
*** jpena is now known as jpena|lunch11:40
*** sshnaidm|afk is now known as sshnaidm11:41
*** iurygregory has quit IRC11:49
*** jcapitao_lunch is now known as jcapitao11:57
*** iurygregory has joined #zuul11:59
*** rlandy has joined #zuul12:00
*** rlandy is now known as rlandy|ruck12:01
*** rfolco has joined #zuul12:08
*** Goneri has joined #zuul12:17
*** jpena|lunch is now known as jpena12:43
openstackgerritClark Boylan proposed zuul/nodepool master: Omnibus nodepool container image fixes  https://review.opendev.org/74194213:14
clarkball the fixes squashed together. I expect that may be landable to get an image out even if it isn't the exact end state we want13:14
*** holser has quit IRC13:20
*** holser_ has joined #zuul13:20
*** holser_ has quit IRC13:32
*** holser has joined #zuul13:33
felixedelzuul-maint: Could I get a review on two small PF4 fixes/improvements https://review.opendev.org/#/c/740914/ and https://review.opendev.org/#/c/740923/ ?13:35
*** bhavikdbavishi has quit IRC13:42
clarkbugh cryptography literally just did a release so that wheel isn't in our arm cache13:51
clarkband the change I just pushed is trying to build cryptography. I guess we'll see if it can get that done within the timeout time13:51
clarkbfelixedel: I'm trying but admittedly somewhat lost. In https://review.opendev.org/#/c/740923/1/web/src/App.jsx do we rely on activeClassName on line 102 to do the equivalent of the old check for isActive via react magic?14:01
clarkbI think I'm ok approving that one except I don't quite understand exactly how we're checking which item is the active item14:01
clarkbfelixedel: and for the other change I feel like I'm even more lost :)14:04
felixedelclarkb: The activeClassname property comes with the NavLink component. It's a component from react-router that allows custom styling of the currently active Route. So in that case I would call it "react-router magic" ;-)14:06
felixedelAh, now I see what you mean. The isActive was a leftover from PF3 and not used anymore in PF4.14:07
fungidigging into a bit of an odd situation in opendev... we restarted the scheduler on 2020-07-17 with current master branch of zuul/zuul deployed, zuul-web was not restarted at that time (last restarted 2020-06-16). github webhooks continue to be processed by zuul-web and submitted to the scheduler via rpc, but the scheduler does not log seeing them since it was restarted14:10
clarkbyup after some googling I've discovered that NavLink does magic :)14:10
felixedelPF4 wants you to implement an onClick handler on the navitem to highlight the current one. But that won't work if somebody visits a page directly. Since I didn't find a proper solution for that with PF4 components I came up with the react-router solution14:11
clarkbfelixedel: I left an idea on  https://review.opendev.org/#/c/740923 but +2 otherwise.14:11
funginot finding any related exceptions/tracebacks in debug logs for either the scheduler nor zuul-web14:11
clarkbfungi: my hunch is that we've chagned something related to the zk switch and old zuul-web is still relying on gearman where it isn't expected to14:11
clarkbfungi: but I've not been able to keep up with all of the zkification changes14:11
clarkbnow trying to google around PageHeader and better understand that14:12
tobiashnothing has changed yet regarding gearman vs zk14:12
clarkboh page header is local not part of react14:13
clarkbtobiash: thanks for confirming14:13
felixedelclarkb: https://www.patternfly.org/v4/documentation/react/components/nav14:13
clarkbaha thanks14:13
tobiashclarkb: do you see "Github Webhook Received" in the scheduler logs?14:13
fungitobiash: not since the scheduler restart, no14:13
tobiashthat should be logged before it is even trying to access github14:13
felixedelclarkb: For the whole PageHeader I took this demo as example https://www.patternfly.org/v4/documentation/react/demos/pagelayout14:13
tobiashhrm, can you get a list of gear functions?14:14
tobiashthere should be a function "github:<connection name>:payload" function registered by the scheduler14:14
fungitobiash: oh, wait, grep failure on my part. we do get it from geard:14:14
fungi2020-07-21 14:09:57,953 DEBUG zuul.GithubGearmanWorker: [e: dab55200-cb5b-11ea-8377-eaa62efdaa50] Github Webhook Received14:14
tobiashcan you grep for dab55200-cb5b-11ea-8377-eaa62efdaa50 ?14:15
felixedelclarkb: Where are you lost in the other change (740914) ? :)14:15
fungitobiash: yeah, just did: http://paste.openstack.org/show/796160/14:16
clarkbfungi: mostly just how logoComponent translates to "construct this as an intelligent link"14:16
clarkbdigging through pf docs now to see how that works14:16
tobiashfungi: are there exceptions between the start and end timestamp (without filtering)?14:17
fungitobiash: just a sec, finding a better example (a pull_request event for something we should be running jobs on)14:20
corvusoh we're over here now14:20
tobiashfungi: that's ok, there should be more in between the last two lines14:21
tobiashregardless of configured jobs14:21
tobiashhrm, exceptions should be logged with the event id there actually14:23
fungitobiash: yeah, nothing between those lines, you can see the line numbers to the left of the timestamps in this paste: http://paste.openstack.org/show/796162/14:23
corvusfungi: for comparison i looked at an older event; i think this is typical:  zgrep 9e9a8e00-c7f7-11ea-94ba-01b971d1bdde debug.log.4.gz14:24
tobiashbetween starting and finished event processing this runs: https://opendev.org/zuul/zuul/src/branch/master/zuul/driver/github/githubconnection.py#L36814:25
corvusyeah and we don't see either the type or unhandled message, could self.connector._stopped be true?14:26
fungicorvus: agreed, we're not seeing the zuul.GithubConnection line or anything after it14:26
tobiashbut this should always log something14:26
tobiashcorvus: yes, that's the only explanation I have atm14:26
tobiashbut why should _stopped be true?14:26
corvusit looks like it's only set if we're shutting down14:27
tobiashyeah14:27
tobiashmaybe a thread dump helps (or even repl to check that variable)14:27
corvustobiash: the threading around the event processor is a little weird...14:28
corvustobiash: it runs in a thread pool, and its run method has a try/finally but no exception14:28
corvustobiash: so what happens if _process_event raises an exception?14:28
tobiashthat's a good question, maybe that doesn't log it then14:29
corvustobiash: would it propogate up through the future and log here? https://opendev.org/zuul/zuul/src/branch/master/zuul/driver/github/githubconnection.py#L719-L72714:29
tobiashexceptions are propagated and raised when calling result() yes14:30
tobiashso yes, we would see "Exception moving GitHub event"14:31
corvusok, just wanted to make sure we covered all the exits from that method14:31
fungidefinitely not getting those in the logs14:31
tobiashhttps://docs.python.org/3/library/concurrent.futures.html#concurrent.futures.Future.result14:32
tobiash"If the call raised, this method will raise the same exception."14:32
tobiashfungi: can you check the logs at that timestamp without grepping for the event id?14:34
tobiashmaybe there is an exception but without the event id14:34
fungitobiash: those are line numbers from the log14:34
tobiashbecause I think this is broken: https://opendev.org/zuul/zuul/src/branch/master/zuul/driver/github/githubconnection.py#L37814:34
fungitobiash: they're sequential, so no lines appear in the log between them14:34
tobiashah ok14:34
openstackgerritTobias Henkel proposed zuul/zuul master: Fix accessing the installation map in _process_event  https://review.opendev.org/74221114:37
tobiashI think this should fix this ^14:37
corvustobiash: how do we hit that without getting an exception?14:37
tobiashthat's what I don't understand, does try/finally catch all exceptions?14:37
corvustobiash: the exception should still be raised because it's unhandled14:38
corvusoh14:38
tobiashthat's what I thought as well14:38
corvusfungi: the exception would be *after* the finished line14:39
tobiashoh right14:39
fungiahh, well, my workstation's display locked up so i'm in the middle of rebooting it while the opendev conference is on a quick break, but i can look again shortly14:39
corvusi'll check14:39
tobiashI guess you'll have plenty of hits when searching for "Exception moving GitHub event"14:40
fungitobiash: i did grep the log for that string earlier and had no hits14:40
tobiashweird14:40
corvustobiash: any idea why this isn't tested?14:41
tobiashhrm, I think the events in the test cases don't have installation ids14:42
tobiashso the tests don't enter this condition14:42
corvusi concur with fungi there are no hits for that string, and there's nothing interesting within 100 lines of the "Finished event" log line; so it's still a mystery how we're getting out of there with no output14:44
corvustobiash: i think we'll learn a lot if we can reproduce this with a test; do you think you can add/update a test to use an installation id?14:46
fungiyeah, no Traceback lines at all in the log after the event i'm looking at14:46
fungi[e: 27912c80-cb4c-11ea-9cf1-66c0ab8a82ba]14:46
tobiashcorvus: I'll try to add a test case but maybe it makes sense in a followup since it's broken atm14:46
corvustobiash: a followup is fine, but i think i'd like to see the results from the test case before we restart opendev (there's still something we don't understand about this, so we may not have the full story).14:48
corvus(iow, i'm happy to approve the expected fix now, but don't want to take 2 outages if we find something else when we do the test case)14:49
corvustobiash: also, are you running this in prod yet?  or do you not have installation ids?14:49
fungiyeah, i mean the situation for us is not great, but it's also not urgent since we don't really do much with github14:49
tobiashno, that's not yet deployed in our prod14:49
fungiand i agree, not restarting twice would be preferable14:50
tobiashk, if it's not so urgent I'll try the test case before merging the fix14:50
corvusi think it's fine to merge the fix14:50
corvusi just don't want to restart with the fix until we've at least run the test case locally :)14:51
fungii concur14:51
corvus(because i figure since there's part of this we still don't understand, there's a 10% chance the test case might uncover something else)14:51
fungiso i guess connection.installation_map still exists (or else we'd be raising exceptions)?14:54
tobiashno, it doesn't14:54
fungiso something must be swallowing the exception from _process_event() as it's not inside a try/except block there14:57
tobiashok, I have a quick and dirty local reproducer which fails the test but doesn't print the exception as well14:59
tobiashand the fix proposed makes that test green14:59
tobiashnow I'll check why the exception is hidden14:59
tobiashoh I know why: https://opendev.org/zuul/zuul/src/branch/master/zuul/driver/github/githubconnection.py#L366 this sets a valid return code and prevents the exception from bubbling up15:02
tobiashjust removing that try/finally makes the exception visible15:02
clarkbtobiash: can we and an exception handler there that simply logs the error then let finally run?15:03
clarkbthat way we at least get the logging15:03
tobiashthe exception handler there is just for logging the finished message15:04
openstackgerritFabien Boucher proposed zuul/zuul master: Remove shebang for base/library/command.py|zuul_console.py  https://review.opendev.org/72895515:04
corvustobiash: wow i didn't expect that15:04
tobiashI think it's the return which inhibits all exceptions15:05
corvustobiash: yeah, that works under normal circumstances (even just a plain function call -- doesn't need a thread pool)15:06
corvustil.15:06
corvusclarkb, fungi: seems like https://review.opendev.org/742211 is gtg if you want to +3 it15:06
corvusunless you want to wait for the exception handler logging15:06
fungii think i'm good with it, i just wanted to understand first how we weren't getting an exception raised from that15:07
clarkbcorvus: done15:07
clarkbya I think now that we know what the problems are we can easily address those in a followup15:07
tobiashyepp, confirmed the return in the finally inhibits the exception even if there is an explicit raise in the handler15:07
tobiashbecause after thinking about a return 'repairs' the exception state if it's within the finally clause15:08
tobiashfix for the event handling is then just unindenting the return :)15:08
*** bhavikdbavishi has joined #zuul15:11
openstackgerritTobias Henkel proposed zuul/zuul master: Don't mask exceptions in _process_event  https://review.opendev.org/74222215:13
openstackgerritTobias Henkel proposed zuul/zuul master: Add repository and installation id to events in tests  https://review.opendev.org/74222315:13
tobiashlogging fix and the test case (tested with one test case locally) ^15:14
tobiashftr: "If the finally clause executes a return, break or continue statement, the saved exception is discarded" from https://docs.python.org/3/reference/compound_stmts.html#the-try-statement15:18
corvusall approved, thanks tobiash!15:20
tobiashthanks for saving me from catching this in prod :)15:20
*** bhavikdbavishi has quit IRC15:20
*** bhavikdbavishi has joined #zuul15:23
fungitobiash: yw! thankfully it's not as severe an impact for us as it would have been for you or others15:27
openstackgerritMerged zuul/zuul master: Fix brand logo link for dashboards deployed on a sub-url  https://review.opendev.org/74091415:28
tobiashcorvus: do you want more reviews on the change queue changes since you didn't +w them?15:36
tobiash(https://review.opendev.org/718531 and child)15:38
corvustobiash: oh, i was going through in series and didn't want to +w until i got to the end.  but then the really complicated change was at the end and i was out of brainpower, and i've forgotten to get back to it.15:38
corvussorry15:38
corvustobiash: i'll try to review the last change after we cut the release15:39
tobiashthanks!15:40
*** yolanda has quit IRC15:42
corvuswe got an error on the stable/3.x branch: https://zuul.opendev.org/t/zuul/build/86a2c3b858d34aa2a1c7aa63c517b501/console#2/0/56/ubuntu-bionic15:42
corvus/usr/bin/python3: No module named pip15:42
corvusclarkb, fungi: does that make sense to you?15:43
corvustobiash: oh -- https://review.opendev.org/74207415:43
clarkbcorvus: I374dac18b9b7e376d924b11f4661355ea7c4d149 and I3dec251d19dd7b3807848a54e6a20a8e89d30a4e probably need to be included?15:44
*** yolanda has joined #zuul15:44
corvusclarkb: i think tobiash's change may be enough?15:44
corvusi think the move to pre is more or less an optimization15:45
clarkbcorvus: I think its two different jobs15:45
clarkbquickstart and log streamer15:45
tobiashat least 742074 has a green check15:45
clarkbmaybe we only run log streamer under specific cases15:45
clarkbya 742974 looks good15:45
clarkb*74207415:45
corvusi approved it and rebased the other 2 changes on it15:46
zbr|rovertobiash: can you please look again at https://review.opendev.org/#/c/739482/ ? i think i addressed your concerns.15:48
openstackgerritTobias Henkel proposed zuul/zuul master: Block localhost shell tasks in untrusted playbooks  https://review.opendev.org/74222915:49
corvuszbr|rover: you gonna look into redux?15:49
tobiashremote: Block localhost shell tasks in untrusted playbooks  https://review.opendev.org/74223015:49
clarkbI've approved both after skimming and confirming they look the way I expected15:53
clarkbcorvus: tobiash ^15:53
felixedelclarkb: Basically the logoComponent specifies which component (or tag) should be used for the link element. By default that's an <a>. The patch changes this to a <Link> so we can utilize react-router's capabilities to get the correct relative URL16:00
clarkbfelixedel: yup I finally tracked that down and left a link to the docs in the change that explains the <a> vs <Link> thing16:01
tobiashzbr|rover: auto reload still seems to be off by default (tried in multiple browsers and with private more)16:02
tobiashtoggling seems to behave correctly now16:03
felixedelclarkb: Regarding your idea on https://review.opendev.org/#/c/740923/. I think this should be part of a separate change if we decide to do so. But I think other websites also use a URL schema like builds/ and build/:id (which would also reflect the URLs of the API).16:06
openstackgerritMerged zuul/zuul master: Fix accessing the installation map in _process_event  https://review.opendev.org/74221116:06
*** marios has quit IRC16:07
corvusclarkb: if you're in a web-space, https://review.opendev.org/740345 and parent could use a +w16:13
openstackgerritMerged zuul/zuul master: Don't mask exceptions in _process_event  https://review.opendev.org/74222216:16
zbr|rovertobiash: going to test the default value now.16:20
openstackgerritMerged zuul/zuul master: Add repository and installation id to events in tests  https://review.opendev.org/74222316:22
*** hamalq has joined #zuul16:27
*** hamalq has quit IRC16:28
*** bhavikdbavishi has quit IRC16:28
openstackgerritSorin Sbarnea (zbr) proposed zuul/zuul master: Enable optional pre-wrapping on console and output  https://review.opendev.org/72360316:28
*** hamalq has joined #zuul16:29
*** bhavikdbavishi has joined #zuul16:30
openstackgerritSorin Sbarnea (zbr) proposed zuul/zuul master: Enable optional pre-wrapping on console and output  https://review.opendev.org/72360316:33
corvuszbr|rover: if you're interested in learning about redux i can answer questions and give you pointers16:40
zbr|rovercorvus: clearly that I am because i wasn't able to figure it out from my initial attempts.16:42
zbr|roversadly the attention lifespan is close to a goldfish, i always endup being pinged to fix something else before i managed to get it right.16:43
zbr|rovergood part is the the ^ above adds preferences w/o need for redux, at least it will make the other change easier to review.16:43
zbr|rovermaking wrapping configurable works with pure css variables quite nice16:44
zbr|roveri was impressed to get it working from 1st attempt16:44
corvuszbr|rover: :)  okay so the way i see it is that it's a pretty simple system that has been made unecessarily complex and hard to understand.  basically there's a thing called a reducer which is just a function that updates a central state object.  redux calls the reducer and updates the state object with the value it returns.  there's an action which is really just a string constant.  there's a function16:46
corvusassociated with the action which sends that string constant to redux, and it calls the appropriate reducer.  so when you "dispatch" an "action", redux calls the "reducer" for the action and updates the state.16:46
zbr|roverclarkb: do you remember asking me about cmd changes? here is how it looks now, without pre: https://sbarnea.com/ss/Screen-Shot-2020-07-21-17-45-42.64.png16:46
corvuszbr|rover: it looks like the timezone feature uses redux, and is fairly self-contained, you can probably use that as a model for most of it.16:46
corvuszbr|rover: the extra part will be that we'll need to set the initial/default value of the state to something loaded from localStorage, and write to localStorage whenever it's updated.  that means that the reducer function should write out to local storage.16:47
zbr|roverprobably we need a generic preferences loader that loads all of them, so we do not have to reimplement it for each option we add.16:48
corvuszbr|rover: agreed, we can have a preferences object and each of the preferences being an attribute on the object16:49
corvus(then serialize that object to json for storage)16:49
zbr|roveri kinda like keeping settings as individual entries in localStorage, makes it much easier to debug and tune.16:50
zbr|roverdo we have any performance limitations from using individual keys for each option?16:50
zbr|roveri know it supports few MB of data, so I doubt "blob" approach is really needed.16:50
corvuszbr|rover: it's easy to debug a json serialized string in storage.  that lets us save/load the entire preference blob in one line.  i think it's going to be a lot more concise.16:51
*** bolg has quit IRC17:01
*** jpena is now known as jpena|off17:05
*** yolanda has quit IRC17:10
*** bhavikdbavishi has quit IRC17:10
*** yolanda has joined #zuul17:13
*** nils has quit IRC17:15
*** hamalq has quit IRC17:16
*** hamalq has joined #zuul17:16
openstackgerritTobias Henkel proposed zuul/zuul master: Block localhost shell tasks in untrusted playbooks  https://review.opendev.org/74222917:18
tobiashclarkb, corvus: that needed a small fix for the zuul-stream tests ^17:19
tobiashthere was a bug lurking in the test that hit us now: https://review.opendev.org/#/c/742229/2/playbooks/zuul-stream/templates/ansible.cfg.j217:19
tobiashremote: Block localhost shell tasks in untrusted playbooks  https://review.opendev.org/74223017:21
corvustobiash, clarkb: i re +3d; i think clarkb is afk but i'm sure he'd be okay with that fix :)17:24
*** jcapitao has quit IRC17:26
fungiis that a backport from master?17:30
fungiahh, yeah now i see 74222917:32
tobiashyes, cherry pick17:32
tobiashI didn't really have to 'backport' it :)17:32
fungifair17:33
fungiif you cherry-pick -x then it will add a line to the commit message referencing the original commit id too17:34
tobiashthanks for the hint for next time, this time I was lazy and just hit the cherry pick button in gerrit17:36
fungican sometimes help with clarity, though in this case i think we all understand that basically everything on that temporary stable/3.x branch is taken from changes in master17:36
fungiand it's not like we expect to be doing this all the time17:37
fungiahh, that was the cherry-pick button, got it17:37
tobiashwell, we already have two cherry picks on the stable branch...17:37
tobiashbut anyway, thanks for that tip, I never used cherry-pick -x yet :)17:39
openstackgerritMerged zuul/zuul master: Keep active nav links highlighted while browsing the page  https://review.opendev.org/74092317:39
tobiashremote: https://review.opendev.org/742260 Create virtualenvs in series to avoid cache race17:43
tobiashclarkb, corvus: the stable branch needs another cherry-pick for the stream tests ^17:43
tobiashfungi: now used cherry-pick -x :)17:43
tobiashbut it looks odd having that comment beneath the change id so I reordered that part of the commit message17:44
fungioh, yeah the change-id does have to be in the last "paragraph"17:44
openstackgerritSorin Sbarnea (zbr) proposed zuul/zuul master: Enable optional pre-wrapping on console and output  https://review.opendev.org/72360317:52
openstackgerritMerged zuul/zuul master: Block localhost shell tasks in untrusted playbooks  https://review.opendev.org/74222918:15
*** vishalmanchanda has quit IRC18:26
corvusfelixedel: i've noticed on the status page that pageup/down doesn't seem to scroll18:51
clarkbneither does spacebar fwiw18:52
fungiwas anyone able to work out why the x11 primary selection buffer no longer gets populated when console stream text is highlighted with the pointer?18:53
clarkbfungi: fwiw that rarely works for me in ff or chrome18:57
clarkbI have to right click copy and then right click paste in my DE for whatever reason18:57
fungii think some javascripty lib must be capturing mouseclicks or selection events19:10
ianwi like whatever happened to the UI since i last looked; one weird thing on mine is that the hosts under "Results" are much smaller font size than everything else (https://imgur.com/a/91hKUDD).  it looks a bit weird to me19:47
corvusianw: https://review.opendev.org/74012119:47
corvusand child19:48
clarkbalso I think it broke the whitelabeled setup for openstack19:48
clarkbI'm not sure if that is just browser cache problems though19:48
corvusclarkb: wfm19:48
clarkbah yup it seems to workfor me now too. likely was just caches19:48
ianwcorvus: cool ... the site preview link there doesn't work? https://9f9df69279350fd5a76c-d6fd3b06d50c034d0364bbf684ea1b1c.ssl.cf1.rackcdn.com/740121/1/check/zuul-build-dashboard-opendev/8132036/npm/html/19:51
corvusianw: maybe try the child?19:52
ianwcorvus: yeah, that seems to19:53
ianwi still feel like the size is wrong with the child change19:54
corvusianw: oh, yeah, i think my change doesn't fix that19:54
ianw... interesting ... if i narrow the window, the font gets bigger, make it wider, and it gets smaller19:55
ianwi also don't see any buttons next to "Results" which i feel like maybe i should with that child change19:55
corvusianw: it's in the console tab, not the results section19:56
ianwcorvus: here's what i'm seeing ... https://imgur.com/a/KcrbuAy20:02
ianwwhen i pull it inwards, i think the font size goes from 12 to 16 point20:02
ianw(to be clear, 16px makes it look consistent)20:04
corvusianw: yep i'm seeing it too20:11
ianwi have something that fixes it for me ...20:11
corvusi fixed a similar issue on a different page20:11
corvus2 weeks ago20:11
corvuswas just hoping someone would approve the change20:12
corvusi think we should merge that, and merge the build page refactor before the backlog becomes too large20:12
openstackgerritIan Wienand proposed zuul/zuul master: ui: make list-group-item-heading same size as other fonts  https://review.opendev.org/74227420:12
corvustobiash: https://review.opendev.org/74227120:13
corvustobiash: i think that's only other thing we said we were going to put in 3.19.1?  so we should land that and then cut a release?20:13
tobiash++20:14
tobiashclarkb: you mean selecting in the live log stream?20:17
tobiashI think there we need to handle the key combimations ourselves if I remember correctly20:19
corvusianw: are you going to leave a review on https://review.opendev.org/740345 and parent?20:21
ianwcorvus: i can, if "i played with this" is sufficient :)20:22
corvusi think so :)20:23
mnaseris there a good example of a change that does multiarch image builds?20:25
mnaseror switched an image to using multiarch20:25
tobiashmnaser: I think nodepool does multiarch in the meantime (if the jobs are fixed already)20:29
tobiashmnaser: or maybe not yet, this is the latest attempt to fix those: https://review.opendev.org/#/c/74194220:30
mnaseroh wow that seems to be quite a performance hit20:31
mnaseri was looking to build a small golang docker image, maybe that might be a lot less of an impact20:31
mnaserok ill try it for a very tiny golang project and see what happens20:33
corvusmnaser: we're building under qemu, so anything we have to compile is really slow20:33
corvusotherwise, it's not too bad20:33
mnasercorvus: if a cloud has nested virt, does that actually have an impact/use a speed up?20:33
mnaseroh wait, nevermind20:34
mnaserthis is a different architecture20:34
clarkbcorrwct neated virt only helps for same arch20:34
clarkband I cant type20:34
mnaserok, seems to work, though i had a GOARCH=amd64 that i just had to remove20:51
*** y2kenny has joined #zuul21:12
mnasertime="2020-07-21T21:03:36Z" level=fatal msg="Error initializing image from source docker://127.0.0.1:60319/vexxhost/node-labeler:latest: unsupported docker v2s2 media type: \"application/vnd.oci.image.layer.v1.tar+gzip\""21:22
mnaserdid i do something wrong?21:22
corvusmnaser: that msg is not familiar to me; more context?21:22
mnasercorvus: i was trying to enable multi-arch builds for this very small golang project - https://review.opendev.org/#/c/742276/221:23
corvusi think our reno setup is really unhappy with multiple branches21:23
corvusthis is the preview build for the tip of 3.x: https://7c3f904deee39351719a-9c035ca54e1e355b36ad4d338836f375.ssl.cf1.rackcdn.com/742271/1/gate/zuul-tox-docs/ac36a3a/docs/reference/releasenotes.html21:24
clarkbcorvus: reno jobs should only run from master branch21:24
corvusit's empty21:24
mnaserand the image built fine, but then the push-to-intermediate-registry blew up21:24
clarkbat least that is how openstack has dealt with reno confusion there21:24
clarkbmnaser: is your intermediate registry up to date? there were a number of bugs fixed around it21:24
corvusclarkb: so the next time we land a master change after 3.19.1 is tagged, it should show up there?21:24
mnaserclarkb: patch is on opendev.. so it should be? :)21:25
corvusclarkb: but in the mean time, we don't have a way to preview what the 3.19.1 reno section will have?21:25
clarkbcorvus: yes aiui reno will evaluate all the branches21:25
clarkbI'm not sure about the preview question21:25
openstackgerritMerged zuul/zuul master: Web: Adjust console tab type sizes for pf4  https://review.opendev.org/74012121:25
corvuscause as it stands, i don't actually know if we have any release notes for 3.19.1, and it seems like we should21:25
openstackgerritMerged zuul/zuul master: Add an icon next to result buttons on the console log  https://review.opendev.org/74034521:25
mnaserand my job is `vexxhost-build-docker-image` which is nothing but a child of `opendev-build-docker-image`21:25
corvusmnaser: https://bugzilla.redhat.com/show_bug.cgi?id=1794167 seems relevant?21:29
openstackbugzilla.redhat.com bug 1794167 in ansible-role-tripleo-modify-image "buildah builds images that podman can't pull (unsupported docker v2s2 media type: "")" [High,Closed: errata] - Assigned to aschultz21:29
corvusmnaser: maybe we need to add the --format docker argument to our buildx invocation?21:30
corvusi don't know why we haven't hit this with nodepool though21:30
corvusclarkb: i guess i should tag locally and do a reno build?21:31
mnasercorvus: maybe it's because something something the base image i use is golang and it has something something that causes this?21:31
corvusmnaser: could be21:31
clarkbcorvus: ya that may be the easiest thing? I don't really have better ideas. smcginnis may though?21:32
corvussmcginnis: (wondering what's the best way to find out what the release notes will look like for a release on a non-master branch -- the preview job doesn't seem to produce any output: https://7c3f904deee39351719a-9c035ca54e1e355b36ad4d338836f375.ssl.cf1.rackcdn.com/742271/1/gate/zuul-tox-docs/ac36a3a/docs/reference/releasenotes.html )21:32
smcginniscorvus: Typically you need to have some sort of landing page that will pull in the notes for that branch.21:37
smcginniscorvus: Kind of like what we do for stable branch release notes: https://opendev.org/openstack/cinder/raw/branch/master/releasenotes/source/ussuri.rst21:38
smcginniscorvus: So looks like you would need a file that includes a directive to pull in stable/3.x21:38
corvushrm; i just want all the versions on one page21:39
corvusi tagged 3.19.1 locally, ran tox-docs, and got no output just like the preview21:40
smcginniscorvus: You can do that. You would just need something like an index.rst that includes that directive for each branch you want to show.21:40
corvusthis is an ephemeral branch though21:40
corvusit's about to be deleted21:40
corvuswill it show up in master after i merge the branch into master?21:41
smcginnisdhellmann might know a better way, but doesn't look like he's present.21:41
smcginniscorvus: Where is the release notes source docs? I see the notes in zuul/zuul, but not the rst files.21:41
corvusi added a branch directive to that, and it only pulled in 3.19.021:41
corvussmcginnis: doc/source/reference/releasenotes.rst21:42
smcginnisI *think* if it doesn't specify the branch, it should just pick up local changes.21:42
smcginnisHmm21:42
smcginnisTrying to build that one locally.21:44
corvusthis is unfortunate -- i merged 3.x into master locally and ran the build, and got the same thing as we're currently publishing: https://zuul-ci.org/docs/zuul/reference/releasenotes.html21:44
corvus(which is incorrect, that should have all of the releases ever)21:44
corvusi'm afraid we've somehow made a hash of this :(21:45
smcginnisYeah, something doesn't look right here.21:45
corvushttps://zuul-ci.org/docs/zuul/3.19.0/reference/releasenotes.html21:45
corvusthat's the page from the last release, so what we're publishing should look like that plus the in-development section at the top21:45
smcginniscorvus: Have there been release notes added since 3.19.0? I don't see one being added in the patch you referenced.21:46
corvussmcginnis: i suspect not (that was the original question i set out to answer)21:46
fungiif you locally tag a 3.19.1 on stable/3.x and then merge it back into master do you get release notes for it? or is that what you're trying?21:46
corvusfungi: that's what i tried and got the same thing that is currently published21:47
corvusfungi: (the in-development section only)21:47
smcginnisSo I think if you don't specify branch, it only goes back to the last tag. So if there aren't any other release notes added, that may make sense.21:48
corvussmcginnis: so if we assume there are no release notes, and some kind of behavior switch has happened to cause reno to only show the most recent notes, then both builds are "correct" according to those rules -- 3.x (3.19.1) has zero notes, and master has only the in-development ones.  but the critical error is that master should also have all the previous releases.21:48
corvussmcginnis: until recently, that has not been the case -- our builds had all the releases https://zuul-ci.org/docs/zuul/3.19.0/reference/releasenotes.html21:49
smcginnisYeah, based on the 3.19.0 release notes page you linked.21:49
corvus(i think we can live with 3.19.1 having a screwed up release notes page, but the master one being broken is a showstopper)21:49
fungiand if you tag 3.19.1 on the stable/3.x branch but don't merge it back to master what does reno on master do?21:50
clarkbdid adding the branch change reno behavior?21:50
*** Goneri has quit IRC21:50
corvusfungi: checking; clarkb i have lots of branches locally, i'd think this wouldn't be any different21:51
fungioh, good point, maybe tag 3.19.1 on the stable/3.x branch but don't merge it back to master *and* then delete the stable/3.x branch21:51
fungijust to rule out reno treating stable/.* branch presence specially21:51
corvusfungi: same behavior (in development only) with 3.19.1 tagged on stable/3.x and not merged to master then running tox -edocs on master21:51
corvusi'll delete stable now21:51
fungiif that's the case, then may also want to try with less-recent reno if there's been a very recent release... checking21:52
corvusi deleted my local branch; i wonder if it looks at remotes?21:52
corvussame behavior21:53
fungireno 3.1.0 was 2020-05-1421:53
smcginnisLooking at a git diff 3.19.0...HEAD, I'm not seeing anything that looks like it should have affected this.21:53
corvustrying with reno 3.0.121:55
corvussame21:56
fungidoes deleting the 3.19.1 tag restore the old behavior?21:57
corvusfungi: nope, same behavior21:57
fungithat's really strange21:57
fungiso the only things for it to be keying on at that point are recent commits in master21:58
corvusi just tried it in a clean checkout and got the same behavior21:59
smcginniscorvus: You had tested with 3.0.1, not 3.1.0, right?22:00
corvussmcginnis: both22:00
smcginnisI see a couple of potential suspects commits to reno, but they were both added in 3.1.0.22:01
corvuslet me try a clean tree with 3.0.122:01
corvus(the clean tree test got 3.1.0 for obvious reasons)22:01
smcginnisTiming-wise, 3.0.1 should have been what was used for 3.19.0, but you could try reno 3.0.0 to be really safe.22:02
corvus3.0.1 on clean tree has the same behavior22:04
corvuswill try with 3.0.022:04
*** decimuscorvinus has quit IRC22:06
corvussame behavior22:06
fungii wonder how we got it to build those currently published release notes then22:07
corvusme too22:07
*** decimuscorvinus has joined #zuul22:11
fungiyeah, even if i reset master to 3.19.0 i get the same22:13
corvusi'm stumped22:14
fungiactually, no, it's not building the release notes at all for me22:14
fungi`tox -e docs` is expected to build them?22:14
corvusfungi: yeah, if i build 3.19.0 i get a blank page (versus master where i get only the notes after 3.19.0)22:15
corvusfungi: and yes22:15
corvusre tox22:15
corvuswhat does this mean from reno: got versions []22:15
fungiyep, okay, that matches what i'm seeing then22:15
smcginnisI think that means it scanned through and didn't find anything.22:15
smcginnisWhich explains why it's blank, but doesn't explain what the heck changed.22:16
corvusgot versions ['3.19.0-77']22:16
corvusthat's what i get on master22:16
corvusand [] on the blank pages22:16
fungii wonder if some recent behavior change in setuptools could be at fault22:17
openstackgerritSean McGinnis proposed zuul/zuul master: Add reno configuration settings  https://review.opendev.org/74229622:18
fungithough no, reno should be looking at git versions not python versionbs22:18
smcginnisThat should do it ^^22:18
smcginnisBut I still don't like that we don't know why that is now needed.22:18
corvussmcginnis: i agree that works as expected22:19
corvuslet me try redoing the 3.19.1 tag and see what happens22:20
corvusi'm going to output some test results here, so to be clear, i'll repeat the first one i did:22:21
corvusmaster, no tags or stable branch, with smcginnis patch: works as expected (in development followed by all previous releases)22:21
corvusstable/3.x with smcginnis patch: works as expected (3.19.0 and previous releases -- no release notes on 3.19.1 branch yet)22:22
corvusstable/3.x with smcginnis patch and a new release note: works as expected (in development followed by all previous releases)22:24
smcginnisSo all looks right now?22:25
corvusstable/3.x with smcginnis patch and a new release note, tagged as 3.19.1: works as expected (3.19.1 and all previous releases)22:25
corvussmcginnis: so far so good; going to test a few more things22:25
smcginnisWhew22:25
corvusbuild on master with smcginnis patch, and previous stable/3.x branch and tag also existing locally: works as expected (in development from master and 3.19.0 and previous releases)22:26
corvusbuild on master with smcginnis patch and stable/3.x (tagged as 3.19.1) merged into master: not as expected: it only shows in development features22:30
corvussmcginnis: ^ if we merge stable/3.x back into master, i'd expect it to find all the releases (3.19.1 from the stable branch and 3.19.0 and all prior from the shared branch history), but it doesn't; instead it only finds the in-development notes.  any idea why?22:32
corvusi feel like "stop_at_branch_base: false" says don't do that22:35
fungiis that even after deleting the stable/3.x branch?22:35
corvusno didn't try that22:36
corvusfungi: same behavior22:36
corvussmcginnis: is there a way to get debug output from reno?22:37
corvus(maybe if i run it outside of sphinx?)22:37
fungii want to say the reason openstack stopped merging stable branch tags back into master was that it "confused" reno. i'll grant when i suggested the temporary branch i wasn't considering the need for release notes in things we tagged on it appearing on the master branch release notes22:37
fungithough maybe there's a way to make reno not get confused by it these days22:38
corvusi don't know, but my perspective is that i never want to not have release notes22:38
corvusi'm simple that way22:38
corvusi found:  reno -v report >/dev/null22:41
corvusi'm trying to see if there are any clues in that output22:41
fungior maybe reno has a provision for manual additions? that's one of the reasons i've been wary about it in the past is over concern that there's no way to fix up whatever it generates or add stuff after the fact22:41
corvusfungi: manual additions of what?22:42
fungireleases22:42
corvusfungi: manually tell it to add each of the previous tags?22:42
corvusthat sounds tedious22:42
fungijust the one on the temporary branch i mean22:42
corvusoh22:42
corvusi would rather the tail (reno) not wag the dog (git) on this22:43
fungithough maybe the other concern is that you want the tag from the temporary branch to also appear in master's history?22:43
corvusyes, i think that most closely expresses our view of the development history in graph form22:43
fungigot it. i also hadn't considered that desire when suggesting the temporary branch because openstack had given up on merging tags back into master when it adopted reno22:44
fungibut point is, reno seems mostly adapted to openstack's release workflows22:45
corvusignoring null-merge commits22:45
corvustreating b'7ce9cd026606a2488e6af3d632aa2c4a21ebff54' as a null-merge because parent b'd1e48428f8d3e9f95089ccce66d4268ed532e6f8' has tag(22:45
corvuss) ['3.19.1']22:45
corvusi wonder if that's relevant22:45
fungihttps://review.openstack.org/470023 introduced that check22:46
corvus+ignore_null_merge: false22:48
corvusthat does correct the behavior22:48
*** rlandy|ruck is now known as rlandy|ruck|bbl22:48
corvusi feel like the erroneous behavior doug describes is the behavior that i desire22:48
corvuswell actually22:49
corvusit ended up putting the in-development notes under 3.19.122:49
corvusi'm assuming if we merged 3.19.1 into master after we made a release, it might work as expected?22:50
corvusthat's not ideal though22:51
funginot sure, it might just put all the 4.0.0 release notes under 3.19.122:51
corvusso at the moment, we still have no way to build a page with all the notes of all the releases.22:52
fungithe mechanism by which reno tries to match notes file additions up with tags in which they appear is likely seeing those files getting added before the 3.19.1 tag merged to master22:52
corvusto be honest, i'm wondering if we should revert everything in master back to 3.19.0, make our patches, make a 3.19.1 release, then unrevert everything after22:52
corvusand never do this again22:52
fungii'm questioning the utility of reno for projects which don't follow openstack's release workflows22:53
fungithe assumptions it makes about git history are fragile22:53
fungigranted, i also thought openstack had a merged release notes for all its versions rather than per-branch release notes22:54
fungibut apparently they do not22:55
fungimaybe if reno grew a means of being able to retroactively associate specific notes with specific releases, to correct its autodetection failures22:56
*** y2kenny has quit IRC22:59
*** Goneri has joined #zuul22:59
corvusfungi: to answer an earlier question -- merging 3.19.1 after 4.0.0 is tagged looks correct23:00
corvusso it's just that the un-tagged notes get "tagged" once we merge 3.19.1 into master23:00
corvusif they already show up in 4.0.0, then it's fine23:00
corvusmaybe we could remove the in-development release notes, tag 3.19.1, merge it to master, then re-add the release notes23:01
fungioh, i hadn't thought of that. could actually work23:01
corvusit's easier than removing all the code :/23:02
corvusi'll try that23:02
fungiright, the blunt force solution would be a squashed revert of everything after 3.19.0, cherry-picking all commits (except .gitreview) from stable/3.x into master, deleting stable/3.x branch, tagging 3.19.1 on master, and then reverting the revert with some merge conflict resolution for the stuff picked earlier23:02
fungiwhich also basically creates two new bisect event horizons, so messy23:02
corvusstill working on that test23:09
corvusnope that doesn't work23:10
corvusas soon as we un-revert the notes, they end up in 3.19.123:10
corvusthough...23:10
corvusi guess we can re-add them with different uids23:11
corvusthat does look like it will work23:15
openstackgerritPierre-Louis Bonicoli proposed zuul/zuul-jobs master: Use ansible_distribution* facts instead of ansible_lsb  https://review.opendev.org/74231023:15
corvuswe're going to have an increasing amount of "unable to find release notes file" spam, but we can live with that; we might be able to add those to an ignore list23:15
corvuszuul-maint: it looks like there's a difficulty using reno with releases made from temporary branches; i think we can work around it.  here's the process:23:17
corvus1) add release notes to stable/3.x about the security update and the other important fixes.  this can be done in one new commit.23:17
corvus2) add the reno options to stable/3.x23:17
corvus3) release 3.19.1 from stable/3.x23:18
corvus4) remove the pending release notes from master23:18
corvus5) merge stable/3.x into master and delete the branch23:18
corvus6) re-add the pending release notes to master under a new filename23:18
corvus[eol]23:18
corvusfungi, smcginnis: ^ that look good?23:18
corvusit's been a long day for me, i need to eod, i'll pick this up tomorrow23:21
clarkbfor step 5 its a ours/theirs to just tie the history but not change cod eright?23:33
fungiit's probably all the same since the changes on stable/3.x since branching are just cherry-picks (aside from the .gitreview adjustment and release note additions)23:34
fungicorvus: that seems reasonable. i guess tagging before we remove the release notes on master is fine so long as it's before we merge stable/3.x back in23:36
*** Goneri has quit IRC23:36
fungier, so long as they're removed from master before we merge back in23:36
corvusclarkb: yes.  currently there's a minor conflict related to the quickstart, but i expect us to resolve in favor of the master branch; fungi yes i think that's fine based on seeing the correct output from reno on the stable branch (did not include orphaned master notes)23:39
*** iurygregory has quit IRC23:42
dmsimardo/ long time no see23:43
dmsimardI know there are people who like TTYs and CLIs here :P currently iterating on CLI for ara and thought you might be interested: http://paste.openstack.org/show/796192/23:44
dmsimardit's very rough, it might not be the right columns in the right order but it's there and it works23:45
*** tosky has quit IRC23:47
openstackgerritPierre-Louis Bonicoli proposed zuul/zuul-jobs master: Avoid to use 'length' filter with null value  https://review.opendev.org/74231623:52

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!