| *** mgoddard has quit IRC | 01:01 | |
| *** mgoddard has joined #zuul | 01:05 | |
| *** saneax has joined #zuul | 01:37 | |
| *** sanjayu_ has joined #zuul | 01:41 | |
| *** saneax has quit IRC | 01:44 | |
| *** sanjayu_ has quit IRC | 02:27 | |
| *** bhavikdbavishi has joined #zuul | 03:34 | |
| *** bhavikdbavishi1 has joined #zuul | 03:41 | |
| *** bhavikdbavishi has quit IRC | 03:42 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 03:42 | |
| *** raukadah is now known as chandankumar | 04:10 | |
| *** jamesmcarthur has joined #zuul | 04:21 | |
| *** bolg has joined #zuul | 04:24 | |
| *** threestrands has joined #zuul | 05:00 | |
| *** jamesmcarthur has quit IRC | 05:04 | |
| *** jamesmcarthur has joined #zuul | 05:06 | |
| *** jamesmcarthur has quit IRC | 05:10 | |
| *** tflink has quit IRC | 05:24 | |
| *** tflink has joined #zuul | 05:31 | |
| *** bjackman has joined #zuul | 05:42 | |
| *** AJaeger has quit IRC | 05:53 | |
| -openstackstatus- NOTICE: Due to a failure on the logs.openstack.org volume, old logs are unavailable while partition is recovered. New logs are being stored. ETA for restoration probably ~Mon Jul 22 12:00 UTC 2019 | 06:08 | |
| *** ChanServ changes topic to "Due to a failure on the logs.openstack.org volume, old logs are unavailable while partition is recovered. New logs are being stored. ETA for restoration probably ~Mon Jul 22 12:00 UTC 2019" | 06:08 | |
| *** AJaeger has joined #zuul | 06:18 | |
| *** AJaeger has quit IRC | 06:18 | |
| *** AJaeger has joined #zuul | 06:19 | |
| *** ChanServ changes topic to "Discussion of the project gating system Zuul | Website: https://zuul-ci.org/ | Docs: https://zuul-ci.org/docs/ | Source: https://git.zuul-ci.org/ | Channel logs: http://eavesdrop.openstack.org/irclogs/%23zuul/ | Weekly updates: https://etherpad.openstack.org/p/zuul-update-email" | 06:27 | |
| -openstackstatus- NOTICE: logs.openstack.org volume has been restored. please report any issues in #openstack-infra | 06:27 | |
| *** pcaruana has joined #zuul | 06:32 | |
| *** smyers has quit IRC | 06:33 | |
| *** jpena has joined #zuul | 06:34 | |
| *** smyers has joined #zuul | 06:52 | |
| *** bolg has quit IRC | 06:57 | |
| *** jamesmcarthur has joined #zuul | 07:07 | |
| *** bolg has joined #zuul | 07:18 | |
| *** bolg has quit IRC | 07:21 | |
| *** tosky has joined #zuul | 07:21 | |
| *** jhesketh has quit IRC | 07:22 | |
| *** jhesketh has joined #zuul | 07:24 | |
| *** gtema has joined #zuul | 07:25 | |
| *** jamesmcarthur has quit IRC | 07:26 | |
| *** jamesmcarthur has joined #zuul | 07:26 | |
| *** bolg has joined #zuul | 07:45 | |
| *** jangutter has joined #zuul | 07:52 | |
| *** bolg has quit IRC | 07:54 | |
| *** gtema has quit IRC | 07:56 | |
| *** gtema has joined #zuul | 07:56 | |
| *** jamesmcarthur has quit IRC | 07:56 | |
| *** jamesmcarthur has joined #zuul | 07:58 | |
| *** hashar has joined #zuul | 08:02 | |
| *** yolanda has joined #zuul | 08:06 | |
| *** bkorren has joined #zuul | 08:09 | |
| *** jamesmcarthur has quit IRC | 08:13 | |
| *** threestrands has quit IRC | 08:14 | |
| *** zbr|out is now known as zbr | 08:22 | |
| *** gtema has quit IRC | 08:33 | |
| *** gtema has joined #zuul | 08:34 | |
| *** gtema has quit IRC | 08:38 | |
| *** jpena is now known as jpena|brb | 08:56 | |
| *** mgoddard has quit IRC | 09:01 | |
| *** mgoddard has joined #zuul | 09:04 | |
| *** bhavikdbavishi has quit IRC | 09:11 | |
| *** hashar has quit IRC | 09:17 | |
| *** bolg has joined #zuul | 09:35 | |
| *** jpena|brb is now known as jpena | 09:46 | |
| *** gtema has joined #zuul | 10:07 | |
| *** AshBullock has joined #zuul | 10:08 | |
| *** mgoddard has quit IRC | 10:11 | |
| *** mgoddard has joined #zuul | 10:18 | |
| *** bkorren has quit IRC | 11:03 | |
| openstackgerrit | Sagi Shnaidman proposed zuul/nodepool master: Fix nodepool container failure https://review.opendev.org/672012 | 11:04 |
|---|---|---|
| sshnaidm | tristanC, ^^ | 11:05 |
| *** gtema has quit IRC | 11:10 | |
| *** jpena is now known as jpena|lunch | 11:33 | |
| *** AshBullock has quit IRC | 11:41 | |
| *** hashar has joined #zuul | 12:14 | |
| *** rfolco|rover has joined #zuul | 12:28 | |
| *** rlandy has joined #zuul | 12:31 | |
| *** irclogbot_3 has quit IRC | 12:39 | |
| *** electrofelix has joined #zuul | 12:40 | |
| *** irclogbot_1 has joined #zuul | 12:44 | |
| *** jpena|lunch is now known as jpena | 12:47 | |
| *** sanjayu_ has joined #zuul | 12:59 | |
| *** bjackman_ has joined #zuul | 13:01 | |
| *** bjackman has quit IRC | 13:02 | |
| *** needscoffee is now known as kmalloc | 13:36 | |
| *** jeliu_ has joined #zuul | 13:46 | |
| *** bhavikdbavishi has joined #zuul | 14:00 | |
| *** bhavikdbavishi1 has joined #zuul | 14:04 | |
| *** bhavikdbavishi has quit IRC | 14:05 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 14:05 | |
| *** michael-beaver has joined #zuul | 14:12 | |
| corvus | mordred, tristanC: when either of you are around and have a moment i'd like to ask for some advice about https://review.opendev.org/671912 -- specifically, i know it's terrible, and i was wondering how we should structure the code to load the data file after having loaded the build api result and the manifest. should we put logic like that in the reducer? it seems like it's a good place because it's a FSM | 14:26 |
| corvus | with all the data available to it, but it currently isn't used for anything other than data updates... | 14:26 |
| mordred | corvus: I'm not sure yet - but I'm reading/digesting | 14:30 |
| mordred | corvus: I think probably so - but I feel like I need more coffee in my brain | 14:36 |
| corvus | mordred: yeah -- it seems like a good place, but i don't really know if there's some reason to keep the reducer as "pure" data copying or something... | 14:37 |
| corvus | mordred: the other idea i had was maybe i could rework how that big chain of async methods in fetchBuild to make it more modular so it somehow wasn't calling dispatch automatically, then presumably its component methods would be more resuable... but i'm not sure how that would work, and it might end up being just another event system on top of the one that's there | 14:39 |
| mordred | corvus: yeah - I think I want to hear tristanC's thoughts on this one before we go down that road | 14:41 |
| corvus | mordred: this doesn't bode well for the idea of doing anything in the reducer: https://redux.js.org/basics/reducers#handling-actions | 14:43 |
| tristanC | corvus: mordred: i'm not entirely sure how redux objects are supposed to be organized, i used https://github.com/tootsuite/mastodon/tree/master/app/javascript/mastodon for inspiration | 14:45 |
| tristanC | and most of the reducers only manage one type of REQUEST/SUCCESS/FAIL actions | 14:46 |
| tristanC | thus i would add a new log reducer to request the manifest on BUILD_FETCH_REQUEST action | 14:47 |
| tristanC | oh, and reducer should only manage state update, doing the query and parsing the result should happen in the action using pure function | 14:49 |
| corvus | okay, so i think i need to see how i can rework the async methods in build.js to be reusable for both the build page and the view page | 14:51 |
| corvus | because the build page is going to fetch the API and the manifest, and the view page is going to fetch API, manifest, and logfile. i don't want just want to copy the code for the first two | 14:51 |
| corvus | (though, to be fair, that would probably be the cleanest thing ...) | 14:52 |
| *** bjackman_ has quit IRC | 14:52 | |
| *** jamesmcarthur has joined #zuul | 14:53 | |
| corvus | tristanC, mordred: oh the other question i had -- if we implement this as a separate page, and we do an internal Link to move from the build page to the view page, is all the existing build state data going to remain in the redux store? so if we fetch the API info for the build for the build page, then click on a logfile to show it, is the data about the build still going to be there so we don't have to | 14:56 |
| corvus | fetch it again? or does the transition that happens for an internal link clear that out? | 14:56 |
| tristanC | actually, you can't dispatch new actions from within a reducer, so fetching the manifest and the log should be triggered in the `then` block of the build api query | 14:58 |
| tristanC | corvus: if you use fetchBuildIfNeeded, then the view page will not fetch the build info when coming form the build page | 14:59 |
| tristanC | iirc, all the api response are kept in the store and re-used when possible, except for builds (filtered list) query | 15:00 |
| corvus | tristanC: cool, thanks; i think i have what i need to try again now :) | 15:02 |
| *** jamesmcarthur has quit IRC | 15:03 | |
| *** jamesmcarthur has joined #zuul | 15:04 | |
| openstackgerrit | Sagi Shnaidman proposed zuul/nodepool master: Fix nodepool container failure https://review.opendev.org/672012 | 15:07 |
| *** bolg has quit IRC | 15:07 | |
| *** jamesmcarthur has quit IRC | 15:08 | |
| clarkb | tobiash: tristanC https://review.opendev.org/#/c/671858/ may interest you as well. Updates the xterm.js version (they fixed the reason we capped then I had to chagne a thing to make it work which I am not 100% sure is correct) | 15:22 |
| egustafson | corvus: just circling back with adding a zuul user to the executor container, I added the zuul user and kicked another build and the error (rsync/chown) still appears. | 15:47 |
| openstackgerrit | Merged zuul/nodepool master: Fix nodepool container failure https://review.opendev.org/672012 | 16:01 |
| *** jpena is now known as jpena|off | 16:02 | |
| Shrews | tobiash: hrm, we don't really have a good way in place to test https://review.opendev.org/671704 do we? | 16:17 |
| Shrews | A very large part of me wants to see that logic in openstacksdk | 16:17 |
| Shrews | oh, i see you and mordred already discussed this in #openstack-sdk | 16:19 |
| *** michael-beaver has quit IRC | 16:21 | |
| *** mattw4 has joined #zuul | 16:27 | |
| *** mattw4 has quit IRC | 16:30 | |
| *** mattw4 has joined #zuul | 16:31 | |
| *** bhavikdbavishi has quit IRC | 16:45 | |
| tobiash | Shrews: a solution in openstacksdk would be great as well | 16:50 |
| *** igordc has joined #zuul | 16:54 | |
| *** hashar has quit IRC | 17:09 | |
| *** chandankumar is now known as raukadah | 17:18 | |
| *** panda has quit IRC | 17:25 | |
| *** panda has joined #zuul | 17:26 | |
| *** rlandy has quit IRC | 17:52 | |
| corvus | tristanC, mordred: okay, i think i worked out the right combination of promises and thunks to let me write a method like this: http://paste.openstack.org/show/754728/ | 17:58 |
| corvus | so that would run for the build page, and then we'd do the same thing with one more chained to the end for the logfile for the log viewer page | 17:58 |
| * SpamapS apologizes for non-participation lately. y'all are doing such cool stuff | 17:59 | |
| corvus | (though, actualyl, we don't need the build output for the log viewer, so that kind of modularity is even better | 17:59 |
| corvus | SpamapS: it ebbs and flows for all of us :) | 17:59 |
| tristanC | corvus: that looks good to me | 18:06 |
| tristanC | then let react update/display the content once the store is updated | 18:07 |
| corvus | yep | 18:09 |
| corvus | maybe i can get it to launch the output and manifest requests at the same time | 18:09 |
| mordred | corvus: somethign something async | 18:10 |
| corvus | since they each only depend on the build object | 18:10 |
| *** electrofelix has quit IRC | 18:12 | |
| *** jeliu_ has quit IRC | 18:23 | |
| tristanC | corvus: redux thunk dispatch are already async, thus this should make both request at the same time: http://paste.openstack.org/show/754730/ | 18:41 |
| tristanC | e.g. fetchBuild returns a Promise | 18:42 |
| *** jeliu_ has joined #zuul | 18:53 | |
| egustafson | hi folks, i tried a previous suggestion to add a 'zuul' user on the executor container to get around an rsync issue in the gather log phase (POST) but it did not appear to help. i tried adding a zuul and a genadm (used on the external build node) and I received the same failure. To make it easier to read, I took a screenshot of the output and it appears to be in the 'collect log output' play in the 'fetch-output' role. The | 18:56 |
| egustafson | screenshot is uploaded at: https://imgur.com/a/kqWitOq | 18:56 |
| *** jeliu_ has quit IRC | 18:57 | |
| *** sanjayu_ has quit IRC | 18:58 | |
| *** igordc has quit IRC | 19:27 | |
| *** igordc has joined #zuul | 19:30 | |
| corvus | egustafson: did you try running "zuul-executor keep" and running an rsync manually into that directory? | 19:52 |
| *** igordc has quit IRC | 20:03 | |
| *** EmilienM is now known as EmilienM|pto | 20:05 | |
| egustafson | i did and I picked apart the rsync command, displayed in the screenshot, and it did not appear correct. '-S' appears to specify a sparse file and with arguments so i'm not sure what 'none' represents and '-o' appears to not need arguments as well (Port=22 is its own switch). -o implies preserve owner. Nonetheless, i wanted to say i did manage to get it to complete without error by copying the pub key of executor which the | 20:05 |
| egustafson | genadm user already had to root/.ssh and i also changed the user in the etc_nodepool/nodepool.yaml from 'username: genadm' to 'username: root' | 20:05 |
| *** mattw4 has quit IRC | 20:08 | |
| *** igordc has joined #zuul | 20:08 | |
| *** mattw4 has joined #zuul | 20:08 | |
| *** michael-beaver has joined #zuul | 20:11 | |
| *** jeliu_ has joined #zuul | 20:22 | |
| corvus | tristanC, mordred: if you have a minute to review https://review.opendev.org/671874 -- if we get that merged and into the opendev base jobs, then we can test the WIP stuff against production a bit easier. | 20:41 |
| corvus | (it's not impossible to test now, but it's a little hacky) | 20:41 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: WIP download and display a log file https://review.opendev.org/671912 | 20:42 |
| corvus | tristanC, mordred: ^ i think that is less terrible than what we started with this morning :) if you have other structural suggestions, i'm all ears; otherwise, i'll start working on hooking that up to the tree view | 20:43 |
| *** armstrongs has joined #zuul | 20:44 | |
| *** armstrongs has quit IRC | 20:58 | |
| mordred | corvus: 671874 lgtm | 21:01 |
| *** mattw4 has quit IRC | 21:05 | |
| *** mattw4 has joined #zuul | 21:05 | |
| *** pcaruana has quit IRC | 21:10 | |
| *** hwangbo has joined #zuul | 21:17 | |
| *** mattw4 has quit IRC | 21:20 | |
| SpamapS | Has any thought been given to rotating repo keys? | 21:27 |
| clarkb | iirc that was one of the motivations for tobiash's master key | 21:27 |
| SpamapS | Yeah I remember something about that | 21:27 |
| clarkb | but then that ran into problems with python cryptography not supporting feeding in a seed | 21:27 |
| SpamapS | but at this point, I have a (not really but don't make me explain it)compromised master key. | 21:28 |
| SpamapS | hypothetical: somebody with root access to zuul-scheduler is terminated. | 21:28 |
| clarkb | I think the currently supported method is delete the key(s) then restart the scheduler. Zuul will regenerate all the keys then you can reencrypt stuff | 21:29 |
| clarkb | the reencrypt stuff is a problem regardless when rekeying right? | 21:29 |
| clarkb | otherwise in your hypothetical you'd not be protected | 21:30 |
| *** mattw4 has joined #zuul | 21:32 | |
| clarkb | and in that hypothetical you probably want to change the encrypted content too so ya I don't think you can get away from that | 21:32 |
| corvus | SpamapS: key rotation is stubbed out, but not implemented: https://opendev.org/zuul/zuul/src/branch/master/zuul/lib/keystorage.py#L85 | 21:42 |
| corvus | the idea is that the current system is forward-compatible with that, but no one has written the thing to actually deal with a second key | 21:42 |
| *** mattw4 has quit IRC | 22:10 | |
| *** jeliu_ has quit IRC | 22:12 | |
| *** mattw4 has joined #zuul | 22:13 | |
| SpamapS | corvus: beautiful. That's all I need for now. | 22:17 |
| SpamapS | Just need a plan going forward. | 22:17 |
| SpamapS | We just had to figure out key-rotation in git-crypt .. which is how we encrypt most things... and I was hoping we'd have a better story than what git-crypt has. :-D | 22:17 |
| fungi | i mean, keep in mind that if you need to rotate keys, you may need to rotate the content they encrypted too | 22:30 |
| SpamapS | that's the point. I want to rotate the key when I rotate the content. ;) | 22:30 |
| SpamapS | in some instances (mostly when a high-permission user leaves) | 22:31 |
| fungi | the assumption being you shouldn't reuse the key because it could be known... you need to also assume it could have been used to decrypt/leak secrets which also need replacing | 22:31 |
| fungi | so just rotating the zuul keys isn't really thorough enough for some risk profiles | 22:31 |
| SpamapS | yep | 22:31 |
| fungi | so, yeah, at that point reencrypting is necessary anyway, doing so with a replaced key doesn't really make it more work | 22:32 |
| SpamapS | The zuul part is pretty easy... the other end of those secrets is the hard part. ;) | 22:34 |
| clarkb | fungi: ya that was my point earlier | 22:34 |
| fungi | it's getting late here ;) | 22:35 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: WIP download and display a log file https://review.opendev.org/671912 | 22:49 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: WIP download and display a log file https://review.opendev.org/671912 | 22:49 |
| *** tosky has quit IRC | 22:49 | |
| corvus | but having good rotation in zuul will make that more possible; right now, you might have to shut the whole system down and force-merge content updates or something | 22:51 |
| corvus | with rotation, you could add the new key, update content, and remove the old. you'd still be vulnerabile during that window, but you would be able to rotate through everything (starting with the zuul keys, then going down through what they encrypt) without downtime | 22:52 |
| fungi | yeah, i completely agree. i just think it's our responsibility to remind sysadmins that rotating those keys also implies rotating the sensitive data they encrypted | 22:52 |
| corvus | good idea; whenever we figure out how to actually do rotation, we can incorporate that reminder | 22:53 |
| fungi | and so no point in zuul providing automation to reencrypt your secrets with the new keys | 22:53 |
| fungi | because if you consider the old keys compromised, then you ought to consider the data they were protecting equally compromised | 22:54 |
| corvus | mordred, tristanC: ^ https://review.opendev.org/671912 links from the tree view to the log viewer. it's pretty fast too, because it only fetches whatever data are missing (so if you go to a build page, then click back and forth between several log files, it only fetches the new log files; it doesn't have to reload any build api/manifest data) | 22:54 |
| corvus | fungi: that's a good point too | 22:55 |
| SpamapS | corvus:right, what I want is the multiple-active-keys workflow. That way once we've confirmed denial of access to the zuul-scheduler and underlying storage of keys, we can make a new one, and incrementally re-encrypt things to the new key without breaking the world. | 23:00 |
| corvus | next steps for logs are: 1) review https://review.opendev.org/671874 and get it merged so we have real data so we can use the build preview to see the system working; 2) start adding in some OSLA-style htmlification | 23:00 |
| SpamapS | Also there's "I need to change the key for a compromise I know about" and there's also "I need to rotate the keys so I can limit the exposure of accidental compromise/exposure." | 23:01 |
| SpamapS | The former is totally fine to do with force pushes and disruption. Part of life. But the latter should be a non-event. | 23:01 |
| *** threestrands has joined #zuul | 23:42 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!