| *** jesusaur has quit IRC | 00:00 | |
| *** jesusaur has joined #zuul | 00:11 | |
| manjeets | ianw, still there one more question https://zuul-ci.org/docs/zuul/admin/nodepool_static.html is host-key a private key for which public should be node authorized_keys in static_node ? | 00:51 |
|---|---|---|
| manjeets | should be added to authorized_keys in static_node that;s being added to nodepool ? | 00:52 |
| ianw | manjeets: host-key will be the host key of the server connecting to | 00:53 |
| ianw | zuul will authenticate as the zuul user (or whatever you're running as) so the authorized_keys you want on the static node is the public key of the zuul user's private key | 00:54 |
| ianw | does that make sense? | 00:54 |
| manjeets | ianw, zuul user is actually on static target node we add here https://zuul-ci.org/docs/zuul/admin/nodepool_static.html? | 00:57 |
| manjeets | it login as username:zuul or whatever i give to ip given in name: field ? | 00:58 |
| manjeets | it logs into static node from somewhere in one of the containers ? so public key of privatekey for that container should be added ? | 00:59 |
| ianw | manjeets: yes, the zuul executor logs in to the static node, so it's public key needs to be in the authorized_keys of the static node | 01:00 |
| manjeets | what is this host-key in https://zuul-ci.org/docs/zuul/admin/nodepool_static.html ? sorry i still don't get it ? | 01:01 |
| ianw | manjeets: that's the, well host key of the remote host | 01:05 |
| ianw | you know when you log into a new machine, and it prompts you "do you want to trust this" and gives you the fingerprint | 01:05 |
| ianw | that | 01:05 |
| manjeets | ohk got it thanks ! | 01:06 |
| manjeets | I screwed previous run by thinking its sorta public key | 01:06 |
| manjeets | dumb I | 01:06 |
| manjeets | thanks ianw | 01:06 |
| *** rlandy is now known as rlandy|biab | 01:09 | |
| *** openstackgerrit has quit IRC | 02:22 | |
| *** rlandy|biab is now known as rlandy | 02:45 | |
| *** rlandy has quit IRC | 02:45 | |
| *** bhavikdbavishi has joined #zuul | 04:10 | |
| *** bhavikdbavishi1 has joined #zuul | 04:22 | |
| *** bhavikdbavishi has quit IRC | 04:23 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 04:23 | |
| *** bhavikdbavishi has joined #zuul | 04:24 | |
| *** chandankumar has joined #zuul | 04:33 | |
| *** bhavikdbavishi has quit IRC | 04:34 | |
| *** bhavikdbavishi has joined #zuul | 04:35 | |
| *** chandankumar is now known as chkumar|ruck | 04:36 | |
| *** bhavikdbavishi1 has joined #zuul | 04:54 | |
| *** bhavikdbavishi has quit IRC | 04:54 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 04:54 | |
| *** bhavikdbavishi has quit IRC | 04:59 | |
| *** bjackman has joined #zuul | 05:09 | |
| *** ianychoi_ has joined #zuul | 06:30 | |
| *** ianychoi has quit IRC | 06:33 | |
| *** bhavikdbavishi has joined #zuul | 06:55 | |
| *** bhavikdbavishi has quit IRC | 07:58 | |
| *** quiquell has joined #zuul | 08:19 | |
| quiquell | Good morning | 08:19 |
| quiquell | fungi: You there ? | 08:19 |
| fungi | quiquell: not really, packing up and about to go catch a bus to the airport | 08:21 |
| quiquell | fungi: ack, next week then, have a good trip !!! | 08:22 |
| *** openstackgerrit has joined #zuul | 08:28 | |
| openstackgerrit | Quique Llorente proposed openstack-infra/zuul master: Setup model before connection https://review.openstack.org/618484 | 08:28 |
| *** pcaruana has joined #zuul | 08:39 | |
| *** bhavikdbavishi has joined #zuul | 08:41 | |
| *** jpena|off is now known as jpena | 08:45 | |
| *** bhavikdbavishi has quit IRC | 08:51 | |
| *** hashar has joined #zuul | 08:53 | |
| *** panda|rover|off is now known as panda | 09:03 | |
| *** chkumar|ruck has quit IRC | 09:04 | |
| *** bjackman has quit IRC | 09:13 | |
| *** chandankumar has joined #zuul | 09:15 | |
| *** chandankumar is now known as chkumar|ruck | 09:27 | |
| *** ParsectiX has joined #zuul | 09:35 | |
| *** ParsectiX has quit IRC | 09:37 | |
| *** bjackman has joined #zuul | 09:38 | |
| *** panda is now known as panda|rover | 09:39 | |
| *** ParsectiX has joined #zuul | 09:49 | |
| *** openstackgerrit has quit IRC | 09:52 | |
| *** dkehn has quit IRC | 09:52 | |
| *** dkehn_ has joined #zuul | 09:52 | |
| *** goern has quit IRC | 09:53 | |
| *** lennyb has quit IRC | 09:53 | |
| *** lennyb has joined #zuul | 09:54 | |
| *** gtema has joined #zuul | 09:59 | |
| *** ParsectiX has quit IRC | 10:00 | |
| *** goern has joined #zuul | 10:05 | |
| fungi | quiquell: i have a few minutes now that i'm through airport security. what was your question? | 10:10 |
| quiquell | fungi: Just checking ci of the patch about makedirs .ssh | 10:11 |
| quiquell | fungi: I think the failure there is not related, but don't know for sure | 10:11 |
| quiquell | fungi: At gerritconfig.log "Invalid SSH Key\n" adding admin ssh key | 10:12 |
| quiquell | http://logs.openstack.org/84/617984/6/check/zuul-quick-start/4fe60cf/container_logs/gerritconfig.log | 10:12 |
| quiquell | fungi: but the change is at merger.py I think it's not run there | 10:12 |
| quiquell | fungi: but maybe it mess up the .ssh dir | 10:12 |
| fungi | quiquell: yes, i don't think they're related either based on which job was impacted, but i didn't have an opportunity to dig into the logs from the failure | 10:12 |
| fungi | quiquell: i also just noticed corvus pushed up basically the same change two days ago https://review.openstack.org/617955 | 10:15 |
| quiquell | fungi: Then we just need to merge that, I also refactor working_root stuff | 10:16 |
| fungi | which i've approved now, but i have a feeling it's going to get a verify -2 for failing quickstart job in the gate | 10:16 |
| fungi | starting to dig into what has regressed that job now | 10:17 |
| quiquell | fungi: The difference with corvus is the working_dir, corvus's is passing | 10:17 |
| quiquell | working_root I mean | 10:17 |
| *** ParsectiX has joined #zuul | 10:18 | |
| fungi | ahh, true | 10:18 |
| *** ParsectiX has quit IRC | 10:23 | |
| fungi | the failure looks like zuul-web is returning a 404 not found for requests to /a/accounts/zuul/sshkeys | 10:33 |
| fungi | yeah, as suspected, 617955 is failing in the gate pipeline on the same job | 10:42 |
| fungi | in the same way | 10:43 |
| fungi | so this is likely a regression which has cropped up in the past two days since that change passed in check | 10:44 |
| fungi | but i'm about to board my flight now so need to disconnect. i can try to pick the shovel up again later if nobody else gets a chance | 10:45 |
| *** electrofelix has joined #zuul | 10:46 | |
| *** ParsectiX has joined #zuul | 10:49 | |
| fbo | hi, is there a way to tell Zuul to read variables from some yml files via the job definition ? I have some variables files I want to expose to my job and the files are in a required-project and I don't want to duplicate using job.vars. | 11:11 |
| clarkb | fbo you can use include_vars | 11:14 |
| fbo | clarkb: but include_vars is executed from the executor and seems bubblewrap deny the access to the vars file I expect to read. | 11:18 |
| clarkb | hrm I thought we usee it in places for distro specific flags | 11:20 |
| clarkb | but maybe I am mistaken | 11:20 |
| fbo | Well I need to check again my include_vars path might be wrong | 11:21 |
| *** hashar has quit IRC | 11:32 | |
| quiquell | fungi: ahh so it's regression | 11:42 |
| *** dkehn_ has quit IRC | 11:44 | |
| *** gtema has quit IRC | 11:48 | |
| *** quiquell is now known as quiquell|brb | 11:53 | |
| quiquell|brb | clarkb: Do you know what options at zuul can help up speed thinks up regarding cloning repo and the like ? | 11:56 |
| quiquell|brb | clarkb: Feels like recloning everythign after consequitive changes in the same review | 11:56 |
| *** ParsectiX has quit IRC | 12:10 | |
| *** EmilienM is now known as EvilienM | 12:24 | |
| *** quiquell|brb is now known as quiquell | 12:25 | |
| *** jpena is now known as jpena|lunch | 12:29 | |
| *** gtema has joined #zuul | 12:32 | |
| *** hashar has joined #zuul | 12:42 | |
| tobiash | fbo: include_vars should work if you use a safe path | 12:48 |
| fbo | tobiash: yes I figured out my mistake, works now :) | 12:50 |
| tobiash | :) | 12:50 |
| tobiash | quiquell: what do you mean with recloning? The executor clones from a local cache dir | 12:51 |
| quiquell | tobiash: I don't really know what I mean exactly :-) | 12:52 |
| tobiash | (using hard links if the cached repo and workspace are on the same filesystem) | 12:52 |
| quiquell | tobiash: will do some investigations first to come with more specifics | 12:52 |
| tobiash | Ok | 12:52 |
| quiquell | tobiash: How is this hardlink thing ? do we have a role or something to do that already ? | 12:53 |
| *** bjackman has quit IRC | 12:55 | |
| *** mordred has quit IRC | 12:55 | |
| *** chkumar|ruck has quit IRC | 12:56 | |
| *** ParsectiX has joined #zuul | 13:11 | |
| quiquell | tobiash: The 'console.log' zuul streams is from executor ? | 13:23 |
| *** quiquell is now known as quiquell|lunch | 13:32 | |
| *** quiquell|lunch is now known as quiquell | 13:35 | |
| *** jpena|lunch is now known as jpena | 13:37 | |
| quiquell | I have the zuul executor running at docker and it get stuck I see playbook progressing in the node but not the executor | 13:42 |
| *** rlandy has joined #zuul | 13:46 | |
| *** quiquell is now known as quiquell|lunch | 14:09 | |
| *** ParsectiX has quit IRC | 14:13 | |
| fungi | quiquell|lunch: the "hard linking" in question is simply git clone's default behavior if you give it a source file path instead of a remote url and the destination doesn't traverse mountpoints from the source | 14:16 |
| fungi | it creates hardlinks for all the objects in the tree rather than copying them into new files | 14:16 |
| quiquell|lunch | fungi: nice! will try thanks | 14:18 |
| *** ParsectiX has joined #zuul | 14:19 | |
| fungi | yeah, in theory it's nearly instantaneous since it only has to write out the additional inode entries | 14:28 |
| fungi | er, handles for them i mean | 14:28 |
| quiquell|lunch | fungi: That's going to speed thinks up, the other issue I face is executor get stuck | 14:29 |
| quiquell|lunch | fungi: I see the playbook finishing correctly at the node | 14:29 |
| quiquell|lunch | fungi: but console.log stream is stuck and also job-output.txt at the bubblewrap thingy | 14:29 |
| quiquell|lunch | fungi: And one of the task in the middle (long one) | 14:29 |
| quiquell|lunch | Maybe I have to configure some timeout | 14:30 |
| *** quiquell|lunch is now known as quiquell | 14:30 | |
| *** toabctl has quit IRC | 14:31 | |
| Diabelko | corvus: is there any common place where Zuul publications/presentations land? | 14:33 |
| Diabelko | I faintly recall someone mentioning something like it on the mailing list, but I do not remember the conclusion at all | 14:34 |
| *** toabctl has joined #zuul | 14:34 | |
| fungi | i wonder if we could/should commit them to the zuul-website-media repo | 14:35 |
| quiquell | fungi: But the clone stuff is executed by zuul code and it does not clone from cached repo | 14:38 |
| *** ianychoi_ is now known as ianychoi | 14:41 | |
| fungi | are you talking about on the executor or on a job node? | 14:43 |
| fungi | i think on the executor it does have to copy from the cache instead of linking, for safety reasons, but i'm not in a position to confirm that right now | 14:45 |
| quiquell | fungi: in node I think | 14:46 |
| fungi | the executor needs to create the repository state and then push it onto the job node. zuul itself doesn't perform any cloning i'm aware of on job nodes (though your job definitions may do so) | 14:47 |
| quiquell | fungi: ok, will investigate | 14:48 |
| fungi | quiquell: the role that does the bit i was describing to push the workspace copy onto job nodes is https://git.openstack.org/cgit/openstack-infra/zuul-jobs/tree/roles/mirror-workspace-git-repos/tasks/main.yaml | 14:52 |
| fungi | er, i mean https://git.zuul-ci.org/cgit/zuul-jobs/tree/roles/mirror-workspace-git-repos/tasks/main.yaml | 14:53 |
| quiquell | fungi: thanks so much, want to fix the stuck issue first | 14:58 |
| *** ParsectiX has quit IRC | 15:17 | |
| *** ParsectiX has joined #zuul | 15:18 | |
| *** ParsectiX has quit IRC | 15:23 | |
| *** ParsectiX has joined #zuul | 15:26 | |
| quiquell | fungi: Damn I don't know why my executor get stuck, I see tasks running at node | 15:35 |
| quiquell | fungi: but executor is stuck at one of them like forever | 15:35 |
| quiquell | fungi: Could be related to socket to run shell at cloud ? | 15:36 |
| quiquell | fungi: Maybe I have some timeouts at my cloud that closes the socket | 15:37 |
| quiquell | fungi: after some inactivity | 15:37 |
| *** pcaruana has quit IRC | 15:56 | |
| rlandy | hello ... at what point does zuul apply zuul.change (ref) to the repos on the nodes? https://github.com/openstack-infra/zuul-jobs/blob/master/roles/mirror-workspace-git-repos/tasks/main.yaml#L35 looks like it checks out master | 16:00 |
| *** ParsectiX has quit IRC | 16:10 | |
| *** quiquell is now known as quiquell|off | 16:13 | |
| *** j^2 has joined #zuul | 16:17 | |
| rlandy | fungi: ^^ hi - can you point me to where the zuul change gets applied? | 16:25 |
| *** hashar has quit IRC | 16:32 | |
| *** hashar has joined #zuul | 16:34 | |
| *** openstackgerrit has joined #zuul | 16:37 | |
| openstackgerrit | Markus Hosch proposed openstack-infra/zuul master: Allow to select the merge method in GitHub https://review.openstack.org/617949 | 16:37 |
| *** electrofelix has quit IRC | 16:56 | |
| *** mordred has joined #zuul | 17:20 | |
| SpamapS | quiquell|off: if you could look at what's running under the executor process... maybe 'docker exec zuul-executor-container ps auxfw' ... it should be running bwrap -> ansible-playbook | 17:20 |
| SpamapS | rlandy: zuul prepares all projects' git trees in the executor | 17:21 |
| SpamapS | rlandy: and then https://git.openstack.org/cgit/openstack-infra/zuul-jobs/tree/roles/mirror-workspace-git-repos/tasks/main.yaml sends the prepared git workspaces to job nodes | 17:22 |
| rlandy | SpamapS: ok - this task? https://git.openstack.org/cgit/openstack-infra/zuul-jobs/tree/roles/mirror-workspace-git-repos/tasks/main.yaml#n9 | 17:24 |
| SpamapS | rlandy: right, that's what copies the git states as they are from executor -> nodes | 17:25 |
| SpamapS | the git state is updated inside the executor here.... https://github.com/openstack-infra/zuul/blob/master/zuul/executor/server.py#L2285-L2303 | 17:26 |
| SpamapS | and most of the deep stuff happens in updateRepo in the merger.. https://github.com/openstack-infra/zuul/blob/master/zuul/merger/merger.py#L511-L563 | 17:28 |
| rlandy | SpamapS: hmmm ... I am following a local run and I see u'checkout': u'master' on that step - not the change branch I was looking for | 17:28 |
| SpamapS | So, if the change is stacked on top of master, that's normal. | 17:29 |
| SpamapS | You'll checkout master, and then merge the change on top of it. | 17:29 |
| SpamapS | https://github.com/openstack-infra/zuul/blob/master/zuul/merger/merger.py#L602 | 17:29 |
| SpamapS | In fact you won't realy "merge" at that stage, you'll just checkout the ref. | 17:30 |
| SpamapS | the merge happens at a different point | 17:30 |
| rlandy | ok - let me rerun and I'll watch again - thank you | 17:31 |
| SpamapS | rlandy: anyway, when the scheduler enqueues your change, it adds change info to the job, and the executor will definitely only run the job on the change that is referred to. I've never seen that not work as expected. | 17:31 |
| SpamapS | It's worth noting that the way a job is triggered, what type of pipeline it is, and a few other things, all will affect what git states get sent out. | 17:32 |
| SpamapS | Like, if you trigger on a commit, not a change (for instance, in a post-merge pipeline), that won't have a zuul.change. | 17:33 |
| SpamapS | Or if you are in a 'dependent' pipeline, there may be multiple changes merged before yours. | 17:33 |
| SpamapS | Anyway, just to say: please don't hesitate to ask here if you're not sure what you're seeing. There are a lot of factors. | 17:33 |
| *** gtema has quit IRC | 17:35 | |
| *** jpena is now known as jpena|off | 17:46 | |
| *** quiquell|off has quit IRC | 17:51 | |
| *** hashar has quit IRC | 17:53 | |
| *** ParsectiX has joined #zuul | 17:58 | |
| rlandy | SpamapS: thanks - that info was very helpful | 18:03 |
| rlandy | I see the ref now ... | 18:05 |
| rlandy | 2018-11-16 00:28:10.956994 | secondary -> localhost | To git+ssh://zuul@198.72.124.122//home/zuul/src/git.openstack.org/openstack-infra/tripleo-ci | 18:05 |
| rlandy | 2018-11-16 00:28:10.957428 | secondary -> localhost | 803a036..ce9ab51 master -> master | 18:05 |
| *** ParsectiX has quit IRC | 18:14 | |
| manjeets | hi guys, I'm trying to locate logs for nodepool services (logs like if executor ever sshed to machine allocated from nodepool) etc, I'm trying to run a job on static baremetal machine bu using ip and username etc. but dont see any error happen on where all the zuul services were executed, but I get NODE_FAILURE on job | 18:29 |
| manjeets | so where exactly in examples_logs container I can get nodepool specific logs (like was the machine ever got allocated) or ssh success etc | 18:30 |
| SpamapS | manjeets: nodepool's logs are usually configured with a logging.conf... are you using the docker compose quick start? | 18:31 |
| manjeets | SpamapS, yes using docker compose | 18:31 |
| *** rfolco has quit IRC | 18:31 | |
| SpamapS | manjeets: ok, the container that it spins up for "logs" is for the job logs, not zuul and nodepool component logs | 18:33 |
| manjeets | ohk but where can I find nodepool, zuul logs, Its streaming on machine I'm running but hard to locate error from there like NODE_FAILURE | 18:34 |
| SpamapS | manjeets: did you try just 'docker logs launcher' ? | 18:35 |
| manjeets | SpamapS, you mean zuul/nodepool-launcher ? | 18:36 |
| SpamapS | oh, right | 18:36 |
| * SpamapS hasn't used it, I'm spinning it up for the first time right now actually ;) | 18:36 | |
| manjeets | SpamapS, I found some logs for nodepool launcher | 18:39 |
| manjeets | thanks ! | 18:39 |
| SpamapS | np | 18:40 |
| manjeets | SpamapS, from logs I figured out there was host-key mismatch error, I think it could be bug in upstream the way nodepool set up things | 19:31 |
| manjeets | so I changed the host-key in nodepool.yaml and kill docker-compose run and re ran docker-compose up | 19:32 |
| manjeets | since host-key for static node was right it was still matching with wrong key (I guess one stored in db or somewhere which older') | 19:32 |
| manjeets | then I killed all the containers, and images and rebuilt then error was gone | 19:33 |
| SpamapS | manjeets: I haven't done much with static nodes, but that sounds plausible. If nothing else it may be confusing. | 19:38 |
| *** rlandy has quit IRC | 20:47 | |
| *** panda|rover is now known as panda|rover|off | 21:03 | |
| *** goern has quit IRC | 21:17 | |
| *** j^2 has quit IRC | 21:18 | |
| manjeets | Hi SpamapS quick question on ssh private key path in executor config section in zuul.conf | 21:43 |
| manjeets | so private_key_path here is private key in executor container whose public key should be added to static node (added for jobs) ?) | 21:44 |
| SpamapS | manjeets: correct | 21:45 |
| manjeets | SpamapS, after lots of error I was able to use a static baremetal node and execute a sample command as part of test using that pipeline | 22:14 |
| manjeets | its cool ! thanks for you help | 22:14 |
| SpamapS | manjeets: congrats! | 22:43 |
| *** EvilienM is now known as EmilienM | 22:55 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!