| ianw | ok, it must have been similar enough for me to be confused ;) i think if have them both as something obviously different that's great | 00:01 |
|---|---|---|
| dmsimard | ianw: for the key, I'm thinking we can probably set it up in an unrelated location -- it's just for the duration of the installation | 00:04 |
| dmsimard | As long as the location of the key matches in the .repo file | 00:04 |
| dmsimard | I was renaming the key but it felt... weird | 00:05 |
| dmsimard | Like, we could put it in /tmp and it wouldn't matter | 00:05 |
| ianw | that seems fine too | 00:06 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul-jobs master: Vendor the RDO repository configuration for installing OVS https://review.openstack.org/624817 | 00:18 |
| dmsimard | clarkb, ianw: ^ this sets up the key temporarily in /tmp/ to ensure it doesn't clash with the key installed by packages | 00:18 |
| *** dkehn has joined #zuul | 01:42 | |
| *** dkehn has left #zuul | 01:42 | |
| *** dkehn has joined #zuul | 01:42 | |
| *** ssbarnea|rover has quit IRC | 02:06 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/nodepool master: Switch devstack jobs to Xenial https://review.openstack.org/624855 | 03:47 |
| openstackgerrit | Ian Wienand proposed openstack-infra/nodepool master: Switch devstack jobs to Xenial https://review.openstack.org/624855 | 03:48 |
| openstackgerrit | Ian Wienand proposed openstack-infra/nodepool master: Switch devstack jobs to Xenial https://review.openstack.org/624855 | 03:51 |
| *** bhavikdbavishi has joined #zuul | 04:16 | |
| *** swest has joined #zuul | 05:42 | |
| *** swest has quit IRC | 05:48 | |
| *** swest has joined #zuul | 05:50 | |
| *** swest has quit IRC | 05:55 | |
| *** bhavikdbavishi has quit IRC | 06:00 | |
| *** swest has joined #zuul | 06:12 | |
| *** quiquell|off is now known as quiquell | 06:59 | |
| *** bhavikdbavishi has joined #zuul | 06:59 | |
| *** rlandy|bbl has quit IRC | 06:59 | |
| *** smyers has quit IRC | 07:03 | |
| *** bhavikdbavishi has quit IRC | 07:12 | |
| *** pcaruana has joined #zuul | 07:12 | |
| *** smyers has joined #zuul | 07:25 | |
| *** hashar has joined #zuul | 08:02 | |
| *** goern has joined #zuul | 08:09 | |
| *** Alvass has joined #zuul | 08:23 | |
| *** themroc has joined #zuul | 08:28 | |
| Alvass | Hi, I'm having some problems with nodepool configuration when trying to use winrm for the host | 08:42 |
| Alvass | the zuul-executor seem to crash in server.py, row 1262 with 'NoneType is not iterable'. When trying to get the host key to set know_hosts for ssh. Not sure but this seems like a bug. | 08:44 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: refactor build page to use a reducer https://review.openstack.org/624894 | 08:55 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: refactor build page using a container https://review.openstack.org/624895 | 08:55 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add errors from the job-output to the build page https://review.openstack.org/624896 | 08:55 |
| *** jpena|off is now known as jpena | 09:04 | |
| tristanC | mordred: corvus: using the "local" tenant of the zuul-build-dashboard-multi-tenant shows the failed task output on the build page of ^ To make this work with openstack, we'll need cors header from the logserver. | 09:10 |
| tristanC | actually, the zuul-build-dashboard also works with logs.openstack.org because it's already serving from there | 09:20 |
| *** gtema has joined #zuul | 09:25 | |
| *** ssbarnea|rover has joined #zuul | 09:34 | |
| *** sshnaidm has quit IRC | 09:37 | |
| *** ssbarnea|rover has quit IRC | 09:53 | |
| *** bhavikdbavishi has joined #zuul | 09:54 | |
| *** gtema has quit IRC | 10:01 | |
| *** electrofelix has joined #zuul | 10:04 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add errors from the job-output to the build page https://review.openstack.org/624896 | 10:16 |
| *** sshnaidm has joined #zuul | 10:20 | |
| *** bhavikdbavishi has quit IRC | 10:34 | |
| *** ssbarnea|rover has joined #zuul | 10:43 | |
| *** sshnaidm has quit IRC | 10:50 | |
| *** hashar has quit IRC | 11:07 | |
| *** dkehn has quit IRC | 11:17 | |
| *** Alvass has quit IRC | 11:17 | |
| *** hashar has joined #zuul | 11:25 | |
| *** tobias-urdin is now known as tobias-urdin_afk | 11:41 | |
| *** tobias-urdin_afk is now known as tobias-urdin | 11:42 | |
| *** tobias-urdin is now known as tobias-urdin_afk | 11:43 | |
| *** dkehn has joined #zuul | 12:04 | |
| *** sshnaidm has joined #zuul | 12:12 | |
| *** bhavikdbavishi has joined #zuul | 12:20 | |
| *** gtema has joined #zuul | 12:36 | |
| *** tobias-urdin_afk is now known as tobias-urdin | 12:53 | |
| *** gtema has quit IRC | 13:00 | |
| *** jpena is now known as jpena|lunch | 13:01 | |
| *** rlandy has joined #zuul | 13:03 | |
| *** themroc has quit IRC | 13:05 | |
| *** themroc has joined #zuul | 13:06 | |
| *** themroc has quit IRC | 13:06 | |
| ssbarnea|rover | how can I find what happens with zuul on a job? it seems stuck at https://zuul.openstack.org/stream/0dc38407c7c94063afc53ad126cfe932?logfile=console.log -- and 624381 seems stuck | 13:12 |
| ssbarnea|rover | this was supposed to be last task on post, but even after 40mins, zuul is still waiting for it to finish | 13:13 |
| *** hashar has quit IRC | 13:18 | |
| *** bhavikdbavishi has quit IRC | 13:25 | |
| pabelanger | ssbarnea|rover: which review was that for, it is done now, but if nothing is showing in logs a zuul admin may need to look at executor logs | 13:57 |
| *** irclogbot_3 has quit IRC | 14:02 | |
| ssbarnea|rover | pabelanger: this is serious problem because we have high number of POST_FAILUREs, and we assumed that something wrong was happening with "du", and this was an attempt to fix it. Guess what, it didn't and my impression is that may not be related to that. | 14:03 |
| ssbarnea|rover | pabelanger: the behaviours was always the same: post was failing and last thing seen was this du command which happens to be the last command in the last task. | 14:03 |
| ssbarnea|rover | now I am 100% sure that du executes before that and zuul gets stuck. | 14:04 |
| ssbarnea|rover | the mentioned bug has a link to logstash: http://logstash.openstack.org/#/dashboard/file/logstash.json?query=(message:%20%5C%22FAILED%20with%20status:%20137%5C%22%20OR%20message:%20%5C%22FAILED%20with%20status:%20143%5C%22%20OR%20message:%20%5C%22POST-RUN%20END%20RESULT_TIMED_OUT%5C%22)%20AND%20tags:%20%5C%22console%5C%22%20AND%20voting:1 | 14:06 |
| *** smyers has quit IRC | 14:08 | |
| *** hashar has joined #zuul | 14:08 | |
| ssbarnea|rover | i checked and I was not able to find any builder, branch or job pattern. it just happens, and often enough to be of concern. 177 counts in one week. | 14:09 |
| *** irclogbot_3 has joined #zuul | 14:12 | |
| *** smyers has joined #zuul | 14:12 | |
| pabelanger | ssbarnea|rover: lets move to #openstack-infra to debug | 14:13 |
| ssbarnea|rover | sure, good idea. i was not sure where it fits. | 14:13 |
| *** swest has quit IRC | 14:23 | |
| *** albin_vass has joined #zuul | 14:47 | |
| *** quiquell is now known as quiquell|lunch | 14:58 | |
| *** panda|off is now known as panda | 14:58 | |
| *** quiquell|lunch is now known as quiquell | 15:10 | |
| *** andreykurilin has joined #zuul | 15:21 | |
| andreykurilin | hi folks! Can someone help me to solve the issue? I'm trying to access the variable defined by 'parent' job, but i'm getting strange error | 15:22 |
| pabelanger | sure, if you want to link your log file | 15:23 |
| andreykurilin | https://review.openstack.org/#/c/624409/5/tests/ci/playbooks/run-rally-task.yaml (originally, I tried "{{ devstack_localrc.get('USE_PYTHON3') or False }}", but simplified this temporary) | 15:23 |
| andreykurilin | and here is a usage of my local variable https://review.openstack.org/#/c/624409/3/tests/ci/playbooks/roles/snapshot-available-os-resources/tasks/main.yaml | 15:24 |
| andreykurilin | error: http://logs.openstack.org/09/624409/5/check/rally-task-simple-job/1971602/job-output.txt.gz#_2018-12-13_15_00_15_671511 | 15:24 |
| andreykurilin | ps: I do not need a local variable and would like to use directly the parent one from my roles, but it raised the same issue and I tried to play with accessing it | 15:25 |
| pabelanger | andreykurilin: lets move this to #openstack-infra | 15:29 |
| andreykurilin | ok | 15:29 |
| andreykurilin | pabelanger: it looks like I cannot send messages there which is strange since I have registered nick. hm | 15:32 |
| pabelanger | andreykurilin: ah, yes. you need to register nick with freenode | 15:33 |
| pabelanger | it is needed now to cut down on spam | 15:33 |
| pabelanger | but, thought it was also enabled here | 15:33 |
| andreykurilin | pabelanger: i already have registered nick. strange, will recheck everything | 15:36 |
| *** albin_vass has quit IRC | 16:02 | |
| *** jpena|lunch is now known as jpena | 16:08 | |
| *** morgan is now known as kmalloc | 16:09 | |
| *** andreykurilin has quit IRC | 16:12 | |
| *** andreykurilin has joined #zuul | 16:12 | |
| *** quiquell is now known as quiquell|off | 16:22 | |
| *** hashar has quit IRC | 16:50 | |
| dmsimard | clarkb: btw I just did a recheck on https://review.openstack.org/#/c/624526/ for https://review.openstack.org/#/c/624817/ | 16:50 |
| clarkb | dmsimard: thanks | 16:51 |
| clarkb | dmsimard: if the tripleo jobs come back happy I think we should merge your change. That should be a decent reliability improvement | 17:08 |
| *** hashar has joined #zuul | 17:09 | |
| *** hashar has quit IRC | 17:14 | |
| *** arxcruz is now known as arxcruz|off|next | 17:34 | |
| *** arxcruz|off|next is now known as arxcruz|next_yea | 17:34 | |
| *** arxcruz|next_yea is now known as arxcruz|next_yr | 17:35 | |
| *** ianychoi has quit IRC | 17:42 | |
| *** panda is now known as panda|off | 17:42 | |
| *** hashar has joined #zuul | 17:51 | |
| *** sshnaidm is now known as sshnaidm|off | 18:01 | |
| *** rlandy is now known as rlandy|biab | 18:09 | |
| *** hashar has quit IRC | 18:22 | |
| *** electrofelix has quit IRC | 18:34 | |
| *** jpena is now known as jpena|off | 18:38 | |
| *** electrofelix has joined #zuul | 18:44 | |
| *** electrofelix has quit IRC | 18:51 | |
| SpamapS | I dare you to read this Ansible task and not get a little angry at the universe in general for having made me do this: http://paste.openstack.org/show/737251/ | 19:21 |
| *** rlandy|biab is now known as rlandy | 19:23 | |
| dmsimard | SpamapS: that would be more readable in python :/ | 19:43 |
| *** pcaruana has quit IRC | 19:47 | |
| *** hashar has joined #zuul | 19:48 | |
| *** hashar has quit IRC | 19:48 | |
| SpamapS | dmsimard: yep | 19:55 |
| SpamapS | But it shouldn't have to be like this | 19:55 |
| SpamapS | it's just "put this map in this other map at this location" | 19:55 |
| SpamapS | Sometimes I wish there was a more lightweight "put python here". | 19:56 |
| *** gtema has joined #zuul | 20:12 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Extract out common config parsing for ConfigPool https://review.openstack.org/621642 | 20:23 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Extract common config parsing for ProviderConfig https://review.openstack.org/625094 | 20:23 |
| *** ianychoi has joined #zuul | 20:59 | |
| *** gtema has quit IRC | 21:11 | |
| *** tobias-urdin has quit IRC | 21:32 | |
| SpamapS | Hm I just had something super weird happen. | 22:39 |
| SpamapS | Landed changes to my post jobs.. | 22:39 |
| SpamapS | but it's still running the old ones on commit to that branch. | 22:39 |
| SpamapS | merger fail? | 22:39 |
| SpamapS | Like, it's still running the old playbooks, the old content, despite a new sha. | 22:40 |
| clarkb | SpamapS: or an ordering/race problem. If the trigger for post is processed before the code is merged event that updates config it may kick off jobs with the old config? | 22:40 |
| clarkb | hrm I would've expect the merger to pull in the up to date playbooks in that case though | 22:41 |
| clarkb | (since it should be running against $sha1 with thoseplaybooks) | 22:41 |
| SpamapS | It has happened twice now. | 22:41 |
| SpamapS | I was sort of in the "oh maybe.. sequence..." but now.. | 22:42 |
| clarkb | are there config loading errors? that may possibly force it to fallback to the last successfully loaded config | 22:42 |
| clarkb | (though I think it would actually result in jobs not running instead) | 22:43 |
| SpamapS | You can see logs and stuff here https://zuul.gdmny.co/builds | 22:43 |
| SpamapS | (though the sha's are in private repos so you can't see those) | 22:43 |
| SpamapS | no, gate landed the changes. | 22:43 |
| clarkb | no alarm bell for broken configs | 22:43 |
| clarkb | SpamapS: in your job logs, there should be the inventory file that zuul logged for you (this comes from our base jobs though so you may not have that I guess). In that inventory should be the job inheritance path for that job | 22:44 |
| clarkb | SpamapS: does that offer any clues like maybe the job is actually defined elsewhere | 22:44 |
| SpamapS | I do have the inventory file | 22:44 |
| clarkb | one thing that can happen like for tag events is those don't map to branches so if you have multiple branches with overlapping config for those pipelines zuul gets confused and runs the first found iirc | 22:45 |
| clarkb | I don't think this is uspposed to be an issue for post since post should be able to be associated with a branch. But maybe that is github vs gerrit difference | 22:45 |
| SpamapS | http://paste.openstack.org/show/737271/ | 22:45 |
| SpamapS | and I have `branches: master` in that job | 22:46 |
| clarkb | that may confuse it in post depending on the available branch context data on that trigger (I have no idea if that works with the github triggers) | 22:46 |
| clarkb | though I'd expect the confusion to be job doesn't run at all because branch doesn't match | 22:47 |
| clarkb | SpamapS: tobiash had all these paged in recently (maybe look at recent changes from tobiash too?) but aiui github sends and event that says this merged. Zuul is then supposed to send the please update my repo event to the mergers. THis moves master ahead and results in new config being loaded | 22:48 |
| SpamapS | The trigger is on push | 22:50 |
| SpamapS | trigger: | 22:50 |
| SpamapS | github: | 22:50 |
| SpamapS | - event: push | 22:51 |
| clarkb | there was a similar thing that led to me writing https://review.openstack.org/621665 which ended up being unnecessary after tobiash dug more. This was to handle deleted branches on the mergers (which weren't being updated) | 22:51 |
| SpamapS | but the weird thing is that this has run like, 50 times in the last 10 days. | 22:51 |
| clarkb | there was another change that I thought improved that specific issue but having a hard time finding that, but wonder if it is related to this | 22:51 |
| SpamapS | We've been using it to push our site live. It's been nice. :-P | 22:51 |
| SpamapS | to me it looks like the job itself just isn't updated in the config | 22:52 |
| SpamapS | I wonder if the scheduler had a problem. | 22:52 |
| clarkb | the sequencing there (from memory) is that the scheduler will ask the mergers to give it the new updates for .zuul.yaml etc, then it compiles that and applies to subsequent runs of jobs | 22:53 |
| clarkb | so ti could be in the scheduler or the merger potentially | 22:53 |
| clarkb | I would look for logs around when the push trigger happens on the scheduler and merger to see if it says anything about loading its configs | 22:57 |
| SpamapS | thing is, that might explain 1 | 22:59 |
| SpamapS | but this is multiple fails in a row. :( | 22:59 |
| clarkb | unless its a persistent failure to update | 22:59 |
| clarkb | then each push would fail to update | 22:59 |
| clarkb | (potentially) | 22:59 |
| *** rlandy has quit IRC | 23:03 | |
| SpamapS | failure to update what? | 23:03 |
| clarkb | the repo for .zuul.yaml config updates | 23:04 |
| * SpamapS also has just discovered that his dev ELK instance filled up about 6 hours ago.. whoops | 23:04 | |
| clarkb | elasticsaerch will do that for you openstack runs 6 * 1TB disk instances and keeps 10 days of data | 23:04 |
| SpamapS | clarkb: Not sure I follow.. but.. I'm going to try restarting the scheduler next. :-P | 23:04 |
| SpamapS | the weird thing is, the *contents*.. the working dirs.. are all correct. | 23:05 |
| clarkb | SpamapS: the way zuul updates its configuration is the scheduler asks a merger to cat the contents of the various possible zuul config files. If that fails the scheduler won't know what the new config is | 23:05 |
| SpamapS | just not the zuul config or playbooks. | 23:05 |
| clarkb | (and may keep running with the old config) | 23:05 |
| SpamapS | ah ok | 23:06 |
| SpamapS | I'll dig around in the logs for something like that | 23:06 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!