| *** shanemcd has quit IRC | 00:02 | |
| *** shanemcd has joined #zuul | 00:04 | |
| *** ssbarnea has quit IRC | 00:29 | |
| *** rlandy|bbl is now known as rlandy | 01:59 | |
| *** swest has quit IRC | 02:33 | |
| *** swest has joined #zuul | 02:47 | |
| *** bhavikdbavishi has joined #zuul | 03:10 | |
| *** threestrands has joined #zuul | 03:46 | |
| *** rlandy has quit IRC | 05:30 | |
| *** swest has quit IRC | 06:11 | |
| *** swest has joined #zuul | 06:12 | |
| *** chkumar|off is now known as chandankumar | 06:21 | |
| *** threestrands has quit IRC | 06:52 | |
| *** bhavikdbavishi has quit IRC | 07:02 | |
| *** pcaruana|elisa| has joined #zuul | 07:40 | |
| *** pcaruana|elisa| has quit IRC | 07:59 | |
| *** bhavikdbavishi has joined #zuul | 08:01 | |
| *** pcaruana has joined #zuul | 08:05 | |
| *** bhavikdbavishi has quit IRC | 08:09 | |
| *** bhavikdbavishi has joined #zuul | 08:09 | |
| *** ssbarnea has joined #zuul | 08:55 | |
| *** rcarrillocruz has quit IRC | 09:05 | |
| *** rcarrillocruz has joined #zuul | 09:11 | |
| *** nilashishc has joined #zuul | 09:16 | |
| *** electrofelix has joined #zuul | 09:32 | |
| *** panda|off is now known as panda | 09:56 | |
| *** chandankumar has quit IRC | 10:09 | |
| *** chandankumar has joined #zuul | 10:31 | |
| *** pcaruana has quit IRC | 11:05 | |
| *** bhavikdbavishi has quit IRC | 11:16 | |
| *** nilashishc has quit IRC | 11:45 | |
| *** pcaruana has joined #zuul | 11:52 | |
| *** bhavikdbavishi has joined #zuul | 11:54 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Add tox functional testing for drivers https://review.openstack.org/609515 | 12:25 |
|---|---|---|
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Add tox functional testing for drivers https://review.openstack.org/609515 | 12:26 |
| *** goern has joined #zuul | 12:40 | |
| *** rlandy has joined #zuul | 12:41 | |
| *** zul has joined #zuul | 12:58 | |
| *** hashar has joined #zuul | 13:02 | |
| *** zul has quit IRC | 13:04 | |
| *** zul has joined #zuul | 13:06 | |
| Shrews | tobiash: if you have a moment, my moving nodepool unit tests to a subdir in 609515 has caused failures in test_launcher.py for some reason. I can't quite see why, and I have to run to an eye appointment right now. Can you take a quick look and give me a hint? I've already fixed the TestDrivers failure, just not pushed it up. | 13:36 |
| Shrews | bbiab | 13:36 |
| *** hashar has quit IRC | 13:49 | |
| *** goern_ has joined #zuul | 13:51 | |
| tobiash | looking | 13:54 |
| *** goern has quit IRC | 13:54 | |
| *** goern_ is now known as goern | 13:54 | |
| *** bhavikdbavishi has quit IRC | 13:58 | |
| tobiash | weird, running the failing tests individually works | 14:09 |
| mordred | tobiash, Shrews: while you're looking at weird test failures - any clue why things are failing here: https://review.openstack.org/#/c/612169/ ? | 14:16 |
| mordred | they both seem random | 14:16 |
| tobiash | mordred: that looks like a zk connection problem we still see occationally | 14:18 |
| tobiash | Shrews: looks like the drivers list is empty sometimes when I enhance the error message: RuntimeError: fake: unknown driver, available: {} | 14:19 |
| mordred | tobiash: :( | 14:23 |
| mordred | tobiash, Shrews: maybe a global variable that's getting stepped on? | 14:25 |
| tobiash | Shrews, mordred: in the failing cases Drivers.load tells me it's already loaded (which is not true | 14:27 |
| mordred | tobiash: well - Drivers.load does use a singleton list | 14:28 |
| mordred | so if two tests are running and the file is loaded in each of them - that state is going to stick around | 14:28 |
| tobiash | and at some point it initializes it to an empty list | 14:28 |
| tobiash | that's why it says it;s already loaded | 14:28 |
| tobiash | but I have no clue why this happens only in this change | 14:29 |
| mordred | yah - me either | 14:30 |
| mordred | unless moving the tests around just happened to change the ordering such that 2 tests that each want to load drivers wind up in the same process memory | 14:30 |
| tobiash | mordred: I fixed that emtpy list and now I get: FileNotFoundError: [Errno 2] No such file or directory: '/home/cloud/nodepool-test/nodepool/tests/driver' | 14:30 |
| tobiash | ah, I think I have a clue... | 14:31 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: Add tox functional testing for drivers https://review.openstack.org/609515 | 14:33 |
| tobiash | Shrews, mordred: fixes these driver paths | 14:34 |
| tobiash | Shrews, mordred: specifically https://review.openstack.org/#/c/609515/14/nodepool/tests/unit/test_drivers.py | 14:34 |
| mordred | tobiash: ah - because of the extra directory | 14:36 |
| tobiash | that's why I don't like global variables ;) | 14:36 |
| Shrews | tobiash: yeah, that was the TestDrivers fix i mentioned. That doesn't fix the TestLauncher failures | 15:29 |
| tobiash | Shrews: it did fix it locally | 15:29 |
| Shrews | how? | 15:30 |
| tobiash | It had a side effect of breaking driver loading | 15:30 |
| Shrews | oh, i see | 15:31 |
| Shrews | tobiash: thx! | 15:31 |
| tobiash | No problem | 15:32 |
| Shrews | mordred: for your failing thing, is it possible you may be seeing another task manager problem that clarkb put up a fix for (https://review.openstack.org/#/c/612835/1/nodepool/tests/test_sdk_integration.py)? | 15:36 |
| Shrews | mordred: the zk stuff was misleading in that case | 15:37 |
| Shrews | mordred: i ask only because i saw the words "task manager" in your review, not that I've investigated | 15:37 |
| fungi | corvus: tobiash: we have some renewed interest in https://review.openstack.org/578557 | 15:40 |
| fungi | does it need anything else besides the todos in the commit message (documentation and release note?) | 15:40 |
| tobiash | We run it at least since a while in production | 15:41 |
| tobiash | fungi: I think only docs and reno is missing | 15:41 |
| corvus | tobiash, fungi: agreed | 15:43 |
| panda | does zuul compile the list of playbooks to run for a job dynamically or statically ? I mean when does it decide which pre-run/post-run to run ? | 16:03 |
| corvus | panda: yes -- right before it starts the job. it takes into account all job variants which apply to the change. | 16:04 |
| corvus | (job variants may or may not apply based on what branch the change is on) | 16:05 |
| panda | corvus: is there a way to see this list in the job itself ? | 16:05 |
| corvus | panda: yes, there's currently some debugging information emitted under the "zuul._inheritance_path" ansible variable. it's not a stable api and is only meant for human consumption, but it should tell you what's going on. | 16:06 |
| corvus | for example, in http://logs.openstack.org/85/605585/5/check/system-config-run-docker/3bb43f1/zuul-info/inventory.yaml you can see http://paste.openstack.org/show/733779/ which shows that it built up the final job it ran from 4 sources | 16:08 |
| corvus | (the numbers are line numbers, so you can track down which jobs / variants were used) | 16:08 |
| panda | corvus: I know, that shows how many configurations you traverse, but doesn't show the list of playbooks are actually going to be run | 16:13 |
| panda | also to parse this corretly I would need to know the exact version of the file was used | 16:14 |
| corvus | panda: well, the list of playbooks run is encoded in either the ara report, or the json output. eg: http://logs.openstack.org/85/605585/5/check/system-config-run-docker/3bb43f1/job-output.json.gz | 16:15 |
| corvus | panda: however, you used the future tense, and i'm using the past tense... so we may not be on the same page -- what is it that you want to do? | 16:15 |
| panda | corvus: ooooohhh right! the json console | 16:16 |
| panda | corvus: is ther a way to get this json in the run itself, not as an artifact at the end of it ? | 16:18 |
| corvus | panda: what are you trying to do? | 16:19 |
| panda | corvus: we are tracking down the requirements to reproduce locally the jobs. THere are essential part of the two nodes job for example (the multinode-bridge) that are done in pre-run playbooks. We need to run them too before the reproducer can kick in, or the initial environment will not be genuine. | 16:22 |
| corvus | panda: ah, so you want to make a post-playbook to run at the end of the job which processes the job-output.json of the currently running job to generate a reproducer script which gets uploaded along with the logs? | 16:23 |
| panda | corvus: yup | 16:24 |
| *** pall is now known as pabelanger | 16:24 | |
| panda | and then it will probably create a black hole of recursion | 16:24 |
| corvus | panda: https://zuul-ci.org/docs/zuul/user/jobs.html#var-zuul.executor.log_root is the directory containing job-output.json; the job should be able to read it, but it will only be partially written at that point. i don't remember if there will be anything about the currently running playbook in it, or if it will only be written up to the previous playbook if you read it mid-stream. | 16:26 |
| corvus | panda: so that's worth a try :) | 16:26 |
| panda | corvus: that can be taken into account, the only problme is if mid-stream the json is not parsable | 16:27 |
| corvus | panda: i think it should always be valid json | 16:27 |
| corvus | (i'm pretty sure we take care to make sure that's the case) | 16:27 |
| panda | corvus: wow, perfect, I think we have a lot more than a shot, thanks! | 16:27 |
| corvus | panda: we control the generation of that file via a custom ansible callback plugin in zuul, so if you need more information, we can probably add it | 16:28 |
| corvus | panda: see http://git.zuul-ci.org/cgit/zuul/tree/zuul/ansible/callback/zuul_json.py#n73 | 16:29 |
| corvus | (that's how all the pre/post stuff gets into the json) | 16:30 |
| panda | corvus: I see a dump only for v2_playbook, is it updated at every step ? | 16:33 |
| panda | (not really familiar with the zuul code unfortunately) | 16:34 |
| corvus | panda: that's more ansible than zuul -- but i *think* that means that we dump to the file after the completion of each playbook, so the file should have everything up to the end of the previous playbook, and nothing about the current playbook when you read it. | 16:36 |
| SpamapS | hrm.. having pbrx issues | 16:36 |
| SpamapS | http://paste.openstack.org/show/733781/ | 16:36 |
| Shrews | mordred: i'm becoming more convinced that we need to account for task manager threads in our test thread whitelist | 16:37 |
| panda | corvus: perfect, will check, thanks. | 16:37 |
| mordred | Shrews: nod. so - hopefully we're not too far off from getting rid of the task manager threads | 16:37 |
| mordred | SpamapS: does your project have an entry_points section in its setup.cfg? | 16:38 |
| mordred | SpamapS: (we should make that section of code more resilient and skip the per-console-script image creation if there are no console-script defined) | 16:39 |
| clarkb | Shrews: we already do | 16:39 |
| clarkb | Shrews: well sort of, we explicitly stop the taskmanager threads in nodepool | 16:40 |
| clarkb | Shrews: the reason the openstacksdk taskmanager didn't work is those don't get registered with the driver so we don't call stop on them iirc | 16:40 |
| clarkb | which is why my fix to use the nodepool taskmanager made the problem go away | 16:40 |
| SpamapS | mordred: my project is zuul.. so.. yes | 16:41 |
| SpamapS | oh | 16:41 |
| SpamapS | looks like the bash script has a bug | 16:42 |
| SpamapS | big surprise | 16:42 |
| SpamapS | stand down | 16:42 |
| Shrews | clarkb: but something weird is up with mordred's change to the task manager. it "feels" similar, but i can't quite yet put a finger on the issue | 16:42 |
| *** pabelanger is now known as pall | 16:43 | |
| *** sshnaidm|ruck is now known as sshnaidm|afk | 16:48 | |
| manjeets | corvus, Hi https://github.com/openstack-infra/zuul/blob/master/doc/source/admin/examples/etc_zuul/main.yaml#L10 this maps to https://github.com/openstack-infra/zuul/blob/master/doc/source/admin/examples/etc_zuul/zuul.conf#L23 ? | 16:51 |
| manjeets | and in https://github.com/openstack-infra/zuul/blob/master/doc/source/admin/examples/etc_zuul/zuul.conf#L23 I can use local file path in base url ? | 16:51 |
| *** rfolco|rover is now known as rfolco|ruck | 16:58 | |
| corvus | manjeets: exactly | 17:01 |
| manjeets | corvus, but what driver should be used ? | 17:03 |
| corvus | manjeets: 'git', just line on line 25 | 17:04 |
| mordred | tobiash: my brain just clicked with something I think you were talking about yesterday or the day before ... regarding camel cased statsd keys | 17:40 |
| mordred | tobiash: it's not actually openstacksdk that creates the camel cased keys - it's nodepool.task_manager._transform_task_name | 17:40 |
| tobiash | mordred: maybe we want to change that? | 17:41 |
| mordred | maybe - I'm actually working on what to do about the task manager statsd logging when there is no longer a task manager | 17:42 |
| *** goern has quit IRC | 17:44 | |
| mrhillsman | just want to say that the notifications on the dashboard are awesome | 17:46 |
| mrhillsman | big high five, thumbs up, etc for whoever implemented that | 17:46 |
| mrhillsman | not sure if this is the place but a timestamp would be very useful; i say that because i just noticed notifications exist and not sure if i already corrected the issue i got notification for or not | 17:48 |
| AJaeger | mrhillsman: notifications? I haven't found those yet - tell me more, please | 17:50 |
| fungi | what notifications? i completely missed that, whatever it is | 17:50 |
| fungi | now i'm intrigued | 17:51 |
| AJaeger | good, I'm not the only one ;) | 17:51 |
| mrhillsman | i do not see any zuul.o.o | 17:51 |
| mrhillsman | but on a new install i just put up via the docker containers | 17:51 |
| mrhillsman | in the top-right next to "Tenant tenant-name" | 17:52 |
| mrhillsman | there is a bell icon | 17:52 |
| fungi | what sort of notifications is it displaying? | 17:52 |
| mrhillsman | that lets me know i have some config errors | 17:52 |
| fungi | aha! | 17:52 |
| mrhillsman | not sure if they are old or new, pretty sure they are new | 17:52 |
| mrhillsman | i mean old lol | 17:52 |
| mrhillsman | but they are there :) | 17:53 |
| mrhillsman | and i like it | 17:53 |
| fungi | yeah, config errors info page is relatively new | 17:53 |
| mrhillsman | hehe | 17:53 |
| AJaeger | http://zuul.openstack.org/api/config-errors shows we have no errors ;) | 17:53 |
| fungi | i guess i missed that you'd get a little bell icon on the main status if there are config errors | 17:53 |
| fungi | that's neat | 17:53 |
| mrhillsman | being able to configure the notifications and having a timestamp would be extra lovely | 17:54 |
| mrhillsman | i was trying to find out more to see if i could add a patch | 17:55 |
| mrhillsman | ooo, i think i found it :) | 17:55 |
| *** sshnaidm|afk is now known as sshnaidm|off | 18:00 | |
| corvus | mrhillsman: ah, the notifications represent *current* config errors -- so they're never out of date. so if you see an error there, it's current. if you fix an error, it will go away. | 18:09 |
| mrhillsman | good to know, thx corvus | 18:10 |
| manjeets | corvus, for some reason zuul web server always shows loading nothing even though patches are now getting streamed | 18:24 |
| manjeets | https://pasteboard.co/HLbAMDA.png | 18:24 |
| corvus | manjeets: try doing a shift-reload | 18:27 |
| manjeets | corvus, no difference | 18:34 |
| corvus | manjeets: hrm. maybe one of the folks more familiar with the javascript stuff can suggest a debugging strategy. | 18:35 |
| corvus | manjeets: oh, one thing to make sure of is whether zuul-web is running, and if there are any errors in its logs | 18:36 |
| corvus | manjeets: if you're using containers, it should be the "web" container | 18:37 |
| manjeets | ok corvus and using that zuul-config I'm trying to post a comment on a patch in ci-sandbox using my own account | 18:38 |
| manjeets | http://paste.openstack.org/show/733859/ | 18:38 |
| manjeets | but it returns host key verification failed ? I didn't get that same key is used for streaming events which works fine | 18:38 |
| corvus | manjeets: until we merge https://review.openstack.org/608453, you'll need to put the host key into ~/.ssh/known_hosts in the executor container | 18:39 |
| clarkb | corvus: thoughts on cutting a zuul release nowish since ifnra has been running nodepool and zuul for about a week with no major issues (openstacksdk being the only one but that is fixed indepdnently) | 18:39 |
| corvus | clarkb: yes that seems like a very good idea | 18:41 |
| corvus | clarkb: hrm, we did add "Filter file comments for existing files" after the current sha | 18:44 |
| manjeets | oh thanks corvus another issue I see is may be because of image used for container some container I am able to get in but for executor it returns | 18:44 |
| manjeets | rpc error: code = 2 desc = oci runtime error: exec failed: container_linux.go:247: starting container process caused "exec: \"bash\": executable file not found in $PATH" | 18:44 |
| corvus | manjeets: you need to run '/bin/sh' instead of bash | 18:44 |
| manjeets | lol dumb me | 18:45 |
| corvus | clarkb: ideally, i'd like to land the pep8 job changes and test that out before release, but that's going to take a while. we could release 3.3.0 with openstack's sha now and count on releasing 3.3.1 later. or we could jump ahead and release master as 3.3.0, assuming our test coverage is good there. :) | 18:45 |
| corvus | that's the only substantial code change between openstack's sha and master | 18:46 |
| clarkb | I'd be fine with 3.3.1 after 3.3.0 to pick that up. Simply because oepnstack is running without that fix and things seem mostly happy | 18:46 |
| clarkb | fwiw I'm working on restarting nodepool services today to pick up the sdk changes | 18:47 |
| corvus | yeah, it's not a bug, per-se. it's a safety belt in case someone has a bug in an in-line comment return (like we did). | 18:47 |
| corvus | though, if we do release 3.3.0, it means we can't land the pep8 job change until we are certain it's working correctly (since if we break it with 3.3.0 out, we'll break other installs) | 18:48 |
| SpamapS | oh yay, cherrypy actually serves w/o nginx now | 18:48 |
| SpamapS | (in the pbrx built images) | 18:49 |
| corvus | clarkb: so should we wait just a bit longer to tag nodepool? | 18:52 |
| clarkb | corvus: ya, also nodepool has a very recent tag anyway so is less urngent | 18:52 |
| corvus | clarkb, tobiash, Shrews, mordred: i think we're looking at zuul 3.3.0 @ d4a142b97b5ce4b0c15702f502e55965e89095eb and nodepool 3.3.1 @ 7e1b8a7261d31208d853ecfa255da648ba54b656 [pending the ok after clarkb restarts openstack's nodepool as a sanity check]. look right? | 18:54 |
| *** rfolco has joined #zuul | 18:54 | |
| clarkb | corvus: that is the sha1 that is being installed on the infra nodepool builders right now. | 18:55 |
| *** rfolco|ruck has quit IRC | 18:56 | |
| tobiash | corvus: looks good to me | 18:58 |
| Shrews | corvus: lgtm | 19:04 |
| clarkb | corvus: re zuul release that appears to be what openstack ifnra is running now. I'm happy with that and I think gets a lot of improvements out to users before the summit which will be nice (worry that with summit fast approaching we won't be able to do 3.3.0 with inlien commenting fix before hand) | 19:04 |
| *** pcaruana has quit IRC | 19:05 | |
| corvus | yes, lots of improvements, and one dangerous pitfall :) | 19:09 |
| manjeets | corvus, I added the host key to ~./ssh/known_hosts to executor container still see this http://paste.openstack.org/show/733859/ | 19:14 |
| *** panda is now known as panda|off | 19:20 | |
| *** caphrim007 has joined #zuul | 19:26 | |
| caphrim007 | corvus: am i understanding this correctly that, if a nodeset to a job is not provided, that the job will run on the zuul-executor node itself? | 19:26 |
| *** electrofelix has quit IRC | 19:28 | |
| *** leifmadsen has left #zuul | 19:43 | |
| *** j^2 has joined #zuul | 19:48 | |
| corvus | caphrim007: ansible is always run from the zuul-executor (that's the ansible control host). it's accessible as the implied localhost in the inventory (so to use it, you have to name it explicitly, eg, "hosts: localhost"). any job can perform limited actions on it, and jobs defined in config-repos can perform any action on it. | 20:10 |
| corvus | caphrim007: (this is all true whether or not there are any actual hosts in the inventory) | 20:10 |
| caphrim007 | thanks corvus! | 20:11 |
| corvus | manjeets: hrm, i don't know why that wouldn't work. maybe the wrong key? | 20:11 |
| manjeets | corvus, copied same key from one I had in etc_zuul/ which works for gerrit event streaming | 20:12 |
| corvus | manjeets: the gerrit event streaming doesn't have the same problem (it doesn't require an entry in known_hosts), so that may not be doing what you think. | 20:13 |
| *** rlandy is now known as rlandy|brb | 20:26 | |
| *** rlandy|brb is now known as rlandy | 21:21 | |
| *** rcarrillocruz has quit IRC | 21:56 | |
| *** threestrands has joined #zuul | 22:17 | |
| *** j^2 has quit IRC | 22:39 | |
| *** caphrim007 has quit IRC | 23:01 | |
| mrhillsman | can someone help me understand bare vs contained role? https://zuul-ci.org/docs/zuul/user/config.html?highlight=galaxy#attr-job.roles.name | 23:47 |
| mrhillsman | https://zuul-ci.org/docs/zuul/user/config.html?#attr-job.roles.name | 23:48 |
| clarkb | mrhillsman: bare role is a repo with a single role in it. Think openstack-ansible-nova. Contained is a repo with a bunch of roles like zuul-jobs | 23:50 |
| clarkb | at least that is my understanding of it | 23:50 |
| mrhillsman | ty sir | 23:50 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!
