| *** dkranz has quit IRC | 00:04 | |
| *** odyssey4me has quit IRC | 00:23 | |
| *** odyssey4me has joined #zuul | 00:23 | |
| *** Wei_Liu1 has joined #zuul | 03:03 | |
| *** Wei_Liu has quit IRC | 03:04 | |
| *** Wei_Liu1 is now known as Wei_Liu | 03:04 | |
| dmsimard | mordred: I have a stupid question. What would happen if you created hundreds of thousands of MySQL databases ? The databases would be small but self-contained. | 03:22 |
|---|---|---|
| dmsimard | I guess MySQL is not really the kind of database for that ? | 03:22 |
| dmsimard | I'm considering providing essentially the same thing as the sqlite middleware but for mysql/postgre | 03:23 |
| dmsimard | Instead of having to put everything in one large database, it could be "sharded" and would in theory provide faster seek/scan times ? If the pros don't outweigh the database search time because there'd be so many. | 03:25 |
| dmsimard | Turns out after some reading that's probably a bad idea :D | 03:35 |
| clarkb | dmsimard: each db or table gets its own file dpending how innodb is configured iirc | 04:04 |
| clarkb | chances are your filesystem would not like that at all | 04:05 |
| dmsimard | clarkb: sqlite middleware is kinda like that ? | 04:05 |
| clarkb | dmsimard: sort of, the way we write the files to the fs is very different than how mysql will do it aiui | 04:14 |
| *** elyezer has quit IRC | 04:45 | |
| *** elyezer has joined #zuul | 04:46 | |
| *** nguyenhai has quit IRC | 04:55 | |
| *** nguyenhai has joined #zuul | 04:56 | |
| *** elyezer has quit IRC | 05:14 | |
| *** elyezer has joined #zuul | 05:16 | |
| tobiash | dmsimard: I think with mysql you rather want a db per tenant and not per job | 05:18 |
| *** AJaeger has quit IRC | 06:10 | |
| *** snapiri- has quit IRC | 06:16 | |
| *** snapiri has joined #zuul | 06:16 | |
| *** AJaeger has joined #zuul | 06:31 | |
| *** hashar has joined #zuul | 06:58 | |
| *** Wei_Liu has quit IRC | 07:09 | |
| *** Wei_Liu has joined #zuul | 07:10 | |
| *** electrofelix has joined #zuul | 07:41 | |
| *** jpena|off is now known as jpena | 07:50 | |
| openstackgerrit | Simon Westphahl proposed openstack-infra/zuul master: Allow using remote refs to find commits for change https://review.openstack.org/544964 | 08:25 |
| openstackgerrit | Simon Westphahl proposed openstack-infra/zuul master: Allow using remote refs to find commits for change https://review.openstack.org/544964 | 09:16 |
| *** sshnaidm|off is now known as sshnaidm | 09:30 | |
| *** xinliang has quit IRC | 11:16 | |
| *** xinliang has joined #zuul | 11:27 | |
| *** xinliang has quit IRC | 11:27 | |
| *** xinliang has joined #zuul | 11:27 | |
| *** jpena is now known as jpena|lunch | 11:42 | |
| *** JasonCL has joined #zuul | 12:00 | |
| *** elyezer has quit IRC | 12:00 | |
| *** JasonCL has quit IRC | 12:05 | |
| *** JasonCL has joined #zuul | 12:12 | |
| *** elyezer has joined #zuul | 12:12 | |
| *** rlandy has joined #zuul | 12:14 | |
| *** dkranz has joined #zuul | 12:20 | |
| *** jpena|lunch is now known as jpena | 12:32 | |
| *** maeca has joined #zuul | 12:32 | |
| *** maeca has quit IRC | 12:33 | |
| *** odyssey4me has quit IRC | 12:38 | |
| *** odyssey4me has joined #zuul | 12:38 | |
| *** gouthamr has joined #zuul | 12:49 | |
| *** gouthamr has quit IRC | 12:52 | |
| openstackgerrit | Stephen Finucane proposed openstack-infra/zuul-jobs master: Default warning-is-error to True for non-legacy Sphinx projects https://review.openstack.org/559348 | 12:52 |
| *** gouthamr has joined #zuul | 13:02 | |
| *** gouthamr has quit IRC | 13:15 | |
| *** gouthamr has joined #zuul | 13:17 | |
| *** elyezer has quit IRC | 13:19 | |
| dmsimard | tobiash: yeah it was a stupid idea but made me think about something :D | 13:19 |
| *** elyezer has joined #zuul | 13:20 | |
| *** gouthamr has quit IRC | 13:26 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Fix missing semaphore release on zk error https://review.openstack.org/559745 | 14:06 |
| openstackgerrit | Merged openstack-infra/zuul master: Remove openstack-infra reference https://review.openstack.org/559586 | 14:23 |
| *** gouthamr has joined #zuul | 14:29 | |
| *** gouthamr has quit IRC | 14:31 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul master: Fix streaming decoding boundaries https://review.openstack.org/559326 | 14:37 |
| Shrews | ianw: ^^ adds the final decode you suggested. good eye | 14:38 |
| mordred | dmsimard: I actually don't think mysql would care too much - unless as clarkb mentions you get to enough databases thatyou run out of inodes | 14:58 |
| mordred | ALTHOUGH - mysql 5.8 or 5.10 or wichever is the one they just released - has finally done the drizzle thing and moved the .frm file content into the innodb tablespace | 14:59 |
| mordred | so with a modern enough mysql (or drizzle) there would be no difference - it would just be a new collection of tables | 15:00 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Fix missing semaphore release on zk error https://review.openstack.org/559745 | 15:05 |
| *** gouthamr has joined #zuul | 15:13 | |
| Shrews | mordred: oh did they? i like watching mysql trying to catch up to drizzle | 15:17 |
| Shrews | it's fun | 15:17 |
| mordred | Shrews: yah. stewart tweeted something out a little while ago | 15:19 |
| *** xinliang has quit IRC | 15:19 | |
| *** xinliang has joined #zuul | 15:21 | |
| *** xinliang has quit IRC | 15:21 | |
| *** xinliang has joined #zuul | 15:21 | |
| *** jpena is now known as jpena|brb | 15:41 | |
| *** electrofelix has quit IRC | 15:44 | |
| clarkb | mordred: dmsimard double checking things ext4 has unlimited files per dir it was ext3 specifically that had limits I think so likely fine | 15:55 |
| *** gouthamr has quit IRC | 15:56 | |
| pabelanger | Hmm, paths._is_safe_path for synchronize doesn't seem to include the path to the playbook when checking source value. So doing something like source: ../config caused zuul-executor to fail, even though ../config isn't outside of the work directory | 16:03 |
| clarkb | pabelanger: playbooks are un the trusted/ and untrusted/ dirs iirc and are outside of the work directory | 16:05 |
| pabelanger | oh, maybe that is it | 16:06 |
| pabelanger | https://review.openstack.org/559132/ | 16:08 |
| pabelanger | I think that might help | 16:08 |
| clarkb | pabelanger: I think there are exceptions for other things like copying using template or direct file copy | 16:09 |
| pabelanger | but, that is for trusted only | 16:09 |
| clarkb | but synchronize may not have it as its per module needing to set the flag iirc | 16:09 |
| corvus | pabelanger: are you using a zuul which is running 559132? | 16:09 |
| pabelanger | corvus: not sure, I am testing on zuul.o.o | 16:10 |
| *** hashar has quit IRC | 16:10 | |
| corvus | pabelanger: ah, it's not running it yet. as soon as the etherpad changes land, i'll restart it with that change. | 16:10 |
| pabelanger | cool | 16:10 |
| corvus | pabelanger: yes, it should help this situation. it will be good to repeat your test after restarting. | 16:10 |
| pabelanger | thanks! | 16:10 |
| *** jpena|brb is now known as jpena | 16:24 | |
| openstackgerrit | Merged openstack-infra/nodepool master: Add a backoff for failed builds https://review.openstack.org/558686 | 16:34 |
| corvus | wait can we talk about that one? | 16:34 |
| corvus | ianw, Shrews, jhesketh: that runs counter to nodepool design goals. i'd like to discuss it. | 16:35 |
| corvus | Shrews: how do we recover from that? | 16:36 |
| clarkb | corvus: its specific to the test | 16:36 |
| clarkb | corvus: so the test job detects failure early rather than waiting until the job timeout | 16:36 |
| corvus | oh! | 16:36 |
| Shrews | yeah, just affects our test, not nodepool itself | 16:36 |
| clarkb | (nothing in nodepool itself has changed) | 16:36 |
| corvus | the commit message did not convey that information to me | 16:36 |
| corvus | nevermind then, carry on :) | 16:36 |
| Shrews | :) | 16:36 |
| corvus | sorry for the confusion | 16:36 |
| Shrews | mondays are made for confusion | 16:37 |
| *** hashar has joined #zuul | 16:42 | |
| -openstackstatus- NOTICE: zuul was restarted to update to the latest code; please recheck any changes uploaded within the past 10 minutes | 16:53 | |
| jlk | o/ | 16:55 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Refactor load sensors into drivers https://review.openstack.org/549275 | 16:55 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Refactor load sensors into drivers https://review.openstack.org/549275 | 16:58 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Refactor load sensors into drivers https://review.openstack.org/549275 | 16:59 |
| dmsimard | corvus: are both docs.o.o/infra/zuul and zuul-ci.org/docs kept up to date ? | 16:59 |
| mrhillsman | the console stream requires ingress access to the executor? | 17:00 |
| clarkb | dmsimard: they are both served from the same afs volume aiui | 17:00 |
| pabelanger | mrhillsman: unless you have zuul-fingergw, it will be a gateway to executors | 17:01 |
| clarkb | fingergw still requires ingress to executors, but is a single source rather than arbitrary | 17:01 |
| mrhillsman | was about to ask that | 17:01 |
| mrhillsman | so yeah, no console stream :( | 17:01 |
| dmsimard | clarkb: oh, neat | 17:01 |
| mrhillsman | but i can push the logs somewhere at least | 17:01 |
| pabelanger | yah, ingress still but wouldn't need public IPs, assuming fingergw bridges the 2 networks | 17:02 |
| corvus | dmsimard, clarkb: actually they're different. duplicate publication jobs. i was thinking we probably want to retire docs.o.o/zuul and redirect to zuul-ci.org. | 17:03 |
| clarkb | ah | 17:03 |
| dmsimard | corvus: that's what I thought would be best (even if just from a "branding" perspective) | 17:04 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Refactor load sensors into drivers https://review.openstack.org/549275 | 17:04 |
| corvus | mrhillsman: yeah, for either finger or web console streaming, we need a route from scheduler->executor. but other than that, the executors don't need to be accessible. | 17:04 |
| corvus | mrhillsman: so you might be able to put the scheduler on a vpn that can contact the executors. | 17:05 |
| corvus | let me amend that | 17:05 |
| mrhillsman | hehe | 17:05 |
| corvus | it's not the scheduler that needs that access, it's the zuul-web and zuul-fingergw processes respectively | 17:05 |
| corvus | that probably doesn't substantially change things though. | 17:06 |
| mrhillsman | but it is web->exec vs web<-exec | 17:06 |
| corvus | mrhillsman: correct | 17:06 |
| mrhillsman | yeah, unfortunately cannot let anything in | 17:06 |
| corvus | mrhillsman: is vpn an option? | 17:07 |
| mrhillsman | so, i have everything working right now with the corporate vpn | 17:07 |
| mrhillsman | they are just not going to let any ingress that is not an employee | 17:08 |
| mrhillsman | which so far just leaves me with not having console stream | 17:09 |
| tobiash | mrhillsman: what's your network layout regarding the various zuul serivices? | 17:09 |
| mrhillsman | scheduler, gearman, zookeeper all in public space | 17:10 |
| mrhillsman | executor, merger, n-builder, n-launcher all behind corporate vpn | 17:10 |
| mrhillsman | i want to have something i can start to explore/try the executor-affinity stuff with | 17:11 |
| mrhillsman | if that is the right term | 17:11 |
| mrhillsman | otherwise i am not sure tying in this with an environment that is all in public space will work | 17:12 |
| clarkb | mrhillsman: in this case its fingergw making connections to known port on the executors. Then the data it reads is proxied to the user on the public finger port side. I don't think that is any less secure than copying the logs off host at the end of the job abd allowing arbitrary code execution | 17:12 |
| clarkb | of course if you sell it as arbitrary code execution people might get scared | 17:12 |
| mrhillsman | at least right now it is confirmed console stream won't be available | 17:12 |
| tobiash | with that topology and your restrictions I fear the only possibility is to put zuul-web/zuul-fingergw also behind the corporate vpn and sacrifice global availability of streaming | 17:13 |
| tobiash | but you at least could make streaming possible for your employees | 17:13 |
| mrhillsman | clarkb that may be doable | 17:13 |
| pabelanger | couldn't the executor establish a connection to zuul-fingergw to support this use case? | 17:13 |
| mrhillsman | that makes sense tobiash | 17:13 |
| pabelanger | I would guess that would means something other then finger protocol | 17:14 |
| clarkb | pabelanger: assuming that doesn't violate some different security policy | 17:14 |
| mrhillsman | pabelanger that would be ideal of course but not sure ^ | 17:14 |
| pabelanger | clarkb: yah | 17:14 |
| mrhillsman | sorry, i mean, not sure that is possible without some codefu | 17:15 |
| mrhillsman | ^ for your comment not clark's :) | 17:15 |
| clarkb | fwiw you could totally do that with a reverse ssh tunnel and just run the fingergw as is | 17:15 |
| clarkb | >_> | 17:15 |
| pabelanger | yah, that should work too | 17:16 |
| pabelanger | :) | 17:16 |
| mrhillsman | i think reverse ssh tunnels are security policy violations hehe | 17:16 |
| tobiash | which likely is against policy ;) | 17:16 |
| clarkb | but again no less secure | 17:16 |
| mrhillsman | console streaming is not a deal breaker | 17:16 |
| clarkb | if you set it up to do just a port forward between the two it would be basically equivalent to reversing the direction of the tcp SYN | 17:17 |
| mrhillsman | and since it does connect to a known port that could be something | 17:17 |
| *** jpena is now known as jpena|away | 17:18 | |
| tobiash | is it possible to do streaming via gearman? | 17:18 |
| mrhillsman | this port, traffic from this address, of this protocol, and whatever other restrictions may be feasible | 17:19 |
| mrhillsman | but i do not know if the blanket policy is no ingress | 17:19 |
| mrhillsman | so i am working from that as a worst case scenario | 17:19 |
| mrhillsman | or best case scenario :) | 17:20 |
| *** bhavik1 has joined #zuul | 17:25 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Update Github3.py to 1.0.3 https://review.openstack.org/559798 | 17:27 |
| tobiash | jlk: preparation for your upcoming release ^ ;) | 17:27 |
| *** gouthamr has joined #zuul | 17:27 | |
| jlk | oh cool. | 17:27 |
| jlk | I suppose I should do that today :D | 17:27 |
| jlk | I'll -2 that until 1.0.3 is out :D | 17:27 |
| tobiash | :) | 17:28 |
| jlk | or rather w-1 | 17:28 |
| *** sean-k-mooney has joined #zuul | 17:32 | |
| *** bhavik1 has quit IRC | 17:33 | |
| sean-k-mooney | o/ | 17:34 |
| sean-k-mooney | hi does anyone recognise this error "Implicit role not found in /tmp/978bdac494724e54a327e0a949f1de70/ansible/pre_playbook_0/role_0/zuul-base-jobs" | 17:35 |
| *** gouthamr has quit IRC | 17:35 | |
| pabelanger | sean-k-mooney: which zuul-base-jobs are you using? | 17:38 |
| sean-k-mooney | openstack/zuul-base-jobs | 17:38 |
| sean-k-mooney | master | 17:38 |
| sean-k-mooney | it looks identical to https://git.zuul-ci.org/cgit/zuul-base-jobs/ | 17:38 |
| pabelanger | can you share URL for console log? | 17:39 |
| pabelanger | I have to AFK, but will be able to help when I return | 17:39 |
| sean-k-mooney | pabelanger: unfortuetly i dont have any i am trying to setup a new ci. i have the noop job working but when i treied to get the tox-pep8 job to work i hit this error | 17:39 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: WIP: Add cgroup support to ram sensor https://review.openstack.org/549506 | 17:40 |
| sean-k-mooney | pabelanger: and no worries thanks for trying to help :) | 17:41 |
| tobiash | sean-k-mooney: This message typically is no error but is emitted on every repo which has no /roles directory. This message is just informational for debugging role path issues. | 17:42 |
| tobiash | sean-k-mooney: that's why we have a change up to make this a debug message: https://review.openstack.org/556866 | 17:43 |
| tobiash | sean-k-mooney: do you have an issue you are debugging? | 17:43 |
| sean-k-mooney | tobiash: right so both https://git.zuul-ci.org/cgit/zuul-base-jobs/ and https://github.com/openstack-infra/zuul-base-jobs do not have role direcoties | 17:43 |
| tobiash | that's ok so your real error is probably somewhere else | 17:44 |
| sean-k-mooney | tobiash: im tring to set up a thrid party ci using zuul v3. i have the noop jobs working and it is claiming a instance fine in nodepool but i have not figured out how to get job logs working yet | 17:45 |
| sean-k-mooney | the only log i have so far is the executor log http://paste.openstack.org/show/718762/ | 17:46 |
| corvus | yeah, that's the next thing we need to add to the zfs guide, and the base jobs repo | 17:47 |
| sean-k-mooney | tobiash: i could set up an autohold for the vm but i dont know where to start looking to debug what failed. is there a default location inside the vm that would have more info | 17:47 |
| corvus | was someone working on that last week? | 17:47 |
| tobiash | sean-k-mooney: so the actual error is 'ansible timeout exceeded' | 17:48 |
| sean-k-mooney | tobiash: yep i just have no idea why | 17:49 |
| tobiash | what you could to is to execute 'zuul-executor verbose' | 17:49 |
| tobiash | after that the executor will output ansible debug stuff | 17:49 |
| sean-k-mooney | tobiash: oh ok will that print the ansible output to the executor log | 17:49 |
| tobiash | and you also seem to have your log level to info | 17:49 |
| corvus | oh, right, it was LinuxJedi that we were working with, and it was 2 weeks ago: http://eavesdrop.openstack.org/irclogs/%23zuul/%23zuul.2018-03-27.log.html#t2018-03-27T14:07:45 | 17:49 |
| tobiash | you need to set this to debug | 17:49 |
| corvus | we need a runtime info/debug switch | 17:50 |
| sean-k-mooney | tobiash: ok cool ya i just left it at default but that makes sense i should have tried that. | 17:50 |
| tobiash | corvus: wasn't there a patch for this or did I dream of it? | 17:50 |
| corvus | tobiash: not quite -- we use debug if you run with '-d' (no daemon). but otherwise, the only way to get debug is via a logging config | 17:51 |
| corvus | which i'd like to get rid of for all but the most extreme development debug cases | 17:51 |
| sean-k-mooney | tobiash: ill try running in debug mode and see if i can fix the real error thanks. by the way is there a documented way to reuse things like the devstack and tempest jobs. i had quite alot of issues useing them without including every poject in openstack. | 17:52 |
| sean-k-mooney | the required_projects item in job definitions is kind of virial | 17:52 |
| corvus | sean-k-mooney: not documented yet -- if you can help us with that in openstack-infra, that would be great | 17:52 |
| corvus | sean-k-mooney: i have an idea to make required-projects less viral, but won't be able to implement it for a little while yet | 17:53 |
| corvus | sean-k-mooney: but we should eventually get to the point where something can be in required-projects but not be in main.yaml | 17:53 |
| corvus | (doing that depends on some configuration refactoring that is in progress, but i had to put on hold for a bit because other things came up) | 17:54 |
| sean-k-mooney | corvus: im not sure you would like how im doinging it (create a local repo called openstack-infra/project-config with no jobs). | 17:55 |
| sean-k-mooney | corvus: even then my tenant definition looks like this http://paste.openstack.org/show/718763/ | 17:55 |
| corvus | sean-k-mooney: why do you need to create a local openstack-infra/project config repo? | 17:57 |
| sean-k-mooney | corvus: its a required project in one of the job definitions | 17:58 |
| corvus | sean-k-mooney: can you point out which one, that sounds like a bug | 17:58 |
| sean-k-mooney | let me see if i can find it. it may have been the openstack-zuul-jobs which i have now removed form the list | 17:58 |
| corvus | sean-k-mooney: cool. if it's not a bug (or not something we can fix), you should be able to add it to the config under "untrusted-projects: include: []" | 18:00 |
| sean-k-mooney | corvus: i was getting an error because it contains pipeline definitions and they are not allowed in untrusted-projects. the execption is still trown when it in the include: [] section | 18:01 |
| corvus | sean-k-mooney: that sounds like a zuul bug | 18:03 |
| sean-k-mooney | corvus: ok so yes the dependecy on project-config was from the openstack-zuul-jobs not devstack or tempest directly. https://github.com/openstack-infra/openstack-zuul-jobs/blob/master/zuul.d/jobs.yaml#L89 | 18:03 |
| sean-k-mooney | with the current list of project i do not need my own local copy of openstack-infra/project-config | 18:04 |
| corvus | sean-k-mooney: good! we should still look into that zuul bug :| | 18:05 |
| sean-k-mooney | corvus: its not just the require projets unfortuetly as the same happens with secrets too. i think the exeption is been throwen uncontionally without regards for include: [] or exclude: ... | 18:06 |
| sean-k-mooney | well missing secrets could be a different issue i guess. that was coming from the fact that i allowed a job to be imported that needed a secret but did not import the secret. when i do import the secret however it can decrypt it so its not going to work in either case unless i define my own secret in its place | 18:09 |
| sean-k-mooney | corvus: tobiash anyway thanks for your time ill see if i can reconfigure to debug loging and root cause my actul issue. | 18:10 |
| *** CrayZee has quit IRC | 18:12 | |
| *** jpena|away is now known as jpena|off | 18:27 | |
| *** trishnag has quit IRC | 19:43 | |
| *** trishnag has joined #zuul | 19:53 | |
| pabelanger | clarkb: corvus: retested the "Syncing files from outside the working dir" issue again now that zuul.o.o is updated, still an issue. The files (directory in this case) aren't actually outside of work. In fact, it is actually: {{ zuul.executor.work_root }}/{{ zuul.project.src_dir }}/config but using "../config" triggers the fail message on action plugin. I'm going to try and reproduce with our zuul remote unit | 20:01 |
| pabelanger | testing. | 20:01 |
| clarkb | pabelanger: gotcha so this isn't a require-project it is the actual project under test? | 20:02 |
| pabelanger | copy isn't affect, just synchronize | 20:02 |
| clarkb | pabelanger: you should be able to reproduce it in the zuul test framework | 20:03 |
| pabelanger | clarkb: right, ../config would be in tree but playbooks are run from: http://git.openstack.org/cgit/openstack/windmill/tree/playbooks | 20:04 |
| pabelanger | clarkb: yah, think so too | 20:04 |
| corvus | pabelanger: can you link to the error? | 20:05 |
| pabelanger | corvus: http://logs.openstack.org/61/559761/9/check/windmill-ubuntu-xenial/fcdefee/job-output.txt.gz#_2018-04-09_19_57_22_867305 | 20:06 |
| pabelanger | I still think clarkb is right, the untrusted playbooks are out side of work dir, which is part of the issue here too | 20:10 |
| corvus | pabelanger: i suspect the issue is that code path doesn't resolve CWD into the path. some of them do now -- if something uses find_needle, it will (so generally roles accessing things internally). but generally relative playbook paths won't. | 20:11 |
| pabelanger | even thought config is one directory up and in the same project | 20:11 |
| corvus | pabelanger: i believe if you provided an absolute path it would work. | 20:11 |
| pabelanger | corvus: yah, absolute does work | 20:11 |
| corvus | clarkb, pabelanger, tobiash, mordred: so we may want to take a look at the action modules which use is_safe_path but don't use find_needle, and see if we can resolve relative paths. | 20:12 |
| corvus | basically, we'd need to make sure each of the action plugins honors relative paths in the same way. | 20:12 |
| pabelanger | +1 | 20:13 |
| *** dkranz has quit IRC | 20:30 | |
| mordred | mrhillsman, tobiash: if/when we finish the streaming rework patches the need for the executor to hit the port on the buidl nodes goes away | 20:41 |
| *** elyezer has quit IRC | 20:47 | |
| mrhillsman | my case is a bit different mordred | 20:48 |
| mrhillsman | the executor is near the build nodes and no issues with it accessing them | 20:48 |
| mrhillsman | the issue is external access to the executor | 20:48 |
| mordred | AH - for the websocket itself | 20:49 |
| mordred | nod. that makes sense | 20:49 |
| mordred | and yes - the stream rework would not help you | 20:49 |
| mrhillsman | yeah, :( | 20:49 |
| mrhillsman | hehe | 20:49 |
| mordred | that's a use case we should think about though | 20:50 |
| mordred | I don't have any good ideas to add to the scrollback - but I'll put it in my thinking hole | 20:50 |
| mrhillsman | i still need to get the logs pushed out but other than that, everything looks kosher | 20:50 |
| mrhillsman | ++, it is not a deal breaker at this stage | 20:51 |
| mrhillsman | now i am going to spend this week and probably next digesting config stuff, jobs, etc etc | 20:52 |
| mrhillsman | and hopefully can start contributing some code going forward | 20:52 |
| mrhillsman | have not had much time to just sit and learn | 20:53 |
| mrhillsman | been trial by fire :) | 20:53 |
| Shrews | So it turns out... our async zuul-web process? Yeah, not so async anymore | 20:56 |
| clarkb | Shrews: it blocks on all the db lookups aiui | 20:56 |
| clarkb | Shrews: it will asynchronously handle new connections but once each one hits db it block | 20:56 |
| Shrews | clarkb: zuul-web does not use a database | 20:57 |
| clarkb | isn't that what gives the js its data? which is db queries? | 20:57 |
| Shrews | i know nothing about the js | 20:57 |
| Shrews | I *do* know that it now uses gearman much more, and the code that has been added for that is not doing things in the asyncio manner | 20:58 |
| clarkb | Shrews: ya basically anything with async annotation like the methods taht call gearman are async | 20:59 |
| clarkb | Shrews: but the gearman itself will be synchronous | 20:59 |
| clarkb | so once you entire that method and call gearman you wait | 20:59 |
| Shrews | clarkb: what you are supposed to do is put the gearman calls on the event loop so that it does NOT wait | 20:59 |
| Shrews | then it can switch tasks to something else until gearman responds | 21:00 |
| clarkb | I remember calling this out in review particularly for the db side of things | 21:01 |
| clarkb | but it was working and not falling over so people seemed happy with it | 21:01 |
| clarkb | and looks like gearman is similar situation | 21:01 |
| Shrews | it's easily fixable. i already have an example in the log streaming code that does what we need | 21:02 |
| Shrews | i'll work something up | 21:02 |
| clarkb | also ya doesn't look like zuul-web serves the contents for the build history | 21:02 |
| * clarkb goes looking for that | 21:02 | |
| Shrews | then go back to fixing the other thing i was looking at (adding command socket support to zuul-web) | 21:03 |
| *** harlowja has joined #zuul | 21:03 | |
| Shrews | http://git.openstack.org/cgit/openstack-infra/zuul/tree/zuul/web/__init__.py#n85 | 21:04 |
| Shrews | fwiw | 21:04 |
| clarkb | let me dig up the db side of things so we can address that too if necessary | 21:05 |
| clarkb | Shrews: driver/sql/sqlconnection.py line 201 | 21:07 |
| clarkb | and it is via zuul-web just not in zuul/web | 21:08 |
| Shrews | clarkb: where is that called from in zuul-web? | 21:09 |
| clarkb | Shrews: lines 258-260 in zuul/web/__init__.py | 21:09 |
| clarkb | Shrews: it dynamically builds the groups and handlers from the list of connections | 21:09 |
| clarkb | s/groups/routes/ | 21:09 |
| Shrews | clarkb: ok, that makes me REALLY sad | 21:12 |
| Shrews | the 'async' keyword is not magical | 21:12 |
| clarkb | yes, I know I explicitly called it out in review but people seemed reasonably happy with it. fwiw I'm happy to know there is a straightforward solution to the problem which we can use (I think updating to use that solution is a good thing) | 21:13 |
| Shrews | it basically defines a unit of work. it doesn't make things asynchronous. you have to do that manually | 21:13 |
| Shrews | clarkb: you've just doubled the work load, but yeah, it can be fixed | 21:14 |
| Shrews | *sigh* | 21:14 |
| clarkb | Shrews: you don't have to fix it :) | 21:14 |
| clarkb | Shrews: if you like I can make a change on yours as a follow up | 21:14 |
| clarkb | (I'll use your fix for gearman as a template for fixing $sqldriver) | 21:14 |
| Shrews | i'll fix the stuff directly in zuul/web/__init__.py as an example | 21:14 |
| Shrews | you or whoever added the brokenness can then fix the other stuff :) | 21:15 |
| Shrews | but we should probably watch for this in future changes | 21:15 |
| Shrews | corvus: ^^^ | 21:15 |
| corvus | Shrews: ack. this is new to many of us. :) | 21:16 |
| Shrews | and I'm having to re-learn it thanks to pabelanger! bah :-P | 21:16 |
| corvus | hopefully once we have the "access something from $external_resource" down, we won't have to figure out this pattern too much more. we'll probably have to do it for zk next, but after that, i'm hoping it doesn't grow any more tendrils (and starts to lose some). | 21:17 |
| clarkb | basically asyncio is explicit coordination there is no preemption | 21:18 |
| clarkb | so if you block on a non awaited function call you will block there until done | 21:18 |
| clarkb | this has the upside of making context switching incedibly explicit | 21:19 |
| corvus | yep. so you'd best know how all the functions you call are implemented internally. :) | 21:19 |
| clarkb | the downside is you must be very careful to do it all yourself or you will block | 21:19 |
| Shrews | yep yep yep | 21:19 |
| Shrews | like i said, not magical... just verbose :) | 21:20 |
| *** gouthamr has joined #zuul | 21:20 | |
| clarkb | Shrews: going forward I will make that a -1 instead of a nice to have review point :) | 21:21 |
| clarkb | actually I may have reviewed it after the fact? I do recall bringing it up with mordred and tristanC though | 21:21 |
| Shrews | clarkb: and i'll try to pay more attention to the changes there too | 21:21 |
| corvus | Shrews, clarkb: in gear, we can probably accomplish this with a subclass of Job which overrides handleWorkComplete (and friends) to put an event on the loop. | 21:22 |
| corvus | (er, to be clear, we can do the subclass in zuul-web. we might *also* want to port that to gear itself to make it nicer for asyncio users, but should not be required. we should start in zuul) | 21:23 |
| mordred | corvus: ++ | 21:23 |
| corvus | in fact, that is exactly how zuul's executor client works today -- it does that to put job completion events on *zuul's* event queue | 21:25 |
| corvus | see zuul/executor/client.py | 21:25 |
| Shrews | corvus: well, the GearmanHandler is passed in an already instantiated rpc client. I think I can just make a method inside there that can be reusable within that class context | 21:26 |
| corvus | (though, i lied, it's not Job we would subclass, it's Client) | 21:26 |
| corvus | Shrews: can you subclass the rpc client? | 21:26 |
| corvus | Shrews: oh, er, that doesn't help | 21:27 |
| corvus | Shrews: the rpc client creats its own plain client | 21:27 |
| corvus | it's the RPCClient.gearman attribute which would need to be the subclassed gear.Client | 21:28 |
| corvus | anyway, i'll stop trying to help :) | 21:28 |
| Shrews | corvus: NEVER stop trying to help | 21:28 |
| Shrews | corvus: i mean, yes, stop now. but otherwise... :-P | 21:29 |
| corvus | :) | 21:29 |
| Shrews | i will solve this with containers | 21:29 |
| corvus | :( | 21:29 |
| Shrews | https://review.openstack.org/559326 needs a final +2 to fix streaming multibyte chars | 21:33 |
| clarkb | Shrews: I'll rereview it now | 21:33 |
| clarkb | Shrews: done | 21:35 |
| Shrews | w00t. thx clarkb | 21:36 |
| corvus | late +2 here :) | 21:36 |
| *** hashar has quit IRC | 21:37 | |
| * Shrews gives a late w00t to corvus | 21:40 | |
| Shrews | corvus: is the test_crd_check_unknown test failure a known random thing? http://logs.openstack.org/26/559326/3/gate/zuul-tox-py35/403b698/testr_results.html.gz | 21:53 |
| corvus | Shrews: i can't recall | 21:53 |
| Shrews | that's totally unrelated afaict. imma recheck | 21:54 |
| corvus | Shrews: that error looks like maybe the test is missing a sync point | 21:54 |
| *** rlandy has quit IRC | 21:55 | |
| jhesketh | Morning | 21:57 |
| openstackgerrit | Clark Boylan proposed openstack-infra/zuul master: Make db queries asynchronous in zuul-web https://review.openstack.org/559852 | 21:59 |
| clarkb | Shrews: corvus ^ thats a first stab at addressing this for the generic handlers | 21:59 |
| clarkb | s/generic/connection/ I tried to make it somewhat generic so that you could provide arbitrary connections handlers and not care about implementation details too much | 21:59 |
| clarkb | also mysql is the one thing my local test bed lacks so unsure if that is working currently | 21:59 |
| corvus | it's zuul meeting time in #openstack-meeting-alt | 22:00 |
| Shrews | clarkb: looks about right. probably want to catch the timeout exception there | 22:00 |
| clarkb | Shrews: we already catch it a level higher and set it to a 500 result which I think is sufficient? | 22:01 |
| Shrews | oh probably | 22:01 |
| clarkb | Shrews: I marked it inline to make it more clear | 22:02 |
| clarkb | Shrews: we might also need to make the connection begin() call async but I expect that to be far less of an issue over time (since it will reuse connections iirc) | 22:03 |
| clarkb | ya begin() is more about starting a transaction and the connection details are more abstracted so if that becomes an issue may need more explict connection management instead | 22:06 |
| Shrews | ok, wow. i can't explain all of these failures: http://logs.openstack.org/26/559326/3/gate/zuul-tox-py35/42541bb/testr_results.html.gz | 22:23 |
| clarkb | Shrews: address already in use | 22:27 |
| clarkb | its a race between multiple test processes using port 9000? | 22:27 |
| clarkb | Shrews: can you use port 0 then ask the server for what port it got after its bound? | 22:28 |
| Shrews | clarkb: the test i added does that | 22:29 |
| Shrews | clarkb: oh, not for zuulweb | 22:29 |
| Shrews | yeah | 22:29 |
| clarkb | ya I seemed to recall the finger side doing that | 22:31 |
| clarkb | but its the gateway that conflicts | 22:31 |
| clarkb | er dashboard | 22:31 |
| *** gouthamr has quit IRC | 22:43 | |
| *** JasonCL has quit IRC | 22:44 | |
| *** gouthamr has joined #zuul | 22:44 | |
| *** ChanServ changes topic to "Discussion of the project gating system Zuul | Website: https://zuul-ci.org/ | Docs: https://zuul-ci.org/docs/ | Source: https://git.zuul-ci.org/ | Channel logs: http://eavesdrop.openstack.org/irclogs/%23zuul/ | Weekly updates: https://etherpad.openstack.org/p/zuul-update-email" | 22:53 | |
| *** JasonCL has joined #zuul | 22:59 | |
| *** patriciadomin has quit IRC | 23:04 | |
| *** patriciadomin has joined #zuul | 23:05 | |
| openstackgerrit | Merged openstack-infra/zuul master: Fix streaming decoding boundaries https://review.openstack.org/559326 | 23:11 |
| *** gouthamr has quit IRC | 23:23 | |
| *** JasonCL has quit IRC | 23:46 | |
| *** JasonCL has joined #zuul | 23:49 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!