| corvus | (which is a thing i think we want anyway) | 00:00 |
|---|---|---|
| * SpamapS securing gearman traffic now that we'll have > 1 executor | 00:00 | |
| SpamapS | corvus: yeah, with the FF thing, I was thinking we'd have to have state. | 00:00 |
| corvus | SpamapS: excellent idea! | 00:00 |
| mordred | also - I think skip-clean-check + zuul gating + github status flags is a potentially nice scenario for folks- the status flags make people learn to not force-merge something until it at LEAST has a green check - but gating lets them know that if they're impatient they can click approve and it'll get merged the instnat it DOES have a clean flag | 00:00 |
| SpamapS | you'd have to remember the git state that passed + the jobs | 00:00 |
| corvus | (the secure gearman thing) | 00:00 |
| SpamapS | corvus: aye | 00:00 |
| corvus | SpamapS: yeah, and state for any required-projects | 00:01 |
| mordred | I obviously think removing force-merge from people is preferrable - but understand there are circumstances where it's unpossible - or maybe undesirable | 00:01 |
| corvus | mordred: it's easy: you wait until they're asleep | 00:01 |
| mordred | corvus: :) | 00:01 |
| SpamapS | mordred: I also think there can be a fast-gate | 00:02 |
| mordred | you're a fast gate | 00:02 |
| SpamapS | which is like.. do whatyou can... make sure zuul's config still works.. but... landthis change more or less nao | 00:02 |
| SpamapS | I've a fast gait, it's true. | 00:02 |
| mordred | I have a fat goiter | 00:02 |
| mordred | and some phat gout | 00:03 |
| * SpamapS hands mordred some iodized salt | 00:03 | |
| SpamapS | and some clean socks | 00:03 |
| mordred | mmm. smakelijk! | 00:03 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Add 'build' method to API https://review.openstack.org/592225 | 00:23 |
| SpamapS | hrm | 00:30 |
| SpamapS | I'm trying to do ssl for gearman | 00:30 |
| SpamapS | but it never sends the cert | 00:31 |
| * SpamapS decides to roll back and head to bar | 00:31 | |
| tristanC | corvus: recording the git sha's of the required project at check and doing the comparaison with the last git shas of the gate queue wouldn't be enough for that fast-forward-gate? | 01:25 |
| corvus | tristanC: yeah, something like that. we just need to store all that info in the build db -- and we need to require the build db. so i'd probably put implementing that after we decide whether the build db should be in sql or zk. | 01:35 |
| corvus | tristanC: but also, i think if we add it, we should have a nice long thing in the docs about how *external* dependencies may still have changed. so it's not 100% guaranteed not to break or wedge the project's gate. | 01:37 |
| *** jiapei has joined #zuul | 01:37 | |
| corvus | tristanC: it may require zuul to push merges (otherwise the shas may be different) | 01:38 |
| tristanC | that sounds like relational data... are you suggesting zk to remove one required service? | 01:38 |
| corvus | tristanC: i haven't thought about the schema. but yes, i think we should look into storing the build db in zk to reduce the extra dependency. i have no idea if it's a good idea -- only that we should consider it. :) | 01:39 |
| corvus | okay this is really weird -- see my inline comments on https://review.openstack.org/592225 | 01:41 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Add 'build' method to API https://review.openstack.org/592225 | 01:43 |
| corvus | i guess there's a bug in flake8 that cause it to miss that. weird. | 01:43 |
| tristanC | corvus: that is weird, there doesn't seem any reason to happen | 01:44 |
| tristanC | corvus: that zk or sql decision would be good to know, i'd like to add a nodeset and node table to record to record the build resources used by a job | 01:46 |
| tristanC | which also sounds like relational data, so i wonder how it will look through the zk module | 01:47 |
| tristanC | or build resouruces could be stored unstructured, but then how to do reverse lookup (e.g. query which job used that specific node) | 01:48 |
| corvus | tristanC: good point, we should collect the kinds of queries we might want -- because it might be easy to query in one direction (buildset -> build -> node) in zk, but not the other direction (start with a node id) | 01:50 |
| openstackgerrit | neilsun proposed openstack-infra/zuul master: Add type check for zuul conf https://review.openstack.org/591917 | 02:22 |
| *** rlandy|afk is now known as rlandy | 02:44 | |
| *** rlandy has quit IRC | 02:44 | |
| *** jpena|off has quit IRC | 02:45 | |
| *** NeilSun has joined #zuul | 03:04 | |
| *** NeilSun has quit IRC | 03:04 | |
| *** NeilSun has joined #zuul | 03:16 | |
| *** pcaruana has quit IRC | 05:26 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: WIP web: rewrite interface in react https://review.openstack.org/591604 | 06:34 |
| tristanC | corvus: mordred: last PS adds console stream and builds page filter. It seems like it's almost feature complete now, just missing filters settings from querystring or cookies | 06:35 |
| tristanC | note that the console stream uses a smooth scrolling, and the builds filter now support more columns | 06:36 |
| *** jiapei has quit IRC | 06:37 | |
| *** pcaruana has joined #zuul | 06:48 | |
| *** gtema has joined #zuul | 07:16 | |
| *** jpena has joined #zuul | 07:31 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/zuul-jobs master: Build releasenotes when tox.ini changes https://review.openstack.org/592298 | 07:35 |
| openstackgerrit | Markus Hosch proposed openstack-infra/nodepool master: Move sphinx + deps to doc/requirements.txt https://review.openstack.org/591565 | 07:58 |
| *** darkwisebear has joined #zuul | 07:59 | |
| tristanC | corvus: mordred: so using path routing is simple: just replace "HashRouter" with "BrowserRouter" in the index.js file. | 08:09 |
| tristanC | However this need complex server side rules to make the static files not load from the scoped path, and as monty said, it's going to be even more tricky for swift based hosting | 08:10 |
| *** aspiers[m] has quit IRC | 08:11 | |
| tristanC | so i'll keep the change api route discovery from href (with the trick to split at '/t/'), but i'll recommend we keep the hashrouter to make deployment simpler | 08:12 |
| *** darkwisebear has quit IRC | 08:17 | |
| *** darkwisebear has joined #zuul | 08:18 | |
| *** electrofelix has joined #zuul | 08:20 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: WIP web: rewrite interface in react https://review.openstack.org/591604 | 08:38 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: WIP web: use BrowserRouter https://review.openstack.org/592321 | 08:38 |
| *** darkwisebear has quit IRC | 08:39 | |
| *** ianychoi_ has quit IRC | 08:42 | |
| *** panda|off is now known as panda | 08:56 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: WIP web: rewrite interface in react https://review.openstack.org/591604 | 09:37 |
| *** elyezer_ has quit IRC | 09:40 | |
| *** elyezer_ has joined #zuul | 09:42 | |
| *** elyezer_ has quit IRC | 09:48 | |
| *** elyezer_ has joined #zuul | 09:50 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/zuul-jobs master: Cleanup temporary directories https://review.openstack.org/592340 | 10:42 |
| openstackgerrit | Ian Wienand proposed openstack-infra/zuul-jobs master: Create a download script https://review.openstack.org/592341 | 10:42 |
| openstackgerrit | Ian Wienand proposed openstack-infra/zuul-jobs master: upload-logs-swift: Cleanup temporary directories https://review.openstack.org/592340 | 10:47 |
| openstackgerrit | Ian Wienand proposed openstack-infra/zuul-jobs master: upload-logs-swift: Create a download script https://review.openstack.org/592341 | 10:47 |
| *** panda is now known as panda|lunch | 11:02 | |
| *** jpena is now known as jpena|lunch | 11:08 | |
| *** elyezer_ has quit IRC | 11:11 | |
| *** elyezer_ has joined #zuul | 11:12 | |
| *** jpena|lunch is now known as jpena | 12:14 | |
| *** samccann has joined #zuul | 12:35 | |
| *** rlandy has joined #zuul | 12:41 | |
| *** NeilSun has quit IRC | 12:54 | |
| *** elyezer_ has quit IRC | 12:56 | |
| *** elyezer_ has joined #zuul | 13:03 | |
| mordred | tristanC: going through the patch now - the updated pages look pretty awesome! love the scrolling in the stream page | 13:08 |
| openstackgerrit | Markus Hosch proposed openstack-infra/nodepool master: Add list of metrics provided to statsd https://review.openstack.org/590233 | 13:09 |
| tristanC | mordred: nice! the console stream may need some refresh rate delay though, the scrolling may jitter because it's activated for each line. | 13:12 |
| tristanC | and for builds query filter, the api returns 500 when adding multiple filter for a single key | 13:13 |
| corvus | the smooth scrolling looks nice, but i wonder if we can speed it up? when it's moving, it's difficult to read, so less time spent moving means more time available to read | 13:13 |
| mordred | tristanC: the builds query filter is all server-side it seems, yeah? | 13:15 |
| corvus | i'm glad that BrowserRouter is looking promising -- it makes for much friendlier urls, and i'd like to get to the point where "zuul.openstack.org/jobs/devstack" is a url people are comfortable with | 13:15 |
| mordred | of course it is - that was a silly question | 13:15 |
| tristanC | mordred: yes, as it was before | 13:15 |
| mordred | tristanC: I had an idea ... | 13:16 |
| mordred | tristanC: (I recommend running and hiding now) | 13:16 |
| mordred | tristanC: the manifest.json that's there for progressive web apps ... I wonder if we could also read values from it from the dashboard itself - and offer it as a way a deployer could set the api url location | 13:17 |
| mordred | tristanC: in addition to the build-time option | 13:17 |
| tristanC | mordred: probably, not sure how it's supposed to be loaded though | 13:17 |
| mordred | tristanC: so that a deployer _could_ just run from the tarball we build - but have config management splat down an updated manifest | 13:17 |
| mordred | might be a thing a deployer wants to do anyway, to update title and description - since those make their way in to PWAs saved to mobile devies | 13:18 |
| mordred | devices | 13:18 |
| mordred | so I could see, for instance, softwarefactory deploying one with a title "softwarefactory Zuul" and openstack doing "openstack zuul" - and then as a user me saving both as different icons to my android phone | 13:19 |
| mordred | just a thought for the future - I don't think we need to do anythign with that today | 13:19 |
| tristanC | mordred: it will also adds an extra http call on load in addition to the /api/info | 13:20 |
| tristanC | corvus: it does and we can probably make it work with the right rewrite rule, but i don't think it will be possible to just host the files on swift then | 13:21 |
| *** darkwisebear has joined #zuul | 13:21 | |
| tristanC | corvus: is zuul.openstack.org/#/job/devstack really less friendlier? | 13:21 |
| mordred | btw - the status page looks great on my phone | 13:22 |
| *** elyezer_ has quit IRC | 13:22 | |
| tristanC | mordred: oh yes, though the "kebab" collapse menu isn't working yet | 13:22 |
| mordred | zomg. I just navigated to the preview dashboard on my phone, clicked "save to home screen" - and I now have a zuul "app" on my phone | 13:24 |
| mordred | I agree - the collapse menu it totally not working :) | 13:24 |
| corvus | tristanC: yes it is -- # or #! don't mean anything to humans (or if they do, they mean "an anchor in a page") so they make that part of the url unintuitive. you'd have to expect people to remember where in the url to place the #, and whether it should be # or #! or what | 13:24 |
| *** elyezer_ has joined #zuul | 13:25 | |
| corvus | so what's the thing about the static external option (eg swift) that makes it tricky for browser router? | 13:27 |
| mordred | they don't know how to serve the file paths | 13:27 |
| mordred | it's the same with the current dashboard for multi-tenant dashboard | 13:27 |
| corvus | oh, so this isn't an additional problem, it's just the same caveat? | 13:28 |
| mordred | yah | 13:28 |
| mordred | well - we'll need to do some more work to make single-tenant work with this setup | 13:28 |
| mordred | because we're just letting react-scripts handle our webpack config for us | 13:28 |
| tristanC | corvus: yes, /job/devstack isn't a file | 13:28 |
| mordred | I'm not sure we should do that work until someone actually wants to deploy the dashboard to swift, because I really like the lower complexity of just having the webpack config be whatever react-scrits think it should be | 13:29 |
| mordred | the current actual deployment scenarios we have all involve either serving from zuul itself or serving from apache with rewrites, all of which work with the react-scripts approach - and should work fine with browserrouter | 13:30 |
| tristanC | also, even when serving the index.html for non existing file path, then the static file are sill loaded incorrectly, e.g. from "/job/devstack/static/bundle.js" | 13:32 |
| corvus | yeah, i agree those are the things we should focus on; we don't need to trade in complexity for a hypothetical deployment on swift | 13:32 |
| tristanC | corvus: well the complexity comes from using browserrouter (which doesn't work without server settings), the easier way is using hashrouter (which would also work fine on swift) | 13:33 |
| tristanC | for the static file location, it's either relative to the request path (it's the "homepage": "./" setting in package.json) | 13:33 |
| corvus | tristanC: we accept complexity in the code to make things friendlier for users. | 13:34 |
| corvus | tristanC: users > deployers > developers -- that's the order of importance here | 13:34 |
| tristanC | either it's an absolute path (default to /static) which break sub path deployment | 13:34 |
| corvus | tristanC: regarding complexity, i was agreeing with mordred we don't have to solve all of the static-external deployment issues now since we don't have a use case for them | 13:34 |
| corvus | tristanC: but we should accept the complexity of browserrouter because it improves the user experience | 13:35 |
| tristanC | alright, so i think the main issue is static file location that can no longer be relative to requested path | 13:36 |
| corvus | tristanC: in which case does that happen? | 13:38 |
| tristanC | corvus: when you load "/job/devstack", even if the server serve "/index.html", the static files are loaded from "/job/static/" because the homepage is set to "./" in the packages.json | 13:39 |
| tristanC | if we don't set homepage to "./", then it loads static file from "/static" by default iirc | 13:40 |
| *** darkwisebear has quit IRC | 13:41 | |
| tristanC | which i think wouldn't work for the job preview file for example | 13:41 |
| *** elyezer_ is now known as elyezer | 13:42 | |
| tristanC | on the other hand we could json patch the homepage to "./" for job preview | 13:49 |
| tristanC | eventually, it will be an issue for sub path deployment, in which case deployer will have to put zuul static file in /static, or re-build the ui using the desired homepage path | 13:50 |
| tristanC | i mean, that's my current understanding of browserrouter deployment, there may be other solution | 13:52 |
| *** elyezer has quit IRC | 13:52 | |
| *** elyezer has joined #zuul | 13:54 | |
| SpamapS | hm.. does the github endpoint live in zuul-web now? | 13:56 |
| * SpamapS is trying to set up a new github app and going a little cross-eyed | 13:56 | |
| corvus | SpamapS: yes, it's been there for a while :) | 14:01 |
| corvus | SpamapS: we're down to only one webapp | 14:02 |
| SpamapS | Good good | 14:02 |
| SpamapS | I thought so | 14:02 |
| SpamapS | just couldn't read that ball while juggling it ;) | 14:02 |
| pabelanger | could I get a +3 on https://review.openstack.org/591457/ adds node info into emit-job-header role | 14:22 |
| *** elyezer has quit IRC | 14:30 | |
| *** elyezer has joined #zuul | 14:32 | |
| *** darkwisebear has joined #zuul | 14:38 | |
| rcarrillocruz | so folks, i'm' getting a host authenticity error https://ansible.softwarefactory-project.io/logs/7/7/bc3f0e852d069096ae27ff3c8b2d7bc3a83f6447/check/ansible-role-tests-vqfx-devel-py2/1ed6d14/job-output.json.gz | 14:46 |
| rcarrillocruz | trying to understand the logic | 14:46 |
| rcarrillocruz | i see zuul server.py expects a known_hosts prepopulated | 14:46 |
| rcarrillocruz | how's the workflow till it reaches the point of doing the ansible -m setup | 14:47 |
| mordred | rcarrillocruz: the known hosts entry comes to zuul from nodepool - and zuul executor server writes whatever nodepool sends it into the known_hosts file | 14:49 |
| mordred | rcarrillocruz: line 1451 in zuul/executor/server.py is where it writes the file | 14:49 |
| rcarrillocruz | so that means, the ssh-keyscan from nodepool runs, but some garbage is put on the record for that node, therefore zuul fails as it doesn't have it on that file ? | 14:50 |
| pabelanger | maybe something to do with ssh-ed25519 key? | 14:50 |
| mordred | rcarrillocruz: yah - if ssh-keyscan is not working I would expect things to go poorly | 14:50 |
| rcarrillocruz | what i'm guessing yeah, maybe it expects rsa | 14:50 |
| rcarrillocruz | ? | 14:50 |
| rcarrillocruz | it's a net appliance, so snowflakes ahead | 14:50 |
| mordred | oh goodie :) | 14:51 |
| * rcarrillocruz looks at nodeutils | 14:51 | |
| mordred | you can set host-key-checking to false on the pool | 14:51 |
| mordred | and it will skip the keyscan | 14:51 |
| rcarrillocruz | gah...thing is we share the pool with others in SF | 14:52 |
| rcarrillocruz | pabelanger , tristanC , nhicher ^ | 14:52 |
| rcarrillocruz | would you be ok with that | 14:53 |
| mordred | I just did an ssh-keyscan on mirror02.us-west-1.packethost.openstack.org which has an ecdsa key | 14:53 |
| pabelanger | rcarrillocruz: I think we can create a specific pool for it | 14:53 |
| rcarrillocruz | or maybe it's the right time to just use vexxhost pool pabelanger | 14:53 |
| mordred | andit got both the ecdsa-sha2-nistp256 and the ssh-rsa keys | 14:53 |
| rcarrillocruz | mordred: i can do a quick ssh-keyscan and paste, sec | 14:53 |
| pabelanger | rcarrillocruz: sure, we can test that | 14:53 |
| mordred | rcarrillocruz: yah - I would expect it to work - however, it's possible the net appliance is doing something weird with ssh hostkeys | 14:54 |
| mordred | by weird, I do of course mean broken - but it's a vendor appliance, so we should expect it to do basic networking things incorrectly :) | 14:54 |
| rcarrillocruz | nm, i even had a keyscan from a couple hours ago | 14:55 |
| rcarrillocruz | http://paste.openstack.org/show/728202/ | 14:55 |
| rcarrillocruz | it does return an rsa | 14:55 |
| rcarrillocruz | ... | 14:55 |
| mordred | ah! | 14:57 |
| mordred | rcarrillocruz: ssh-keyscan does not read ed25519 keys by default | 14:58 |
| mordred | it can be specified | 14:58 |
| mordred | ssh-keyscan -t rsa,ecdsa,ed25519 | 14:58 |
| mordred | so maybe this wants to either be a thing we just add to the keyscan command we do - or perhaps make it a config option if we think automatically scanning for ed25519 keys is something we shouldn't do | 14:59 |
| rcarrillocruz | sohmm, wait... but nodeutils doesn't seem to use ssh-keyscan, it does use paramiko for doing the scan? | 14:59 |
| mordred | yah. Im guessing same thing - looking | 15:00 |
| Shrews | comment there says only rsa is returned by paramiko | 15:01 |
| rcarrillocruz | yah... | 15:01 |
| mordred | https://github.com/paramiko/paramiko/issues/626 | 15:01 |
| Shrews | http://git.openstack.org/cgit/openstack-infra/nodepool/tree/nodepool/nodeutils.py#n110 | 15:01 |
| rcarrillocruz | line 110 | 15:01 |
| mordred | https://github.com/paramiko/paramiko/issues/626#issuecomment-297272384 | 15:02 |
| rcarrillocruz | will spin up an instance on my cloud account and try to run that snippet of paramiko, to see how it blows up | 15:03 |
| mordred | Shrews: do you remember why we did it with paramiko and not with keyscan? it was just to avoid needing openssh client installed on launchers? | 15:04 |
| *** darkwisebear has quit IRC | 15:05 | |
| mordred | I'm also curious as to why paramiko is finding an rsa key but ssh in ansible is finding the ed25519 | 15:06 |
| rcarrillocruz | i would have sworn we had ssh-keyscan at some point ? | 15:06 |
| pabelanger | maybe add debug line for nodepool for list host keys found | 15:07 |
| rcarrillocruz | gah | 15:09 |
| rcarrillocruz | so yeah | 15:09 |
| rcarrillocruz | >>> sock.connect(('38.145.33.81', 22)) | 15:09 |
| rcarrillocruz | >>> t = paramiko.transport.Transport(sock) | 15:09 |
| rcarrillocruz | >>> t.start_client() | 15:09 |
| rcarrillocruz | >>> t.get_remote_server_key() | 15:09 |
| rcarrillocruz | <paramiko.ed25519key.Ed25519Key object at 0x7f8237637b50> | 15:09 |
| rcarrillocruz | it doesn't return rsa | 15:09 |
| Shrews | mordred: i do not remember. either that was original code, or i seem to (maybe incorrectly) recall pabelanger adding something to that bit | 15:10 |
| Shrews | pabelanger: do you recall anything about that? | 15:10 |
| mordred | rcarrillocruz: weird - is the opposite thign happening then? is openssh preferring rsa? | 15:11 |
| rcarrillocruz | lulz, so... http://paste.openstack.org/show/728205/ | 15:11 |
| mordred | so keyscan is finding the ed25519 key but ansible is trying to do rsa | 15:12 |
| mordred | rcarrillocruz: is it 'rsa' instead of 'ssh-rsa' maybe? | 15:12 |
| pabelanger | rcarrillocruz: try again | 15:12 |
| pabelanger | that might be ssh trying to boot up | 15:12 |
| pabelanger | Shrews: let me look | 15:12 |
| rcarrillocruz | niet, " Incompatible ssh peer (no acceptable host key)" | 15:13 |
| rcarrillocruz | when calling start_client | 15:14 |
| rcarrillocruz | i think the banner is a net appliance shenanigan | 15:14 |
| rcarrillocruz | that's a common random failure when dealing with this kind of stuff | 15:14 |
| Shrews | pabelanger: hrm, looks like i added that bit of code. not sure if that was in v2 or not | 15:15 |
| pabelanger | Shrews: https://review.openstack.org/445055/ | 15:15 |
| pabelanger | it original was keyscan | 15:15 |
| rcarrillocruz | ok | 15:15 |
| pabelanger | but corvus asked for paramiko | 15:15 |
| rcarrillocruz | so i close socket | 15:15 |
| rcarrillocruz | start from scratch | 15:15 |
| rcarrillocruz | it worked now | 15:15 |
| pabelanger | and me too | 15:15 |
| Shrews | lol | 15:15 |
| rcarrillocruz | http://paste.openstack.org/show/728208/ | 15:16 |
| rcarrillocruz | mordred: ^ | 15:16 |
| pabelanger | Shrews: mordred: so, maybe we do use ssh keyscan :) | 15:16 |
| rcarrillocruz | so folks, are you ok adding that preferred_keys line | 15:16 |
| rcarrillocruz | i.e. forcing to give rsa | 15:16 |
| rcarrillocruz | since well, we expected it to be rsa per comment anyways (even if it wasn't returning rsa) | 15:16 |
| pabelanger | rcarrillocruz: so, why don't we get rsa without it? | 15:17 |
| pabelanger | or do we, but it is malformed | 15:17 |
| corvus | rcarrillocruz: i'm confused -- we believe that nodepool is returning the ed key, and ansible is trying to use the ed key -- where does rsa come into this? | 15:18 |
| rcarrillocruz | no, that was me doing a snippet of code to force paramiko to return rsa | 15:18 |
| rcarrillocruz | the failure shows ed | 15:18 |
| corvus | rcarrillocruz: right, i'm saying forget about rsa -- why doesn't ed work? | 15:19 |
| rcarrillocruz | so i'm not sure if it's because of it and it should be rsa | 15:19 |
| corvus | nodepool is using ed. ansible is using ed. why isn't it working? | 15:19 |
| rcarrillocruz | i don't know , but if you are saying that ed is ok to go, then pabelanger we could look locally for the known hosts and see what's going on on nodepool? | 15:19 |
| corvus | i think that, and seeing what known_hosts zuul wrote out, might be worthwhile. because i'm not sure we fully understand the problem yet, and an incomplete fix could make things worse | 15:20 |
| *** pbrobinson has quit IRC | 15:21 | |
| corvus | if we wanted to change nodepool's behavior, i'd suggest that we'd probably want to try to scan *all* the key types and add all of them to known hosts. that's probably the only safe thing to do. | 15:21 |
| corvus | (and that, aiui from mordred's bug report, would mean looping over all the types we know of and attempting a connection on each) | 15:21 |
| *** pbrobinson has joined #zuul | 15:21 | |
| mordred | corvus: yah - and to do that, we'd need to either switch to ssh-keyscan or we'd have to do that ina loop in python - but we have to use a private variable to accomplish it | 15:21 |
| rcarrillocruz | yeah, that's what it looks like, putting preferred_keys over a loop | 15:21 |
| mordred | corvus: yah | 15:22 |
| mordred | that said - I would like to understand a bit more where it's falling down | 15:22 |
| rcarrillocruz | but anyways, if you say ed is fine, then the issue must come from known_hosts | 15:22 |
| corvus | mordred: i think preferred_keys in a loop as rcarrillocruz says should do it without a private var? | 15:22 |
| mordred | corvus: no, t._preferred_keys = ['ssh-rsa'] | 15:22 |
| corvus | mordred: oooh. "neat" | 15:23 |
| mordred | it's not a public variable | 15:23 |
| mordred | yeah | 15:23 |
| mordred | "yay" | 15:23 |
| corvus | rcarrillocruz: yeah, i don't think nodepool/zuul want to care about which keys are used, so if your device wants to use an ed key, we shouldn't get in the way. | 15:23 |
| mordred | ++ | 15:23 |
| pabelanger | so, if you have an RSA ssh key, you also need the RSA hostkey on the remote side right? | 15:29 |
| pabelanger | because, if so. nodepool is only returning a ssh-ed25519 hostkey from the server | 15:30 |
| pabelanger | and, my guess is the ssh key rcarrillocruz is using might be ssh-rsa | 15:30 |
| pabelanger | so, when ansible connection, it fails due to ssh-rsa host key missing in known_hosts | 15:30 |
| pabelanger | rcarrillocruz: if you generate a ssh-ed25519 ssh key, I think it might work as expected | 15:31 |
| pabelanger | https://github.com/paramiko/paramiko/issues/626 | 15:32 |
| pabelanger | doesn't look like we can get all hostkeys with paramiko right now | 15:32 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: IGNORE testing pbrx https://review.openstack.org/592550 | 15:33 |
| Shrews | mordred: ^^ is the pbrx depends on | 15:34 |
| mordred | corvus, Shrews, pabelanger: what do y'all think about announcing pbrx patches in here | 15:34 |
| mordred | at least for the period of time while "publish zuul images" is dependent on pbrx work | 15:35 |
| Shrews | mordred: i'd definitely like them "somewhere" | 15:35 |
| rcarrillocruz | pabelanger: 'if you have an ssh rsa key', i guess you mean the zuul executor, that's what is doing the gather facts that fails in the job | 15:36 |
| pabelanger | rcarrillocruz: oh, yes right | 15:36 |
| rcarrillocruz | and yes, sf zuul executor has an rsa key iiuc | 15:37 |
| mordred | Shrews, pabelanger, corvus: remote: https://review.openstack.org/592554 Add pbrx patch announcements to #zuul | 15:37 |
| pabelanger | rcarrillocruz: so, let me see if we can make nodescan return multiple hostkeys | 15:37 |
| corvus | pabelanger: keys used for authentication and encryption are different. the user can have an rsa key and connect to a host with an ed key | 15:39 |
| corvus | pabelanger: the first questions to answer are: it looks like nodepool found an ed key -- did it? dit it pass it to zuul and did zuul write it to known_hosts correctly? if so, why did ansible fail? | 15:43 |
| Shrews | so, i thought paramiko ONLY returned rsa, which is what we pass to zuul. but ansible seems to be expecting ed25519... so i'm confused | 15:45 |
| Shrews | but i'm also split-brain right now on two things | 15:45 |
| corvus | Shrews: it may be that paramiko returns the first key, which just happened to be rsa when we were testing it and wrote that comment? rcarrillocruz did a paramiko test by hand and got an ed key by default. but anyway, yes, that is the first question to answer :) | 15:47 |
| Shrews | ah | 15:47 |
| Shrews | so maybe a paramiko update "fixed" things | 15:47 |
| corvus | Shrews: apparently it only added ed support a year or two ago? so that's quite possible | 15:48 |
| rcarrillocruz | i really think the issue is on the known_hosts | 15:48 |
| rcarrillocruz | like | 15:48 |
| rcarrillocruz | this is what i tried | 15:48 |
| pabelanger | corvus: yah, we only get back ssh-ed25519 from server, and pass that into zk | 15:48 |
| rcarrillocruz | i ran a ssh-keyscan against my appliance node | 15:48 |
| pabelanger | working to see if known_host file is written properly | 15:48 |
| rcarrillocruz | put the ed entry on ssh/known_hosts | 15:48 |
| rcarrillocruz | ansible -m setup works, it doesn't fail or asks for yay/nay to accept host | 15:48 |
| rcarrillocruz | wellll | 15:50 |
| rcarrillocruz | wait | 15:50 |
| rcarrillocruz | the connection-type of the node is network_cli, i.e. it should be blacklisted for gathering facts purposes | 15:51 |
| rcarrillocruz | git blame 6eda43970 | 15:52 |
| *** pbrobinson has quit IRC | 15:52 | |
| rcarrillocruz | i added that specifically, as network nodes are not OS nodes, therefore no python, therefore ansible -m setup failed | 15:52 |
| rcarrillocruz | but for some reason here the gathre facts is still being run | 15:53 |
| rcarrillocruz | do we collect the setup inventory file as part of the job results? | 15:55 |
| corvus | rcarrillocruz: the error you linked wasn't in the setup call | 15:55 |
| corvus | rcarrillocruz: that was a normal pre playbook | 15:56 |
| pabelanger | 2018-08-16 14:11:13,751 DEBUG zuul.AnsibleJob: [build: 1ed6d14c6c0e4eaf9d47d4954f201c81] Ansible command: ANSIBLE_CONFIG=/tmp/tmp2yxuinem/1ed6d14c6c0e4eaf9d47d4954f201c81/ansible/setup_playbook/ansible.cfg ansible '*' -v -m setup -i /tmp/tmp2yxuinem/1ed6d14c6c0e4eaf9d47d4954f201c81/ansible/setup-inventory.yaml -a 'gather_subset=!all' | 15:56 |
| pabelanger | that actually works | 15:56 |
| rcarrillocruz | corvus: so it got passed the setup call ? | 15:56 |
| pabelanger | so, I think the ssh key is working as expected | 15:56 |
| pabelanger | rcarrillocruz: yes, I think so | 15:56 |
| *** pbrobinson has joined #zuul | 15:57 | |
| pabelanger | is it possible this is a gather_facts issue? | 15:58 |
| pabelanger | in the pre.yaml file, we don't have any gather_facts filters | 15:58 |
| rcarrillocruz | yah | 16:00 |
| rcarrillocruz | so, we don't have the pre pre job setup issue | 16:00 |
| rcarrillocruz | since we bypass it with network_cli blacklisting | 16:00 |
| rcarrillocruz | but within playbooks context we do a gather facts later on | 16:00 |
| pabelanger | yah, that seems right. And the failure we are getting from ansible, is wrong error message | 16:03 |
| *** gtema has quit IRC | 16:06 | |
| pabelanger | corvus: mordred: Shrews: to recap, nodepool / zuul is doing the right thing with ssh keys. But the ansible error makes it look like we had bad SSH keys | 16:07 |
| pabelanger | going to refactor some base jobs later today for the network device to and filter gathering of facts | 16:08 |
| *** jpena is now known as jpena|off | 16:19 | |
| mordred | pabelanger, rcarrillocruz: wow - what a fun issue! | 16:24 |
| *** panda|lunch is now known as panda|off | 16:28 | |
| *** myoung is now known as myoung|lunch | 16:28 | |
| *** rlandy is now known as rlandy|brb | 16:33 | |
| *** pcaruana has quit IRC | 16:34 | |
| *** sshnaidm is now known as sshnaidm|bbl | 16:35 | |
| *** openstackgerrit has quit IRC | 16:49 | |
| *** openstackgerrit has joined #zuul | 16:58 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: DNM: test swift logs https://review.openstack.org/592581 | 16:58 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: DNM: test swift logs https://review.openstack.org/592581 | 17:01 |
| openstackgerrit | Merged openstack/pbrx master: Clear up "tag" vs "image" https://review.openstack.org/592536 | 17:08 |
| *** rlandy|brb is now known as rlandy | 17:11 | |
| *** electrofelix has quit IRC | 17:15 | |
| *** myoung|lunch is now known as myoung | 17:44 | |
| pabelanger | I find it difficult a PR in github, cannot depend on another PR. It ends up pullling in the commits from the parent for the child | 17:51 |
| mordred | pabelanger: yup, this is correct | 17:58 |
| mordred | pabelanger: you can use depends-on: in zuul though | 17:58 |
| pabelanger | yah | 17:59 |
| *** mmedvede has quit IRC | 18:13 | |
| *** mmedvede has joined #zuul | 18:17 | |
| *** elyezer has quit IRC | 18:29 | |
| *** rlandy has quit IRC | 18:32 | |
| *** myoung is now known as myoung|brb | 18:34 | |
| *** rlandy_ has joined #zuul | 18:35 | |
| *** elyezer has joined #zuul | 18:36 | |
| SpamapS | corvus: hey, regarding ssl and gearman... does the ssl in geard do any authorization with the client cert? Like, how can make sure only my executors can talk to gearman, and not some rando things that just happen to have client certs signed by my CA? | 18:40 |
| SpamapS | reading the code, it's not clear | 18:41 |
| corvus | SpamapS: er, i believe anything signed by the ca is accepted... i'm not 100% on that | 18:41 |
| corvus | SpamapS: yeah, that's my understanding from reading https://docs.python.org/2/library/ssl.html#ca-certificates | 18:42 |
| SpamapS | Same | 18:42 |
| SpamapS | so I guess the simple answer is I should just make a bastion-CA that is just for gearman. | 18:43 |
| corvus | SpamapS: now there *is* some code in there about ACLs, but that's some geard-private stuff that i never got around to using for real, and therefore, never proposed as a gear protocol extension. | 18:43 |
| corvus | SpamapS: i think that's probably the thing to do. | 18:43 |
| SpamapS | yeah, pretty straight forward and I can automate it. | 18:43 |
| *** sshnaidm|bbl is now known as sshnaidm | 18:43 | |
| SpamapS | finally got my executor to scp its logs | 18:44 |
| SpamapS | now need to scale out executors | 18:44 |
| corvus | SpamapS: (however, *if* you wanted to nerd-snipe yourself into extending the gear protocol with acls, there's a bunch of code already written :) | 18:44 |
| SpamapS | nneewwwp | 18:44 |
| corvus | good call | 18:44 |
| SpamapS | bring on the zookeeper | 18:44 |
| corvus | ya, that's likely the long term thing anyway | 18:45 |
| openstackgerrit | David Shrewsbury proposed openstack/pbrx master: Implement basic image push https://review.openstack.org/592648 | 18:46 |
| Shrews | mordred: ^^^ seems to work: https://hub.docker.com/r/shrews/ | 18:47 |
| mordred | Shrews: WOAH | 18:47 |
| Shrews | we might want to add the ability to specify a non-dockerhub repo in the future | 18:48 |
| Shrews | mordred: the issue we have with jobs now is the whole job artifact thing. for now, if we want to push, we have to build first | 18:48 |
| Shrews | in the same job | 18:48 |
| Shrews | oh, i need to push the -base image too | 18:49 |
| Shrews | doh | 18:49 |
| mordred | Shrews: yes- I think that's just gonna be where we're at for a bit - but is probably ok for now | 18:50 |
| Shrews | pabelanger: fyi ^^ | 18:51 |
| *** harlowja has joined #zuul | 18:51 | |
| pabelanger | woah, Yay! | 18:52 |
| openstackgerrit | David Shrewsbury proposed openstack/pbrx master: Implement basic image push https://review.openstack.org/592648 | 18:59 |
| *** myoung|brb is now known as myoung | 19:19 | |
| pabelanger | I just stumbled across: https://mergify.io/ | 19:22 |
| clarkb | pabelanger: thats sileht and jd's thing born out of gnocchi's move to github and not having zuul anymore | 19:22 |
| pabelanger | yah | 19:23 |
| *** rlandy_ is now known as rlandy | 19:40 | |
| *** pcaruana has joined #zuul | 20:13 | |
| *** ssbarnea has quit IRC | 20:45 | |
| *** pcaruana has quit IRC | 20:53 | |
| *** samccann has quit IRC | 20:56 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/zuul-website master: Add a promotional message banner and events list https://review.openstack.org/591870 | 21:05 |
| mordred | Shrews: that patch is entirely too small and simple | 21:08 |
| mordred | Shrews: when you get a sec, could you add a release note? | 21:09 |
| Shrews | mordred: is it wrong? did i miss something? | 21:15 |
| Shrews | oh, i missed the snark | 21:16 |
| Shrews | :) | 21:16 |
| openstackgerrit | David Shrewsbury proposed openstack/pbrx master: Implement basic image push https://review.openstack.org/592648 | 21:19 |
| mordred | pabelanger, corvus, SpamapS: ^^ if you have any interest in reviewing that | 21:21 |
| Shrews | we need to add use of a prefix to the zuul job | 21:32 |
| Shrews | oh, nm. it already has it | 21:33 |
| Shrews | our pbrx-build-zuul-containers just isn't using it | 21:34 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/zuul-website master: Add a promotional message banner and events list https://review.openstack.org/591870 | 21:55 |
| *** harlowja has quit IRC | 22:14 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/zuul-website master: Add a promotional message banner and events list https://review.openstack.org/591870 | 22:24 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/zuul-website master: Add a promotional message banner and events list https://review.openstack.org/591870 | 22:26 |
| *** sshnaidm is now known as sshnaidm|off | 22:45 | |
| openstackgerrit | Merged openstack/pbrx master: Implement basic image push https://review.openstack.org/592648 | 23:17 |
| ianw | can we/do we write ansible modules/libraries in zuul-jobs as python3 only? | 23:30 |
| *** elyezer has quit IRC | 23:34 | |
| *** elyezer has joined #zuul | 23:36 | |
| mordred | ianw: not yet - we're still running things in python2 on the remote nodes | 23:38 |
| mordred | ianw: and for nodes like centos7 that still only have python2 by default we should be careful to make sure stuff will still work on python2 | 23:38 |
| ianw | yeah, what i thought. we should probably run unit tests... | 23:43 |
| mordred | yah | 23:47 |
| *** rlandy is now known as rlandy|bbl | 23:53 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!