Thursday, 2019-06-27

*** sanjayu_ has quit IRC00:02
*** saneax has joined #zuul00:06
*** openstack has joined #zuul13:14
*** ChanServ sets mode: +o openstack13:14
AJaegermnaser: AFAIU required-projects checks the repo out. IF the repo is a trusted repo, it's master, not the depends-on change...13:15
mnaserAJaeger: ah okay gotcha, so i cant do anything then in that case :)13:20
fungimnaser: and to be clear, that's a security feature. trusted projects contain security-sensitive data and controls which could be compromised if zuul allowed speculative execution of configuration from a trusted project in an untrusted context (e.g. a pre-review "check" pipeline)13:29
jangutterI believe a while ago I saw something that checked the commit header for things like 'DNM'/'WIP'/'POC' and specifically skipped jobs or reacted differently to the review. Sorry for being vague, but does this trigger anyone's memory?13:29
jangutterSpecifically, the use case is to only run linters on reviews with those strings in the header on check, and skip other jobs.13:31
fungidoesn't sound familiar to me. it's also definitely not something i'd want for my workflow since i often push do-not-merge changes specifically to exercise particular jobs13:32
jangutteryeah, I wonder if I'm misremembering something like DNM was not allowed to pass gate... but it could totally be a flight of imagination.13:36
*** pcaruana has quit IRC13:46
*** pcaruana has joined #zuul13:47
mnaserfungi: does that mean using secrets in untrusted jobs is a security issue ?13:48
mnaserBecause if so I think I know of a project that is pushing consumable artifacts for an OpenStack project that has in repo secrets...13:49
pabelangermnaser: no, you can have secrets in untrusted. Just means project needs to be careful about which other projects are allowed to inherit jobs, and ensure nobody merges echo $secret job. Check would be safe, but gate would expose them13:51
*** sgw has joined #zuul13:57
corvuspabelanger: that's not entirely true; secrets are tightly bound to the playbooks in the job they are defined, so child jobs can't access the secrets of parent jobs.14:02
corvusmhu: i don't think we have that yet, but i agree it would be nice.  i think atm you would need to crawl all the project definitions using the api14:03
pabelangercorvus: ah right. I was most thinking, if the parent job wrote the secret to disk some place14:03
corvusyes, that requires care and thought14:05
fungimnaser: the difference is that secrets-using playbooks in trusted repos can be run (non-speculatively) or similarly inherited by jobs in post-review pipelines for untrusted repos. secrets-using playbooks in untrusted repos can't be used by jobs in other untrusted repos (modulo the recently-merged ability for config projects to expressly override the allowed-projects list)14:12
fungihttps://zuul-ci.org/docs/zuul/user/config.html#attr-job.allowed-projects14:13
mhucorvus: thanks!14:13
*** jeliu_ has joined #zuul14:29
openstackgerritJames E. Blair proposed zuul/zuul-operator master: Add .gitreview  https://review.opendev.org/66794214:38
openstackgerritJames E. Blair proposed zuul/zuul-operator master: Add .gitreview and .zuul.yaml  https://review.opendev.org/66794214:39
openstackgerritJames E. Blair proposed zuul/zuul-operator master: Add .gitreview and .zuul.yaml  https://review.opendev.org/66794214:41
corvusi'm self-approving that ^14:42
openstackgerritMerged zuul/zuul-operator master: Add .gitreview and .zuul.yaml  https://review.opendev.org/66794214:42
fungiwfm14:45
*** bhavikdbavishi has joined #zuul14:57
*** bhavikdbavishi has quit IRC15:02
*** bhavikdbavishi has joined #zuul15:10
*** hashar has joined #zuul15:13
*** bhavikdbavishi has quit IRC15:14
*** pcaruana has quit IRC15:14
*** bhavikdbavishi has joined #zuul15:15
*** gtema_ has joined #zuul15:22
*** portdirect has quit IRC15:30
*** portdirect has joined #zuul15:31
*** chandankumar is now known as raukadah15:34
*** mattw4 has joined #zuul15:43
fbohi, is there a way to force zuul ansible_python_interpreter to python3 ?15:53
fbowe have tried to set https://zuul-ci.org/docs/nodepool/configuration.html#attr-diskimages.python-path in nodepool w/o luck15:54
corvusfbo: some changes just went into nodepool to be able to set that on a per-image basis, that's probably the best way to ....15:54
corvusoh :(15:54
corvusfbo: you'll need to be running the newest releases of zuul and nodepool for that (or git master)15:55
corvusjust released last week15:55
fbocorvus: ah :) ok that's the reason then. Thanks15:55
openstackgerritJames E. Blair proposed zuul/nodepool master: Switch functional testing to a devstack consumer job  https://review.opendev.org/66502315:58
*** sgw has quit IRC16:01
*** jangutter has quit IRC16:02
*** jangutter has joined #zuul16:02
*** sshnaidm|ruck is now known as sshnaidm|off16:06
*** pwhalen has quit IRC16:09
*** igordc has joined #zuul16:11
*** sgw has joined #zuul16:20
*** jpena|mtg is now known as jpena|off16:21
*** mattw4 has quit IRC16:26
*** panda has quit IRC16:32
*** hashar has quit IRC16:35
*** panda has joined #zuul16:36
raukadahHello16:54
raukadahis it possible to create a symlink in image from nodepool config?16:55
raukadahon openstack provider16:55
clarkbraukadah: yes, the disk image builder elements can create a symlink at image build time16:55
tobiashcorvus: is the functional testing change ready for review?16:58
clarkbraukadah: what you'll need to do is createa disk image build element that creates the files and/or symlinks that you want16:58
raukadahclarkb: do I need to use some specific element for that?16:58
clarkbraukadah: you'll likely need to write your own16:59
corvustobiash: yes!  literally right this minute the whole next phase of the stack returned green :)16:59
raukadahI just wanted to create a symlink  from /usr/bin/python to /usr/bin/python316:59
clarkbraukadah: I would recommend against that16:59
corvustobiash: (i will rebase the removal change now)16:59
clarkbraukadah: python upstream has stated python2 == python and python3 != python (as far as executable names go)16:59
clarkbraukadah: that will liekly create system confusion if other applications expect python2 at `python`17:00
openstackgerritJames E. Blair proposed zuul/nodepool master: Remove devstack plugin functional test jobs  https://review.opendev.org/66715617:00
corvusthat's the rebase of the removal17:00
raukadahclarkb: what if all the application assumes python3 == python then17:01
clarkbraukadah: then those applications are buggy17:01
clarkb(unless you are on arch)17:01
clarkbmy biggest concern is that if you are on a platform (basically anything but arch) where python == python2 any system applications that expect python to be python2 will break17:02
clarkband so would be preferable to tell python3 applications to use `python3`17:02
clarkbanother potential option is to use a virtualenv17:02
clarkb(since that will isolate the system python concerns)17:02
clarkbbut in any case you'll need an element17:03
clarkbthere is no generic make a symlink element17:03
corvusoh, my, the tentacles extend to openstacksdk too17:03
raukadahclarkb: ok, it is the same case which fbo was discussing earlier17:03
clarkbraukadah: ok so for that you can set that as an attribute of the nodepool config, but it tells ansible to use `python3` not a symlink from python to python317:04
clarkblet me find a link17:04
*** jeliu_ has quit IRC17:05
clarkbraukadah: https://zuul-ci.org/docs/nodepool/configuration.html#attr-diskimages.python-path set that to /usr/bin/python317:05
clarkb(this requires an up to date nodepool and zuul install, most recent releases for both)17:06
*** hwangbo has quit IRC17:06
raukadahclarkb: the nodepool and zuul is not update in sf17:07
raukadahclarkb: https://softwarefactory-project.io/r/#/c/15834/ I tried that so got suggested to create a symlink17:07
pabelangeryou can set ansible_python_interpreter in your job, if you need to change it too17:10
pabelangerthat doesn't need a new release of nodepool17:10
clarkbpabelanger: I think zuul overrides it though17:10
corvusnope17:11
pabelangerclarkb: nope17:11
corvuswe use it in opendev17:11
clarkboh so the hardcoded value in zuul prior to the recent release can be overridden in the job directly?17:11
clarkbOh the hardcoded value is what the executor runs but the rmeote node can be set got it17:11
clarkbor something like that, the remote python is determined at runtime, not prior to starting ansible on the executor17:12
*** hwangbo has joined #zuul17:14
corvusclarkb: i think the locality of setting it as a host-var overrode the hardcoded global setting17:14
raukadahclarkb: pabelanger https://review.rdoproject.org/r/#/c/21270/4/zuul.d/standalone-jobs.yaml@2617:15
raukadahi have tried this17:15
pabelangerclarkb: yah, it gets a little tricky, with more then 1 nodeset using the job. I kinda wish you could set a nodeset.vars, but with new python-path in nodepool, that isn't needed17:16
pabelangerraukadah: is there logs from the job run?17:17
clarkbout of curiousity what is the problem with using python2 there? Does ansible not run under rhel8's python2?17:17
clarkbor maybe we are trying to avoid installing python2 on rhel8?17:18
pabelangerthere is no python2 on rhel817:18
clarkboh I thought there was a python217:18
clarkbbut primary was 317:18
pabelangerno, last I heard it was all removed17:18
clarkbthe internet tells me that you can install python2 on rhel817:19
pabelangerhowever, I haven't tried rhel8 GA17:19
clarkbit isn't there by default but it is available as a package17:19
pabelangerah17:19
openstackgerritJames E. Blair proposed zuul/nodepool master: Remove devstack plugin functional test jobs  https://review.opendev.org/66715617:19
raukadahtrying to capture the stream out, the job is queued17:19
pabelangerclarkb: my guess, is product reasons not to install python217:19
pabelangerbut, that is a wild guess17:19
fungiof course, having your job install python2 so that ansible can use it is a bit of a catch-22, so would probably need to install it in the image instead17:20
clarkbfungi: yes would have to be part of the image build17:20
pabelangerthere is some good news, with ansible 2.8, they've make it a little smarter to look for python the remote node.17:21
pabelangerThere is a new 'auto' setting you can setup for ansible_python_interpreter that will use any python found17:21
pabelangerit searches multiple paths for it17:21
openstackgerritJames E. Blair proposed zuul/nodepool master: Remove devstack plugin functional test jobs  https://review.opendev.org/66715617:21
corvuswe have oodles of ways of selecting the correct python for ansible in zuul, so noone should need to update base images17:24
tobiashcorvus: I left a question on 66502317:24
clarkbcorvus: ya the question was more why is a symlink preferred to actually installing python217:25
clarkbsounds like it should be possible to address without changing the image though so yay17:25
corvustobiash: replied, thanks17:26
tobiash++17:27
openstackgerritDirk Mueller proposed zuul/nodepool master: Add Python 3.7 testing  https://review.opendev.org/66772017:29
*** rlandy is now known as rlandy|brb17:29
openstackgerritJames E. Blair proposed zuul/nodepool master: Reparent nodepool-functional-openstack-src  https://review.opendev.org/66799517:29
openstackgerritJames E. Blair proposed zuul/nodepool master: Reparent nodepool-functional-openstack-src  https://review.opendev.org/66799517:37
*** mattw4 has joined #zuul17:40
corvustobiash: fyi see -1 on 66772017:41
tobiashOh, that's right17:43
openstackgerritDirk Mueller proposed zuul/nodepool master: Add Python 3.7 testing  https://review.opendev.org/66772017:43
*** rlandy|brb is now known as rlandy17:45
*** gtema_ has quit IRC17:47
*** bhavikdbavishi has quit IRC17:48
*** mattw4 has quit IRC17:50
openstackgerritDirk Mueller proposed zuul/zuul master: Add Python 3.7 testing  https://review.opendev.org/66800617:52
*** electrofelix has quit IRC17:53
*** pcaruana has joined #zuul17:54
*** mattw4 has joined #zuul18:12
*** jeliu_ has joined #zuul18:15
*** mattw4 has quit IRC18:17
*** mattw4 has joined #zuul18:24
*** mattw4 has quit IRC18:30
*** themroc has quit IRC18:44
SpamapSso18:47
SpamapSwe're broken right now18:48
SpamapSinstall-nodejs18:48
SpamapSI've never heard of this 'command' command, and neither have my servers. :-/18:50
fungii lack context18:50
openstackgerritClint 'SpamapS' Byrum proposed zuul/zuul-jobs master: Revert "install-nodejs: add support for RPM-based OSes"  https://review.opendev.org/66802118:50
funginow i have context!18:50
fungithis is the first i've heard of `command` but it seems to be a shell builtin?18:53
fungicommand [-pVv] command [arg ...] Run command with args suppressing the normal shell function lookup. Only builtin commands or commands found in the PATH are executed. [...]18:54
clarkbyes looks like bash builtin18:54
fungifrom bash(1)18:54
fungidoes not seem to be posix bourne18:55
fungimaybe copied from zsh?18:55
fungiat any rate, we should probably not assume an ansible shell task is running in a shell which has that builtin18:56
clarkbI'm probably blind but where is `command` in that revert?18:56
fungiroles/install-nodejs/tasks/main.yaml18:56
fungicommand: command -v node18:57
clarkbbah gerrit search only searches the new side of the diff18:57
fungiIf the -V or -v option is supplied, the exit status is 0 if command was found, and 1 if not.18:57
clarkbtype also works but that is anohter builtin. In any case should use a shell ratherthan command task I agree18:57
fungiapparently a bashism for which(1)18:57
fungithe manpage for which(1) indicates this is its default behavior18:58
fungiEXIT STATUS: 0 if all specified commands are found and executable; 1 if one or more specified commands is nonexistent or not executable18:59
clarkbI think the struggle with which is it isnt always installed18:59
fungiahh18:59
clarkbso the tradeoff here is expecting bash to exist or expecting which to exist18:59
clarkbpersonally I expect both to exist but I know that is not true of all installations18:59
corvusSpamapS: +2 with a comment that we should add jobs to test that role when we next take a crack at it.  but we should nail down the particulars of your environment to make sure we have it covered.19:00
fungidoes ansible not have a feature to test for the existence of an executable in the path?19:00
clarkbI've approved the revert19:00
clarkbfungi: I am not aware of one19:00
corvussounds like a system with non-bash as the default probably would have caught that19:00
clarkbcorvus: ya I think ubuntu would've caught it19:01
* clarkb checks if dash has command19:01
fungidouble-approved i guess19:01
clarkboh wow dash has command too19:01
corvusit's possible that a solution which would satisfy everything would be to specify those ansible shell tasks run with bash19:01
clarkbcorvus: in this case it is a command task which is extra problematic19:02
fungiahh, yeah, command doesn't run in the context of a shell at all, right?19:02
corvuser, then if it does exist, what exactly does ansible run?19:03
corvus(i thought a shell builtin would have to be run in a shell)19:03
clarkbcorvus: yes it should be in a shell task19:03
clarkbbut then we'd also have to ensure that the shell run (which is sh by default which varies on all platforms) has `command` or similar19:04
clarkbmy suse sh is bash. ubuntu sh is dash. I know some docker images have bourne shell as sh and so on19:04
corvusso does "command: command" even work?19:04
corvuswith "shell:" you can specify which shell to use19:05
*** gtema_ has joined #zuul19:05
clarkbif ansible runs command in a shell then it will work if that shell has command, I don't know if ansible runs command tasks in a shell or not19:05
corvus({shell: 'command foo', args: {executable: '/bin/bash}})19:05
corvusclarkb: interesting question; i assumed since it didn't allow a shell argument that it did not use a shell, and was like an exec....19:06
corvusfatal: [localhost]: FAILED! => {"changed": false, "cmd": "command node", "msg": "[Errno 2] No such file or directory: 'command': 'command'", "rc": 2}19:07
clarkbso ya I don't think that ever worked then19:07
fungiagreed19:07
corvusmoving the zuul-jobs test jobs into zuul-jobs is my next move-to-zuul-tenant task, so i'll try to set up more docs/examples around that, and we can start to improve testing there19:09
openstackgerritMerged zuul/zuul-jobs master: Revert "install-nodejs: add support for RPM-based OSes"  https://review.opendev.org/66802119:19
corvusSpamapS: ^19:19
openstackgerritJeff Liu proposed zuul/zuul-operator master: Add playbooks folder and debug zuul job  https://review.opendev.org/66802919:20
corvustobiash, SpamapS, mordred, tristanC: jeliu_ is an intern at red hat who is starting to help us with work on the operator... for now, we're working on setting up some basic testing infrastructure, then as the spec comes together, he'll be working on fleshing out the implementation19:20
*** gtema_ has quit IRC19:21
*** gtema_ has joined #zuul19:22
fungithat's exciting. welcome jeliu_!19:26
tobiashWelcome jeliu_ !19:28
jeliu_Thanks! Excited to be here!19:29
SpamapScorvus:thanks so much for reverting so quickly!19:33
SpamapSProbably should add a local gated mirror to make these less urgent.19:33
SpamapSIs there any reason to use the `command` command in the `command` module in Ansible?19:34
tobiashSo many commands19:35
SpamapSI command you to use command.19:36
clarkbSpamapS: no I think it will only work when run in a shell (and only if that shell has that builtin so not bourne or csh (probably))19:37
clarkbjeliu_: hello19:37
clarkband welcome19:37
SpamapSSo the real problem is that the code path isn't tested.19:37
fungiSpamapS: yeah, we don't currently exercise every role in zuul-jobs on changes to zuul-jobs19:38
jeliu_clarkb Thanks!19:39
fungiSpamapS: corvus has some ideas on how we can do functional testing of roles and playbooks (i think as a follow on to the current integration testing?)19:40
fungiand was looking for volunteers to work on implementation19:40
SpamapSI'd be quite happy to be a 3rd party test.19:40
SpamapSone of the other engs here noticed another problem with the change19:42
fungiawesome19:42
SpamapSthe original installed apt-transport-https, but the new one's Debian tasks dont.19:42
fungiwhich is okay for newer debian releases, but probably not <buster?19:43
clarkbor <= xenial19:43
fungi(i can't remember when https went into the apt package)19:43
SpamapS+    install_nodejs_node_installed.rc == 1 or install_nodejs_npm_installed == 119:44
SpamapSthat's also a bug19:44
fungiyeah, https://packages.debian.org/apt-transport-https confirms it was an actual package through stretch, and only becomes a transitional package as of buster19:44
SpamapSit should be != 0, not == 119:45
SpamapSand the second one needs .rc19:45
SpamapSThat led to the conditional result failing19:46
SpamapSjeliu_:hey howdy. Sorry for pointing out bugs publicly. I'm sure this was a challenging task. We'll pile on to make sure the next round goes well. :)19:47
SpamapSAlso I just want to remind everyone that zuul-jobs is not tagged, doesn't do release controls, and as a result, can really screw things up for users who are consuming it directly. Careful reviews are definitely needed.19:47
fungiyep. can still leave comments about all these points on https://review.opendev.org/63104919:47
fungieven though it's merged (and reverted)19:47
SpamapS(I am going to resurrect my old task to move to a gated mirror for our deploy)19:48
fungiSpamapS: jeliu_ is working on the zuul-operator for kubernetes. the patch which got reverted was mhu's19:48
*** hwangbo has quit IRC19:49
*** hwangbo has joined #zuul19:50
SpamapSoh19:51
SpamapSderp19:51
hwangbohow much work would need to be done to support dependency cycles? I saw that it was discussed roughly in this thread http://lists.zuul-ci.org/pipermail/zuul-discuss/2018-November/000637.html19:57
clarkbhwangbo: I believe it requires adding direct merge support to zuul19:57
clarkbbecause hitting the merge button in gerrit/github would sequence changes/PRs19:58
clarkband could fail halfway through19:59
*** hwangbo has quit IRC20:04
*** hwangbo has joined #zuul20:04
*** hwangbo has quit IRC20:05
*** hwangbo has joined #zuul20:05
fungiyes, the risk is that you decide to merge changes a,b,c together and then you call gerrit to submit a and it works and then you call gerrit to submit b and it's rejected, and you're left with a merged but not b or c20:05
fungiso zuul would need to push the merge commits into gerrit instead to avoid that20:06
*** jeliu_ has quit IRC20:09
*** hwangbo has quit IRC20:23
openstackgerritJames E. Blair proposed zuul/zuul-jobs master: Split job definitions into multiple files  https://review.opendev.org/66804020:36
openstackgerritJames E. Blair proposed zuul/zuul-jobs master: Add a simple test of the install-nodejs role  https://review.opendev.org/66804320:43
corvusi think i'd like to add a feature which allows an admin to specify that zuul should load in-repo config from a non-standard location for a particular project.  that way, we can put a project stanza in zuul-jobs, but put it in a file like "zuul-jobs-testing.d/project.yaml", and tell the opendev zuul to additionally load files in "zuul-jobs-testing.d" just for the zuul-jobs repo (and just in the zuul tenant).20:46
corvusthat way no other users of zuul-jobs see that file, but we can still do in-repo config of the project stanza for zuul-jobs.20:46
corvusadd to that the change i want to do to have a files-matcher which automatically matches if the current job config is changed, and i think the testing story for zuul-jobs will get even better20:46
corvusi bet we could add https://review.opendev.org/668043 to 75% of our roles without writing any additional playbook code and get Pretty Good Coverage (TM)20:52
openstackgerritJames E. Blair proposed zuul/zuul-jobs master: DNM: install-nodejs: add support for RPM-based OSes  https://review.opendev.org/66804820:53
corvusin fact... let's see if it would have caught that ^20:54
openstackgerritJames E. Blair proposed zuul/zuul-jobs master: Move upload-git-mirror test job in-repo  https://review.opendev.org/66805021:01
corvusfungi, clarkb, SpamapS: check it out!  http://logs.openstack.org/48/668048/1/check/zuul-jobs-test-install-nodejs/dd2dab4/ara-report/result/ce8c4427-7c3f-463a-9e26-7276869bf8d3/21:03
*** gtema_ has quit IRC21:04
corvusi'm waiting for the git-mirror job results to arrive, then i'll push up a final version of that for review21:05
*** pcaruana has quit IRC21:05
*** sgw has quit IRC21:12
openstackgerritJames E. Blair proposed zuul/zuul-jobs master: Add a simple test of the install-nodejs role  https://review.opendev.org/66804321:13
openstackgerritJames E. Blair proposed zuul/zuul-jobs master: Move upload-git-mirror test job in-repo  https://review.opendev.org/66805021:13
openstackgerritJames E. Blair proposed zuul/zuul-jobs master: Add zuul-jobs-tox-linters  https://review.opendev.org/66805221:13
openstackgerritJames E. Blair proposed zuul/zuul-jobs master: Move upload-git-mirror test job in-repo  https://review.opendev.org/66805021:16
openstackgerritJames E. Blair proposed zuul/zuul-jobs master: Add zuul-jobs-tox-linters  https://review.opendev.org/66805221:16
openstackgerritJames E. Blair proposed zuul/zuul-jobs master: Move upload-git-mirror test job in-repo  https://review.opendev.org/66805021:27
openstackgerritMerged zuul/nodepool master: Switch functional testing to a devstack consumer job  https://review.opendev.org/66502321:39
openstackgerritJames E. Blair proposed zuul/zuul-jobs master: Add base integration roles  https://review.opendev.org/66806121:44
openstackgerritJames E. Blair proposed zuul/zuul-jobs master: Add base integration roles  https://review.opendev.org/66806122:00
openstackgerritJames E. Blair proposed zuul/zuul-jobs master: Move upload-git-mirror test job in-repo  https://review.opendev.org/66805022:03
openstackgerritJames E. Blair proposed zuul/zuul-jobs master: Add base integration roles  https://review.opendev.org/66806122:03
*** mattw4 has joined #zuul22:23
*** mattw4 has quit IRC22:28
*** mattw4 has joined #zuul22:28
*** igordc has quit IRC22:36
*** mattw4 has quit IRC22:37
*** rlandy has quit IRC22:57
SpamapScorvus:zomg I love the idea of setting the paths to look for config in the tenant config stanza for the project. That kind of works in the same way I wanted namespacing to work.22:58
SpamapScorvus:also nice job on the test!22:59
*** tosky has quit IRC23:00

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!