| *** caphrim007_ has joined #openstack-jjb | 00:03 | |
| *** caphrim007 has quit IRC | 00:06 | |
| *** caphrim007_ has quit IRC | 00:08 | |
| openstackgerrit | Anil Belur proposed openstack-infra/jenkins-job-builder master: Add support for build status Job Filter plugin https://review.openstack.org/548196 | 00:40 |
|---|---|---|
| *** caphrim007 has joined #openstack-jjb | 01:21 | |
| *** caphrim007 has quit IRC | 01:21 | |
| larainema | thanks zxiiro, joined | 01:22 |
| *** caphrim007 has joined #openstack-jjb | 01:22 | |
| *** caphrim007_ has joined #openstack-jjb | 01:26 | |
| *** caphrim007 has quit IRC | 01:29 | |
| *** caphrim007_ has quit IRC | 06:36 | |
| *** ckonstanski has quit IRC | 07:17 | |
| *** hashar has joined #openstack-jjb | 08:18 | |
| *** electrofelix has joined #openstack-jjb | 08:36 | |
| openstackgerrit | Sorin Sbarnea proposed openstack-infra/jenkins-job-builder master: Update artifact_resolver to make use of convert_mapping_to_xml() https://review.openstack.org/473994 | 09:08 |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder master: Update dynamic_scriptler_param_common to use convert_xml() https://review.openstack.org/475968 | 09:14 |
| ssbarnea | https://review.openstack.org/#/c/519340/ -- on python-jenkins but important for jjb. | 09:22 |
| ssbarnea | electrofelix: ^^ | 09:22 |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder master: Update artifact_resolver to make use of convert_mapping_to_xml() https://review.openstack.org/473994 | 09:27 |
| electrofelix | ssbarnea: yeah, I've been uncomfortable about that patch as it's making folders look identical to jobs with no plan on how to distinguish them subsequently. To manage jobs/views in folders, you need to be able to create the folders before the jobs which is important for updating multiple jobs in parallel | 09:55 |
| ssbarnea | electrofelix: in fact folders are special job (project) types | 09:56 |
| electrofelix | they are but you cannot create a job nested in a folder without the parent folder existing | 09:56 |
| ssbarnea | electrofelix: the issue about having to create folder before job is valid, but we can fix in the future | 09:56 |
| electrofelix | so you need a way to distinguish between them | 09:56 |
| ssbarnea | by auto-creating folders. | 09:57 |
| ssbarnea | now, is a know limitation, aka bug. | 09:57 |
| electrofelix | if fixing in the future means undoing that patch, then no that doesn't work either.. need a path to a solution identified first, doesn't need to be implemented | 09:57 |
| electrofelix | adding stuff doesn't mean you can remove it in the future, can that will break people, once it's in, it's in for a very long time | 09:58 |
| ssbarnea | electrofelix: i doubt we need to undo this patch, i do find its API quite clear. | 09:59 |
| ssbarnea | electrofelix: why we would have to change it? | 10:00 |
| electrofelix | in jjb, how do you plan to determine if a job folder exists that is required for a job? or determine whether actually it's a job that cannot contain others? | 10:01 |
| ssbarnea | electrofelix: i was more concerned about the issue with multibranches because sub-jobs of multibranch are not real jobs, cannot be configured individually. | 10:01 |
| electrofelix | then don't include the two patches together | 10:01 |
| ssbarnea | electrofelix: you mean that you are ok with previous patchset? | 10:03 |
| ssbarnea | i think that we still need params to tell get_all_jobs what we are looking for. Do we want any "entity" that behaves like a job or we want to get all configurable jobs? | 10:04 |
| ssbarnea | clearly there is no way to delete a multibranch-branch-job using API because this is more of a "virtual-job". | 10:04 |
| electrofelix | I mean that if someone has a suggestion for how to distinguish folders in the future that will allow determining them sensibly without needing to make a change now (maybe that is just inspecting the class info which seems horrible to me, but if from app perspective it might be ok) then yes by all means lets land the previous iteration. | 10:05 |
| electrofelix | Similarly it seems like it might be better to return all associated jobs and let the application decide how to handle the workflow multi branch project piece, assuming it returns enough info. The alternative is that for every unusual job or view type python-jenkins will need to add special handling, seems the wrong level? | 10:07 |
| electrofelix | ssbarnea: does the original patch mean that the 'jobs' attribute gets exposed as part of the folder entries back to the calling application, I've a feeling it doesn't | 10:10 |
| ssbarnea | electrofelix: ok, lets make changes atomic, one at a time. easy one https://review.openstack.org/#/c/548547/ | 10:23 |
| electrofelix | ssbarnea: I'm spinning up my test env to inspect what is returned, if it has the 'jobs' attribute exposed by default then I'm happy to approve | 10:25 |
| *** electrofelix has quit IRC | 10:42 | |
| *** electrofelix has joined #openstack-jjb | 11:57 | |
| electrofelix | ssbarnea: confirmed we can use he attribute 'jobs' to distinguish a folder, next question, has anyone tested the behaviour of what happens when you say delete unmanaged jobs in jjb with this change applied? | 12:02 |
| electrofelix | I suspect the previous behaviour most likely was to ignore folders, now I suspect this would result in folders being deleted, which since jjb doesn't properly manage them, might be quite a surprise to anyone using jjb | 12:03 |
| ssbarnea | electrofelix: nope, but i can test now. i considered these two unrelated. | 12:03 |
| electrofelix | given existing behaviour of jjb, it's likely most folders in use are not managed by jjb | 12:04 |
| ssbarnea | electrofelix: can we first review/merge this https://review.openstack.org/#/c/87090/ ? it would help debugging the change above | 12:08 |
| electrofelix | does that need to be updated to call 'get_all_jobs()'? looks to sometimes use 'get_jobs()' https://review.openstack.org/#/c/87090/10/jenkins_jobs/cli/subcommand/list.py | 12:11 |
| ssbarnea | electrofelix: good news, "jenkins-jobs delete myfolder/subfolder/deepjob" does the right thing. | 12:12 |
| electrofelix | ssbarnea: btw, you can use 'git review -x 87090' to apply other changes | 12:12 |
| ssbarnea | electrofelix: i know, but got too many to stack now, hopefully we will deal with them soon. | 12:12 |
| ssbarnea | also "jenkins-jobs delete myfolder" works as expected, removing folder with all jobs included in it. | 12:14 |
| ssbarnea | but there is nothing special in python code about, everything is done by jenkins. | 12:15 |
| electrofelix | ssbarnea: think https://git.openstack.org/cgit/openstack-infra/jenkins-job-builder/tree/jenkins_jobs/builder.py#n200 protects us from deleting unmanaged folders | 12:17 |
| ssbarnea | indeed | 12:19 |
| electrofelix | Does that code cause issues for the multibranch stuff? | 12:21 |
| *** electrofelix has quit IRC | 12:41 | |
| openstackgerrit | Sorin Sbarnea proposed openstack-infra/jenkins-job-builder master: Update release to make use of convert_mapping_to_xml() https://review.openstack.org/479928 | 12:44 |
| *** hashar is now known as hasharAway | 14:46 | |
| *** electrofelix has joined #openstack-jjb | 15:46 | |
| openstackgerrit | Sorin Sbarnea proposed openstack-infra/jenkins-job-builder master: Add multibranch project-type https://review.openstack.org/403940 | 16:04 |
| zxiiro | ssbarnea: I think that entry.py line should be a separate patch to keep the patchset on topic imo | 16:10 |
| ssbarnea | zxiiro: sure, will do that. are you going to be here the next hour? | 16:11 |
| zxiiro | ssbarnea: yeah I'm around all day. Wednesdays are (thankfully) quiet days for me. | 16:11 |
| zxiiro | electrofelix: can you take a look at ssbarnea's multibranch patch? https://review.openstack.org/403940 I'd like your opinion on it and we're trying to get it in before the end of the week. | 16:12 |
| ssbarnea | just found a bug in it, give me few minutes to fix it before doing the review. | 16:13 |
| openstackgerrit | Sorin Sbarnea proposed openstack-infra/jenkins-job-builder master: Add multibranch project-type https://review.openstack.org/403940 | 16:15 |
| openstackgerrit | Sorin Sbarnea proposed openstack-infra/jenkins-job-builder master: Added main entry point https://review.openstack.org/548649 | 16:17 |
| ssbarnea | zxiiro: now the bug is fixed. the trigger was not working correctly. | 16:20 |
| zxiiro | ssbarnea: FYI you shouldn't workflow +1 until we're ready to merge. (although to be fair that patch is pretty trivial so I think we can just merge it). | 16:20 |
| zxiiro | ssbarnea: we usually try to wait for 2x +2's CR before we merge a patch as best practice. | 16:20 |
| ssbarnea | ok, i will keep in mind. | 16:20 |
| zxiiro | yep. np usually the 2nd person who +2's will +1 Workflow. | 16:21 |
| zxiiro | Sometimes I just merge obvious patches like fixing typos or adding docs. | 16:21 |
| ssbarnea | we reached 25 open CRs on JJB, quite a record I think. | 16:22 |
| ssbarnea | well, i have a filter that lists only verified ones, others do not count. | 16:23 |
| ssbarnea | zxiiro: this is one that you should find useful, https://review.openstack.org/#/c/87090/ | 16:24 |
| ssbarnea | with this in, it would be easy to list (and eventually delete) all jobs that are not managed by jjb. | 16:26 |
| zxiiro | ssbarnea: there's a dashboard... let me find you the link | 16:28 |
| zxiiro | ssbarnea: I think it's this one https://review.openstack.org/#/projects/openstack-infra/jenkins-job-builder,dashboards/important-changes:important-changes-dashboard | 16:30 |
| zxiiro | ssbarnea: yeah i plan on verifying the list patch today | 16:31 |
| zxiiro | ssbarnea: it's been in my queue :) | 16:31 |
| ssbarnea | thanks | 16:31 |
| zxiiro | ssbarnea: there was also this one https://review.openstack.org/#/projects/openstack-infra/jenkins-job-builder,dashboards/important-changes:review-inbox-dashboard | 16:31 |
| zxiiro | but looks like it's blank right now | 16:31 |
| zxiiro | probably because we caught up with everythign that needs review | 16:31 |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder master: Added main entry point https://review.openstack.org/548649 | 16:36 |
| electrofelix | for the multibranch patch I think that it should be split up to have the scm parts in the scm module and use dispatch to link in, also the style with the explicit setting of XML attrs instead of all being in the same call doesn't fit with the rest of the codebase | 16:55 |
| *** hasharAway is now known as hashar | 17:04 | |
| zxiiro | electrofelix: oh yeah we also started a new mailing list just for jjb discussion https://groups.google.com/forum/#!forum/jenkins-job-builder | 17:05 |
| *** caphrim007 has joined #openstack-jjb | 17:29 | |
| *** ckonstanski has joined #openstack-jjb | 17:40 | |
| *** hashar is now known as hasharDinner | 17:58 | |
| ssbarnea | electrofelix: the scm implementation from multibranch is not compatible with the ones from normal jobs. on clear example is gerrit which is supported only in multibranch. | 18:23 |
| ssbarnea | if i remember well even those overlapping have different representation at xml level. | 18:24 |
| ssbarnea | this is probably why original patch had "scm" under "multibranch" attribute instead of root. | 18:25 |
| ssbarnea | electrofelix: here is the deal: classic modules map scm data into <scm> element but mutibranch one uses <sources><data><jenkins.branch.BranchSource>... | 18:38 |
| ssbarnea | implementations are very different, i check "git" one, and they seem to have almost nothing in common. | 18:40 |
| ssbarnea | I could create a "sources" module and move the scm part there, but I doubt I cam make "scm" module produce two very different xml formats. | 18:41 |
| ssbarnea | or if I do, it will be a total mess | 18:41 |
| electrofelix | ssbarnea: last time I checked, these plugins also work with the pipeline jobs, basically anything that supports the jenkins DSL, and the git one is the same between both the multibranch and freestyle meaning any options in the normal git scm are avaialble in the multibranch pipeline workflow project as well AFAICS | 19:04 |
| electrofelix | While we can't define them for the pipeline ones via jjb, it looks like these are intended to become the standard way to use SCMs, but maybe it's only limited to the git one being avilable for both | 19:05 |
| ssbarnea | electrofelix: not true for multibranch, i checked 5mins ago. | 19:06 |
| ssbarnea | yes, scm implementation is comatible between freetyle, pipeline, and others, but not mutibranch ones. | 19:06 |
| ssbarnea | multibranch ones are using "branch-api" plugin, which uses this <sources> | 19:07 |
| ssbarnea | initially I didn't understand why, but it starts to make some sense, classic scm always reffered to a specific branch to build. | 19:07 |
| ssbarnea | sources, is more flexible, as it has rules for how to get the list of branches (which is dynamic), CRs, PRs,... | 19:08 |
| electrofelix | ok, but you'll note that the current multibranch code is missing all the options supported for git repos, but within Jenkins UI you can access them all | 19:09 |
| electrofelix | so it's using the same git plugin behind the scenes for all of them, with the multibranch having additional options on top | 19:10 |
| ssbarnea | in fact neither of us is right, is something in-between. some options are common/shared, some are unique to each others. | 19:13 |
| electrofelix | wonderful, that sounds like the worst of both worlds, so that means we need to move some of the git options in the scm module into a common helper function and reference from both? | 19:14 |
| ssbarnea | freestyle git: has branches to build unique, supports multiple repositories (instead of only one). some Behaviours (traits) are shared, but some are unique. | 19:14 |
| ssbarnea | the "Additional Behaviours" is is hard to compare, but is clearly different from one to another. GIT behaviours: "Discover branches", "Filter by <...>" seem to be available only to mutibranch-git repos. | 19:19 |
| ssbarnea | the behaviour dropdown drives me crazy... | 19:19 |
| ssbarnea | there are 20+ options listed there, cannot even copy/paste them, cannot screenshot, html source is a mess. | 19:20 |
| ssbarnea | electrofelix: what would you recommend? | 19:21 |
| electrofelix | clean up the existing patch, add a subsequent patch to move the scm entries out to separate functions within that file, and then look to build a list of what options are common for the git plugins between freestyle and multibranch pipeline and see if there is anything that obviously separates them from other the options? | 19:33 |
| openstackgerrit | Sorin Sbarnea proposed openstack-infra/jenkins-job-builder master: Add multibranch project-type https://review.openstack.org/403940 | 19:42 |
| electrofelix | btw, why did you need '__main__' on the cli entry point? | 19:43 |
| zxiiro | ssbarnea: I think I am caught up on the code reviews but let me know if there's something that needs looking at. I see you guys have been hashing out the multibranch patch. | 19:51 |
| ssbarnea | electrofelix: i needed it in order to make pycharm debuger load the module as cli. so I could set a breakpoint anywhere and just run the custom cli I wanted. | 19:53 |
| ssbarnea | somehow it didn't want to start without having a main on the module | 19:53 |
| ssbarnea | electrofelix: see http://s3.sbarnea.com/ss/RunDebug_Configurations_2018-02-28_19-55-59.png | 19:56 |
| ssbarnea | there is no option to tell it which function from the module to run, it would fail without a main | 19:57 |
| electrofelix | ah, I see, guessing you didn't try configuring pycharms to use ~/.tox/py27/bin/python and a test configuration to call 'jenkins-jobs' with it? | 20:05 |
| *** electrofelix has quit IRC | 20:17 | |
| *** major12 has joined #openstack-jjb | 20:34 | |
| major12 | Hello | 20:36 |
| major12 | Is there a good example how to group job templates ? | 20:37 |
| major12 | I've just started with jjb and have something like - job-template: !include: 1.yaml - job-template: !include: 2.yaml - job-template: !include: 3.yaml - job-template: !include: 4.yaml | 20:38 |
| major12 | https://gist.github.com/anonymous/23285a6303afbebdd2e9dd9768dbcf4f | 20:39 |
| major12 | It looks bad since some configs will include 20-30 templates | 20:39 |
| zxiiro | major12: I think you're looking for job groups? https://docs.openstack.org/infra/jenkins-job-builder/definition.html#job-group | 20:40 |
| major12 | Can job templates be placed inside job group? | 20:41 |
| zxiiro | major12: yes that's the entire purpose of a job group. to put job templates inside of them. | 20:43 |
| zxiiro | major12: then in your project definition you just call the job-group once and it pulls in all of the job templates. | 20:43 |
| major12 | hmm... documentation says that jobs can be placed in job groups | 20:44 |
| major12 | do you have example of config? | 20:44 |
| zxiiro | major12: the example in that docs are using job-templates | 20:44 |
| major12 | in doc @ your link only jobs are inside job group. job templates are still on root level of yaml config | 20:46 |
| zxiiro | major12: the job group are defining job templates take a look at the names | 20:46 |
| zxiiro | major12: the first job template is called "{name}-unit-tests" and the job group lists "{name}-unit-tests" in it's list. it's using a job-template | 20:47 |
| zxiiro | major12: when you call {name}-unit-tests. you're telling JJB to loko up the job template named {name}-unit-tests and pull it into your job group. | 20:48 |
| zxiiro | major12: I think you're confusion might be that the "job-group" has a "jobs" definiton. that jobs definition includes job-templates. it doesn't mean single jobs. You can define both job-templates and jobs in them. | 20:50 |
| major12 | Cool | 20:53 |
| major12 | I will try now | 20:53 |
| major12 | what key should be used in job-group? for jobs we have key "jobs:" | 20:58 |
| major12 | this one is not wortking https://gist.github.com/anonymous/11872f78fa0813cec347c1366c8c1076 | 21:00 |
| major12 | no syntax errors but no jobs rendered | 21:00 |
| zxiiro | major12: you're still getting it wrong. look at the example again | 21:05 |
| zxiiro | major12: DO NOT declare at "job-templates inside of the job group | 21:05 |
| major12 | I want to move job template definitions out from main yaml config | 21:06 |
| major12 | now I can do that in way shown here https://gist.github.com/anonymous/23285a6303afbebdd2e9dd9768dbcf4f | 21:06 |
| zxiiro | major12: if i use your examlpe. this is how you should code it https://pastebin.com/1t8X6bx2 | 21:06 |
| major12 | and this looks "not so perfect" to me since for 20 job templates I will have 60 lines in config | 21:07 |
| zxiiro | major12: then split it into several files | 21:07 |
| major12 | 1 for constant "- job-template" | 21:07 |
| zxiiro | major12: you don't have to define everything in 1 file | 21:08 |
| zxiiro | major12: we have over 2500 jobs defined in our system. We organize it by project and many projects reuse the same 20 or so job-templates defined in global-jjb https://github.com/opendaylight/releng-builder/tree/master/jjb | 21:10 |
| zxiiro | major12: use the file system to your advantage and organize your repo so that you don't have to define 20 job templates in one file :) | 21:10 |
| *** hasharDinner has quit IRC | 21:11 | |
| major12 | Do you include all the job-templates every time? Or your configuration is a little bit separated - one part in jjb configs and second in bash (command line parameters for jjb run) ? | 21:15 |
| zxiiro | major12: no you only declare job templates once jjb will find it as long as it's in the directory. | 21:16 |
| zxiiro | major12: in all of your project definitions you just state the name of the job-template | 21:17 |
| zxiiro | major12: think of "job-templates" and "macros" as libraries you can pull into a "project" section. | 21:17 |
| zxiiro | major12: declare once, use x times everywhere else. | 21:17 |
| major12 | I understand that templates are declared once. Let me rephrase a little. When our deploying the jobs you are pointing jjb to folder with all job templates or just to specific file/folder which contain templates only for this project ? | 21:21 |
| zxiiro | major12: jenkins-jobs supports a recursive option. As long as you organize all of your files inside the same top level directory it will find all of hte files. you just run jjb once | 21:22 |
| zxiiro | major12: I recommend making a git repo just for jjb definitions and have all of your users work against that repo. | 21:23 |
| zxiiro | major12: then you can have jjb run against the repo to deploy all of the jobs in that repo. | 21:23 |
| major12 | Yes sure. I'm concerned that e.g. I have 2 projects with some common code base of templates. And if first project will commit something bad to common/ (like job intead of job-template) this might affect secont project. So code review should be "must" to avoid such situations | 21:26 |
| zxiiro | major12: that's why we put it into a git repo and setup verify jobs to validate syntax and have committers review the code | 21:27 |
| zxiiro | major12: you should have a team of folks with deep understanding of the jobs perform code review on changes | 21:27 |
| zxiiro | major12: here's an example. https://git.opendaylight.org/gerrit/#/q/project:releng/builder | 21:29 |
| zxiiro | major12: releng/builder is our jjb repo for our project and every chagne to jjb is code reviewed by a person and verified by Jenkins | 21:29 |
| zxiiro | major12: we also make global-jjb available as an open source project in case we have some of our reusable job templates might be useful for you http://docs.releng.linuxfoundation.org/projects/global-jjb/en/latest/index.html#lfreleng-global-jjb | 21:31 |
| zxiiro | major12: The CI jobs contain jobs for verifying and merging jjb | 21:31 |
| zxiiro | job templates* | 21:31 |
| zxiiro | we're also working on some jobs for managing global jenkins configuration that JJB cannot manage. | 21:32 |
| major12 | Thanks a lot for your help ! | 21:33 |
| major12 | Could you recoment some tool for main jenkins config ? | 21:34 |
| zxiiro | major12: I'm not aware of any. that's why we started developing those CI jobs in the link I gave you. | 21:35 |
| zxiiro | currently the jenkins-cfg job only supports global jenkins variables but I'm about to finish up a patch to let us manage openstack cloud plugin configuration. | 21:36 |
| major12 | We always can browse XML by python and update some nodes based on our config - but it would be great to have more standard solution | 21:37 |
| major12 | Thanks a lot one more time ! | 21:41 |
| zxiiro | major12: np and good luck! | 21:41 |
| *** major12 has left #openstack-jjb | 21:42 | |
| *** jplf has quit IRC | 23:18 | |
| *** jplf has joined #openstack-jjb | 23:22 | |
| *** zxiiro has quit IRC | 23:26 | |
| *** abelur has quit IRC | 23:26 | |
| *** abelur has joined #openstack-jjb | 23:27 | |
| *** zxiiro has joined #openstack-jjb | 23:28 | |
| *** caphrim007 has quit IRC | 23:50 | |
| *** caphrim007_ has joined #openstack-jjb | 23:51 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!
