| *** sdake has joined #zuul | 00:03 | |
| *** sdake has quit IRC | 00:05 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/nodepool master: [wip] add openstackci-mirrors element for centos/ubuntu testing https://review.openstack.org/634366 | 00:09 |
|---|---|---|
| *** sdake has joined #zuul | 00:10 | |
| *** sdake has quit IRC | 00:44 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/nodepool master: [wip] add openstackci-mirrors element for centos/ubuntu testing https://review.openstack.org/634366 | 01:02 |
| *** sdake has joined #zuul | 01:14 | |
| *** sdake has quit IRC | 01:27 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/nodepool master: [wip] add openstackci-mirrors element for centos/ubuntu testing https://review.openstack.org/634366 | 01:43 |
| openstackgerrit | Ian Wienand proposed openstack-infra/nodepool master: [wip] add openstackci-mirrors element for centos/ubuntu testing https://review.openstack.org/634366 | 02:01 |
| openstackgerrit | Ian Wienand proposed openstack-infra/nodepool master: [wip] add openstackci-mirrors element for centos/ubuntu testing https://review.openstack.org/634366 | 02:31 |
| *** bhavikdbavishi has joined #zuul | 02:54 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/nodepool master: [wip] add openstackci-mirrors element for centos/ubuntu testing https://review.openstack.org/634366 | 02:56 |
| openstackgerrit | wes hayutin proposed openstack-infra/zuul-jobs master: WIP: skip broken updates for krb5 https://review.openstack.org/634622 | 03:59 |
| ianw | corvus / zuul-root : i really hope it's not just EOD delusions, but i'm really struggling to understand what the errors described in https://storyboard.openstack.org/#!/story/2004927 are trying to tell me | 05:35 |
| ianw | "The task includes an option with an undefined variable. The error was: 'value' is undefined" | 05:35 |
| openstackgerrit | Chandan Kumar proposed openstack-infra/zuul-jobs master: WIP: skip broken updates for krb5 https://review.openstack.org/634622 | 05:41 |
| SpamapS | nice to see the EC2 driver getting +2's. Just in time for me to start looking to move to GCE. ;) | 06:09 |
| tobiash | SpamapS: looking forward to see a gce driver ;) | 06:13 |
| tobiash | ianw: that's caused by the jinja in the commit message | 06:14 |
| tobiash | ianw: will be fixed by https://review.openstack.org/633930 | 06:14 |
| tobiash | ianw: ansible evaluates the whole variable 'zuul' when accessing something from it and the commit message (which is part of it) contains jinja stuff which is evaluated | 06:16 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Add spec for multi ansible version support https://review.openstack.org/623927 | 06:24 |
| SpamapS | tobiash: There was an experimental one I think. | 06:28 |
| tobiash | SpamapS: If it's there I don't find it | 06:30 |
| tobiash | there is an azure driver however ;) | 06:30 |
| ianw | tobiash: ohhhh!!! i would never have figured that out :) | 06:36 |
| tobiash | ianw: some others also ran into this problem ;) | 06:38 |
| tobiash | ianw: you can work around this easily by removing the jinja stuff from your commit message :) | 06:38 |
| ianw | yeah, and that explains "value" appearing there ... so confusing because i was thinking something is doing foo['value'] instead of foo[value] etc | 06:39 |
| ianw | anyway, i owe you a beverage for that one! | 06:39 |
| tobiash | ianw: I didn't discover nor fix it, I'm just following what's going on here and I remembered that one ;) | 06:40 |
| *** quique|rover|off is now known as quiquell|rover | 06:49 | |
| *** bhavikdbavishi has quit IRC | 06:49 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Increase timeout for zuul to settle in tests https://review.openstack.org/634634 | 07:09 |
| tobiash | I think that should make our tests more stable ^ | 07:10 |
| *** saneax has joined #zuul | 07:11 | |
| tobiash | example is here: http://logs.openstack.org/27/623927/5/check/tox-py36/187c532/testr_results.html.gz | 07:12 |
| tobiash | all failed test cases ran into 'Timeout waiting for zuul to settle' for no special reason | 07:12 |
| tobiash | they all timeouted while preparing jobs, so I think the current timeout is too short for slower nodes | 07:13 |
| *** sshnaidm|off is now known as sshnaidm | 07:30 | |
| *** pcaruana has joined #zuul | 07:35 | |
| *** pcaruana has quit IRC | 07:42 | |
| *** pcaruana has joined #zuul | 07:43 | |
| *** quiquell|rover is now known as quique|rover|brb | 07:57 | |
| *** bhavikdbavishi has joined #zuul | 08:11 | |
| *** quique|rover|brb is now known as quiquell|rover | 08:16 | |
| *** gtema has joined #zuul | 08:23 | |
| tobiash | the quickstart frequently fails with stderr: 'fatal: unable to access 'https://git.zuul-ci.org/zuul-jobs/': Couldn't connect to server' | 08:24 |
| tobiash | 4 of the last zuul-quickstart failures failed either by ^ or by stderr: 'fatal: unable to access 'https://git.zuul-ci.org/zuul-jobs/': Could not resolve host: git.zuul-ci.org' | 08:26 |
| AJaeger | tobiash: I canot connect to https://git.zuul-ci.org/zuul-jobs/ either ;( | 08:27 |
| AJaeger | infra-root ^ | 08:27 |
| AJaeger | oh, I can - via git... | 08:28 |
| tobiash | the browser friendly url is https://git.zuul-ci.org/cgit/zuul-jobs/tree/ | 08:28 |
| AJaeger | yeah, that one works - and git clone from the other as well... | 08:28 |
| tobiash | so that could be some region specific problem? | 08:29 |
| AJaeger | yeah, could be - where did the jobs run? All same cloud region? | 08:29 |
| tobiash | have to check | 08:29 |
| tobiash | the four failures are in ORD, GRA1, GRA1, BHS1 | 08:31 |
| *** bjackman has joined #zuul | 08:32 | |
| AJaeger | inconclusive ;( | 08:33 |
| tobiash | found further occurrences in DFW, BHS1, DFW, DFW | 08:36 |
| AJaeger | so, that means it could be on the hosting side ;( | 08:42 |
| tobiash | maybe | 08:45 |
| *** jpena|off is now known as jpena | 08:45 | |
| openstackgerrit | Chandan Kumar proposed openstack-infra/zuul-jobs master: WIP: skip broken updates for krb5 https://review.openstack.org/634622 | 09:01 |
| *** tobias-urdin has joined #zuul | 09:06 | |
| *** bjackman has quit IRC | 09:18 | |
| *** bjackman has joined #zuul | 09:36 | |
| openstackgerrit | Chandan Kumar proposed openstack-infra/zuul-jobs master: WIP: skip broken updates for krb5 https://review.openstack.org/634622 | 09:42 |
| *** chkumar|ruck has quit IRC | 09:58 | |
| *** bjackman has quit IRC | 10:01 | |
| *** bjackman has joined #zuul | 10:03 | |
| *** hashar has joined #zuul | 10:06 | |
| *** sanjayu_ has joined #zuul | 10:09 | |
| *** saneax has quit IRC | 10:10 | |
| *** gtema has quit IRC | 10:11 | |
| *** gtema has joined #zuul | 10:22 | |
| *** chandan_kumar has joined #zuul | 10:24 | |
| *** chandan_kumar is now known as chkumar|ruck | 10:24 | |
| *** gtema has quit IRC | 10:27 | |
| *** gtema has joined #zuul | 10:27 | |
| *** sdake has joined #zuul | 10:35 | |
| *** bjackman has quit IRC | 10:40 | |
| *** bjackman has joined #zuul | 10:48 | |
| *** sdake has quit IRC | 10:51 | |
| *** bhavikdbavishi has quit IRC | 11:02 | |
| *** bhavikdbavishi has joined #zuul | 11:15 | |
| *** gtema has quit IRC | 11:17 | |
| *** gtema has joined #zuul | 11:30 | |
| *** rf0lc0 is now known as rfolco | 12:26 | |
| *** jpena is now known as jpena|lunch | 12:40 | |
| *** EvilienM is now known as EmilienM | 12:44 | |
| mordred | tobiash, tristanC: on the ec2 driver, why is boto in test-requirements.txt when boto3 is in requirements.txt? what's the difference? | 12:52 |
| tobiash | mordred: I've probably overlooked that | 12:53 |
| *** rlandy has joined #zuul | 12:53 | |
| tobiash | SpamapS: maybe you know ^ | 12:53 |
| tobiash | mordred: what's in the test-requirements is 'moto | 12:54 |
| tobiash | whatever that is | 12:54 |
| mordred | oh - so it is | 12:54 |
| mordred | I think that's a mock library for boto | 12:54 |
| mordred | makes more sense to my eyes now | 12:54 |
| tobiash | ah yepp 'Moto is a library that allows your python tests to easily mock out the boto library' | 12:54 |
| tobiash | mordred: I think you'll like https://review.openstack.org/634614 | 12:56 |
| mordred | tobiash: ooh | 12:56 |
| tobiash | that eliminates all remaning direct prints to stderr | 12:57 |
| tobiash | during the tests | 12:57 |
| mordred | that's so exciting | 12:58 |
| mordred | corvus, Shrews: https://review.openstack.org/#/c/535558 has 2x+2 now - but I left off the +A in case you wanted to look | 12:58 |
| tobiash | mordred: and this might or might not fix our executor memleak: https://review.openstack.org/634598 | 13:02 |
| tobiash | but it's something easy we can try | 13:03 |
| mordred | tobiash: seems reasonable to do even if it doesn't | 13:05 |
| tobiash | yes | 13:07 |
| *** bhavikdbavishi has quit IRC | 13:23 | |
| sshnaidm | can someone tell what means this error message in patch? tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001 finger://60273432fe24/9e3c5d73043b4d378b39b31f9fc4565d : DISK_FULL in 3h 13m 35s | 13:25 |
| sshnaidm | I don't think there is no space on disk.. | 13:25 |
| tobiash | sshnaidm: diskfull means that the job needed more space on the executor than it is allowed | 13:25 |
| sshnaidm | tobiash, on executor itself? | 13:26 |
| tobiash | yes | 13:26 |
| tobiash | the local workspace | 13:26 |
| sshnaidm | tobiash, and where can I configure/fix it?\ | 13:26 |
| tobiash | sshnaidm: it's a global zuul configuration | 13:26 |
| tobiash | sshnaidm: https://zuul-ci.org/docs/zuul/admin/components.html#attr-executor.disk_limit_per_job | 13:27 |
| tobiash | sshnaidm: openstack's limit seems to be here: http://git.openstack.org/cgit/openstack-infra/system-config/tree/manifests/site.pp#n812 | 13:28 |
| sshnaidm | tobiash, thanks.. how can I see what exactly executor save there? | 13:28 |
| tobiash | sshnaidm: you'd might need support from infra-root | 13:30 |
| sshnaidm | tobiash, it's not openstacks one, I run all this locally | 13:31 |
| tobiash | ah ok | 13:31 |
| *** jpena|lunch is now known as jpena | 13:31 | |
| tobiash | it counts the job's workspace minus cached repos | 13:31 |
| sshnaidm | tobiash, so disk full - is it related to executor machine or host where job is running? | 13:32 |
| tobiash | executor machine | 13:32 |
| sshnaidm | tobiash, and where is jobs workspace on executor machine? | 13:33 |
| tobiash | sshnaidm: normally it's in /var/lib/zuul-executor/jobs or similar | 13:34 |
| tobiash | ah no, default is /tmp | 13:34 |
| tobiash | https://zuul-ci.org/docs/zuul/admin/components.html#attr-executor.job_dir | 13:34 |
| sshnaidm | tobiash, I have /tmp/tmpt1wo_v42 but it's empty.. maybe because job finished? | 13:37 |
| sshnaidm | nothing else in /tmp | 13:37 |
| tobiash | sshnaidm: you can run 'zuul-executor keep' to not delete job workspaces | 13:38 |
| tobiash | but you should not run for a long time with 'keep' ;) | 13:38 |
| sshnaidm | tobiash, I see, thanks :) | 13:38 |
| tobiash | (on the executor) | 13:38 |
| sshnaidm | will try again | 13:38 |
| *** quiquell|rover is now known as quique|rover|lun | 13:38 | |
| *** quique|rover|lun is now known as quique|rover|lch | 13:38 | |
| *** panda|off is now known as panda | 13:40 | |
| *** quique|rover|lch is now known as quique|rover|eat | 13:40 | |
| sshnaidm | tobiash, is it possible to configure executor to run ansible with more verbosity? | 13:42 |
| tobiash | 'zuul-executor verbose' | 13:43 |
| sshnaidm | tobiash, and both "keep" and "verbose" could be passed to daemon, right? | 13:44 |
| tobiash | yes | 13:45 |
| tobiash | you can disable it at runtime then with 'zuul-executor nokeep' or 'zuul-executor unverbose' or just a restart of the executor | 13:46 |
| sshnaidm | tobiash, ack | 13:49 |
| *** sanjayu_ has quit IRC | 13:59 | |
| *** quique|rover|eat is now known as quiquell|rover | 14:02 | |
| openstackgerrit | Quique Llorente proposed openstack-infra/zuul master: Escape jinja2 stuff from inventory https://review.openstack.org/633930 | 14:11 |
| *** dmsimard has quit IRC | 14:18 | |
| *** dmsimard has joined #zuul | 14:18 | |
| jkt | sshnaidm: fyi, I had the same problem. It wasn't about "disk is full", but rather "my git repos are bigger than that configurable limit" | 14:19 |
| sshnaidm | jkt, ack, I updated it to 5GB now, let's see | 14:21 |
| *** gtema has quit IRC | 14:28 | |
| *** chandankumar has joined #zuul | 14:28 | |
| *** pwhalen has quit IRC | 14:29 | |
| *** jpena is now known as jpena|off | 14:34 | |
| *** pwhalen has joined #zuul | 14:38 | |
| *** chkumar|ruck has quit IRC | 14:38 | |
| *** SpamapS has quit IRC | 14:40 | |
| sshnaidm | 167M /tmp/...trusted/project_1/git.openstack.org/openstack-infra/project-config | 14:40 |
| sshnaidm | it takes most of default 250 M | 14:40 |
| *** jpena|off is now known as jpena | 14:40 | |
| *** SpamapS has joined #zuul | 14:40 | |
| *** chandankumar has quit IRC | 14:41 | |
| openstackgerrit | Quique Llorente proposed openstack-infra/zuul master: Escape jinja2 stuff from inventory https://review.openstack.org/633930 | 14:42 |
| arxcruz | hey guys, can we have some love on https://review.openstack.org/#/c/607077/ | 14:48 |
| arxcruz | I would love to see this on zuul :D | 14:48 |
| tobiash | arxcruz: I've added this to my review queue | 14:51 |
| arxcruz | tobiash: <3 | 14:52 |
| openstackgerrit | Quique Llorente proposed openstack-infra/zuul master: Escape jinja2 stuff from inventory https://review.openstack.org/633930 | 14:56 |
| *** sean-k-mooney has quit IRC | 14:57 | |
| *** rlandy has quit IRC | 14:58 | |
| *** rfolco is now known as rfolco_lunch | 15:01 | |
| openstackgerrit | Quique Llorente proposed openstack-infra/zuul master: Escape jinja2 stuff from inventory https://review.openstack.org/633930 | 15:02 |
| *** swest has quit IRC | 15:08 | |
| *** rlandy has joined #zuul | 15:12 | |
| *** sdake has joined #zuul | 15:13 | |
| *** rlandy has quit IRC | 15:13 | |
| *** rlandy has joined #zuul | 15:14 | |
| openstackgerrit | Sorin Sbarnea proposed openstack-infra/zuul-jobs master: Make install-docker compatible with centos https://review.openstack.org/633948 | 15:21 |
| openstackgerrit | Sorin Sbarnea proposed openstack-infra/zuul-jobs master: Make install-docker compatible with centos https://review.openstack.org/633948 | 15:21 |
| *** quiquell|rover is now known as quique|rover|off | 15:24 | |
| *** rfolco_lunch is now known as rfolco | 15:25 | |
| Shrews | tristanC: i think we could land https://review.openstack.org/590335 if you add the missing doc stuff | 15:55 |
| openstackgerrit | Merged openstack-infra/nodepool master: Amazon EC2 driver https://review.openstack.org/535558 | 16:13 |
| dmsimard | ^ SpamapS: woot, thanks for picking that up | 16:14 |
| Shrews | speaking of which, SpamapS, i totally missed that there were NO docs for that Amazon driver. Could you add that please? I shouldn't have approved it w/o that | 16:14 |
| *** sdake has quit IRC | 16:15 | |
| *** bhavikdbavishi has joined #zuul | 16:15 | |
| Shrews | well, several of us missed it I guess :) | 16:15 |
| tobiash | oops | 16:18 |
| Shrews | yeah, big oops | 16:18 |
| pabelanger | oh neat, ec2 landed | 16:24 |
| *** sdake has joined #zuul | 16:25 | |
| mordred | Shrews: bother | 16:27 |
| * mordred hears SpamapS is going to write us a GCE driver next :) | 16:27 | |
| *** pwhalen has quit IRC | 16:31 | |
| *** pwhalen has joined #zuul | 16:32 | |
| openstackgerrit | Merged openstack-infra/zuul master: Make git clone timeout configurable https://review.openstack.org/633936 | 16:33 |
| *** bhavikdbavishi has quit IRC | 16:34 | |
| *** bhavikdbavishi has joined #zuul | 16:34 | |
| *** sdake has quit IRC | 16:38 | |
| *** bhavikdbavishi1 has joined #zuul | 16:41 | |
| *** bhavikdbavishi has quit IRC | 16:42 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 16:42 | |
| SpamapS | Docs smocks. | 16:42 |
| SpamapS | ;) | 16:42 |
| SpamapS | I'll add them yeah. | 16:43 |
| corvus | yes, i'd love to review the ec2 driver. | 16:43 |
| corvus | Shrews: maybe you could wait till i wake up next time? :) | 16:44 |
| SpamapS | also I believe I have a local bugfix to submit which I've been holding off on to keep focus. | 16:47 |
| SpamapS | Also, honestly, thank you Shrews.. I've been off mainline and it will be good to not have to rebase to deploy new versions. ;) | 16:49 |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/nodepool master: EC2: Do not tag an instance until it is running https://review.openstack.org/634740 | 16:52 |
| *** bjackman has quit IRC | 16:52 | |
| * SpamapS madly writes docs before somebody decides to go and revert it | 16:53 | |
| corvus | SpamapS: thank you. :) | 16:54 |
| openstackgerrit | Merged openstack-infra/zuul master: Identify the gerrit source when logging stream-events errors. https://review.openstack.org/633567 | 16:55 |
| *** sdake has joined #zuul | 16:57 | |
| *** sdake has quit IRC | 17:06 | |
| *** dmellado has quit IRC | 17:07 | |
| *** dmellado has joined #zuul | 17:08 | |
| *** dmellado has quit IRC | 17:08 | |
| *** dmellado has joined #zuul | 17:09 | |
| *** sdake has joined #zuul | 17:23 | |
| sshnaidm | did anyone see problem when zuul finger stream stuck, but job continues to run and passed - "[Zuul] Log Stream did not terminate" | 17:26 |
| SpamapS | I love how writing docs reveals bugs in the code. :-P | 17:26 |
| *** chandankumar has joined #zuul | 17:59 | |
| *** hashar has quit IRC | 17:59 | |
| *** pcaruana has quit IRC | 18:00 | |
| SpamapS | For instance, we used flavor-name still, but it really should be instance-type. | 18:03 |
| *** jpena is now known as jpena|off | 18:04 | |
| *** bhavikdbavishi has quit IRC | 18:11 | |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/nodepool master: Remove unused fields from AWS driver https://review.openstack.org/634768 | 18:12 |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/nodepool master: Add documentation for AWS/EC2 driver https://review.openstack.org/634769 | 18:12 |
| SpamapS | Shrews: ^ ask and ye shall receive. | 18:12 |
| SpamapS | corvus: ^ | 18:12 |
| fungi | sshnaidm: that can happen if the console log streamer on the node is interrupted and not restarted, for example if the node is rebooted or the oom killer gets it | 18:14 |
| *** chandankumar has quit IRC | 18:19 | |
| daniel2 | Im trying to refactor a very old nodepool.yaml file, it appears that a lot of variables names have changed, is there a page that would have a changelog of the changes over time? I can't find much on that. | 18:21 |
| fungi | daniel2: how old is the old nodepool deployment? (what version was the config working in?) | 18:23 |
| openstackgerrit | Merged openstack-infra/nodepool master: Improve connection timeout log message https://review.openstack.org/632704 | 18:35 |
| daniel2 | fungi: about 2 years old, this is my first time working with nodepool though | 18:36 |
| *** bjackman has joined #zuul | 18:37 | |
| fungi | daniel2: do you know what version of nodepool that was? maybe 0.something? | 18:38 |
| zbr|ssbarnea | need reciew on https://review.openstack.org/633948 which enables us to reuse the install-docker role on centos, tested via https://review.rdoproject.org/r/#/c/18627/ tx | 18:39 |
| fungi | daniel2: most of the changes with the data model and overall design happened with the jump from nodepool 0.5.0 to 3.0.0 | 18:39 |
| daniel2 | Well, when did they drop gearman for zookeeper? | 18:39 |
| fungi | that was in 3.0.0 | 18:40 |
| daniel2 | https://github.com/a10networks-ci/project-config/blob/master/nodepool/nodepool.yaml | 18:40 |
| daniel2 | This is the actual config I'm trying to convert. | 18:40 |
| daniel2 | I noticed that a lot of the openstack stuff was moved to clouds.yaml. | 18:41 |
| fungi | right, nodepool 3.0.0 dropped support for jenkins too, so no longer needs any of the jenkins authentication/configuration bits | 18:42 |
| fungi | and yes the openstack api interactions moved to using the openstack sdk so you end up configuring the sdk with your credentials and endpoint info | 18:43 |
| fungi | (via clouds.yaml) | 18:43 |
| daniel2 | I've been going over the documentation, I'm figuring that any variables that dont exist in the latest docs have been completely removed. | 18:44 |
| daniel2 | There are no hidden settings are there? | 18:44 |
| SpamapS | daniel2: not by design. :) | 18:45 |
| daniel2 | famous last words, thanks for the clarification! | 18:45 |
| fungi | daniel2: there is an optional secure.conf where you can configure database credentials and connection info | 18:45 |
| daniel2 | So you said dropped jenkins support, so what is it suppose to use in its place? | 18:46 |
| fungi | daniel2: zuul uses ansible to directly execute jobs on test nodes as of v3 (well, really starting with zuul v2.5 but that was experimental and changed radically in v3) | 18:47 |
| Shrews | we also had a nodepool 2.5 (iirc) that was part zk, part gearman. only the builder used zookeeper, but launcher was still gearman based | 18:47 |
| Shrews | i think | 18:47 |
| Shrews | so long ago | 18:48 |
| fungi | Shrews: if we did, it was never tagged that i can see | 18:48 |
| fungi | it may have just been an interim state after nodepool 0.5.0 | 18:48 |
| Shrews | yah, i don't think we tagged anything | 18:49 |
| Shrews | that may have been on the feature/zuulv3 branch too | 18:50 |
| openstackgerrit | Merged openstack-infra/zuul master: Properly close streams https://review.openstack.org/634598 | 18:51 |
| fungi | oh, is the nodepool secure.conf [database] section on our builders and launchers cruft now? i can't find it mentioned in the docs... on our servers it refers to a mysql running on localhost but we're not running mysql on them | 18:51 |
| Shrews | fungi: not cruft, but optional now (was required) | 18:52 |
| daniel2 | So whats the earlier version I can go to that supports jenkins? Sounds like refactoring to ansible isn't much of an option right now for us. | 18:52 |
| Shrews | i think we intend to store zk creds there | 18:52 |
| fungi | daniel2: yes, nodepool>=3 is intended to be used in conjunction with zuul>=3 which implies rewriting your jobs in ansible | 18:53 |
| daniel2 | Looking at github it goes from 0.5.0 to 3.0.0 | 18:53 |
| corvus | (keep in mind, that if your jobs already are, or can be turned into simple shell scripts, you can have ansible run that shell script pretty easily. this is what we did for openstack) | 18:53 |
| fungi | i believe the last release of nodepool with jenkins support was 0.5.0 (the tag message confirms that too) | 18:53 |
| daniel2 | I just transitioned from operations/sre to developer recently, so I see both sides of the fence. I believe things should be maintained and updated, my colleagues dont always agree with that :) | 18:54 |
| SpamapS | daniel2: ^5 | 18:55 |
| fungi | daniel2: keep in mind nodepool 0.5.0 was tagged barely over one year ago, so it's not exactly ancient but we've not kept up with what newer jenkins api expectations might be in the intervening timeframe so you may need a jenkins version contemporary with it | 18:55 |
| *** bjackman has quit IRC | 18:56 | |
| corvus | we made the change in zuulv3 so that we *could* support the system. maintaining it as a bolt-on afterthought to jenkins wasn't sustainable in the long term. | 18:56 |
| daniel2 | Yeah, so I'm going to propose standing everything up with the older stuff, and later on setting up the newer environment to move over time. | 18:56 |
| fungi | (which quite likely is no longer supported by jenkins upstream and therefore may have security vulnerabilities you'll want to find ways to mitigate) | 18:56 |
| fungi | i honestly don't know what the lts timeframe for jenkins is | 18:57 |
| fungi | nor to what extent they might make breaking changes in an lts series which might make nodepool need to adapt | 18:57 |
| SpamapS | just | 18:58 |
| SpamapS | breathe | 18:58 |
| SpamapS | ;) | 18:58 |
| corvus | daniel2: the best way to make the future move easier is to avoid using jenkins plugins. each one of those is more lock-in to jenkins. anything you can move into a shell builder is portable. you don't need to have jenkins run ansible (but you can) -- jenkins running shell, then switching to ansible running shell is fine. | 18:58 |
| * SpamapS gets a little antsy whenever he thinks about using Jenkins | 18:58 | |
| corvus | (the converse is true too -- if you migrate to shell or ansible, you won't be locked into zuul either) | 19:00 |
| openstackgerrit | Merged openstack-infra/zuul master: Add web/node_modules to dockerignore https://review.openstack.org/633402 | 19:03 |
| mordred | SpamapS: you've angered the pep8 gods | 19:08 |
| *** dkehn has quit IRC | 19:10 | |
| pabelanger | mordred: do you have any ideas why we see 'Waiting for logger' on node that was added using 'add_host' ? https://object-storage-ca-ymq-1.vexxhost.net/v1/a0b4156a37f9453eb4ec7db5422272df/logs/periodic-1hr/github.com/ansible-network/windmill-config/master/windmill-config-deploy/a6f79da/job-output.html#l114 | 19:18 |
| pabelanger | I believe the firewalls are correctly open, as we eventually see the task output, just not in realtime | 19:19 |
| mordred | pabelanger: I'm guessing you didn't run zuul_console on that node | 19:19 |
| mordred | pabelanger: if you add a zuul_console: task after the add_host - it should start working for you | 19:20 |
| pabelanger | mordred: started: https://object-storage-ca-ymq-1.vexxhost.net/v1/a0b4156a37f9453eb4ec7db5422272df/logs/periodic-1hr/github.com/ansible-network/windmill-config/master/windmill-config-deploy/a6f79da/job-output.html#l76 | 19:20 |
| pabelanger | Going to dig more today | 19:21 |
| mordred | pabelanger: hrm. | 19:21 |
| mordred | pabelanger: I'd expect that to work | 19:21 |
| pabelanger | I can only think I some how messed up firewall access | 19:21 |
| openstackgerrit | Merged openstack-infra/nodepool master: EC2: Do not tag an instance until it is running https://review.openstack.org/634740 | 19:23 |
| SpamapS | mordred: I kneel before the pep8 alter and beg forgiveness | 19:27 |
| * SpamapS will fix.. damnit. | 19:27 | |
| tobiash | SpamapS: the doc build is broken too ;) | 19:35 |
| openstackgerrit | Merged openstack-infra/zuul master: Move isChangeReadyToBeEnqueued after pipeline requirements https://review.openstack.org/634191 | 19:48 |
| SpamapS | looks like I left a few bits off the docs too | 20:32 |
| * SpamapS diving back in | 20:33 | |
| openstackgerrit | Merged openstack-infra/zuul master: Test coverage to exclude pipeline with untrusted project https://review.openstack.org/631990 | 20:38 |
| tobiash | change 590697 would need a second review. It hardens zuul against git failures during job preparation. | 20:43 |
| openstackgerrit | Merged openstack-infra/zuul master: Properly log requests in SimpleHTTPRequestHandler https://review.openstack.org/634614 | 20:45 |
| daniel2 | corvus: what do you mean by jenkins plugins? like the nodepool jenkins plugin? | 20:46 |
| corvus | daniel2: https://plugins.jenkins.io/ | 20:46 |
| daniel2 | We aren't using that. When I described the ansible method and using shell scripts, which is literally what we do with jenkins currently, I think he was open to it. | 20:46 |
| daniel2 | corvus: yes, I know what jenkins plugins are. | 20:46 |
| corvus | daniel2: i guess i don't understand your question :) | 20:46 |
| daniel2 | You said dont use jenkins plugins and the only plugin I could think of that we would be using is the nodepool one. | 20:47 |
| daniel2 | I was kind of just trying to clarify | 20:47 |
| corvus | daniel2: ah, i meant plugins used in jenkins jobs, so like using a maven plugin or something | 20:48 |
| corvus | daniel2: if your stuff is mostly shell scrips already, then migration should be relatively straightforward | 20:48 |
| daniel2 | Honestly, instead of trying to "convert" the old config, I'm going to just rewrite it from scratch. I feel it's the best method than trying to plug everything in./ | 20:51 |
| corvus | daniel2: quite possibly true | 20:51 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: DNM: Test for 633948 https://review.openstack.org/634805 | 20:51 |
| openstackgerrit | Merged openstack-infra/zuul master: web: clear status when changing tenant https://review.openstack.org/634541 | 20:58 |
| openstackgerrit | Merged openstack-infra/zuul master: Fix stuck job caused by exception during repo update https://review.openstack.org/590697 | 21:18 |
| tobiash | \o/ | 21:22 |
| pabelanger | could I get a few reviews on https://review.openstack.org/592160/ please | 21:24 |
| corvus | pabelanger: what's it for? | 21:24 |
| corvus | (so not everyone in the channel has to click the url) | 21:25 |
| pabelanger | being able for a job to run a list of playbooks | 21:25 |
| corvus | i'd like to review that, but need to finish some stuff first; i'll leave a comment asking folks not to +w it | 21:25 |
| pabelanger | ack | 21:26 |
| corvus | more reviews welcome, but if no one else gets around to it, i can +3 it later today or tomorrow if it looks good | 21:27 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Use node v10 instead of node v8 https://review.openstack.org/632165 | 21:27 |
| pabelanger | wfm, thanks | 21:28 |
| SpamapS | Hm, there's something in the way the zuul web UI uses URL routing that makes a click on a link under the same domain as the web UI try to fetch /api/info .... makes it hard to host logs on a different path in the same vhost. | 21:35 |
| daniel2 | So I've got nodepool running, however, I can't seem to find where private-key would go now. | 21:35 |
| daniel2 | Where you put the path to the ssh key created when install nodepool/zuul. | 21:36 |
| corvus | tristanC, tobiash, mordred: ^ i think SpamapS issue is related to the javascript app system thingy... what's it called? | 21:36 |
| mordred | corvus: we call that "something something javascript something" | 21:36 |
| corvus | it's the thing that makes it work on your phone | 21:36 |
| mordred | oh - the progressive web app | 21:37 |
| corvus | service worker! | 21:37 |
| mordred | yes! | 21:37 |
| corvus | (thanks, i got there via pwa) | 21:37 |
| mordred | I'm surprised that the links are getting intercepted by the js routing though | 21:37 |
| mordred | in the react we have to use the Link construct to make it do the things | 21:38 |
| mordred | I'm not doubting it is, obviously, it's just surprising behavior - and one we should definitely figure out and fix as I think SpamapS use case is a valid one | 21:38 |
| corvus | daniel2: nodepool doesn't use a private key anymore; only the zuul-executor | 21:38 |
| daniel2 | ah thank you for that! | 21:39 |
| corvus | daniel2: (nodepool will connect to the nodes it creates, but only to verify that it can and to fetch ssh host keys, it doesn't actually log in) | 21:39 |
| mordred | SpamapS: out of curiosity, can you tell me the root of the path you're using? and also - you're doing single-tenant/whitelabel - are you doing suburl too? | 21:40 |
| corvus | daniel2: (if you are using nodepool v2 ready scripts to do things on hosts before putting them in service, in zuul v3 you can use a zuul pre-playbook for that) | 21:40 |
| daniel2 | corvus: that was actually about to be another question of mine | 21:42 |
| daniel2 | Is there any guides on converting those? | 21:43 |
| mordred | SpamapS: can you also share how you're doing the whitelabeling? | 21:43 |
| corvus | daniel2: not explicitly, no, but that would be a good addition | 21:43 |
| corvus | daniel2: https://zuul-ci.org/docs/zuul/user/config.html#attr-job.pre-run if you do that in a base job, then it automatically happens for every job | 21:43 |
| corvus | daniel2: (a benefit over the old system is that failures here are now visible to users, but we still retain some of the robustness -- zuul will try 3 times to run a job with a failing pre-playbook before reporting a failure to the user) | 21:44 |
| corvus | daniel2: (and that's adjustable: https://zuul-ci.org/docs/zuul/user/config.html#attr-job.attempts ) | 21:45 |
| corvus | old nodepool would just respawn nodes indefinitely, which made things look good, until they got really really bad. | 21:46 |
| daniel2 | Ah, my next step is to rewrite the layout.yaml file, so that gave me the info I needed. | 21:48 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Return artifacts as dicts and add metadata https://review.openstack.org/634825 | 22:30 |
| corvus | mordred, tobiash, SpamapS, pabelanger: ^ i'd love some feedback on that idea -- i was just writing roles for the docker registry idea and realized that those changes would make things much nicer | 22:31 |
| mordred | corvus: I like the way the example in the docs reads | 22:34 |
| corvus | pabelanger: +2 on 592160, but i have a request for 2 improvements. could be an amend or a followup. | 22:43 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: WIP: Add intermediate registry push/pull roles https://review.openstack.org/634829 | 22:54 |
| ianw | are we aware of current log streaming issues? | 22:57 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Return artifacts as dicts and add metadata https://review.openstack.org/634825 | 22:59 |
| corvus | ianw: not generally, it's come up in a few recent questions, but i think so far they've all been attributed to local job actions. | 23:01 |
| SpamapS | mordred: I am doing single-tenant whitelabel yes. I do so with nginx. Not sure if I do suburl. | 23:01 |
| SpamapS | The /logs path is intercepted at the k8s ingress level, so it never actually talks to zuul-web, it just goes right to my logs proxy. | 23:02 |
| corvus | SpamapS: suburl meaning serving zuul under something other than / (eg, http://zuul.openstack.org does not use a suburl) | 23:02 |
| corvus | http://example.com/zuul would be a suburl | 23:02 |
| SpamapS | http://paste.openstack.org/show/743138/ <-- that's my nginx config | 23:03 |
| SpamapS | and it all ends up under a single host, so no, no suburl. | 23:03 |
| SpamapS | when I inspect links, they don't have any on-click attached that I can see, so I assume it's something deeper | 23:05 |
| openstackgerrit | Ian Wienand proposed openstack-infra/nodepool master: [wip] add openstackci-mirrors element for centos/ubuntu testing https://review.openstack.org/634366 | 23:12 |
| corvus | SpamapS: yeah, i suspect it's something service-worker related -- aiui, they can do that sort of thing. but tristanC, mordred, and tobiash are more knowledgable about that than i am (as i have told you everything) | 23:14 |
| corvus | istr tobiash proposed a change to just turn off the service worker because of issues he was having | 23:14 |
| SpamapS | K, for now I've trained my users to shift-refresh when they see the Loading ... | 23:16 |
| SpamapS | But would obviously like that to be temporary. | 23:16 |
| openstackgerrit | Ian Wienand proposed openstack-infra/nodepool master: [wip] add openstackci-mirrors element for centos/ubuntu testing https://review.openstack.org/634366 | 23:22 |
| *** rlandy is now known as rlandy|bbl | 23:22 | |
| *** sdake has quit IRC | 23:28 | |
| *** sdake has joined #zuul | 23:30 | |
| * SpamapS finding so many unused attr's in the aws driver | 23:31 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/nodepool master: [wip] add openstackci-mirrors element for centos/ubuntu testing https://review.openstack.org/634366 | 23:34 |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/nodepool master: Remove unused fields from AWS driver https://review.openstack.org/634768 | 23:36 |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/nodepool master: Add documentation for AWS/EC2 driver https://review.openstack.org/634769 | 23:36 |
| SpamapS | ^^ docs with even more fields (maybe even all of them) and removed even more unused things | 23:36 |
| *** panda has quit IRC | 23:40 | |
| *** panda has joined #zuul | 23:41 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!