| pabelanger | so, it seems like we cannot force delete nodes in nodepool that are locked and in-use | 01:53 |
|---|---|---|
| pabelanger | which, is something we could do in v2 | 01:53 |
| pabelanger | http://zuulv3.openstack.org/static/stream.html?uuid=6b1bef6cc9be4c96ba896df5d6ac8f26&logfile=console.log | 01:53 |
| pabelanger | I am guessing that is going to timeout for the post playbook in 3 hours | 01:53 |
| pabelanger | due to networking issue in inap | 01:53 |
| pabelanger | and, have no idea / way of having zuul abort that just right now | 01:54 |
| pabelanger | maybe I could try killing the ansible process | 01:54 |
| clarkb | kill the ansible process | 01:54 |
| clarkb | ya | 01:54 |
| pabelanger | yah | 01:54 |
| pabelanger | let me try that | 01:54 |
| pabelanger | k, that did it | 01:56 |
| *** threestrands has joined #zuul | 02:09 | |
| openstackgerrit | Rui Chen proposed openstack-infra/nodepool feature/zuulv3: Fix nodepool cmd TypeError when no arguemnts https://review.openstack.org/519582 | 03:30 |
| openstackgerrit | Rui Chen proposed openstack-infra/nodepool feature/zuulv3: Apply floating ip for node according to configuration https://review.openstack.org/518875 | 03:31 |
| *** bhavik has joined #zuul | 04:05 | |
| *** bhavik has quit IRC | 04:11 | |
| *** threestrands has quit IRC | 06:34 | |
| *** tobiash has quit IRC | 06:43 | |
| *** tobiash has joined #zuul | 06:44 | |
| openstackgerrit | Merged openstack-infra/nodepool feature/zuulv3: Be consistent with the ZK data model https://review.openstack.org/519706 | 06:52 |
| *** mmedvede has quit IRC | 07:59 | |
| *** mmedvede has joined #zuul | 08:06 | |
| *** nguyentrihai has joined #zuul | 13:33 | |
| tobiash | dmsimard: I've left comments on https://review.openstack.org/#/c/518374/1 | 13:39 |
| dmsimard | tobiash: yes, that's fair, it's copy/pasta from a thing I did in the past where I heavily standardized on quoting everything :P | 13:47 |
| dmsimard | that's why I threw a WIP on there | 13:47 |
| dmsimard | also need to write integration tests for it | 13:48 |
| tobiash | dmsimard: do you think the service needs to be enables? | 13:49 |
| tobiash | the only use case I currently see in enabling (instead of just starting) is that you also support jobs that reboot the node | 13:49 |
| dmsimard | mordred, pabelanger: I forgot to mention yesterday, but another approach I was thinking about for testing base roles/playbooks is to somehow have them self-tested. I need to think about the "somehow" -- maybe like a "<role>/tests/main.yaml" that gets included when we know we're inside a testing environment that involves that role (i.e, not a nova patch but a <role> patch) | 13:50 |
| dmsimard | It still doesn't help us testing things that can't be speculatively tested, but it makes it easier to "scale" because we don't have to bounce back and forth between z-j, o-z-j and project-config to define integration tests for the different roles | 13:51 |
| dmsimard | Wow you people thought about everything https://docs.openstack.org/infra/zuul/feature/zuulv3/admin/tenants.html?highlight=shadow#attr-tenant.untrusted-projects.%3Cproject%3E:.include | 14:12 |
| dmsimard | everyone++ | 14:12 |
| mordred | dmsimard: :) | 14:32 |
| mordred | dmsimard: yah, I had a similar thought while talking to folks in sydney but forgot to mention when we were tlaking yesterday - thanks for having the thought AND saying something :) | 14:32 |
| mordred | dmsimard: basically - for a lot of the roles in base job we should be able to make synthetic test playbooks (kind of like how roles on galaxy will do 'unit' testing) | 14:33 |
| mordred | dmsimard: that should at least allow us to have tests that the role does the thing it says - and those can obviously be safely run as tests even speculatively | 14:33 |
| dmsimard | mordred: the tests would run, but not with the patch in progress | 14:35 |
| mordred | dmsimard: the tests would totally run with the patch in progress | 14:39 |
| mordred | dmsimard: the non-speculative is only for the job config | 14:39 |
| mordred | but if the job is "run the test playbook from {{ zuul_work_dir }}/tests" - then that will run in the correctly checked out copy of the repo on the remote node | 14:39 |
| mordred | dmsimard: isn't it great how much this particular area makes the brain ache | 14:40 |
| dmsimard | mordred: ok so you can speculatively (typing any derivative of the speculative word hurts my brain) test changes to code in roles and playbooks but not jobs themselves | 14:44 |
| SpamapS | pabelanger: clarkb I noticed the "how can I abort this?" discussion yesterday. Seems like 'abort' would be a good thing for zuul client to implement. | 14:46 |
| rcarrillocruz | are we good to merge https://review.openstack.org/#/c/500800/7 | 14:47 |
| dmsimard | tobiash: yeah, I'll get a new patchset up once I have a chance. I wanted to get something up to show someone | 14:47 |
| rcarrillocruz | Shrews , mordred ^ | 14:47 |
| rcarrillocruz | i'm trying to move that stack a bit | 14:47 |
| rcarrillocruz | upper changes will be need a rebase, will do as things move along | 14:47 |
| dmsimard | mordred: btw in the objective of making zuul-jobs re-usable outside of openstack, we already have a couple "bugs". I'd like to document those somewhere.. do you have a preference ? | 14:51 |
| dmsimard | Bugs range from assumptions because things are there in the default openstack nodepool images, to assuming we're running certain jobs on ubuntu and stuff like that. | 14:52 |
| tristanC | dmsimard: mordred: https://etherpad.openstack.org/p/downstream-zuul-jobs-issues | 14:59 |
| dmsimard | bah I created https://etherpad.openstack.org/p/zuul-jobs-issues but let's use yours sure :p | 14:59 |
| mordred | tristanC, dmsimard: I think that's a great idea ... and also those are likely great things to make synthetic role tests too (no reason we can put a tests dir in zuul-jobs with playbooks to test the roles - and even define a job in zuul-jobs that can be used run those playbooks | 15:08 |
| dmsimard | mordred: yeah, I'll send a WIP to self-test a role, I have an idea | 15:08 |
| mordred | tristanC, dmsimard: I also think it would be pretty cool to get third-party ci going for zuul-jobs | 15:09 |
| dmsimard | mordred: yeah I was discussing that on our end.. how do you see that happening ? Do you care if, for example, our zuul would report a syntax failure on our zuul because of a change in zuul-jobs for example ? | 15:09 |
| dmsimard | fwiw we're planning on including as little as possible to avoid exactly that kind of issue where a commit in z-j would break our zuul | 15:10 |
| dmsimard | with shadow and include https://docs.openstack.org/infra/zuul/feature/zuulv3/admin/tenants.html?highlight=shadow#attr-tenant.untrusted-projects.%3Cproject%3E:.include | 15:10 |
| mordred | dmsimard: yah - I think that would be important to show - since zuul-jobs really shouldn't be introducing syntax-errors like that (unless you just happened to have a pre-existing job with a conflicting name, but even then it would be good for us to be aware of so that we can learn how to communicate abot stuff) | 15:11 |
| mordred | dmsimard: it should be safe foryou to consume zuul-jobs without needing to do a filtered include (although shadow is important) - we avoid defining pipelines there already | 15:12 |
| mordred | dmsimard: so if it's not safe to do that we should probably talk about how we can make changes to make consuming zuul-jobs as easy as possible (obviously do what you need to - just saying I think the upstream intent for that repo should be that you should not NEED to) | 15:13 |
| dmsimard | right | 15:16 |
| rcarrillocruz | speaking of which, anyone aware of other groups using zuulv3 where they keep their jobs / configs | 15:19 |
| rcarrillocruz | i'd like to get more ideas on how they reuse openstack zuul jobs | 15:19 |
| rcarrillocruz | and the likes | 15:19 |
| rcarrillocruz | tobiash: you folks have a github or repo somewhere for this? | 15:19 |
| tristanC | rcarrillocruz: this is how we configure upstream zuul-jobs: https://softwarefactory-project.io/r/#/c/10231/1/ansible/roles/sf-repos/templates/config/zuulV3/_main.yaml.j2 | 15:21 |
| rcarrillocruz | oh neat | 15:21 |
| rcarrillocruz | thx | 15:21 |
| tobiash | rcarrillocruz: I don't have anything on github currently, but I have it similar like you | 15:22 |
| tobiash | just using exclude instead of include | 15:22 |
| dmsimard | rcarrillocruz: openstack-zuul-jobs is not meant to be used outside of OpenStack, zuul-jobs is | 15:22 |
| tobiash | hrm, similar like tristanC ;) | 15:22 |
| rcarrillocruz | yeah, ino | 15:22 |
| rcarrillocruz | thx | 15:23 |
| tristanC | rcarrillocruz: though we are looking forward a better git driver so that we don't need a gerrit connection to openstack | 15:23 |
| rcarrillocruz | you can't use github? | 15:23 |
| rcarrillocruz | haven't tried, but was thinking about it | 15:23 |
| tobiash | but I'm just I'm in process of starting to make real usage of zuul-jobs | 15:23 |
| tobiash | but currently fighting with deploying openshift in a productive environment as a platform to host zuul etc | 15:24 |
| tristanC | rcarrillocruz: probably, but it sounds better to get it dirrectly from openstack.org | 15:24 |
| tobiash | so that's currently a bit suspended until I have that running again | 15:24 |
| mordred | rcarrillocruz: well - to use github you'd need to get us to add your github app to the openstack zuul-jobs repo on github | 15:24 |
| mordred | rcarrillocruz: but that's why we want to get the git driver updated so thatyou can just point at the git url for it and have that work | 15:25 |
| rcarrillocruz | ic | 15:25 |
| rcarrillocruz | btw tristanC , saw your providerconfig refactor | 15:26 |
| rcarrillocruz | i was thinking about poking at an ansible driver | 15:26 |
| rcarrillocruz | but then i figured is a bit in flux the the driver interface | 15:26 |
| rcarrillocruz | is that going to merge? | 15:26 |
| rcarrillocruz | i 've seen a few +2 | 15:26 |
| rcarrillocruz | i could +A, but not sure if that's on hold for a reason | 15:26 |
| tristanC | rcarrillocruz: i'd love to have the providerconfig refactor merged yes | 15:27 |
| * mordred wants to merge many of tristanC's patches soon | 15:27 | |
| tristanC | rcarrillocruz: and fwiw, the opencontainer driver is using ansible to create and delete slave | 15:28 |
| mordred | tristanC, rcarrillocruz: think Shrews and I were waiting on jeblair to take a look at the refactor patch since it's introducing a fundamental api | 15:28 |
| rcarrillocruz | mordred: gotcha | 15:28 |
| rcarrillocruz | tristanC: ah cool | 15:28 |
| rcarrillocruz | thinking of creating a very generic one | 15:29 |
| tristanC | mordred: happy to fix what's needed | 15:29 |
| rcarrillocruz | where we could hook up playbook for create/delete | 15:29 |
| tristanC | rcarrillocruz: yes sure, that would be useful | 15:29 |
| rcarrillocruz | what i do at ansible is network modules | 15:29 |
| rcarrillocruz | and vendors are like 'omg, CI is cool' | 15:29 |
| rcarrillocruz | like they're totally discovering this thing like it's a brand new thing | 15:29 |
| rcarrillocruz | and we use nodepool internally for ourselves | 15:29 |
| rcarrillocruz | and we use network appliacnces as VMs | 15:29 |
| rcarrillocruz | but the idea is to get vendors as 3rd party CI to use Nodepool to test stuff on gear we can't possibly test | 15:30 |
| tristanC | rcarrillocruz: right now i used a json dump in the playbook so that it can communicate some information back to nodepool | 15:30 |
| rcarrillocruz | so i see a great use case here to write an ansible driver that will jsut leverage our modules do pre-test and post-test stuff | 15:30 |
| tristanC | (last task of https://review.openstack.org/#/c/468753/25/nodepool/driver/oci/playbooks/init.yml ) | 15:31 |
| rcarrillocruz | heh, so you pass out data via a file | 15:31 |
| tristanC | rcarrillocruz: there is surely a better way, but that works ok | 15:32 |
| rcarrillocruz | no wonder, output of ansible-playbook is horrible (unless a custom callback is written) | 15:32 |
| rcarrillocruz | i see totally why you do it, ansible is terrible to give text output in an easy way to consume | 15:32 |
| tristanC | rcarrillocruz: don't tell me about it, i just wrote some awfull code to remove the std(out|err)_lines lists, in particular when ansible-playbook calls are nested | 15:34 |
| rcarrillocruz | lulz | 15:34 |
| rcarrillocruz | i could share similar stories on our DCI hooks, some bash awk tthing :D | 15:34 |
| tristanC | for what it worth, my gory remove_ansible_std_lines_lists http://paste.openstack.org/show/626409/ | 15:35 |
| mordred | rcarrillocruz, tristanC: yah - this is why we wrote zuul_return and had playbooks that need to communicate with zuul just write out a json file :) | 15:37 |
| tristanC | mordred: rcarrillocruz: yes, hence having a generic nodepool ansible driver to facilate that would be neat to herit from | 15:38 |
| mordred | yah - a generic ansible driver also strikes me as a nice easy button for people who have 'weird' setups but where just writing a little ansible would be an easy thing for them to do | 15:39 |
| rcarrillocruz | like | 15:39 |
| rcarrillocruz | we have internally also an ESX | 15:39 |
| rcarrillocruz | i don't feel like writing a driver for it, but is cooler to just use ansible to use vmware modules to do it | 15:40 |
| mordred | yah | 15:40 |
| rcarrillocruz | i looked at linchpin, but dunno... the layout syntax is a bit odd to me | 15:40 |
| mordred | and then maybe you reach a point where you're like "I could really use a native nodepool driver" | 15:40 |
| rcarrillocruz | and anyways, for our use case of physical network gear, we need plain ansible | 15:40 |
| mordred | rcarrillocruz: ++ | 15:41 |
| pabelanger | SpamapS: ++ | 16:01 |
| *** hashar has joined #zuul | 17:01 | |
| SpamapS | A generic ansible driver would solve a lot actually. | 17:05 |
| SpamapS | Not that it's hard to write python. | 17:05 |
| SpamapS | also, I'd really like to have a "generic thing pool" not just nodes. I have an instance where I have a pool of accounts on a cloud, and I want to only hand out one at a time. Strikes me that it's similar to the use case for static nodes. | 17:06 |
| SpamapS | s/only hand out one at a time/hand out each one to one job at a time/ | 17:07 |
| SpamapS | So if we boiled down nodepool to thingpool, and then nodepool is just "things that turn into inventories" .. and you can have a job also request thingset's and just get back values that are useful in whatever context your job needs them in, that would be sweet. | 17:08 |
| SpamapS | Like if one wanted to run jobs on kubernetes, just handing out namespaces with their own creds on the k8s, would be super useful. | 17:09 |
| pabelanger | yah, I'd love to see a ansible nodepool driver, I know linchpin was tossed around a few times, but think that might be a little over the top in this context. Then you could do oci / docker / containers via ansible. | 17:11 |
| rcarrillocruz | dummy question, i believe there's a noop job that runs from executor, so there's no need for a nodepool node | 17:12 |
| rcarrillocruz | where is that defined | 17:12 |
| rcarrillocruz | as a base job in project-config ? | 17:12 |
| rcarrillocruz | my grep is failing if so | 17:12 |
| pabelanger | rcarrillocruz: not noop, but you can just define a job without nodesets, IIRC. Look for rtfd post hook job | 17:14 |
| pabelanger | that just runs curl from executor | 17:14 |
| *** hashar is now known as hasharAway | 17:28 | |
| rcarrillocruz | Thx, will take a look | 17:34 |
| Shrews | should just be... nodes: [] | 17:38 |
| tobiash | rcarrillocruz: noop is a special job defined in zuul itself | 17:39 |
| rcarrillocruz | so i can totally use 'noop' for test, despite not being a job defined as a playbook anywhere | 17:40 |
| rcarrillocruz | ? | 17:40 |
| tobiash | Yes | 17:41 |
| rcarrillocruz | i'm looking at projec-config base jobs, but there's quite a bit of openstackisms, and i'd like to start small | 17:41 |
| rcarrillocruz | oh neat | 17:41 |
| rcarrillocruz | let me try then | 17:41 |
| tobiash | noop is just an empty job that does nothing | 17:41 |
| tobiash | also it's totally valid to have a real job without nodes | 17:42 |
| tobiash | if a job does only things it is allowed to do on the executor | 17:43 |
| rcarrillocruz | wootz! | 17:46 |
| rcarrillocruz | https://github.com/rcarrillocruz-org/zuul-tested-repo/pull/7 | 17:46 |
| rcarrillocruz | making progress | 17:46 |
| tobiash | Success in 0s :) | 17:47 |
| tobiash | Every job should be that fast | 17:47 |
| rcarrillocruz | specially integration tests , lulz | 17:47 |
| rcarrillocruz | hmm, try putting nodeset withj empty nodes | 18:07 |
| rcarrillocruz | not getting anything spit | 18:07 |
| rcarrillocruz | looks like that's equivalent to not dfefining a nodeset in a job | 18:08 |
| rcarrillocruz | whatever is | 18:08 |
| rcarrillocruz | not getting anything reported back on the PR for this | 18:08 |
| rcarrillocruz | https://github.com/rcarrillocruz-org/zuul-tested-repo/pull/8/files | 18:08 |
| rcarrillocruz | what should I need to get a basic hello-world | 18:08 |
| pabelanger | rcarrillocruz: try: http://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul.d/jobs.yaml#n848 | 18:15 |
| Shrews | rcarrillocruz: did you define a base job in your config repo? not sure what happens if you don't do that | 18:16 |
| rcarrillocruz | I didn't | 18:23 |
| rcarrillocruz | Probably that | 18:24 |
| rcarrillocruz | Will create one | 18:24 |
| dmsimard | Shrews: does the "rate" config handle things other than "provider" specific requests ? | 19:02 |
| dmsimard | Shrews: like, zookeeper operations | 19:02 |
| clarkb | dmsimard: it shouldn't | 19:03 |
| Shrews | dmsimard: nope, just provider | 19:03 |
| dmsimard | I know that rate is basically the amount of time to wait between requests (say, to create an instance you need to list machines, list ports, images, networks, create instance, floating ip, etc.) | 19:03 |
| clarkb | dmsimard: its purely to provide you with a mechanism to work around api rate limits in providers | 19:03 |
| dmsimard | ok | 19:03 |
| dmsimard | just making sure | 19:03 |
| clarkb | because many clouds rate limit their apis | 19:03 |
| dmsimard | FWIW I'm working on the zuul_json issue (yes, it's still not fixed) out of Software Factory's zuul v3 | 19:04 |
| dmsimard | reproduced it \o/ http://paste.openstack.org/raw/626431/ | 19:16 |
| *** tobiash has quit IRC | 19:27 | |
| *** tobiash has joined #zuul | 19:27 | |
| dmsimard | broadening the scope of the issue a bit... | 19:51 |
| dmsimard | When there is a log url returned in a zuul job comment (comes from here http://git.openstack.org/cgit/openstack-infra/zuul/tree/zuul/executor/server.py?h=feature/zuulv3#n731 ), it looks like this for a "successful" job | 19:53 |
| dmsimard | 2017-11-15 19:50:28,735 DEBUG zuul.AnsibleJob: [build: 32f9b42db83b4ac29b5d04fa9cbf3de0] Sending result: {"data": {"zuul": {"log_url": "https://some-server/logs/7/7/4/check/break-things/32f9b42/"}}, "result": "SUCCESS"} | 19:53 |
| dmsimard | When we look at a result from a job which returned a finger:// URL, we see this instead: | 19:53 |
| dmsimard | 2017-11-15 19:48:19,795 DEBUG zuul.AnsibleJob: [build: 723f63806cda4e38bba50b1142122cc6] Sending result: {"data": {}, "result": "POST_FAILURE"} | 19:54 |
| dmsimard | We should probably not return a finger URL (ever?), it has no purpose. Should we make sure that the log url is at the very least properly sent back, even if that log location ends up being empty because the log upload failed ? | 19:54 |
| dmsimard | Oh damn now I understand | 19:55 |
| dmsimard | The executor actually uses the json file, and it uses that to work out details from the job | 19:56 |
| dmsimard | In this particular case the json file is broken so everything explodes.. but why does it rely on the json file ? It doesn't know where the logs will be uploaded until the role to upload the logs is actually ran perhaps ? | 19:57 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Make build-python-release job https://review.openstack.org/513925 | 20:25 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Remove old python-sdist job https://review.openstack.org/513926 | 20:30 |
| mordred | just got a fascinating test job fail .. | 20:33 |
| mordred | tobiash, Shrews, clarkb, pabelanger: have we seen anything like http://logs.openstack.org/25/513925/2/check/tox-py35-on-zuul/83b3855/job-output.txt.gz#_2017-11-15_20_27_57_762232 | 20:34 |
| mordred | I'm inclined to think it was some really weird hiccup ... | 20:34 |
| dmsimard | mordred: haven't seen that before | 20:35 |
| mordred | dmsimard: yes - the role to upload the logs is the thing that tells zuul where the logs are | 20:35 |
| dmsimard | mordred: is there anything in that job that modifies the git config ? | 20:35 |
| mordred | dmsimard: I mean, there's TONS of things in the zuul test that do all sorts of things with git config - but that's a patch to zuul-jobs, so really shouldn't have been able to break anything | 20:36 |
| mordred | I'm going to recheck and see if it persists | 20:36 |
| Shrews | mordred: facinating | 20:36 |
| mordred | right? | 20:36 |
| Shrews | nope, never seen that | 20:36 |
| pabelanger | also new to me | 20:37 |
| dmsimard | maybe it's racey and different git config steps on each other ? | 20:39 |
| mordred | Shrews, pabelanger, dmsimard: happened again | 20:40 |
| mordred | http://logs.openstack.org/26/513926/2/check/tox-py35-on-zuul/26cb2ad/job-output.txt.gz#_2017-11-15_20_38_00_851739 | 20:40 |
| mordred | I'm going to make a dummy patch to zuul to see if it's just unhappy for some reason | 20:41 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: DNM no-op patch to test the current gate job https://review.openstack.org/520194 | 20:42 |
| Shrews | mordred: only thing "odd" (which isn't really odd) about that change is that the job name and role are identically named | 20:43 |
| Shrews | mordred: i have no idea if that would affect anything or not | 20:44 |
| Shrews | i mean, obviously it shouldn't | 20:44 |
| tobiash | mordred: in the first link two test cases use the same tmp dir | 20:54 |
| tobiash | so there might be something broken in tmp dir setup and tear down | 20:55 |
| tobiash | http://logs.openstack.org/25/513925/2/check/tox-py35-on-zuul/83b3855/job-output.txt.gz#_2017-11-15_20_27_57_764177 | 20:57 |
| tobiash | And 2017-11-15 20:27:57.764822 | 20:58 |
| mordred | tobiash: spectacular | 20:58 |
| tobiash | http://logs.openstack.org/25/513925/2/check/tox-py35-on-zuul/83b3855/job-output.txt.gz#_2017-11-15_20_27_57_764822 | 20:58 |
| tobiash | Two different test cases, same tmp dir | 20:58 |
| tobiash | That looks weird | 20:58 |
| tobiash | But I'm currently on mobile phone and cannot really assist in tracing this | 20:59 |
| mordred | tobiash: well- what's even better is that the no-op test to zuul WORKED | 21:00 |
| mordred | so somehow that patch to zuul-jobs and the tox-py27-on-zuul is doing $something | 21:00 |
| * mordred is about to jump on the phone - will debug more when off | 21:00 | |
| dmsimard | completely unrelated but I was wondering... what we do right now (although it is a workaround to a current problem) in RDO is that in some circumstances we *pull* logs from the Nodepool VM to the logserver (i.e, the logserver has a key that allows connection to the nodepool VMs). Is that something we could consider to skip pulling the logs back to the executors and then the executor pushes to the logserver ? | 21:06 |
| dmsimard | This means the executor delegates a task to the logserver which has the logserver pull the logs | 21:07 |
| dmsimard | I guess there's some files that are bound to be local on the executor (job-output, ara reports, etc.) but maybe it'd avoid some of the disk space issues we've seen | 21:08 |
| dmsimard | I suppose it'd save on bandwidth (pull once instead of pull/push) and save time but maybe there are considerations I'm not seeing. | 21:08 |
| pabelanger | dmsimard: yah, in our case, logs.o.o can't login to nodepool nodes. We could change that, but so after haven't. | 21:10 |
| pabelanger | but agree it would save some bandwidth | 21:10 |
| dmsimard | pabelanger: we could create a new keypair and add it to infra-root-keys in the nodepool config or something. | 21:11 |
| dmsimard | Anyway, just a random thought :) | 21:11 |
| dmsimard | pabelanger: I thought about it when I saw the DISK_FULL code bits :p | 21:12 |
| pabelanger | dmsimard: yah, so far this is still much like jenkins, where we pull all logs back to executor and publish | 21:12 |
| pabelanger | the other issue, would mean we need to run ansible on logs.o.o, from zuul | 21:13 |
| pabelanger | which, might not be something we want to do | 21:13 |
| pabelanger | currently, we just need ssh running on logs.o.o and push to it | 21:13 |
| dmsimard | pabelanger: we're not really running ansible on logs.o.o | 21:14 |
| dmsimard | pabelanger: ansible still runs from the executor, but the logs.o.o server is in the inventory for the purpose of being able to delegate tasks to it | 21:15 |
| dmsimard | pabelanger: but thinking about it now, it would be hard to lock down tasks/roles from just using logs.o.o inappropriately | 21:16 |
| dmsimard | i.e, "rm -rf /" on logs.o.o | 21:16 |
| pabelanger | right | 21:16 |
| dmsimard | hmmm | 21:17 |
| dmsimard | it's probably possible to do already.. | 21:17 |
| pabelanger | i don't think we want to delegate_to: control plane servers, might open ourself to an attack | 21:17 |
| * dmsimard tries something (that doesn't involve rm -rf" | 21:17 | |
| pabelanger | dmsimard: well, we do add_host: logs.o.o today. let me think | 21:18 |
| pabelanger | yah, so we do pull from logs.o.o to executor today. I thought we pushed | 21:19 |
| pabelanger | which, makesense, because we disabled facts | 21:20 |
| SpamapS | hmmmm | 21:27 |
| SpamapS | http://paste.openstack.org/show/626445/ | 21:27 |
| SpamapS | Looping a command showed no output, but that command definitely prints stuff when it fails. | 21:27 |
| dmsimard | SpamapS: probably a bug in the zuul stream callback | 21:28 |
| pabelanger | SpamapS: check job-output.json, it likey is there | 21:31 |
| *** threestrands has joined #zuul | 21:32 | |
| *** threestrands has quit IRC | 21:32 | |
| *** threestrands has joined #zuul | 21:32 | |
| *** hasharAway has quit IRC | 21:34 | |
| pabelanger | I know we have work to do on stream.html, but would be nice not to start streaming until all merge operations are done. As not to get the could not find build message | 21:39 |
| mordred | dmsimard: one of the reasons for the proposal to switch how log collection works is to allow people to make such a decision at their deployment in their base job | 22:23 |
| mordred | dmsimard: like - we're not super comfortable putting keys onto build nodes after code has been run from the internet - but other people may be in a position where they're ok with that ... or also one might want to upload logs to swift and can provide the credentials in such a way that it doesn't involve writing secrets to disk on a remote potentially rooted host | 22:24 |
| mordred | dmsimard: which is all to say, I doubt we'll choose to skip copying back to the executor for openstack, but once we get the log collection changes pushed through (I'm gonna try to get back to working on that tomorrow) - it should empower zuul admins to choose how they want to deal with that | 22:25 |
| dmsimard | mordred: sure, leaving the door open to other options is great. | 22:31 |
| *** rcarrillocruz has quit IRC | 22:34 | |
| *** rcarrillocruz has joined #zuul | 22:41 | |
| mnaser | mordred: I know this isn't the place for me to ask but a "best practice" doc for zuul role writing would be nice | 22:49 |
| mnaser | for example i created this - https://review.openstack.org/#/c/519489/ - based off the python one | 22:49 |
| mnaser | and i see you have the change in the python one to a best practice, so i dont know, probably be a very useful asset for those trying to write up roles | 22:50 |
| mnaser | i.e. when to use zuul_work_dir, role specific var, zuul.project.src_dir vs src/{{ zuul.project.canonical_name }} | 22:50 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Make build-python-release job https://review.openstack.org/513925 | 22:54 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Remove old python-sdist job https://review.openstack.org/513926 | 22:54 |
| mordred | mnaser: agree - I think that's definitely a thing we want | 22:54 |
| mordred | tobiash, Shrews, dmsimard: https://review.openstack.org/520236 Change override-branch to override-checkout ... I believe the issue that was happening earlier is that zuul was checking out the master branch for zuul rather than feature/zuulv3 | 22:55 |
| mordred | we'll see how the job goes now | 22:56 |
| SpamapS | mnaser: I've taken to not using zuul. in roles at all. Makes it easier for me to know what variables to fake when I'm running local tests. | 23:35 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!