| *** mattw4 has quit IRC | 00:19 | |
| *** michael-beaver has quit IRC | 00:35 | |
| *** armstrongs has quit IRC | 01:16 | |
| *** mattw4 has joined #zuul | 01:33 | |
| *** bjackman__ has joined #zuul | 01:37 | |
| *** bjackman__ has quit IRC | 02:17 | |
| *** mattw4 has quit IRC | 02:23 | |
| *** noorul has joined #zuul | 02:35 | |
| noorul | clarkb: hi | 02:35 |
|---|---|---|
| *** noorul has quit IRC | 02:44 | |
| *** noorul has joined #zuul | 02:45 | |
| *** noorul has quit IRC | 03:04 | |
| *** bjackman__ has joined #zuul | 03:06 | |
| *** noorul has joined #zuul | 03:09 | |
| *** bjackman__ has quit IRC | 03:12 | |
| *** noorul has quit IRC | 03:13 | |
| *** bjackman__ has joined #zuul | 03:56 | |
| *** raukadah is now known as chkumar|rover | 04:04 | |
| openstackgerrit | Andreas Jaeger proposed zuul/zuul-jobs master: Handle sphinx PDF files https://review.opendev.org/678077 | 04:06 |
| openstackgerrit | Andreas Jaeger proposed zuul/zuul-jobs master: Handle sphinx PDF files https://review.opendev.org/678077 | 04:39 |
| openstackgerrit | Felix Schmidt proposed zuul/zuul master: Improve visualization retry attempts in UI https://review.opendev.org/678147 | 05:16 |
| *** noorul has joined #zuul | 05:26 | |
| noorul | Is there a way to setup zuul so that I can test the changes quickly live? | 05:27 |
| noorul | Now I am using docker-compose. It takes time to build the container. | 05:28 |
| *** noorul has quit IRC | 05:36 | |
| *** noorul has joined #zuul | 05:42 | |
| *** bjackman__ has quit IRC | 06:03 | |
| *** bjackman has joined #zuul | 06:04 | |
| *** titusjose has joined #zuul | 06:13 | |
| *** saneax has joined #zuul | 06:18 | |
| openstackgerrit | Andreas Jaeger proposed zuul/zuul-jobs master: Handle sphinx PDF files https://review.opendev.org/678077 | 06:30 |
| *** jpena|off is now known as jpena | 06:59 | |
| AJaeger | zuul-maint, could you review the change above, please? I tested it now and it works... | 07:09 |
| *** jangutter has quit IRC | 08:15 | |
| *** jangutter has joined #zuul | 08:33 | |
| openstackgerrit | Monty Taylor proposed zuul/zuul master: Write json log file by seeking and appending https://review.opendev.org/676717 | 08:41 |
| openstackgerrit | Merged zuul/zuul-jobs master: Handle sphinx PDF files https://review.opendev.org/678077 | 08:57 |
| openstackgerrit | Monty Taylor proposed zuul/zuul master: Write errors from ansible execution into json log https://review.opendev.org/676723 | 08:59 |
| *** hashar has joined #zuul | 09:13 | |
| mordred | clarkb, corvus: I went ahead and updated 676717 to remove the incorrect comment if you feel like re-voting on it | 09:18 |
| mordred | clarkb: and if you coudl review https://review.opendev.org/#/c/676516 - I don't see any reason to not land it and 676717 | 09:19 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Optionally allow zoned executors to process unzoned jobs https://review.opendev.org/673840 | 10:52 |
| *** rfolco is now known as rfolco|brb | 11:10 | |
| *** rfolco|brb is now known as rfolco | 11:17 | |
| openstackgerrit | Felix Schmidt proposed zuul/zuul master: Make direct-push configurable on project-level https://review.opendev.org/677109 | 11:21 |
| openstackgerrit | Felix Schmidt proposed zuul/zuul master: Implement push job in merger https://review.opendev.org/677110 | 11:21 |
| openstackgerrit | Felix Schmidt proposed zuul/zuul master: Push changes in GerritReporter if direct-push is enabled https://review.opendev.org/677111 | 11:21 |
| *** bjackman has quit IRC | 11:23 | |
| openstackgerrit | Felix Schmidt proposed zuul/zuul master: Improve visualization retry attempts in UI https://review.opendev.org/678147 | 11:25 |
| *** jpena is now known as jpena|lunch | 11:32 | |
| *** hashar has quit IRC | 11:41 | |
| *** hashar has joined #zuul | 12:07 | |
| fungi | noorul: the fastest way to test changes to zuul is to write unit tests which exercise your changes. that has at least a couple of benefits: 1. you don't need to wait for container builds because tox will just reference the files directly from a virtualenv (which also gets cached so it doesn't have to be rebuilt on subsequent tox invocations, and also you can ask it to only run your specific tests so you don't | 12:14 |
| fungi | have to wait for the full battery to complete), and 2. if you push the tests with your changes then you can prevent those features from regressing in the future | 12:14 |
| fungi | noorul: another option is to install zuul system-wide and just manually reinstall your updated source (or edit the files directly in /usr/lib/python*), but i expect the same is possible in a container image too | 12:16 |
| fungi | but the unit test approach is nice because you can follow a test-driven development model where you write the test first to describe what behavior you want, and then keep re-testing and fixing your implementation until the test passes | 12:17 |
| fungi | you can even (ab)use opendev's zuul deployment to this end by commenting out every job except one of the unit test jobs and then tweak the tox configuration to only run your new tests and push up your changes as work in progress and see what the job result ends up being | 12:19 |
| fungi | especially useful if you're in a situation where you can't easily run the unit tests locally | 12:19 |
| *** rlandy has joined #zuul | 12:20 | |
| *** rlandy is now known as rlandy|ruck | 12:20 | |
| *** hashar_ has joined #zuul | 12:23 | |
| *** hashar_ has quit IRC | 12:44 | |
| *** hashar has quit IRC | 12:44 | |
| *** jpena|lunch is now known as jpena | 12:49 | |
| *** hashar has joined #zuul | 13:00 | |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Annotate QueueItem logger https://review.opendev.org/678226 | 13:30 |
| noorul | fungi: How do I set log level to debug? | 13:49 |
| fungi | noorul: https://opendev.org/opendev/puppet-zuul/src/branch/master/files/logging.conf#L15 | 13:50 |
| fungi | and make sure there's a corresponding [handler_debug] if you want to redirect it to a different file | 13:51 |
| fungi | zuul just uses the python stdlib logging module | 13:51 |
| fungi | so it's a python logging config, not anything zuul-specific | 13:51 |
| noorul | fungi: Where can I set the log_level ? | 13:57 |
| noorul | fungi: Which config file? zuul.conf? | 13:57 |
| fungi | noorul: logging.conf | 13:58 |
| noorul | fungi: Is there an example? | 13:58 |
| fungi | did you see the url i posted? that's how we're setting up debug logging in our (opendev) deployment | 13:58 |
| fungi | not sure what other sort of example you're expecting | 13:59 |
| Shrews | https://zuul-ci.org/docs/zuul/admin/components.html#attr-gearman_server.log_config may be useful | 14:00 |
| Shrews | errr, https://zuul-ci.org/docs/zuul/admin/components.html#attr-scheduler.log_config | 14:00 |
| Shrews | so many configs :) | 14:00 |
| * Shrews hopes fungi is bracing for hurricane season | 14:01 | |
| fungi | Shrews: thanks, i was searching the docs for "logging" and not finding those | 14:02 |
| fungi | and yeah, i need to pull a bunch of plywood out of the shed and get it ready for boarding up windows | 14:03 |
| fungi | i just hope i don't end up having to deal with a hurricane when i'm supposed to be at ansiblefest | 14:03 |
| Shrews | yeah, that would be inconvenient for sure | 14:04 |
| fungi | i guess our docs don't actually spell out that zuul uses python logging | 14:04 |
| fungi | noorul: https://docs.python.org/3.5/library/logging.config.html is the config specification for those files | 14:06 |
| fungi | i'll push up some extra words | 14:06 |
| Shrews | don't they? i thought there was something... | 14:07 |
| Shrews | fungi: ah, yeah, something for nodepool logging (https://zuul-ci.org/docs/nodepool/installation.html#configuration) but not zuul i guess | 14:08 |
| fungi | even better, words i can just copy! | 14:08 |
| Shrews | \o/ | 14:09 |
| fungi | does zuul also support dictconfig format? | 14:09 |
| Shrews | that i do not know | 14:09 |
| fungi | zuul/ansible/logconfig.py: logging.config.dictConfig(self._config) | 14:10 |
| fungi | git grep says yes | 14:10 |
| noorul | fungi: I followed the Zuul from scratch guide | 14:11 |
| noorul | fungi: Do I need to start gearman separately | 14:11 |
| fungi | zuul from scratch is mostly geared toward the minimum necessary to get zuul running, so is unlikely to incorporate a lot of information on optional configuration | 14:11 |
| fungi | gearman should be started automatically in the scheduler container as a child process of the scheduler daemon if you configure zuul to provide a gearman server | 14:12 |
| noorul | fungi: I get the following exception Exception: Cat job <gear.Job 0x7fc7e2669780 handle: b'H:::ffff:127.0.0.1:1' name: merger:cat unique: cb912586b66c4a0eb0444e56bbc7d35e> failed | 14:13 |
| clarkb | mordred: thanks approved the failed/skipped change and +2'd the json editing | 14:13 |
| clarkb | anyone else want to review 676717? I think that may help with memory on executors | 14:14 |
| fungi | noorul: i think that means a gearman server is running, but the merger process which attempted to handle that particular concatenation failed. you probably need to look in your merger/executor logs for 0x7fc7e2669780 | 14:15 |
| fungi | the gearman protocol itself doesn't provide a lot of facilities for communicating detailed failure information, so you have to go hunting for an actual error message at the client end | 14:16 |
| *** chkumar|rover is now known as raukadah | 14:24 | |
| noorul | fungi: executor is not starting | 14:28 |
| fungi | that might be the problem then | 14:28 |
| openstackgerrit | Jeremy Stanley proposed zuul/zuul master: Document logging configuration schema https://review.opendev.org/678243 | 14:30 |
| noorul | fungi: fixed the executor | 14:33 |
| noorul | fungi: the web is returning 404 when I hit port 9000 | 14:33 |
| fungi | noorul: that's expected, we've had some requests to add an index page to the api but there isn't one yet | 14:34 |
| fungi | actual api calls should not return 404, however | 14:34 |
| noorul | fungi: How do I get to the status page? | 14:34 |
| openstackgerrit | Merged zuul/nodepool master: Add a dib-cmd option for diskimages https://review.opendev.org/672196 | 14:35 |
| fungi | oh, nevermind, 9000 is not the api socket it's the base zuul-web socket i guess | 14:35 |
| fungi | try /tenants | 14:36 |
| fungi | our deployment redirects from / to /tenants automatically | 14:36 |
| fungi | however it's all rendered via javascript these days, so you need to make sure that /static/js/ is being served | 14:37 |
| fungi | also /static/css/, probably basically anything in /static | 14:39 |
| clarkb | tristanC: fyi not sure if 672196 affects your non dib building | 14:39 |
| tristanC | clarkb: thanks for the headsup, this will let us specify the custom disk-image-create script explicitely | 14:42 |
| tobiash | \o/ Now I can remove my PATH env var hack :) | 14:42 |
| fungi | noorul: on our deployment (opendev) if i fetch http://127.0.0.1:9000/tenants/ from the command-line on the scheduler i see it linking to css and js files which i can also successfully fetch, so that's probably what you want to check | 14:42 |
| fungi | noorul: i guess it also does the same without /tenants on ours | 14:43 |
| tristanC | clarkb: tobiash: there is a risk the cli interface change, perhaps we should document and support a stable disk-image-create usage argparse? | 14:43 |
| tobiash | tristanC: that fact is at least documented in that patch | 14:44 |
| fungi | noorul: yeah, looks like the redirect from / to /tenants/ must be encoded in the javascript, so checking the base url should work too | 14:45 |
| tobiash | no idea though if it is possible or desired to keep the disk-image-create cli format stable (although I never saw any change that would impact how nodepool calls dib) | 14:45 |
| clarkb | tristanC: you mean parse dibs args in nodepool? | 14:47 |
| clarkb | I dont think we should do that | 14:48 |
| tristanC | clarkb: i mean document how nodepool use the disk-image-create script and add to release note upgrade if that change | 14:48 |
| tobiash | tristanC: If this ever happend we definitely need to add a release note | 14:50 |
| clarkb | on this particular case a test only flag was removed so I think that is fine. Documenting the values present in the string ibterpolation would be good | 14:50 |
| clarkb | (and maybe we can extend it further to make the whole command overrideable) | 14:51 |
| tobiash | right, that seems useful (I did several tries to parse the args correctly on my first wrapper) ;) | 14:51 |
| tristanC | our wrapper is actually just converting the cli arg to ansible vars: https://softwarefactory-project.io/cgit/config/tree/nodepool/elements/virt-customize/disk-image-create | 14:52 |
| tristanC | thus perhaps nodepool could support an ansible playbook to build image, and set the vars directly, e.g.: image_output=/var/... image_formats=['qcow2', ...] | 14:53 |
| *** mattw4 has joined #zuul | 14:54 | |
| clarkb | tristanC: I dont think it should call ansible directly but instead make more than the binary path executable configurable | 14:54 |
| clarkb | then you can configure ansoble yourself with the correct parameters | 14:54 |
| tristanC | clarkb: that works for me too | 14:55 |
| clarkb | and people can use packer or whatever if they want | 14:55 |
| clarkb | as long as the files end up on disk with the correct nameit should work | 14:55 |
| *** sgw has joined #zuul | 14:56 | |
| noorul | fungi: not found | 15:06 |
| noorul | fungi: I think I am missing something | 15:07 |
| noorul | fungi: But I remember following exact steps | 15:07 |
| fungi | noorul: i don't know what to suggest off the top of my head, and am waaay behind schedule on things i need to get done today, but hopefully someone else in here has suggestions | 15:09 |
| noorul | It will be great if someone can help | 15:09 |
| noorul | I followed this https://zuul-ci.org/docs/zuul/admin/zuul-from-scratch.html | 15:09 |
| clarkb | did you look at zuul-web logs? | 15:11 |
| noorul | clarkb: Yes, http://paste.openstack.org/show/762690/ | 15:12 |
| openstackgerrit | Merged zuul/zuul master: Set failed, unreachable, skipped statuses in json plugin https://review.opendev.org/676516 | 15:12 |
| *** hashar has quit IRC | 15:15 | |
| clarkb | I'm betting sudo pip3 install . does not build the js ssets | 15:16 |
| clarkb | assuming you ran that command to install zuul | 15:16 |
| noorul | clarkb: Yes, you are right | 15:17 |
| clarkb | https://zuul-ci.org/docs/zuul/developer/javascript.html#deploying maybe that helps? | 15:17 |
| noorul | clarkb: but /tenants is also returning 404 | 15:18 |
| clarkb | noorul: yes, those urls serve javascript which render that info in your browser | 15:21 |
| clarkb | there are separate api urls if you just want the json responses | 15:21 |
| clarkb | try /api/tenants and /api/info | 15:22 |
| fungi | yeah, i can confirm, `wget -qO- http://127.0.0.1:9000/api/info` works on our scheduler and returns: {"info": {"websocket_url": null, "stats": {"prefix": null, "type": "graphite", "url": null}, "capabilities": {"job_history": false}}} | 15:23 |
| clarkb | our server does have job history, odd that that is false? | 15:24 |
| openstackgerrit | David Shrewsbury proposed zuul/zuul master: Mark nodes as USED when deleting autohold https://review.opendev.org/664060 | 15:25 |
| noorul | clarkb: That is giving me response | 15:25 |
| noorul | clarkb: So I need to run web on another port | 15:27 |
| clarkb | no | 15:27 |
| clarkb | you need to build the javascript assets as described at https://zuul-ci.org/docs/zuul/developer/javascript.html#deploying | 15:28 |
| noorul | I think after installing yarn | 15:29 |
| noorul | We just have to run pip3 install . again | 15:29 |
| noorul | correct? | 15:29 |
| clarkb | you need to run yarn build first | 15:31 |
| clarkb | then run pip3 install . again | 15:31 |
| pabelanger | if zuul can see yarn, it should buid it | 15:32 |
| pabelanger | looking at https://opendev.org/zuul/zuul/src/branch/master/zuul/_setup_hook.py | 15:32 |
| *** jpena is now known as jpena|off | 15:32 | |
| noorul | ok | 15:32 |
| pabelanger | build* | 15:32 |
| clarkb | oh cool | 15:33 |
| clarkb | I think we have a few docs updates to make around that in the from scratch and javascript pages then | 15:33 |
| clarkb | from scratch should include a install yarn step prior to pip install | 15:33 |
| clarkb | and javascript page can mention that if yarn is present it will b called by pip | 15:33 |
| clarkb | noorul: is that something you might be interested in pushing if your testing confirms that is the fix? | 15:34 |
| noorul | clarkb: Need to get approval from management | 15:34 |
| noorul | clarkb: Right now in the middle of hackathon | 15:34 |
| noorul | clarkb: Trying to make zuul work with stash and demo it | 15:35 |
| noorul | ran yarn build and then pip3 install . | 15:38 |
| noorul | restarted zuul-web | 15:38 |
| noorul | no luck | 15:38 |
| clarkb | did yarn build succeed? | 15:38 |
| noorul | Yes | 15:39 |
| *** mattw4 has quit IRC | 15:39 | |
| noorul | It says build folder is ready to be deployed | 15:39 |
| clarkb | you might need to pip install -U . to get pip to "upgrade" it | 15:39 |
| clarkb | it might see that it hasn't changed from its perspective and not update it? | 15:39 |
| noorul | Trying that | 15:40 |
| noorul | Super | 15:41 |
| noorul | it worked | 15:41 |
| pabelanger | yay | 15:41 |
| clarkb | ya so thats a pip thing since the git repo didn't change, the version doesn't change and pip says "I'm done" | 15:41 |
| *** jamesmcarthur has joined #zuul | 15:47 | |
| noorul | Shrews: Is there an example /etc/zuul/scheduler-logging.yaml | 15:57 |
| Shrews | noorul: i believe fungi gave you a link for an example | 15:58 |
| noorul | That is .conf | 15:58 |
| noorul | https://opendev.org/opendev/puppet-zuul/src/branch/master/files/logging.conf#L15 | 15:58 |
| noorul | In this document https://zuul-ci.org/docs/zuul/admin/components.html#attr-scheduler.log_config | 15:58 |
| noorul | It mentions yaml | 15:58 |
| fungi | noorul: see https://review.opendev.org/678243 | 15:59 |
| fungi | i pushed that to address our earlier discussion | 15:59 |
| *** kmalloc is now known as needscoffee | 16:00 | |
| *** needscoffee is now known as needsSoMuchMoreC | 16:00 | |
| Shrews | noorul: the log_config parameter is contained in the main zuul.yaml file (and can appear in various sections of that file for each service). That parameter should point to the logging .conf file | 16:00 |
| *** needsSoMuchMoreC is now known as needscoffee | 16:00 | |
| Shrews | i don't know what scheduler-logging.yaml is | 16:01 |
| Shrews | unless it's the yaml version of the logging configuration | 16:01 |
| Shrews | we don't use that style afaik | 16:01 |
| zbr | can someone correct me about: at this moment zuul requires python2 on all nodes, right? Can we make it make it smarter like ansible2.8 where python3 interpreter is also detected and used? | 16:02 |
| *** jangutter has quit IRC | 16:03 | |
| fungi | pretty sure even without ansible 2.8 jobs (including the base job) can set the python path | 16:04 |
| zbr | what interest me is: to be able to use nodepool images which have only python3 on them, without any "python" or "python2" executable, and be able to use these without having to configure ansible_python_interpreter. | 16:05 |
| fungi | we do that currently for jobs where we don't want ansible to run python2.7 | 16:05 |
| clarkb | fungi: I think opendev is still all python2 | 16:06 |
| clarkb | but zuul does respect the python version reported by nodepool iirc | 16:06 |
| fungi | well, if ansible has a feature where it is able to guess what python to use without being told, then that would likely be independent of zuul | 16:06 |
| zbr | fungi: that is exactly what I want: to avoid using py27, in fact I want to remove it from some images we have on rdo. | 16:06 |
| clarkb | and jobs may be able to further override from there | 16:06 |
| zbr | fungi: ansible guess very well but only starting with 2.8 :( | 16:06 |
| zbr | this means that once default is 2.8, the override is no longer needed. | 16:07 |
| zbr | but there is no other requirement from zuul itself for also having a py27 installed. | 16:07 |
| zbr | that was my guess too but someone questioned that and needed to double check. | 16:08 |
| zbr | thanks! | 16:08 |
| clarkb | zbr https://zuul-ci.org/docs/nodepool/configuration.html#attr-diskimages.python-path | 16:11 |
| zbr | clarkb: i guess that ideally this should be a list. | 16:12 |
| clarkb | no I think it is to be used to pick a speciric version | 16:13 |
| clarkb | since you are building the image you know ehat python should be used | 16:14 |
| zbr | yep, can be controlled by that no big deal. still that default is quickly becoming outdated. | 16:15 |
| zbr | but i know that changing a default could alienate people | 16:15 |
| clarkb | we tried to default to 3 but ansible doesnt reliably work with 3 in all cases | 16:15 |
| clarkb | the default is largely there to ensure ansible functions aiui then people can opt into 3 when they know ansible will work with it | 16:16 |
| zbr | clarkb: when it was that? or more exactly, which version of ansible. 26-27 work decently well with py3, only bit missing I know about is auto-detection of python3 executable. | 16:17 |
| clarkb | its the roles and libraries aiui not ansible itself | 16:17 |
| clarkb | though ansible itself had problems in tye past too | 16:17 |
| zbr | but if some people are using some less-mainstream modules which where not ported... that is another issue. | 16:17 |
| clarkb | ansivle core should work now | 16:17 |
| clarkb | its everything else that may not | 16:18 |
| zbr | sure, i guess we can make try to change default in january. | 16:18 |
| noorul | Shrews: Maybe doc is out dated | 16:25 |
| fungi | noorul: not necessarily outdated, see https://review.opendev.org/678243 which will explain the difference | 16:30 |
| *** mattw4 has joined #zuul | 16:31 | |
| *** AJaeger_ has joined #zuul | 16:32 | |
| *** AJaeger has quit IRC | 16:35 | |
| *** AJaeger_ is now known as AJaeger | 16:38 | |
| noorul | fungi: cool, thanks! | 16:39 |
| noorul | fungi: something new | 16:40 |
| fungi | not really new either, merely undocumented | 16:42 |
| fungi | which is why i pushed that documentation change | 16:43 |
| *** saneax has quit IRC | 16:43 | |
| *** saneax has joined #zuul | 16:45 | |
| *** AJaeger_ has joined #zuul | 16:47 | |
| *** AJaeger has quit IRC | 16:51 | |
| *** saneax has quit IRC | 16:56 | |
| openstackgerrit | Radosław Piliszek proposed zuul/zuul master: Make files matcher match changes with no files https://review.opendev.org/678273 | 16:59 |
| yoctozepto | fungi: ^ | 17:01 |
| *** saneax has joined #zuul | 17:01 | |
| fungi | neat | 17:01 |
| yoctozepto | at least it does not suffer from the problems of the predecessor | 17:01 |
| *** igordc has joined #zuul | 17:09 | |
| *** jamesmcarthur has quit IRC | 17:11 | |
| *** jamesmcarthur has joined #zuul | 17:11 | |
| *** saneax has quit IRC | 17:13 | |
| *** saneax has joined #zuul | 17:15 | |
| *** jamesmcarthur has quit IRC | 17:22 | |
| *** saneax has quit IRC | 17:31 | |
| *** michael-beaver has joined #zuul | 17:32 | |
| pabelanger | so, it took a few hours, but we now have a propose-updates job for ansible-network, for helping automate some tasks that require pull requests to be created: https://logs.zuul.ansible.com/ff/ff38d08253a1c797455cb18b599fc5b29c103afd/post/propose-network-content-collector-vyos-vyos/6e5beb9/job-output.html#l6064 | 17:38 |
| pabelanger | not as nice at gerrit / git-review, but seems to be working | 17:38 |
| pabelanger | we wrote a collections migration script to move network bits from ansible/ansible into new platform collections | 17:39 |
| *** dkehn has joined #zuul | 17:58 | |
| *** AJaeger_ is now known as AJaeger | 18:14 | |
| openstackgerrit | Clark Boylan proposed zuul/zuul master: Stop executor if log streamer dies https://review.opendev.org/678049 | 18:24 |
| clarkb | tobiash: ^ is that better? | 18:24 |
| noorul | clarkb: I am getting exception http://paste.openstack.org/show/762695/ | 18:36 |
| noorul | I am not sure why it is looking for that file | 18:38 |
| *** armstrongs has joined #zuul | 18:39 | |
| clarkb | noorul: ssh is how zuul connects to test nodes, it needs an ssh key to do that | 18:40 |
| titusjose | I am attempting to write a nodepool driver for GCE. Although I was able to follow the existing nodepool driver implementations and figured out that nodepool is using ZK to store the metadata about the resource created on the provider. I am not able to figure out how zuul would be using the metadata(hostname/ip , ssh key etc) about a node with Ansible. | 18:40 |
| titusjose | Can someone point out on how this plumbing of ansible figuring out the node hostname/ip ssh keys etc for the node returned from nodepool is done? | 18:41 |
| noorul | clarkb: I thought it will be looking at zuul users home folder /var/lib/zuul/.ssh | 18:41 |
| clarkb | titusjose: zuul submits node requests to nodepool via zookeeper. Nodepool reads those and attempts to fulfill them. If it fulfills them it sends a response to zuul on the node request with those details | 18:41 |
| titusjose | And how does zuul expose those details it got from nodepool in to ansible? | 18:42 |
| clarkb | noorul: the docker compose stuff uses /var/ssh | 18:43 |
| clarkb | titusjose: zuul writes an ansible inventory with the information it received from zookeeper from nodepool | 18:43 |
| clarkb | titusjose: https://opendev.org/zuul/zuul/src/branch/master/zuul/executor/server.py#L654-L686 there | 18:44 |
| clarkb | titusjose: you shouldn't need to worry about that if the GCE driver in nodepool adheres to the existing interfaces | 18:44 |
| clarkb | (all of that should just work) | 18:44 |
| noorul | clarkb: I am running using the scratch doc | 18:45 |
| clarkb | noorul: I'm not sure why it chooses that dir but that is the dir it seems to use | 18:45 |
| clarkb | noorul: what is your configs [executor] private_key_file value set to? | 18:46 |
| openstackgerrit | Clark Boylan proposed zuul/zuul master: Document js tool installation in scratch doc https://review.opendev.org/678286 | 18:49 |
| clarkb | noorul: ^ patch to address the problem you ran into earlier | 18:49 |
| *** armstrongs has quit IRC | 18:51 | |
| openstackgerrit | Clark Boylan proposed zuul/zuul master: Stop executor if log streamer dies https://review.opendev.org/678049 | 18:53 |
| noorul | clarkb: private_key_file=/var/lib/zuul/.ssh/nodepool_rsa | 18:55 |
| noorul | clarkb: The tenant configuration is stored in /etc/zuul/main.yaml | 18:56 |
| clarkb | oh sorry this is in the merger process | 18:56 |
| noorul | clarkb: Does that mean, if I have to add a new tenant, I need to restart scheduler? | 18:57 |
| clarkb | no | 18:57 |
| clarkb | the merger process is going to use the key associated with the gerrit/github/bitbucket connection | 18:57 |
| clarkb | you need to check the sshkey value of that connection | 18:57 |
| clarkb | (I thought this was the key to ssh to the nodes but it is the key to connect to the git server instead) | 18:58 |
| clarkb | tenant configuration lives in the yaml files which are loaded when triggered to do so | 18:58 |
| clarkb | if you hcange zuul.conf then you hvae to restart | 18:58 |
| noorul | clarkb: https://zuul-ci.org/docs/zuul/admin/zuul-from-scratch.html | 19:01 |
| noorul | clarkb: tenant_config points /etc/zuul/main.yaml | 19:01 |
| noorul | clarkb: I cant see merger section there | 19:02 |
| clarkb | noorul: in the zuul.conf file | 19:02 |
| clarkb | noorul: what connections do you have defined in that file? | 19:03 |
| noorul | http://paste.openstack.org/show/762848/ | 19:04 |
| clarkb | noorul: it come sfrom line 24 of that file | 19:04 |
| noorul | Oh the bitbucket has /var/ssh/zuul reference | 19:04 |
| noorul | clarkb: Regarding tenant config, how can I load it dynamically | 19:06 |
| fungi | `zuul --help` should give you command-line help on the various rpc subcommands. one of them is for reconfiguring | 19:07 |
| fungi | https://zuul-ci.org/docs/zuul/admin/client.html | 19:08 |
| clarkb | also sighup against the scheduler works (though the utility command is preferred) | 19:08 |
| fungi | er, for some reason i thought there was a reconfig | 19:08 |
| clarkb | fungi: I thought there was too | 19:08 |
| fungi | i'm not seeing it in the client doc though | 19:09 |
| clarkb | I think it is zuul-scheduler reconfigure | 19:09 |
| clarkb | not zuul reconfigure | 19:09 |
| pabelanger | zuul-scheduler full-reconfigure | 19:09 |
| noorul | So this can be automated using a git repo | 19:10 |
| clarkb | pabelanger: thanks | 19:10 |
| pabelanger | https://zuul-ci.org/docs/zuul/admin/components.html#operation | 19:10 |
| pabelanger | noorul: yup, gitops should be fine | 19:10 |
| fungi | aha, i was looking for the wrong cli | 19:13 |
| *** jamesmcarthur has joined #zuul | 19:28 | |
| SpamapS | With nodepool+aws+cloudimages we tried to bump our min-ready from 0 to 3 and got this | 19:30 |
| SpamapS | http://paste.openstack.org/show/762851/ | 19:30 |
| SpamapS | This is on 3.8.0, but we saw it on 3.6.something too | 19:30 |
| SpamapS | Just looks like an assumption. AWS has no diskimage ability, thus no attribute. | 19:31 |
| SpamapS | Shrews:^ ? | 19:32 |
| *** jamesmcarthur has quit IRC | 19:32 | |
| *** jamesmcarthur has joined #zuul | 19:32 | |
| clarkb | SpamapS: ya it is checking that the image for a label is present. But should only do that check against dib if building images for that label | 19:33 |
| clarkb | there is a hsort circuit in there though | 19:34 |
| clarkb | SpamapS: https://opendev.org/zuul/nodepool/src/branch/master/nodepool/launcher.py#L1021-L1023 are you missing that config flag? | 19:34 |
| clarkb | oh I see I think it is a bug in the aws driver | 19:35 |
| clarkb | SpamapS: https://opendev.org/zuul/nodepool/src/branch/master/nodepool/driver/aws/config.py#L168-L170 | 19:35 |
| clarkb | I don't know enough of aws' image management to say if we should always se tthat value to false or not | 19:37 |
| SpamapS | clarkb:well there you go | 19:37 |
| SpamapS | There's no image mgmt | 19:37 |
| SpamapS | Some day maybe. ;) | 19:37 |
| SpamapS | clarkb:do you want to do the patch? :) | 19:37 |
| clarkb | ya one sec | 19:38 |
| openstackgerrit | Clark Boylan proposed zuul/nodepool master: Set manage_images to false on aws https://review.opendev.org/678292 | 19:39 |
| SpamapS | :-D | 19:39 |
| fungi | was aws supposed to have image management? | 19:40 |
| SpamapS | I wrote some follow-up code that went nowhere, but it landed originally w/o img mgmt | 19:40 |
| SpamapS | it's possible a git add -p went awry, and then reviewers missed it | 19:41 |
| *** jamesmcarthur has quit IRC | 19:43 | |
| *** jamesmcarthur has joined #zuul | 19:45 | |
| noorul | Is there a documentation for running log server? | 19:47 |
| clarkb | noorul: the quickstart docs work through it with the docker compose hosted logs server | 19:49 |
| *** jamesmcarthur has quit IRC | 19:50 | |
| fungi | zuul itself doesn't provide a log server, but yeah there is an example of one built from conventional parts in the quickstart | 19:50 |
| fungi | you can also point zuul at a swift object store (or a compatibly-configured ceph deployment apparently) | 19:50 |
| fungi | i wouldn't be surprised if someone comes up with other drivers for, e.g., amazon s3 | 19:51 |
| noorul | clarkb: I am using the scratch doc | 19:53 |
| noorul | clarkb: I am not using docker now | 19:53 |
| clarkb | I know | 19:53 |
| clarkb | I'm saying that that document goes through it, I'm not aware of another document | 19:53 |
| fungi | zuul-from-scratch is a walkthrough for setting up zuul, and zuul doesn't itself provide any file server to host logs | 19:54 |
| noorul | fungi: Got it | 19:54 |
| fungi | but it can take advantage of one if you provide it and configure jobs to archive to it | 19:54 |
| fungi | zuul-quickstart has a container which provides a log server (with i think sshd and apache?) for the example jobs to archive their build logs on | 19:55 |
| fungi | zuul-quickstart also provides a contanier with a gerrit server to be able to trigger events and accept result reporting, but that's technically not a component of zuul either | 19:56 |
| AJaeger | Zuul experts, I see on https://review.opendev.org/#/c/678248/ "Unable to freeze job graph: 'NoneType' object has no attribute 'decrypt'" after the change is merged - and the promote job is not run. Any ideas? | 20:05 |
| clarkb | secrets have the decrypt attribute so that tells me that a secret isn't being set | 20:06 |
| AJaeger | that would be promote-stx-tox-docs or promote-stx-api-ref - I'M checking | 20:07 |
| AJaeger | found it - wrong secret name | 20:08 |
| *** weshay is now known as weshay_MOD | 20:10 | |
| AJaeger | thanks, clarkb for hint | 20:10 |
| *** jamesmcarthur has joined #zuul | 20:14 | |
| *** jamesmcarthur has quit IRC | 20:18 | |
| *** jamesmcarthur has joined #zuul | 20:19 | |
| *** igordc has quit IRC | 20:37 | |
| titusjose | clarkb: For AWS driver I see providers.[aws].pools.labels.key-name as the keypair that would be used to create an instance. What is the assumption that exists here that Ansible should know to pick up the corresponding private key of the keypair? | 21:08 |
| clarkb | titusjose: I think it is the other way around. Nodepool is expected to provide the public side of the key that zuul has | 21:09 |
| *** michael-beaver has quit IRC | 21:10 | |
| clarkb | titusjose: for opendev the way we set it up is to have our image build process bake the public key into authorized_keys for zuul. Then we use the key-name metadata based stuff to add our administrator keys | 21:10 |
| clarkb | (but you could use the more dynamic system to provide keys for the zuul user too) | 21:10 |
| clarkb | poking at docs that may be something we need to be more explicit about | 21:14 |
| clarkb | I'm not sure where best to expand docs for that https://zuul-ci.org/docs/zuul/admin/components.html#attr-executor.private_key_file seems like the correct reference point but likely to be lost if wanting to know the bigger picture | 21:19 |
| *** needscoffee is now known as kmalloc | 21:32 | |
| noorul | clarkb: I am getting the exception http://paste.openstack.org/show/763007/ | 21:36 |
| noorul | clarkb: in executor log | 21:36 |
| clarkb | noorul: you need to set https://zuul-ci.org/docs/zuul/admin/components.html#attr-merger.git_user_email and https://zuul-ci.org/docs/zuul/admin/components.html#attr-merger.git_user_name in zuul.conf | 21:38 |
| *** igordc has joined #zuul | 21:45 | |
| *** jamesmcarthur has quit IRC | 21:46 | |
| *** jamesmcarthur has joined #zuul | 21:48 | |
| *** jamesmcarthur has quit IRC | 21:50 | |
| *** jamesmcarthur has joined #zuul | 21:51 | |
| openstackgerrit | Clark Boylan proposed zuul/zuul master: Set git user config in from scratch document https://review.opendev.org/678312 | 21:54 |
| clarkb | noorul: ^ is a docs update to reflect that | 21:54 |
| *** jamesmcarthur has quit IRC | 22:07 | |
| *** jamesmcarthur has joined #zuul | 22:07 | |
| *** a5hik has joined #zuul | 22:12 | |
| *** jamesmcarthur has quit IRC | 22:13 | |
| *** rlandy|ruck has quit IRC | 22:21 | |
| *** jamesmcarthur has joined #zuul | 22:31 | |
| *** jamesmcarthur has quit IRC | 22:37 | |
| *** jamesmcarthur has joined #zuul | 22:39 | |
| *** jamesmcarthur has quit IRC | 22:46 | |
| *** jamesmcarthur has joined #zuul | 22:50 | |
| *** igordc has quit IRC | 23:00 | |
| *** jamesmcarthur has quit IRC | 23:01 | |
| *** jamesmcarthur has joined #zuul | 23:15 | |
| *** jamesmcarthur has quit IRC | 23:17 | |
| pabelanger | Hmm | 23:19 |
| pabelanger | anybody seem this before | 23:19 |
| *** jamesmcarthur has joined #zuul | 23:19 | |
| pabelanger | http://paste.openstack.org/show/763008/ | 23:19 |
| pabelanger | refs/remotes/origin/HEAD will become dangling! | 23:19 |
| clarkb | pabelanger: I think that means origin deleted what was HEAD? | 23:20 |
| clarkb | did someone switch from master to devel and delete master? | 23:20 |
| clarkb | something like that | 23:20 |
| clarkb | if you delete the repo and force zuul to reclone I would expect that to fix it | 23:21 |
| pabelanger | Oh | 23:21 |
| clarkb | (you could also do surgery and update to match the new HEAD | 23:21 |
| pabelanger | I did delete some branches | 23:21 |
| pabelanger | but not master | 23:21 |
| clarkb | well whatever head is | 23:21 |
| clarkb | (might not be master) | 23:21 |
| pabelanger | okay, that might be it | 23:22 |
| pabelanger | I deleted a bunch of branches in github | 23:22 |
| *** jamesmcarthur has quit IRC | 23:22 | |
| pabelanger | that must have messed up mergers | 23:22 |
| *** jamesmcarthur has joined #zuul | 23:22 | |
| clarkb | you can check what head is by catting it | 23:25 |
| clarkb | then cross check against your deletions | 23:25 |
| clarkb | but I think the fix is likely to have zuul reclone | 23:25 |
| pabelanger | yah, I forced zuul to reclone | 23:26 |
| pabelanger | should try and figure out a nicer way to handle that | 23:27 |
| pabelanger | I completely forgot deleting branches affected zuul | 23:27 |
| clarkb | well I think it is only a problem if HEAD is gone | 23:27 |
| clarkb | it should generally be fine | 23:27 |
| pabelanger | HEAD should be master in this case | 23:27 |
| clarkb | but changes to HEAD don't propogate | 23:27 |
| clarkb | pabelanger: I understand refs/remotes/origin/HEAD will become danglin as a result of git remote prune means "if I delete the branches you've asked me to delete then refs/remotes/origin/HEAD will pointto a branch that does not exist anymore" | 23:28 |
| pabelanger | yah, I'm just surprised that HEAD was not master | 23:29 |
| pabelanger | I need to go back and look | 23:30 |
| pabelanger | in this case, it was just a random branch that developer created to create a PR from | 23:30 |
| *** jamesmcarthur has quit IRC | 23:32 | |
| *** mattw4 has quit IRC | 23:37 | |
| *** jamesmcarthur has joined #zuul | 23:38 | |
| *** jamesmcarthur has quit IRC | 23:42 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!