| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul-jobs master: add-build-sshkey: remove previously authorized build-sshkey https://review.openstack.org/632620 | 00:56 |
|---|---|---|
| SpamapS | another option would be to add a `dequeue-trigger:` to pipelines. | 00:57 |
| SpamapS | And then make sure there are sufficient actions to just catch what gets sent whenever a label is removed. | 00:58 |
| SpamapS | or maybe cancel-trigger would be more clear | 00:58 |
| openstackgerrit | zhouxinyong proposed openstack/pbrx master: Fix author-email in setup.cfg https://review.openstack.org/632847 | 02:03 |
| *** bhavikdbavishi has joined #zuul | 02:09 | |
| *** bhavikdbavishi has quit IRC | 02:37 | |
| *** bjackman has quit IRC | 02:54 | |
| *** dkehn has quit IRC | 03:11 | |
| *** bhavikdbavishi has joined #zuul | 03:27 | |
| *** bhavikdbavishi has quit IRC | 03:31 | |
| *** bhavikdbavishi has joined #zuul | 03:35 | |
| *** rlandy|bbl is now known as rlandy | 03:53 | |
| *** spsurya has joined #zuul | 04:33 | |
| *** bjackman has joined #zuul | 04:58 | |
| SpamapS | whoa, this is weird | 05:09 |
| SpamapS | I pulled in someone else's branch into my fork, submitted a new PR for it, and Zuul got confused and is running with change set to the original | 05:10 |
| SpamapS | oh.. and circleci got confused too.... | 05:11 |
| * SpamapS is so stealing the promote pipeline | 05:21 | |
| *** quiquell|off is now known as quiquell | 06:01 | |
| quiquell | SpamapS: good morning this is good to merge https://review.openstack.org/#/c/623294 | 06:13 |
| quiquell | Testing patches are in the comments | 06:13 |
| quiquell | Maybe you can help | 06:13 |
| *** badboy has joined #zuul | 06:22 | |
| *** saneax has joined #zuul | 06:22 | |
| quiquell | AJaeger: multinode job working at test reviews https://review.openstack.org/#/c/623294 | 06:28 |
| quiquell | Is good to merge | 06:28 |
| quiquell | tristanC: ^ | 06:40 |
| quiquell | sshnaidm|afk: did you have a testing tenant? | 06:42 |
| quiquell | Openatack tenant I mean | 06:42 |
| sshnaidm|afk | quiquell, yep | 06:43 |
| quiquell | Want to try to start to experiment with the repro ci there | 06:43 |
| badboy | hi guys | 06:57 |
| badboy | Any ideas on hot to debug Zuul's log: | 06:57 |
| badboy | WARNING zuul.Scheduler: Tenant example-tenant isn't loaded | 06:57 |
| badboy | the tenant_config is correct in /etc/zuul/main.yaml | 06:58 |
| badboy | :q | 06:58 |
| badboy | ups ;) | 06:59 |
| quiquell | badboy: It takes time to startup | 07:01 |
| quiquell | badboy: in case the message is at start up | 07:02 |
| quiquell | badboy: Tenant is loaded after you see it parses the config in the logs | 07:02 |
| quiquell | badboy: but takes time | 07:02 |
| badboy | quiquell: it's not on startup, it's everytime I open Zuul's webpage | 07:03 |
| badboy | the dashboard | 07:03 |
| quiquell | badboy: Well it means something is not good not at the main.yaml but at the project zuul config | 07:03 |
| quiquell | badboy: you can check in the logs | 07:03 |
| badboy | quiquell: can it be that gerrit is unavailable? | 07:04 |
| badboy | quiquell: because I know it's down for maintenance | 07:04 |
| quiquell | quiquell: yep | 07:04 |
| quiquell | quiquell: that would be very verbose at logs too | 07:04 |
| badboy | 2019-01-23 23:04:31,532 DEBUG zuul.RPCListener: Received job zuul:status_get 2019-01-23 23:04:31,532 WARNING zuul.Scheduler: Tenant example-tenant isn't loaded 2019-01-23 23:04:31,573 DEBUG zuul.RPCListener: Received job zuul:config_errors_list | 07:05 |
| badboy | sorry for spamming | 07:05 |
| quiquell | badboy: Have you see anything regarding gerrit connections ? | 07:09 |
| quiquell | badboy: At startup you see how it get of the refspecs and the like | 07:09 |
| quiquell | badboy: If something with the key or connection is no good you see it there in the scheduler | 07:09 |
| badboy | quiquell: the gerrit I'm trying to connect is currently offline | 07:10 |
| quiquell | badboy: maybe there is a mirror or the like, so you can change it | 07:11 |
| quiquell | badboy: but without mirror there is nothing to do I think | 07:11 |
| badboy | quiquell: it's my internal dev gerrit | 07:11 |
| badboy | quiquell: as the is a proof of concept I wouldn't want to connect to a production gerrit | 07:12 |
| badboy | quiquell: so I guess I have to wait for it to become online | 07:12 |
| quiquell | badboy: Do you have a lot of projects if not maybe you can mimic your internal gerrit at localhost and connect to it | 07:13 |
| quiquell | badboy: with empty projects | 07:13 |
| quiquell | badboy: or importing the projects if they are not huge | 07:13 |
| badboy | quiquell: that's another option but I would have to reconfigure it so that is resembles the dev gerrit and I'm not the one who configured it | 07:14 |
| badboy | quiquell: but thx for the hint | 07:14 |
| quiquell | quiquell: well you can also change your /etc/resolv.conf to point to your local gerrit :-) | 07:15 |
| quiquell | badboy: take a look at zuul quickstart it's good to test stuff | 07:15 |
| quiquell | badboy: https://zuul-ci.org/docs/zuul/admin/quick-start.html | 07:16 |
| badboy | quiquell: I've already tried the docker approach | 07:16 |
| quiquell | badboy: ack | 07:16 |
| badboy | quiquell: it working great | 07:16 |
| badboy | quiquell: now I want to do that on my own machines without docker | 07:16 |
| quiquell | badboy: was good for us a our team, we are using it for other matters | 07:16 |
| quiquell | badboy: like baremetal ? | 07:17 |
| badboy | like vms | 07:17 |
| quiquell | I see | 07:18 |
| badboy | quiquell: I cannot wrap my head around the web dashboard config for nginx or apache | 07:18 |
| badboy | quiquell: do you have any working vhost config for that? | 07:19 |
| quiquell | badboy: nah, I am just using docker version of it for my team purposes | 07:19 |
| quiquell | badboy: and reporting possible issues | 07:20 |
| badboy | quiquell: i see | 07:20 |
| quiquell | badboy: I am not a zuul developer, maybe some exporadical patches | 07:20 |
| tristanC | badboy: vhost configuration depends on if you want whitelabel tenant interface or multi-tenant | 07:22 |
| quiquell | tristanC: Good morning sir | 07:29 |
| tristanC | quiquell: 'morning | 07:31 |
| badboy | tristanC: hi, I would be good to post an example config in the docs | 07:33 |
| tristanC | badboy: aren't those enough https://zuul-ci.org/docs/zuul/admin/installation.html#web-deployment-options ? | 07:34 |
| tristanC | badboy: you may want to give those a try too: https://review.openstack.org/#/c/608850/2/doc/source/admin/installation.rst | 07:35 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Apply requested tags locally for docker build https://review.openstack.org/632790 | 07:36 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Document upload-logs reliance on add-fileserver https://review.openstack.org/632572 | 07:37 |
| badboy | tristanC: thx, will look into that | 07:38 |
| quiquell | tristanC: It's this good now ? https://review.openstack.org/#/c/623294/ | 07:38 |
| quiquell | tristanC: Do I need to test something else ? | 07:38 |
| badboy | tristanC: /usr/share/zuul doesn't exist | 07:38 |
| badboy | tristanC: where does the "pip3 install ." put the web files? | 07:39 |
| *** quiquell is now known as quiquell|brb | 07:42 | |
| tristanC | badboy: iirc you need yarn installed to get pip to build the web files, and iirc pip puts them in site-packages/zuul/web/static | 07:44 |
| tristanC | badboy: you may want to grab a pre-build version from http://tarballs.openstack.org/zuul/zuul-content-latest.tar.gz | 07:45 |
| badboy | tristanC: so the vhost config shou point to /usr/local/lib/python3.6/site-packages/zuul/web/static, right? | 07:45 |
| tristanC | quiquell|brb: it lgtm | 07:46 |
| tristanC | badboy: that would work i guess | 07:46 |
| badboy | tristanC: worth noting in the docs I guess | 07:47 |
| badboy | tristanC: on Ubuntu 18.04 it's /usr/local/lib/python3.6/dist-packages/zuul/web/static/ | 07:48 |
| *** quiquell|brb is now known as quiquell | 08:18 | |
| quiquell | tristanC: can you workflow it? | 08:18 |
| *** themroc has joined #zuul | 08:18 | |
| tristanC | quiquell: actually i'm not a zuul maintainer :) | 08:19 |
| quiquell | tristanC: then some blessing with +1 would be good too | 08:20 |
| quiquell | Thanka | 08:20 |
| *** badboy has quit IRC | 08:36 | |
| *** badboy has joined #zuul | 08:42 | |
| *** jpena|off is now known as jpena | 08:42 | |
| *** avass has joined #zuul | 08:43 | |
| *** hashar has joined #zuul | 08:44 | |
| badboy | where did I go wrong? | 08:48 |
| badboy | paramiko.ssh_exception.SSHException: Signature verification (ssh-ed25519) failed. | 08:49 |
| *** ianychoi has quit IRC | 08:55 | |
| *** bjackman has quit IRC | 09:08 | |
| *** bjackman has joined #zuul | 09:12 | |
| *** bjackman has quit IRC | 09:20 | |
| *** bjackman has joined #zuul | 09:26 | |
| *** bjackman has quit IRC | 09:30 | |
| *** sanjayu_ has joined #zuul | 09:36 | |
| *** saneax has quit IRC | 09:38 | |
| tristanC | badboy: we had the same issue recently, it seems like paramiko doesn't like non ssh-rsa keys, removing the offending type from ~zuul/.ssh/known_hosts should fix it | 09:39 |
| *** luizbag has joined #zuul | 09:50 | |
| *** bjackman has joined #zuul | 09:50 | |
| *** electrofelix has joined #zuul | 09:51 | |
| quiquell | tristanC: this is host key verification or priv key verification ? | 09:52 |
| quiquell | tristanC: priv key the issues is different | 09:52 |
| *** sshnaidm|afk is now known as sshnaidm | 09:53 | |
| tristanC | quiquell: it seems like host key verification | 09:53 |
| quiquell | badboy, tristanC: Maybe this is related too https://storyboard.openstack.org/#!/story/2004842 | 09:53 |
| avass | hmm, is there any way to get a 'worker-id', like a number, from nodepool to be able to set a different workspace depending on worker-id on a static node when max-parallel jobs is set higher than 1? | 09:55 |
| *** ssbarnea|rover has joined #zuul | 10:02 | |
| *** ssbarnea|bkp2 has quit IRC | 10:04 | |
| quiquell | avass: There is a web api at nodepool, maybe there is something there than you can use | 10:05 |
| quiquell | avass: https://zuul-ci.org/docs/nodepool/operation.html#web-interface | 10:06 |
| *** sanjayu_ has quit IRC | 10:07 | |
| *** ssbarnea|bkp2 has joined #zuul | 10:07 | |
| *** sanjayu_ has joined #zuul | 10:08 | |
| quiquell | corvus, fungi: Are you up for a +w https://review.openstack.org/#/c/623294/ ? it's look ok now | 10:10 |
| *** ssbarnea|rover has quit IRC | 10:10 | |
| quiquell | ianw: ^ ? | 10:10 |
| avass | quiqell: thanks I'll take a look at that | 10:12 |
| quiquell | avass: yw | 10:14 |
| panda | any feedback on http://lists.zuul-ci.org/pipermail/zuul-discuss/2019-January/000715.html before I start fixing 10 millions lines in tests ? | 10:22 |
| *** bhavikdbavishi has quit IRC | 10:22 | |
| openstackgerrit | Jean-Philippe Evrard proposed openstack-infra/zuul-jobs master: Allow different filenames for Dockerfiles https://review.openstack.org/632979 | 10:31 |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/zuul master: Scheduler: make autohold hold_list configurable https://review.openstack.org/632498 | 10:34 |
| quiquell | tristanC: Do you know if zuul just open one gerrit connection per gerrit server or multiple ? | 11:01 |
| *** ssbarnea|rover has joined #zuul | 11:23 | |
| *** ssbarnea|bkp2 has quit IRC | 11:24 | |
| *** hashar has quit IRC | 11:30 | |
| *** bhavikdbavishi has joined #zuul | 11:36 | |
| tristanC | quiquell: only one iirc, but perhaps multiple channels when requestion change... why do you ask? | 11:39 |
| quiquell | tristanC: we were running zuul with a user and get too much concurrent connections | 11:40 |
| quiquell | tristanC: looks like we where not graceful shutting down, now it's all good | 11:40 |
| sshnaidm | tristanC, do you know what is policy for building zuul/nodepool-launcher for example? I see it's 6 days old | 11:43 |
| sshnaidm | and all other zuul/* containers are pretty old, a few days before | 11:43 |
| quiquell | sshnaidm, tristanC: openstackzuul-pbrx-push-container-images is failing | 11:45 |
| sshnaidm | quiquell, link..? | 11:46 |
| quiquell | http://zuul.openstack.org/builds?project=openstack-infra%2Fnodepool&job_name=openstackzuul-pbrx-push-container-images | 11:46 |
| tristanC | sshnaidm: you mean docker.io/zuul namespace? i don't know how this works | 11:46 |
| *** gtema has joined #zuul | 11:46 | |
| quiquell | denied: requested access to the resource is denied | 11:47 |
| quiquell | sshnaidm: is at post pipeline, so it does not vote in the gates :-/ | 11:48 |
| quiquell | sshnaidm: maybe they reached a limit at dockerhub | 11:48 |
| quiquell | sshnaidm: or password changed | 11:48 |
| sshnaidm | quiquell, yeah, asking on infra channel | 11:50 |
| quiquell | sshnaidm: Let's see | 11:50 |
| *** ssbarnea|bkp2 has joined #zuul | 11:52 | |
| *** ssbarnea|rover has quit IRC | 11:54 | |
| *** hashar has joined #zuul | 11:58 | |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Default private_ipv4 to use public_ipv4 address when null https://review.openstack.org/623294 | 12:16 |
| *** panda is now known as panda|lunch | 12:17 | |
| *** gtema has quit IRC | 12:18 | |
| *** quiquell is now known as quiquell|lunch | 12:36 | |
| *** ssbarnea|rover has joined #zuul | 12:37 | |
| *** ssbarnea|bkp2 has quit IRC | 12:39 | |
| *** jpena is now known as jpena|lunch | 12:40 | |
| *** pcaruana has quit IRC | 12:40 | |
| *** jesusaur has quit IRC | 12:48 | |
| *** quiquell|lunch is now known as quiquell | 12:54 | |
| badboy | should I be worried about something like this in Zuul's logs? | 12:55 |
| badboy | AttributeError: 'MergeJob' object has no attribute 'updated' | 12:55 |
| *** panda|lunch is now known as panda | 12:55 | |
| *** quiquell is now known as quiquell|brb | 12:59 | |
| *** bhavikdbavishi has quit IRC | 12:59 | |
| *** bhavikdbavishi has joined #zuul | 13:00 | |
| *** bjackman has quit IRC | 13:09 | |
| *** gtema has joined #zuul | 13:09 | |
| *** quiquell|brb is now known as quiquell | 13:15 | |
| *** pcaruana has joined #zuul | 13:20 | |
| *** hashar has quit IRC | 13:22 | |
| odyssey4me | sshnaidm It could useful to have userdata_file as another option? Either that, or some way to have it grab a file instead of specifying the content directly in nodepool.yaml | 13:24 |
| sshnaidm | odyssey4me, I thought about it, but wonder - where can I put the file so that it could be passed to nodepool? | 13:25 |
| odyssey4me | sshnaidm well, the file would need to be local to nodepool at the specified path | 13:25 |
| odyssey4me | how it gets there is up to the user | 13:25 |
| sshnaidm | odyssey4me, so it should be on nodepool host itself | 13:25 |
| odyssey4me | sshnaidm yeah - it useful when using a bash script to have that... trying to use the native yaml formatted config gets a bit complicated | 13:26 |
| sshnaidm | odyssey4me, yeah, it could be useful I think | 13:27 |
| odyssey4me | sshnaidm I wonder though, whether a new attribute makes sense - or whether there's another way to do it | 13:31 |
| *** gtema has quit IRC | 13:31 | |
| odyssey4me | in ansible, for example, we use a jinja lookup - something like this... userdata: "{{ lookup('file', /path/to/file) }}" | 13:32 |
| odyssey4me | that pulls the file into the value of the var | 13:32 |
| *** gtema has joined #zuul | 13:32 | |
| sshnaidm | odyssey4me, mm.. could it be used as "instance-properties: userdata: /path/to/file" ? | 13:32 |
| sshnaidm | odyssey4me, I think I saw some parameter for file in particular | 13:33 |
| odyssey4me | oh really? | 13:33 |
| sshnaidm | odyssey4me, yeah.. need to find that, if I'm not wrong.. | 13:33 |
| *** rlandy has joined #zuul | 13:37 | |
| *** jpena|lunch is now known as jpena | 13:43 | |
| *** bjackman has joined #zuul | 13:49 | |
| odyssey4me | sshnaidm did you mean in nodepool itself, or in shade or elsewhere? | 13:50 |
| sshnaidm | odyssey4me, I mean userdata file parameter to nova client | 13:51 |
| sshnaidm | odyssey4me, or maybe it's possible via meta-data: https://zuul-ci.org/docs/nodepool/configuration.html#attr-providers.[openstack].pools.labels.instance-properties | 13:52 |
| *** badboy has quit IRC | 13:52 | |
| sshnaidm | odyssey4me, I'm a little bit confused with all these options.. | 13:52 |
| Shrews | panda: i replied to that email | 13:55 |
| Shrews | sshnaidm: what option is confusing you? | 14:03 |
| panda | Shrews: thanks. | 14:04 |
| odyssey4me | sshnaidm yeah, meta-data is different - it's for properties of the server like this sort of stuff: https://docs.openstack.org/nova/queens/user/config-drive.html#openstack-metadata-format | 14:09 |
| *** gtema has quit IRC | 14:15 | |
| *** ssbarnea|rover has quit IRC | 14:19 | |
| *** bjackman has quit IRC | 14:21 | |
| *** ssbarnea|rover has joined #zuul | 14:22 | |
| *** bjackman has joined #zuul | 14:24 | |
| *** avass has quit IRC | 14:33 | |
| sshnaidm | odyssey4me, so I suppose the easiest option will be just to read file in path and to pass string to cliend with current userdata var | 14:39 |
| sshnaidm | odyssey4me, but still requires a different option | 14:40 |
| SpamapS | odyssey4me: shouldn't you be generating your nodepool yaml via config management? So you can just use ansible and jinja and get what you need there.. no? | 14:51 |
| odyssey4me | SpamapS I do happen to be, yes - but using "{{ lookup('file', /path/to/script) }}" is resulting in an invalid yaml formatted file | 14:52 |
| odyssey4me | At this point I'm thinking of just having a simple curl | bash in userdata instead. | 14:53 |
| SpamapS | json encode it | 14:54 |
| SpamapS | you'll be fine | 14:54 |
| SpamapS | Also, how big is your userdata? because, it should be like, 1-2 lines of bash usually. | 14:54 |
| SpamapS | big userdata is an anti-pattern. | 14:54 |
| odyssey4me | SpamapS It's a bit of a script - yeah, I know, it's a bit of config and required package installs to make sure jenkins/ansible can do things on it. It's longer than it has to be for improved readability. | 14:55 |
| SpamapS | Interesting | 14:56 |
| odyssey4me | In particular this is for Ironic/OnMetal hosts where we can't implement things in a diskimage. | 14:56 |
| SpamapS | (I've seen some pretty awful abuses of userdata, so don't feel bad. ;-) | 14:56 |
| odyssey4me | It's not *that* bad. :p https://raw.githubusercontent.com/rcbops/rpc-gating/85ca0b3f7c462a949677300908db299502f3b371/scripts/user_data_pubcloud.sh | 14:56 |
| SpamapS | And yeah typically it's a script that will do something like install python and an SSH key and ping somewhere so ansible can start pwning the box. | 14:57 |
| pabelanger | odyssey4me: SpamapS: it feels like this would be a good use case for ready-scripts, which nodepool used to support | 14:57 |
| pabelanger | we used to have bash script, that would setup things like dns / mirrors before handing it over to zulu | 14:57 |
| pabelanger | zuul* | 14:57 |
| SpamapS | Either way, I think if you add a |to_json after your lookup, you'll get valid yaml. | 14:57 |
| odyssey4me | pabelanger yeah, and with userdata support, there is kinda a ready script function now | 14:57 |
| SpamapS | (it should encode it as a json string with everything escaped properly) | 14:58 |
| SpamapS | and a json string is a valid yaml string. | 14:58 |
| odyssey4me | SpamapS hmm, lemme have a go at that - thanks | 14:58 |
| odyssey4me | I tried to_yaml and that didn't work out | 14:58 |
| pabelanger | odyssey4me: don't really know user_data well, but maybe just curl it onto baremetal, rather then push from nodepool? | 14:58 |
| SpamapS | pabelanger: yeah, same difference really.. the point is that you still need something to customize a stock image so your config management can start doing the heavy lifting. | 14:58 |
| SpamapS | odyssey4me: yeah, to_json is more strict. | 14:59 |
| odyssey4me | pabelanger this tells nodepool when creating an instance what to run when the instance boots - it uses cloud-init to execute it... | 15:00 |
| odyssey4me | cloud-init has its own yaml formatted config, but any script can be run... and the built-in stuff is yet another syntax to learn, so meh | 15:00 |
| SpamapS | that's way way way more than you need for ansible to work btw. | 15:00 |
| pabelanger | odyssey4me: okay, understand now | 15:00 |
| SpamapS | odyssey4me: does OnMetal not support nova SSH keys? | 15:01 |
| odyssey4me | SpamapS yes, of course - the intent is absolutely to cut that set down | 15:01 |
| odyssey4me | SpamapS it does, but we want *all* our keys, not just the jenkins account keys | 15:01 |
| SpamapS | yeah, put those keys in the ansible | 15:01 |
| SpamapS | You can have a pre-ansible that is basically this bash script. | 15:01 |
| SpamapS | All you need is python and an SSH key. | 15:02 |
| SpamapS | It's the single most important feature of Ansible. :) | 15:02 |
| SpamapS | like I said, should be 1-2 lines of bash in userdata. | 15:03 |
| * mordred has still never actually used a cloud-init userdata script | 15:06 | |
| corvus | you can put multiple ssh keys in an openstack 'keypair'. openstack-infra has all of the infra-root keys in one. | 15:06 |
| mordred | corvus: ++ | 15:07 |
| mordred | yup. there is nothing about openstack keypairs that requires you only put one key in it | 15:07 |
| pabelanger | I think we use the raw task on a nodepool that ricardo built for network images, in an effort to bootstrap them, since they don't support python | 15:08 |
| pabelanger | would need to check again | 15:08 |
| pabelanger | nodepool node* | 15:08 |
| odyssey4me | pabelanger yep, a raw/script task works if python is not present on the target node | 15:09 |
| SpamapS | TIL you can install python with raw. | 15:09 |
| odyssey4me | https://github.com/rcbops/rpc-gating/blob/master/playbooks/setup_openstack_instances.yml#L147-L182 | 15:09 |
| mordred | really shouldn't need any userdata scripts with ansible as long as you put ssh keys in to keypairs | 15:10 |
| SpamapS | mordred: um, I've been meaning to tell you.. we got really drunk in Atlanta once and I took a video of you using a userdata script to bootstrap imagemagick to crop some... sensitive photos... | 15:10 |
| mordred | because once an ssh key is on, the rest can be done with ansible | 15:10 |
| mordred | SpamapS: that seems ... unlikely :) | 15:10 |
| SpamapS | mordred: totally happened | 15:11 |
| mordred | SpamapS: but - you never know with karaoke bars in atlanta | 15:11 |
| SpamapS | and you did it from a mac | 15:11 |
| mordred | SpamapS: yeah. that definitely sounds like a thing i'd do | 15:11 |
| SpamapS | you even editted the script with textmate | 15:11 |
| mordred | I bet I found it to be a very pleasant workflow | 15:11 |
| SpamapS | I've found trolling you to be a very pleasant workflow. | 15:11 |
| * SpamapS retreats to breakfasting w/ kids | 15:12 | |
| odyssey4me | thanks for the pointer SpamapS - using to_json appears to be working :) | 15:12 |
| odyssey4me | thanks you all for being enablers ;) | 15:12 |
| electrofelix | looking at adding some stats to zuul merger server locally (cause it's v2 thought might be of interest for v3) to try and capture variations in how long the merger:update and merger:merge jobs take and be able to detect unusual spikes for certain projects being processed | 15:13 |
| SpamapS | odyssey4me: now say 3 hail spaghetti monster's and add a task to your project management system to de-userdata your solutions. Ra-men. | 15:13 |
| electrofelix | but I'm a bit of a novice as to statsd, so I just copied the statsd call used on the pipelines for resident_time and I'm wondering if that would show me when it takes longer for certain projects compared to others to be processed | 15:15 |
| corvus | electrofelix: if you pastebin what you have i can translate :) | 15:16 |
| electrofelix | this is to help see the problem with some projects in github with lots of branches/refs and subsequently whether changes made or cleanup done helps fix | 15:16 |
| electrofelix | corvus: https://pastebin.com/bvUnpiDy | 15:17 |
| corvus | electrofelix: the statsd part of that looks okay, but in a dependent pipeline, or with a long series of dependent changes, it will record the entire operation across all branches and repos under the project under test. | 15:19 |
| corvus | electrofelix: so if you have A->B->C, then the cumulative value to update all 3 repos gets logged under A. | 15:20 |
| electrofelix | so need to move to the mergeChange to be from the start/end of each item, thanks | 15:21 |
| electrofelix | one thing I wasn't sure about was why existing resident time related stats multiply by 1000 | 15:21 |
| *** bhavikdbavishi has quit IRC | 15:22 | |
| *** bhavikdbavishi has joined #zuul | 15:22 | |
| electrofelix | the other stat we're hoping to add is to capture the time queued of these jobs being performed by the merger, with a goal of trying to understand when we need additional mergers as well as the benefit when one is added, unless I've missed a stat capturing that already? | 15:24 |
| corvus | electrofelix: statsd values are integers, so it's in microseconds (which is the javascript-standard way of representing time) | 15:24 |
| corvus | electrofelix: in v2, i couldn't say. in v3, you just look at the merger queue backlog. | 15:26 |
| electrofelix | corvus: I suspect we don't have the sustained traffic for us to have a consistent queue length, it's more like there are certain busy periods and then quiet periods but we're not always watching | 15:28 |
| electrofelix | corvus: we've some problems that might be due to only running one merger but it's kind of speculative as whether that is what is happening so was going to add some enqueue/dequeue info and make sure the onMergeCompleted does the same as onBuildCompleted for processing stats | 15:29 |
| corvus | electrofelix: right, in v3, zuul emits stats of the queue length, so if the yellow line goes up and to the right, add mergers: http://grafana.openstack.org/d/T6vSHcSik/zuul-status?panelId=31&fullscreen&orgId=1 | 15:30 |
| electrofelix | ah, yep that's missing from v2 alright | 15:31 |
| electrofelix | I might have a look at what's needed to backport the same stat | 15:31 |
| electrofelix | would the other stats for capturing length of time to process merger tasks be of interest for v3? | 15:32 |
| corvus | electrofelix: i think so, yes | 15:33 |
| dmsimard | The new zuul pipeline for promote (and the docker use case) really reminds me of the way we do container promotions in RDO although we have an intermediary registry between the builds and dockerhub | 15:34 |
| dmsimard | We build containers, push them to a private registry, have a bunch of jobs test those containers (which are tagged after a hash, not unlike how you tag them with the gerrit change) and then if everything is successful at the end, they are uploaded to dockerhub with an appropriate tag | 15:35 |
| *** pcaruana has quit IRC | 15:35 | |
| dmsimard | I guess I don't know where I am going with this, but the challenge we had was the step "have a bunch of jobs test those containers" which might span different projects or changes | 15:35 |
| dmsimard | I remember talking about this with mordred and inc0 in denver a long time ago | 15:36 |
| corvus | dmsimard: yeah, i think that's the next thing to tackle. we could build something like you describe with a zuul job graph, so that, with one change, you build containers, stick them in a local registry, then several dependent jobs use those containers. | 15:37 |
| corvus | dmsimard: but that only exists within a single change, so depends on each change starting by building all the necessary containers (including those of changes ahead of it in the queue). it would be nice to avoid that and re-use the built artifacts of changes ahead. | 15:37 |
| corvus | that, however, runs into issues with parallelization. | 15:39 |
| dmsimard | yeah, we've ended up evolving our promotion process to be asynchronous | 15:39 |
| dmsimard | jobs will eventually run against this set of container images (matching a specific tag) and report their results -- there is an outside process that will query the results and decide to promote or not based on criteria (i.e, job X Y Z must be successful) | 15:40 |
| corvus | dmsimard: that is an interesting approach. i believe that in the not too distant future we will be able to do the same with zuul's synchronous gate. | 15:42 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Bind mount ld.so.cache into bwrap context https://review.openstack.org/633019 | 15:42 |
| corvus | that is to say, accomplish the same ends, not perform the same process. :) | 15:42 |
| mordred | corvus, tobiash: ^^ found the issue with the quick-start | 15:42 |
| mordred | tobiash: you were right - it was bubblewrap related | 15:43 |
| dmsimard | corvus: yes -- I think the take away is that our promotion process requires more than one job to be successful against a set of artifacts | 15:43 |
| *** ianychoi has joined #zuul | 15:44 | |
| corvus | mordred: that should probably be rebased into the dockerfile stack... | 15:45 |
| corvus | mordred: but how about we squash all 5 changes together into one? | 15:45 |
| corvus | mordred: i'd rather not push 4 broken images to dockerhub :) | 15:46 |
| corvus | also we need to fix the nodepool change | 15:46 |
| mordred | corvus: can do - I was going to push up an updated quick-start patch with a depends-on just to make sure it worked | 15:46 |
| mordred | then start squashing things | 15:46 |
| mordred | but I can go ahead and squash | 15:46 |
| corvus | dmsimard: yeah, though the more than one job part is easy and exists today, the artifacts built by different projects and spanning changes is the tricky part. | 15:48 |
| *** quiquell is now known as quiquell|off | 15:51 | |
| mordred | corvus: what strategy do we want to use for the nodepool jobs? just add a secret to nodepool? | 15:51 |
| mordred | with the zuulzuul secret? | 15:51 |
| mordred | corvus: if so - since you just added zuulzuul to project-config, are you in an easy position to push up a zuulzuul secret to nodepool? | 15:53 |
| *** pcaruana has joined #zuul | 15:55 | |
| mordred | corvus: or do we want to make a zuul-build-image base job in project-config with an allowed-projects list? | 15:56 |
| *** gtema has joined #zuul | 15:56 | |
| corvus | mordred: i say copy to np for now; i will do | 16:00 |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool master: Add dockerhub secret https://review.openstack.org/633025 | 16:02 |
| corvus | mordred: feel free to squash ^ | 16:02 |
| mordred | corvus: awesome. updates coming | 16:05 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool master: Build images with Dockerfile instead of pbrx https://review.openstack.org/632187 | 16:11 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Build zuul containers with dockerfile not pbrx https://review.openstack.org/631840 | 16:11 |
| mordred | corvus: ^^ there - 2 changes :) | 16:11 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool master: Stop building an explicit nodepool-base image https://review.openstack.org/632188 | 16:12 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Stop building an explicit zuul-base image https://review.openstack.org/632189 | 16:13 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Use node v10 instead of node v8 https://review.openstack.org/632165 | 16:14 |
| openstackgerrit | Merged openstack-infra/nodepool master: Add a timeout for the image build https://review.openstack.org/629923 | 16:15 |
| openstackgerrit | Merged openstack-infra/nodepool master: Revert "Revert "Cleanup down ports"" https://review.openstack.org/614370 | 16:15 |
| mordred | corvus: I think we want to see the zuul change quick-start job succeed before landing the nodepool change - but in general, https://review.openstack.org/#/q/status:open+topic:replace-pbrx should be fairly solid now I thnik | 16:16 |
| corvus | mordred: agreed | 16:16 |
| mordred | I went through and abandoned all of the various things that got squashed | 16:17 |
| Shrews | fyi, a failed fedora update on my work laptop seems to have put it in an unusable state. i will likely be spending much of my day recovering from that | 16:18 |
| mordred | Shrews: I've never suffered from that before | 16:18 |
| Shrews | mordred: it's such fun | 16:18 |
| mordred | Shrews: and certainly not multiple times since the switch to systemd | 16:18 |
| mordred | Shrews: I definitly didnt' have to reinstall my laptop from scratch the last time because the failure mode was undebuggable | 16:19 |
| * Shrews suspects a *tiny* bit of sarcasm | 16:20 | |
| mordred | Shrews: I NEVER use sarcasm constantly | 16:21 |
| *** themroc has quit IRC | 16:21 | |
| Shrews | yup, this is hosed big time. /me looks for his usb key with ubuntu | 16:21 |
| dmsimard | fedora updates have been working really well for me but I haven't moved to f29 yet | 16:23 |
| corvus | Shrews: oh, well, when you get back, those 2 changes you wanted to restart with have landed :) | 16:26 |
| Shrews | corvus: ack. definitely gonna have to re-install it looks like, so not likely to take on doing that today | 16:30 |
| corvus | Shrews: good luck! | 16:31 |
| tobiash | hrm, my zuul suddenly started to timeout many jobs after two minutes | 16:31 |
| *** mhu has joined #zuul | 16:37 | |
| mordred | tobiash: that does not seem like a good new feature | 16:38 |
| tobiash | I have no idea | 16:38 |
| tobiash | and unfortunately zuul doesn't log the timeout it thinks it's enforcing | 16:38 |
| corvus | tobiash: did we land that tenant max timeout thing? | 16:40 |
| tobiash | corvus: no | 16:40 |
| tobiash | let me check if it's in my current deployment | 16:40 |
| *** panda is now known as panda|off | 16:40 | |
| *** sanjayu_ has quit IRC | 16:41 | |
| tobiash | nope | 16:41 |
| corvus | no leap seconds recently... :) | 16:41 |
| tobiash | it started together with a reconfig | 16:41 |
| tobiash | but could be a coincidence | 16:41 |
| tobiash | almost 400 timeouts in the last hour | 16:42 |
| corvus | tobiash: the inventory file should have the timeout value | 16:42 |
| tobiash | unfortunately I get only finger links | 16:43 |
| tobiash | so no logs | 16:43 |
| tobiash | but I could enable keep | 16:43 |
| *** pcaruana has quit IRC | 16:48 | |
| mordred | corvus: woot! https://review.openstack.org/#/c/631840/ passed quick-start and correctly used the built images | 16:57 |
| mordred | corvus: tox failure though | 16:57 |
| corvus | mordred: yeah, they've been failing a lot lately :( | 16:58 |
| mordred | yeah | 16:58 |
| corvus | that particular failure looks debuggable. let's recheck that, but i'll dig into that one. | 16:59 |
| mordred | kk | 16:59 |
| *** bhavikdbavishi has quit IRC | 17:01 | |
| tobiash | hrm, looks like a second reconfigure fixed it | 17:03 |
| tobiash | shall I blame cosmic radiation? | 17:03 |
| *** gtema has quit IRC | 17:18 | |
| *** bhavikdbavishi has joined #zuul | 17:20 | |
| *** mhu has quit IRC | 17:25 | |
| jkt | it seems that I cannot pass secrets from a tenant-specific repo towards a base job which is defined in a trusted config repo shared by two tenants | 18:15 |
| jkt | right? :) | 18:15 |
| jkt | my use case: have the base-job defined in that shared repo, and supply tenant-specific credentials for log upload over SCP | 18:18 |
| *** jpena is now known as jpena|off | 18:28 | |
| dmsimard | pabelanger: do you still use molecule with zuul ? how is that working out for you ? | 18:32 |
| SpamapS | jkt: My understanding of the rules: secrets can only be consumed in playbooks that are in the same repo as the secret is defined in. | 18:32 |
| pabelanger | dmsimard: yup, we are working towards doing it with our ansible-network roles | 18:32 |
| SpamapS | jkt: so if your base job is in the shared repo, the secret needs to be in the shared repo too. | 18:33 |
| dmsimard | pabelanger: have an example working somewhere ? curious to see what it looks like in practice | 18:33 |
| dmsimard | pabelanger: there's some amount of overlap between molecule and zuul as far as testing roles go | 18:33 |
| kmalloc | o/ | 18:33 |
| pabelanger | dmsimard: http://git.openstack.org/cgit/openstack/ansible-role-nodepool has tox -emolecule entry point | 18:33 |
| kmalloc | just wanted to stop in and say hi :) | 18:33 |
| * kmalloc is catching up on everything. | 18:33 | |
| dmsimard | pabelanger: with ara enabled too, isn't that fancy | 18:34 |
| pabelanger | dmsimard: yah, agree with overlad. I just added a test for ansible-role-nodepool, to show it work by default with docker, but with ansible-network roles, we'll skip docker and just use nodesets from zuul via local connection | 18:34 |
| pabelanger | dmsimard: of course :) | 18:34 |
| dmsimard | pabelanger: right, by overlap I meant for node provisioning | 18:34 |
| pabelanger | yup, people in ansible have been talking about adding some sort of node provision for zuul, but I push back a little on that. giving molecule a working inventory is better option IMO | 18:35 |
| pabelanger | dmsimard: http://logs.openstack.org/34/632034/1/check/windmill-tox-molecule/97eae0b/ is a recent run | 18:37 |
| pabelanger | look under logs folder for molecule ARA | 18:37 |
| *** bjackman has quit IRC | 18:43 | |
| *** bjackman has joined #zuul | 18:43 | |
| mordred | SpamapS, jkt: we *just* released an update to that secret usage policy - which is the pass-to-parent feature | 18:46 |
| mordred | SpamapS, jkt: with that - you can define a job in one repo that is designed to take a secret, and then define a child job in another repo that has the secret and attaches it to the job with the 'pass-to-parent' flag set to true | 18:47 |
| mordred | jkt: so - you can now do the thing you want to do | 18:48 |
| corvus | mordred: http://logs.openstack.org/40/631840/16/check/zuul-quick-start/27e9337/job-output.txt.gz#_2019-01-24_17_14_30_158354 is an interesting error | 18:51 |
| corvus | mordred: that run was with psutil 5.4.8. the previous successful run was with psutil 5.5.0 | 18:51 |
| corvus | 5.5.0 is still the latest on pypi | 18:52 |
| corvus | but either way, the error is no gcc? | 18:53 |
| *** manjeets_ has joined #zuul | 18:59 | |
| *** manjeets has quit IRC | 19:00 | |
| SpamapS | mordred: oh right, is that in 3.5.0 ? | 19:02 |
| jkt | mordred: thanks a lot, I'll give this a try | 19:05 |
| corvus | SpamapS: yes it is | 19:08 |
| corvus | https://zuul-ci.org/docs/zuul/releasenotes.html#new-features and https://zuul-ci.org/docs/zuul/user/config.html#attr-job.secrets.pass-to-parent | 19:09 |
| corvus | jkt: ^ | 19:09 |
| *** dkehn has joined #zuul | 19:10 | |
| * Shrews is not having a good/fun day, fwiw | 19:10 | |
| *** bhavikdbavishi has quit IRC | 19:11 | |
| *** manjeets_ is now known as manjeets | 19:17 | |
| SpamapS | Cool! | 19:19 |
| mordred | corvus: hrm. so the 'no gcc' error usually happens when we didnt' build a wheel for something we should have during the builder image phase | 19:21 |
| corvus | mordred: is it possible some dependency pinned psutil 5.4.8 between the two phases? | 19:23 |
| mordred | possibly ... but pip is claiming it's installing psutil because of requirements.txt | 19:23 |
| mordred | corvus: here: http://logs.openstack.org/40/631840/16/check/zuul-quick-start/27e9337/job-output.txt.gz#_2019-01-24_17_14_30_137735 | 19:23 |
| mordred | corvus: /output/requirements.txt is a verbatim copy of zuul's | 19:23 |
| corvus | oh. wow. | 19:24 |
| mordred | corvus: so for some reason, confronted with 'psutil' - pip decided to install 5.4.8 | 19:24 |
| corvus | mordred: and maybe that changed between the two phases? | 19:24 |
| corvus | mordred: like, maybe we hit an out of date mirror or something? | 19:25 |
| mordred | yeah | 19:25 |
| mordred | corvus: makes me kind of want to output a constraints file in the build step based on what we build | 19:25 |
| mordred | corvus: and supply it to the pip install | 19:25 |
| mordred | so that we'd get an error ore like "couldn't find 5.5.0 at pypi.org" instead of "gcc command not found" | 19:26 |
| corvus | mordred: that sounds reasonable; it's intended to be the same build, we don't want anything to change between the two phases | 19:27 |
| corvus | mordred: btw, are we using the mirror in opendevorg/python-builder ? | 19:27 |
| mordred | yeah. I think for now just rechecking this should be fine - that seems like an internet heisenbug | 19:27 |
| corvus | agreed -- recheck issued | 19:28 |
| mordred | corvus: no - I haven't figured out a good way to do that yet | 19:28 |
| mordred | corvus: it's on my list | 19:28 |
| corvus | mordred: oh, right... per-region... i think i'm beginning to see the problem. | 19:28 |
| mordred | corvus: yeah- and you can't bind-mount with docker build :( | 19:29 |
| mordred | corvus: I thnik the answer might be to do a build-arg with the mirror url | 19:29 |
| corvus | mordred: ah nice, i like that. | 19:29 |
| mordred | and have the assemble script splat out a mirror config if it's set | 19:29 |
| corvus | that's way better than the crazy ideas i was about to say | 19:30 |
| mordred | hehe | 19:30 |
| corvus | (they involved ansible mutating images) | 19:30 |
| corvus | forget i said anything | 19:30 |
| mordred | corvus: already forgotten | 19:30 |
| *** sshnaidm is now known as sshnaidm|off | 19:36 | |
| tobiash | ++ for mirror per buildarg, but you should take care to not persist this in the image | 19:45 |
| mordred | tobiash: yah - very much so | 19:47 |
| openstackgerrit | Elod Illes proposed openstack-infra/zuul-jobs master: Fix Editable Git install package name parsing https://review.openstack.org/633049 | 19:47 |
| mordred | tobiash: I think in the builder image it's fine - but not the final image | 19:47 |
| tobiash | Yes | 19:48 |
| mordred | corvus, tobiash: https://review.openstack.org/#/c/631840/ is green! | 20:02 |
| corvus | mordred: +3 on the nodepool side; +2 on zuul | 20:04 |
| mordred | corvus: woot | 20:05 |
| mordred | corvus: I also re-did https://review.openstack.org/632532 to make it two different jobs like you somewhat suggested | 20:06 |
| corvus | slick | 20:08 |
| jkt | corvus, mordred: I still have to define *some* secret in the parent job, right? I'm getting an error if I do not do that | 20:22 |
| corvus | jkt: shouldn't need to; i'll dig up some examples | 20:29 |
| corvus | jkt: just to double check, are you running from dockerhub images or source? | 20:29 |
| *** luizbag has quit IRC | 20:31 | |
| corvus | jkt: here's a child job: http://git.openstack.org/cgit/openstack-infra/system-config/tree/.zuul.yaml#n47 | 20:33 |
| corvus | jkt: which passes a secret to its parent: http://git.zuul-ci.org/cgit/zuul-jobs/tree/zuul.yaml#n32 | 20:33 |
| jkt | corvus: source, 3.5.0 | 20:34 |
| jkt | corvus: thanks, it works | 21:08 |
| jkt | corvus: the only undocumented thing was the need to override zuul_logserver_root as in https://gerrit.cesnet.cz/plugins/gitiles/ci/project-config/+/027a8427a1040993477cbbad4409a92da36cb400/playbooks/base/post-logs.yaml#11 | 21:08 |
| jkt | corvus: my secret definition is at https://gerrit.cesnet.cz/plugins/gitiles/ci/project-config-public/+/ef90bbe53714ffb75f063dc20b88fd646a0e506d/zuul.d/secrets.yaml , it seems that the {{secret}}.path as passed to add-fileserver it not picked up by the upload-logs role | 21:09 |
| jkt | there's also that rather cosmetic issue that I have to have a small pass-through job in both tenants just to pass the secret, https://gerrit.cesnet.cz/plugins/gitiles/ci/project-config-public/+/ef90bbe53714ffb75f063dc20b88fd646a0e506d/zuul.d/jobs.yaml | 21:11 |
| jkt | but that's just a cosmetic thing | 21:11 |
| *** ssbarnea|bkp2 has joined #zuul | 21:32 | |
| *** ssbarnea|rover has quit IRC | 21:33 | |
| openstackgerrit | Elod Illes proposed openstack-infra/zuul-jobs master: Fix Editable Git install package name parsing https://review.openstack.org/633049 | 21:37 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Drop pip.conf into target source dir https://review.openstack.org/633068 | 21:39 |
| mordred | corvus: ^^ and https://review.openstack.org/633069 | 21:40 |
| *** jesusaur has joined #zuul | 21:45 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Drop pip.conf into target source dir https://review.openstack.org/633068 | 22:13 |
| *** ssbarnea|rover has joined #zuul | 22:14 | |
| *** pwhalen has joined #zuul | 22:15 | |
| *** ssbarnea|bkp2 has quit IRC | 22:16 | |
| corvus | mordred: left several thoughts on https://review.openstack.org/633068 | 23:04 |
| *** spsurya has quit IRC | 23:04 | |
| *** bjackman has quit IRC | 23:35 | |
| *** bjackman has joined #zuul | 23:36 | |
| *** dkehn has quit IRC | 23:37 | |
| mordred | corvus: awesome. I stopped having thoughts a few hours ago | 23:48 |
| corvus | mordred: excellent choice. have shots instead. | 23:49 |
| mordred | corvus: I like all of your thoughts | 23:49 |
| mordred | corvus: I'll start to whittle them down a little bit more tomorrow | 23:49 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!