| pabelanger | for the most part, it is close, but not 100%. However, if you are used to ARA, that output will be the same | 00:00 |
|---|---|---|
| SpamapS | ParsectiX: For most cases I'd recommend writing very simple playbooks that mostly just call things your developers would run directly. | 00:04 |
| pabelanger | SpamapS: when you have cycles, do you mind sharing your ngnix? I have one up now, but it is pretty basic. Going to look into apache one and try to covert and update docs | 00:13 |
| SpamapS | pabelanger: sure I have it right here.. | 00:18 |
| SpamapS | pabelanger: http://paste.openstack.org/show/743138/ | 00:19 |
| SpamapS | white labeled for the GoodMoney tenant. | 00:19 |
| *** spsurya has quit IRC | 00:26 | |
| pabelanger | SpamapS: thanks! | 00:26 |
| *** rlandy has quit IRC | 01:00 | |
| *** ParsectiX has quit IRC | 01:42 | |
| openstackgerrit | Merged openstack-infra/zuul master: Fix secondary exception in fingergw https://review.openstack.org/632582 | 03:08 |
| *** bhavikdbavishi has joined #zuul | 05:22 | |
| *** saneax has joined #zuul | 05:28 | |
| *** saneax has quit IRC | 05:51 | |
| *** spsurya has joined #zuul | 06:08 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul-jobs master: add-build-sshkey: remove previously authorized build-sshkey https://review.openstack.org/632620 | 06:24 |
| *** quiquell|off is now known as quiquell | 06:26 | |
| quiquell | Hello | 06:54 |
| quiquell | Is this ok now https://review.openstack.org/623294 | 06:54 |
| quiquell | AJaeger, clarkb, tristanC: ^ | 06:55 |
| tristanC | quiquell: lgtm, but is there a job log that validate the new role works as expected? | 06:59 |
| quiquell | tristanC: we have a job at RDO that exercise that, have to fix it, will paste when finished | 07:02 |
| quiquell | It uses static nodepool provider | 07:03 |
| quiquell | And was working fine | 07:03 |
| tristanC | quiquell: adding the link as review comment would likely help | 07:09 |
| quiquell | Sure but yesterday the job broke for unrelated issues have to fix it, make it pass | 07:09 |
| AJaeger | quiquell: this needs testing inside OpenStack, see the previous links I gave to the review. We need to ensure it does not break our multinode jobs. | 07:19 |
| quiquell | AJaeger: ack, will do a testing review | 07:20 |
| quiquell | Aja | 07:21 |
| quiquell | AJaeger: tripleo multinode job is good option? | 07:21 |
| quiquell | Or is there less complex job to run | 07:22 |
| quiquell | AJaeger: ahh ok the policy link, will check thanks | 07:25 |
| *** bjackman has joined #zuul | 07:27 | |
| *** saneax has joined #zuul | 07:29 | |
| AJaeger | quiquell: devstack job perhaps? Sorry, can't help further here this week. | 07:32 |
| *** saneax has quit IRC | 07:36 | |
| quiquell | AJaeger: no problem will test it right, thanks | 07:37 |
| quiquell | tristanC, AJaeger: btw, do you know if there are limitations at sshkeys for the gerrit connection ? looks like some no-password encrypted keys are not working | 07:38 |
| *** saneax has joined #zuul | 07:38 | |
| quiquell | some of them works, the different is calling ssh-keygen from within the container zuul quickstart container <- this ones always work | 07:38 |
| tristanC | quiquell: what is not working? | 07:54 |
| *** hashar has joined #zuul | 08:08 | |
| *** jpena|off is now known as jpena | 08:14 | |
| *** themroc has joined #zuul | 08:25 | |
| *** panda|off is now known as panda | 08:28 | |
| quiquell | tristanC: is exactly this https://github.com/paramiko/paramiko/issues/1015 | 08:35 |
| quiquell | tristanC: there is a bug at paramiko about RSA keys with header BEGIN OPENSSH PRIVATE KEY | 08:35 |
| quiquell | tristanC: in this key it thinks that it's not a RSA but a ed25519 key | 08:35 |
| quiquell | tristanC: solution is to regenerate key with ssh-keygen -m PEM -t rsa | 08:50 |
| quiquell | tristanC: do you know if there is any fix for that at zuul or a paramiko alternative ? | 08:51 |
| quiquell | tristanC: created bug with alternatives of solutions https://storyboard.openstack.org/#!/story/2004842 | 09:06 |
| tobiash | quiquell: I think we should update docs and monitor how paramiko reacts on this | 09:31 |
| quiquell | tobiash: I suspect no one has yet use zuul with new versions of openssh | 09:32 |
| quiquell | tobiash: I can put a review for the doc, where is the proper doc for this ? | 09:32 |
| quiquell | tobiash: I mean with a key generated by new versions of openssh | 09:33 |
| tobiash | quiquell: you can grep for ssh-keygen but I'm not sure if there is a proper place in the docs that will be read by people facing this issue | 09:38 |
| tobiash | quiquell: maybe we should just wait a bit to see if paramiko has a solution and update it then | 09:38 |
| quiquell | tobiash: well the issue is from 2017, looks like it needs a big refactor :-( | 09:39 |
| openstackgerrit | Quique Llorente proposed openstack-infra/zuul-jobs master: Default private_ipv4 to use public_ipv4 address when null https://review.openstack.org/623294 | 09:40 |
| openstackgerrit | Quique Llorente proposed openstack-infra/zuul-jobs master: DNM: Test multinode https://review.openstack.org/632672 | 09:40 |
| quiquell | tobiash: this is good to test multinode after a change at the related role ? https://review.openstack.org/#/c/632672 | 09:41 |
| tobiash | quiquell: yes, many changes to zuul-jobs are currently tested by using the roles via depends-on | 09:43 |
| quiquell | tobiash: ok let's see if devstack-multinode passes I think review is ready to merge | 09:43 |
| quiquell | tobiash: thanks | 09:43 |
| tobiash | quiquell: once it passes you should note that with a link in a comment | 09:45 |
| quiquell | tobiash: ack | 09:46 |
| quiquell | tobiash: also for the current sprint we need this https://review.openstack.org/#/c/630649/ | 09:46 |
| quiquell | tobiash: is there any testing missing there ? | 09:46 |
| tobiash | quiquell: you have a +2 from me but you need more reviews ;) | 09:48 |
| quiquell | AJaeger, tobiash: +2? https://review.openstack.org/#/c/630649/ ? | 09:48 |
| quiquell | corvus: ^ | 09:49 |
| quiquell | tobiash: I always forgot about time zones | 09:49 |
| quiquell | tobiash: another one, we were thingking about adding an option to zuul autohold or zuul.conf to bypass job result filter | 09:49 |
| quiquell | tobiash: wdyt ? | 09:49 |
| tobiash | quiquell: I see that pabelanger and SpamapS already had a look at this change so they probably want to have a second look on it | 09:50 |
| tobiash | quiquell: you want to hold successful builds? | 09:50 |
| quiquell | tobiash: yep | 09:51 |
| quiquell | panda: ^ | 09:51 |
| *** bjackman has quit IRC | 09:51 | |
| tobiash | what's your use case for holding successful builds? | 09:53 |
| tobiash | what's the criteria you want to filter? | 09:53 |
| quiquell | tobiash: is a local zuul node to run tripleo CI we always want to poke at nodes after they finish | 09:54 |
| tobiash | isn't that contradicting automated ci? | 09:55 |
| *** bjackman has joined #zuul | 09:55 | |
| quiquell | tobiash: maybe, but this is about using zuul to debug tripleo jobs | 09:56 |
| quiquell | tobiash: sometimes even with a success job you wan't to check if it was doing the correect thing or do you want to test stuff at the final stage | 09:56 |
| tobiash | for debugging you can add a fail task to the last playbook and filter for failed jobs then ;) | 09:56 |
| quiquell | tobiash: yep that was our plan, but was thinking that maybe it's better to put this option upstream at zuul | 09:57 |
| panda | tobiash: quiquell I'm working on this https://review.openstack.org/632498 | 09:57 |
| panda | tobiash: quiquell I was preparing an email to discuss the use cases | 09:57 |
| quiquell | panda: ack, then all the info is there, I will shut up | 09:58 |
| quiquell | panda: let's just discuss at e-mail | 09:58 |
| *** bhavikdbavishi has quit IRC | 10:00 | |
| tobiash | SpamapS: what permissions on aws does the ec2 driver require? | 10:00 |
| *** jpena is now known as jpena|brb | 10:08 | |
| quiquell | tobiash: this multinode failures is related to the changes of the review ? http://logs.openstack.org/72/632672/1/check/devstack-multinode/aa85096/job-output.txt.gz | 10:12 |
| tobiash | quiquell: I'm not familiar with these jobs | 10:13 |
| tobiash | AJaeger: ^ | 10:13 |
| *** bjackman has quit IRC | 10:23 | |
| *** bjackman has joined #zuul | 10:28 | |
| *** pcaruana has joined #zuul | 10:37 | |
| *** andreaf has quit IRC | 10:54 | |
| *** andreaf has joined #zuul | 10:56 | |
| *** hashar has quit IRC | 11:01 | |
| *** jpena|brb is now known as jpena | 11:07 | |
| sshnaidm | what does mean message " INFO - Configuration syntax error in dynamic layout" | 11:12 |
| sshnaidm | how can I find where exactly the "syntax error" in zuul config? | 11:13 |
| sshnaidm | together with "INFO - Configuration syntax error not related to change context. Error won't be reported." | 11:13 |
| *** avass has joined #zuul | 11:19 | |
| tobiash | sshnaidm: during startup? | 11:21 |
| sshnaidm | tobiash, when submitting a patch to gerrit | 11:24 |
| sshnaidm | tobiash, zuuls doesn't run jobs and posts these messages | 11:24 |
| tobiash | hrm, that's weird, normally the configuration errors should be there | 11:24 |
| sshnaidm | just curious where to look for more meaningful message | 11:24 |
| tobiash | maybe the debug log holds the real errors | 11:25 |
| quiquell | AJaeger, tristanC, tobiash: So devstack-multinode is passing https://review.openstack.org/#/c/632672/ | 11:53 |
| *** bhavikdbavishi has joined #zuul | 11:54 | |
| quiquell | Humm to use finger at my zuul instance do I have to open port 79 ? | 12:04 |
| quiquell | Nah is here https://zuul-ci.org/docs/zuul/admin/components.html | 12:10 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: Improve connection timeout log message https://review.openstack.org/632704 | 12:11 |
| *** hashar has joined #zuul | 12:11 | |
| pabelanger | quiquell: yes, you will get parmiko issue on fedora with out -m PEM, since it is a newer version of SSH. Agree with tobiash we should document this and work upstream to properly fix RFC 4716 | 12:13 |
| quiquell | pabelanger: thanks so much | 12:14 |
| *** bjackman has quit IRC | 12:18 | |
| quiquell | pabelanger: this looks good to you ? https://review.openstack.org/#/c/623294/ it has multinode testing review at the latest comments | 12:23 |
| quiquell | pabelanger: also this https://review.openstack.org/#/c/630649/ it has already a +2 | 12:24 |
| quiquell | pabelanger: thanks | 12:24 |
| pabelanger | quiquell: not sure on the multi-node-bridge job, I haven't reallhy been involved with it. clarkb might be a good person to review | 12:26 |
| quiquell | pabelanger: the nodepool one ? | 12:26 |
| pabelanger | will review 630649 again this morning after coffee | 12:26 |
| quiquell | pabelanger: ack thanks again | 12:26 |
| *** panda is now known as panda|launch | 12:32 | |
| *** jpena is now known as jpena|lunch | 12:33 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/nodepool master: openstack: Implement user data configuration https://review.openstack.org/632706 | 12:37 |
| *** gtema has joined #zuul | 12:40 | |
| odyssey4me | heh, I see someone else already did it | 12:51 |
| odyssey4me | I've posed a question in https://review.openstack.org/630649 - perhaps mordred is best qualified to answer. | 12:52 |
| *** panda|launch is now known as panda | 12:53 | |
| quiquell | sshnaidm: ^ | 12:55 |
| sshnaidm | odyssey4me, hi, it's my patch, userdata is valid | 12:56 |
| sshnaidm | odyssey4me, tested with cloud config | 12:56 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: WIP: Support userdata for instances in aws https://review.openstack.org/632712 | 12:56 |
| odyssey4me | sshnaidm yeah, I see that - good patch :) | 12:58 |
| sshnaidm | odyssey4me, well, you're partially correct, but: https://github.com/openstack/python-novaclient/blob/master/novaclient/v2/servers.py#L662-L663 | 12:59 |
| sshnaidm | odyssey4me, zuul uses nova client and this gets "userdata" | 12:59 |
| odyssey4me | ok, so the nova library will change it - but as far as I know nodepool uses the openstacksdk, not the nova client library | 12:59 |
| sshnaidm | odyssey4me, I thin it's the same in this case | 13:00 |
| odyssey4me | sshnaidm https://github.com/openstack/openstacksdk/blob/7f4b2b4068b004d1c82af267e32cb3eeb90f016a/openstack/compute/v2/server.py#L122 | 13:00 |
| pabelanger | odyssey4me: sshnaidm: seems to say user_data: https://docs.openstack.org/openstacksdk/latest/user/model.html#server | 13:02 |
| pabelanger | I think that is the normalized data structure | 13:02 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: WIP: Make public ip configurable in aws https://review.openstack.org/632715 | 13:02 |
| sshnaidm | odyssey4me, isn't it output? | 13:03 |
| sshnaidm | pabelanger, odyssey4me https://github.com/openstack-infra/nodepool/blob/master/nodepool/driver/openstack/provider.py#L340 | 13:03 |
| sshnaidm | pabelanger, no, it's "userdata" sure, because it works :) | 13:03 |
| odyssey4me | sshnaidm I think user_data is the correct argument, and userdata is a shim | 13:03 |
| pabelanger | will defer to mordred :) | 13:05 |
| odyssey4me | so here's the path: https://github.com/openstack-infra/nodepool/blob/master/nodepool/driver/openstack/provider.py#L340 -> https://github.com/openstack/openstacksdk/blob/46cbbfd372d4ba63c2256bca8e8359a045ae455b/openstack/compute/v2/_proxy.py#L423-L433 -> https://github.com/openstack/openstacksdk/blob/46cbbfd372d4ba63c2256bca8e8359a045ae455b/openstack/compute/v2/server.py#L122 | 13:06 |
| odyssey4me | I think userdata is working, because openstacksdk will load the nova client if it's there to load, and the nova client has an alias for user_data called userdata | 13:07 |
| sshnaidm | odyssey4me, hmm.. it's used even in test: https://github.com/openstack/openstacksdk/blob/ec90fb64020bf41b5935bb78a62f323958c742fe/openstack/tests/unit/cloud/test_create_server.py#L410 | 13:10 |
| odyssey4me | yeah, I suspect mordred will know the history better - this may even induce some rage against inconsistencies :p | 13:11 |
| pabelanger | I believe non underscore variables are legacy, and variables with underscore are new way moving forward. I seem to recall mordred saying they may be removed in future | 13:11 |
| odyssey4me | clearly both are used, and quite honestly I would rather not use anything that comes out of the service client libraries if possible... openstacksdk is lighter and better | 13:12 |
| Shrews | actually, the correct path is here: http://git.openstack.org/cgit/openstack/openstacksdk/tree/openstack/cloud/openstackcloud.py#n6787 | 13:12 |
| Shrews | userdata is correct as a parameter to create_server() there | 13:12 |
| odyssey4me | heh, thanks Shrews | 13:13 |
| odyssey4me | it seems that perhaps 'userdata' is the non-encoded parameter, which http://git.openstack.org/cgit/openstack/openstacksdk/tree/openstack/cloud/openstackcloud.py#n6906 then encodes | 13:14 |
| odyssey4me | and adds as a user_data argument | 13:14 |
| odyssey4me | which is totally not confusing at all :p | 13:15 |
| sshnaidm | odyssey4me, yeah, better to be userdata64 in this case :) | 13:15 |
| sshnaidm | so userdata is good to go? | 13:15 |
| odyssey4me | yeah, it seems so - I'll +1 and comment on my own question | 13:15 |
| odyssey4me | oh, I see that tobiash already commented | 13:16 |
| sshnaidm | pabelanger, can you vote please? https://review.openstack.org/#/c/630649/ | 13:17 |
| *** rlandy has joined #zuul | 13:27 | |
| *** jpena|lunch is now known as jpena | 13:34 | |
| openstackgerrit | Benedikt Löffler proposed openstack-infra/zuul master: Report retried builds in a build set via mqtt. https://review.openstack.org/632727 | 14:01 |
| mordred | odyssey4me: fwiw, openstacksdk will _not_ load python-novaclient under any circumstances | 14:01 |
| odyssey4me | mordred ah, I was thinking of ye olde shade :p | 14:02 |
| * mordred is still reading/digesting scrollback | 14:02 | |
| mordred | odyssey4me: yah - thank goodness we're off of the client libs now! | 14:02 |
| *** panda is now known as panda|brb | 14:03 | |
| quiquell | mordred: Can you help with https://review.openstack.org/#/c/623294/ it's already working and testing patches testing multinode is working too | 14:06 |
| *** panda|brb is now known as panda | 14:08 | |
| mordred | quiquell: lgtm | 14:08 |
| quiquell | mordred: AJaeger, clarkb maybe you can help too to merge that ^ | 14:09 |
| *** badboy has joined #zuul | 14:14 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Use node v10 instead of node v8 https://review.openstack.org/632165 | 14:14 |
| badboy | mordred: I've been struggling to setup a working proof of concept of Zuul on a single machine and digging through the docs | 14:17 |
| badboy | mordred: unfortunately there are a few things missing i.e. no gearman and zookeeper info | 14:17 |
| badboy | mordred: would it be possible to mention that you need to install those yourself if you don't have them anywhere on the network? | 14:18 |
| badboy | mordred: or maybe a simple guide on how to install these? | 14:18 |
| badboy | mordred: just my two cents ;) | 14:18 |
| *** avass has quit IRC | 14:20 | |
| *** avass has joined #zuul | 14:21 | |
| pabelanger | badboy: have you looked at the quick-start tutorial? https://zuul-ci.org/docs/zuul/admin/quick-start.html that is likey best working example today | 14:22 |
| badboy | pabelanger: yes, the docker example works just fine but replicating that behavior on a non-contenerized environment is that simple | 14:23 |
| badboy | s/is that/in't that/ | 14:24 |
| badboy | well isn't | 14:24 |
| Shrews | those are also mentioned in https://zuul-ci.org/docs/zuul/admin/installation.html#external-dependencies | 14:24 |
| badboy | Shrews: true, are mentioned except Zookeeper | 14:25 |
| Shrews | badboy: 2nd paragraph under Nodepool | 14:25 |
| Shrews | but perhaps deserves a section of its own, too | 14:26 |
| badboy | Shrews: +1 | 14:26 |
| badboy | but there's also an Installation from scratch doc which describes installing nodepool and zuul | 14:26 |
| badboy | maybe adding installation guide of gearman and zookeper would be a good idea? | 14:27 |
| quiquell | Shrews: Can you help +w this https://review.openstack.org/#/c/623294/ ? <- multinode zuul-jobs role | 14:28 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Use node v10 instead of node v8 https://review.openstack.org/632165 | 14:30 |
| pabelanger | badboy: you can get gearman from zuul-scheduler today, with a config setting. But yah, zookeeper is hard dependency | 14:32 |
| pabelanger | https://zuul-ci.org/docs/zuul/admin/components.html#attr-gearman_server.start | 14:32 |
| *** quiquell is now known as quiquell|lunch | 14:33 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul master: Explicitly callout ZooKeeper as ext dependency https://review.openstack.org/632732 | 14:33 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul master: Explicitly callout ZooKeeper as ext dependency https://review.openstack.org/632732 | 14:34 |
| openstackgerrit | Merged openstack-infra/nodepool master: Support userdata for instances in openstack https://review.openstack.org/630649 | 14:36 |
| mordred | tobiash, Shrews, corvus: https://review.openstack.org/#/c/632577 is green, so I think the stack leading up to it is good to review/land (I wanted to see the quick-start working with the new images first) | 14:37 |
| tobiash | mordred: what is that awk magic supposed to do: https://review.openstack.org/#/c/632577/2/playbooks/quick-start/run.yaml ? | 14:37 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Allow nodepool to use zuul-*-image jobs https://review.openstack.org/632186 | 14:38 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Stop building an explicit zuul-base image https://review.openstack.org/632189 | 14:39 |
| mordred | tobiash: that's finding the names of all of the local images that start with zuul or nodepool but don't start with zuul/ | 14:40 |
| tobiash | mordred: I ran that docker build locally and my local images don't have a name at all, just ids | 14:41 |
| tobiash | that's why I'm confused | 14:41 |
| Shrews | mordred: https://review.openstack.org/631840 commit message seems outdated w/o the curl/gnupg2 stuff | 14:41 |
| mordred | tobiash: oh? hrm. maybe I was wrong/sad about that working - let me retry locally | 14:41 |
| tobiash | and I cannot find any evidence in the build log that it tagged these images | 14:42 |
| mordred | tobiash: yeah - I thnik you're almost certainly right - I thnik my tagged images may have been from a different build :( | 14:42 |
| tobiash | mordred: I think you need to go through the target build cycle | 14:43 |
| mordred | yeah. grump. oh well | 14:43 |
| tobiash | but the overhead of that should be minimal due to the cache | 14:43 |
| mordred | oh - totally - I'm just grumpy that I need to do it :) | 14:44 |
| tobiash | :) | 14:44 |
| openstackgerrit | Benedikt Löffler proposed openstack-infra/zuul master: Report retried builds in a build set via mqtt. https://review.openstack.org/632727 | 14:45 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Use node v10 instead of node v8 https://review.openstack.org/632165 | 14:45 |
| Shrews | mordred: did you want to correct that commit message, or just let it go? | 14:53 |
| *** hashar has quit IRC | 14:54 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Build zuul containers with dockerfile not pbrx https://review.openstack.org/631840 | 14:54 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Remove zuul-migrate and zuul-bwrap images https://review.openstack.org/632167 | 14:54 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Switch to zuul-jobs docker jobs https://review.openstack.org/632173 | 14:54 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Update quick-start job to build images with dockerfile https://review.openstack.org/632577 | 14:54 |
| mordred | Shrews: corrected | 14:54 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Use node v10 instead of node v8 https://review.openstack.org/632165 | 14:54 |
| mordred | and lets see if that version of the quick-start job patch works | 14:54 |
| tobiash | SpamapS: are you using custom AMIs or the default ones provided by amazon? | 14:56 |
| *** badboy has quit IRC | 15:01 | |
| *** quiquell|lunch is now known as quiquell | 15:03 | |
| pabelanger | CI related news, looks like travisCI has new overlords | 15:08 |
| *** bjackman has joined #zuul | 15:10 | |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Add docker image build jobs https://review.openstack.org/632172 | 15:16 |
| openstackgerrit | Merged openstack-infra/zuul master: Replace build-essential with gcc/g++ https://review.openstack.org/632576 | 15:23 |
| *** saneax has quit IRC | 15:25 | |
| *** themroc has quit IRC | 15:28 | |
| *** bjackman has quit IRC | 15:28 | |
| *** quiquell is now known as quiquell|off | 15:30 | |
| *** themroc has joined #zuul | 15:30 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Update quick-start job to build images with dockerfile https://review.openstack.org/632577 | 15:42 |
| *** bjackman has joined #zuul | 15:58 | |
| *** ParsectiX has joined #zuul | 16:05 | |
| *** ParsectiX has quit IRC | 16:05 | |
| *** ParsectiX has joined #zuul | 16:05 | |
| *** ParsectiX has quit IRC | 16:06 | |
| *** luizbag has joined #zuul | 16:27 | |
| *** sshnaidm is now known as sshnaidm|afk | 16:46 | |
| *** themroc has quit IRC | 16:46 | |
| corvus | mordred, tobiash, Shrews: i'd like to restart openstack's zuul, check that it's working, make a release, then dive back into images with ya'll | 16:47 |
| tobiash | :) | 16:48 |
| mordred | corvus: I would also like all of those things | 16:48 |
| SpamapS | tobiash: I am using Ubuntu's stock AMI's | 16:48 |
| *** hashar has joined #zuul | 16:48 | |
| SpamapS | tobiash: I update to the latest ones manually every once in a while (thanks for the reminder, it's been a few weeks) | 16:49 |
| tobiash | SpamapS: how do you install python on these? | 16:49 |
| SpamapS | I did take a stab at making nodepool-builder work with EC2.. it's actually not hard at all and I had it half-working before I got distracted | 16:49 |
| SpamapS | tobiash: I have site variables that set ansible_python_interpreter=/usr/bin/python3 | 16:49 |
| tobiash | ah ok | 16:50 |
| tobiash | that works? | 16:50 |
| SpamapS | quite well yes | 16:50 |
| SpamapS | I had to fix one bug, but that landed months ago | 16:50 |
| tobiash | so maybe we don't need ansible_python_interpreter anymore? | 16:50 |
| SpamapS | (we weren't using site variables on the ansible -m setup call) | 16:50 |
| SpamapS | No you still have to set that. | 16:51 |
| SpamapS | Ansible just doesn't know what to do with a python3-only box. | 16:51 |
| tobiash | that makes me sad | 16:51 |
| SpamapS | I'm guessing that's one of those things they won't be able to solve until 3.0 | 16:51 |
| mordred | yeah - but we could potentially stop setting it to python2 | 16:51 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Update quick-start job to build images with dockerfile https://review.openstack.org/632577 | 16:52 |
| tobiash | maybe we get autodetection in 2.8: https://github.com/ansible/ansible/issues/45852 | 16:53 |
| SpamapS | Oh yeah that'd be nice, I kind of expected it'd have to wait until some interface changes. | 16:53 |
| tobiash | maybe we just have to wait until python2 is not supported anymore ;) | 16:55 |
| *** avass has quit IRC | 17:07 | |
| corvus | i'm going to tag 9e679eadedf2b64955b0511cada91018a1a0e30a as zuul 3.5.0 | 17:11 |
| tobiash | ++ | 17:12 |
| mordred | ++ | 17:12 |
| pabelanger | ++ | 17:23 |
| pabelanger | SpamapS: tobiash: good to know about site var for ansible_python_interpreter, going to try that out myself | 17:25 |
| pabelanger | but would also be nice to expose that at job level | 17:25 |
| tobiash | what do you think about adding provider/pool specific hostvars in nodepool? | 17:26 |
| SpamapS | IIRC it has to be done at the site var level because zuul runs -m setup early on, before job vars are in the picture. | 17:26 |
| corvus | okay, 3.5.0 and the release/security announcements are out | 17:26 |
| SpamapS | tobiash: I'd think nodesets are the better place for that. | 17:26 |
| pabelanger | corvus: thanks! going to try updating now | 17:27 |
| tobiash | SpamapS: I'm not sure, nodesets are typically job focused so having infrastructure related hostvars could make sense | 17:27 |
| tobiash | corvus: cool :) | 17:28 |
| SpamapS | Anything that the ansible fact gathering doesn't already set that you can think of? | 17:28 |
| tobiash | python interpreter, maybe proxy vars | 17:28 |
| tobiash | or mirror urls | 17:29 |
| SpamapS | Yeah that all makes sense. | 17:29 |
| SpamapS | the interpreter not so much, I'd think that's image-specific. | 17:30 |
| SpamapS | (but having image vars is also not an awful idea) | 17:30 |
| tobiash | yeah | 17:30 |
| SpamapS | Though for that, there's always userdata, which IIRC is landing in openstack now or has landed. | 17:31 |
| tobiash | it has landed | 17:33 |
| tobiash | SpamapS: what do you think about InstanceInitiatedShutdownBehavior='stop'|'terminate' in the aws driver? | 17:35 |
| pabelanger | corvus: fungi: at what point do we start publishing CVE for zuul? Or guidelines around it? | 17:36 |
| tobiash | SpamapS: forget it | 17:36 |
| tobiash | I've read that wrong | 17:36 |
| pabelanger | was just asked about it downstream, and don't really know | 17:36 |
| fungi | pabelanger: we have instructions in the user documentation about how to report suspected vulnerabilities. for requesting a cve assignment, it's a judgement call and anyone can do that | 17:37 |
| fungi | i started drafting instructions for zuul maintainers on process for triaging and handling reported vulnerabilities but need to pick that back up | 17:38 |
| pabelanger | fungi: okay, thanks. I'm guessing, doing a CVE has pros and cons too? Like look, we have a security issue but also, look a security issue | 17:38 |
| fungi | though the process the openstack vmt uses is fairly applicable, i think (with a few minor tweaks anyway) | 17:39 |
| corvus | apparently people can request cves for things in systems that they don't maintain that aren't even security vulnerabilities | 17:40 |
| *** gtema has quit IRC | 17:41 | |
| fungi | yes, that's exactly true | 17:41 |
| mordred | cough *yaml* cough | 17:41 |
| fungi | and anyone else can dispute them too | 17:41 |
| corvus | pabelanger: would requesting cves be helpful? | 17:41 |
| fungi | the point of the cve databse seems to be lost on a lot of people. it's really for making sure there's a short reference identifier multiple parties can use as a reference when discussing or collaborating around the same vulnerability | 17:43 |
| pabelanger | corvus: I am not sure, I can ask more with internal humans. It was just the first question asked when I indicated a new security issue was found, should upgrade to 3.5.0 | 17:44 |
| corvus | fungi: you mean it's not a scorecard? | 17:45 |
| fungi | right, whether or not that unique id maps to an actual vulnerability or a mistake is somewhat orthogonal | 17:46 |
| fungi | s/mistake/mistaken report/ | 17:46 |
| mordred | tobiash, corvus, Shrews: woot! https://review.openstack.org/#/c/632577 is green now - this time legitimately and for real I think | 17:46 |
| fungi | a lot of people incorrectly assume that an unaddressed cve for a piece of software is a sign the software is insecure somehow | 17:47 |
| corvus | mordred: it depends on a nodepool change | 17:47 |
| corvus | mordred: and that nodepool change is going to need updating to copy the zuul secret/job/etc | 17:48 |
| corvus | but i'll go ahead and start looking at the zuul change :) | 17:48 |
| mordred | corvus: yeah - it's mostly to show that the zuul change will work with compose ... however .. | 17:49 |
| mordred | I think it's still not solid | 17:49 |
| pabelanger | corvus: seems CVE request was more just to find addional info, like sha1 for potential cherry-pick. However, ML discussion also worked for today. So, think we are okay on that front for now | 17:49 |
| corvus | mordred: ooh, tell me what it's missing before i add my +2 :) | 17:50 |
| mordred | (the compose change, not the dockerfile change) | 17:50 |
| corvus | pabelanger: oh, sorry, we usually point to the gerrit change in our security announcements, i forgot to do that. | 17:50 |
| mordred | corvus: in the compose change, we're only tagging as change_xxx and not as latest, so docker compose is still pulling latest | 17:50 |
| mordred | corvus: I think I need to add tag: latest to the build variables | 17:50 |
| corvus | mordred: is there a compose change up? | 17:51 |
| mordred | corvus: yes - https://review.openstack.org/#/c/632577 | 17:51 |
| corvus | thx | 17:51 |
| mordred | corvus: but I think it's still bong | 17:51 |
| mordred | s/I think// | 17:52 |
| corvus | mordred: what does that have to do with compose? | 17:52 |
| corvus | that uses the build-docker-image role... | 17:52 |
| mordred | corvus: compose/quick-start | 17:52 |
| mordred | gah | 17:52 |
| mordred | s/compose/quick-start/ | 17:52 |
| corvus | ok. that makes sense. | 17:53 |
| mordred | sorry - the comose invocation in that change is not consuming the images built by the build step | 17:53 |
| corvus | i thought you were talking about using docker-compose to build. which is a thing that can be done. | 17:53 |
| mordred | yeah - I started looking at that too ... and I think I might have to | 17:54 |
| corvus | mordred: i think we can have build-docker-image tag the images latest as well | 17:54 |
| corvus | mordred: or, rather, have it tag the images it builds with whatever is is docker_images[].tag (which defaults to latest) | 17:54 |
| mordred | corvus: which do you think is better here? for compose, are we ok growing a dependency on the nodepool source being cloned adjacent to the zuul source? | 17:55 |
| corvus | i don't think having the extra local tags will affect the operation of the other 2 roles, and it would make things more consistent between the local and eventual upstream env. and it would make this work, yeah? | 17:55 |
| mordred | corvus: yah | 17:55 |
| corvus | mordred: for the purposes of quick-start, i think no. | 17:55 |
| mordred | corvus: yah. so in that case, I think putting build things into the compose file is going to make things less good - let's do the build role thing | 17:56 |
| corvus | (we don't expect people to *do* anything with the zuul source code, we only have zuul cloned because that's where it happens to live) | 17:56 |
| mordred | yah | 17:56 |
| corvus | (and if someone said "we should put quick-start in a repo of it's own" i think that's compelling enough we'd have a serious conversation about it) | 17:56 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Apply requested tags locally for docker build https://review.openstack.org/632790 | 18:01 |
| mordred | corvus: ^^ | 18:01 |
| tobiash | SpamapS: is it correct that ec2 picks the availability zone by the selecting the subnet? | 18:01 |
| tobiash | as I understood each az has its own subnets? | 18:01 |
| tobiash | so if I want nodepool to spread over az's in aws I'd have to either don't define a subnet or have three pools? | 18:03 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Update quick-start job to build images with dockerfile https://review.openstack.org/632577 | 18:03 |
| mordred | corvus: ^^ and there is the quick-start change with depends-on on that zuul-jobs change added | 18:04 |
| *** hashar is now known as hasharDinner | 18:04 | |
| mordred | corvus: I think it might be nice to put $something into the quick-start job to verify that the compose used the image we built - but I'm not sure what atm | 18:04 |
| corvus | mordred: small fix needed on 790 | 18:05 |
| mordred | corvus: ++ | 18:06 |
| corvus | mordred: what does it look like if the job doesn't use it? does the local image registry have extra zuul images? maybe we could do a docker image list and make sure it looks right? | 18:06 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Apply requested tags locally for docker build https://review.openstack.org/632790 | 18:06 |
| mordred | corvus: yeah - I think that would work | 18:07 |
| mordred | corvus: look for images with change_ in their tags | 18:07 |
| mordred | corvus: and make sure the sha of those matches the :latest versions of the same images | 18:07 |
| corvus | mordred: ya, i think that would do it | 18:07 |
| corvus | mordred: while you're there, can you have the quick-start job emit docker image list into the logs? | 18:08 |
| corvus | (that'll help us double check this and identify problems in the future) | 18:08 |
| mordred | corvus: yeah. in fact, why don't I do that real quick while I work on the other thing | 18:08 |
| corvus | cool, then we'll have something to compare | 18:09 |
| *** saneax has joined #zuul | 18:11 | |
| *** saneax has quit IRC | 18:11 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Update quick-start job to build images with dockerfile https://review.openstack.org/632577 | 18:11 |
| SpamapS | tobiash: correct, subnets do not span az's | 18:15 |
| SpamapS | tobiash: note that if you specify multiple AZ's in OpenStack, they don't spread either. You get one AZ for all your nodes for each request (which is particularly broken if you have large node requests and multiple AZ's, because min-ready's choose a random AZ) | 18:17 |
| *** bhavikdbavishi has quit IRC | 18:17 | |
| SpamapS | tobiash: so arguably it's better to have 1:1 pool:AZ | 18:17 |
| *** bhavikdbavishi has joined #zuul | 18:18 | |
| *** saneax has joined #zuul | 18:18 | |
| *** saneax has quit IRC | 18:18 | |
| tobiash | SpamapS: got it | 18:19 |
| *** ssbarnea|bkp2 has joined #zuul | 18:28 | |
| *** ssbarnea|rover has quit IRC | 18:29 | |
| *** gtema has joined #zuul | 18:29 | |
| *** themroc has joined #zuul | 18:41 | |
| corvus | mordred: good news and bad news! | 18:44 |
| *** jpena is now known as jpena|off | 18:44 | |
| corvus | mordred: good news: we're using the built container images now! http://logs.openstack.org/77/632577/7/check/zuul-quick-start/90b3631/job-output.txt.gz#_2019-01-23_18_26_19_019043 | 18:45 |
| mordred | yes! | 18:45 |
| *** bhavikdbavishi has quit IRC | 18:45 | |
| corvus | mordred: bad news is they don't work | 18:45 |
| mordred | I agree - they were definitely correctly used | 18:45 |
| corvus | mordred: http://logs.openstack.org/77/632577/7/check/zuul-quick-start/90b3631/container_logs/executor.log | 18:45 |
| corvus | (but even that is good news -- testing is working!) | 18:45 |
| corvus | 2019-01-23 18:27:53,112 DEBUG zuul.AnsibleJob: [build: d80f72a4e2dc45ac86481294ba7611ca] Ansible output: b'/usr/local/bin/python: error while loading shared libraries: libpython3.7m.so.1.0: cannot open shared object file: No such file or directory' | 18:45 |
| corvus | (sorry, can't deep link that one, but that's from that executor log ^) | 18:46 |
| mordred | ooh. well that's certainly weird | 18:46 |
| mordred | corvus: I am investigating that locally | 18:49 |
| mrhillsman | q: static driver does not accept range of ip addresses; i have to create one entry per ip? | 18:53 |
| tobiash | mrhillsman: currently yes | 18:54 |
| mrhillsman | if yes does anyone have a snazzy way to manage that or ideas/implementations of how they are managing that | 18:54 |
| tobiash | mrhillsman: you could use yaml anchors to simplify it at least a little bit | 18:55 |
| *** mugsie has quit IRC | 18:55 | |
| mrhillsman | will look into that; ty sir | 18:56 |
| *** mugsie has joined #zuul | 18:57 | |
| tobiash | yw | 18:57 |
| tobiash | mordred: corvus: I guess that's because /usr/local is not mounted into the bwrap | 18:58 |
| tobiash | that probably needs to be either added to the (un)trusted-ro-paths or added into zuul as default mount | 18:59 |
| tobiash | hrm, I lied, the whole /usr is mounted | 19:01 |
| pabelanger | 3.5 deployed and restarted! thanks | 19:05 |
| pabelanger | Hmm, I thought we included the version number of zuul in jobs someplace | 19:06 |
| *** gtema has quit IRC | 19:07 | |
| *** electrofelix has quit IRC | 19:11 | |
| corvus | pabelanger: no, but it's on the status page | 19:26 |
| *** spsurya has quit IRC | 19:55 | |
| *** dkehn has joined #zuul | 19:58 | |
| *** themr0c has joined #zuul | 20:10 | |
| *** themroc has quit IRC | 20:13 | |
| *** luizbag has quit IRC | 20:21 | |
| Shrews | soooo, if folks are feeling adventurous, i think we have 2 nodepool reviews up that we should get in and restart the openstack builders and launchers with them: https://review.openstack.org/614370 and https://review.openstack.org/629923 | 20:39 |
| Shrews | those would be worthy of tagging a new nodepool release if there are no issues found | 20:39 |
| Shrews | or not | 20:40 |
| tobiash | oh the port revert revert :) | 20:48 |
| corvus | we usually like to wait for 4 reverts before adding new features... ;) | 20:52 |
| tobiash | Shrews: I just noticed that I'll have to adapt my dib wrapper for windows image builds due to --logfile, good to know | 20:53 |
| tobiash | Shrews: am I reading this correctly that we don't kill dib on a timeout? | 20:54 |
| Shrews | tobiash: right. i think our signal handling up front prevents that from working | 20:55 |
| corvus | Shrews: that seems strange... zuul-executor kills ansible all the time. | 20:56 |
| Shrews | corvus: that's weird. both p.kill() and p.terminate() caused tests to hang for me | 20:56 |
| Shrews | how does zuul-executor kill the subprocess? | 20:57 |
| tobiash | Shrews: pgid = os.getpgid(self.proc.pid) | 20:58 |
| tobiash | os.killpg(pgid, signal.SIGKILL) | 20:58 |
| corvus | i think we start ansible in a new process group | 20:59 |
| corvus | maybe we need to do that here | 20:59 |
| tobiash | more context here: https://git.zuul-ci.org/cgit/zuul/tree/zuul/executor/server.py#n1775 | 20:59 |
| *** themr0c has quit IRC | 20:59 | |
| Shrews | corvus: would that affect killing the subprocess if we kill the builder process itself? i thought that was the reason for the current design (copied from v2) | 21:00 |
| Shrews | (signal propogates to the child something something) | 21:01 |
| corvus | Shrews: that's unclear to me right now. what we do in zuul, however, would end up sending the signal to all the *dib* subprocesses, which probably isn't what we want | 21:01 |
| corvus | iiuc, we want to send dib a nice friendly signal to tell it to start killing its subprocesses | 21:01 |
| corvus | (if, instead, we just want to give up on dib ever doing anything useful after that point, and kill it all, then we should do what zuul is doing) | 21:01 |
| corvus | (we do it this way in zuul because, in the case of a job timeout, there is not ansible cleanup we want to run, and we know we can clean up behind ansible. but we're not doing things like mounting filesystems.) | 21:02 |
| corvus | Shrews: if i wanted to replicate what you saw, what test should i run? | 21:03 |
| Shrews | corvus: the newly added test | 21:03 |
| corvus | k. imma gonna poke at that a bit. | 21:03 |
| Shrews | corvus: add the p.kill() call to the "except subprocess.TimeoutExpired" section | 21:04 |
| * Shrews gladly accepts additional code poking | 21:05 | |
| *** hasharDinner is now known as hashar | 21:09 | |
| *** rlandy is now known as rlandy|brb | 21:10 | |
| Shrews | corvus: oh, hrm, i'm seeing something different when using ttrun instead of tox. | 21:16 |
| corvus | Shrews: yeah, i'm looking at whether we need to adjust wait_for_builds | 21:16 |
| * tobiash just ran an experimental job on a 72 core vm in aws | 21:18 | |
| tobiash | htop looks awesome... | 21:18 |
| corvus | Shrews: i'm also seeing that sometimes wait_for_builds sees dib files, but sometimes it doesn't. i'm wondering if, with the pkill, we kill the process before it writes the files that waitForBuild is looking for | 21:20 |
| corvus | sorry, i made up 'wait_for_builds' it's waitForBuild | 21:21 |
| Shrews | corvus: agreed. the fake dib doesn't write the manifest until the end of the script | 21:21 |
| corvus | Shrews: maybe we should skip checking for the file if the state isn't ready? | 21:21 |
| Shrews | corvus: i'm going to try a flag to override that test | 21:21 |
| Shrews | or that | 21:22 |
| corvus | Shrews: based on me hacking out that section of the code, it's looking very promising locally. :) | 21:23 |
| corvus | i'll leave a review comment for posterity and leave it to you to push up a new ps | 21:23 |
| Shrews | corvus: yeah. thx for the 2nd pair of eyes there | 21:25 |
| corvus | Shrews: np, glad it wasn't actually signal handler debugging :) | 21:26 |
| Shrews | i'm mad i missed that the first time | 21:26 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Add a timeout for the image build https://review.openstack.org/629923 | 21:27 |
| mordred | in case anyone was wondering, /bin/bah is not a valid command | 21:34 |
| Shrews | mordred: /sbin/bah? | 21:35 |
| corvus | lol | 21:36 |
| Shrews | mordred: ah, you must install the bah package. it also gives you /bin/flerp | 21:36 |
| *** hashar has quit IRC | 21:39 | |
| mordred | Shrews: ++ | 21:43 |
| mordred | corvus: so - python and ansible both work fine in the image | 21:43 |
| mordred | I haven' | 21:43 |
| mordred | I haven't gotten it to break in the same way it's breaking when running zuul-executor | 21:43 |
| mordred | so I'm guessing it has something to do with the run when it's setup in zuul - which means perhaps either our callback plugin or the ara callback plugin | 21:44 |
| corvus | mordred: there's a playbook to run the zuul-quick-start job locally | 21:45 |
| mordred | yeah. I'm getting close to doing that | 21:45 |
| corvus | mordred: playbooks/quick-start/localtest.yaml | 21:46 |
| mordred | corvus: I was hoping I could trigger it just by running simple python or ansible things ... you know, if it was linker related or something | 21:46 |
| corvus | mordred: it could be an inside bwrap thing | 21:46 |
| mordred | yeah. could be | 21:46 |
| corvus | mordred: running zuul-bwrap in the image might be helpful | 21:46 |
| mordred | good call | 21:46 |
| corvus | it's not entirely straightforward, lemme conjure an invocation | 21:46 |
| mordred | might just be easier to run the localtest playbook | 21:47 |
| corvus | mordred: actually it might be easy for this case | 21:47 |
| corvus | mordred: "zuul-bwrap /tmp python" might work | 21:47 |
| corvus | mordred: or "zuul-bwrap /tmp ansible ..." | 21:47 |
| corvus | (it gets trickier to run zuul-bwrap if you want to mount secrets or deal with ssh-agent, etc, but i don't think we need all that for this) | 21:49 |
| SpamapS | TIL that unlabeling an approved PR in github does *not* prevent Zuul from merging it. | 21:49 |
| SpamapS | I wonder if I don't have the right requirements on the pipeline | 21:50 |
| SpamapS | hrm no, I guess we don't check requirements after gate tests pass. | 21:50 |
| corvus | SpamapS: i think the requirements are validated only on entry | 21:50 |
| SpamapS | why does removing +A on gerrit prevent merge? | 21:50 |
| SpamapS | (or does it?) | 21:50 |
| corvus | SpamapS: because gerrit enforces it | 21:50 |
| SpamapS | Ohhhhhhh | 21:50 |
| *** rlandy|brb is now known as rlandy | 21:51 | |
| SpamapS | I wonder if closing the PR would prevent the merge | 21:51 |
| SpamapS | kinda need a way for my users to stop the presses :-P | 21:51 |
| corvus | we *could* evaluate requirements after entry as well, though we'd need a way to maintain the current behavior too (openstack, for example, relies on it for the "clean check" behavior, where a change must have a zuul+1 to enqueue into gate) | 21:52 |
| mordred | SpamapS: does closing the PR cancel the jobs? I know in gerrit abandoning a change does | 21:52 |
| SpamapS | I don't know, that's worth playing around with a bit. | 21:53 |
| corvus | whether it currently behaves that way or not, it sounds entirely reasonable :) | 21:53 |
| mordred | yeah. in fact, I'd say if it _doesn't_ currently behave that way - it should | 21:58 |
| pabelanger | SpamapS: Yah, I ran into that issue too, haven't fixed it yet either | 23:04 |
| pabelanger | too bad branch protections don't enforce labels, would be another way to do it | 23:05 |
| *** bjackman has quit IRC | 23:06 | |
| *** bjackman has joined #zuul | 23:07 | |
| *** rlandy is now known as rlandy|bbl | 23:29 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!