| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Add upload-logs-gcs role https://review.opendev.org/703711 | 00:02 |
|---|---|---|
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Add index_links option to zuul manifest https://review.opendev.org/705580 | 00:02 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Google logs: Link to index.html files https://review.opendev.org/705586 | 00:02 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Add index_links option to zuul manifest https://review.opendev.org/705580 | 00:08 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Google logs: Link to index.html files https://review.opendev.org/705586 | 00:08 |
| *** tosky has quit IRC | 00:22 | |
| *** mattw4 has quit IRC | 00:25 | |
| *** wxy-xiyuan has joined #zuul | 00:27 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-jobs master: install-docker: enable setting docker userland proxy https://review.opendev.org/702753 | 00:27 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-operator master: Replace existing operator tasks with the new dhall function https://review.opendev.org/702106 | 00:27 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-operator master: Replace existing operator tasks with the new dhall function https://review.opendev.org/702106 | 00:57 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-operator master: Generate TLS certificats for the gearman service https://review.opendev.org/702716 | 00:57 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-operator master: Handle service restart when connections are changed https://review.opendev.org/703624 | 00:58 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-operator master: Add tenant reconfiguration when main.yaml changed https://review.opendev.org/703631 | 00:58 |
| *** rfolco has joined #zuul | 01:07 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-jobs master: install-openshift: honor the docker_userland_proxy toggle https://review.opendev.org/705589 | 01:12 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-operator master: Add OpenShift SCC and functional test https://review.opendev.org/702758 | 01:13 |
| *** jamesmcarthur has joined #zuul | 01:30 | |
| *** jamesmcarthur has quit IRC | 01:43 | |
| *** jamesmcarthur has joined #zuul | 01:44 | |
| *** jamesmcarthur has quit IRC | 01:53 | |
| *** jamesmcarthur has joined #zuul | 02:03 | |
| *** jamesmcarthur has quit IRC | 02:07 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-jobs master: install-openshift: honor the docker_userland_proxy toggle https://review.opendev.org/705589 | 02:08 |
| *** jamesmcarthur has joined #zuul | 02:15 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-operator master: Add CONTRIBUTE file https://review.opendev.org/705535 | 02:17 |
| *** jamesmcarthur has quit IRC | 02:45 | |
| *** jamesmcarthur has joined #zuul | 02:46 | |
| *** rfolco has quit IRC | 02:50 | |
| *** jamesmcarthur has quit IRC | 03:13 | |
| *** jamesmcarthur has joined #zuul | 03:13 | |
| *** bhavikdbavishi has joined #zuul | 03:15 | |
| *** bhavikdbavishi1 has joined #zuul | 03:18 | |
| *** bhavikdbavishi has quit IRC | 03:20 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 03:20 | |
| *** rlandy has quit IRC | 03:30 | |
| *** jamesmcarthur has quit IRC | 03:32 | |
| *** jamesmcarthur has joined #zuul | 03:48 | |
| *** jamesmcarthur_ has joined #zuul | 03:52 | |
| *** jamesmcarthur has quit IRC | 03:56 | |
| *** jamesmcarthur_ has quit IRC | 03:57 | |
| *** jamesmcarthur has joined #zuul | 03:58 | |
| *** jamesmcarthur has quit IRC | 04:04 | |
| *** jamesmcarthur has joined #zuul | 04:11 | |
| *** jamesmcarthur has quit IRC | 04:18 | |
| *** jamesmcarthur has joined #zuul | 04:48 | |
| *** jamesmcarthur has quit IRC | 05:00 | |
| *** jamesmcarthur has joined #zuul | 05:00 | |
| *** jamesmcarthur has quit IRC | 05:02 | |
| *** shanemcd has quit IRC | 05:06 | |
| *** saneax has joined #zuul | 05:08 | |
| *** shanemcd has joined #zuul | 05:10 | |
| *** bolg has joined #zuul | 05:21 | |
| *** yolanda has quit IRC | 05:24 | |
| *** jamesmcarthur has joined #zuul | 05:33 | |
| *** evrardjp has quit IRC | 05:33 | |
| *** evrardjp has joined #zuul | 05:34 | |
| *** raukadah is now known as chkumar|rover | 05:37 | |
| *** felixedel has joined #zuul | 06:34 | |
| *** felixedel has quit IRC | 06:40 | |
| *** felixedel has joined #zuul | 07:28 | |
| *** mugsie has quit IRC | 07:37 | |
| *** mugsie has joined #zuul | 07:41 | |
| *** evrardjp has quit IRC | 07:46 | |
| *** evrardjp has joined #zuul | 07:50 | |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: WIP: Fix mqtt log url reporting when report-build-page is active https://review.opendev.org/703983 | 08:08 |
| *** tosky has joined #zuul | 08:40 | |
| *** jpena|off is now known as jpena | 08:47 | |
| *** mhu has joined #zuul | 08:55 | |
| *** avass has joined #zuul | 09:20 | |
| *** bhavikdbavishi has quit IRC | 10:00 | |
| *** bhavikdbavishi has joined #zuul | 10:14 | |
| swest | zuul-maint: anyone up for a second +2 review? https://review.opendev.org/#/c/704758/ | 10:15 |
| *** bhavikdbavishi has quit IRC | 10:52 | |
| *** felixedel has quit IRC | 10:55 | |
| *** felixedel has joined #zuul | 11:10 | |
| *** tosky has quit IRC | 11:14 | |
| *** mugsie has quit IRC | 11:14 | |
| *** saneax has quit IRC | 11:14 | |
| *** wxy-xiyuan has quit IRC | 11:14 | |
| *** AJaeger has quit IRC | 11:14 | |
| *** webknjaz has quit IRC | 11:14 | |
| *** adam_g has quit IRC | 11:14 | |
| *** mgoddard has quit IRC | 11:14 | |
| *** aluria has quit IRC | 11:14 | |
| *** zbr has quit IRC | 11:14 | |
| *** evgenyl has quit IRC | 11:14 | |
| *** dcastellani has quit IRC | 11:14 | |
| *** reiterative has quit IRC | 11:14 | |
| *** stevthedev has quit IRC | 11:14 | |
| *** Shrews has quit IRC | 11:14 | |
| *** dustinc has quit IRC | 11:14 | |
| *** decimuscorvinus has quit IRC | 11:14 | |
| *** mnasiadka has quit IRC | 11:14 | |
| *** andreaf has quit IRC | 11:14 | |
| *** donnyd has quit IRC | 11:14 | |
| *** kmalloc has quit IRC | 11:14 | |
| *** logan- has quit IRC | 11:14 | |
| *** irclogbot_2 has quit IRC | 11:15 | |
| *** openstackstatus has quit IRC | 11:16 | |
| *** ttx has quit IRC | 11:17 | |
| *** smcginnis|FOSDEM has quit IRC | 11:18 | |
| *** dmellado has quit IRC | 11:18 | |
| *** bstinson has quit IRC | 11:18 | |
| *** Miouge has quit IRC | 11:18 | |
| *** timburke has quit IRC | 11:18 | |
| *** tristanC has quit IRC | 11:18 | |
| *** arxcruz has quit IRC | 11:18 | |
| *** irclogbot_1 has joined #zuul | 11:18 | |
| *** johanssone has quit IRC | 11:18 | |
| *** tosky has joined #zuul | 11:18 | |
| *** mugsie has joined #zuul | 11:18 | |
| *** saneax has joined #zuul | 11:18 | |
| *** wxy-xiyuan has joined #zuul | 11:18 | |
| *** AJaeger has joined #zuul | 11:18 | |
| *** webknjaz has joined #zuul | 11:18 | |
| *** adam_g has joined #zuul | 11:18 | |
| *** mgoddard has joined #zuul | 11:18 | |
| *** aluria has joined #zuul | 11:18 | |
| *** zbr has joined #zuul | 11:19 | |
| *** evgenyl has joined #zuul | 11:19 | |
| *** dcastellani has joined #zuul | 11:19 | |
| *** reiterative has joined #zuul | 11:19 | |
| *** stevthedev has joined #zuul | 11:19 | |
| *** Shrews has joined #zuul | 11:19 | |
| *** dustinc has joined #zuul | 11:19 | |
| *** decimuscorvinus has joined #zuul | 11:19 | |
| *** mnasiadka has joined #zuul | 11:19 | |
| *** andreaf has joined #zuul | 11:19 | |
| *** donnyd has joined #zuul | 11:19 | |
| *** kmalloc has joined #zuul | 11:19 | |
| *** logan- has joined #zuul | 11:19 | |
| *** arxcruz has joined #zuul | 11:20 | |
| *** johanssone has joined #zuul | 11:20 | |
| *** timburke has joined #zuul | 11:20 | |
| *** dmellado has joined #zuul | 11:21 | |
| *** donnyd has quit IRC | 11:21 | |
| *** mnaser has quit IRC | 11:21 | |
| *** mnaser has joined #zuul | 11:22 | |
| *** smcginnis|FOSDEM has joined #zuul | 11:23 | |
| *** tristanC has joined #zuul | 11:23 | |
| *** dcastellani has quit IRC | 11:23 | |
| *** donnyd has joined #zuul | 11:23 | |
| *** ttx has joined #zuul | 11:23 | |
| *** dcastellani has joined #zuul | 11:26 | |
| *** Miouge has joined #zuul | 11:27 | |
| *** bstinson has joined #zuul | 11:31 | |
| *** sshnaidm|afk is now known as sshnaidm | 11:31 | |
| *** hashar has joined #zuul | 11:32 | |
| mordred | swest: +A | 11:34 |
| *** armstrongs has joined #zuul | 11:36 | |
| swest | mordred: \o/ thanks! | 11:36 |
| *** kgz has joined #zuul | 11:40 | |
| *** armstrongs has quit IRC | 11:51 | |
| *** armstrongs has joined #zuul | 11:54 | |
| *** zbr is now known as zbr|lunch | 12:06 | |
| *** armstrongs has quit IRC | 12:07 | |
| *** AJaeger has quit IRC | 12:11 | |
| *** AJaeger has joined #zuul | 12:24 | |
| openstackgerrit | Antoine Musso proposed zuul/zuul master: web: humanize time durations https://review.opendev.org/705120 | 12:24 |
| *** jpena is now known as jpena|lunch | 12:31 | |
| *** rfolco has joined #zuul | 12:31 | |
| *** jpena|lunch is now known as jpena|off | 12:40 | |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: DNM: Reproduce merger failure when changing default branch https://review.opendev.org/705659 | 12:47 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Allow different branches for config projects https://review.opendev.org/705663 | 12:57 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Add branch filter to tenant configuration https://review.opendev.org/705664 | 12:57 |
| *** rlandy has joined #zuul | 12:58 | |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Add branch filter to tenant configuration https://review.opendev.org/705664 | 13:06 |
| *** jamesmcarthur has quit IRC | 13:15 | |
| *** jamesmcarthur has joined #zuul | 13:16 | |
| tobiash | corvus: this implements what we discussed in berlin which would make it easier for downstream projects to update zuul-jobs in a controlled manner ^ | 13:16 |
| tobiash | it also would make testing of trusted jobs easier | 13:16 |
| *** jamesmcarthur has quit IRC | 13:25 | |
| *** jpena|off is now known as jpena | 13:28 | |
| *** jamesmcarthur has joined #zuul | 13:28 | |
| *** pcaruana has quit IRC | 13:43 | |
| *** jamesmcarthur has quit IRC | 13:44 | |
| openstackgerrit | Antoine Musso proposed zuul/zuul master: ansible manager: only failed if last ansible failed https://review.opendev.org/704699 | 14:06 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Add branch filter to tenant configuration https://review.opendev.org/705664 | 14:13 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Increase timeout of zuul-build-image https://review.opendev.org/705689 | 14:17 |
| *** jamesmcarthur has joined #zuul | 14:18 | |
| *** rfolco has left #zuul | 14:32 | |
| *** michael-beaver has joined #zuul | 14:54 | |
| *** pcaruana has joined #zuul | 14:56 | |
| *** hashar has quit IRC | 14:57 | |
| *** zbr|lunch is now known as zbr | 15:04 | |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Correct info log on item report https://review.opendev.org/705709 | 15:11 |
| corvus | tobiash: ^ i was curious how often we're hitting gerrit-unable-to-merge errors these days, and found that it's difficult to tell :/ | 15:12 |
| corvus | tobiash: what's up with push/depends? do i just need to review the changes there? | 15:16 |
| *** zxiiro has joined #zuul | 15:21 | |
| *** Goneri has joined #zuul | 15:23 | |
| tobiash | corvus: I thing it's best to start with the spec (https://review.opendev.org/643309) | 15:25 |
| corvus | tobiash: k. i'm about halfway through now :) | 15:26 |
| tobiash | cool :) | 15:26 |
| tobiash | there is also an implementation for cdep (https://review.opendev.org/685354) and also for direct push | 15:26 |
| tobiash | direct push is https://review.opendev.org/#/q/topic:direct-push+(status:open+OR+status:merged) | 15:27 |
| mordred | neat! | 15:28 |
| tobiash | +3 on 705709 | 15:28 |
| corvus | i need to grab breakfast, biab. | 15:30 |
| *** felixedel has quit IRC | 15:30 | |
| tobiash | zuul-maint: I've seen quite a few zuul-build-image timeouts so I think we should increase the timeout: https://review.opendev.org/705689 | 15:32 |
| *** jamesmcarthur has quit IRC | 15:33 | |
| *** jamesmcarthur_ has joined #zuul | 15:33 | |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Add branch filter to tenant configuration https://review.opendev.org/705664 | 15:37 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Evaluate CODEOWNERS settings during canMerge check https://review.opendev.org/644557 | 15:41 |
| tristanC | tobiash: could this be related to the changes we did to the manage-ansible pip cache? | 15:46 |
| *** sshnaidm is now known as sshnaidm|afk | 15:46 | |
| tobiash | tristanC: this hasn't been merged yet | 15:46 |
| openstackgerrit | Jan Kubovy proposed zuul/zuul master: WIP: Store unparsed branch config in Zookeeper https://review.opendev.org/705716 | 15:47 |
| tristanC | tobiash: https://review.opendev.org/#/c/703126/ got merged | 15:47 |
| tobiash | ah yes, this definitely had an impact | 15:48 |
| tobiash | however the build time of that job was always close to the timeout so increasing the timeout makes still sense | 15:50 |
| corvus | tobiash: okay, left some comments. 2 big things: 1) i don't understand the configuration settings, but that should be straightforward to correct. 2) it seems like the spec is advocating *not* using zuul-push in order to be able to revert. is that your current thinking? that you don't want to push reverts? | 15:54 |
| tobiash | corvus: yes, imho force push should be avoided and can do more harm than it's useful in this case, but we should still combine it with direct push (at least for gerrit) to rule out merge algorithmic sources for failures | 15:55 |
| corvus | (there are 2 reasons for us to implement zuul-push: a) so we can push a revert if circular dependencies fail. b) so we can test the exact git shas that are merged. however, (b) requires significantly more work in that we have to be able to distribute those shas to all the workers, so it's a substantial redesign) | 15:55 |
| corvus | tobiash: you and felix are both saying "force push" and i keep saying "regular push" :) | 15:56 |
| corvus | er, not felix, simon, sorry | 15:56 |
| corvus | "push a revert" is not the same as "force-push a rewind" | 15:57 |
| corvus | i would advocate "push a revert" so that we don't have to deal with people being confused by head pointer rewinding | 15:57 |
| tobiash | hrm I'm not sure I really like that better | 15:58 |
| tobiash | pushing a revert also means we're pushing something untested | 15:58 |
| corvus | that's fair, it still has some problems, but nothing is perfect. :) | 15:58 |
| tobiash | and can confuse users even more | 15:58 |
| corvus | well, the idea would be to get back to the previous state, which presumably was tested | 15:59 |
| *** mhu has quit IRC | 15:59 | |
| tobiash | in my eyes the best compromise would be to do nothing there except a gate reset which will automatically enqueue the remaining parts again | 16:00 |
| clarkb | re codeowners, it would be nice to support that in gerrit too if possible | 16:01 |
| *** bolg has quit IRC | 16:01 | |
| tobiash | because pushing a revert can fail as well | 16:01 |
| tobiash | clarkb: how is that related to gerrit? | 16:01 |
| clarkb | tobiash: typically codeowners is per dir, so would give gerrit users fine grained control over approvals in subtress (but would require going outside of gerrit's structured approvals so might get messy) | 16:01 |
| clarkb | (this is why I say "if possible") | 16:02 |
| tobiash | clarkb: codeowners in github are enforced by github (and github doesn't give us the required api to check for that so we mimick that) | 16:02 |
| clarkb | ah | 16:03 |
| corvus | tobiash: at any rate, i'm getting the idea that direct-push may be harder with some of the newer drivers, and/or that people may be uncomfortable with enabling it. circumstances have changed since the last time we talked about this in depth. i think i'm willing to try the approach you suggest for now to see how things work in practice. hopefully with my logging fix we can get a better handle on how likely | 16:04 |
| corvus | that is to happen, and with a little bit of experience with circular deps in practice, we can understand whether we need to do something to handle reverts, or if crash-and-burn is okay. so perhaps we should shelve direct-push for now and just proceed with circular deps as best-effort. mordred, what do you think? | 16:04 |
| corvus | clarkb, tobiash: iiuc, codeowners is required for zuul to correctly predict github's behavior and is therefore user-invisible. extending it to gerrit would be a user-visible feature where we would be "enhancing" gerrit via zuul. | 16:05 |
| tobiash | corvus, clarkb: yes, exactly tht | 16:06 |
| corvus | pabelanger: have you had a chance to review the circular dep spec? | 16:08 |
| clarkb | ya for some reason I thought it was only bot enforced wtih github too | 16:08 |
| tobiash | corvus: I think direct push could still be useful on its own (optionally for certain drivers) in order to avoid cases where zuul is happy to merge and gerrit is not because of an automatically resolvable conflict | 16:08 |
| corvus | clarkb: probably used to be :) | 16:08 |
| corvus | tobiash: ok, so look at direct-push as something unrelated to circ-dep except that it can reduce the chances of errors because cgit > jgit? | 16:09 |
| tobiash | yes, that are my thoughts | 16:09 |
| corvus | tobiash: ok. that sounds reasonable. it's still a lot of code/complexity for that. do you have an idea how often we're hitting errors like that these days? (also, opendev's gerrit is very old, maybe jgit is better now?) | 16:11 |
| corvus | tobiash: what gerrit version are you running? | 16:11 |
| tobiash | all other sources of errors are network related and especially with github we saw cases where we got an error back from github (500) and the merge was still done. In this case no matter what we do we do the wrong thing. | 16:11 |
| tobiash | corvus: let me check | 16:11 |
| corvus | ESUCCESS: Error: Success! | 16:11 |
| tobiash | well, actually that's a more theoretical question, I cannot remember such a case in our system (or I didn't get reports) | 16:12 |
| tobiash | corvus: we have almost the same version of gerrit running ;) | 16:13 |
| tobiash | but most of our projects run on github nowadays | 16:13 |
| *** mattw4 has joined #zuul | 16:18 | |
| tobiash | especially the github 500 with still successfull merge pulled me to the conclusion that the safest thing we can do in case of a failed merge is a gate reset which will pull in current repo states and does the right thing moving forward. | 16:18 |
| tobiash | which is also the thing zuul currently does :) | 16:19 |
| openstackgerrit | Merged zuul/zuul master: Correct info log on item report https://review.opendev.org/705709 | 16:22 |
| openstackgerrit | Andy Ladjadj proposed zuul/nodepool master: add ebs-optimized support for aws provider https://review.opendev.org/705755 | 16:42 |
| *** noorul has joined #zuul | 16:42 | |
| noorul | ls | 16:42 |
| noorul | ls -la | 16:42 |
| noorul | Oops! sorry about that :( | 16:42 |
| pabelanger | corvus: I'll make some time this evening | 16:42 |
| noorul | ofosos: hi | 16:42 |
| *** jpena is now known as jpena|brb | 16:45 | |
| *** chkumar|rover is now known as raukadah | 16:48 | |
| sugaar | hi, in zuul.conf [connection "mysql"] where needs to be provided the dns name of the ip address of the database? is that "name"? or connection "mysql"? i tried searching on https://zuul-ci.org/docs/zuul/howtos/zuul-from-scratch.html?highlight=zuul%20conf but I couldn't find it | 16:51 |
| corvus | sugaar: it's here: https://zuul-ci.org/docs/zuul/reference/drivers/sql.html#attr-%3Csql%20connection%3E.dburi | 16:52 |
| corvus | the "name" is the name of the connection inside of zuul (that's how you add it to pipelines) but isn't used for connecting to the db. that's dburi. | 16:53 |
| sugaar | mm | 16:56 |
| sugaar | I think I understand, so in "dburi=mysql+pymysql://zuul:%(zuul_mysql_password)s@localhost/zuul" the "mysql" between "dburi=" and "+pymysq" is the dns that gets translated to the ip address right? | 16:57 |
| corvus | sugaar: actually it's the 'localhost' in that | 16:58 |
| corvus | sugaar: the mysql+mysql part tells sqlalchemy "use mysql dialect with the pymysql driver" | 16:58 |
| sugaar | so I didn't understand it that well XD | 16:59 |
| sugaar | I get it now | 16:59 |
| corvus | then it's "://user:password@host/database" | 16:59 |
| sugaar | thanks corvus | 16:59 |
| corvus | sugaar: you're welcome | 16:59 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Google logs: Link to index.html files https://review.opendev.org/705586 | 17:01 |
| mnaser | its kindof a bummer you dont really have a way to get a functional out of the box image with nodepool/dib without having to figure out how to get your own elements up and going. | 17:05 |
| clarkb | mnaser: I'm not sure that is true | 17:08 |
| mnaser | example: no python-tox | 17:08 |
| clarkb | mnaser: the ubuntu element for example shoudl give you a VM that is ssh able if you set ssh key metadata | 17:08 |
| mnaser | clarkb: in my case the tox-* jobs dont work because python-tox is not installed | 17:09 |
| mnaser | and ensure-tox installs it inside --user which is well not within the usual $PATH | 17:09 |
| mnaser | (it seems openstack-infra drops it inside the infra-package-needs) | 17:09 |
| corvus | huh, the purpose of ensure-tox was specifically to handle this case | 17:09 |
| mordred | yeah - if we need to update paths - then we should do that | 17:10 |
| mnaser | ensure-tox runs pip install with --user and therefore it installs into ~/.local/bin/tox | 17:10 |
| mordred | yeah. we should do whatever we need to do to make sure then that ~/.local/bin is in the path | 17:10 |
| mnaser | so unless i need to somehow rewire $PATH in my nodepool images, it doesnt really work | 17:11 |
| mordred | otherwise we have a bunch of useless roles in zuul-jobs | 17:11 |
| clarkb | mnaser: I think what corvus and mordred are getting at is that the zuul-jobs role(s) need to be updated instead | 17:11 |
| mnaser | i assumed that the jobs were ok because openstack-infra uses ensure-tox :) | 17:11 |
| mordred | yah. we should make sure those roles do the right thing | 17:11 |
| clarkb | because they install tox to handle this case and if that isn't working the bug is there (not in your image) | 17:11 |
| mordred | yup | 17:11 |
| mnaser | (sorry, i mean its baked in) | 17:11 |
| mnaser | i'm not sure what the best way to go about solving this though | 17:12 |
| mordred | which is to say - I agree with your assessment that this should all just work - and if it's not working it's a definite bummer :) | 17:12 |
| pabelanger | the issue is .local/bin isn't in PATH for centos (maybe fedora), but should be fine for ubuntu | 17:12 |
| mnaser | sounds like it might be a whack-a-mole thing | 17:12 |
| mnaser | .local/bin isnt in path for debian too (in my case) | 17:12 |
| mordred | mnaser: I think we can have ensure-tox (and similat things) drop ~/.local/bin into the path | 17:12 |
| pabelanger | so need to update bashrc | 17:12 |
| mordred | yes | 17:12 |
| mordred | that is what ensure-tox should do | 17:12 |
| mnaser | so ensure-tox should make sure ~/.local/bin is in path *if* it installs tox | 17:13 |
| mordred | and we need to switch between adding it to bashrc or bash_profile based on distro - but I've actually got a patch up doing that for a different thing | 17:13 |
| pabelanger | but yes, we still bake in tox for zuul.a.c today. haven't got around to fixing ensure-tox | 17:13 |
| mordred | mnaser: yes | 17:13 |
| pabelanger | I think the better fix, is to set tox_executable to full path, and avoid modifying PATH | 17:14 |
| corvus | pabelanger: why is that better? | 17:14 |
| mordred | https://review.opendev.org/#/c/704266/1/roles/ensure-python/tasks/pyenv.yaml | 17:14 |
| mordred | there's an example of modifying bash rc files | 17:14 |
| pabelanger | corvus: it is possible something in local/bin could conflict with distro, and break? | 17:15 |
| mordred | it's not 100% solid yet - but that would be the idea | 17:15 |
| pabelanger | the last time we tried to modify PATH, we had some fallout for executor side | 17:15 |
| corvus | mordred: would it make sense to make that a role and then have all the ensure-* roles include_role that? | 17:15 |
| mordred | corvus: yes | 17:15 |
| mnaser | so an add-to-path role or something | 17:16 |
| corvus | pabelanger: i don't think we'd be modifying anything on the executor, this would be on the worker nodes. | 17:16 |
| openstackgerrit | Andy Ladjadj proposed zuul/nodepool master: add ebs-optimized support for aws provider https://review.opendev.org/705755 | 17:16 |
| mordred | pabelanger: I think we can a) put ~/.local/bin at the end of path so it doesn't conflict with things BUT b) nothing shoudl be in local/bin other than things that jobs think they installed - so I'd like to see putting it at the front fail first and understand what the situation is | 17:16 |
| corvus | mordred: ++ | 17:17 |
| mordred | but in any case - we should definitely fix enxure-tox (and friends) so that mnaser doesn't have a sad about base images not working :) | 17:17 |
| corvus | mordred: is the path part of that patch gtg, or does it need more work? | 17:17 |
| pabelanger | yes, that was a poor example. I was trying to say, there are reasons for .local/bin, not in path by default (I don't know what that is), but if our job changes that, that is some drift on default installs. I'd be okay with some way to opt into it | 17:17 |
| mordred | corvus: I'm not sure - I wrote that as a morning drive-by - the distro to file mapping should be solid - but I need to test it more | 17:18 |
| clarkb | pabelanger: distros tend to not put random stuff in path, they put only what they control | 17:18 |
| mnaser | ok so i can try and refactor that into an add-path-to-path .. role? | 17:18 |
| mordred | mnaser: yah | 17:18 |
| corvus | mnaser: ++ | 17:18 |
| mnaser | anyone wanna volunteer come up with a better name | 17:19 |
| mnaser | heh | 17:19 |
| clarkb | if the names are case sensitive: add-path-to-PATH | 17:19 |
| corvus | mnaser: we can write a special test job for this that does an "apt-get remove tox" or whatever to make sure this is all tested in zuul-jobs | 17:19 |
| mordred | pabelanger: it's a good idea from a distro side to not have local.bin in path for safety. but for us, we're the user, so if we're installing software into ~/.local, I think it's fair for us to add it to path | 17:19 |
| tobiash | we could also link the tox binary into /usr/bin/ | 17:19 |
| mordred | I agree we should _not_ do that if we don't install things | 17:19 |
| clarkb | tobiash: that is what I do | 17:19 |
| corvus | mnaser: if you can get patches up for the role and ensure-tox, i'll write the test job to verify it all. | 17:19 |
| clarkb | tobiash: I install stuff in a venv then ln -s them to a location in my path | 17:20 |
| mordred | the nice thing about --user and adding to path is that it means you don't actually need root for ensure-tox to work | 17:20 |
| clarkb | (its actually ~/bin but same idea) | 17:20 |
| mnaser | corvus: ok will do | 17:20 |
| mordred | which, although we have it - is still nice in terms of widest applicability for people running in environments where root may be more scarce - and nothing about installing tox should require root - and path and ~/.bashrc exist to allow stuff like this - so there's no reason to restrict ourselves from using them | 17:20 |
| mnaser | i can also do a depends-on check | 17:21 |
| mnaser | because our zuul is connected to opendev's gerrit | 17:21 |
| mnaser | \o/ | 17:21 |
| pabelanger | we should do what we did in ensure-twine: https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/ensure-twine/tasks/main.yaml#L16 but using tox_executable as fact | 17:21 |
| corvus | mnaser: ++ | 17:21 |
| pabelanger | that results is the same issue, but twine works | 17:21 |
| mnaser | oh actually what do folks thing about setting a fact inside ensure-tox ? | 17:22 |
| corvus | mnaser: it won't persist between plays | 17:22 |
| mnaser | ah yes | 17:22 |
| corvus | and we want to be able to have ensure-tox in pre-run and tox in run | 17:22 |
| corvus | pabelanger: yes, that approach is the other option. i think we should settle on one for all the ensure-* roles though. | 17:23 |
| corvus | if we add-to-path then the end-use roles like tox or twine can be simpler. | 17:23 |
| *** jpena|brb is now known as jpena | 17:24 | |
| mnaser | corvus, mordred ok with 1 change that includes add-to-path and updating ensure-tox or rather me split em? | 17:26 |
| corvus | wfm | 17:27 |
| pabelanger | another option, is update zuul user at DIB time, to have correct PATH | 17:27 |
| clarkb | pabelanger: I think we should avoid relying on image build specifics | 17:27 |
| *** noorul has quit IRC | 17:28 | |
| clarkb | if the jobs can ssh in they should be able to run | 17:28 |
| clarkb | the reason for that is wedon't require dib | 17:30 |
| clarkb | and don't support it or other automated image building with other provider backends | 17:30 |
| pabelanger | good point | 17:30 |
| *** michael-beaver has quit IRC | 17:33 | |
| *** evrardjp has quit IRC | 17:33 | |
| *** evrardjp has joined #zuul | 17:34 | |
| openstackgerrit | Mohammed Naser proposed zuul/zuul-jobs master: ensure-tox: add ~/.local/bin to $PATH https://review.opendev.org/705782 | 17:35 |
| mnaser | corvus, mordred ^ i'll test this locally too | 17:35 |
| *** jamesmcarthur_ has quit IRC | 17:35 | |
| mordred | mnaser: woot! | 17:36 |
| clarkb | mnaser: I don't think you need the distro switch | 17:36 |
| clarkb | that behavioris bash defined not distro defined | 17:36 |
| clarkb | also it won't apply on command or shell tasks by default | 17:36 |
| clarkb | (because they use sh instead which may not be bash) | 17:36 |
| *** jamesmcarthur has joined #zuul | 17:37 | |
| mnaser | clarkb: what do you suggest instead? | 17:38 |
| clarkb | mnaser: I think drop the distro checks and instead write a .shrc and source that in .bashrc | 17:40 |
| mordred | clarkb: I'm pretty certain you need the distro checks | 17:40 |
| clarkb | mordred: why? bash the program determines what it sources | 17:41 |
| clarkb | not the distros | 17:41 |
| mordred | because just sourcing the .shrc in .bashrc isn't gauranteed to workk on the platforms where .bashrc doesn't get sourced automatically | 17:41 |
| mordred | clarkb: I'm pretty certain that's not 100% true | 17:41 |
| mordred | the distros go out of their way to make different default choices | 17:41 |
| pabelanger | what is user is not using bash as shell in 705782? | 17:42 |
| pabelanger | if* | 17:42 |
| clarkb | reading more .profile is the sh login setup. .bash_profile is login setup for bash and it falls back to .profile if .bash_profile is not present | 17:42 |
| clarkb | rc is for interactive sessions that are not a login | 17:42 |
| clarkb | that is bash behavior not distro behavior | 17:42 |
| *** jamesmcarthur has quit IRC | 17:42 | |
| clarkb | mordred: specific to homedir files | 17:42 |
| pabelanger | maybe we need a new role, setup-zuul-user. that base-minimal jobs can include, to problem setup user things, like shell. Having ensure-foo randomily modify bash doesn't seem clean | 17:43 |
| clarkb | mordred: what the distros do in /etc is probably weird | 17:43 |
| mordred | ok. I'd be happy to be wrong about that of course | 17:43 |
| clarkb | I think this means that .profile might work as long as we get a login shell | 17:43 |
| mordred | pabelanger: I do not believe we support users not using bash at all | 17:43 |
| clarkb | as that would cover bourne and bash | 17:44 |
| pabelanger | mordred: do you know what windows uses? | 17:44 |
| mordred | pabelanger: powershell? | 17:44 |
| mordred | I don't expect these to work without modication on windows :) | 17:44 |
| pabelanger | mostly thinking of tobiash, but have no idea is they use these jobs | 17:45 |
| mordred | clarkb: so I think the question is whether we get logic shells for each task | 17:45 |
| clarkb | mordred: yup | 17:45 |
| mordred | pabelanger: yeah - it's a good thing to think about - but I think in general most of these already dont' even come close to working on windows | 17:45 |
| *** jamesmcarthur has joined #zuul | 17:46 | |
| pabelanger | ack, good to know | 17:46 |
| mordred | pabelanger: that said - I actually disagree with you about a thing for base-minimal ... because of a point you made earlier :) I think we should only modify path if we actually install something to a new location - and we won't know we've done that until we install or don't install something | 17:46 |
| clarkb | and if we don't get a login shell I would write .shrc and .bashrc and source one from the other | 17:46 |
| mordred | so - although it does touch more things, I think we can modify things less frequently if we just modify things at point of use | 17:47 |
| pabelanger | mordred: yah, I think the thing I am worried about, is modifying PATH on CI jobs, for tox. Because we install it from RPM and not pip, we don't actually need that on zuul.a.c now ( I just remember we stopped using pip install tox in dib). So, it would nice to only modify if pip install is invoked | 17:48 |
| mordred | pabelanger: yes - I think we should absolutely do that | 17:48 |
| clarkb | pabelanger: the job won't install it into the local/bin dir if you've installed it globally | 17:48 |
| clarkb | and mnaser mentioned above only doing the path update if the install happens | 17:48 |
| mordred | yeah - and if we don't hit the "I need to install this" condition, then we shoudlnt' modify path | 17:48 |
| mordred | yeah | 17:49 |
| pabelanger | clarkb: 705782 needs to be upgraded, we modify PATH regardless | 17:49 |
| clarkb | pabelanger: it needs updating anyway since that won't work with default command and shell tasks on a bunch of systems (including ubuntu) | 17:49 |
| mordred | pabelanger: ++ | 17:49 |
| *** jamesmcarthur has quit IRC | 17:51 | |
| *** jamesmcarthur has joined #zuul | 17:51 | |
| *** igordc has joined #zuul | 17:53 | |
| clarkb | this sh stuff is bringing back memories helping univeristy students set up their computing environments | 17:53 |
| clarkb | related, you want the root shell to be statically compiled. This is why a lot of systems still use not bash for sh. | 17:55 |
| clarkb | If you want to discover why this is important do a make world on freebsd after setting root's shell to bash from csh | 17:55 |
| clarkb | mordred: ^ did you hear about that? chromakode discovered that the hard way working on our drizzle db server | 17:56 |
| clarkb | its funny to think back on but at the time that was pain | 17:56 |
| mnaser | nothing is easy :< | 17:58 |
| corvus | not even goat farming | 17:58 |
| pabelanger | but, goat simulator is fun | 17:59 |
| openstackgerrit | Andy Ladjadj proposed zuul/nodepool master: add ebs-optimized support for aws provider https://review.opendev.org/705755 | 17:59 |
| *** tosky has quit IRC | 18:01 | |
| *** jamesmcarthur has quit IRC | 18:12 | |
| *** jamesmcarthur has joined #zuul | 18:13 | |
| *** jamesmcarthur has quit IRC | 18:19 | |
| *** jamesmcarthur has joined #zuul | 18:25 | |
| fungi | if goat farming doesn't work out, there's always long-haul trucking. something alluring about life on the open road | 18:33 |
| *** jamesmcarthur has quit IRC | 18:34 | |
| *** jamesmcarthur has joined #zuul | 18:35 | |
| *** adamw has quit IRC | 18:36 | |
| *** jpena is now known as jpena|off | 18:37 | |
| *** jamesmcarthur has quit IRC | 18:40 | |
| *** jamesmcarthur has joined #zuul | 18:40 | |
| mnaser | poo | 18:41 |
| mnaser | "Destination /home/zuul/.bash_profile does not exist !" under debian | 18:41 |
| mnaser | maybe i should add debian to fedora/ubnutu list mordred ? | 18:41 |
| openstackgerrit | Mohammed Naser proposed zuul/zuul-jobs master: ensure-tox: add ~/.local/bin to $PATH https://review.opendev.org/705782 | 18:44 |
| *** jamesmcarthur has quit IRC | 18:48 | |
| mnaser | WARNING: The scripts tox and tox-quickstart are installed in '/home/zuul/.local/bin' which is not on PATH. | 18:48 |
| mnaser | hmm | 18:48 |
| mnaser | thats with the depends-on patch above | 18:48 |
| mnaser | i think clarkb theory was correct bc bash is not used? | 18:48 |
| fungi | why not append to ~/.profile? | 18:49 |
| mnaser | clarkb: what files should i ideally update? it seems like you had a grasp on things | 18:49 |
| clarkb | mnaser: I left a comment on the change | 18:49 |
| clarkb | but ya .profile should be used by both sh and bash. If we don't get login shells then update .shrc and .bashrc (and have one source the other) | 18:50 |
| mnaser | to avoid touching a lot of pieces, i wonder if i can start by seeing if it works with .profile only initially | 18:51 |
| fungi | long story short, making sure things happen for specific users on login to various distros is nontrivial | 18:52 |
| fungi | but yeah, i recommend trying ~/.profile next | 18:52 |
| * fungi is still catching up | 18:52 | |
| mnaser | (i can only try on debian right now fwiw) | 18:52 |
| openstackgerrit | Mohammed Naser proposed zuul/zuul-jobs master: ensure-tox: add ~/.local/bin to $PATH https://review.opendev.org/705782 | 18:52 |
| clarkb | the breakdown is .profile and .bash_profile are sourced by login shells. .shrc and .bashrc are sourced by non login shells. If .bash_profile does not exist then .profile will be sourced by bash, but that is not the case with .shrc and .bashrc | 18:54 |
| clarkb | so we have to account for login vs not (depending on how ansible shell tasks work) and whether or not files exist | 18:54 |
| clarkb | definitely start by trying .profile and if it works its probably simplest. If not then .shrc + .bashrc may be required | 18:55 |
| mnaser | ^ that one didnt work | 18:55 |
| fungi | poop | 18:57 |
| mnaser | (by that one, means adding into .profile and this is on debian buster) | 18:58 |
| clarkb | probalby because it ships a default .bash_profile | 18:58 |
| clarkb | ? | 18:58 |
| clarkb | though the sh shell tasks should pull in .profile if they use a login shell so actually probably because not a login shell | 18:59 |
| mnaser | i run debian sid on my laptop which i imagine isnt far away from buster | 19:00 |
| mnaser | and uh | 19:00 |
| mnaser | http://paste.openstack.org/show/789129/ | 19:00 |
| *** mattw4 has quit IRC | 19:00 | |
| mnaser | i am confused now | 19:00 |
| clarkb | mnaser: probably not a login shell | 19:00 |
| clarkb | that file is only sourced if it is a login shell | 19:01 |
| mnaser | ah ok | 19:01 |
| *** mattw4 has joined #zuul | 19:02 | |
| mnaser | i'm starting to feel discovering the tox location might be more robust | 19:02 |
| mnaser | whats annoying is .shrc doesnt exist in debian it seems | 19:03 |
| *** michael-beaver has joined #zuul | 19:04 | |
| fungi | not existing isn't the same as not getting used | 19:05 |
| mnaser | well in that case i'd have to make 2 branches of code | 19:06 |
| fungi | if we append then that should cover cases where it exists as well as those where it doesn't (as long as our append action will create nonexistent files) | 19:06 |
| mnaser | one that makes sure the file exists / at least empty, and the other is lineinfile | 19:06 |
| mnaser | oh lineinfile has create | 19:06 |
| mnaser | "If specified, the file will be created if it does not already exist." | 19:06 |
| fungi | ahh, so ansible doesn't have something like `echo ... >> foo` where foo is appended to when it exists but created when it doesn't? | 19:06 |
| mnaser | "By default it will fail if the file is missing." | 19:06 |
| fungi | aha, so it does | 19:07 |
| fungi | just needs an extra toggle | 19:07 |
| mnaser | ok so bashrc and shrc | 19:07 |
| fungi | add it to .shrc, and then add a line to .bashrc to source .shrc | 19:07 |
| fungi | or optionally add it to both .shrc and .bashrc, since adding an entry to .bashrc to source .shrc would need some idempotency checks | 19:08 |
| mnaser | yeah i think adding it to both might be easier and cleaner | 19:08 |
| mnaser | so i dont end up sourcing other things | 19:08 |
| fungi | to avoid sourcing contents of .shrc multiple times | 19:08 |
| openstackgerrit | Mohammed Naser proposed zuul/zuul-jobs master: ensure-tox: add ~/.local/bin to $PATH https://review.opendev.org/705782 | 19:08 |
| mnaser | so nice to test it in another repo on another gerrit :) | 19:09 |
| mnaser | ah i forgot something | 19:09 |
| openstackgerrit | Mohammed Naser proposed zuul/zuul-jobs master: ensure-tox: add ~/.local/bin to $PATH https://review.opendev.org/705782 | 19:09 |
| pabelanger | mnaser: we should only update PATH, if pip install is done. | 19:10 |
| mnaser | pabelanger: yep, i plan to do that once i get it work minimally :p | 19:10 |
| pabelanger | ++ | 19:10 |
| mnaser | sigh | 19:12 |
| mnaser | that didnt even work | 19:12 |
| mnaser | btw we should totally install using pip3 first before pip but thats a whole another thing | 19:13 |
| mnaser | so the shell command notes: "It is almost exactly like the command module but runs the command through a shell (/bin/sh) on the remote node." | 19:27 |
| mnaser | so it means command doesnt actually use a shell at all? | 19:27 |
| pabelanger | yah, that is different with shell task | 19:27 |
| mnaser | looks like it actually uses subprocess.Popen | 19:30 |
| pabelanger | if tox role using command? | 19:30 |
| pabelanger | err | 19:31 |
| pabelanger | is* | 19:31 |
| pabelanger | I wonder | 19:31 |
| pabelanger | if we do use set_fact for full path | 19:32 |
| mnaser | yes it is | 19:32 |
| pabelanger | then set cacheable: true | 19:32 |
| mnaser | (using command) | 19:32 |
| pabelanger | it should persist over playbooks | 19:32 |
| pabelanger | because, we have fact cache | 19:32 |
| pabelanger | maybe that is better way | 19:32 |
| mnaser | ya either that or just have a actual `environment` in every single place.. | 19:34 |
| mnaser | (that we have command) | 19:34 |
| mnaser | and we append the path there | 19:34 |
| pabelanger | tox_environment should work | 19:35 |
| pabelanger | but, then you still have fact issue | 19:35 |
| mnaser | yeah you'd have to carry the fact that you have a local install | 19:36 |
| pabelanger | If cacheable works, I think that is miminal change | 19:37 |
| mnaser | if cacheable works then i think we can just fact of tox_executable | 19:37 |
| pabelanger | agree | 19:38 |
| openstackgerrit | Mohammed Naser proposed zuul/zuul-jobs master: ensure-tox: save tox_executable fact https://review.opendev.org/705816 | 19:41 |
| mnaser | ..testing here | 19:41 |
| *** avass has quit IRC | 19:43 | |
| openstackgerrit | Mohammed Naser proposed zuul/zuul-jobs master: ensure-tox: save tox_executable fact https://review.opendev.org/705816 | 19:46 |
| *** armstrongs has joined #zuul | 19:46 | |
| mnaser | mordred: i see you did work on pyenv, i'm considering building debian python packages for different python versions (for example, buster doesnt have 3.6 but has 3.7...) | 19:53 |
| mnaser | i'm not sure if there's been discussions around how to ship python quickly independent of the distro | 19:54 |
| fungi | there were a bunch of ideas tossed about last week before fosdem | 19:54 |
| mordred | mnaser: I don't know about quickly ... but I've been extremely pleased as a pyenv user myself | 19:55 |
| fungi | one challenge is going to be caching the interpreter builds somewhere so that they're not recompiled from source in every job | 19:55 |
| clarkb | I could never get pyenv to work on suse fwiw | 19:55 |
| clarkb | (becuse of distro patching) | 19:55 |
| clarkb | python upstream will take patches to build python on weird random platforms but not linxu distros apparently | 19:55 |
| mordred | well - it works nicely on debuntu and osx | 19:55 |
| mnaser | mordred: i think the issue here is building python on every commit | 19:55 |
| mnaser | if we cache whatever pyenv builds across builds then i'm all for it | 19:56 |
| *** armstrongs has quit IRC | 19:56 | |
| mordred | yeah. I think it would be pretty possible to cache the python builds from pyenv - they go into directories | 19:56 |
| clarkb | mordred: ya its a problem due to how suse packages its curses stuff | 19:56 |
| mordred | mnaser: you know - I have an idea | 19:56 |
| mnaser | a cache would totally be something i know a lot of users be interested in | 19:56 |
| fungi | something like the wheel build jobs opendev uses, but cache tarballs of the pyenv result which a job can fetch and splat out into its filesystem | 19:58 |
| clarkb | one thing you need to be careful of is the linkking to all those external libs | 19:58 |
| clarkb | the cache would have to be per distro release arch | 19:58 |
| fungi | it would need to be per-platform, yep | 19:58 |
| mordred | fungi: yes | 19:58 |
| clarkb | fungi: ya | 19:58 |
| mordred | clarkb: and yes | 19:58 |
| mnaser | could even be a zuul job really | 19:58 |
| mordred | so - my thought is - do it like the cache tarball idea fungi just had - but splat the tarball into an otherwise empty docker image | 19:58 |
| mordred | so that we get transit and region caching easily | 19:59 |
| mordred | then have the untarring be a docker export | 19:59 |
| clarkb | doesn't dockerhub already provide python version specific iamges? | 19:59 |
| mnaser | ^ | 19:59 |
| mordred | yes - but those are images | 19:59 |
| mordred | they have an install of python baked into the image | 19:59 |
| mordred | that's not what we want here - we want a python that's built for the target host system | 19:59 |
| mordred | (because of the shared libs thing clarkb mentioned) | 19:59 |
| clarkb | right I guess if using docker why not just run a container then | 20:00 |
| mordred | right - because that adds a bunch of other things into the mix - you'd need to bind-mount the world, then you'd be using the libs from the image and not the host | 20:00 |
| mordred | I'm just suggesting using docker as a transport mechanism for the tarball content - not as an actual runnable container image | 20:00 |
| mordred | since it's already addressable and we already have per-region caching mirrors set up | 20:01 |
| clarkb | ya I guess my grip with that is docker is pretty terrible at http | 20:01 |
| mordred | BUT - we could also just use tarballs - they won't be that big | 20:01 |
| clarkb | I would rather use anything else to get http caching :) | 20:01 |
| mordred | heh | 20:01 |
| clarkb | we have to live with it for docker itself though | 20:01 |
| tristanC | fwiw i heard nixos and cachix is a good solution to distribute and share such things too. | 20:02 |
| mordred | yeah - mostly just we're already dealing with it - what's another couple of images | 20:02 |
| fungi | if we do it like our wheels, we'd distribute via afs and cache with apache | 20:02 |
| fungi | er, cache with afs, serve with apache | 20:02 |
| mordred | fungi: that's a very good point | 20:02 |
| clarkb | tristanC: fwiw you can run nix without nixos | 20:02 |
| clarkb | tristanC: and that may be more applicable here, except that doesn't use the host libs like mordred wants | 20:02 |
| clarkb | (all the software would be in nix managed space isntead) | 20:02 |
| fungi | though if we do it to dockerhub we could still cache with apache | 20:02 |
| tristanC | clarkb: yeah, that's how i'm using it, you don't even need to be root to install things | 20:03 |
| mnaser | the other dockerhub advantage is well.. other users can take advantage of it | 20:03 |
| mordred | clarkb, tristanC: we'd also have to figure out region mirror/cachine and whatnot | 20:03 |
| mnaser | aka i dont need to build them too | 20:03 |
| mordred | mnaser: you could also take advantage of http hosted tarballs | 20:04 |
| mordred | in either case we'd need an untar step | 20:04 |
| mnaser | except dockerhub can sustain the traffic and our infra.. maybe not =P | 20:04 |
| mordred | uh | 20:04 |
| mnaser | but maybe i can just borrow opendev's mirrors for my own zuul | 20:04 |
| mordred | I'd go the other way | 20:04 |
| mnaser | it doesnt run far away | 20:04 |
| mnaser | =P | 20:04 |
| clarkb | mnaser: see earlier mention about dockerhub http woes | 20:04 |
| clarkb | mnaser: it fails a lot | 20:04 |
| mordred | mnaser: you can also run your own afs client and run your own mirror | 20:05 |
| mnaser | that sounds like work monty | 20:05 |
| mordred | our afs volumes are public | 20:05 |
| mordred | mnaser: :) | 20:05 |
| mnaser | but that might be something that i could look into | 20:05 |
| clarkb | our caching has significantly improved the reliability of dockerhub by not actually requesting data from them often :) | 20:05 |
| mordred | well - in any case - I think the first step is some zuul jobs that do a crap-load of pyenv install steps and a bunch of tarring | 20:06 |
| clarkb | the other thing to consider is if using pyenv version on $distro is worthwhile for testing | 20:06 |
| clarkb | seems like python gets patched a lot of various distros so using the distro python is useful if you expect people to deploy using a distro python | 20:06 |
| mordred | clarkb: I think it's a better choice than distro python myself - but I've gone off the deepend | 20:06 |
| clarkb | mordred: right but you aren't your users | 20:07 |
| mordred | clarkb: well - but the issue at hand here is testing something on an os that doesn't have a particular python version | 20:07 |
| clarkb | if users are going to use ubuntu or rhel python then testing that is useful. If they use container images from dockerhub then testing those container images is useful | 20:07 |
| mordred | I don't think this is useful for versions of python that do exist in the distro | 20:07 |
| clarkb | mordred: right and that is only useful if we expect users to do that in the wild | 20:07 |
| mordred | yeah | 20:07 |
| mordred | mnaser: what is driving your desire for using a python that's not in a given distro? | 20:08 |
| clarkb | otherwise I guess we can say "we test pythonx.y" but then when peopel deploy it on rhel with all the patching there it still may not work | 20:08 |
| fungi | i think that software which wants to test "does this work with the default python3 on ubuntu 18.04" should use the default python3 on ubuntu 18.04, and folks who want to test "does this work with upstream python3.6" should use a pyenv build of 3.6 on whatever platform | 20:08 |
| mordred | clarkb: to be fair - I'd be fine with that and would be happy pointing the finger ata . distro that has broken python | 20:08 |
| fungi | those are not the same thing, and we ought to make both possible | 20:08 |
| mnaser | mordred: in this case, a zuul user wans to run their unit tests for py35, py36, py37 .. the ensure-python role does apt install python$version-dev .. python3.6 doesnt exist in debian buster in this case | 20:08 |
| mordred | fungi: ++ | 20:08 |
| openstackgerrit | Merged zuul/zuul master: Allow skipping child jobs from paused job again https://review.opendev.org/704758 | 20:08 |
| fungi | even if it means compiling python3.6 on ubuntu 18.04 which already provides a packaged python3.6 interpreter | 20:09 |
| fungi | it's a conscious decision for one or the other | 20:09 |
| mordred | the unittest case is a really good example where caring about the distro version of python is less likely the target | 20:09 |
| clarkb | right I'm not opposed to making these possible but I think this is a subtle thing that few users end up considering | 20:09 |
| clarkb | and it is actually important when you deploy to production | 20:09 |
| mordred | you want python 3.5, 3.6 - you don't necessarily care about the distro the node is running | 20:09 |
| fungi | (disclaimer: i test my personal software on compiled upstream python versions independent of my system python packages) | 20:09 |
| mordred | fungi: yeah - you and me both | 20:10 |
| mnaser | yes, in this case, the user doesnt care about the distro, they want "does unit tests run on python X" | 20:10 |
| mordred | yah. | 20:10 |
| mordred | so it seems like a worthwhile thing to be able to provide people | 20:10 |
| clarkb | for that use case I think I'm with tristanC actually | 20:10 |
| mordred | even if there are some caveats for when people should or shouldn't use it | 20:10 |
| clarkb | tools like nix exist for that specific problem space | 20:10 |
| fungi | clarkb: it's an argument i've gone round and round with the openstack tc on when the pti got rewritten to be specific to python versions independent of distro platforms | 20:11 |
| mordred | clarkb: they might - but every time I've tried to use nix I've fallen down a rabbit hole of complexity | 20:11 |
| mordred | I like pyenv because it's so stinking simple | 20:11 |
| mordred | but - that's just me | 20:11 |
| clarkb | nix exists to allow anyone with a linux install (and not even root) to grab software of arbitrary and possibly conflicting versions and run it | 20:11 |
| clarkb | mordred: the issue is pyenv doesn't solve this for ruby or perl or golang or rust | 20:11 |
| fungi | i still just make altinstall in a git clone of cpython | 20:11 |
| mnaser | i'm almost tempted to make a custom dib image with a bunch of python versions and use a psecific nodeset for that | 20:12 |
| fungi | with various tags checked out for the versions i want | 20:12 |
| mnaser | buuuuut that seems like it solves things for me and not everyone | 20:12 |
| clarkb | mordred: python is a subset of the problem (and we hit it first because we doa lot of python) | 20:12 |
| mordred | clarkb: indeed | 20:12 |
| mordred | well - I will try nix again and see if it pisses me off less this time | 20:12 |
| mordred | I WANT to like it | 20:13 |
| mordred | just so far I have not | 20:13 |
| fungi | it seems like docker images are the way folks test arbitrary language ecosystem interpreter versions these days | 20:13 |
| mordred | yup | 20:13 |
| mordred | "directory /nix does not exist; creating it by running 'mkdir -m 0755 /nix && chown mordred /nix' using sudo" | 20:13 |
| fungi | for example, i can no longer compile python3.4 on my dev workstation due to requiring a too-old libssl | 20:13 |
| mnaser | yeah.. its really easy to use docker | 20:14 |
| mordred | this is why I love the docker.io/library/python base images | 20:14 |
| mnaser | also most ci tools let you use docker images to run jobs inside btw | 20:14 |
| mnaser | which i notice is a huge driver for a lot of poeple :\ | 20:14 |
| clarkb | mnaser: so does zuul fwiw | 20:14 |
| mnaser | but i mean in a much more native way | 20:14 |
| fungi | we just don't make a big deal about it because it's just a command line either way | 20:14 |
| fungi | but sure, maybe we should advertise it more visibly | 20:15 |
| clarkb | right its the difference between "being able to do it" and "being forced to do it" | 20:15 |
| fungi | "we let you call `docker run ...` | 20:15 |
| fungi | right, a lot of recent ci systems are built to expect and *only* support using docker images | 20:15 |
| mnaser | yep, i'm just saying it is an easy "way out" for users .. for example, this python issue would be simplified for the use case of running containers | 20:16 |
| tristanC | mordred: nix can be a bit complex, but i find it quite simple for things like `nix-env -i python3.7` . then with a common `ensure-nix` role, you have a generic solution to the requirements installation problem | 20:16 |
| fungi | which of course makes the decision on whether or not to do something in docker a lot easier when you | 20:16 |
| mnaser | s/running containers/running unittests/ | 20:16 |
| fungi | 're using one of those | 20:16 |
| clarkb | mnaser: right so another option is to write zuul jobs that act in that manner | 20:16 |
| clarkb | mnaser: then all users need to do is supply an image name and command | 20:16 |
| mnaser | yep | 20:16 |
| clarkb | we can make that simple for users too | 20:16 |
| clarkb | (In the case of the k8s pod provider I think that is basically how it works too right?) | 20:17 |
| mnaser | not really because the nodesets/labels are defined inside nodepool | 20:17 |
| clarkb | ah | 20:17 |
| clarkb | so ya would still be a job level thing then | 20:18 |
| tristanC | mordred: fwiw, i'm using nix in a container, here is my recipe: https://github.com/podenv/hub/blob/master/runtimes/nixos/create.dhall#L19 | 20:18 |
| mnaser | mordred, clarkb, fungi, tristanC: for what it's worth, i think at the end of the day, our users dont care how python shows up, as long as it shows up quickly (no long build times) and the right version. | 20:20 |
| clarkb | mnaser: right that was my earlier concern :) | 20:21 |
| mnaser | we're the ones who care the most about the how, so if nix is a solution, i dont mind its extra complexity if it solves the issue | 20:21 |
| clarkb | mnaser: because distros publish python they call pythonX.Y that is not compatibile with pythonX.Y | 20:21 |
| clarkb | (that subtle thing means it can be important that you test the right pythonX.Y) | 20:21 |
| mnaser | i think in that case we'd have like tox-py3-ubuntu-bionic or whatever, and tox-py37 | 20:22 |
| fungi | it's "better" these days, but i'm still scarred by red hat backporting python3.1 features into their "python2.6" | 20:22 |
| mnaser | and ubuntu bionic py3 means whatever it gets there | 20:22 |
| fungi | mnaser: sounds great to me, yep | 20:23 |
| openstackgerrit | Clark Boylan proposed zuul/zuul-jobs master: Use unique loop vars to avoid conflicts https://review.opendev.org/705337 | 20:23 |
| clarkb | mnaser: ya I think being explicit is a good way to address that | 20:24 |
| mordred | clarkb, mnaser: ++ | 20:26 |
| *** felixedel has joined #zuul | 20:29 | |
| mnaser | now going a few steps back | 20:29 |
| *** felixedel has quit IRC | 20:29 | |
| mnaser | mordred, pabelanger, corvus, clarkb: https://review.opendev.org/#/c/705816/ resolved it for me in a depends-on change | 20:29 |
| mnaser | its not ideal but because command uses Popen it's hard to pass the envirnment on | 20:30 |
| *** hashar has joined #zuul | 20:31 | |
| mordred | mnaser: how does tox_executable persist everywhere it needs to? | 20:31 |
| pabelanger | cachable: true | 20:31 |
| pabelanger | should put it in fact cache | 20:31 |
| mordred | OH | 20:31 |
| mnaser | ^ yep | 20:31 |
| mordred | TIL | 20:31 |
| mnaser | and i have it working here, i can link in private if anyone wants to see the run | 20:31 |
| mordred | honestly - I think that's a nice solution to that specific issue - especially given command and environments | 20:32 |
| mnaser | ya the alternative was going and adding environment to a whole bunch of other calls | 20:32 |
| mordred | I'm a little sad we can't extend PATH sanely though | 20:32 |
| mordred | but - that's life | 20:32 |
| mnaser | same, it seems so .. trivial | 20:32 |
| clarkb | is cache based on inventory name or real name? | 20:33 |
| clarkb | because that may break if based on inventory name since a lot of jobs share the inventory name | 20:33 |
| clarkb | (I think it is inventory name based on ansible cache behavior we have seen on bridge.o.o) | 20:34 |
| clarkb | or is that all contained in the per build stuff? | 20:34 |
| clarkb | if its per build then ++ good fix | 20:34 |
| hashar | hi ! | 20:36 |
| mnaser | clarkb: i think this is all in the per build stuff / inside the bwrap in zuul-executor | 20:37 |
| mnaser | https://opendev.org/zuul/zuul/src/branch/master/zuul/executor/server.py#L352-L354 | 20:38 |
| corvus | mnaser, mordred: wow, there's a few hundred lines of scrollback and 705816 isn't what we talked about earlier. can you tldr why we're changing the approach? | 20:43 |
| mnaser | corvus: tl;dr is none of the solutions involving adding export PATH worked because command uses subprocess.Popen which reads environment from the 'parent' process | 20:44 |
| corvus | got it, thx. | 20:44 |
| mnaser | i tried .profile, .bashrc/.shrc, .bash_profile, etc | 20:44 |
| mnaser | so using cacheable makes the fact survive multiple tasks/plays so it lives the entire lifetime of the job (as this stuff goes inside the bwrap of zuul-executor) | 20:45 |
| mnaser | and its working for me (i tested it with a depends-on) | 20:45 |
| corvus | mnaser: one more q: how does this compare to pabelanger's earlier suggestion of doing a test in the run role (like twine) instead of using a cacheable fact? | 20:45 |
| mnaser | corvus: i inspected how twine did it. it executes a `which` and uses the output which i suspect wont be much more productive if we dont have this inside $PATH | 20:46 |
| mnaser | we *could* work around by running which with an overridden $PATH, and use that output | 20:47 |
| corvus | kk, and i see pabelanger is on board. 705816 lgtm as is, just wanted to make sure i understood | 20:47 |
| corvus | mordred: ^ can you leave a vote there too? | 20:47 |
| mnaser | alright i'm heading off (eu tz), i'll check in buffer tomorrow if there's anything developed since then :) | 20:49 |
| * mnaser & | 20:49 | |
| mordred | corvus: yeah - I think it's great (although I'm sad there isn't a better way to manipulate PATH more systemically) | 20:58 |
| corvus | huzzah, consensus! | 21:00 |
| openstackgerrit | Merged zuul/zuul-jobs master: ensure-tox: save tox_executable fact https://review.opendev.org/705816 | 21:11 |
| *** adamw has joined #zuul | 21:17 | |
| *** zxiiro has quit IRC | 21:20 | |
| fungi | mordred: well, there is... adjust files in /etc | 21:38 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Add disallowed-labels tenant option https://review.opendev.org/705856 | 21:48 |
| corvus | i'm thinking about how to automate deployment of gerrit's zuul, and i think that having a dedicated tenant with the option of requesting a special node with service account access to the k8s cluster may be useful, but i don't want to have to exhaustively list all the allowed nodes for the main tenant. I'd rather just put "disallow-labels: deployment.*" on the main tenant and be done with it. so i wrote | 21:50 |
| corvus | that change ^ | 21:50 |
| corvus | tristanC: ^ you have experience with allowed-labels i think? | 21:51 |
| mordred | fungi: well - I'm not convinced that would do it - we keep persistent connections open ... so I'm honestly not sure which files get read when and what we'd need to do to re-read them | 21:53 |
| openstackgerrit | Merged zuul/zuul-jobs master: upload-afs: rename to upload-afs-roots; add afs-upload-synchronize https://review.opendev.org/705368 | 21:56 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: web: link to index.html if index_links is set https://review.opendev.org/705585 | 21:59 |
| *** sgw has quit IRC | 22:04 | |
| *** adamw has quit IRC | 22:04 | |
| *** adamw has joined #zuul | 22:05 | |
| *** sugaar has quit IRC | 22:12 | |
| fungi | mordred: good point | 22:24 |
| corvus | mordred, fungi: we could probably hup the ssh connection but i don't think we want to do that on a typical job (sort of defeats the purpose of the optimization in the first place) | 22:27 |
| pabelanger | https://review.opendev.org/653130/ is something that we do in zuul.a.c, just haven't pushed more on it. (also specific to test-setup role) | 22:30 |
| corvus | yeah like that. handy if you need it, but we probably don't want to do it when we don't | 22:30 |
| *** sgw has joined #zuul | 22:34 | |
| *** hashar has quit IRC | 22:34 | |
| ianw | corvus: were you generally ok with making a new principal for project.tarballs and migrating per https://storyboard.openstack.org/#!/story/2006598 #38607 ? | 22:38 |
| corvus | ianw: lets move to infra | 22:49 |
| *** Goneri has quit IRC | 22:57 | |
| *** tosky has joined #zuul | 22:57 | |
| *** jamesmcarthur has joined #zuul | 23:00 | |
| *** jamesmcarthur has quit IRC | 23:20 | |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Add disallowed-labels tenant option https://review.opendev.org/705856 | 23:35 |
| *** zxiiro has joined #zuul | 23:36 | |
| *** tosky has quit IRC | 23:55 | |
| *** igordc has quit IRC | 23:55 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!