| *** yolanda has quit IRC | 00:12 | |
| *** yolanda has joined #zuul | 00:13 | |
| *** jamesmcarthur has joined #zuul | 00:45 | |
| *** jamesmcarthur has quit IRC | 01:13 | |
| *** jamesmcarthur has joined #zuul | 01:20 | |
| *** jamesmcarthur has quit IRC | 01:30 | |
| *** jamesmcarthur has joined #zuul | 02:12 | |
| *** yolanda has quit IRC | 02:32 | |
| *** yolanda has joined #zuul | 02:37 | |
| *** jamesmcarthur has quit IRC | 02:38 | |
| *** yolanda has quit IRC | 02:43 | |
| *** yolanda has joined #zuul | 02:44 | |
| *** jamesmcarthur has joined #zuul | 02:47 | |
| *** rfolco has quit IRC | 02:49 | |
| *** jamesmcarthur has quit IRC | 02:56 | |
| *** jamesmcarthur has joined #zuul | 02:59 | |
| *** yolanda has quit IRC | 03:17 | |
| *** yolanda has joined #zuul | 03:22 | |
| *** jamesmcarthur has quit IRC | 03:25 | |
| *** toabctl has quit IRC | 04:21 | |
| *** toabctl has joined #zuul | 04:24 | |
| *** bhavikdbavishi has joined #zuul | 05:00 | |
| *** raukadah is now known as chkumar|ruck | 05:10 | |
| *** sshnaidm is now known as sshnaidm|afk | 05:29 | |
| *** pcaruana has joined #zuul | 05:29 | |
| *** pcaruana has quit IRC | 05:37 | |
| *** bhavikdbavishi1 has joined #zuul | 05:38 | |
| *** bhavikdbavishi has quit IRC | 05:40 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 05:40 | |
| *** bjackman has joined #zuul | 05:42 | |
| *** jamesmcarthur has joined #zuul | 05:47 | |
| *** pcaruana has joined #zuul | 05:50 | |
| *** jamesmcarthur has quit IRC | 06:03 | |
| *** jamesmcarthur has joined #zuul | 06:06 | |
| *** jamesmcarthur has quit IRC | 06:21 | |
| *** jamesmcarthur has joined #zuul | 06:45 | |
| *** AJaeger has quit IRC | 06:47 | |
| *** jamesmcarthur_ has joined #zuul | 06:51 | |
| *** jamesmcarthur_ has quit IRC | 06:53 | |
| *** jamesmcarthur_ has joined #zuul | 06:53 | |
| *** jamesmcarthur_ has quit IRC | 06:53 | |
| *** jamesmcarthur_ has joined #zuul | 06:54 | |
| *** jamesmcarthur has quit IRC | 06:54 | |
| *** jamesmcarthur_ has quit IRC | 07:04 | |
| *** AJaeger has joined #zuul | 07:04 | |
| *** jamesmcarthur has joined #zuul | 07:05 | |
| *** saneax has joined #zuul | 07:09 | |
| *** jamesmcarthur has quit IRC | 07:10 | |
| *** tosky has joined #zuul | 07:17 | |
| *** bjackman has quit IRC | 07:17 | |
| *** bjackman has joined #zuul | 07:18 | |
| *** jpena|off is now known as jpena | 07:25 | |
| *** themroc has joined #zuul | 07:29 | |
| *** panda has quit IRC | 07:30 | |
| *** panda has joined #zuul | 07:33 | |
| *** zbr is now known as zbr|flu | 07:47 | |
| *** fbo has joined #zuul | 07:47 | |
| *** saneax has quit IRC | 08:42 | |
| *** AJaeger has quit IRC | 08:42 | |
| *** persia has quit IRC | 08:42 | |
| *** mordred has joined #zuul | 08:43 | |
| *** AJaeger has joined #zuul | 08:43 | |
| *** saneax has joined #zuul | 08:43 | |
| *** openstackgerrit has quit IRC | 08:45 | |
| *** persia has joined #zuul | 08:47 | |
| *** irclogbot_0 has quit IRC | 08:49 | |
| *** irclogbot_3 has joined #zuul | 08:53 | |
| *** sshnaidm|afk is now known as sshnaidm | 09:52 | |
| *** bhavikdbavishi has quit IRC | 10:22 | |
| *** bhavikdbavishi has joined #zuul | 10:24 | |
| *** bhavikdbavishi has quit IRC | 10:33 | |
| *** bhavikdbavishi has joined #zuul | 10:36 | |
| *** pcaruana has quit IRC | 10:43 | |
| *** pcaruana has joined #zuul | 10:43 | |
| *** bjackman has quit IRC | 11:13 | |
| *** electrofelix has joined #zuul | 11:23 | |
| *** jpena is now known as jpena|lunch | 11:40 | |
| ianw | tobiash: could you do me a favour and check in on https://github.com/philpep/testinfra/issues/479 about installing the opendev-zuul github app. i think i'm saying the right thing, and i think we're setup system-config wise correctly to run the jobs, but it's my first journey down this particular road | 11:41 |
|---|---|---|
| ianw | if/when i get it worked out, i might write a bit of a user-focused guide on github integration | 11:41 |
| tobiash | ianw: sounds correct | 11:45 |
| tobiash | does the owner know about zuul? | 11:45 |
| ianw | tobiash: yes, we've discussed it before; they want some testing against ansible devel branch which we can provide easily | 11:48 |
| tobiash | ah cool | 11:48 |
| ianw | tobiash: oh, and thanks :) | 11:48 |
| ianw | tobiash: once the app is installed ... if zuul was started without it, will it "reconnect" ... i was looking at it and it seems to poll the available apps at start but not sure if it re-polls | 11:49 |
| ianw | sorry, the app threads poll the available projects | 11:49 |
| ianw | (i.e. currently we get 2019-08-01 14:50:25,604 ERROR zuul.GithubConnection: No installation ID available for project philpep/testinfra) | 11:49 |
| tobiash | ianw: you might need a reconfigure of zuul | 11:49 |
| ianw | ok, good to know, thanks | 11:50 |
| *** tributarian has joined #zuul | 11:52 | |
| *** rlandy has joined #zuul | 12:00 | |
| *** rlandy is now known as rlandy|rover | 12:00 | |
| *** rfolco has joined #zuul | 12:03 | |
| *** aprice has quit IRC | 12:14 | |
| *** aprice has joined #zuul | 12:14 | |
| *** iamweswilson has quit IRC | 12:15 | |
| *** iamweswilson has joined #zuul | 12:15 | |
| *** jpena|lunch is now known as jpena | 12:30 | |
| *** electrofelix has quit IRC | 12:37 | |
| *** themroc has quit IRC | 12:41 | |
| *** electrofelix has joined #zuul | 12:47 | |
| zbr|flu | can zuul pass tags to ansible when running jobs? | 13:05 |
| zbr|flu | ianw: i am also interested about a zuul integration with github, i already have few repos where I would like to test it. | 13:06 |
| pabelanger | zbr|flu: not today, but should be able to support if somebody was to add it | 13:10 |
| zbr|flu | pabelanger: ok. for the moment i will stick to travis and azure-pipelines for github projects, they both have very good integrations. | 13:11 |
| pabelanger | zbr|flu: I mean, if that is the only thing stopping you from migrating to zuul. can whip up a change for it | 13:13 |
| zbr|flu | pabelanger: my only concerns are regarding UX for those trying to make a small contribution. If would work I have no constraints to migrate some of my projects to zuul: like selinux or pytest-molecule. I cannot speak for molecule itself as I do not have the power to make decisions there. Still, gundalow returned today, so thinks may get moving. | 13:17 |
| zbr|flu | personally I would prefer to play with a test (or small) project, with multiple CI in parallel, so we would assure it works well. | 13:17 |
| *** frickler has quit IRC | 13:18 | |
| *** chkumar|ruck is now known as raukadah | 13:30 | |
| corvus | mordred: can you +3 https://review.opendev.org/673327 ? | 13:51 |
| corvus | that will get the log stuff moving | 13:51 |
| corvus | clarkb: ^ | 13:52 |
| clarkb | corvus: is that ready now? | 14:00 |
| corvus | clarkb: yeah, we sent the deprecation notice 2 weeks ago | 14:00 |
| corvus | http://lists.zuul-ci.org/pipermail/zuul-announce/2019-July/000046.html | 14:00 |
| clarkb | corvus: looks like it hit a merge conflict | 14:05 |
| *** openstackgerrit has joined #zuul | 14:05 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: config: add tenant.toDict() method and REST endpoint https://review.opendev.org/621344 | 14:05 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Remove download_artifact_name parameter https://review.opendev.org/673327 | 14:11 |
| corvus | clarkb: ^ still look good? | 14:11 |
| *** tributarian has quit IRC | 14:12 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Use human-readable names for artifact returns https://review.opendev.org/672382 | 14:13 |
| corvus | that one got caught up in the whitespace addition patch | 14:13 |
| clarkb | yup looks like AJaeger and I both got it | 14:14 |
| corvus | clarkb: can you get https://review.opendev.org/672382 while you're at it? | 14:15 |
| clarkb | Done | 14:18 |
| corvus | this isn't quite as time-sensitive, but i'd like to get topic:zuul-release merged before the next release. they're all easy docs changes. | 14:22 |
| corvus | something weird happened with the tests on that, which i haven't debugged yet, but if those could get another +2, i'll figure out what needs to happen to merge them | 14:23 |
| corvus | i'd like to make a release after they land and we verify everything is okay with the url reporting switch in opendev | 14:23 |
| *** tributarian has joined #zuul | 14:28 | |
| openstackgerrit | Merged zuul/zuul-jobs master: Remove download_artifact_name parameter https://review.opendev.org/673327 | 14:30 |
| fungi | what's up with the failures on 675411? does anybody know yet? | 14:33 |
| clarkb | corvus said he would debug if that stack got reviews | 14:33 |
| clarkb | I have +2'd that stack now along with tobiash | 14:33 |
| clarkb | corvus: ^ fyu | 14:34 |
| fungi | see, that's what i get for trying to skim scrollback in an airport gate area | 14:34 |
| fungi | i should know better, and read closer ;) | 14:34 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Collect container logs from test-registry job https://review.opendev.org/675926 | 15:00 |
| *** mattw4 has joined #zuul | 15:01 | |
| *** mattw4 has quit IRC | 15:06 | |
| openstackgerrit | Merged zuul/zuul-jobs master: Use human-readable names for artifact returns https://review.opendev.org/672382 | 15:07 |
| tobiash | corvus, Shrews: I have an interesting question/challenge regarding nodepool images. We're in transition of moving workload away from ceph onto nvme disks. So with this workflow a new image gets first copied onto the compute node if it's not yet cached there. | 15:16 |
| tobiash | the problem we face there is the size of the images (up to 160gb) | 15:16 |
| tobiash | which makes it problematic to just upload the image and then use it for all new nodes (this would kill just ceph and the network) | 15:17 |
| tobiash | corvus, Shrews: what do you think about a hook in nodepool that gets called after the upload and before the first usage? | 15:18 |
| corvus | clarkb, fungi: you may have input ^ | 15:18 |
| tobiash | this could enable two use cases: first it makes it possible to validate the image itself by a script to make sure it can be booted, second it could facilitate our use case by carefully pre-populating the image to all compute nodes before nodepool uses the image | 15:19 |
| clarkb | you mean have nova callback to nodepool to tell it when it is safe to usethe image? | 15:19 |
| clarkb | fwiw I believe boot from volume is meantto address this problem | 15:20 |
| tobiash | I mean adding an optional hook into nodepool itself that calls a shell script that can do whatever | 15:20 |
| tobiash | and based on the return code treat the image upload as success and else as failed | 15:21 |
| tobiash | clarkb: we explicitly go awai from boot from volume because of ceph (it is a severe scalability issue for our workload) | 15:21 |
| tobiash | we just need a way to distribute the imaged in a controlled way to the compute nodes | 15:22 |
| tobiash | our idea here is to create temporary vms using anti affinity to pre-populate the image cache | 15:22 |
| tobiash | so tldr the idea is to add an optional post-upload script into the nodepool image config that can do such things | 15:24 |
| corvus | clarkb: this sounds like something we dealt with in infra-cloud; but my recollection is that we dealt with it by making sure we used qcow instead of raw images, and eliminating an extra layer of caching on the hypervisors? so maybe that still doesn't help with a 160G image? | 15:25 |
| corvus | (my memory of this is pretty fuzzy though) | 15:25 |
| tobiash | we will switch to qcow but this will bring down our image size from 160gb to 100gb which still requires special thought | 15:26 |
| corvus | tobiash: assuming no one has better ideas about how to avoid the need in the first place, a hook sounds reasonable to me | 15:28 |
| tobiash | corvus: cool, I'll look into that | 15:28 |
| tobiash | thanks | 15:28 |
| mordred | yeah - I think a hook seems reasonable - let's deployers handle cases that are specific to their env | 15:28 |
| mordred | s/let's/lets/ | 15:28 |
| clarkb | corvus: ya in infracloud we were uploading qcoes then told nova not to convert them to raw | 15:29 |
| clarkb | as that was an extra set of data to move around | 15:29 |
| corvus | yeah that was it | 15:30 |
| corvus | tobiash: ^ you may want to be aware of that when you switch | 15:30 |
| clarkb | I have no objections to a sanity checker nodepool exec | 15:31 |
| tobiash | yeah, we're planning to switch to qcow and I have the expectation that this makes it better as well | 15:31 |
| tobiash | but given our image sizes I'm pretty sure we still need some pre-popuplation before spawning 100 jobs | 15:31 |
| corvus | sounds likely | 15:31 |
| tobiash | thanks for confirming that qcow is the right way :) | 15:33 |
| tobiash | (plus post-upload hook) | 15:33 |
| *** AJaeger is now known as AJaeger_ | 15:51 | |
| *** jpena is now known as jpena|off | 15:53 | |
| corvus | regarding the registry-related errors on those doc changes -- it looks like somehow one of the layers that was pushed into the intermediate registry is bad | 16:05 |
| corvus | the build that pushed it did not report any errors, but i can reproduce the issue when fetching it. | 16:06 |
| corvus | the push did not happen anywhere near the time that the garbage collection cron runs, so it seems unlikely to be corruption caused by that | 16:06 |
| corvus | it might be worth trying to see if we can find the object for that layer in swift, just to see if there's anything interesting about it. i'm not sure how difficult that will be | 16:07 |
| *** bhavikdbavishi has quit IRC | 16:08 | |
| *** bhavikdbavishi has joined #zuul | 16:10 | |
| corvus | it's a 0 byte object in swift | 16:19 |
| corvus | i'll see if i can dig up some logs from the intermediate registry | 16:20 |
| mordred | corvus: WEIRD | 16:21 |
| *** hwangbo has joined #zuul | 16:23 | |
| corvus | i'm not seeing anything obvious in the registry log | 16:31 |
| corvus | i'm not sure what else to debug here. either the registry failed to store the layer, or swift deleted the data. i lean toward the first option. | 16:32 |
| corvus | it might be the case that the swift backend for the docker registry is not 100% reliable | 16:32 |
| mordred | :( | 16:34 |
| corvus | in happier news, the 'report the zuul url' change is in place in opendev -- here's an example: https://review.opendev.org/#/c/675923/ | 16:35 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Revert "Try out reporting the build page" https://review.opendev.org/675942 | 16:37 |
| corvus | we can clean that up now ^ | 16:37 |
| clarkb | corvus: in that revert do you want to leave off those success urls for npm html? | 16:38 |
| clarkb | and rely on hte build page to link to the artifacts instead? | 16:38 |
| corvus | clarkb: we could go either way -- the build page is still going to get reported, the success-url just determines what that status page links to at this point. | 16:40 |
| clarkb | corvus: ok I +2'd it with that note | 16:41 |
| clarkb | will let you decide if you want to change it or approve it | 16:41 |
| clarkb | as for the intermediate registry do we delete the image that is in there? | 16:41 |
| corvus | clarkb: i think we'll get another set of builds/images as the parent changes merge, so those changes should get unstuck | 16:43 |
| *** armstrongs has joined #zuul | 16:55 | |
| corvus | wow, i'm seeing a bunch of test timeouts | 16:56 |
| openstackgerrit | Merged zuul/zuul-jobs master: Collect container logs from test-registry job https://review.opendev.org/675926 | 16:58 |
| corvus | 2 runs in ovh-bhs1 took >1h, 1 run in gra1 took 1h, 1 run in rax-dfw took 54m | 16:58 |
| corvus | last week the same job took 45m in limenstone and rax-ord | 16:59 |
| corvus | GitPython==2.1.13 -> GitPython==3.0.0 | 17:01 |
| corvus | good news is it works, the bad news is (if my theory is right) it's 1.5x slower | 17:03 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Cap GitPython <3.0.0 https://review.opendev.org/675991 | 17:04 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Add release note for Pagure driver https://review.opendev.org/675410 | 17:04 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Move admin-rules setting in tenants doc https://review.opendev.org/675411 | 17:05 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Make auth docs more boring https://review.opendev.org/675412 | 17:05 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Don't capitalize Token in docs https://review.opendev.org/675413 | 17:05 |
| corvus | i went ahead and rebased that docs stack on it, so we'll get lots of data | 17:05 |
| *** mattw4 has joined #zuul | 17:10 | |
| mordred | corvus: I hope your theory isn't right | 17:14 |
| corvus | yeah. i'm not sure what we should do if it is. | 17:14 |
| *** electrofelix has quit IRC | 17:21 | |
| *** saneax has quit IRC | 17:46 | |
| fungi | tobiash: the other thing we did, which is probably also obvious and therefore not especially helpful advice, is to devote as much network bandwidth as possible to the interfaces compute hosts use to communicate with wherever the original images are stored | 17:47 |
| *** jamesmcarthur has joined #zuul | 17:53 | |
| fungi | allison price at osf just brought this bit of praise to my attention: https://twitter.com/gposton/status/1159027714122059779 | 17:55 |
| *** sgw has quit IRC | 18:03 | |
| corvus | \o/ | 18:05 |
| fungi | word is spreading (somehow, somewhere!) | 18:06 |
| mordred | yay! | 18:14 |
| mordred | I have liked the posts | 18:15 |
| mordred | hopefully doing so is an action that has positive impact | 18:15 |
| *** jamesmcarthur has quit IRC | 18:30 | |
| *** jamesmcarthur has joined #zuul | 18:32 | |
| tobiash | fungi: yes, we have separated nics for storage and user traffic | 18:33 |
| *** jamesmcarthur has quit IRC | 18:34 | |
| *** jamesmcarthur has joined #zuul | 18:34 | |
| tristanC | zbr|flu: here is a story for ansible tag in zuul job: https://storyboard.openstack.org/#!/story/2001353 | 18:50 |
| *** jamesmcarthur has quit IRC | 19:08 | |
| SpamapS | tristanC: neat. My workaround for that has been to break everything up into roles and includable/importable playbooks and compose. Tags are pretty nice though, if you prefer a more monolithic playbook style. | 19:12 |
| *** igordc has joined #zuul | 19:26 | |
| *** igordc has quit IRC | 19:41 | |
| corvus | mordred, clarkb, tobiash: good/bad news -- it does look like gitpython3 caused the 1.5x slowdown. i think we should merge https://review.opendev.org/675991 and start figuring out why | 19:45 |
| tobiash | yay | 19:46 |
| clarkb | I'm good woth capping to work aroubd it for now | 19:46 |
| *** bhavikdbavishi has quit IRC | 19:49 | |
| tristanC | corvus: are we still expecting a zuul release this week? | 19:54 |
| corvus | tristanC: yes, i haven't heard of any issues with the build page reporting in opendev (we made the switch a couple hours ago). assuming that is still the case, once we merge everthing under "topic:zuul-release" (mostly some docs changes) i think we'll be ready | 19:55 |
| corvus | maybe tomorrow or wednesday? | 19:55 |
| tristanC | corvus: excellent, thanks a lot! | 19:56 |
| corvus | i opened an issue with gitpython: https://github.com/gitpython-developers/GitPython/issues/906 | 19:57 |
| corvus | i've confirmed that the single unit test tests.unit.test_merger_repo.TestMergerRepo.test_ensure_cloned takes 1.57x under gitpython 3 | 20:06 |
| corvus | so i think that's a pretty good narrow and reproducible test case | 20:07 |
| *** igordc has joined #zuul | 20:11 | |
| corvus | looking at the timestamps in those tests, a simple repo reset takes 43ms under 2, and 60ms under 3. so the 1.5 factor is still holding | 20:14 |
| openstackgerrit | Merged zuul/zuul master: Cap GitPython <3.0.0 https://review.opendev.org/675991 | 20:36 |
| corvus | it's looking like maybe the number of git process executions has ~doubled | 20:39 |
| corvus | based on running some simple ops through yappi | 20:39 |
| corvus | oh yeah, there's tons of extra 'git rev-parse' invocations | 20:41 |
| SpamapS | bingo | 20:42 |
| corvus | ugh, 2/3 of the git blame commands i'm running are just showing pep8 changes | 20:52 |
| *** pcaruana has quit IRC | 20:56 | |
| *** mattw4 has quit IRC | 21:03 | |
| *** mattw4 has joined #zuul | 21:04 | |
| corvus | way more than 2/3 | 21:16 |
| corvus | every thread i pull ends at a pep8 patch | 21:16 |
| corvus | finally -- i think this is the problem: https://github.com/gitpython-developers/GitPython/pull/808/files | 21:24 |
| corvus | specifically: https://github.com/gitpython-developers/GitPython/pull/894/files#diff-c276fc3c4df38382ec884e59657b869dR450-R458 | 21:27 |
| corvus | that causes a bunch of extra rev-parse calls | 21:28 |
| *** mattw4 has quit IRC | 21:31 | |
| *** mattw4 has joined #zuul | 21:32 | |
| *** mattw4 has quit IRC | 21:38 | |
| *** mattw4 has joined #zuul | 21:39 | |
| *** tosky has quit IRC | 21:55 | |
| corvus | tristanC: https://review.opendev.org/675461 might be interesting to you when you have a sec (low importance) | 22:16 |
| clarkb | corvus: is the git repo config file not implied to be at the git_dir + /config? | 22:21 |
| clarkb | I guess I had assumed that assumption was always true | 22:21 |
| clarkb | also I'm not quite sure I understand how git rev-parse config is supposed to do something use ful there | 22:22 |
| clarkb | oh its --git-path | 22:25 |
| openstackgerrit | Merged zuul/zuul master: Add release note for Pagure driver https://review.opendev.org/675410 | 22:26 |
| openstackgerrit | Merged zuul/zuul master: Move admin-rules setting in tenants doc https://review.opendev.org/675411 | 22:34 |
| openstackgerrit | Merged zuul/zuul master: Make auth docs more boring https://review.opendev.org/675412 | 22:58 |
| *** mattw4 has quit IRC | 23:00 | |
| *** mattw4 has joined #zuul | 23:00 | |
| *** rlandy|rover is now known as rlandy|rover|bbl | 23:12 | |
| openstackgerrit | Merged zuul/zuul master: Revert "Try out reporting the build page" https://review.opendev.org/675942 | 23:31 |
| *** armstrongs has quit IRC | 23:40 | |
| *** mattw4 has quit IRC | 23:48 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!