| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Return artifact metadata in API endpoint https://review.openstack.org/638052 | 00:12 |
|---|---|---|
| *** rlandy has quit IRC | 00:16 | |
| *** jamesmcarthur has quit IRC | 00:22 | |
| *** jamesmcarthur has joined #zuul | 00:23 | |
| *** jamesmcarthur has quit IRC | 00:27 | |
| *** sdake has joined #zuul | 00:59 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Don't set child jobs to skipped on post failure https://review.openstack.org/638054 | 01:35 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: push-to-registry: handle undefined docker_images https://review.openstack.org/638057 | 01:49 |
| *** cognifloyd has joined #zuul | 02:07 | |
| mrhillsman | anyone running or tried running zuul on all arm? | 02:23 |
| clarkb | I'm not aware of anyone doing that but I cant think of any proble areas for doing that | 02:30 |
| *** sdake has quit IRC | 02:36 | |
| *** jamesmcarthur has joined #zuul | 02:37 | |
| fungi | yeah, in theory should work fine | 02:40 |
| *** bhavikdbavishi has joined #zuul | 03:10 | |
| dmsimard | Looking at this line: https://github.com/openstack-infra/zuul/blob/5271b592afe708d33fc4b3d08d9a2cc97ae0ddfc/.zuul.yaml#L51 where does "zuul.project.src_dir" come from ? | 03:16 |
| dmsimard | I mean, if that is rendered at runtime, could it be different depending on which project triggered the job ? | 03:17 |
| *** jamesmcarthur has quit IRC | 03:21 | |
| *** jamesmcarthur has joined #zuul | 03:22 | |
| *** jamesmcarthur has quit IRC | 03:27 | |
| *** jamesmcarthur has joined #zuul | 03:44 | |
| clarkb | yes its the source dir of the triggering project | 03:52 |
| clarkb | 03:53 | |
| *** jamesmcarthur has quit IRC | 03:59 | |
| *** jamesmcarthur has joined #zuul | 04:00 | |
| *** jamesmcarthur has quit IRC | 04:05 | |
| *** jamesmcarthur has joined #zuul | 04:09 | |
| *** jamesmcarthur has quit IRC | 04:22 | |
| *** jamesmcarthur has joined #zuul | 04:25 | |
| *** jamesmcarthur has quit IRC | 04:30 | |
| *** jamesmcarthur has joined #zuul | 04:53 | |
| *** cognifloyd has left #zuul | 04:59 | |
| *** jamesmcarthur has quit IRC | 04:59 | |
| *** kratec has joined #zuul | 05:31 | |
| jhesketh | tobiash: (sorry for the slow reply), regarding 636146 I'm fine with merging it, but initially wanted to leave it up longer because my review was shortly after it was published and wanted to give others a chance to weigh in etc | 05:36 |
| tobiash | jhesketh: thanks | 05:38 |
| *** k4ch0 has joined #zuul | 06:00 | |
| *** badboy has joined #zuul | 06:04 | |
| badboy | #gentoo-pl | 06:04 |
| *** k4ch0 has quit IRC | 06:22 | |
| *** sdake has joined #zuul | 06:25 | |
| *** sdake has quit IRC | 06:27 | |
| *** saneax has joined #zuul | 06:31 | |
| openstackgerrit | Merged openstack-infra/zuul master: Don't exclude config if excluded in earlier tenant https://review.openstack.org/636146 | 06:41 |
| *** quiquell|off is now known as quiquell|rover | 06:49 | |
| *** hasharAway has quit IRC | 07:09 | |
| *** jamesmcarthur has joined #zuul | 07:18 | |
| *** jamesmcarthur has quit IRC | 07:22 | |
| *** quiquell|rover is now known as quique|rover|brb | 07:36 | |
| *** chandankumar is now known as chkumar|ruck | 07:59 | |
| *** pcaruana has joined #zuul | 07:59 | |
| *** quique|rover|brb is now known as quiquell|rover | 08:04 | |
| *** avass has joined #zuul | 08:12 | |
| *** ruffian_sheep has joined #zuul | 08:14 | |
| *** hashar has joined #zuul | 08:28 | |
| openstackgerrit | Jan Kundrát proposed openstack-infra/nodepool master: Implement a Runc driver https://review.openstack.org/535556 | 08:29 |
| *** gtema has joined #zuul | 08:41 | |
| *** electrofelix has joined #zuul | 08:45 | |
| *** jpena|off is now known as jpena | 09:00 | |
| *** saneax has quit IRC | 09:04 | |
| *** saneax has joined #zuul | 09:05 | |
| *** nilashishc has joined #zuul | 09:30 | |
| *** themroc has joined #zuul | 09:38 | |
| *** nilashishc has quit IRC | 09:43 | |
| *** panda|off is now known as panda | 09:46 | |
| quiquell|rover | tobiash: ping, do you have a minute ? | 10:26 |
| *** bhavikdbavishi has quit IRC | 10:31 | |
| tobiash | quiquell|rover: sorry, full day workshop today | 10:35 |
| quiquell|rover | tobiash: ack np | 10:37 |
| *** ianychoi has quit IRC | 10:59 | |
| *** ruffian_sheep has quit IRC | 11:48 | |
| *** hashar has quit IRC | 11:56 | |
| *** avass has quit IRC | 11:59 | |
| *** jpena is now known as jpena|lunch | 12:12 | |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: CLI: fail if trying to enqueue/dequeue a change for the wrong project https://review.openstack.org/636662 | 12:31 |
| *** hashar has joined #zuul | 12:31 | |
| *** rlandy has joined #zuul | 13:00 | |
| *** saneax has quit IRC | 13:10 | |
| *** saneax has joined #zuul | 13:11 | |
| *** jamesmcarthur has joined #zuul | 13:18 | |
| *** quiquell|rover is now known as quique|rover|mtg | 13:20 | |
| *** jamesmcarthur has quit IRC | 13:22 | |
| *** jamesmcarthur has joined #zuul | 13:22 | |
| *** ianychoi has joined #zuul | 13:22 | |
| *** electrofelix has quit IRC | 13:31 | |
| *** jamesmcarthur has quit IRC | 13:33 | |
| *** jpena|lunch is now known as jpena | 13:34 | |
| *** spsurya has joined #zuul | 13:44 | |
| *** jamesmcarthur has joined #zuul | 13:51 | |
| *** jamesmcarthur has quit IRC | 13:51 | |
| *** jamesmcarthur has joined #zuul | 13:52 | |
| *** sdake has joined #zuul | 14:10 | |
| *** sdake has quit IRC | 14:18 | |
| iurygregory | Morning everyone o/, does anyone has idea how to set a configuration file to be used by tox when running zuulv3? We have functional tests in python-ironicclient https://github.com/openstack/python-ironicclient/blob/master/tools/run_functional.sh but i didnt figure out how could i made it run using zuulv3, any tips? My zuulv3 patch is https://review.openstack.org/#/c/633010/ | 14:28 |
| *** sdake has joined #zuul | 14:34 | |
| *** chkumar|ruck is now known as kmrchdn | 14:35 | |
| dmsimard | iurygregory: https://zuul-ci.org/docs/zuul-jobs/jobs.html#job-tox maybe tox_environment or tox_extra_args ? | 14:39 |
| *** electrofelix has joined #zuul | 14:40 | |
| iurygregory | dmsimard, tks man! going to give a try =) | 14:40 |
| *** gtema has quit IRC | 14:52 | |
| *** jamesmcarthur has quit IRC | 14:53 | |
| *** sdake has quit IRC | 14:58 | |
| *** quique|rover|mtg is now known as quiquell | 15:06 | |
| *** saneax has quit IRC | 15:33 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: use-buildset-registry: support running before docker installed https://review.openstack.org/638180 | 15:36 |
| *** gtema has joined #zuul | 15:43 | |
| *** kmrchdn is now known as chandankumar | 15:48 | |
| *** nilashishc has joined #zuul | 15:54 | |
| *** quiquell is now known as quiquell|off | 15:56 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: Split docker mirror config into its own role https://review.openstack.org/638195 | 16:13 |
| jkt | what is the usual Zuul idiom for running pieces of a job as root on target nodes? My nodes are containers maanged by nodepool's runc (not-yet-merged) if that matters | 16:17 |
| jkt | do I use become: true, sudo, etc? | 16:17 |
| *** nilashishc has quit IRC | 16:20 | |
| pabelanger | jkt: become: true in your ansible playbooks should be enough | 16:21 |
| jkt | pabelanger: and that is subject to whatever ansible rules, defaulting probably to sudo, right, thanks | 16:22 |
| pabelanger | jkt: yup, you have the ability to control it via a playbook using become_method | 16:23 |
| fungi | also in openstack we've found it's helpful to revoke sudo privileges from the zuul user at some point in a lot of jobs to prevent accidental use of sudo from creeping into the code being exercised by our jobs | 16:23 |
| jkt | fungi: I'll look into this, thanks! | 16:26 |
| fungi | jkt: we use the revoke-sudo role from the zuul-jobs standard library for that purpose: https://zuul-ci.org/docs/zuul-jobs/roles.html#role-revoke-sudo | 16:34 |
| *** nilashishc has joined #zuul | 16:42 | |
| *** themroc has quit IRC | 16:45 | |
| *** nilashishc has quit IRC | 16:57 | |
| jkt | ...so the fact that runc containers have r/o rootfs is something as a blocker for this :) | 17:02 |
| *** nilashishc has joined #zuul | 17:06 | |
| mordred | jkt: is that a fundamental feature of runc containers? or just how the nodepool driver is doing them? (I haven't dug in to the runc driver yet) | 17:11 |
| *** pcaruana has quit IRC | 17:13 | |
| *** nilashishc has quit IRC | 17:17 | |
| *** nilashishc has joined #zuul | 17:18 | |
| *** chandankumar is now known as kmrchdn | 17:19 | |
| *** nilashishc has quit IRC | 17:23 | |
| *** nilashishc has joined #zuul | 17:23 | |
| *** nilashishc has quit IRC | 17:28 | |
| *** nilashishc has joined #zuul | 17:35 | |
| *** nilashishc has quit IRC | 17:49 | |
| *** panda is now known as panda|off | 17:59 | |
| *** sdake has joined #zuul | 18:03 | |
| *** hashar has quit IRC | 18:03 | |
| *** gtema has quit IRC | 18:04 | |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: push-to-registry: handle undefined docker_images https://review.openstack.org/638057 | 18:06 |
| *** jpena is now known as jpena|off | 18:11 | |
| *** themroc has joined #zuul | 18:25 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: use-buildset-registry: support running before docker installed https://review.openstack.org/638180 | 18:35 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: Split docker mirror config into its own role https://review.openstack.org/638195 | 18:35 |
| corvus | pabelanger: can you please review https://review.openstack.org/638054 and parent? | 18:38 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: use-buildset-registry: support running before docker installed https://review.openstack.org/638180 | 18:48 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: Split docker mirror config into its own role https://review.openstack.org/638195 | 18:48 |
| *** sdake has quit IRC | 18:58 | |
| pabelanger | +2 | 19:00 |
| *** sdake has joined #zuul | 19:14 | |
| *** saneax has joined #zuul | 19:28 | |
| *** rfolco has quit IRC | 19:32 | |
| *** electrofelix has quit IRC | 19:38 | |
| *** hashar has joined #zuul | 19:38 | |
| *** rfolco has joined #zuul | 19:39 | |
| *** sdake has quit IRC | 19:50 | |
| *** saneax has quit IRC | 19:53 | |
| openstackgerrit | Merged openstack-infra/zuul master: Support boolean values in get_default https://review.openstack.org/637158 | 19:56 |
| daniel2 | So looking at the image.log file from the image builder, it is super hard to find what is causing a failed build. Any ideas on how to find that faster? | 19:58 |
| daniel2 | I literally have to go through thousands of lines of logs to find the one line that shows it failed. | 19:58 |
| tobiash | daniel2: just look at the last hundred ;) | 19:59 |
| daniel2 | Well it doesn't help that it restarts the build after it fails | 19:59 |
| daniel2 | is there a way to make it stop after fail? | 19:59 |
| tobiash | it does, it just does some extensive cleanup that also spits out logs | 19:59 |
| daniel2 | Well, no, I mean, once it fails it restarts the build which floods the image.log file. | 20:00 |
| tobiash | daniel2: you can configure nodepool to create one log file per image build | 20:00 |
| daniel2 | ah yes that makes sense | 20:01 |
| *** jamesmcarthur has joined #zuul | 20:02 | |
| tobiash | daniel2: this is the relevant part of a logging config that does that: http://paste.openstack.org/show/745513/ | 20:03 |
| tobiash | corvus: do you have any objection about making it possible to disable the disk accountant in the executor? | 20:04 |
| corvus | tobiash: what's the problem? | 20:04 |
| tobiash | I have a hunch that it is very expensive when running large jobs and having many large repos in the cache | 20:05 |
| daniel2 | tobiash: Thats actually what its set to, but its just reusing /var/log/nodepool/image/image.log | 20:05 |
| openstackgerrit | Merged openstack-infra/zuul master: Return artifact metadata in API endpoint https://review.openstack.org/638052 | 20:05 |
| openstackgerrit | Merged openstack-infra/zuul master: Don't set child jobs to skipped on post failure https://review.openstack.org/638054 | 20:05 |
| tobiash | and we're very short on io performance on the executors | 20:05 |
| tobiash | but have large disks | 20:05 |
| tobiash | so I think in our setup it might harm more than it provides us safety | 20:06 |
| tobiash | daniel2: hrm, let me recheck our logging config, as our builder has one log file per build | 20:08 |
| corvus | tobiash: i think it would be okay (probably just reuse the max size setting for that). we could also try renicing the thread *, or increasing the loop delay. (* apparently individual threads can have their own niceness in linux) | 20:08 |
| daniel2 | So I noticed that there are two log configs in /etc/nodepool, one is logging.conf and the other is builder-logging.conf | 20:08 |
| daniel2 | So I'm unsure if its even picking up the builder-logging.conf file | 20:09 |
| daniel2 | I'm working with old stuff created by my colleague 3 years ago. They were on nodepool 0.3.0, I updated it to 0.5.0 for now. | 20:09 |
| tobiash | corvus: k, I'll try it via the max size setting, also renicing won't make much difference in our setup because of shared ceph cluster where I'm currently trying to avoid any unneccessary io | 20:10 |
| tobiash | daniel2: scratch that, my logging config looks outdated in this regard | 20:12 |
| daniel2 | It looks like it just rotates after a certain time, not after a build. | 20:12 |
| tobiash | daniel2: current nodepool versions just do this by default | 20:13 |
| tobiash | but that probably won't help you as you're probably still on zuulv2? | 20:14 |
| daniel2 | yeah.. | 20:16 |
| daniel2 | I tried to push on just upgrading but got shot down, so I do what I can. | 20:17 |
| daniel2 | I'm low on the totem pole. | 20:18 |
| daniel2 | Why did nodepool/zuul drop jenkins anyways? | 20:18 |
| daniel2 | Jenkins is a pretty solid tool. | 20:18 |
| mordred | daniel2: many reasons, but importantly it's not solid at scale | 20:19 |
| daniel2 | ah I guess that's a good point | 20:19 |
| daniel2 | I just started this job a month and a half ago. I've gotten a pretty good crash course into nodepool. I might take some spare time to at least make a POC in order to push updating. | 20:20 |
| mordred | daniel2: nothing like on-the-job learning! :) | 20:20 |
| daniel2 | I haven't worked with openstack since grizzly when I was a NOC Engineer for HP Public Cloud. | 20:21 |
| daniel2 | Actually, I think I worked with Havana a little bit at Mirantis, and that's where I learned how to work with Gerrit. | 20:21 |
| mordred | daniel2: oh - we've worked together before then. I was also at HP Public Cloud. | 20:22 |
| daniel2 | mordred: Did you work there in 2014? I was there Feb 2014 to Feb 2015. | 20:23 |
| mordred | daniel2: yup. Nov2011 to Aug2015. although by Feb 2014 I wasn't as directly involved with the public cloud | 20:24 |
| daniel2 | mordred: Well most of the service teams hated us in the NOC :) | 20:25 |
| mordred | :) | 20:25 |
| daniel2 | I wrote a portal for the NOC using the openstack bindings and Django. It was one of my biggest accomplishments. It was meant just for the NOC but eventually started to spread around the company. It was capable of tracking down anything to anywhere, instances to routers to networks to customers, etc. | 20:26 |
| tobiash | corvus, jkt: regarding the update thread and checkout parallelization (635496, 637996). It seems like we have somewhat conflicting conditions. So a one thread count to rule them all may not be possible. What do you think about defaulting to the cpu count (which should be a reasonable default for most installations) and still make that overidable? | 20:31 |
| *** spsurya has quit IRC | 20:42 | |
| jkt | tobiash: I'm fine with that, but just note that e.g. 3.7+'s implementation actually uses 5*CPUs in their thread-backed pool implementation, https://docs.python.org/dev/library/concurrent.futures.html#concurrent.futures.ThreadPoolExecutor | 20:49 |
| jkt | their reasoning is IMHO relevant, they are also talking about overlapping IO | 20:49 |
| *** themroc has quit IRC | 20:50 | |
| corvus | tobiash: what's the conflict? ie, why not go with 5x or 8x? | 20:51 |
| tobiash | jkt: I think that might be ok for us as long we have that threadpool per executor and not per job | 20:53 |
| tobiash | So go with 5x or 8x and have a common threadpool for updates and clones sounds like a good compromise | 20:55 |
| corvus | tobiash: the existing change has a common thread pool, right? | 20:56 |
| corvus | the only issue under discussion is what the level of parallelization should be? | 20:57 |
| tobiash | corvus: for updates yes, jkt's followup currently has one per job if I read that correct | 20:57 |
| corvus | oh, i haven't looked at that | 20:57 |
| tobiash | I think that was the modt important part I stumbled accross | 20:57 |
| tobiash | I think it makes sense to combine those threadpools an have 8x cpu maybe with a limit at 64 threads or so | 21:00 |
| corvus | tobiash: then, yes, i think i agree with you. python can't handle more than 1000 threads (even 1000 is probably too much). so we need to plan for having about 100 job threads and certainly no more than 100 (64 sounds better) io threads. | 21:01 |
| openstackgerrit | Ian Wienand proposed openstack-infra/nodepool master: Use a pipeline for dib stats https://review.openstack.org/638265 | 21:01 |
| corvus | (with 1k threads, the overhead for dealing with the gil, etc, is too much, cpu-bound threads get starved even when there's available cpu) | 21:02 |
| tobiash | Yes, I've saw this in nodepool (v2) a few times | 21:03 |
| *** jamesmcarthur has quit IRC | 21:05 | |
| *** sdake has joined #zuul | 21:12 | |
| *** jamesmcarthur has joined #zuul | 21:17 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: use-buildset-registry: support running before docker installed https://review.openstack.org/638180 | 21:28 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: Split docker mirror config into its own role https://review.openstack.org/638195 | 21:29 |
| *** takamatsu_ has joined #zuul | 21:31 | |
| *** takamatsu has quit IRC | 21:31 | |
| *** sdake has quit IRC | 21:38 | |
| *** sdake has joined #zuul | 21:39 | |
| *** takamatsu_ has quit IRC | 21:45 | |
| *** takamatsu has joined #zuul | 21:46 | |
| *** sdake has quit IRC | 21:48 | |
| *** sdake has joined #zuul | 21:49 | |
| jkt | tobiash: sure, that makes sense, I should have probably marked these patches as RFC or something | 21:53 |
| jkt | tobiash: I'll see if I can make these thread pools global, not per-job | 21:53 |
| *** jamesmcarthur has quit IRC | 21:55 | |
| *** jamesmcarthur has joined #zuul | 21:59 | |
| *** sdake has quit IRC | 21:59 | |
| *** sdake has joined #zuul | 22:01 | |
| *** hashar has quit IRC | 22:09 | |
| *** hashar has joined #zuul | 22:10 | |
| *** hashar has quit IRC | 22:10 | |
| *** hashar has joined #zuul | 22:11 | |
| corvus | i wonder if we should increase the priority of node requests for childred of paused jobs | 22:36 |
| corvus | otherwise, we may leave a paused job idling for a while, which is inefficient overall | 22:36 |
| *** hashar has quit IRC | 22:37 | |
| mordred | corvus: yeah. seems like an unterrible idea | 22:47 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: use-buildset-registry: support running before docker installed https://review.openstack.org/638180 | 22:48 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: Split docker mirror config into its own role https://review.openstack.org/638195 | 22:48 |
| *** rlandy is now known as rlandy|bbl | 23:14 | |
| *** jamesmcarthur has quit IRC | 23:55 | |
| *** jamesmcarthur has joined #zuul | 23:56 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!