| pabelanger | SpamapS: mordred: okay, thanks. I'll guess I'll try and figure out how to deal with the python version via ansible (for site-package) | 00:02 |
|---|---|---|
| *** dmsimard has quit IRC | 00:25 | |
| *** sdake has quit IRC | 00:37 | |
| *** sdake has joined #zuul | 00:41 | |
| *** sdake has quit IRC | 00:41 | |
| *** dmsimard has joined #zuul | 00:57 | |
| *** sdake has joined #zuul | 01:12 | |
| SpamapS | pabelanger: we could probably make the zuul CLI print it out | 01:20 |
| SpamapS | But as mordred says, maybe best is just to send everything to cherrypy | 01:21 |
| *** sdake has quit IRC | 01:48 | |
| *** sdake has joined #zuul | 01:50 | |
| *** manjeets has quit IRC | 02:15 | |
| *** manjeets has joined #zuul | 02:16 | |
| *** sdake has quit IRC | 02:19 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add /{tenant}/buildsets route https://review.openstack.org/630035 | 03:08 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add buildsets page https://review.openstack.org/630041 | 03:08 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add /{tenant}/buildset/{uuid} route https://review.openstack.org/630078 | 03:10 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul-jobs master: add-build-sshkey: remove previously authorized build-sshkey https://review.openstack.org/632620 | 03:12 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: Implement zookeeper-auth https://review.openstack.org/619155 | 03:23 |
| *** rlandy|bbl is now known as rlandy | 03:43 | |
| *** sdake has joined #zuul | 03:43 | |
| *** sdake has quit IRC | 03:44 | |
| *** rlandy has quit IRC | 04:06 | |
| *** sdake has joined #zuul | 04:10 | |
| *** sdake has quit IRC | 05:18 | |
| *** chandankumar has joined #zuul | 05:45 | |
| *** chandankumar is now known as chkumar|ruck | 05:46 | |
| *** swest has joined #zuul | 06:21 | |
| *** swest has quit IRC | 06:27 | |
| *** swest has joined #zuul | 06:28 | |
| *** quiquell|off is now known as quiquell|rover | 06:36 | |
| *** saneax has joined #zuul | 06:37 | |
| quiquell|rover | tobiash: thanks, didn't have much time to give youlove to it, just rebase love | 06:37 |
| quiquell|rover | tobiash: if you have some time today maybe you can help me with it | 06:39 |
| tobiash | quiquell|rover: if you want I could fix it later today or this evening | 06:41 |
| quiquell|rover | I will put back .keep | 06:42 |
| tobiash | ok | 06:42 |
| quiquell|rover | What I didn't figure out is | 06:42 |
| quiquell|rover | How to use ansible zuul test case | 06:43 |
| quiquell|rover | And hold node | 06:43 |
| quiquell|rover | Hold build sorry | 06:43 |
| quiquell|rover | Y call release in the unit test but build is still there | 06:43 |
| quiquell|rover | I have to find why is that | 06:44 |
| quiquell|rover | Maybe you can give me a clue on that | 06:44 |
| tobiash | quiquell|rover: holding the builds with ansible test cases is not widely used atm | 06:46 |
| tobiash | I noticed that you have to call release for each playbook | 06:46 |
| tobiash | do you really need to hold the build in the test case? | 06:46 |
| quiquell|rover | I need that to get the inventory | 06:46 |
| quiquell|rover | If not it's vanished | 06:46 |
| quiquell|rover | Will look into it | 06:46 |
| tobiash | you can also instruct it to keep the build dir | 06:47 |
| quiquell|rover | Hummm | 06:47 |
| quiquell|rover | That works for me | 06:47 |
| tobiash | look in test_v3 for that, I thing keep should be used somewhere in there | 06:47 |
| tobiash | or you can just put the change message into a file within a test playbook | 06:48 |
| *** chkumar|ruck has quit IRC | 06:48 | |
| tobiash | and assert within the playbook that the content is correct | 06:48 |
| *** chandankumar has joined #zuul | 06:48 | |
| tobiash | that'll work too | 06:48 |
| *** chandankumar is now known as chkumar|ruck | 06:49 | |
| *** quiquell|rover is now known as quique|rover|r-- | 06:53 | |
| quique|rover|r-- | Thanks!! Will look at it | 06:53 |
| *** gtema has joined #zuul | 06:54 | |
| *** pcaruana has joined #zuul | 07:02 | |
| *** bjackman has joined #zuul | 07:11 | |
| *** quique|rover|r-- is now known as quiquell|rover | 07:18 | |
| *** bjackman has quit IRC | 07:21 | |
| *** bjackman has joined #zuul | 07:23 | |
| *** quiquell|rover is now known as quiquell|rover|b | 07:33 | |
| *** quiquell|rover|b is now known as quique|rover|brb | 07:33 | |
| *** quique|rover|brb is now known as quiquell|rover | 07:58 | |
| *** hashar has joined #zuul | 08:04 | |
| *** gtema has quit IRC | 08:08 | |
| *** themroc has joined #zuul | 08:17 | |
| *** gtema has joined #zuul | 08:36 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add /{tenant}/buildsets route https://review.openstack.org/630035 | 08:39 |
| *** jpena|off is now known as jpena | 08:45 | |
| *** zbr|ssbarnea has joined #zuul | 08:49 | |
| *** chkumar|ruck has quit IRC | 08:53 | |
| *** zbr has joined #zuul | 09:11 | |
| *** zbr|ssbarnea has quit IRC | 09:12 | |
| *** saneax has quit IRC | 09:15 | |
| *** panda|off is now known as panda | 09:17 | |
| quiquell|rover | tobiash: Got it working thanks !!! | 09:20 |
| tobiash | :) | 09:20 |
| quiquell|rover | Le me also add the poor .keep file | 09:20 |
| *** saneax has joined #zuul | 09:22 | |
| openstackgerrit | Quique Llorente proposed openstack-infra/zuul master: Escape jinja2 stuff from inventory https://review.openstack.org/633930 | 09:22 |
| openstackgerrit | Quique Llorente proposed openstack-infra/zuul master: Escape jinja2 stuff from inventory https://review.openstack.org/633930 | 09:22 |
| quiquell|rover | tobiash: ^ let's see now | 09:23 |
| quiquell|rover | tobiash: do I have to add more test to that ? | 09:23 |
| quiquell|rover | maybe trying to parse inventory with ansible | 09:23 |
| *** saneax has quit IRC | 09:28 | |
| *** saneax has joined #zuul | 09:29 | |
| *** bjackman_ has joined #zuul | 09:31 | |
| *** bjackman has quit IRC | 09:33 | |
| *** chandankumar has joined #zuul | 09:34 | |
| tobiash | quiquell|rover: commented | 09:35 |
| *** chandankumar is now known as chkumar|ruck | 09:35 | |
| quiquell|rover | tobiash: thanks ! let's do it | 09:36 |
| *** zbr has quit IRC | 09:40 | |
| *** zbr|ssbarnea has joined #zuul | 09:40 | |
| *** chandankumar has joined #zuul | 09:50 | |
| *** bjackman_ has quit IRC | 09:51 | |
| *** chkumar|ruck has quit IRC | 09:51 | |
| *** chandankumar is now known as chkumar|ruck | 09:51 | |
| *** bjackman_ has joined #zuul | 09:57 | |
| *** zbr|ssbarnea has quit IRC | 10:04 | |
| *** zbr|ssbarnea has joined #zuul | 10:07 | |
| openstackgerrit | Quique Llorente proposed openstack-infra/zuul master: Escape jinja2 stuff from inventory https://review.openstack.org/633930 | 10:23 |
| *** bjackman_ has quit IRC | 10:29 | |
| *** bjackman_ has joined #zuul | 10:29 | |
| openstackgerrit | Fabien Boucher proposed openstack-infra/zuul master: [WIP] - URLTrigger based on the timer trigger https://review.openstack.org/635241 | 10:57 |
| *** fdegir has quit IRC | 10:57 | |
| *** fdegir has joined #zuul | 10:58 | |
| *** sshnaidm|afk is now known as sshnaidm | 11:17 | |
| *** chkumar|ruck has quit IRC | 11:29 | |
| *** chandankumar has joined #zuul | 11:30 | |
| *** chandankumar is now known as chkumar|ruck | 11:30 | |
| *** rfolco is now known as rfolco_doctor | 11:38 | |
| *** bjackman__ has joined #zuul | 11:39 | |
| *** bjackman_ has quit IRC | 11:41 | |
| openstackgerrit | Quique Llorente proposed openstack-infra/zuul master: Escape jinja2 stuff from inventory https://review.openstack.org/633930 | 11:47 |
| quiquell|rover | tobiash: ^ now is all covered I think | 11:47 |
| quiquell|rover | tobiash: also the ansible expansion | 11:47 |
| *** electrofelix has joined #zuul | 11:58 | |
| lennyb | Hi, I am running zuul2 and I see a lot of my jobs in 'queued' state. I have enough free nodes in Jenkins, so I don`t understand why job is not triggered. No errors in logs. | 11:59 |
| tobiash | lennyb: maybe you need to restart jenkins | 12:00 |
| tobiash | lennyb: that solved many problems ages ago ;) | 12:00 |
| tobiash | at least I had often similar problems back then and a jenkins restart solved these most of the time | 12:01 |
| *** electrofelix has quit IRC | 12:04 | |
| *** swest has quit IRC | 12:04 | |
| *** electrofelix has joined #zuul | 12:05 | |
| *** hashar has quit IRC | 12:10 | |
| tobiash | mordred: looking at my graphs it looks like that 634598 was good because of clean code but didn't solve the memleak on the executors :( | 12:13 |
| lennyb | tobiash: thanks, but it did not help. | 12:15 |
| tobiash | lennyb: hrm, maybe zuul lost its connection to jenkins | 12:15 |
| tobiash | lennyb: or maybe gearman is stuck | 12:15 |
| lennyb | gearman plugin passed test. it happens a lot, restarting zuul 'fixes' the issue, but I am loosing a lot of commits. | 12:16 |
| mordred | tobiash: :( | 12:16 |
| *** swest has joined #zuul | 12:20 | |
| tobiash | mordred: that's the rss of our executor containers during the last 6 months: https://paste.pics/7a57a7ae4a3973198b41e4db5005724a | 12:20 |
| tobiash | mordred: at one point in time it started | 12:20 |
| tobiash | mordred: we switched from alpine to ubuntu at this time | 12:20 |
| tobiash | mordred: so maybe a memleak in python itself or a different java-like memory management in glibc based python | 12:21 |
| mordred | tobiash: so the leak correlates with the switch from alpine? | 12:21 |
| tobiash | yes | 12:21 |
| tobiash | exactly with the date when we rolled it out | 12:22 |
| mordred | you are using the python shipped by ubuntu in your containers, yes? | 12:22 |
| tobiash | yes | 12:22 |
| mordred | and before with alpine were you using the python:alpine images? or normal alpine with the alpine provided python? | 12:22 |
| tobiash | so could be either alpine vs ubuntu or py37 vs py36 as alpine was at 3.7 at that time afaik | 12:22 |
| tobiash | it was normal alpine | 12:23 |
| mordred | yeah. there's several things possible there | 12:23 |
| tobiash | maybe I should try py37 from universe to rule that out | 12:23 |
| mordred | could be patches to python applied by ubuntu, or libc differences, or python versions | 12:23 |
| mordred | yeah - I know that 3.6 introduced a new dict impl | 12:23 |
| mordred | so it's possible 3.7 contains fixes for it that are important for our heavy dict usage | 12:23 |
| tobiash | possibly | 12:24 |
| mordred | it'll be interesting to see if we see the same behavior when we start deploying from the python:slim debian-based containers with 3.7 | 12:24 |
| mordred | tobiash: it also seems like your immediate baseline memory is higher even before it starts leaking too | 12:25 |
| mordred | but maybe that's just musl vs glibc | 12:25 |
| mordred | we're still running on 3.5 on xenial at the moment | 12:26 |
| tobiash | ah | 12:27 |
| mordred | http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=64003&rra_id=all <-- there's memory graphs for one of our executors - doesn't look like we're seeing the same sorts of increases you are | 12:28 |
| tobiash | yes, that looks different | 12:29 |
| mordred | python version and libc version are the primary differences that I'd expect to be relevant | 12:30 |
| tobiash | but at least we're updating zuul faster than the executors could oom ;) | 12:30 |
| tobiash | yes | 12:30 |
| tobiash | so think I'll try py37 first | 12:31 |
| mordred | ++ - that'll be a good data point | 12:31 |
| tobiash | it's available as a package in universe so should be relatively easy to use that | 12:31 |
| *** bjackman__ has quit IRC | 12:32 | |
| SpamapS | tobiash: you should also consider trying jemalloc | 12:33 |
| SpamapS | https://zapier.com/engineering/celery-python-jemalloc/ | 12:34 |
| mordred | SpamapS: you are awake at an absurdly early hour | 12:35 |
| SpamapS | mordred: tell me about it :-P | 12:35 |
| mordred | SpamapS: I have very recent memories of being awake at the same hour in my timezone | 12:35 |
| SpamapS | Got my 4 hours in :-P | 12:35 |
| mordred | me too! | 12:36 |
| tobiash | SpamapS: thanks, reading | 12:36 |
| *** bjackman has joined #zuul | 12:38 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Perform per repo locking on the executor https://review.openstack.org/635495 | 12:40 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Optionally parallelize update threads https://review.openstack.org/635496 | 12:40 |
| *** jpena is now known as jpena|lunch | 12:44 | |
| * SpamapS may try to snag another hour before the sun comes up | 12:44 | |
| *** sdake has joined #zuul | 12:46 | |
| *** sdake has quit IRC | 12:51 | |
| *** sdake has joined #zuul | 12:52 | |
| *** sshnaidm is now known as sshnaidm|afk | 12:52 | |
| *** rlandy has joined #zuul | 13:04 | |
| *** sshnaidm|afk is now known as sshnaidm | 13:11 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: WIP Run python with jemalloc in containers https://review.openstack.org/635504 | 13:14 |
| tobiash | mordred: lol, just thought about doing the same thing ^ | 13:14 |
| mordred | tobiash, SpamapS: ^^ there's a patch inspired by that blogpost - completely untested, and might be a terrible idea | 13:14 |
| mordred | tobiash: :) | 13:14 |
| tobiash | mordred: the packaged versions of jemalloc are very old as it seems | 13:15 |
| tobiash | both on bionic and stretch | 13:16 |
| tobiash | 3.6.0 is from early 2015 :-/ | 13:16 |
| tobiash | so you might want to build it from source ;) | 13:16 |
| mordred | yah - although in that post even 3.5 seemed to have a significant impact on his swap usage | 13:17 |
| tobiash | yes | 13:17 |
| mordred | tobiash: totally. if it winds up being meaningful for zuul, I think building it from source in a builder image wouldn't be terrible | 13:17 |
| tobiash | ++ | 13:17 |
| tobiash | the question is who wants to try this in production ;) | 13:18 |
| mordred | what could possibly go wrong? :) | 13:18 |
| tobiash | mordred: but I had the same ENV trick in mind, so that would also affect/improve ansible memory usage | 13:19 |
| tobiash | I think I'll try that after the py37 experiment | 13:19 |
| pabelanger | SpamapS: Yah, in fact, that is what I am doing today. So, likely okay just having cherrypy serve it, and deal with it if / when becomes an issue | 13:34 |
| *** sdake has quit IRC | 13:47 | |
| *** jpena|lunch is now known as jpena | 13:48 | |
| *** rfolco_doctor is now known as rfolco | 13:58 | |
| openstackgerrit | Quique Llorente proposed openstack-infra/zuul master: Escape jinja2 stuff from inventory https://review.openstack.org/633930 | 13:59 |
| quiquell|rover | tobiash: ^ all good now | 13:59 |
| tobiash | quiquell|rover: thanks, will look later | 13:59 |
| quiquell|rover | cool thanks | 13:59 |
| *** quiquell|rover is now known as quique|rover|eat | 14:06 | |
| *** bjackman has quit IRC | 14:23 | |
| *** irclogbot_3 has joined #zuul | 14:31 | |
| *** bjackman has joined #zuul | 14:33 | |
| *** gtema has quit IRC | 14:35 | |
| *** sdake has joined #zuul | 14:35 | |
| *** quique|rover|eat is now known as quiquell|rover | 14:38 | |
| *** bjackman has quit IRC | 14:42 | |
| *** bjackman has joined #zuul | 14:42 | |
| *** bjackman has quit IRC | 14:48 | |
| *** sdake has quit IRC | 14:54 | |
| *** ParsectiX has joined #zuul | 14:59 | |
| *** sdake has joined #zuul | 15:09 | |
| tobiash | mordred: according to my graphs the scheduler seems to be leaky too | 15:15 |
| tobiash | or it's really just java-like behavior... | 15:16 |
| *** quiquell|rover is now known as quiquell|off | 15:27 | |
| *** gtema has joined #zuul | 15:33 | |
| *** zbr has joined #zuul | 15:35 | |
| *** swest has quit IRC | 15:37 | |
| *** zbr|ssbarnea has quit IRC | 15:37 | |
| tobiash | mordred: -rwxr-xr-x. 1 root root 4.3M Feb 7 15:42 /usr/local/lib/libjemalloc.so.2 | 15:56 |
| tobiash | 4.3M for a malloc implementation | 15:56 |
| tobiash | impressive | 15:56 |
| tobiash | mordred: so now I've restarted one executor as reference, updated one to py37 and one to py37+jemalloc | 15:58 |
| tobiash | looking forward to the graphs tomorrow... | 15:58 |
| clarkb | tobiash: with the scheduler I think it is more that as it expands to load configs python can never really release that memory again | 15:59 |
| tobiash | hrm, that graph is continuously rising during the day and rising again the next day | 16:01 |
| tobiash | just like it wouldn't reuse that memory | 16:01 |
| clarkb | tobiash: http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=64792&rra_id=all that one? | 16:03 |
| clarkb | it looks pretty stable | 16:03 |
| openstackgerrit | Fabien Boucher proposed openstack-infra/zuul master: Propose the URL driver to implement an URL change Require Filter https://review.openstack.org/635554 | 16:03 |
| tobiash | yeah, maybe it stabilizes once it doesn't get more memory ;) | 16:03 |
| tobiash | hm, ok my executor has a limit of 4gb currently | 16:05 |
| tobiash | maybe it would stabilize around that level | 16:05 |
| *** pcaruana has quit IRC | 16:21 | |
| *** themroc has quit IRC | 16:22 | |
| *** chkumar|ruck has quit IRC | 16:38 | |
| *** chandankumar has joined #zuul | 16:39 | |
| *** saneax has quit IRC | 16:40 | |
| dmsimard | Is the zuul inventory file available during the runtime of a job ? | 16:42 |
| dmsimard | I know the Ansible vars are available, I mean the literal file :) | 16:42 |
| jkt | so, I'm playing with a pre-run playbook to basically call `git submodule update --init --recursive`, after fixing the repo URLs etc | 16:43 |
| jkt | I understand that I have to add all dependant projects into tenant config and job's required_project stanza | 16:43 |
| dmsimard | Ah, got it: {{ zuul.executor.log_root }}/zuul-info/inventory.yaml | 16:44 |
| jkt | my problem is that even that initial step where zuul-executor prepares these git repos takes 2.5 minutes on this VM which is allegedly already backed by SSDs | 16:45 |
| clarkb | dmsimard: ya a privileged play copies it to that location | 16:45 |
| clarkb | dmsimard: so the running job doesn't read that exact file (eg don't expect updates to it to reflect in the ansible runes) | 16:45 |
| openstackgerrit | Fabien Boucher proposed openstack-infra/zuul master: URLTrigger driver time based https://review.openstack.org/635567 | 16:48 |
| corvus | dmsimard: https://zuul-ci.org/docs/zuul/user/jobs.html#var-zuul.executor.inventory_file | 16:50 |
| corvus | jkt: i think we've talked about this, but just in case -- is your executor's job_dir on the same filesystem as it's git_dir? https://zuul-ci.org/docs/zuul/admin/components.html#attr-executor.job_dir and https://zuul-ci.org/docs/zuul/admin/components.html#attr-executor.git_dir | 16:53 |
| jkt | corvus: we talked about this, but I don't remember that particular point | 17:03 |
| jkt | corvus: and yup, same fs | 17:04 |
| jkt | doh, nope | 17:04 |
| tobiash | jkt: not same fs is a real performance killer ;) | 17:05 |
| jkt | ok, actually, yes, this is all rootfs, so no tmpfs for /tmp | 17:05 |
| tobiash | jkt: also same mount point (if you're runnning containerized)? | 17:05 |
| jkt | tobiash: I am not using containers; this is on centos 7, with an xfs / and nothing mounted at /tmp | 17:06 |
| tobiash | ok | 17:06 |
| jkt | so both /var/lib/zuul/executor-git and /tmp should be rootfs | 17:06 |
| tobiash | jkt: maybe this could help: https://review.openstack.org/635496 | 17:06 |
| jkt | tobiash: have you noticed that you guys come up with a solution to any of my problems just hours before I need one? :) | 17:07 |
| tobiash | :) | 17:07 |
| *** sdake has quit IRC | 17:07 | |
| tobiash | jkt: how many repos does your job have? | 17:07 |
| jkt | about a zillion, well, actually, ~150 | 17:08 |
| jkt | :( | 17:08 |
| tobiash | that change parallelizes the initial git fetch of all repos of the job | 17:08 |
| tobiash | ok, so then I'd expect quite some speedup in your case | 17:08 |
| jkt | the biggest offender are the Boost C++ libraries | 17:09 |
| jkt | their build system hasn't liked my attempts at pruning the dep tree *at all* | 17:09 |
| corvus | tobiash: left a question on that about whether we can eliminate the config option :) | 17:09 |
| jkt | also, their submodules point to ../relative/whatever.git instead of just ../relative/whatever | 17:10 |
| tobiash | corvus: fine for me, I just wanted to retain status quo with the config option | 17:10 |
| *** sdake has joined #zuul | 17:10 | |
| jkt | this is what I have now, http://paste.openstack.org/show/744695/ | 17:11 |
| tobiash | I guess we could start with 1*cpus? | 17:11 |
| corvus | tobiash: ah, yeah, i expect that it would be okay to improve things by default for people. that sounds good. | 17:11 |
| *** pwhalen has quit IRC | 17:12 | |
| tobiash | corvus: I just need to think about how to get the correct number of cpus in my cgroups restricted depoyment | 17:13 |
| tobiash | multiprocessing will return 16, but the executor is restricted to 8 | 17:13 |
| jkt | so, the numbers: 1min 21s for updating all repos in /var/lib/zuul/executor-git/ , which should be a no-op in this case because there are no changes | 17:13 |
| tobiash | but I have that same problem with the starting builds sensor anyway | 17:14 |
| jkt | 30s for cloning to /tmp | 17:14 |
| tobiash | jkt: the executor doesn't know upfront if it's a noop, but I guess the parallelization could make this 1min 21s much smaller | 17:14 |
| jkt | 10s for checking out the respective branches in there | 17:14 |
| tobiash | so yes, that sounds like the parallelization will help there quite a lot | 17:15 |
| jkt | sounds like I should build from git once again :) | 17:16 |
| jkt | does the REST API of zuul-web need that JS Build stack? | 17:16 |
| jkt | or is that just for the dashboard web app? | 17:16 |
| tobiash | zuul-web serves the dashboard webapp | 17:17 |
| tobiash | or do you offload it to apache? | 17:18 |
| jkt | I don't offload that, nope | 17:18 |
| jkt | I'm wondering about the /api/tenant/.../ etc | 17:18 |
| tobiash | the api works without, but is quite useless without the ui | 17:19 |
| jkt | ah, I would probably lose that console streamer as well, then | 17:21 |
| tobiash | yes | 17:22 |
| tobiash | and the status page | 17:22 |
| *** pwhalen has joined #zuul | 17:23 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Optionally parallelize update threads https://review.openstack.org/635496 | 17:29 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Optionally parallelize update threads https://review.openstack.org/635496 | 17:30 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Parallelize update threads https://review.openstack.org/635496 | 17:31 |
| openstackgerrit | Mohammed Naser proposed openstack-infra/nodepool master: docker: don't daemonize when starting images https://review.openstack.org/635584 | 17:39 |
| *** jpena is now known as jpena|off | 17:59 | |
| *** jesusaur has quit IRC | 18:07 | |
| *** gtema has quit IRC | 18:11 | |
| tobiash | corvus, SpamapS: do we want to target the multi ansible spec for next week? I think I've addressed all comments so far. | 18:19 |
| SpamapS | tobiash: Where's that spec? It doesn't seem to have topic:multi-ansible | 18:21 |
| SpamapS | or did it merge? | 18:21 |
| tobiash | SpamapS: https://review.openstack.org/623927 | 18:22 |
| tobiash | I'll set the topic | 18:22 |
| SpamapS | thanks, just couldn't find it ;) | 18:23 |
| tobiash | :) | 18:23 |
| corvus | tobiash, SpamapS: to be clear, i think the utility of having the executor run the install script on startup isn't for quick-start -- i agree, that should be at cointainer build-time. it's for the more traditional pip-install configuration. so that someone who does 'pip instal zuul' has it almost as easy as someone who does 'docker run zuul' | 18:24 |
| tobiash | corvus: I can clarify that in the spec | 18:25 |
| corvus | i mention it here because i think the three of us have been going back and forth on the spec :) | 18:26 |
| tobiash | ok :) | 18:26 |
| SpamapS | I see. | 18:26 |
| corvus | basically it makes "pip install zuul; zuul-executor" continue to work | 18:27 |
| SpamapS | A fair point, and a fine bridge for folks who aren't ready to containerize stuff. | 18:27 |
| corvus | i think we still get new users not using containers. though we've gotten a lot of quick-start users showing up. | 18:29 |
| SpamapS | Yeah, I see why it's a thing and I don't mind keeping that working for a while. | 18:30 |
| corvus | i could see dropping auto-install as a teaching method -- to make sure that people know that's what's going on. but it feels like maybe that may not be worth the additional friction for new users. | 18:32 |
| electrofelix | hughsaunders: are you still working on nodepool-agents plugin for jenkinsci to make use of nodepool? wondering if you're around to chat about some questions I have about making use of it (also asked in https://groups.google.com/d/msg/jenkinsci-dev/VA2i-_L350E/uVhO2xRLGQAJ) | 18:34 |
| tobiash | and probably even existing users | 18:35 |
| SpamapS | honestly | 18:39 |
| SpamapS | this feels almost major version-ish | 18:39 |
| SpamapS | It's more than a feature | 18:39 |
| corvus | no argument here | 18:40 |
| corvus | if folks think multi-ansible is v4, and zookeeper-everywhere is v5, that wfm. :) | 18:40 |
| corvus | tobiash: so yeah, i'll issue a last call for review on the spec monday with the aim to merge it by the end of the week if there are no major revisions required. sound good? | 18:42 |
| clarkb | the problem is wheels not running installation code right? so we somehow have to install portable virtualenvs (or something like that) into the wheel or have the wheel install (or maybe process start up) do the setup? | 18:42 |
| tobiash | corvus: sounds good :) | 18:42 |
| clarkb | I guess the easiest things is having the executor do it on first run | 18:43 |
| clarkb | and it can noop if it is already done for it | 18:43 |
| *** ParsectiX has quit IRC | 18:44 | |
| tobiash | clarkb: yes, it's similarly described in the spec | 18:46 |
| clarkb | yup, just making sure I understand. Is the contention question around whether or not zuul should do the installation on startup? I don't really think there is a straightforward method that avoids that (other than requiring a multistep install process) | 18:47 |
| *** jesusaur has joined #zuul | 18:52 | |
| *** sdake has quit IRC | 18:58 | |
| tobiash | clarkb: we'll support both, manual installation (by calling something like zuul-manage-ansible) and on startup installation (for the lazy pip install use case) | 19:01 |
| *** remi_ness has joined #zuul | 19:04 | |
| *** ParsectiX has joined #zuul | 19:28 | |
| SpamapS | ugh, looks like my pbrx docker builds are failing now because of something that changed in alpine | 19:29 |
| openstackgerrit | Merged openstack-infra/zuul master: Update git connection logging https://review.openstack.org/635204 | 19:29 |
| * SpamapS accelerates switch to dockerfile builds | 19:29 | |
| *** saneax has joined #zuul | 19:38 | |
| *** jesusaur has quit IRC | 19:40 | |
| *** jesusaur has joined #zuul | 19:41 | |
| *** sdake has joined #zuul | 19:47 | |
| *** sdake has quit IRC | 19:49 | |
| *** pwhalen has quit IRC | 19:50 | |
| *** sdake has joined #zuul | 19:51 | |
| openstackgerrit | Mohammed Naser proposed openstack-infra/zuul master: docker: add state folder https://review.openstack.org/635618 | 19:52 |
| *** sdake has quit IRC | 19:53 | |
| *** sdake has joined #zuul | 19:54 | |
| openstackgerrit | Mohammed Naser proposed openstack-infra/zuul master: docker: start process in foreground https://review.openstack.org/635619 | 19:55 |
| mordred | mnaser: ^^ lgtm ... for the state folder - should we mark that as a volume? | 19:58 |
| mnaser | mordred: it looks like most of what goes in the state folder is command control socket, pid files and things that aren't really as much of state | 19:59 |
| tobiash | mordred: I'm just asking myself how I could overlook the foreground thingy ^ | 19:59 |
| mnaser | so i think it should live within the container (imho) | 19:59 |
| mordred | tobiash: yah - same here | 19:59 |
| mordred | mnaser: kk | 19:59 |
| tobiash | mordred: how did the quickstart then work? I'd expected if zuul daemonizes pid 1 exits and that should exit the container... | 20:00 |
| mordred | tobiash: maybe because we're using dumb-init it still works? | 20:00 |
| tobiash | ah, that's the reason | 20:00 |
| tobiash | dumb-init is pid1 | 20:00 |
| mordred | mnaser: actually - at least on the scheduler, /var/lib/zuul contains the keys, which are precious and shoudl go on a volume | 20:01 |
| mnaser | tobiash, mordred quick start works because someone explicitly dropped the -d in the command in docker-compose | 20:01 |
| mnaser | mordred: ah that's true | 20:01 |
| mnaser | i mean tbh i'd feel that /var/lib/zuul/.ssh is the real thing to use in this case | 20:02 |
| mordred | mnaser: no - I mean the per-project encryption keys | 20:02 |
| mnaser | oh | 20:02 |
| mnaser | i was thinking ssh keys | 20:02 |
| mordred | mnaser: there's /var/lib/zuul/keys which have those | 20:02 |
| mnaser | in that case volume makes a lot of sense.. assuming 'volume' does a mkdir | 20:03 |
| tobiash | mnaser: then you could just completely drop the command in the docker-compose for zuul-web and executor | 20:03 |
| mordred | mnaser: I believe it does - I think it makes the mount point | 20:03 |
| mordred | tobiash: ++ | 20:03 |
| mnaser | ok well i can revise the patch and update the quickstart | 20:03 |
| mordred | cool | 20:03 |
| tobiash | mnaser: yes, volume creates that dir | 20:03 |
| tobiash | mnaser: thanks | 20:04 |
| mordred | mnaser: yeah - thanks - great improvements | 20:04 |
| SpamapS | thank god for dumb-init eh? ;-) | 20:04 |
| tobiash | I'm using tini | 20:05 |
| tobiash | but that's basically doing the same | 20:05 |
| openstackgerrit | Mohammed Naser proposed openstack-infra/zuul master: docker: start process in foreground https://review.openstack.org/635619 | 20:07 |
| openstackgerrit | Mohammed Naser proposed openstack-infra/zuul master: docker: add state folder https://review.openstack.org/635618 | 20:07 |
| mnaser | mordred, tobiash ^ added depends-on on the nodepool change too | 20:07 |
| mordred | mnaser: ++ | 20:07 |
| *** remi_ness has quit IRC | 20:07 | |
| tobiash | mnaser: btw, depends-on with changeid is deprecated ;) | 20:11 |
| mnaser | tobiash: i know but i'm old school and lame | 20:11 |
| mnaser | :) | 20:11 |
| mnaser | another fun one | 20:13 |
| mnaser | zuul-fingergw tries to drop perms into user 'zuul' | 20:13 |
| tobiash | ... which doesn't exist... | 20:13 |
| mnaser | http://paste.openstack.org/show/744710/ | 20:14 |
| tobiash | mnaser: I think that should be made optional. e.g. when running in openshift fingergw would run on an unprovileged port with perms already dropped | 20:15 |
| clarkb | shouldn't the zuul user exist in the imges though? | 20:15 |
| corvus | i added the -d in the compose file, because the quick-start was made when we built pbrx containers which didn't use -d | 20:16 |
| corvus | the zuul user does not exist in the images | 20:16 |
| corvus | it runs as "root" | 20:16 |
| clarkb | shoul we create the zuul user? | 20:17 |
| tobiash | I don't think so | 20:17 |
| mnaser | but the root isn't a real root i guess inside a container | 20:17 |
| clarkb | mnaser: depends on whether or not you namespace users but ya | 20:17 |
| *** saneax has quit IRC | 20:17 | |
| tobiash | it just makes things mode complicated if running e.g. in openshift | 20:17 |
| corvus | mordred has discussed possibly creating a zuul user. regardless of that, the immediate issue is that we have an overly aggressive default | 20:17 |
| corvus | i feel like i just spelled out how to fix this | 20:18 |
| corvus | let me dig up irc logs | 20:18 |
| corvus | http://eavesdrop.openstack.org/irclogs/%23zuul/%23zuul.2019-02-01.log.html#t2019-02-01T19:50:09 | 20:20 |
| corvus | from 19:50 through 20:00 | 20:20 |
| clarkb | the problem is that if you assume unprivileged ports then you need a proxy of some sort. If you don't assume privileged ports then you have to do something like the code currently does. Maybe key off the port value? also I should read ^ | 20:20 |
| corvus | i don't see any of the described patches uploaded :( | 20:21 |
| corvus | dkehn: were you planning on pushing up the change to run fingergw in quick-start? | 20:21 |
| clarkb | ah yup I think the described fix would work | 20:22 |
| pabelanger | tobiash: what issues would use see with zuul user in openshift container? | 20:22 |
| dkehn | corvus: once I get it totally working | 20:23 |
| tobiash | pabelanger: openshift by default starts the process with a random uid and gid=0 which has no relationship to the passwd in the container. Further you cannot switch the user in this case. | 20:23 |
| tobiash | so if possible we should not make any assumptions on the users. If we can manage this the image will be generic enough to run anywhere | 20:24 |
| clarkb | do we also need to override the port for running in a container or will we assume privelege within the container network namespace for the current user? | 20:24 |
| clarkb | that assumption is likely safe given how many people use docker | 20:24 |
| tobiash | granted the executor needs privileges, but the others don't and should imho also work with default restrictions | 20:24 |
| tobiash | in openshift you need to run fingergw on an unprivileged port like 1079 and have the service dispatch from 79 to 1079 | 20:25 |
| pabelanger | tobiash: ah, good to know. When I was trying to run docker nodepool from ansible, I ended up crafting the following systemd execstart: http://paste.openstack.org/show/744712/ passing -u flag, because docker image (pbxr) was root. This was the only way I could get volumes to properly work, as they had permissions of nodepool:nodepool (1001). Since then, I haven't really looped back to trying to run docker under | 20:26 |
| pabelanger | systemd | 20:26 |
| pabelanger | I always thought, if user inside container was nodepool, I could drop the -u flag | 20:26 |
| pabelanger | (but likely wrong) | 20:27 |
| *** sshnaidm is now known as sshnaidm|off | 20:29 | |
| tobiash | when working in openshift folders that need to be written by zuul need to be either mounted as a volune or chowned <foo>:0 and g+w if it's inside the container rootfs (a process has uid=random and gid=0 by default) | 20:29 |
| tobiash | ftr, more info on that topic is here: https://docs.okd.io/latest/creating_images/guidelines.html#openshift-specific-guidelines | 20:31 |
| *** sdake has quit IRC | 20:31 | |
| tobiash | most important the chapter "Support Arbitrary User IDs" | 20:31 |
| *** sdake has joined #zuul | 20:32 | |
| pabelanger | tobiash: yah, I don't believe I setup any specific docker volumes via docker, was only trying to bindmount directly to filesystem | 20:33 |
| tobiash | in the end a docker volume also just gets bind mounted into the filesystem | 20:35 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Remove default user for fingergw https://review.openstack.org/635632 | 20:37 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Remove default zookeeper hosts https://review.openstack.org/635633 | 20:37 |
| corvus | i think those should be in 3.7, so i set a WIP on the first, but we can go aheand and review/discuss them | 20:38 |
| mnaser | if you go with the github application route, how do you 'add' the ssh key to clone things (or does it clone using the api token or something?) | 20:41 |
| tobiash | mnaser: zuul doesn't use ssh in that case | 20:42 |
| mnaser | tobiash: ok so that option should technically be ignored when not using a webhook (perhaps i should push up a doc change)? | 20:43 |
| tobiash | mnaser: it gets per installation tokens on demand that expire every hour | 20:43 |
| mnaser | https://zuul-ci.org/docs/zuul/admin/drivers/github.html#attr-<github%20connection>.sshkey | 20:43 |
| mnaser | this one that is | 20:43 |
| tobiash | I wonder why there is even a ssh key | 20:44 |
| tobiash | because even with the api token method it should use that and https | 20:44 |
| mnaser | tobiash: the web hook thing says you need a user with the ssh key | 20:44 |
| mnaser | https://zuul-ci.org/docs/zuul/admin/drivers/github.html#web-hook | 20:44 |
| tobiash | have to re-read the source | 20:44 |
| pabelanger | is that the ssh key you define, if not using github app? | 20:44 |
| tobiash | yes | 20:45 |
| mnaser | so the docs should probably clarify that | 20:45 |
| mnaser | i guess | 20:45 |
| tobiash | mnaser: so rereading the code, yes when using api token you need the ssh key | 20:46 |
| tobiash | mnaser: but more important, when using app key a defined ssh key will break you | 20:46 |
| openstackgerrit | Mohammed Naser proposed openstack-infra/zuul master: doc: clarify sshkey option usage in github connection https://review.openstack.org/635640 | 20:47 |
| mnaser | tobiash: ^ | 20:47 |
| mnaser | also mordred mind +w https://review.openstack.org/#/c/635584/ to unblock similar zuul change (i added a depends-on) | 20:48 |
| tobiash | mnaser: https://git.zuul-ci.org/cgit/zuul/tree/zuul/driver/github/githubconnection.py#n969 | 20:48 |
| mnaser | yep thats what i noticed too | 20:48 |
| tobiash | if you declare an ssh key, it is taken regardless of the auth method | 20:48 |
| tobiash | and since you cannot attach an ssh key to an app this will break zuul | 20:48 |
| corvus | mnaser, tobiash: if you're embarking on the '-d' route, remember that has implications for logging | 20:49 |
| corvus | mordred: ^ | 20:49 |
| tobiash | we probably should document/verify conflicting settings in the github driver | 20:49 |
| tobiash | corvus: which implications do you mean? I think I forgot them | 20:50 |
| corvus | mnaser, tobiash, mordred: tell me you've thought through the logging issue; otherwise maybe we ought to give those changes a little more thoughtL | 20:50 |
| mnaser | are we talking about docker implications or zuul implications | 20:51 |
| corvus | both | 20:51 |
| tobiash | corvus: do you mean the no-logging-on-invalid-log-config-problem? | 20:51 |
| corvus | tobiash: i mean the fact that "-d" doesn't just mean run-in-foreground | 20:52 |
| corvus | the "d" in "-d" stands for "debug" | 20:52 |
| mnaser | o | 20:52 |
| corvus | so that change is actually "run zuul in full debug mode, in the foreground" | 20:52 |
| mnaser | okay, that's a different story, i was under the impression it was foreground | 20:52 |
| corvus | it's both, because zuul was written before containers existed, and this option is as old as zuul :) | 20:53 |
| tobiash | hrm, we're running zuul in -d mode since the beginning :) | 20:53 |
| tobiash | but we're still running it in debug logging mode... | 20:53 |
| mnaser | https://github.com/openstack-infra/zuul/blob/485f1205a358c4c2297967ca70454d923b8c7b04/zuul/cmd/__init__.py#L141-L145 | 20:53 |
| mnaser | L145 is where we go debug | 20:54 |
| corvus | i'm generally in favor of the change, but i'd like *someone* to assure me they've worked through the implications :) | 20:54 |
| mnaser | i feel like the right thing™ is to add a -f option | 20:54 |
| mnaser | having said that, i don't think i have time to do the small rewrite to add -f and make -d do -f + debug | 20:56 |
| mnaser | though it seem relatively trivial | 20:56 |
| pabelanger | I run -d with log_config setup in zuul.conf, and belive logging works correctly | 20:56 |
| tobiash | ah, we're running with log config: https://github.com/openstack-infra/zuul/blob/485f1205a358c4c2297967ca70454d923b8c7b04/zuul/cmd/__init__.py#L137 | 20:56 |
| tobiash | so we need to think about what happens with no log config and -d | 20:57 |
| mnaser | you get debug logging to stdout (that's what i have right now) | 20:57 |
| tobiash | so judging from the code the only difference I see is debug vs non-debug logging | 20:58 |
| tobiash | I think I would have noticed other problems if there were any | 21:00 |
| tobiash | I know that e.g. signal handling and command socket work with -d | 21:01 |
| pabelanger | it also worked well under systemd, been using it for a while in local testing | 21:01 |
| tobiash | corvus: so the question is, is it ok to have debug logging in the container if no log config is configured? | 21:01 |
| corvus | tobiash: i think we'd probably want the normal (info i think) logging -- same as you'd get running a daemon with no logging config | 21:03 |
| corvus | so i think that means mnaser's suggestion of splitting the arg into f and d may be the best path | 21:03 |
| tobiash | ok, I'll push up a change that adds -f with info and leaves -d as it is (except the description) | 21:03 |
| corvus | then Dockerfile should use the foreground option only | 21:07 |
| tobiash | kk | 21:17 |
| tobiash | corvus: do we want to retain debug logging in the quick start? | 21:20 |
| tobiash | or switch this to info too? | 21:20 |
| corvus | tobiash: i could go either way, but maybe let's try info. | 21:20 |
| corvus | if we get a lot of people showing up with questions and we say "enable debug logging" maybe we switch :) | 21:21 |
| tobiash | ... or improve logging | 21:21 |
| corvus | even better | 21:21 |
| tobiash | so I think it's even beneficial to use info. Then we get a better feeling if info is sufficient for finding problems. | 21:22 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Add foreground option https://review.openstack.org/635649 | 21:25 |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: [WIP] web: add tenant and project scoped, JWT-protected actions https://review.openstack.org/576907 | 21:26 |
| corvus | tobiash: i think the same situation applies in nodepool | 21:28 |
| tobiash | corvus: already on it | 21:28 |
| corvus | cool | 21:28 |
| corvus | i wiped 635584 | 21:29 |
| *** sdake has quit IRC | 21:29 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: docker: don't daemonize when starting images https://review.openstack.org/635584 | 21:33 |
| clarkb | tobiash: my only concern was that side effecting self.args like that might not be safe, but we already use that method of setting nodaemon elsewhere so should be fine. +2 | 21:34 |
| tobiash | clarkb: I had the same thoughts but this way was the way of least impact ;) | 21:35 |
| clarkb | tobiash: https://review.openstack.org/#/c/635584/2 comment on that though | 21:36 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: docker: don't daemonize when starting images https://review.openstack.org/635584 | 21:36 |
| *** sdake has joined #zuul | 21:36 | |
| tobiash | clarkb: updated | 21:37 |
| clarkb | +2 thanks | 21:37 |
| tobiash | clarkb, corvus: not urgent, but it would be great if you could add 616306 (resource usage stats) to your review queue. I think that would be useful for the openstack deployment too | 21:39 |
| clarkb | ya I'll add it to my todo list | 21:40 |
| tobiash | thanks :) | 21:46 |
| *** sdake has quit IRC | 21:51 | |
| *** sdake has joined #zuul | 21:55 | |
| *** sdake has quit IRC | 21:56 | |
| *** sdake has joined #zuul | 21:57 | |
| openstackgerrit | Mohammed Naser proposed openstack-infra/zuul master: doc: fix sqlalchemy database url docs path https://review.openstack.org/635670 | 22:17 |
| *** sdake has quit IRC | 22:34 | |
| *** panda is now known as panda|off | 22:59 | |
| *** sdake has joined #zuul | 23:23 | |
| *** ParsectiX has quit IRC | 23:48 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add buildsets page https://review.openstack.org/630041 | 23:52 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add buildset page https://review.openstack.org/630079 | 23:53 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add /{tenant}/buildset/{uuid} route https://review.openstack.org/630078 | 23:53 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add buildset page https://review.openstack.org/630079 | 23:53 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!