| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: web: add OpenAPI documentation https://review.opendev.org/535541 | 01:05 |
|---|---|---|
| *** altlogbot_1 has quit IRC | 02:41 | |
| *** altlogbot_2 has joined #zuul | 02:44 | |
| *** altlogbot_2 has quit IRC | 03:31 | |
| *** altlogbot_1 has joined #zuul | 03:33 | |
| *** raukadah has quit IRC | 04:00 | |
| *** chandankumar has joined #zuul | 04:01 | |
| *** badboy has joined #zuul | 05:22 | |
| *** quiquell has joined #zuul | 06:28 | |
| *** abraverm has quit IRC | 06:53 | |
| *** themroc has joined #zuul | 06:56 | |
| ianw | I feel like "zuul enqueue-ref --tenant openstack --trigger timer --pipeline periodic --project openstack/example_project --ref refs/heads/master" doesn't work | 07:04 |
| ianw | it requires the --newrev, which i'm not sure if it did before? | 07:05 |
| *** altlogbot_1 has quit IRC | 07:13 | |
| *** altlogbot_2 has joined #zuul | 07:19 | |
| tobiash | ianw: as far as I remember it always required newrev, but I didn't use enqueue-ref a lot | 07:28 |
| *** arxcruz|rover is now known as arxcruz | 07:37 | |
| ianw | tobiash: yeah, i'm trying to remember -- the docs say it doesn't | 07:38 |
| ianw | https://zuul-ci.org/docs/zuul/admin/client.html#manual-enqueue-examples | 07:38 |
| tobiash | interesting | 07:39 |
| *** hashar has joined #zuul | 07:42 | |
| *** abraverm has joined #zuul | 07:44 | |
| *** dmellado has quit IRC | 07:44 | |
| *** dmellado has joined #zuul | 07:44 | |
| *** jpena|off is now known as jpena | 07:49 | |
| *** pcaruana has joined #zuul | 08:24 | |
| *** altlogbot_2 has quit IRC | 09:11 | |
| *** altlogbot_2 has joined #zuul | 09:17 | |
| *** zbr is now known as zbr|rover | 09:18 | |
| quiquell | zbr|rover: CI with linter reproducer | 09:40 |
| quiquell | http://logs.rdoproject.org/48/20548/22/check/tox-linters/b6ca1c5/job-output.txt.gz | 09:40 |
| quiquell | please don't add a patch set to the review I am working on it | 09:40 |
| quiquell | just tell me if you find something | 09:40 |
| *** themr0c has joined #zuul | 09:55 | |
| *** themroc has quit IRC | 09:56 | |
| *** gtema has joined #zuul | 10:22 | |
| *** gtema has quit IRC | 10:30 | |
| *** themr0c has quit IRC | 10:34 | |
| *** gtema has joined #zuul | 10:36 | |
| *** bhavikdbavishi has joined #zuul | 10:43 | |
| *** gtema has quit IRC | 11:16 | |
| *** gtema has joined #zuul | 11:17 | |
| *** altlogbot_2 has quit IRC | 11:18 | |
| *** altlogbot_1 has joined #zuul | 11:22 | |
| *** jpena is now known as jpena|lunch | 11:32 | |
| *** rfolco|ruck has joined #zuul | 11:50 | |
| *** bhavikdbavishi has quit IRC | 11:57 | |
| *** rlandy has joined #zuul | 12:11 | |
| *** altlogbot_1 has quit IRC | 12:19 | |
| *** themroc has joined #zuul | 12:20 | |
| *** altlogbot_3 has joined #zuul | 12:21 | |
| *** jpena|lunch is now known as jpena | 12:29 | |
| *** pcaruana has quit IRC | 12:29 | |
| *** gtema has quit IRC | 12:36 | |
| *** altlogbot_3 has quit IRC | 13:15 | |
| *** pcaruana has joined #zuul | 13:16 | |
| *** altlogbot_3 has joined #zuul | 13:17 | |
| *** altlogbot_3 has quit IRC | 13:21 | |
| Shrews | tobiash: what does it take to enable jemalloc on an ubuntu system? | 13:23 |
| Shrews | is it enough to just install libjemalloc? | 13:23 |
| *** altlogbot_1 has joined #zuul | 13:27 | |
| Shrews | maybe using LD_PRELOAD, I guess | 13:28 |
| Shrews | ah, yep. that's how our zuul executors do it | 13:29 |
| tobiash | yes, installing and ld_preload should do it | 13:30 |
| *** ianychoi has quit IRC | 13:48 | |
| *** ianychoi has joined #zuul | 13:52 | |
| *** fdegir has joined #zuul | 13:52 | |
| *** ianychoi has quit IRC | 14:04 | |
| arxcruz | tristanC: around? can we talk tomorrow about the zuul runner? me and rlandy are very interested on it, and we want to test it for tripleo jobs :) | 14:07 |
| pabelanger | morning, if humans are around for reviews this morning. I think we should first land the stack of test improvements at https://review.opendev.org/#/q/status:open+project:zuul/zuul+branch:master+topic:test-improvements | 14:10 |
| pabelanger | then we should be good to open zuul again to merge new features | 14:11 |
| rlandy | arxcruz: tristanC usually shows up at about 6pm my time. I'll ping him then | 14:12 |
| *** quiquell is now known as quiquell|off | 14:15 | |
| mordred | pabelanger: done | 14:16 |
| pabelanger | tyty | 14:17 |
| mordred | pabelanger: thanks for digging in to the flaky tests! | 14:18 |
| *** ianychoi has joined #zuul | 14:18 | |
| pabelanger | np, tobiash was a big help there, I just got the little ones :) | 14:20 |
| pabelanger | but yah, we should be in much position now | 14:20 |
| tobiash | \o/ | 14:28 |
| Shrews | So, it feels to me that maybe something external changed in the Nov-Dec timeframe to cause these nodepool issues. A python update or similar... if you look at the builder mem usage graph, we see the same mem usage jump at that point as in the launchers: http://cacti.openstack.org/cacti/graph.php?action=zoom&local_graph_id=63751&rra_id=0&view_type=tree&graph_start=1525617663&graph_end=1557153663 | 14:42 |
| Shrews | I don't think there has been any change to the builder code that would cause such a jump | 14:43 |
| tobiash | Shrews: did you do the upgrade to bionic at that time? | 14:44 |
| Shrews | tobiash: not sure | 14:44 |
| Shrews | tobiash: no, they are xenial | 14:45 |
| corvus | i'm going to be afk for significant chunks of time this week as i recover from the last 3 weeks and prepare for a presentation coming up soon; i'll try to check in periodically to see if anything is blocked on me, but don't expect a lot out of me this week | 14:47 |
| openstackgerrit | Merged zuul/zuul master: Bump tox jobs timeout to 3600 https://review.opendev.org/656771 | 15:20 |
| clarkb | corvus: weshould consider a release with the memory leak fix | 15:36 |
| corvus | clarkb: i can do that now | 15:36 |
| corvus | opendev is running 7e29b8a9106da95f5e654a88def61e43d0138b52 and there is one non-test change past that: "Fix for yappi output" | 15:38 |
| corvus | that's sort of an advanced-user-only debug tool | 15:38 |
| corvus | should we just tag 7e29b8a9106da95f5e654a88def61e43d0138b52 or master? | 15:39 |
| clarkb | I'm ok with master if yappi fix is the only non test change | 15:41 |
| clarkb | those fixes were exercised against nodepool too aiui | 15:41 |
| *** themroc has quit IRC | 15:41 | |
| *** chandankumar is now known as raukadah | 15:41 | |
| tobiash | i'm fine with either | 15:42 |
| Shrews | i am also fine with a nodepool release (to make pabelanger happy) on commit f58a2a | 15:44 |
| pabelanger | yay | 15:44 |
| Shrews | pabelanger: that does not have your network-per-label change, though | 15:45 |
| Shrews | b/c that just merged | 15:45 |
| corvus | ah cool, i'm glad the logger issue was fixed with an adapter :) | 15:45 |
| pabelanger | Shrews: I'm okay with that for now | 15:46 |
| Shrews | i do hope to restart all nodepool processes today after the jemalloc stuff propogates | 15:47 |
| corvus | for zuul does this look right? git tag -s 3.8.1 cff3bc89c90445a055156f91556afa70ab0ebaeb | 15:47 |
| SpamapS | https://review.opendev.org/587731 <-- hey this looks like an important fix for gear that has been sitting since January. | 15:48 |
| clarkb | lgtm | 15:48 |
| *** abraverm has quit IRC | 15:49 | |
| corvus | SpamapS: oh thanks! | 15:49 |
| SpamapS | corvus:np, I only found it because I had to re-do my gertty and scoped down to just zuul+gear ;) | 15:50 |
| corvus | we will have no release notes for zuul 3.8.1 -- do we want to add a "bug fixes:" note about the memory leak? | 15:51 |
| corvus | (it seems like since that's the main reason for the release, it's probably a worthy enough bug for a note) | 15:51 |
| SpamapS | +1 | 15:51 |
| SpamapS | as a user I love to see bugfix only releases now and then. ;) | 15:52 |
| corvus | for nodepool, does this look right? git tag -s 3.6.0 f58a2a2b68c58f9626170795dba10b75e96cd551 | 15:53 |
| Shrews | yep | 15:54 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Add release note for memory leak https://review.opendev.org/657397 | 15:57 |
| corvus | clarkb, tobiash, SpamapS: ^ | 15:57 |
| corvus | nodepool 3.6.0 pushed | 15:58 |
| corvus | 15:58 < openstackgerrit> Merged opendev/gear master: add missing str to bytes conversion for Python3 https://review.opendev.org/587731 | 15:58 |
| corvus | let's merge that release note and tag the result as zuul 3.8.1 | 16:00 |
| clarkb | +2'd | 16:01 |
| pabelanger | http://logs.openstack.org/13/656213/2/check/zuul-tox-remote/12fbed8/testr_results.html.gz | 16:02 |
| pabelanger | test_command has failed a few times now, I think we'll need to dig into why | 16:03 |
| pabelanger | looks to be related to zuul_stream | 16:03 |
| pabelanger | unfortantely, I don't have much time today (maybe this week) to look into it | 16:03 |
| openstackgerrit | Merged zuul/zuul master: Update timeout_handler to check for path first https://review.opendev.org/656782 | 16:10 |
| Shrews | pabelanger: yeah, i've seen that one lately. it confused me | 16:12 |
| Shrews | i was having issues getting tox remote jobs to run locally before the summit happened | 16:13 |
| *** mattw4 has joined #zuul | 16:25 | |
| *** hashar has quit IRC | 17:01 | |
| *** jpena is now known as jpena|off | 17:10 | |
| openstackgerrit | Merged zuul/zuul master: Make test_playbook more robust https://review.opendev.org/656177 | 17:12 |
| openstackgerrit | Merged zuul/zuul master: Fix race in test_job_node_failure_resume https://review.opendev.org/656178 | 17:24 |
| pabelanger | zuul-maint: if you could add https://review.opendev.org/655474/ to your review pipeline, that should get us per commit wheels for pypi. A discussion can be found at: http://lists.zuul-ci.org/pipermail/zuul-discuss/2019-April/000892.html | 17:25 |
| *** jamesmcarthur has joined #zuul | 17:25 | |
| tobiash | pabelanger: does this job work with unreleased versions? | 17:27 |
| pabelanger | tobiash: yah, it should | 17:31 |
| pabelanger | when I looked at build output of our release job, I didn't see anything of concern | 17:31 |
| pabelanger | we'll propose .dev tarballs, which should be then okay to publish to pypi | 17:31 |
| tobiash | ok | 17:31 |
| *** hashar has joined #zuul | 17:48 | |
| *** jamesmcarthur has quit IRC | 18:00 | |
| openstackgerrit | Merged zuul/zuul master: Update assertNodepoolState() to retry zk requests https://review.opendev.org/656213 | 18:15 |
| *** tjgresha_nope has quit IRC | 18:21 | |
| *** tjgresha has joined #zuul | 18:21 | |
| *** tjgresha has quit IRC | 18:22 | |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Match tag items against containing branches https://review.opendev.org/578557 | 18:30 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Use implied branch matcher for implied branches https://review.opendev.org/640272 | 18:31 |
| pabelanger | tobiash: do you want to review https://review.opendev.org/655588/ | 18:59 |
| pabelanger | https://review.opendev.org/655188/ also a fix for github driver if people want to review | 19:01 |
| *** abraverm has joined #zuul | 19:14 | |
| *** hashar_ has joined #zuul | 19:32 | |
| *** maxamillion_ has joined #zuul | 19:34 | |
| *** quiquell has joined #zuul | 19:39 | |
| *** maxamillion has quit IRC | 19:39 | |
| *** hashar has quit IRC | 19:39 | |
| *** quiquell|off has quit IRC | 19:39 | |
| *** maxamillion_ is now known as maxamillion | 19:39 | |
| *** hashar_ is now known as hashar | 19:49 | |
| pabelanger | starting upgrade to 3.6.0 of nodepool | 20:04 |
| daniel2 | So after 3 months of this dang project, my coworker finally wants me to upgrade nodepool to latest and start working with ansible. | 20:09 |
| daniel2 | :D | 20:10 |
| pabelanger | daniel2: nice | 20:12 |
| openstackgerrit | Merged zuul/zuul master: Add release note for memory leak https://review.opendev.org/657397 | 20:15 |
| pabelanger | So, have had a 2nd user not get the depends-on syntax correct for github | 20:15 |
| pabelanger | what they end up doing is, copying what they see in github ui | 20:16 |
| pabelanger | so, in this case, by original commit had, depends-on: https://github.com/ansible/awx/pull/3799 | 20:16 |
| pabelanger | but github ui renders it as: depends-on: ansible/awx#3799 | 20:17 |
| pabelanger | so, users then copy that syntax for commit / comment, but zuul doens't work as expected | 20:17 |
| pabelanger | tobiash: maybe you have had this issue?^ | 20:17 |
| Shrews | pabelanger: oh, b/c the github ui turns that into a clickable link, yeah? | 20:17 |
| pabelanger | Shrews: Yah | 20:18 |
| pabelanger | github magic | 20:18 |
| pabelanger | https://github.com/ansible-network/sandbox/pull/32 | 20:18 |
| pabelanger | that is the fqdn for github.com | 20:18 |
| pabelanger | in commit message | 20:18 |
| pabelanger | but rendered in short form | 20:18 |
| tobiash | Yeah, it's bad that github changes this | 20:20 |
| corvus | pabelanger: it may be worth making some documentation you can give to users to highlight that? | 20:20 |
| tobiash | I get frequently the same question | 20:20 |
| pabelanger | corvus: agree | 20:20 |
| *** evgenyl has joined #zuul | 20:30 | |
| evgenyl | Hi everyone, is this zuul repo on docker hub an official one? https://hub.docker.com/r/zuul/ | 20:38 |
| evgenyl | I'm wondering if there are any plans on adding tags for the images and how safe is it to always take the latest? | 20:45 |
| corvus | evgenyl: yes it is! | 20:47 |
| corvus | evgenyl: :latest corresponds to the latest commit. we generally think that zuul is safe to run continuously deployed; several of us run it that way. | 20:48 |
| corvus | evgenyl: we do plan on tagging releases there as well, but we haven't set that up yet. | 20:48 |
| *** pcaruana has quit IRC | 20:51 | |
| openstackgerrit | Merged zuul/zuul master: Make paused status bar blue https://review.opendev.org/655588 | 20:52 |
| evgenyl | corvus: Awesome; do you have any recommendations on zuul installation? I'm thinking about taking the playbooks + docker-compose configs and adjusting them for my needs, or are there some other ready-to-use playbooks available? | 21:02 |
| *** mattw4 has quit IRC | 21:05 | |
| corvus | evgenyl: pabelanger has some playbooks in the "windmill" project, but i don't think they use images (yet?) -- i think they run from pypi releases. i know some folks are running from images, but i'm not sure if they have published playbooks yet. | 21:16 |
| corvus | evgenyl: i think starting with the docker-compose file is a fine option. | 21:16 |
| *** Miouge has quit IRC | 21:16 | |
| evgenyl | corvus: I see, thank you! | 21:21 |
| *** mattw4 has joined #zuul | 21:22 | |
| pabelanger | yah, I should update testing for docker images again | 21:24 |
| homeski | I'm setting up the zuul all-in-one server | 21:41 |
| homeski | I'm see errors in executor.log | 21:41 |
| homeski | with permission denied when trying to clone the repos I have set in the tenant config... Yet when I manually try to use the key and ssh command that is failing, it works | 21:42 |
| homeski | any tips? | 21:42 |
| *** mattw4 has quit IRC | 21:43 | |
| *** mattw4 has joined #zuul | 21:44 | |
| *** hashar has quit IRC | 21:45 | |
| homeski | actually.. I set executor.private_key_file thinking that is the key being used, but after re-reading docs I believe I'm wrong :) | 21:50 |
| homeski | I set connection.sshkey for the gerrit driver, but I don't think it's actually being used | 21:53 |
| *** jamesmcarthur has joined #zuul | 22:22 | |
| homeski | finally got it going... I had git 1.8.x installed.. looks like I needed 2.x | 22:27 |
| *** jamesmcarthur has quit IRC | 22:51 | |
| SpamapS | homeski:that's a good callout.. we may want to list that in the doc | 22:51 |
| homeski | I'm trying on centos7 .. looks like instead of `sudo yum install -y git` you need `sudo yum install -y git2u-all` | 22:52 |
| homeski | when using the ius repo | 22:53 |
| clarkb | We run our zuul on xenial which is older git too. I dont think newer git is required | 22:54 |
| corvus | we have 2.7.4 on our executors | 22:55 |
| homeski | so all the logs look fine right now .. executor, scheduler, zookeeper, nodepool | 23:00 |
| clarkb | corvus: huh is that something ubunut keeps up to date maybe? | 23:01 |
| *** zbr|rover has quit IRC | 23:01 | |
| homeski | when I reply 'recheck' on a zuul-job repo, I see scheduler.log spit out "Updating ...' | 23:01 |
| homeski | but that's it, doesn't seem the node gets launched, or any updates to gerrit by zuul from there | 23:01 |
| homeski | ah nvm, I'm, getting errors when loading zuul-config yaml `extra keys not allowed @ data['failure']['mysql']` | 23:03 |
| corvus | homeski: also in the web interface you should be able to see that error by clicking on the little alarm bell in the top right corner, after navigating to any page for that tenant | 23:04 |
| corvus | (eg, the status page for the tenant) | 23:05 |
| homeski | ok thx, yeah I'm currently mucking around in the terminal | 23:05 |
| corvus | yeah, it's the same info; mostly i wanted to make sure you knew that wasn't the *only* way to see errors like that :) | 23:06 |
| corvus | clarkb: i'll tag the zuul release tomorrow and send out release announcements for both | 23:07 |
| homeski | got a zuul 'Build succeeded.' comment :) | 23:12 |
| homeski | in the quickstart guide there are 'code-review, verified, workflow' labels | 23:13 |
| homeski | but when installing gerrit from scratch, there is only 'code-review' | 23:13 |
| homeski | is it further setup of gerrit to get those, it | 23:13 |
| homeski | it's not documented | 23:14 |
| tristanC | rlandy: arxcruz: o/ | 23:17 |
| rlandy | tristanC: hi | 23:22 |
| rlandy | tristanC: we were looking at the zuul runner review again | 23:23 |
| rlandy | weshay: ^^ | 23:23 |
| rlandy | as another option to the reproducer the ci team has worked on | 23:23 |
| rlandy | tristanC: it looks like there are a still a lot of open questions there | 23:24 |
| corvus | homeski: yes, the zuul-quickstart sets up the Verified label | 23:25 |
| rlandy | tristanC: this review ... https://review.opendev.org/#/c/644770/ | 23:25 |
| corvus | homeski: https://opendev.org/zuul/zuul/src/branch/master/doc/source/admin/examples/playbooks/setup.yaml#L49-L67 and https://opendev.org/zuul/zuul/src/branch/master/doc/source/admin/examples/playbooks/project.config#L71-L78 are relevant | 23:26 |
| homeski | thanks! | 23:27 |
| tristanC | rlandy: yes, the final zuul-runner cli isn't fully specified yet | 23:27 |
| tristanC | rlandy: corvus find it too complicated, and I don't know yet how to make it simpler | 23:29 |
| rlandy | tristanC: lol - that is exactly the complaint with the reproducer - people find it complicated | 23:29 |
| rlandy | they liked it when we could juts hand them a few playbooks to run | 23:29 |
| rlandy | but we tried that approach and it's not workable | 23:30 |
| tristanC | rlandy: https://review.opendev.org/607082 would give you the list of playbooks to run to reproduce a job | 23:31 |
| rlandy | tristanC: ok - I'll leave arxcruz some notes with your answers above. But it looks like we will hot similar problems that we had to solve with the reproducer | 23:33 |
| tristanC | rlandy: i can have a look at the stack again today, and can provide instructions how to set it up with your reproducer env | 23:33 |
| rlandy | thanks | 23:33 |
| weshay | rlandy I see.. :) | 23:36 |
| rlandy | weshay: yep - looks like we will hit similar issues | 23:37 |
| tristanC | rlandy: weshay: well, fwiw, the current zuul-runner can reproduce a job with just 2 command (pip install && zuul-runner ...) without needing any services or setup | 23:39 |
| weshay | that is a bold statement :) | 23:41 |
| rlandy | I guess so could w - run bash script | 23:42 |
| rlandy | it's my understanding that to get the zuul runner to work in all the same situations, the system won;t be any less complicated | 23:44 |
| weshay | tristanC so when we attempt w/ tripleo we can bug you for support? | 23:48 |
| tristanC | weshay: yes sure | 23:49 |
| weshay | k.. we'll poke at it over the next few days | 23:49 |
| tristanC | weshay: i'll update the test-day document with clear instructions | 23:50 |
| weshay | tristanC rock on | 23:51 |
| *** rlandy has quit IRC | 23:59 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!