| mnaser | pabelanger: neat! | 00:02 |
|---|---|---|
| mnaser | i dont think 8 might be as bad | 00:03 |
| pabelanger | I think DIB is doing RHEL8, but haven't looked. ianw would know | 00:04 |
| ianw | yes, but only from upstream images | 00:04 |
| pabelanger | yah, figured will be some time for centos-minimal | 00:04 |
| pabelanger | for v8 | 00:04 |
| ianw | i think we know the pain points from rhel8 & fedora28, so it should hopefully not be too bad | 00:05 |
| pabelanger | sounds like a good time to try fedora-30 images | 00:10 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: WIP: Fix gerrit errors from production https://review.opendev.org/683006 | 00:10 |
| *** jamesmcarthur has quit IRC | 00:14 | |
| *** mattw4 has quit IRC | 00:21 | |
| openstackgerrit | Ian Wienand proposed zuul/zuul master: Support nodes setting 'auto' python-path https://review.opendev.org/682275 | 00:29 |
| *** Goneri has quit IRC | 00:45 | |
| *** jamesmcarthur has joined #zuul | 00:58 | |
| *** jamesmcarthur has quit IRC | 01:10 | |
| *** jamesmcarthur has joined #zuul | 01:10 | |
| *** jamesmcarthur has quit IRC | 01:15 | |
| *** jamesmcarthur has joined #zuul | 01:49 | |
| *** jamesmcarthur has quit IRC | 01:58 | |
| *** rlandy|bbl is now known as rlandy | 02:01 | |
| *** rlandy has quit IRC | 02:01 | |
| *** jamesmcarthur has joined #zuul | 02:28 | |
| *** roman_g has quit IRC | 02:35 | |
| *** bhavikdbavishi has joined #zuul | 02:51 | |
| *** bhavikdbavishi1 has joined #zuul | 02:54 | |
| *** bhavikdbavishi has quit IRC | 02:56 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 02:56 | |
| *** jamesmcarthur has quit IRC | 03:38 | |
| *** jamesmcarthur has joined #zuul | 03:40 | |
| *** jamesmcarthur has quit IRC | 03:44 | |
| *** jamesmcarthur has joined #zuul | 04:09 | |
| *** jamesmcarthur has quit IRC | 04:16 | |
| *** pcaruana has joined #zuul | 04:42 | |
| *** bolg has joined #zuul | 04:43 | |
| *** kerby has quit IRC | 04:45 | |
| *** jamesmcarthur has joined #zuul | 04:51 | |
| *** jamesmcarthur has quit IRC | 04:56 | |
| *** swest has joined #zuul | 05:00 | |
| *** bolg has quit IRC | 05:00 | |
| *** jamesmcarthur has joined #zuul | 05:17 | |
| *** jamesmcarthur has quit IRC | 05:22 | |
| *** bhavikdbavishi has quit IRC | 05:26 | |
| *** bhavikdbavishi has joined #zuul | 05:27 | |
| *** noorul has joined #zuul | 06:31 | |
| noorul | I am trying to sync project src folders to /tmp/ using http://paste.openstack.org/show/777607/ | 06:32 |
| noorul | I gave delegate_to: "{{ inventory_hostname }}" | 06:33 |
| noorul | I thought sync directory will be mapped to the node rather than on the executor | 06:34 |
| *** themroc has joined #zuul | 06:53 | |
| noorul | Any idea on the remote node where will be the projects src copied to ? | 06:58 |
| AJaeger | for project openstack/horizon in opendev.org server, they are copied to "{{ ansible_user_dir }}/{{ zuul.projects['opendev.org/openstack/horizon'].src_dir }}" | 07:16 |
| AJaeger | That's a variable you can use in ansible | 07:17 |
| AJaeger | Ah, you use zuul_projects already, so that should be fine. You can look at the inventory log file to see what is where | 07:18 |
| *** jamesmcarthur has joined #zuul | 07:18 | |
| *** jamesmcarthur has quit IRC | 07:23 | |
| *** tosky has joined #zuul | 07:23 | |
| *** jpena|off is now known as jpena | 07:33 | |
| mnasiadka | seems Zuul job queue is ridiculously long - and very long running tripleo jobs at the top... | 07:41 |
| *** roman_g has joined #zuul | 07:42 | |
| *** noorul has quit IRC | 07:53 | |
| *** zbr is now known as zbr|ruck | 07:57 | |
| *** noorul has joined #zuul | 08:05 | |
| *** hashar has joined #zuul | 08:09 | |
| *** arxcruz|ruck is now known as arxcruz|rover | 08:20 | |
| *** lennyb has quit IRC | 08:21 | |
| mordred | mnasiadka: the joys of release crunch time I believe | 08:22 |
| mnasiadka | mordred: might be, but still over 5 hours jobs don't look too good :) | 08:25 |
| mordred | mnasiadka: agree, it's never awesome. just be glad openstack is less trendy than a few years ago where release crunch would lead to 48 hour backed up queues :) | 08:28 |
| mnasiadka | mordred: well, non-trendiness has it's perks it seems :) | 08:28 |
| mordred | my expectations may be skewed - I'm now thrilled that release crunch is ONLY doing 5 hour gate backups | 08:28 |
| mordred | right? | 08:28 |
| mordred | I LOVE being less trendy :) | 08:29 |
| *** gtema has joined #zuul | 08:31 | |
| *** lennyb has joined #zuul | 08:34 | |
| *** noorul has quit IRC | 08:38 | |
| *** noorul has joined #zuul | 08:38 | |
| noorul | mordred: hi | 09:05 |
| noorul | mordred: Y'day we discussed about required_projects and how zuul pulls in code from those | 09:05 |
| noorul | mordred: If the current project has different release scheme number scheme, how can we handle that | 09:06 |
| *** jangutter has joined #zuul | 09:09 | |
| *** tobias-urdin has joined #zuul | 09:12 | |
| *** gtema has quit IRC | 09:13 | |
| mordred | noorul: that's the situtation where you'll need to use override checkout - because you'll need to tell zuul which branches should be tested with each other | 09:26 |
| noorul | mordred: Yes, but the dependency will vary release to release | 09:35 |
| mordred | noorul: yes - you'll likely want to make some child jobs that each have a branch matcher and then an override checkout | 09:37 |
| mordred | let me make you a for-instance real quick | 09:37 |
| mordred | noorul: http://paste.openstack.org/show/777615/ <-- if you do that, it creates variants of the job definition that match on various different branch values of the first project | 09:40 |
| mordred | noorul: so it says, "if the patch to the project is on release_1.0.0, please check out release_2.0 of some/other/project - and if the patch is on release_1.2.0 then please check out release_3.0 of some/other/project - but for other ones, check out matching branches" | 09:42 |
| openstackgerrit | Ian Wienand proposed zuul/zuul master: Add a more conversational overview to README.rst https://review.opendev.org/683085 | 09:45 |
| AJaeger | noorul: there's also https://zuul-ci.org/docs/zuul/user/config.html#attr-pragma.implied-branches | 09:47 |
| mordred | oh yeah. that too. thanks AJaeger :) | 09:53 |
| *** noorul has quit IRC | 09:54 | |
| *** hashar has quit IRC | 09:59 | |
| *** hashar has joined #zuul | 10:00 | |
| *** bhavikdbavishi has quit IRC | 10:03 | |
| openstackgerrit | Fabien Boucher proposed zuul/zuul master: Add reference pipelines file for Github driver https://review.opendev.org/672712 | 10:04 |
| *** hashar has quit IRC | 10:21 | |
| *** noorul has joined #zuul | 10:42 | |
| *** jangutter has quit IRC | 10:44 | |
| *** noorul has quit IRC | 10:47 | |
| *** pcaruana has quit IRC | 10:54 | |
| *** hashar has joined #zuul | 10:55 | |
| *** jangutter has joined #zuul | 11:00 | |
| *** noorul has joined #zuul | 11:03 | |
| *** jpena is now known as jpena|lunch | 11:04 | |
| *** jangutter has quit IRC | 11:05 | |
| noorul | mordred: Cool :) | 11:05 |
| *** pcaruana has joined #zuul | 11:17 | |
| *** bhavikdbavishi has joined #zuul | 11:41 | |
| *** bhavikdbavishi1 has joined #zuul | 11:43 | |
| *** bhavikdbavishi has quit IRC | 11:45 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 11:45 | |
| noorul | If I have a job running for a PR and I push a change, shouldn't the old job get killed? | 11:48 |
| *** jangutter has joined #zuul | 11:53 | |
| *** rfolco has joined #zuul | 12:04 | |
| *** Goneri has joined #zuul | 12:07 | |
| *** noorul has quit IRC | 12:08 | |
| *** jamesmcarthur has joined #zuul | 12:18 | |
| *** bhavikdbavishi has quit IRC | 12:18 | |
| *** bhavikdbavishi has joined #zuul | 12:19 | |
| *** bhavikdbavishi has quit IRC | 12:21 | |
| *** noorul has joined #zuul | 12:23 | |
| *** jpena|lunch is now known as jpena | 12:23 | |
| *** bhavikdbavishi has joined #zuul | 12:23 | |
| *** jamesmcarthur has quit IRC | 12:30 | |
| *** jamesmcarthur has joined #zuul | 12:30 | |
| *** rlandy has joined #zuul | 12:31 | |
| mnaser | noorul: yes | 12:35 |
| *** jamesmcarthur has quit IRC | 12:36 | |
| tobias-urdin | how could i best build a job to test an upgrade of a multi-repo project, what i need is project1-version1 install and test, project2 run upgrade, project1-version2 run test | 12:37 |
| tobias-urdin | where project1-version2 needs to support depends-on checkout | 12:37 |
| *** armstrongs has joined #zuul | 12:38 | |
| pabelanger | can have job that runs from project1-version2, but first switch branch, install, switch back | 12:39 |
| pabelanger | depends-on will be correct | 12:40 |
| armstrongs | i noticed a bug in the new console output. If you have an ansible playbook fail with undefined variable error, zuul will fail but on the console log it as a pass in the ui. I checked the ara report and that logs it correctly as a fail. Is that something you guys are aware of or have come across. im on Zuul version: 3.10.2.dev23 b53b6bad | 12:40 |
| noorul | mnaser: That is not happening in my case. Is this functionality tightly coupled with source driver? | 12:40 |
| tobias-urdin | pabelanger: oh, not sure why i was thinking about checking out the project twice instead of swapping branches, thanks | 12:42 |
| pabelanger | noorul: yes, we sometimes see that happen in github driver (haven't fixed it yet), the key logic is in https://opendev.org/zuul/zuul/src/branch/master/zuul/driver/github/githubmodel.py#L41 for example | 12:44 |
| pabelanger | same would be with bitbucket | 12:44 |
| pabelanger | tobias-urdin: yah, that is the method grenade in openstack does IIRC | 12:45 |
| tobias-urdin | pabelanger: i'll sneak peak grenade, that might help me getting started | 12:46 |
| *** fdegir has quit IRC | 12:47 | |
| mordred | tobias-urdin: zuul prepares appropriate state including depends-on for all of the branches in the repo to support that use case - so even the branches zuul isn't checking out for you should be set up properly if you change branch to them | 12:47 |
| *** fdegir has joined #zuul | 12:48 | |
| mordred | (which is just me agreeing with pabelanger with too many words) | 12:48 |
| tobias-urdin | thanks :) | 12:49 |
| *** bhavikdbavishi1 has joined #zuul | 12:50 | |
| *** bhavikdbavishi has quit IRC | 12:51 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 12:51 | |
| mnaser | i can probably look this up but has the zuul base library moved on from past 2.6 ? | 13:00 |
| mnaser | i ask because https://opendev.org/zuul/zuul-base-jobs/commit/7deaf1142f8755063f58591aa5e5367e76212075 | 13:01 |
| mnaser | (uploading base-jobs into a repo on github with their scanning on makes it unhappy because of CVE-2019-10156) | 13:01 |
| pabelanger | no, we still have zuul that run ansible 2.5 | 13:07 |
| pabelanger | so we need to keep zuul-jobs to support it | 13:07 |
| pabelanger | sorry, 2.6* | 13:07 |
| pabelanger | mnaser: https://review.opendev.org/650431/ drop 2.5 in zuul | 13:09 |
| pabelanger | and stack adds 2.9 support | 13:09 |
| pabelanger | which I need to rebase | 13:09 |
| *** pcaruana has quit IRC | 13:28 | |
| *** recheck has quit IRC | 13:28 | |
| *** recheck has joined #zuul | 13:29 | |
| *** jamesmcarthur has joined #zuul | 13:36 | |
| *** jamesmcarthur has quit IRC | 13:36 | |
| *** jamesmcarthur has joined #zuul | 13:36 | |
| noorul | Is ansible 2.9 released? | 13:44 |
| pabelanger | no | 13:51 |
| pabelanger | soon(tm) | 13:51 |
| pabelanger | rc1 is likely today | 13:51 |
| *** armstrongs has quit IRC | 13:54 | |
| *** pcaruana has joined #zuul | 13:54 | |
| *** saneax has joined #zuul | 14:01 | |
| *** avass has joined #zuul | 14:02 | |
| *** saneax has quit IRC | 14:04 | |
| *** saneax has joined #zuul | 14:05 | |
| pabelanger | now that we have 3.10.2, we got some UI fixes. I noticed this today | 14:09 |
| pabelanger | ansible-test-network-integration-vyos-python37 (3. attempt) | 14:09 |
| pabelanger | should that say (3rd attempt)? | 14:09 |
| *** saneax has quit IRC | 14:12 | |
| mordred | maybe? | 14:15 |
| *** noorul has quit IRC | 14:15 | |
| pabelanger | k, wasn't sure if . was a magic bit that auto did it | 14:16 |
| mordred | I also don't know | 14:17 |
| *** hashar has quit IRC | 14:20 | |
| *** themroc has quit IRC | 14:24 | |
| fungi | tobias-urdin: i believe grenade does (or did) maintain separate clones of repositories under /opt/stack/old and /opt/stack/new, but as long as they're cloned from the copies zuul has prepared on the node they can still checkout those branches and get the same prepared heads | 14:26 |
| fungi | mnaser: fwiw, i completely ignore/disable all of github's vulnerability notification features. the signal-to-noise ratio is so low as to be basically useless for all the projects i've gotten notified about in the past | 14:28 |
| *** avass has quit IRC | 14:32 | |
| *** pcaruana has quit IRC | 14:33 | |
| *** michael-beaver has joined #zuul | 14:35 | |
| *** mmedvede has quit IRC | 14:36 | |
| *** mmedvede has joined #zuul | 14:37 | |
| flaper87 | what var should I use to get the src dir on the worker node? | 14:37 |
| flaper87 | `zuul.project.src_dir` ? | 14:38 |
| mordred | flaper87: yes. that will be the src dir of the project that triggered the job | 14:38 |
| mordred | flaper87: (obviously it won't have as important a meaning for multi-repo jobs that can be triggered by different projects) | 14:38 |
| corvus | flaper87: that will be a relative path -- if you need an absolute path on the worker, you can prepend ansible_user_dir. | 14:39 |
| mordred | flaper87: it shoudl be noted that the zuul.project.src_dir value is a relative path | 14:39 |
| mordred | corvus: jinx | 14:39 |
| flaper87 | sweet, perfect, thanks! :) | 14:42 |
| * flaper87 will use ansible_user_dir + zuul.... | 14:42 | |
| AJaeger | flaper87: example: "{{ ansible_user_dir }}/{{ zuul.projects['opendev.org/openstack/horizon'].src_dir }}" | 14:56 |
| *** tosky has quit IRC | 14:57 | |
| *** tosky has joined #zuul | 14:58 | |
| openstackgerrit | David Shrewsbury proposed zuul/nodepool master: WIP: Do not overwrite image upload zk data on delete https://review.opendev.org/681857 | 14:59 |
| openstackgerrit | Kerby proposed zuul/nodepool master: AWS driver: add ability to determine AMI id using filters https://review.opendev.org/683183 | 15:25 |
| *** jangutter has quit IRC | 15:29 | |
| *** Goneri has quit IRC | 15:47 | |
| openstackgerrit | David Shrewsbury proposed zuul/nodepool master: Do not overwrite image upload ZK data on delete https://review.opendev.org/681857 | 15:53 |
| *** noorul has joined #zuul | 15:55 | |
| Shrews | I think that image leak fix ^^^ is finally ready. The test was... annoying. I was hoping to verify image deletion at the provider level, but that's not possible to do right now since we don't share provider data (in this case, images) across threads. PS2 shows the failure before the fix, PS3 should pass with the fix. | 15:57 |
| *** mgoddard has quit IRC | 15:58 | |
| *** mgoddard has joined #zuul | 15:59 | |
| Shrews | oh, the test comment is a bit off | 16:00 |
| noorul | I have nodepool config http://paste.openstack.org/show/777709/ | 16:01 |
| mnaser | noorul: neat! that seems functional as long as the right users are there? | 16:02 |
| noorul | I would like to run a job on "10.29.12.20", but without making any changest to nodepool | 16:02 |
| noorul | *changes | 16:02 |
| mnaser | i think you'll have to make a specific node label for that use case to be possible | 16:02 |
| noorul | I thought there will be another way | 16:03 |
| pabelanger | yes, need specific label | 16:03 |
| openstackgerrit | David Shrewsbury proposed zuul/nodepool master: Do not overwrite image upload ZK data on delete https://review.opendev.org/681857 | 16:03 |
| pabelanger | then create a nodeset in zuul, to use it | 16:03 |
| pabelanger | wait | 16:04 |
| pabelanger | noorul: just create a new nodeset, that used 10.29.12.20 as the label | 16:05 |
| *** openstackgerrit has quit IRC | 16:06 | |
| noorul | pabelanger: Let me try that | 16:07 |
| noorul | Is there a way to cancel the running job? | 16:07 |
| clarkb | you can use the zuul dequeue command, or push a new patchset or abandon the change (though I don't know if these last two things work with the bitbucket driver) | 16:08 |
| *** jamesmcarthur has quit IRC | 16:12 | |
| sean-k-mooney | you might also be able to configure jobs to look at labels like workflow. | 16:14 |
| sean-k-mooney | *pipelines | 16:14 |
| sean-k-mooney | if it a sepereate zuul install. | 16:15 |
| sean-k-mooney | but i know i have used workfow to ignore WIP patches locally in the past not sure if it would kick out a job that is in the pipeline queue after it was added but before it started | 16:16 |
| *** jamesmcarthur has joined #zuul | 16:21 | |
| *** noorul has quit IRC | 16:22 | |
| *** noorul has joined #zuul | 16:24 | |
| mnaser | so GitHub allows you to provide a link with pre-populated fields for creating a new app | 16:27 |
| mnaser | would it be beneficial to include that in our docs? | 16:27 |
| mnaser | so "go create a new app and check all these boxes" or "click this link and fill out the few missing fields" | 16:27 |
| fungi | seems like a reasonable improvement to me, as long as that api is fairly stabilized | 16:28 |
| *** Goneri has joined #zuul | 16:29 | |
| noorul | dequeue brought my zuul instance down | 16:30 |
| clarkb | noorul hrm I hust used it yesterday with no problems on a freshly installed 3.10.2 zuul | 16:31 |
| fungi | noorul: likely there will be a traceback in the log which will tell you why it crashed | 16:33 |
| *** mattw4 has joined #zuul | 16:34 | |
| rlandy | hello - zuul timer question: https://zuul-ci.org/docs/zuul/admin/drivers/timer.html says 'The first weekday is Monday.' iiuc, cron considers Sunday to be day 0. As such, if I want a pipeline to trigger on Saturday and Sunday, would the line be - time: '0 6 * * 5,6'? | 16:34 |
| noorul | It came up after several restarts | 16:35 |
| fungi | noorul: it's more that i'm hoping you can find the reason it crashed so we can determine if it's a bug with, say, interactions between the dequeue rpc subcommand and the bitbucket driver | 16:36 |
| *** kerby has joined #zuul | 16:37 | |
| noorul | fungi: I understand that | 16:38 |
| pabelanger | rlandy: I believe you want 0 6 * * 6,0 | 16:38 |
| pabelanger | what you have is Friday, saturday | 16:39 |
| noorul | fungi: Right now I am trying to bring up zuul to a state where I can demo its capabilities | 16:39 |
| *** mattw4 has quit IRC | 16:39 | |
| *** mattw4 has joined #zuul | 16:39 | |
| rlandy | pabelanger: ok - 'The first weekday is Monday.' is confusing - if that is considered day 0 or day 1 | 16:40 |
| noorul | Is there a plan to merge stash driver soon. I have not seen ofosos for long | 16:40 |
| pabelanger | rlandy: I find contrab.guru helpful too! | 16:40 |
| pabelanger | https://crontab.guru/#0_6_*_*_6,0 | 16:40 |
| rlandy | thanks | 16:40 |
| *** Goneri has quit IRC | 16:41 | |
| *** jamesmcarthur has quit IRC | 16:42 | |
| *** jamesmcarthur has joined #zuul | 16:43 | |
| *** igordc has joined #zuul | 16:44 | |
| noorul | pabelanger: This http://paste.openstack.org/show/777719/ is the error I see at that time | 16:44 |
| pabelanger | is port 19885 open on servers for zuul_console? | 16:45 |
| pabelanger | tcp | 16:45 |
| *** hashar has joined #zuul | 16:46 | |
| noorul | Yes | 16:46 |
| noorul | I could see console log streaming | 16:46 |
| noorul | Is there a way to make the Duration column in the builds UI more readable. In seconds/minutes ? | 16:47 |
| pabelanger | yah, I'd also like to see that updated too. | 16:48 |
| noorul | What is the difference between builds and buildsets? | 16:49 |
| fungi | a buildset is all the builds associated with a particular change item | 16:51 |
| fungi | so for example when you propose a pr and zuul runs builds for each of the jobs, that set of builds it ran together constitute a buildset | 16:51 |
| noorul | That means only relevant to gerrit | 16:51 |
| fungi | why would it only be relevant to gerrit? | 16:51 |
| fungi | can you not run multiple jobs when using other code review systems? | 16:52 |
| noorul | ok, multiple jobs | 16:52 |
| fungi | yes, "change item" was shorthand for gerrit change, github pr, git commit, timer event, et cetera | 16:52 |
| fungi | it's the set of builds which were triggered together | 16:53 |
| noorul | I got it http://zuul.opendev.org/t/zuul/buildset/42f6cc28945e4f32a2d985c904f2eaa4 | 16:54 |
| noorul | Is there a concept of build numbers like Jenkins in Zuul? | 16:59 |
| noorul | Is there a migration path? | 16:59 |
| *** openstackgerrit has joined #zuul | 16:59 | |
| openstackgerrit | Kerby proposed zuul/nodepool master: AWS driver: add ability to determine AMI id using filters https://review.opendev.org/683205 | 16:59 |
| clarkb | noorul: each build and buildset is identified by a uuiid (the last portion of that link you just pasted is one), but there is no monotonically increasing counter for them | 17:01 |
| *** pcaruana has joined #zuul | 17:01 | |
| clarkb | noorul: migration path of the build data? or of the jobs or? | 17:01 |
| *** Goneri has joined #zuul | 17:01 | |
| noorul | Today in Jenkins we use running number | 17:03 |
| noorul | x.y.z-build-number | 17:03 |
| noorul | pabelanger: http://paste.openstack.org/show/777723/ I tried that | 17:07 |
| noorul | pabelanger: but it still picking other node from the nodepool | 17:07 |
| *** zbr|ruck is now known as zbr | 17:07 | |
| pabelanger | label needs to be '10.29.12.20' | 17:08 |
| pabelanger | that is what you have in nodepool | 17:08 |
| noorul | http://paste.openstack.org/show/777709/ | 17:09 |
| noorul | You mean name translates to label in nodeset? | 17:09 |
| pabelanger | yes | 17:10 |
| pabelanger | https://zuul-ci.org/docs/zuul/user/config.html#nodeset | 17:10 |
| noorul | nodeset.nodes.label (required) | 17:12 |
| noorul | The Nodepool label for the node. Zuul will request a node with this label. | 17:12 |
| noorul | In my nodepool config label is central-ut | 17:12 |
| pabelanger | Oh, wait. | 17:12 |
| pabelanger | sorry, this is static | 17:12 |
| pabelanger | yah, so you need per region labels | 17:13 |
| pabelanger | like central-ut-a | 17:13 |
| pabelanger | central-ut-b | 17:13 |
| pabelanger | etc | 17:13 |
| noorul | In nodeset? | 17:13 |
| pabelanger | no, in nodepool | 17:13 |
| noorul | So I have to change nodepool | 17:13 |
| pabelanger | yes | 17:13 |
| *** Goneri has quit IRC | 17:14 | |
| *** pcaruana has quit IRC | 17:14 | |
| noorul | ok, then for the time being I will comment out all other nodes | 17:14 |
| pabelanger | you can keep central-ut, just also create per region labels, if you want that | 17:14 |
| noorul | Where can I find the change-id of the running job? | 17:20 |
| noorul | so that I can pass it as parameter to dequeue | 17:21 |
| noorul | zuul show has no ref / change id information | 17:22 |
| *** jpena is now known as jpena|off | 17:22 | |
| *** pcaruana has joined #zuul | 17:25 | |
| noorul | In openstack python jobs, are you caching python packages somewhere? As the new nodes are created every time package install could take significant time | 17:33 |
| pabelanger | opendev uses regional caches to help speed up somethings | 17:34 |
| *** hashar has quit IRC | 17:34 | |
| pabelanger | but, you can also cache things into our base images | 17:34 |
| noorul | hmm | 17:35 |
| fungi | noorul: yes, in opendev's case we have multiple layers of caching. we run a caching apache proxy in each environment which our job nodes are configured to connect to instead of pypi.org, but also for packages which lack viable prebuilt wheels we maintain a separate wheel cache where we build platform-specific wheels for each of the operating systems we test on and then configure nodes to look there as well as | 17:36 |
| fungi | the pypi proxy | 17:36 |
| fungi | basically every region where we run jobs has a local to it cache of these things | 17:37 |
| fungi | to cut down on trying to fetch them over the internet repeatedly | 17:38 |
| fungi | we do similarly for distro packages, dockerhub images, and so on | 17:38 |
| noorul | how can I determine change id of a running job? | 17:49 |
| corvus | noorul: https://zuul-ci.org/docs/zuul/user/jobs.html#var-zuul.change | 17:50 |
| Shrews | tristanC: the nodepool openshift job seems to be failing quite often. see https://review.opendev.org/683205 and https://review.opendev.org/681857 for examples. | 17:51 |
| corvus | noorul: oh you wanted that for dequeue -- look on the status page | 17:51 |
| noorul | corvus: Is it this command separated value "4,01a7e7e135e04405e1493786fb6887351516b12f" | 17:53 |
| corvus | noorul: yep | 17:53 |
| noorul | s/command/comma | 17:53 |
| corvus | noorul: the 4 is the PR #, the sha is the current tip commit for that PR | 17:54 |
| noorul | corvus: Thank you! | 17:54 |
| noorul | corvus: It worked | 17:54 |
| noorul | I tried this zuul dequeue --tenant central --pipeline check --project ac/monitoring --ref 01a7e7e135e04405e1493786fb6887351516b12f | 17:55 |
| noorul | and zuul crashed | 17:55 |
| fungi | that definitely shouldn't be able to crash the scheduler, so sounds like a definite bug | 17:55 |
| *** pcaruana has quit IRC | 17:55 | |
| SpamapS | Shrews: is there a quick way to make a request for nodepool without running a whole zuul scheduler? | 17:56 |
| fungi | may be we need better error handling around refs mentioned in rpc commands | 17:56 |
| Shrews | SpamapS: not without a small bit of coding | 17:57 |
| Shrews | SpamapS: the nodepool unit test suite is filled with examples of creating a manual NodeRequest | 17:57 |
| AJaeger | zuul-jobs reviewers, the 2 weeks waiting period for https://review.opendev.org/567696 is over - shall we merge it? | 17:57 |
| corvus | noorul: can you elaborate on "zuul crashed"? | 17:57 |
| *** avass has joined #zuul | 17:58 | |
| fungi | noorul: the tracebacks in http://paste.openstack.org/show/777719/ don't look related to the crash. i suppose it's possible the scheduler dies before it can write the relevant traceback to its log | 17:58 |
| noorul | corvus: The UI was not responding | 17:58 |
| fungi | noorul: did the scheduler process stop? | 17:58 |
| Shrews | SpamapS: https://opendev.org/zuul/nodepool/src/branch/master/nodepool/tests/unit/test_launcher.py#L49-L52 is the gist of it | 17:58 |
| noorul | Looks like it stopped, because UI was returning 500 | 17:59 |
| fungi | did you have to start the scheduler again to get it working? | 17:59 |
| noorul | Yes | 17:59 |
| *** Goneri has joined #zuul | 18:12 | |
| tristanC | Shrews: it seems like both failed happen in fortnebula-regionone, where the instance ip 192.168.48 is not reachable ? | 18:14 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Fix gerrit errors from production https://review.opendev.org/683006 | 18:19 |
| *** bhavikdbavishi has quit IRC | 18:19 | |
| corvus | clarkb, mordred: ^ that's the omnibus fix-gerrit-errors-from-yesterday patch | 18:20 |
| *** igordc has quit IRC | 18:20 | |
| *** jamesmcarthur has quit IRC | 18:21 | |
| corvus | Shrews: did all the expected autohold patches land? | 18:21 |
| corvus | Shrews: apparently not, looks like maybe lost in restart; i'll nudge them | 18:21 |
| *** jamesmcarthur has joined #zuul | 18:22 | |
| Shrews | corvus: oh, i assume they had | 18:23 |
| corvus | my fault, sorry | 18:23 |
| *** noorul has quit IRC | 18:23 | |
| corvus | Shrews: they're ahead of the fix i just pushed up, so we should get them in the next restart | 18:24 |
| Shrews | corvus: the last one in that chain needs a +3 https://review.opendev.org/679057 | 18:24 |
| Shrews | corvus: no worries. i forgot about them when i returned to the image leak fix :) | 18:25 |
| *** michael-beaver has quit IRC | 18:25 | |
| corvus | Shrews, tristanC, mhu: that change lgtm, but you can specify the request method in the route map, so you don't need an extra method to dispatch based on that -- maybe we should tidy that up in a later change | 18:26 |
| *** avass has quit IRC | 18:50 | |
| *** Diabelko has quit IRC | 19:09 | |
| *** Diabelko has joined #zuul | 19:15 | |
| daniel2 | How would you modify the cloud config stuff for a nodepool build? | 19:32 |
| Shrews | corvus: wow, such failure on the autohold revamp changes. and i can't seem to pull up logs | 19:33 |
| pabelanger | daniel2: you mean on the one nodepool-builder uses? | 19:33 |
| daniel2 | Yeah | 19:34 |
| pabelanger | you can use any cfgmgmt tool | 19:34 |
| pabelanger | eg: ansible | 19:34 |
| corvus | Shrews: yeah, i dunno what happened with the unit tests; i was just looking at the quickstart | 19:34 |
| Shrews | looks like qs failed on swift upload? | 19:35 |
| corvus | Shrews: where do you see that? | 19:35 |
| Shrews | last entry in https://zuul.opendev.org/t/zuul/build/927433d151a642049c85322a387d535d/log/job-output.txt | 19:35 |
| Shrews | no error message, but that was the last task | 19:36 |
| corvus | Shrews: no that's just where the logs end (we don't get any logs in swift after we start the swift upload, because of restrictions imposed by the space-time continuum) -- if it had failed there, it would be a post_failure | 19:36 |
| corvus | Shrews: try the console tab, it'll take you right to the failed task: https://zuul.opendev.org/t/zuul/build/927433d151a642049c85322a387d535d/console | 19:37 |
| Shrews | corvus: ah. we need ubiquitous logging. make that happen :) | 19:37 |
| corvus | Shrews: i've been reading this book "A Journey Through Time" which has some interesting ideas about how we could solve that | 19:38 |
| Shrews | corvus: i find that error confusing, given the message is "OK" | 19:38 |
| Shrews | corvus: are you able to access the py3* logs? | 19:39 |
| corvus | Shrews: no argument there, but that tells us that something went wrong waiting for zuul to report, and we can dig into the zuul logs for that | 19:40 |
| corvus | Shrews: yeah, but i can tell you they won't be useful. | 19:41 |
| Shrews | i think the node was hit by a cosmic ray | 19:42 |
| corvus | if the job timed out with no test-results.html output, then it's probably some catastrophe that causes all the tests not to work | 19:42 |
| corvus | Shrews: both, apparently | 19:42 |
| corvus | like something happened to zk or somesuch | 19:42 |
| corvus | Shrews: this looks like the actual error with qs: https://zuul.opendev.org/t/zuul/build/927433d151a642049c85322a387d535d/log/container_logs/scheduler.log#2037 | 19:44 |
| corvus | it seems very unlikely related to that change; i'm trying to come up with a theory | 19:46 |
| corvus | the same version of gerrit was used in the last successful gate run | 19:49 |
| *** pcaruana has joined #zuul | 19:51 | |
| Shrews | nothing unusual in the gerrit log | 19:52 |
| corvus | i'm running that job locally to see if it's reproducible | 19:56 |
| *** jamesmcarthur has quit IRC | 20:01 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Remove parent from validate-zone-db https://review.opendev.org/683225 | 20:02 |
| corvus | not a fatal error, but that does appear in the errer logs ^ | 20:02 |
| *** jamesmcarthur has joined #zuul | 20:04 | |
| corvus | Shrews: i got nothin | 20:15 |
| corvus | Shrews: official cause of death: very bad luck | 20:15 |
| Shrews | seems about right | 20:15 |
| corvus | Shrews: i'm reapproving them all | 20:16 |
| corvus | Shrews: my change failed with the same 404 | 20:18 |
| *** jamesmcarthur has quit IRC | 20:20 | |
| Shrews | corvus: it's catching | 20:21 |
| *** jamesmcarthur has joined #zuul | 20:21 | |
| *** jamesmcarthur has quit IRC | 20:22 | |
| corvus | Shrews: all three jobs (the last successful gate job, yours, and mine) have the same sha256 for the gerrit container image | 20:24 |
| Shrews | some sort of race? can't imagine how | 20:25 |
| Shrews | or on what | 20:25 |
| corvus | maybe on secondary index updates? | 20:26 |
| *** pcaruana has quit IRC | 20:31 | |
| *** igordc has joined #zuul | 20:33 | |
| Shrews | hrm, reading https://gerrit-review.googlesource.com/Documentation/config-gerrit.html#index seems there may be potential for some sort of race, but i'm not familiar enough with how/when it is used and if that could potentially affect these ops | 20:35 |
| corvus | it should have had plenty of time to update (it ran an actual job, plus we tried 4 times) | 20:35 |
| corvus | i've also verified that the zuul-scheduler image i tested with locally matches what was used in the most recent passing gate job | 20:39 |
| *** sshnaidm is now known as sshnaidm|off | 20:42 | |
| corvus | Shrews: dockerhub has the image used for your change, maybe i'll try locally with that https://hub.docker.com/layers/zuul/zuul-scheduler/change_661114_latest/images/sha256-d0b18f587bc0235faff43a050dbf5c37671359efeb1ac2235ce1d39b3892a454 | 20:42 |
| openstackgerrit | Merged zuul/zuul-jobs master: Remove parent from validate-zone-db https://review.opendev.org/683225 | 20:47 |
| corvus | Shrews: whoah-hoh! i reproduced locally | 20:48 |
| Shrews | middle-school-me wants to make a joke there | 20:50 |
| Shrews | but yay! | 20:50 |
| Shrews | i wonder if it's an artifact of the way the image is built, or just chance | 20:51 |
| corvus | Shrews: well, 2 image builds since the last one show the problem; and i don't thitk we have an image built since the last change merged which doesn't show it. | 20:52 |
| corvus | i'll see about doing an image build of master right now to see if it shows up | 20:53 |
| SpamapS | Shrews: on the topic of how to create a request.. it's weird to me that we don't have CLI tools to CRUD requests. | 20:55 |
| corvus | SpamapS: we have RD not CU | 20:58 |
| Shrews | SpamapS: RUD is supported by current nodepool cli. only the C | 20:58 |
| Shrews | yeah, i guess not U since hold request was moved to zuul | 20:59 |
| corvus | the only kind of request you could create from a cli would be a hold | 20:59 |
| corvus | (like, request a node and immediately hold it) | 20:59 |
| Shrews | SpamapS: if you think of the way nodepool actually works, requesting nodes asynchronously via CLI doesn't make sense because after the request, you'd have to have another command to remove the fulfilled NodeRequest. And yet another to mark the nodes as USED. | 21:01 |
| corvus | so, potentially a useful debugging and development tool, but i guess not one that has seemed more useful than autoholding from zuul. and when i've hacked on the drivers before, i just use min-ready to generate the requests. | 21:01 |
| Shrews | it's not quite made for such manual (prone to error) interactions | 21:01 |
| Shrews | corvus: looks like it failed again | 21:05 |
| Shrews | wow, this revamp is the pain that just keeps on giving | 21:06 |
| Shrews | oh look... is that beer over there??? | 21:06 |
| corvus | Shrews: yeah, at this point i expect there's a real problem we don't understand yet | 21:07 |
| corvus | i just build new images based on current master, trying that now | 21:07 |
| corvus | yep that fails too | 21:08 |
| corvus | it's super weird that this failed but not the unit tests | 21:09 |
| corvus | (i mean, they did fail, but for other reasons -- not reliably) | 21:10 |
| SpamapS | Shrews: k.. just makes it really hard to test in isolation other than via the test suite. :-P | 21:13 |
| Shrews | SpamapS: yeah, but that's why we use zuul so we can do crossproject testing and not have to test in isolation :) | 21:14 |
| Shrews | but yeah, hard to experiment with it in isolation w/o some coding | 21:15 |
| corvus | SpamapS: you may want to try the min-ready trick -- make a launcher config with 'min-ready: 1' and the launcher will always try to keep a node available. delete the node to get it to run again. | 21:18 |
| corvus | ^C and restart as you make changes | 21:19 |
| SpamapS | corvus: that's true.. I forgot about that. ;) | 21:20 |
| SpamapS | kerby: ^^ | 21:20 |
| Shrews | it's also not hard to write a 20 or so line python script that uses the nodepool zk library to request a node. i think i have one somewhere... | 21:20 |
| corvus | Shrews: okay, i've got a curl command working that should be doing exactly what zuul says it's doing but getting a 404 -- and it's getting that 404 even after my curl command succeeds | 21:23 |
| Shrews | SpamapS: http://paste.openstack.org/show/777952/ | 21:24 |
| corvus | Shrews: i'm assuming, for the moment, that some low-level system or python library has changed in some very subtle way | 21:24 |
| SpamapS | Shrews:maybe that should be in tools. ;) | 21:25 |
| Shrews | SpamapS: i would not want to give folks the impression that is a great way to dynamically request nodes | 21:25 |
| Shrews | it does not account for leaving the node request around after it is fulfilled | 21:26 |
| Shrews | it's a sloppy, quick hack | 21:26 |
| SpamapS | yeah, min-ready trick is better | 21:26 |
| Shrews | corvus: that's wonderful news | 21:27 |
| Shrews | that script is actually quite old. it should be using storeNodeRequest() instead of the direct client call | 21:29 |
| Shrews | SpamapS: ^^ | 21:29 |
| Shrews | but a hack is a hack | 21:29 |
| corvus | Shrews: i'm positive it's not zuul related now -- i have a short python script that reproduces the problem; it works in my py36 venv and fails in the image | 21:37 |
| corvus | now i have 2 containers, one where the simple script works and one where it fails | 21:44 |
| corvus | same python version, same requests version | 21:44 |
| corvus | -urllib3==1.25.4 | 21:45 |
| corvus | +urllib3==1.25.3 | 21:45 |
| corvus | that sounds plausible | 21:45 |
| corvus | i'd just like to call out urllib3's tagline: "Sanity-friendly HTTP client." | 21:46 |
| corvus | Shrews: is that resonating for you right now like it is for me? :) | 21:46 |
| Shrews | corvus: that’s the only difference? | 21:47 |
| corvus | Shrews: in python libs, yes | 21:48 |
| Shrews | not doing much for MY sanity, personally | 21:48 |
| corvus | also the only difference in system package | 21:49 |
| corvus | packages | 21:49 |
| corvus | i still don't know what it's doing wrong. i'm assuming it's doing something weird with "~" | 21:50 |
| SpamapS | https://github.com/urllib3/urllib3/releases/tag/1.25.4 | 21:51 |
| SpamapS | " sethmlarson released this 5 hours ago " | 21:51 |
| SpamapS | https://github.com/urllib3/urllib3/compare/1.25.3...1.25.4 | 21:53 |
| SpamapS | Lots of changes | 21:53 |
| corvus | https://github.com/urllib3/urllib3/pull/1673 looks promising | 21:53 |
| Shrews | Fix for Python 4 (#1669) | 21:55 |
| Shrews | i wasn't aware that was a thing | 21:55 |
| SpamapS | Since you have a reproducer... bisect ftw? | 21:56 |
| corvus | yeah, let me see if i can get that happening on my workstation | 21:57 |
| corvus | yep, that works in my py36 venv | 21:57 |
| corvus | bisect confirms it's https://github.com/urllib3/urllib3/pull/1673/commits/fd8a95af871a4328c474086b897accef620661eb | 22:00 |
| *** mattw4 has quit IRC | 22:01 | |
| *** mattw4 has joined #zuul | 22:02 | |
| corvus | they seem to be going against the "should not" in https://tools.ietf.org/html/rfc3986#section-2.3 | 22:05 |
| corvus | because it does look like they are percent-encoding the tilde | 22:06 |
| *** armstrongs has joined #zuul | 22:27 | |
| corvus | https://github.com/urllib3/urllib3/pull/1684 | 22:30 |
| *** rlandy is now known as rlandy|biab | 22:30 | |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Don't use urllib3 1.25.4 https://review.opendev.org/683250 | 22:34 |
| corvus | zuul-maint: ^ can we merge that asap to unblock the gate? | 22:34 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Fix gerrit errors from production https://review.opendev.org/683006 | 22:35 |
| *** armstrongs has quit IRC | 22:37 | |
| *** igordc has quit IRC | 22:44 | |
| *** igordc has joined #zuul | 23:01 | |
| *** mattw4 has quit IRC | 23:03 | |
| *** mattw4 has joined #zuul | 23:03 | |
| *** tosky has quit IRC | 23:12 | |
| *** rlandy|biab is now known as rlandy|bbl | 23:18 | |
| *** kerby has quit IRC | 23:59 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!