| openstackgerrit | Tony Breeds proposed openstack-infra/openstack-zuul-jobs master: Update the destination path for publishing election files. https://review.openstack.org/509024 | 00:00 |
|---|---|---|
| clarkb | SpamapS: its a synchronize from test node to dest: elections/ so its in the job root not in the work_root | 00:00 |
| SpamapS | interesting | 00:01 |
| * tonyb will debug the requirements job on the elections repo after $meeting | 00:01 | |
| jeblair | well, it expanded to the playbook root; i think the patch checker may guess relative paths wrong? | 00:01 |
| jeblair | path checker even | 00:01 |
| SpamapS | Yeah that's what I was wondering | 00:01 |
| clarkb | tonyb: I'm trying to sort out how it is expected that those contents then by published somewhere | 00:02 |
| SpamapS | is uses os.path.realpath | 00:02 |
| *** xarses has joined #openstack-infra | 00:02 | |
| SpamapS | pretty straightforward expansion with that | 00:02 |
| tonyb | clarkb: Thanks. | 00:02 |
| *** wolverineav has quit IRC | 00:03 | |
| tonyb | Also it seems we just did DST so I have an hour before $meeting \o/ | 00:03 |
| clarkb | tonyb: its not immediately clear to me :( | 00:04 |
| *** jbadiapa_ has quit IRC | 00:04 | |
| tonyb | clarkb: I'm sure we'll work it out ;P | 00:04 |
| SpamapS | it uses both abspath and realpath | 00:04 |
| * tonyb looks too albeit slower than clarkb | 00:04 | |
| SpamapS | so abspath is going to join it with os.getpwd() | 00:05 |
| SpamapS | and realpath will resolve symlinks | 00:05 |
| *** ijw has joined #openstack-infra | 00:05 | |
| *** andreww has joined #openstack-infra | 00:05 | |
| SpamapS | so I'm more suspicious that synchronize just isn't doing the right thing | 00:05 |
| *** andreww has quit IRC | 00:05 | |
| SpamapS | (since I think on localhost os.getpwd() is going to always be the job/playbook root, not work root) | 00:06 |
| jeblair | SpamapS: i believe we set pwd to the work root, but ansible defeats us by using playbook dir as pwd in many cases | 00:07 |
| *** xarses has quit IRC | 00:07 | |
| *** jaypipes has quit IRC | 00:07 | |
| SpamapS | yeah | 00:07 |
| SpamapS | synchronize appears on the surface to be passing dest properly | 00:07 |
| *** andreww has joined #openstack-infra | 00:08 | |
| clarkb | tonyb: looking at other artifact publishing things they have a special base job (which the election static job does not have) | 00:08 |
| SamYaple | does anyone have an example of how to reference a secret in a playbook? | 00:08 |
| SpamapS | if mode == pull that is | 00:09 |
| pabelanger | SpamapS: look at project-config publish-openstack-artifacts | 00:09 |
| pabelanger | SpamapS: sorry | 00:09 |
| pabelanger | SamYaple: ^ | 00:09 |
| jeblair | SpamapS: you've read the docs, right? | 00:09 |
| jeblair | SamYaple: https://docs.openstack.org/infra/zuul/feature/zuulv3/user/jobs.html#secrets | 00:10 |
| SpamapS | hah everybody's SpamYapling | 00:10 |
| SamYaple | jeblair: im reading that yes, i have the secret read | 00:10 |
| SamYaple | just not how to reference it, looking at what pabelanger sent | 00:10 |
| SamYaple | secret ready* | 00:10 |
| clarkb | tonyb: build-openstack-sphinx-docs looks promising, that sets things up for afs publishing (and I think it will look for stuff in doc/build/html or similar | 00:11 |
| jeblair | SamYaple: cool. let me know when you find out what the docs are missing. | 00:11 |
| clarkb | tonyb: so we may have to plumb this second half too | 00:11 |
| jeblair | cause i thought they should answer that question | 00:11 |
| clarkb | tonyb: looks like election content is published to static.o.o so not afs, /me keeps looking | 00:12 |
| *** dingyichen has joined #openstack-infra | 00:12 | |
| tonyb | clarkb: Oh | 00:12 |
| *** lukebrowning has quit IRC | 00:12 | |
| SpamapS | I think this is just ansible's os.getpwd() trickery defeating us | 00:12 |
| SpamapS | but no.. that would make it trip the ansible unsafe path | 00:13 |
| SpamapS | hrrrrrrmmmm | 00:13 |
| tonyb | clarkb: something like publish-openstack-specs ? | 00:13 |
| SamYaple | pabelanger: publish-artifacts-to-fileserver role doesnt seem to exist in project-config | 00:14 |
| jeblair | SamYaple: what question are you trying to answer? | 00:14 |
| SamYaple | ah found it | 00:15 |
| SamYaple | jeblair: im trying to see the secret used somewhere | 00:15 |
| SpamapS | Ok I have to EOD | 00:15 |
| clarkb | tonyb: where is that? | 00:15 |
| SpamapS | nothing obvious jumps out about that synchronize failure | 00:15 |
| jeblair | SamYaple: right, but what's missing from the example in the docs? | 00:15 |
| tonyb | project-config/zuul.d/jobs.yaml:148 | 00:15 |
| SpamapS | definitely seems like the ansible action protection failed us there | 00:15 |
| pabelanger | SamYaple: that exists in zuul-jobs | 00:15 |
| pabelanger | SamYaple: but the jobs uses a secret, like the docs jeblair posted | 00:16 |
| pabelanger | tonyb: clarkb: http://git.openstack.org/cgit/openstack-infra/openstack-zuul-jobs/tree/playbooks/legacy/static-election-publish/post.yaml#n7 is the issue, that should likely be dest: {{ zuul.executor.work_root }}/elections | 00:17 |
| SamYaple | jeblair: the secret has a name which i can configure. it also has a data dict where the key is a variable... im not sure how to actually call this? is the name a variable i can reference in ansible? or the key name in the data dict? | 00:17 |
| SamYaple | pabelanger: i found it it in zuul-jobs, thanks | 00:17 |
| SamYaple | jeblair: i get it now | 00:18 |
| clarkb | pabelanger: yes but what copies that content to static.openstack.org? | 00:18 |
| *** thiagolib has quit IRC | 00:19 | |
| jeblair | SamYaple: okay, what was it you saw in zuul-jobs that is missing from the docs that helped you understand it? | 00:19 |
| *** lukebrowning has joined #openstack-infra | 00:19 | |
| *** jbadiapa_ has joined #openstack-infra | 00:19 | |
| jeblair | SamYaple: (at this point, every question you ask in here is a doc bug. thanks for your help. :) | 00:19 |
| SamYaple | jeblair: you can assign the secret a variable name on the job. you reference the secret by the name set in the secret definition. the data dict then turns into somethign you reference by the variable you gave it in the job definition | 00:19 |
| SamYaple | a bit confusing but it makes sense | 00:20 |
| pabelanger | clarkb: from what I see, nothing. We just copy them to executor and that is it. | 00:20 |
| jeblair | SamYaple: well, the name in the job definition is optional | 00:20 |
| pabelanger | clarkb: so, we likely need a new parent job to handle that. Where should they end up? | 00:20 |
| SamYaple | so thats only if you want to *re*-name it | 00:20 |
| jeblair | SamYaple: ya | 00:20 |
| SamYaple | cool | 00:20 |
| jeblair | SamYaple: if you omit that, then it's just the name of the secret (which is what the docs say there). so we should update that section to mention the possibility of re-naming it. | 00:21 |
| SamYaple | ill throw up a docs patch | 00:21 |
| jeblair | SamYaple: thanks | 00:21 |
| tonyb | pabelanger: ultimately https://governance.openstack.org/election/ | 00:21 |
| clarkb | pabelanger: https://governance.openstack.org/election/ there which is a vhost on static.o.o | 00:21 |
| jeblair | SamYaple: i think the job renaming came after that section was written and we forgot to update it | 00:21 |
| *** mriedem has quit IRC | 00:21 | |
| * tonyb looks for the docs publish job for openstack/governance as it shoudl be preety close to that. | 00:21 | |
| pabelanger | clarkb: tonyb: ya, so it shoulds like we never wrote a publisher for that, unless mordred did | 00:21 |
| *** sdague has quit IRC | 00:22 | |
| tonyb | pabelanger: So should I base it on ... project-config/playbooks/publish/openstack-specs.yaml ? | 00:22 |
| pabelanger | clarkb: tonyb: we'd need to create a new job in project-config to do that, like we did for logs and draft-docs | 00:22 |
| *** dchen has joined #openstack-infra | 00:22 | |
| tonyb | clarkb: ^^ | 00:22 |
| pabelanger | looking | 00:22 |
| jeblair | aha! i get what's causing the base job not defined error! | 00:23 |
| SamYaple | jeblair: well that and its not immediately clear that the secret name is the variable and the data is a dict for the variable after it gets decoded. but ill see if i can reword it/provide enough documentation to make it clear for dense people like myself :) | 00:23 |
| clarkb | jeblair: woo | 00:23 |
| *** lukebrowning has quit IRC | 00:23 | |
| jeblair | it is, sadly, i think not at all related to memory use | 00:23 |
| pabelanger | tonyb: yes, exactly | 00:23 |
| * clarkb is running pip test suite and misses parallelized test runs | 00:23 | |
| jeblair | but i think i can make a quick fix | 00:24 |
| tonyb | Ans at what point does the defn. cease to be legacy? | 00:24 |
| *** andreww has quit IRC | 00:25 | |
| pabelanger | tonyb: we'd likley create a new secret called site_elections in zuul.d/secrets.yaml, which will be copypaste from site_logs but using proper fqdn | 00:25 |
| *** lukebrowning has joined #openstack-infra | 00:25 | |
| tonyb | pabelanger: okay. I'm sure at some point that will make sese to me ;P | 00:26 |
| *** dchen has quit IRC | 00:26 | |
| *** dchen has joined #openstack-infra | 00:26 | |
| pabelanger | tonyb: ya, this will be a tricky job, since you'll be making a new publisher job. If you want to start on it, that is great, and for tomorrow morning I can help move it along | 00:27 |
| tonyb | pabelanger: okay I'll take a run at it. | 00:28 |
| pabelanger | ++ | 00:28 |
| tonyb | IIUC this will alos be needed for all the governnace updates? | 00:28 |
| pabelanger | ya, once you create the new publisher for elections.o.o, any job that needs to upload to it, will need to parent to that publisher job | 00:29 |
| *** lukebrowning has quit IRC | 00:29 | |
| tonyb | pabelanger: Okay I'll do it the other way around create site_governance and use that for the election stuff. as it's just a subdir of that site | 00:30 |
| pabelanger | sure | 00:30 |
| * tonyb goes to read about secrets | 00:30 | |
| pabelanger | I have to drop now, have limited internet access for the next few days. | 00:30 |
| *** lukebrowning has joined #openstack-infra | 00:31 | |
| SamYaple | tonyb: good luck! (its really not that bad) | 00:33 |
| tonyb | SamYaple: thanks ;P | 00:34 |
| *** lukebrowning has quit IRC | 00:35 | |
| *** yamahata has quit IRC | 00:37 | |
| *** lukebrowning has joined #openstack-infra | 00:37 | |
| SamYaple | this looks right for secrets, yes? https://review.openstack.org/#/c/508792/5..6/.zuul.yaml | 00:41 |
| SamYaple | tonyb: ^ that should help you too | 00:41 |
| tonyb | SamYaple: Thanks. | 00:41 |
| *** lukebrowning has quit IRC | 00:42 | |
| *** lnxnut has joined #openstack-infra | 00:42 | |
| jeblair | SamYaple: secret names are also global (for now) so you may want to name it loci_docker_login or something. this is where the in-job-def renaming may come in handy. | 00:50 |
| SamYaple | got it | 00:50 |
| SamYaple | will adjust | 00:50 |
| SamYaple | hey if i declare 'secrets' as part of the job and call that job from the check queue... will it break? or can i do a ' if variable defined' in the playbook | 00:51 |
| jeblair | (and yeah, it makes no sense that secret names are global when they can't be used outside of a project. sorry.) | 00:51 |
| SamYaple | that way i can call the same job from gate check and post | 00:52 |
| jeblair | SamYaple: it will break (zuul will refuse to run the job) | 00:52 |
| SamYaple | got it | 00:52 |
| SamYaple | so ill jsut parent the job | 00:52 |
| jeblair | SamYaple: ya that may be a solution, or possibly just reusing roles | 00:52 |
| ianw | hmm, a whole bunch of dib functional test jobs seemed to just randomly stop, and then the job ends up timing out | 00:53 |
| *** lnxnut has quit IRC | 00:53 | |
| SamYaple | i think im going to switch to a zuul.d layout | 00:53 |
| SamYaple | my config is getting gangly | 00:53 |
| jeblair | SamYaple: secrets tend to put it over the edge, yeah | 00:54 |
| SamYaple | that and having a job we project isn't helping | 00:54 |
| *** lukebrowning has joined #openstack-infra | 00:55 | |
| *** jdandrea_ has joined #openstack-infra | 00:56 | |
| *** lukebrowning has quit IRC | 01:00 | |
| *** edmondsw has joined #openstack-infra | 01:01 | |
| *** lukebrowning has joined #openstack-infra | 01:01 | |
| clarkb | jeblair: the zuul-scheduler is using quite a bit less cpu recently | 01:02 |
| clarkb | looks like memory use leveled off around the same time ish | 01:03 |
| clarkb | possibly due to the kazoo lock fails? | 01:04 |
| *** dingyichen has quit IRC | 01:04 | |
| *** raissa_ has quit IRC | 01:05 | |
| *** raissa_ has joined #openstack-infra | 01:05 | |
| *** kiennt26 has joined #openstack-infra | 01:05 | |
| *** lukebrowning has quit IRC | 01:06 | |
| *** wolverineav has joined #openstack-infra | 01:06 | |
| *** lukebrowning has joined #openstack-infra | 01:07 | |
| *** adisky_ has joined #openstack-infra | 01:07 | |
| clarkb | there is a tcp connection | 01:10 |
| clarkb | looks good on both sides /me tcpdumps to see if traffic flows | 01:10 |
| *** namnh has joined #openstack-infra | 01:11 | |
| clarkb | ok I see traffic flowing | 01:11 |
| *** lukebrowning has quit IRC | 01:12 | |
| *** lukebrowning has joined #openstack-infra | 01:13 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Clear project config cache later https://review.openstack.org/509040 | 01:14 |
| clarkb | jeblair: ^ any ideas? | 01:14 |
| clarkb | I'm fairly stumped and its getting late | 01:14 |
| jeblair | clarkb: sorry what's the issue? | 01:15 |
| *** kiennt26 has quit IRC | 01:15 | |
| *** cuongnv has joined #openstack-infra | 01:16 | |
| clarkb | jeblair: zuul seems stalled out and all the lock node requests are timeing out | 01:17 |
| *** edmondsw has quit IRC | 01:17 | |
| clarkb | noticed that the cpu usage is relatively low compared to before | 01:17 |
| jeblair | clarkb: yeah, it looks like the idleness is because each node action is timing out, and takes 30s to do that | 01:17 |
| clarkb | ya | 01:17 |
| clarkb | the connection is up and packets are flowing though | 01:17 |
| clarkb | er zk connection | 01:17 |
| SamYaple | from a bandwidth perspective, if i publish to dockerhub and then use the registry mirror cache provided by infra per region, does that use less resources than publish to tarballs.o.o and pulling a tarball from there? | 01:18 |
| *** dingyichen has joined #openstack-infra | 01:18 | |
| clarkb | SamYaple: its probably the same ish? we'll proxy cache both of them | 01:18 |
| SamYaple | i think pushing straight to dockerhub would be my vote then, since ill be pushing there anyway | 01:20 |
| clarkb | jeblair: nl02 debug log shows lots of retries happening | 01:20 |
| SamYaple | and any registry infra may setup we can dual push in the future | 01:20 |
| clarkb | in use nodes is 0 so I don't think any requests are succeeding | 01:22 |
| *** ekcs has quit IRC | 01:22 | |
| *** vhosakot has quit IRC | 01:24 | |
| *** lukebrowning has quit IRC | 01:24 | |
| clarkb | hrm | 01:24 |
| clarkb | its zookeeper | 01:24 |
| clarkb | no space left on device according to zk logs | 01:24 |
| clarkb | but I see plenty of disk according to df | 01:24 |
| clarkb | jeblair: did you delete /var/lib/zookeeper/version-2 things? | 01:24 |
| jeblair | clarkb: no; you sure that isn't old? | 01:24 |
| clarkb | oh ya timestamp is for 12 hours ago or so | 01:25 |
| jeblair | whew | 01:25 |
| clarkb | ok so nothing new in zk log | 01:25 |
| jeblair | it's looking like zuul double locked a node | 01:25 |
| jeblair | i'm looking at 0000102036 | 01:25 |
| *** cshastri has joined #openstack-infra | 01:26 | |
| jeblair | requested; allocated; accepted; locked; accepted again ? | 01:26 |
| clarkb | 2017-10-02 23:32:25,350 DEBUG zuul.nodepool: Locking node <Node 0000102036 ubuntu-xenial:ubuntu-xenial> then 2017-10-03 01:18:22,033 DEBUG zuul.nodepool: Locking node <Node 0000102036 primary:ubuntu-xenial> | 01:27 |
| jeblair | http://paste.openstack.org/show/622514/ | 01:30 |
| jeblair | i think the first request disappeared due to zookeeper badness, but zuul locked it anyway (which it was able to do because nodepool offered it in fulfilling a second request) | 01:32 |
| jeblair | so i think the bug here is zuul not checking that the request it's locking a node for still exists | 01:32 |
| *** SumitNaiksatam_ has joined #openstack-infra | 01:32 | |
| clarkb | launcher-debug.log:2017-10-02 23:16:51,400 DEBUG nodepool.DeletedNodeWorker: Unallocating node 0000102036 with missing request 200-0000099439 basically that request there went away and nodepool launcher reacted to that but zuul forged ahead? | 01:33 |
| jeblair | clarkb: yep | 01:33 |
| jeblair | zuul was acting on outdated information | 01:34 |
| jeblair | and by the time zuul got around to locking the node offered in that request, it had been reassigned to another request, so it was actually valid for zuul to lock it | 01:34 |
| *** SumitNaiksatam has quit IRC | 01:35 | |
| *** SumitNaiksatam_ is now known as SumitNaiksatam | 01:35 | |
| jeblair | then eventually zuul gets around to trying to lock it for the second request, and can't, because it owns the lock already | 01:35 |
| clarkb | and that snowballs into being unable to lock for any requests? | 01:35 |
| clarkb | I guess if it handles them serially? | 01:35 |
| jeblair | clarkb: yeah, i think this pattern is repeating due to a lot of queued requests earlier in the day | 01:35 |
| *** lukebrowning has joined #openstack-infra | 01:36 | |
| jeblair | clarkb: i *think* it may have recovered? | 01:36 |
| clarkb | oh wow ya | 01:36 |
| clarkb | 912 nodes in use now | 01:36 |
| clarkb | so maybe this is another source of zuul slowness? | 01:36 |
| jeblair | caused by zuul slowness :( | 01:37 |
| jeblair | anyway, this is fixable. adding to etherpad | 01:37 |
| clarkb | cool, with that I need to get ready for dinner | 01:37 |
| clarkb | thanks for digging into that | 01:37 |
| jeblair | clarkb: thanks for finding it | 01:37 |
| clarkb | (if shrews doesn't dig into fixing that in the morning I may take an attempt at it) | 01:37 |
| jeblair | infra-root: i'd like to land https://review.openstack.org/509040 tonight and restart. we'll have 2 significant zuul scheduler improvements in place for tomorrow | 01:38 |
| clarkb | jeblair: let me review that before dinner then. Also shrews branch matcher fix should go in | 01:39 |
| jeblair | clarkb: yes, thanks, that hasn't landed yet (though is approved) | 01:39 |
| *** lukebrowning has quit IRC | 01:40 | |
| clarkb | jeblair: re 9040 does that serialize all reconfiguration you mean? | 01:40 |
| clarkb | "no other configuration actions can happen between these two steps now." | 01:40 |
| *** wolverineav has quit IRC | 01:41 | |
| *** lukebrowning has joined #openstack-infra | 01:42 | |
| jeblair | clarkb: all reconfiguration should already be serial; i just meant to stress that since the bug was that we were doing something that was arguably part of reconfiguration (clearing the cache) outside that serial line | 01:43 |
| jeblair | iow, it's designed to be serial, and generally is, but that cache clear was not | 01:43 |
| clarkb | gotcha | 01:43 |
| *** camunoz has quit IRC | 01:46 | |
| *** lukebrowning has quit IRC | 01:46 | |
| *** lukebrowning has joined #openstack-infra | 01:48 | |
| clarkb | jeblair: looking at the release() diff merge is going to be an implicit release? why not allow the regex to release that too? | 01:48 |
| *** lnxnut has joined #openstack-infra | 01:50 | |
| clarkb | generally looks good ust that ^ question inline on the change | 01:51 |
| *** lbragstad has joined #openstack-infra | 01:51 | |
| *** lukebrowning has quit IRC | 01:53 | |
| *** lukebrowning has joined #openstack-infra | 01:54 | |
| openstackgerrit | Merged openstack-infra/project-config master: Update Nodepool graphite metric names https://review.openstack.org/508349 | 01:55 |
| *** szaher has quit IRC | 01:57 | |
| *** markmcd has quit IRC | 01:57 | |
| *** lukebrowning has quit IRC | 01:59 | |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack-infra/project-config master: Remove Kolla and Kolla-Ansible jobs https://review.openstack.org/508944 | 02:00 |
| *** lukebrowning has joined #openstack-infra | 02:01 | |
| *** lnxnut has quit IRC | 02:01 | |
| *** baoli has quit IRC | 02:02 | |
| SamYaple | inc0: itll get merged sooner or later..... | 02:02 |
| *** baoli has joined #openstack-infra | 02:03 | |
| SamYaple | ok jeblair, as soon as my zuulv3 patchset series gets merged in LOCI, it should be native v3 with post jobs using secrets! ive very excited to see this happening so quickly | 02:03 |
| SamYaple | thanks for all your help here | 02:03 |
| *** lukebrowning has quit IRC | 02:05 | |
| *** lukebrowning has joined #openstack-infra | 02:07 | |
| *** szaher has joined #openstack-infra | 02:07 | |
| *** kiennt26 has joined #openstack-infra | 02:09 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Switch puppet unit tests base job https://review.openstack.org/508994 | 02:10 |
| *** markmcd has joined #openstack-infra | 02:10 | |
| Jeffrey4l | is there any limitation for the number of running jobs to specify project? | 02:10 |
| inc0 | can I get +workflow on ^? | 02:11 |
| *** lukebrowning has quit IRC | 02:11 | |
| Jeffrey4l | seem some jobs is block for more than 5 hours in zuul. | 02:13 |
| openstackgerrit | Mohammed Naser proposed openstack-infra/openstack-zuul-jobs master: Move tox_envlist variable for releasenotes into job https://review.openstack.org/509045 | 02:14 |
| *** bobh_ has joined #openstack-infra | 02:17 | |
| *** lukebrowning has joined #openstack-infra | 02:18 | |
| openstackgerrit | Sam Yaple proposed openstack-infra/zuul feature/zuulv3: Add additional information about secrets https://review.openstack.org/509047 | 02:20 |
| SamYaple | Jeffrey4l: no! but there is a global number of vms openstack can run, so please don't abuse it | 02:21 |
| Jeffrey4l | SamYaple, do u have any idea why some jobs is running more than 5 hours? | 02:21 |
| SamYaple | Jeffrey4l: zuul has been up and down all day. so it might be a glitch | 02:21 |
| Jeffrey4l | roger. thanks | 02:21 |
| SamYaple | but your gates that you are working on have a timeout of 5400, which is 90 minutes which seems a bit high | 02:22 |
| SamYaple | so it could be that it was queued for 3.5 hours and then took 1.5 hours to run? | 02:22 |
| SamYaple | busy day today :) | 02:22 |
| Jeffrey4l | sometimes build jobs can not be done in 1 hours. | 02:22 |
| *** lukebrowning has quit IRC | 02:23 | |
| Jeffrey4l | inc0 is removing the legacy jobs. and we will try to use some lower timeout when testing something. | 02:24 |
| Jeffrey4l | SamYaple, ^^ | 02:24 |
| openstackgerrit | Sam Yaple proposed openstack-infra/zuul feature/zuulv3: Add additional information about secrets https://review.openstack.org/509047 | 02:25 |
| *** lukebrowning has joined #openstack-infra | 02:26 | |
| SamYaple | Jeffrey4l: you can use what you want, I was just saying what may have happened is all | 02:27 |
| Jeffrey4l | got. | 02:28 |
| inc0 | this timeout was kinda strange, it jsut hung in one place for over an hour | 02:28 |
| inc0 | which was close to zuul having memory issue (close time wise | 02:28 |
| Jeffrey4l | inc0, which place? | 02:28 |
| inc0 | ) | 02:28 |
| inc0 | tacker pip installing | 02:29 |
| inc0 | on one gate, something horizon on another | 02:29 |
| inc0 | both occured ~10min one from another | 02:29 |
| Jeffrey4l | ok. i will check it later. network issue? | 02:30 |
| inc0 | we'll see | 02:30 |
| *** lukebrowning has quit IRC | 02:30 | |
| *** lukebrowning has joined #openstack-infra | 02:32 | |
| openstackgerrit | Tony Breeds proposed openstack-infra/project-config master: Create a publisher for governance.openstack.org https://review.openstack.org/509048 | 02:32 |
| openstackgerrit | Tony Breeds proposed openstack-infra/openstack-zuul-jobs master: Remove the uneeded legacy/static-election-publish playbooks https://review.openstack.org/509049 | 02:33 |
| tonyb | I'm very happy to take corrections ^^^ | 02:33 |
| *** lukebrowning has quit IRC | 02:36 | |
| *** lukebrowning has joined #openstack-infra | 02:38 | |
| *** bnemec has quit IRC | 02:39 | |
| SamYaple | tonyb: from what i saw it looked good. I am not sure if the declaring of non-secrets in the secrets data dict works, but otherwise i think it was all correct syntax wise (I can't speak for some of hte actual tasks though) | 02:39 |
| jeblair | SamYaple, tonyb: secret data doesn't have to be encrypted if that's what you're asking | 02:40 |
| tonyb | SamYaple: where did I declare a non-secret in a secret? | 02:40 |
| jeblair | think of it as a bundle of data | 02:40 |
| jeblair | (the model is borrowed from k8s) | 02:41 |
| tonyb | jeblair: okay I think that's what I did ;P | 02:41 |
| *** lukebrowning has quit IRC | 02:42 | |
| tonyb | does /opt/git exist in the new nodepool images? I have a zuul-cloner failign and I'm trying to work out why ... | 02:43 |
| *** baoli has quit IRC | 02:43 | |
| jeblair | tonyb: no; if zuul-cloner is failing, you probably need to add something to required-projects | 02:43 |
| tonyb | Mind you it's probably better to just rely on related projects and just look there | 02:43 |
| *** lukebrowning has joined #openstack-infra | 02:44 | |
| *** yamamoto has joined #openstack-infra | 02:45 | |
| tonyb | jeblair: okay I though it was there but I'l double check ... more context this is a zull cloner that's being called from a python script from a shell task | 02:45 |
| mordred | jeblair: do you still want to land your change and restart? | 02:45 |
| jeblair | mordred: yeah, i was actually just staging them in a local repo on zuulv3 and am about to restart | 02:46 |
| mordred | jeblair: ok. cool | 02:46 |
| jeblair | we could force-merge, but i still want to keep my backdoor in for more debugging tomorrow, so this works for now i think | 02:46 |
| jeblair | so with that -- i'm going to restart zuul now | 02:46 |
| mordred | jeblair: cool | 02:47 |
| mordred | tonyb, pabelanger: just reading scrollback - I wrote jobs for all of the things that publish to static.o.o earlier today | 02:48 |
| tonyb | mordred: \o/ Yours will probably work :) | 02:48 |
| *** lukebrowning has quit IRC | 02:49 | |
| mordred | tonyb, pabelanger: https://review.openstack.org/#/c/508982/ and https://review.openstack.org/#/c/508983/2 | 02:49 |
| mordred | tonyb: there's likely an issue or two to debug - but that should handle all of the instances | 02:49 |
| *** abelur_ has joined #openstack-infra | 02:49 | |
| tonyb | mordred: Thanks I'll read over them to see how close I was/am | 02:50 |
| mordred | tonyb: woot! | 02:50 |
| *** hongbin has joined #openstack-infra | 02:50 | |
| *** lukebrowning has joined #openstack-infra | 02:51 | |
| *** baoli has joined #openstack-infra | 02:51 | |
| jeblair | re-enqueing changes now | 02:51 |
| SamYaple | jeblair: yeas thats what i was saying, tonyb jeblair clarified for me. I just meant the plain text stuff was "non-secret" was all. but i can be safely ignored :) | 02:53 |
| tonyb | SamYaple: Ahh okay | 02:53 |
| *** lukebrowning has quit IRC | 02:55 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Create static publication base job https://review.openstack.org/508982 | 02:56 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Add jobs for special static publication targets https://review.openstack.org/508983 | 02:56 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Remove centrally defined jobs for openstack/releases https://review.openstack.org/508998 | 02:56 |
| tonyb | mordred: I *think* you need to move the elcetion stuff into 508983 and treat it as speical as IIUC it needs to go into /srv/static/governance/election not /srv/static/election but I guess I need to check system-config to verify that | 02:56 |
| *** baoli has quit IRC | 02:57 | |
| *** lukebrowning has joined #openstack-infra | 02:57 | |
| mordred | tonyb: it's /srv/static/election on the host (I went and checked the original publishers - and also on static.o.o) | 02:58 |
| mordred | tonyb: also - I have fixed the syntax error in https://review.openstack.org/508982 | 02:58 |
| tonyb | mordred: okay cool. | 02:58 |
| *** baoli has joined #openstack-infra | 02:58 | |
| *** lnxnut has joined #openstack-infra | 02:59 | |
| *** bobh_ has quit IRC | 02:59 | |
| *** hongbin has quit IRC | 03:00 | |
| jeblair | #status log zuulv3 restarted with 508786 508787 508793 509014 509040 508955 manually applied; should fix branch matchers, use *slightly* less memory, and fix the 'base job not defined' error | 03:00 |
| *** hongbin has joined #openstack-infra | 03:00 | |
| openstackstatus | jeblair: finished logging | 03:00 |
| jeblair | infra-root: ^ | 03:00 |
| mordred | jeblair: woot | 03:00 |
| * tonyb is happy he wasn't too far wrong. ... and now goes an abandons his work sniff sniff ;P | 03:01 | |
| *** lukebrowning has quit IRC | 03:01 | |
| tonyb | Thanks mordred | 03:02 |
| *** erlon has quit IRC | 03:02 | |
| Jeffrey4l | could we merge this patch? https://review.openstack.org/#/c/508944/ | 03:02 |
| Jeffrey4l | which is helpful for kolla to migrate to zuulv3 jobs. | 03:02 |
| *** lukebrowning has joined #openstack-infra | 03:03 | |
| *** lnxnut has quit IRC | 03:07 | |
| *** lukebrowning has quit IRC | 03:07 | |
| *** lukebrowning has joined #openstack-infra | 03:08 | |
| tonyb | Befoer I take a stab at it ... has anyone started moving projects off of legacy-requirements ? | 03:09 |
| SamYaple | Jeffrey4l: it had a +2+1 but it got rebased. i blame inc0. if anyone can re-workflow https://review.openstack.org/#/c/508944/ for kolla that would be nice | 03:11 |
| *** lukebrowning has quit IRC | 03:13 | |
| *** baoli has quit IRC | 03:14 | |
| *** esberglu has joined #openstack-infra | 03:17 | |
| *** esberglu has quit IRC | 03:17 | |
| *** lbragstad has quit IRC | 03:17 | |
| *** dchen has quit IRC | 03:17 | |
| *** ramishra has joined #openstack-infra | 03:21 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Improve scheduler log messages https://review.openstack.org/509057 | 03:25 |
| *** lukebrowning has joined #openstack-infra | 03:26 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: WIP: Fix TripleO CI jobs https://review.openstack.org/508660 | 03:29 |
| inc0 | SamYaple: got rebased because it got -2 verified due to node_failure:( | 03:29 |
| *** lukebrowning has quit IRC | 03:30 | |
| *** vsaienk0 has joined #openstack-infra | 03:30 | |
| openstackgerrit | Merged openstack-infra/project-config master: Switch jobs to use new release notes job https://review.openstack.org/508763 | 03:31 |
| *** coolsvap has joined #openstack-infra | 03:32 | |
| *** lukebrowning has joined #openstack-infra | 03:32 | |
| SamYaple | inc0: recheck would have fixed that | 03:32 |
| tonyb | I'm a little confused http://git.openstack.org/cgit/openstack-infra/openstack-zuul-jobs/tree/zuul.d/zuul-legacy-jobs.yaml#n10514 indicates that openstack-requirements is a require-project for legacy-requirements but I don't see it in http://logs.openstack.org/08/506308/4/check/legacy-requirements/3042f26/zuul-info/inventory.yaml Should it be there? | 03:33 |
| *** thorst has joined #openstack-infra | 03:33 | |
| SamYaple | and with that im out. i require a breather | 03:34 |
| *** lukebrowning has quit IRC | 03:36 | |
| *** lukebrowning has joined #openstack-infra | 03:38 | |
| *** vsaienk0 has quit IRC | 03:40 | |
| openstackgerrit | Merged openstack-infra/project-config master: Collapse releasenotes jobs to using project template https://review.openstack.org/508769 | 03:41 |
| *** lukebrowning has quit IRC | 03:43 | |
| *** lukebrowning has joined #openstack-infra | 03:45 | |
| *** udesale has joined #openstack-infra | 03:45 | |
| *** Dinesh_Bhor has joined #openstack-infra | 03:47 | |
| *** lukebrowning has quit IRC | 03:49 | |
| *** thorst has quit IRC | 03:50 | |
| *** lukebrowning has joined #openstack-infra | 03:51 | |
| openstackgerrit | Merged openstack-infra/project-config master: Use the openstack-python-jobs-neutron templates https://review.openstack.org/508961 | 03:52 |
| *** dbecker has quit IRC | 03:53 | |
| *** lukebrowning has quit IRC | 03:55 | |
| *** lukebrowning has joined #openstack-infra | 03:57 | |
| *** mrunge has joined #openstack-infra | 04:00 | |
| *** gouthamr has joined #openstack-infra | 04:01 | |
| *** lukebrowning has quit IRC | 04:02 | |
| *** mrunge_ has quit IRC | 04:02 | |
| *** lukebrowning has joined #openstack-infra | 04:03 | |
| *** lnxnut has joined #openstack-infra | 04:04 | |
| *** ijw has quit IRC | 04:06 | |
| *** bhavik1 has joined #openstack-infra | 04:06 | |
| *** ykarel has joined #openstack-infra | 04:07 | |
| *** lukebrowning has quit IRC | 04:08 | |
| *** dbecker has joined #openstack-infra | 04:09 | |
| *** lukebrowning has joined #openstack-infra | 04:10 | |
| *** esberglu has joined #openstack-infra | 04:10 | |
| *** jdandrea_ has quit IRC | 04:11 | |
| *** lukebrowning has quit IRC | 04:14 | |
| *** esberglu has quit IRC | 04:14 | |
| *** lukebrowning has joined #openstack-infra | 04:16 | |
| *** lukebrowning has quit IRC | 04:20 | |
| *** dimak has quit IRC | 04:20 | |
| *** lnxnut has quit IRC | 04:21 | |
| *** oanson has quit IRC | 04:21 | |
| *** dimak has joined #openstack-infra | 04:22 | |
| *** garyk1 has joined #openstack-infra | 04:26 | |
| *** lukebrowning has joined #openstack-infra | 04:27 | |
| *** oanson has joined #openstack-infra | 04:30 | |
| *** sree has joined #openstack-infra | 04:30 | |
| *** lukebrowning has quit IRC | 04:31 | |
| *** hongbin has quit IRC | 04:32 | |
| *** lukebrowning has joined #openstack-infra | 04:33 | |
| *** yamahata has joined #openstack-infra | 04:33 | |
| *** armax has quit IRC | 04:35 | |
| *** lukebrowning has quit IRC | 04:38 | |
| *** lukebrowning has joined #openstack-infra | 04:39 | |
| *** xarses has joined #openstack-infra | 04:41 | |
| *** lukebrowning has quit IRC | 04:44 | |
| *** claudiub|2 has joined #openstack-infra | 04:45 | |
| *** lukebrowning has joined #openstack-infra | 04:46 | |
| *** ijw has joined #openstack-infra | 04:47 | |
| *** lukebrowning has quit IRC | 04:50 | |
| *** ijw_ has joined #openstack-infra | 04:50 | |
| *** gouthamr has quit IRC | 04:50 | |
| *** lukebrowning has joined #openstack-infra | 04:52 | |
| *** ijw has quit IRC | 04:53 | |
| *** ramishra has quit IRC | 04:54 | |
| *** ramishra has joined #openstack-infra | 04:55 | |
| *** lukebrowning has quit IRC | 04:56 | |
| openstackgerrit | Hiroaki Kobayashi proposed openstack-infra/project-config master: Add required-projects to the blazar-dashboard jobs https://review.openstack.org/509078 | 04:57 |
| *** lukebrowning has joined #openstack-infra | 04:58 | |
| *** ijw_ has quit IRC | 04:58 | |
| openstackgerrit | Hiroaki Kobayashi proposed openstack-infra/project-config master: Add required-projects to the blazar-dashboard jobs https://review.openstack.org/509078 | 04:58 |
| openstackgerrit | Hiroaki Kobayashi proposed openstack-infra/project-config master: Add required-projects to the blazar-dashboard jobs https://review.openstack.org/509078 | 05:01 |
| *** pcaruana has joined #openstack-infra | 05:01 | |
| openstackgerrit | Hiroaki Kobayashi proposed openstack-infra/project-config master: Add required-projects to the blazar-dashboard jobs https://review.openstack.org/509078 | 05:01 |
| openstackgerrit | Merged openstack-infra/project-config master: Remove Kolla and Kolla-Ansible jobs https://review.openstack.org/508944 | 05:02 |
| *** lukebrowning has quit IRC | 05:02 | |
| *** markvoelker_ has joined #openstack-infra | 05:03 | |
| *** psachin has joined #openstack-infra | 05:04 | |
| *** lukebrowning has joined #openstack-infra | 05:04 | |
| *** markvoelker has quit IRC | 05:05 | |
| *** lukebrowning has quit IRC | 05:09 | |
| *** rwsu has quit IRC | 05:10 | |
| *** masber has joined #openstack-infra | 05:12 | |
| *** bhavik1 has quit IRC | 05:13 | |
| *** lnxnut has joined #openstack-infra | 05:18 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Move shadow layout to item https://review.openstack.org/509014 | 05:21 |
| *** lukebrowning has joined #openstack-infra | 05:21 | |
| *** pcaruana has quit IRC | 05:23 | |
| *** lukebrowning has quit IRC | 05:26 | |
| *** lukebrowning has joined #openstack-infra | 05:27 | |
| *** lnxnut has quit IRC | 05:28 | |
| *** deduped has joined #openstack-infra | 05:30 | |
| *** nunchuck has joined #openstack-infra | 05:30 | |
| *** lukebrowning has quit IRC | 05:32 | |
| *** rwsu has joined #openstack-infra | 05:33 | |
| *** lukebrowning has joined #openstack-infra | 05:34 | |
| *** oanson has quit IRC | 05:36 | |
| *** mat128 has quit IRC | 05:36 | |
| *** oanson has joined #openstack-infra | 05:36 | |
| *** markvoelker_ has quit IRC | 05:37 | |
| *** markvoelker has joined #openstack-infra | 05:37 | |
| *** lukebrowning has quit IRC | 05:38 | |
| openstackgerrit | garyk proposed openstack-infra/project-config master: Add missing projects to vmware-nsxlib https://review.openstack.org/508779 | 05:39 |
| *** lukebrowning has joined #openstack-infra | 05:40 | |
| *** pgadiya has joined #openstack-infra | 05:40 | |
| *** dfflanders has joined #openstack-infra | 05:41 | |
| *** markvoelker has quit IRC | 05:42 | |
| *** xarses has quit IRC | 05:44 | |
| *** lukebrowning has quit IRC | 05:44 | |
| *** lukebrowning has joined #openstack-infra | 05:46 | |
| *** namnh has quit IRC | 05:49 | |
| *** namnh has joined #openstack-infra | 05:49 | |
| *** lukebrowning has quit IRC | 05:51 | |
| openstackgerrit | garyk proposed openstack-infra/project-config master: Update vmware-nsx to use openstack-*-neutron job https://review.openstack.org/509087 | 05:52 |
| *** lukebrowning has joined #openstack-infra | 05:52 | |
| *** rcernin has joined #openstack-infra | 05:53 | |
| *** jamesdenton has quit IRC | 05:56 | |
| *** sshnaidm|afk is now known as sshnaidm | 05:56 | |
| *** lukebrowning has quit IRC | 05:57 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: WIP: Fix TripleO CI jobs https://review.openstack.org/508660 | 05:58 |
| *** vsaienk0 has joined #openstack-infra | 06:04 | |
| *** aeng has quit IRC | 06:06 | |
| *** pgadiya has quit IRC | 06:14 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Add missing repo to legacy-openstackci-beaker jobs https://review.openstack.org/508857 | 06:15 |
| SpamapS | tonyb: note in the inheritance path: - 'apply variant <Job legacy-requirements branches: None source: openstack-infra/openstack-zuul-jobs/zuul.d/zuul-legacy-project-templates.yaml@master#174>' | 06:16 |
| openstackgerrit | YAMAMOTO Takashi proposed openstack-infra/project-config master: networking-midonet: Add required-projects for tox jobs https://review.openstack.org/509089 | 06:16 |
| *** lukebrowning has joined #openstack-infra | 06:17 | |
| tonyb | SpamapS: okay so does that mean I need to add the require-projecst there rather than where it is? | 06:18 |
| garyk1 | question - anyone know how i can increase the timeout in a project template? | 06:18 |
| *** kiennt26 has quit IRC | 06:19 | |
| tonyb | garyk1: Just guessing but somethign like http://git.openstack.org/cgit/openstack-infra/openstack-zuul-jobs/tree/zuul.d/zuul-legacy-jobs.yaml#n10518 on the template itself? | 06:19 |
| *** kiennt26 has joined #openstack-infra | 06:19 | |
| garyk1 | tonyb: thanks! i'll try that | 06:20 |
| *** ramishra has quit IRC | 06:20 | |
| *** ramishra has joined #openstack-infra | 06:22 | |
| *** lukebrowning has quit IRC | 06:22 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Remove legacy-releases-python35 https://review.openstack.org/509091 | 06:22 |
| *** Apoorva has joined #openstack-infra | 06:23 | |
| openstackgerrit | garyk proposed openstack-infra/project-config master: Increase timeouts for vmware-nsxlib https://review.openstack.org/509092 | 06:23 |
| *** nikhil has quit IRC | 06:24 | |
| *** lnxnut has joined #openstack-infra | 06:24 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Remove legacy static publication jobs https://review.openstack.org/509093 | 06:27 |
| *** isaacb has joined #openstack-infra | 06:29 | |
| *** nmagnezi has joined #openstack-infra | 06:29 | |
| *** jtomasek has joined #openstack-infra | 06:30 | |
| *** Apoorva has quit IRC | 06:30 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Remove legacy static publication jobs https://review.openstack.org/509093 | 06:31 |
| *** itzikb has joined #openstack-infra | 06:31 | |
| itzikb | hi, can someone help me please to figure out why zuul is not happy - https://review.openstack.org/#/c/466528/? | 06:31 |
| *** lukebrowning has joined #openstack-infra | 06:32 | |
| *** lnxnut has quit IRC | 06:34 | |
| *** slaweq has quit IRC | 06:35 | |
| *** slaweq has joined #openstack-infra | 06:36 | |
| *** lukebrowning has quit IRC | 06:37 | |
| eumel8 | itzikb: ssh unreachable - from my experience that works for me after a long night now. recheck is already set, hopefully it will go trough | 06:37 |
| itzikb | eumel8: thanks! :-) | 06:37 |
| *** ociuhandu has joined #openstack-infra | 06:38 | |
| *** lukebrowning has joined #openstack-infra | 06:38 | |
| *** slaweq has quit IRC | 06:40 | |
| eumel8 | http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=63999&rra_id=all | 06:42 |
| eumel8 | but there is still some high load on the zuul server. I think the infra team will continue to fight with zuul-v3 after a break. | 06:42 |
| *** slaweq has joined #openstack-infra | 06:42 | |
| *** lukebrowning has quit IRC | 06:42 | |
| *** pcaruana has joined #openstack-infra | 06:44 | |
| *** lukebrowning has joined #openstack-infra | 06:44 | |
| *** lukebrowning has quit IRC | 06:48 | |
| *** lukebrowning has joined #openstack-infra | 06:51 | |
| sshnaidm | does anybody know where are logs of legacy periodic jobs now? | 06:52 |
| eumel8 | AJaeger: no luck with 508857 | 06:52 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/zuul feature/zuulv3: Improve scheduler log messages https://review.openstack.org/509057 | 06:52 |
| *** dhajare has joined #openstack-infra | 06:52 | |
| *** esberglu has joined #openstack-infra | 06:55 | |
| *** esberglu has quit IRC | 06:55 | |
| *** lukebrowning has quit IRC | 06:55 | |
| *** CHIPPY has joined #openstack-infra | 06:56 | |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Add content to support translation jobs https://review.openstack.org/502207 | 06:59 |
| *** dizquierdo has joined #openstack-infra | 07:06 | |
| *** shardy has joined #openstack-infra | 07:11 | |
| openstackgerrit | Merged openstack-infra/project-config master: Fix OVB jobs config for TripleO https://review.openstack.org/508936 | 07:12 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Move tox_envlist variable for releasenotes into job https://review.openstack.org/509045 | 07:12 |
| *** migi_ is now known as migi | 07:13 | |
| *** tesseract has joined #openstack-infra | 07:17 | |
| openstackgerrit | Merged openstack-infra/project-config master: Update vmware-nsx to use openstack-*-neutron job https://review.openstack.org/509087 | 07:22 |
| openstackgerrit | Merged openstack-infra/project-config master: networking-midonet: Add required-projects for tox jobs https://review.openstack.org/509089 | 07:22 |
| *** ralonsoh has joined #openstack-infra | 07:24 | |
| *** nmagnezi has left #openstack-infra | 07:28 | |
| *** sree has quit IRC | 07:30 | |
| openstackgerrit | Eyal Leshem proposed openstack-infra/project-config master: Add neutron to required-projects for dragonflow https://review.openstack.org/508785 | 07:30 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Print environment in pre playbook of legacy-base job https://review.openstack.org/509009 | 07:30 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Add neutron-lib as required to legacy-tempest-dsvm-neutron-src https://review.openstack.org/508945 | 07:30 |
| *** lnxnut has joined #openstack-infra | 07:31 | |
| *** sree has joined #openstack-infra | 07:31 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Clear project config cache later https://review.openstack.org/509040 | 07:32 |
| ykarel | AJaeger, are there any issue with pypi mirror http://mirror.regionone.tripleo-test-cloud-rh1.openstack.org/pypi? http://logs.openstack.org/79/509079/1/check-tripleo/legacy-tripleo-ci-centos-7-ovb-ha-oooq/2fd15ce/job-output.txt.gz | 07:33 |
| *** isaacb has quit IRC | 07:33 | |
| *** pt_15 has joined #openstack-infra | 07:33 | |
| *** CHIPPY has quit IRC | 07:34 | |
| *** sree has quit IRC | 07:35 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/project-config master: Add required project to puppet-tripleo https://review.openstack.org/509100 | 07:38 |
| sshnaidm | jaosorior, ^^ | 07:38 |
| *** markvoelker has joined #openstack-infra | 07:38 | |
| *** jpena|off is now known as jpena | 07:39 | |
| *** lnxnut has quit IRC | 07:41 | |
| jaosorior | sshnaidm: why do we need puppet-openstack integration? | 07:43 |
| sshnaidm | jaosorior, good question to infra.. | 07:47 |
| *** sshnaidm is now known as sshnaidm|afk | 07:47 | |
| *** egonzalez has joined #openstack-infra | 07:48 | |
| *** esberglu has joined #openstack-infra | 07:48 | |
| *** lukebrowning has joined #openstack-infra | 07:51 | |
| openstackgerrit | Eyal Leshem proposed openstack-infra/project-config master: Add neutron to required-projects for dragonflow https://review.openstack.org/508785 | 07:51 |
| *** sshnaidm|afk has quit IRC | 07:52 | |
| *** ccamacho has joined #openstack-infra | 07:52 | |
| *** esberglu has quit IRC | 07:53 | |
| *** jpich has joined #openstack-infra | 07:54 | |
| AJaeger | ykarel: I just came online and can't help... | 07:56 |
| *** hashar has joined #openstack-infra | 07:57 | |
| *** rwsu has quit IRC | 07:57 | |
| *** bandini has quit IRC | 07:57 | |
| *** kiennt26 has quit IRC | 07:59 | |
| *** kiennt26 has joined #openstack-infra | 08:00 | |
| *** alexchadin has joined #openstack-infra | 08:00 | |
| *** namnh_ has joined #openstack-infra | 08:02 | |
| *** bandini has joined #openstack-infra | 08:02 | |
| *** namnh has quit IRC | 08:04 | |
| *** lukebrowning has quit IRC | 08:04 | |
| *** Hal has joined #openstack-infra | 08:04 | |
| *** Hal is now known as Guest11048 | 08:05 | |
| *** sree has joined #openstack-infra | 08:05 | |
| kazsh | AJaeger: G’day, regarding your comment for zuul v3 https://review.openstack.org/#/c/504104/ , what I need to do is to abandon the current change and make an another change adding zuul.d/project.yaml behalf of jenkins/jobs/projects.yaml & zuul/layout.yaml, correct ? | 08:06 |
| *** lukebrowning has joined #openstack-infra | 08:06 | |
| *** rwsu has joined #openstack-infra | 08:09 | |
| *** lukebrowning has quit IRC | 08:10 | |
| *** markvoelker has quit IRC | 08:12 | |
| *** lukebrowning has joined #openstack-infra | 08:12 | |
| *** ociuhandu has quit IRC | 08:14 | |
| *** isaacb has joined #openstack-infra | 08:14 | |
| *** boris_42_ has quit IRC | 08:15 | |
| *** yboaron_ has joined #openstack-infra | 08:17 | |
| *** lukebrowning has quit IRC | 08:17 | |
| *** lukebrowning has joined #openstack-infra | 08:18 | |
| AJaeger | eumel8: eventually ;) | 08:19 |
| AJaeger | kazsh: yes. | 08:20 |
| AJaeger | kazsh: you use standard jobs, so zuul.d/projects.yaml should be all you need to do... | 08:20 |
| *** oanson has quit IRC | 08:21 | |
| AJaeger | kazsh: and then need PTL+1 on it before we can move forward | 08:21 |
| AJaeger | bbl | 08:21 |
| *** oanson has joined #openstack-infra | 08:22 | |
| *** lukebrowning has quit IRC | 08:23 | |
| *** apevec has joined #openstack-infra | 08:23 | |
| *** snapiri has joined #openstack-infra | 08:25 | |
| kazsh | AJaeger: Thank mate, will do that accordingly. | 08:25 |
| AJaeger | infra-root, really interesting gate pipeline constellation: 508996 is waiting for 3 nodes on top of gate but the changes behind it (508882 and 508934) have same jobs and finished | 08:25 |
| ianw | AJaeger: i feel like i saw something fly by about node ordering, but maybe that was between pipelines | 08:26 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Handle z-c shim copies across filesystems https://review.openstack.org/508772 | 08:27 |
| openstackgerrit | Merged openstack-infra/project-config master: Update releases repo to use openstack-python35-jobs template https://review.openstack.org/508978 | 08:27 |
| openstackgerrit | Merged openstack-infra/project-config master: Create static publication base job https://review.openstack.org/508982 | 08:27 |
| openstackgerrit | Merged openstack-infra/project-config master: Add jobs for special static publication targets https://review.openstack.org/508983 | 08:27 |
| openstackgerrit | Merged openstack-infra/project-config master: Remove legacy Packstack integration jobs https://review.openstack.org/508851 | 08:27 |
| openstackgerrit | Merged openstack-infra/project-config master: Remove duplicate definition of OSA integrated AIO job https://review.openstack.org/508931 | 08:27 |
| openstackgerrit | Merged openstack-infra/project-config master: Remove legacy-rally-dsvm-keystone-v2api-rally job https://review.openstack.org/508833 | 08:27 |
| openstackgerrit | Merged openstack-infra/project-config master: [rally] fix cases when *verify* job should be launched https://review.openstack.org/508834 | 08:27 |
| *** alexchadin has quit IRC | 08:27 | |
| *** lukebrowning has joined #openstack-infra | 08:28 | |
| kklimonda | how are job-output.txt.gz converted into html (with clickable timestampts) ? | 08:29 |
| AJaeger | ianw: is there anything broken with those 3 nodes for the job? | 08:29 |
| *** lucas-afk is now known as lucasagomes | 08:30 | |
| *** efoley has joined #openstack-infra | 08:30 | |
| *** sshnaidm|afk has joined #openstack-infra | 08:32 | |
| ianw | kklimonda: google "os_loganalyze" | 08:33 |
| *** lukebrowning has quit IRC | 08:33 | |
| kklimonda | ah, thanks :) | 08:33 |
| ianw | AJaeger: not that i can see, but ... executor issues have been the issue of the day and i'm not up on debugging them | 08:34 |
| *** lukebrowning has joined #openstack-infra | 08:35 | |
| AJaeger | So, wait longer or kick the change out of the queue? | 08:35 |
| * AJaeger can update commit message and reenqueue - instead of waiting for ages. | 08:35 | |
| *** alexchadin has joined #openstack-infra | 08:35 | |
| ianw | AJaeger: just trying to see if i can tell where it was assigned | 08:35 |
| sshnaidm|afk | hi, is it possible to check which permission images have? I have an error: Permission denied: '/opt/cache/files/cirros-0.3.5-x86_64-disk.img' | 08:36 |
| * AJaeger will wait a bit longer - first run some errands | 08:36 | |
| ianw | sshnaidm|afk: i bet that has to do with jenkins v zuul user, somehow | 08:36 |
| *** lukebrowning_ has joined #openstack-infra | 08:37 | |
| ykarel | AJaeger, Ok | 08:37 |
| ianw | Execute job build-openstack-sphinx-docs (uuid: 4f15233aaab84619b6b34de4202d004d) on nodes <NodeSet OrderedDict([('ubuntu-xenial', <Node 0000114764 ubuntu-xenial:ubuntu-xenial>)]) | 08:39 |
| *** lnxnut has joined #openstack-infra | 08:39 | |
| *** lukebrowning has quit IRC | 08:39 | |
| ianw | | 0000114764 | citycloud-kna1 | nova | ubuntu-xenial | 8cd21062-3546-4a4d-8cae-d8f4fecdd494 | in-use | 00:01:18:01 | 08:40 |
| ianw | it's sitting there | 08:41 |
| sshnaidm|afk | ianw, any thoughts where could it be fixed? | 08:41 |
| *** sshnaidm|afk is now known as sshnaidm | 08:41 | |
| ianw | sshnaidm|afk: i need more details on where you're actually seeing the problem | 08:41 |
| *** lukebrowning_ has quit IRC | 08:41 | |
| *** dingyichen has quit IRC | 08:41 | |
| yboaron_ | Hi guys, | 08:41 |
| yboaron_ | I’m trying to figure out what’s wrong with the following patch: | 08:41 |
| yboaron_ | https://review.openstack.org/#/c/498253/ with no success. | 08:41 |
| *** spectr has joined #openstack-infra | 08:41 | |
| yboaron_ | I can see the error reported by zuul,but can’t see any relevant log file, and no conflict is reported in gerrit. | 08:42 |
| yboaron_ | I would be very grateful for any feedback on this patch failure | 08:42 |
| *** lukebrowning has joined #openstack-infra | 08:42 | |
| leyal | AJaeger , thanks a lot for your help with https://review.openstack.org/#/c/508785/17..18 | 08:43 |
| ianw | AJaeger: that host is up, and zuul has never tried to log into it. this is similar to issues i saw with yolanda the other day | 08:43 |
| *** derekh has joined #openstack-infra | 08:44 | |
| leyal | AJaeger - but i don't sure that i understand your comment there as publish tarball and shpinx-docs are part of openstack-python-jobs-neutron | 08:44 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: WIP: Fix TripleO CI jobs https://review.openstack.org/508660 | 08:46 |
| sshnaidm | ianw, just trying to use it with tempest | 08:46 |
| *** lukebrowning has quit IRC | 08:46 | |
| openstackgerrit | Pavlo Shchelokovskyy proposed openstack-infra/project-config master: Add separate coverage job for ironic-inspector https://review.openstack.org/508129 | 08:47 |
| snapiri | yboaron_: There is an issue with SSH connection to some host. Take a look at the job-output.txt file (it is actually gzipped), I see that ianw is on it :) | 08:47 |
| sshnaidm | ianw, this is actual command that uses it: http://logs.openstack.org/60/508660/12/check/legacy-tripleo-ci-centos-7-scenario003-multinode-oooq-container/5d26f07/logs/undercloud/home/zuul/tempest_output.log.txt.gz#_2017-10-03_07_45_32 | 08:48 |
| *** lnxnut has quit IRC | 08:48 | |
| sshnaidm | I'm fine to fix it, but I don't know where in infras code it's created | 08:48 |
| *** lukebrowning has joined #openstack-infra | 08:48 | |
| ianw | AJaeger: it's all just unknown, i can't see any zuul executor has picked it up. | 08:49 |
| ianw | AJaeger: so, in short, i'd restart it, i have no idea why it's in purgatory | 08:49 |
| openstackgerrit | Hiroaki Kobayashi proposed openstack-infra/project-config master: Add required-projects to the blazar-dashboard jobs https://review.openstack.org/509078 | 08:49 |
| ianw | -rw------- 1 jenkins jenkins 17104896 Oct 2 15:31 /opt/cache/files/cirros-0.3.2-i386-disk.vmdk | 08:50 |
| ianw | that would be an issue i'd say | 08:50 |
| *** lukebrowning has quit IRC | 08:53 | |
| *** tikitavi has joined #openstack-infra | 08:54 | |
| *** lukebrowning has joined #openstack-infra | 08:55 | |
| yboaron_ | snapiri, 10x , will check it | 08:56 |
| *** tosky has joined #openstack-infra | 08:57 | |
| *** lukebrowning has quit IRC | 08:59 | |
| *** hashar has quit IRC | 09:01 | |
| *** lukebrowning has joined #openstack-infra | 09:01 | |
| AJaeger | leyal: I mean: You install networking-sfc in the check and gate pipeline, you need it for the post and release pipeline as well. Otherwise you cannot publish anything | 09:02 |
| AJaeger | ianw: ok, will restart these... | 09:03 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Add openstack-tox-validate job https://review.openstack.org/508996 | 09:04 |
| *** electrofelix has joined #openstack-infra | 09:04 | |
| *** efoley has quit IRC | 09:04 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config master: Open read permissions on /opt/cache/files/* https://review.openstack.org/509116 | 09:05 |
| *** efoley_ has joined #openstack-infra | 09:05 | |
| leyal | AJaeger , Is the tarball task should contain all project dependencies? and which other task mat require a the project dependencies? | 09:05 |
| *** lukebrowning has quit IRC | 09:05 | |
| ianw | sshnaidm: ^^ see that ... it will take some time to make it into images however. your best bet might be a workaround to chown it | 09:05 |
| *** hashar has joined #openstack-infra | 09:06 | |
| sshnaidm | ianw, yeah, I have a workaround, just don't want it to be lost with all these issues.. | 09:06 |
| ianw | sshnaidm: np, we'll keep an eye on 509116 and work from there | 09:07 |
| *** lukebrowning has joined #openstack-infra | 09:07 | |
| *** markvoelker has joined #openstack-infra | 09:09 | |
| AJaeger | leyal: the tarball runs tox -e venv - and your tox command requires some repos to be available. So, you need to set those up | 09:09 |
| *** alexchadin has quit IRC | 09:10 | |
| *** namnh has joined #openstack-infra | 09:12 | |
| *** lukebrowning has quit IRC | 09:12 | |
| AJaeger | ianw: question on your change - should we use zuul:zuul for permissions? And then still do the a+r? | 09:12 |
| *** lukebrowning has joined #openstack-infra | 09:13 | |
| AJaeger | Argh, zuul takes some time to figure out that I rebased 508996... | 09:14 |
| ianw | AJaeger: i dunno, i think the whole thing is probably suspect ... pabelanger might suggest we just get rid of the whole thing, i'm not sure any of it's relevant | 09:14 |
| *** namnh_ has quit IRC | 09:14 | |
| leyal | AJaeger - do you mean the cloning of those repository in the tox_install.sh? | 09:14 |
| AJaeger | leyal: yes. Since you do that, any job that runs tox needs the repos installed | 09:15 |
| *** namnh_ has joined #openstack-infra | 09:15 | |
| *** ociuhandu has joined #openstack-infra | 09:16 | |
| *** namnh has quit IRC | 09:17 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Add openstack-tox-validate job https://review.openstack.org/508996 | 09:18 |
| *** lukebrowning has quit IRC | 09:18 | |
| leyal | AJaeger , I thought that with zuul3 that clone should be reduced as it's done by the infra .. | 09:18 |
| yuval | Heet infra, anyone mind looking at https://review.openstack.org/508807 ? Karbor gate is broken for 2 days because of the LIBS_FROM_GIT issue and the fix in https://review.openstack.org/508344 is not merging | 09:19 |
| *** lukebrowning has joined #openstack-infra | 09:20 | |
| yuval | *hey | 09:20 |
| AJaeger | leyal: yes, migrating your jobs you should get rid of zuul-cloner. But still, you need to tell zuulv3 for it. | 09:20 |
| AJaeger | yuval: 508344 is in gate queue right now. | 09:20 |
| AJaeger | infra-root, can we promote 508344? | 09:20 |
| yuval | AJaeger: it was in gate twice yesterday and failed :\ | 09:20 |
| openstackgerrit | Kazunori Shinohara proposed openstack-infra/project-config master: WIP: Add heat-dashboard project https://review.openstack.org/509119 | 09:21 |
| yuval | AJaeger: meanwhile, we are blocked | 09:21 |
| AJaeger | yuval: getting your changes in will not be quicker, zuul seems unhappy right now ;( | 09:21 |
| *** askb has quit IRC | 09:21 | |
| yuval | AJaeger: alright. Anything I can do to help? | 09:22 |
| AJaeger | yuval: not that I'm aware of ;( Let's see what the rest of the team says later today... | 09:23 |
| AJaeger | ianw: still around and can promote 508344? | 09:23 |
| *** sdague has joined #openstack-infra | 09:23 | |
| *** hashar has quit IRC | 09:23 | |
| *** askb has joined #openstack-infra | 09:23 | |
| *** hashar has joined #openstack-infra | 09:24 | |
| *** yamamoto has quit IRC | 09:24 | |
| * AJaeger fears we need to restart zuul soon but would rather wait for jeblair to be around for that | 09:24 | |
| *** lukebrowning has quit IRC | 09:24 | |
| *** priteau has joined #openstack-infra | 09:25 | |
| leyal | AJaeger: so i think the project-config patch is ok - as in the past we run the zuul_cloner for networking-sfc in some places (like the tarball) without really need that .. | 09:26 |
| AJaeger | leyal: post job runs tox -e ..., so will fail since you install it *everytime*. | 09:26 |
| AJaeger | time to start cooking - bbl... | 09:26 |
| mandre | Hi, can anyone help me figure out why we don't see running jobs for https://review.openstack.org/#/c/508661/? | 09:27 |
| *** igordc has joined #openstack-infra | 09:27 | |
| bcafarel | I was about to ask the same kind of question, I sent a recheck message on https://review.openstack.org/#/c/419212/ but it did not get in the queue | 09:31 |
| *** alexchadin has joined #openstack-infra | 09:31 | |
| openstackgerrit | Thomas Morin proposed openstack-infra/project-config master: Add projects to required-projects for networking-(bagpipe|bgpvpn) https://review.openstack.org/508825 | 09:34 |
| *** lukebrowning has joined #openstack-infra | 09:35 | |
| *** caphrim007 has quit IRC | 09:37 | |
| *** caphrim007 has joined #openstack-infra | 09:37 | |
| *** lukebrowning has quit IRC | 09:40 | |
| *** yamamoto has joined #openstack-infra | 09:41 | |
| *** markvoelker has quit IRC | 09:42 | |
| *** apetrich has quit IRC | 09:42 | |
| *** wolverineav has joined #openstack-infra | 09:44 | |
| *** sree has quit IRC | 09:45 | |
| *** vponomaryov has quit IRC | 09:45 | |
| *** sree has joined #openstack-infra | 09:45 | |
| *** vponomaryov has joined #openstack-infra | 09:45 | |
| yboaron_ | Snapiri,ianw : many thanks for your time and effort, | 09:46 |
| yboaron_ | as per https://review.openstack.org/#/c/498253/, | 09:46 |
| yboaron_ | I saw the ssh time out error - it’s on ‘legacy-tempest-dsvm-octavia-kuryr-kubernetes’ (job-output file: http://logs.openstack.org/53/498253/18/check/legacy-tempest-dsvm-octavia-kuryr-kubernetes/13f3a7c/job-output.txt.gz#_2017-10-02_17_31_15_048702 ) | 09:46 |
| yboaron_ | The ‘legacy-tempest-dsvm-octavia-kuryr-kubernetes’ is non voting , so I assume it should not trigger failure , right ? | 09:46 |
| yboaron_ | I can’t understand why ‘legacy-install-dsvm-default-kuryr-kubernetes’ failed . | 09:46 |
| *** yamamoto has quit IRC | 09:46 | |
| yboaron_ | Didn’t find any error in relevant job output (http://logs.openstack.org/53/498253/18/check/legacy-install-dsvm-default-kuryr-kubernetes/cc0dd55/job-output.txt.gz ) | 09:46 |
| yboaron_ | Highly appreciate if you can advise how to proceed | 09:46 |
| *** lnxnut has joined #openstack-infra | 09:46 | |
| *** yamamoto has joined #openstack-infra | 09:46 | |
| *** lukebrowning has joined #openstack-infra | 09:48 | |
| *** sree has quit IRC | 09:50 | |
| *** pbourke has quit IRC | 09:50 | |
| openstackgerrit | Thomas Morin proposed openstack-infra/project-config master: Add projects to required-projects for networking-(bagpipe|bgpvpn) https://review.openstack.org/508825 | 09:51 |
| *** udesale has quit IRC | 09:51 | |
| *** yamamoto has quit IRC | 09:51 | |
| *** lukebrowning has quit IRC | 09:52 | |
| *** alexchadin has quit IRC | 09:53 | |
| leyal | AJaeger, we planing to remove the zuul-cloner calls see : https://review.openstack.org/#/c/508761/ | 09:53 |
| *** tushar has quit IRC | 09:54 | |
| *** lukebrowning has joined #openstack-infra | 09:54 | |
| *** kjackal_ has joined #openstack-infra | 09:54 | |
| *** lnxnut has quit IRC | 09:56 | |
| *** pbourke has joined #openstack-infra | 09:57 | |
| *** admin0 has joined #openstack-infra | 09:58 | |
| AJaeger | leyal: how will that one work for developers? | 09:58 |
| *** efoley_ has quit IRC | 09:58 | |
| AJaeger | leyal: sorry, don't have time right now for further help... | 09:59 |
| *** lukebrowning has quit IRC | 09:59 | |
| *** lukebrowning has joined #openstack-infra | 09:59 | |
| *** yamamoto has joined #openstack-infra | 10:00 | |
| *** priteau has quit IRC | 10:00 | |
| *** ykarel is now known as ykarel|afk | 10:01 | |
| openstackgerrit | Lucas Alvares Gomes proposed openstack-infra/project-config master: Add Neutron scenario job to networking-ovn https://review.openstack.org/509129 | 10:01 |
| *** isaacb has quit IRC | 10:03 | |
| leyal | AJaeger , isn't it done by the infra now ? | 10:03 |
| *** isaacb has joined #openstack-infra | 10:03 | |
| *** lukebrowning has quit IRC | 10:04 | |
| *** efoley has joined #openstack-infra | 10:05 | |
| *** tushar has joined #openstack-infra | 10:05 | |
| *** deduped has quit IRC | 10:05 | |
| *** nunchuck has quit IRC | 10:05 | |
| *** lukebrowning has joined #openstack-infra | 10:05 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Fix branch matching logic https://review.openstack.org/508955 | 10:08 |
| *** yamahata has quit IRC | 10:08 | |
| *** wolverineav has quit IRC | 10:10 | |
| *** cuongnv has quit IRC | 10:10 | |
| *** leyale has joined #openstack-infra | 10:10 | |
| *** lukebrowning has quit IRC | 10:10 | |
| *** wolverineav has joined #openstack-infra | 10:11 | |
| bcafarel | ok job appeared in the queue for https://review.openstack.org/#/c/419212/, forget my question | 10:11 |
| openstackgerrit | Hiroaki Kobayashi proposed openstack-infra/project-config master: Add required-projects to the blazar-dashboard jobs https://review.openstack.org/509078 | 10:11 |
| *** lukebrowning has joined #openstack-infra | 10:12 | |
| *** sshnaidm is now known as sshnaidm|lnch | 10:14 | |
| *** wolverineav has quit IRC | 10:15 | |
| *** gildub has quit IRC | 10:15 | |
| *** lukebrowning has quit IRC | 10:16 | |
| *** kiennt26 has quit IRC | 10:16 | |
| ianw | AJaeger: i can see 508344 has already failed. at this point, i think we're back to a zuul restart option which is going to to wait for the usa main devs | 10:18 |
| *** lukebrowning has joined #openstack-infra | 10:18 | |
| *** jascott1 has quit IRC | 10:20 | |
| *** jascott1 has joined #openstack-infra | 10:21 | |
| *** claudiub has joined #openstack-infra | 10:22 | |
| *** lukebrowning has quit IRC | 10:22 | |
| *** tumbarka has joined #openstack-infra | 10:23 | |
| *** lukebrowning has joined #openstack-infra | 10:24 | |
| *** claudiub|2 has quit IRC | 10:25 | |
| *** jascott1 has quit IRC | 10:26 | |
| openstackgerrit | YAMAMOTO Takashi proposed openstack-infra/project-config master: tap-as-a-service: Add neutron requirement for tox jobs https://review.openstack.org/509134 | 10:26 |
| openstackgerrit | YAMAMOTO Takashi proposed openstack-infra/project-config master: neutron-vpnaas: Add neutron requirement for tox jobs https://review.openstack.org/509135 | 10:26 |
| *** tushar has quit IRC | 10:26 | |
| *** efoley_ has joined #openstack-infra | 10:28 | |
| *** efoley has quit IRC | 10:28 | |
| *** wolverineav has joined #openstack-infra | 10:29 | |
| *** stakeda has quit IRC | 10:29 | |
| *** lukebrowning has quit IRC | 10:29 | |
| openstackgerrit | YAMAMOTO Takashi proposed openstack-infra/project-config master: neutron-fwaas: Add requirements for tox jobs https://review.openstack.org/509137 | 10:32 |
| *** esberglu has joined #openstack-infra | 10:33 | |
| *** esberglu has quit IRC | 10:33 | |
| *** efoley has joined #openstack-infra | 10:34 | |
| *** efoley_ has quit IRC | 10:34 | |
| *** jkilpatr has quit IRC | 10:34 | |
| *** oanson has quit IRC | 10:38 | |
| *** leyale has quit IRC | 10:39 | |
| *** markvoelker has joined #openstack-infra | 10:39 | |
| *** oanson has joined #openstack-infra | 10:40 | |
| openstackgerrit | YAMAMOTO Takashi proposed openstack-infra/project-config master: networking-odl: Remove cover job from post https://review.openstack.org/509138 | 10:40 |
| openstackgerrit | YAMAMOTO Takashi proposed openstack-infra/project-config master: networking-odl: Add required-projects for tox jobs https://review.openstack.org/509139 | 10:40 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Set BUILD_TIMEOUT for ironic grenade jobs https://review.openstack.org/508882 | 10:40 |
| *** lukebrowning has joined #openstack-infra | 10:43 | |
| openstackgerrit | YAMAMOTO Takashi proposed openstack-infra/project-config master: neutron-fwaas: Add requirements for tox jobs https://review.openstack.org/509137 | 10:43 |
| openstackgerrit | YAMAMOTO Takashi proposed openstack-infra/project-config master: networking-odl: Remove cover job from post https://review.openstack.org/509138 | 10:43 |
| openstackgerrit | YAMAMOTO Takashi proposed openstack-infra/project-config master: networking-odl: Add required-projects for tox jobs https://review.openstack.org/509139 | 10:43 |
| *** sshnaidm|lnch has quit IRC | 10:43 | |
| *** verdurin has joined #openstack-infra | 10:45 | |
| openstackgerrit | YAMAMOTO Takashi proposed openstack-infra/project-config master: neutron-lbaas: Add neutron requirments for tox jobs https://review.openstack.org/509143 | 10:45 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Add openstack-dev/devstack to all dsvm legacy sahara jobs https://review.openstack.org/508934 | 10:47 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Add neutron to dragonflow requiered-project https://review.openstack.org/508856 | 10:47 |
| *** lukebrowning has quit IRC | 10:47 | |
| *** lukebrowning has joined #openstack-infra | 10:49 | |
| *** cshastri has quit IRC | 10:50 | |
| *** lukebrowning has quit IRC | 10:53 | |
| *** lnxnut has joined #openstack-infra | 10:53 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: finish releasenotes job conversion https://review.openstack.org/509145 | 10:54 |
| *** lukebrowning has joined #openstack-infra | 10:55 | |
| *** wolverineav has quit IRC | 10:56 | |
| *** alexchadin has joined #openstack-infra | 10:57 | |
| *** efoley has quit IRC | 10:57 | |
| yamamoto | is this known issue? http://logs.openstack.org/13/507313/24/check/legacy-requirements/3b9fb53/job-output.txt.gz#_2017-10-03_10_29_27_754570 | 10:58 |
| *** jascott1 has joined #openstack-infra | 10:58 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Add required-projects for Cliff legacy tox jobs https://review.openstack.org/508837 | 11:00 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Add missing repo to legacy-openstackci-beaker jobs https://review.openstack.org/508857 | 11:00 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Fix typo in descr of build-openstack-sphinx-docs https://review.openstack.org/508854 | 11:00 |
| *** lukebrowning has quit IRC | 11:00 | |
| *** isaacb has quit IRC | 11:00 | |
| *** efoley has joined #openstack-infra | 11:02 | |
| *** lukebrowning has joined #openstack-infra | 11:02 | |
| *** lnxnut has quit IRC | 11:02 | |
| *** jascott1 has quit IRC | 11:02 | |
| openstackgerrit | Lucas Alvares Gomes proposed openstack-infra/project-config master: Add Neutron scenario job to networking-ovn https://review.openstack.org/509129 | 11:03 |
| *** udesale has joined #openstack-infra | 11:03 | |
| *** jkilpatr has joined #openstack-infra | 11:03 | |
| *** lukebrowning has quit IRC | 11:06 | |
| *** spectr has quit IRC | 11:07 | |
| *** nicolasbock_ has joined #openstack-infra | 11:09 | |
| *** markvoelker has quit IRC | 11:12 | |
| *** dfflanders has quit IRC | 11:13 | |
| *** greghaynes has quit IRC | 11:13 | |
| *** lukebrowning has joined #openstack-infra | 11:14 | |
| *** priteau has joined #openstack-infra | 11:14 | |
| *** lucasagomes is now known as lucas-hungry | 11:16 | |
| *** nicolasbock_ has quit IRC | 11:18 | |
| *** lukebrowning has quit IRC | 11:19 | |
| *** jesusaur has quit IRC | 11:19 | |
| *** eyalb has joined #openstack-infra | 11:19 | |
| *** eyalb has left #openstack-infra | 11:19 | |
| *** efoley has quit IRC | 11:20 | |
| *** efoley_ has joined #openstack-infra | 11:20 | |
| *** jkilpatr has quit IRC | 11:25 | |
| *** ykarel|afk is now known as ykarel | 11:26 | |
| *** sshnaidm|lnch has joined #openstack-infra | 11:26 | |
| *** greghaynes has joined #openstack-infra | 11:26 | |
| *** esberglu has joined #openstack-infra | 11:27 | |
| AJaeger | mordred: do you want to merge https://review.openstack.org/#/c/509007 - or was that just for testing? | 11:28 |
| *** sshnaidm|lnch is now known as sshnaidm | 11:29 | |
| AJaeger | yamamoto: known issue - we need to merge a couple of changes to get this fixed | 11:29 |
| AJaeger | (and release a new reno version) | 11:29 |
| yamamoto | AJaeger: thank you | 11:29 |
| AJaeger | sorry, not reno - fixes for requirements... | 11:29 |
| *** lukebrowning has joined #openstack-infra | 11:30 | |
| AJaeger | yamamoto: https://review.openstack.org/#/c/508898/ is the fix | 11:30 |
| *** nicolasbock_ has joined #openstack-infra | 11:31 | |
| *** esberglu has quit IRC | 11:31 | |
| AJaeger | mordred: I don't understand the error in https://review.openstack.org/#/c/508898/ ;( | 11:31 |
| *** bobh has joined #openstack-infra | 11:31 | |
| *** jesusaur has joined #openstack-infra | 11:33 | |
| AJaeger | mordred: ah, found it - will fix. | 11:34 |
| *** lukebrowning has quit IRC | 11:34 | |
| AJaeger | mordred: left a comment for you - so that you can address jlk's comment as well ;) | 11:35 |
| *** mrunge has quit IRC | 11:39 | |
| openstackgerrit | YAMAMOTO Takashi proposed openstack-infra/openstack-zuul-jobs master: legacy-cross-networking-midonet-python35: Add missing requirements https://review.openstack.org/509153 | 11:39 |
| *** jascott1 has joined #openstack-infra | 11:39 | |
| *** mrunge has joined #openstack-infra | 11:40 | |
| *** jkilpatr has joined #openstack-infra | 11:41 | |
| *** ldnunes has joined #openstack-infra | 11:42 | |
| *** lukebrowning has joined #openstack-infra | 11:42 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Add translation jobs https://review.openstack.org/502208 | 11:42 |
| *** jascott1 has quit IRC | 11:44 | |
| openstackgerrit | YAMAMOTO Takashi proposed openstack-infra/project-config master: neutron-fwaas: Add requirements for tox jobs https://review.openstack.org/509137 | 11:44 |
| openstackgerrit | YAMAMOTO Takashi proposed openstack-infra/project-config master: networking-odl: Remove cover job from post https://review.openstack.org/509138 | 11:44 |
| openstackgerrit | YAMAMOTO Takashi proposed openstack-infra/project-config master: networking-odl: Add required-projects for tox jobs https://review.openstack.org/509139 | 11:44 |
| openstackgerrit | YAMAMOTO Takashi proposed openstack-infra/project-config master: neutron-lbaas: Add neutron requirments for tox jobs https://review.openstack.org/509143 | 11:44 |
| *** eyalb has joined #openstack-infra | 11:45 | |
| *** jkilpatr_ has joined #openstack-infra | 11:46 | |
| *** lukebrowning has quit IRC | 11:47 | |
| *** jkilpatr has quit IRC | 11:47 | |
| *** admin0 has left #openstack-infra | 11:48 | |
| *** lukebrowning has joined #openstack-infra | 11:48 | |
| *** edmondsw has joined #openstack-infra | 11:50 | |
| *** ethfci has quit IRC | 11:50 | |
| *** ethfci has joined #openstack-infra | 11:52 | |
| *** lukebrowning has quit IRC | 11:53 | |
| jokke_ | Are you guys still busy with zuulv3 issues or anyone here who would have time to help me understand why specs.openstack.org is not populating correctly? | 11:54 |
| *** jkilpatr_ has quit IRC | 11:54 | |
| *** lukebrowning has joined #openstack-infra | 11:54 | |
| *** efoley_ has quit IRC | 11:56 | |
| *** udesale has quit IRC | 11:57 | |
| AJaeger | jokke_: still busy - that's a fallout | 11:57 |
| *** thorst has joined #openstack-infra | 11:58 | |
| AJaeger | jokke_: do you have a log file? I'll add to our tracking board... | 11:58 |
| AJaeger | (I thought it was on there but couldn't find...) | 11:58 |
| *** lnxnut has joined #openstack-infra | 11:59 | |
| *** lukebrowning has quit IRC | 11:59 | |
| *** apetrich has joined #openstack-infra | 12:00 | |
| *** thorst_ has joined #openstack-infra | 12:00 | |
| *** deduped has joined #openstack-infra | 12:02 | |
| *** nunchuck has joined #openstack-infra | 12:02 | |
| *** thorst has quit IRC | 12:02 | |
| *** garyk1 has quit IRC | 12:03 | |
| *** isaacb has joined #openstack-infra | 12:04 | |
| *** jascott1 has joined #openstack-infra | 12:04 | |
| openstackgerrit | Eyal Leshem proposed openstack-infra/project-config master: Add neutron to required-projects for dragonflow https://review.openstack.org/508785 | 12:05 |
| *** lukebrowning has joined #openstack-infra | 12:05 | |
| *** eumel8 has quit IRC | 12:05 | |
| *** namnh_ has quit IRC | 12:06 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Fail if Zuul v2 frozen files are touched https://review.openstack.org/509158 | 12:06 |
| AJaeger | team, would like advise on the change above - I've seen too many changes to the frozen files and like to handle it programmatically. This is my first playbook, so, please advise ^ | 12:06 |
| *** pblaho1 has joined #openstack-infra | 12:07 | |
| *** jascott1 has quit IRC | 12:08 | |
| *** jkilpatr_ has joined #openstack-infra | 12:08 | |
| *** efoley has joined #openstack-infra | 12:08 | |
| *** pblaho has quit IRC | 12:09 | |
| *** markvoelker has joined #openstack-infra | 12:09 | |
| *** lnxnut has quit IRC | 12:09 | |
| *** thorst_ has quit IRC | 12:10 | |
| *** tmorin has joined #openstack-infra | 12:10 | |
| *** jaypipes has joined #openstack-infra | 12:10 | |
| *** lukebrowning has quit IRC | 12:10 | |
| tmorin | hi AJaeger, I've a question about your last comment on https://review.openstack.org/#/c/508825 ... are you around? | 12:10 |
| *** gridinv has joined #openstack-infra | 12:11 | |
| AJaeger | tmorin: yes | 12:11 |
| *** lukebrowning has joined #openstack-infra | 12:11 | |
| *** bobh has quit IRC | 12:12 | |
| AJaeger | tmorin: the post jobs (publishing tarballs and docs) and the release jobs use tox - and thus need the same change | 12:12 |
| *** markvoelker has quit IRC | 12:12 | |
| tmorin | AJaeger: my question is: how can I find the exhaustive list of post jobs ? ... | 12:12 |
| *** markvoelker has joined #openstack-infra | 12:12 | |
| tmorin | AJaeger: ... and how can I know which of these needs the required-projects change (which results from the use of zuul-cloner in our tools/tox_install.sh) | 12:12 |
| AJaeger | tmorin: check the job templates in openstack-infra/openstack-zuul-jobs | 12:12 |
| tmorin | AJaeger: I'm guessing perhaps the non-tox job wouldn't need the change | 12:13 |
| AJaeger | tmorin: exactly, non-tox won't need it. But I expect that all your post and release jobs are tox based ;) | 12:13 |
| tmorin | AJaeger: I don't know for instance if the tarball job relies on tox | 12:13 |
| *** isaacb has quit IRC | 12:14 | |
| *** pblaho1 is now known as pblaho | 12:16 | |
| *** lukebrowning has quit IRC | 12:16 | |
| *** pblaho has quit IRC | 12:16 | |
| *** tikitavi has quit IRC | 12:16 | |
| *** pblaho has joined #openstack-infra | 12:17 | |
| tmorin | AJaeger: I see that in https://review.openstack.org/#/c/508822/4/zuul.d/project-templates.yaml there is only one post job (tarball publishing) | 12:17 |
| *** lucas-hungry is now known as lucasagomes | 12:17 | |
| *** dizquierdo has quit IRC | 12:17 | |
| *** lukebrowning has joined #openstack-infra | 12:18 | |
| AJaeger | tmorin: tarball jobs uses tox -e venv | 12:19 |
| openstackgerrit | Pratik Shah proposed openstack-infra/openstack-zuul-jobs master: Modified "required_projects" for openstack/omni project https://review.openstack.org/509159 | 12:19 |
| *** jpena is now known as jpena|lunch | 12:19 | |
| *** thorst has joined #openstack-infra | 12:20 | |
| AJaeger | tmorin: publish-openstack-sphinx-doc and release-notes-jobs also have post jobs with tox usage | 12:20 |
| jokke_ | AJaeger: no, just looking the specs.openstack.org page and there are few items missing that had been merged to the repo | 12:20 |
| *** rkukura has quit IRC | 12:21 | |
| *** wolverineav has joined #openstack-infra | 12:21 | |
| jokke_ | AJaeger: so I see in the docs build gate job that page been populated correctly but not on the actual production site. | 12:22 |
| openstackgerrit | garyk proposed openstack-infra/project-config master: Add missing projects to vmware-nsxlib https://review.openstack.org/508779 | 12:22 |
| *** udesale has joined #openstack-infra | 12:22 | |
| AJaeger | jokke_: which repo? | 12:22 |
| jokke_ | AJaeger: glance-specs | 12:22 |
| *** lukebrowning has quit IRC | 12:23 | |
| tmorin | AJaeger: ok, I'll add required-projects to publish-openstack-python-branch-tarball, publish-openstack-sphinx-doc and release-notes-jobs | 12:23 |
| *** dprince has joined #openstack-infra | 12:23 | |
| openstackgerrit | garyk proposed openstack-infra/project-config master: Add missing projects to vmware-nsxlib https://review.openstack.org/508779 | 12:23 |
| AJaeger | tmorin: yes, please | 12:23 |
| jokke_ | AJaeger: http://logs.openstack.org/03/508003/1/gate/build-openstack-sphinx-docs/f327975/html/latest/specs/queens/approved/index.html vs. http://specs.openstack.org/openstack/glance-specs/specs/queens/approved/index.html | 12:23 |
| *** thorst has quit IRC | 12:24 | |
| *** lukebrowning has joined #openstack-infra | 12:24 | |
| AJaeger | jokke_: that merged on the 29th and did not produce any log files. I need something newer... | 12:24 |
| *** spectr has joined #openstack-infra | 12:24 | |
| *** tpsilva has joined #openstack-infra | 12:25 | |
| *** eyalb has left #openstack-infra | 12:25 | |
| jokke_ | AJaeger: 29th is the last changes to that repo. Should I do a test merge into there so we can see what happens? | 12:25 |
| *** jamesdenton has joined #openstack-infra | 12:26 | |
| AJaeger | jokke_: yes, that would be nice - fix something trivial and tell us afterwards, we can then check logs. | 12:26 |
| *** baoli has joined #openstack-infra | 12:27 | |
| *** lukebrowning has quit IRC | 12:28 | |
| *** dave-mccowan has joined #openstack-infra | 12:29 | |
| *** jascott1 has joined #openstack-infra | 12:29 | |
| openstackgerrit | Thomas Morin proposed openstack-infra/project-config master: Add projects to required-projects for networking-(bagpipe|bgpvpn) https://review.openstack.org/508825 | 12:29 |
| *** lukebrowning has joined #openstack-infra | 12:30 | |
| *** isaacb has joined #openstack-infra | 12:32 | |
| *** erlon has joined #openstack-infra | 12:32 | |
| *** askb has quit IRC | 12:32 | |
| *** jascott1 has quit IRC | 12:33 | |
| sshnaidm | pabelanger, hi, can we fix rh1 pypi mirror today? do you have some time for this? or maybe anybody from infra cores can? | 12:33 |
| jokke_ | AJaeger: https://review.openstack.org/#/c/509161/1 I did ninja this in ;) | 12:33 |
| *** askb has joined #openstack-infra | 12:33 | |
| *** isaacb has quit IRC | 12:34 | |
| AJaeger | jokke_: I thought you would fix a typo or whitespace... | 12:34 |
| AJaeger | jokke_: I'll monitor... | 12:34 |
| odyssey4me | AJaeger jokke_ we don't seem to see any publishing on our specs either (OSA) | 12:34 |
| odyssey4me | nothing has published since newton it would seem: http://specs.openstack.org/openstack/openstack-ansible-specs/ | 12:34 |
| *** lukebrowning has quit IRC | 12:35 | |
| jokke_ | odyssey4me: we have at least up to Pike ok | 12:35 |
| *** pblaho has quit IRC | 12:36 | |
| AJaeger | odyssey4me: let's wait until glance-specs is merged and we have logs - and then you fix it everywhere | 12:36 |
| odyssey4me | FYI we have two specs merging recently (after zuul migration): https://review.openstack.org/499886 & https://review.openstack.org/499882 | 12:36 |
| *** lukebrowning has joined #openstack-infra | 12:36 | |
| *** askb has quit IRC | 12:36 | |
| odyssey4me | AJaeger sure, I've been waiting for the general chaos to die down a little before tackling this. | 12:37 |
| AJaeger | odyssey4me: I need something newer. For those we don't have logs | 12:37 |
| *** rhallisey has joined #openstack-infra | 12:39 | |
| *** rhallisey has quit IRC | 12:39 | |
| *** rhallisey has joined #openstack-infra | 12:40 | |
| *** rlandy has joined #openstack-infra | 12:41 | |
| *** lukebrowning has quit IRC | 12:41 | |
| *** baoli has quit IRC | 12:41 | |
| *** baoli has joined #openstack-infra | 12:42 | |
| *** hemna__ has joined #openstack-infra | 12:42 | |
| *** thorst has joined #openstack-infra | 12:44 | |
| *** kgiusti has joined #openstack-infra | 12:45 | |
| *** alexchadin has quit IRC | 12:46 | |
| *** jcoufal has joined #openstack-infra | 12:46 | |
| *** mat128 has joined #openstack-infra | 12:46 | |
| AJaeger | now we have 508996 on top of gate - and waiting for jobs to run/start for ages | 12:46 |
| *** lukebrowning has joined #openstack-infra | 12:48 | |
| *** pblaho has joined #openstack-infra | 12:48 | |
| AJaeger | jokke_: you can abandon your glance-specs change, I found a cinder-specs log file. | 12:49 |
| *** baoli has quit IRC | 12:49 | |
| *** panda|bbl is now known as panda | 12:49 | |
| *** esberglu has joined #openstack-infra | 12:51 | |
| *** lukebrowning has quit IRC | 12:52 | |
| *** udesale has quit IRC | 12:52 | |
| AJaeger | mordred, it fails in fetch-tox-output, see http://logs.openstack.org/78/7887c6aae2d0faf2bf574897a2f458ba45c33891/post/publish-openstack-specs/ . This was yesterday. Any ideas? ^ | 12:53 |
| *** trown|outtypewww is now known as trown | 12:54 | |
| *** lukebrowning has joined #openstack-infra | 12:54 | |
| *** alexchadin has joined #openstack-infra | 12:55 | |
| leyal | AJaeger , i uploaded new patch based on your comments (https://review.openstack.org/#/c/508785/19) , do you have time to review it? | 12:55 |
| AJaeger | leyal: will do later... | 12:56 |
| *** isaacb has joined #openstack-infra | 12:56 | |
| leyal | AJaeger, thanks | 12:57 |
| *** lukebrowning has quit IRC | 12:58 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Add missing projects for openstackci beaker jobs https://review.openstack.org/508981 | 12:58 |
| openstackgerrit | Lucas Alvares Gomes proposed openstack-infra/openstack-zuul-jobs master: Add release-notes-jobs-neutron https://review.openstack.org/509165 | 12:59 |
| mnaser | AJaeger i think the cinder-specs job is failing because the fetch-tox-output assumes that all env have ran | 12:59 |
| AJaeger | mnaser: any ideas on a fix? | 13:00 |
| *** lukebrowning has joined #openstack-infra | 13:00 | |
| mnaser | AJaeger at first i thought maybe we could drop those extra envs but it looks like cinder-specs actually has unit tests | 13:00 |
| *** Goneri has joined #openstack-infra | 13:01 | |
| AJaeger | mnaser: I think currently all specs publish jobs fail, we can take an easier example and ask them to merge something ;) | 13:01 |
| openstackgerrit | Eyal Leshem proposed openstack-infra/project-config master: Add neutron to required-projects for dragonflow https://review.openstack.org/508785 | 13:03 |
| *** isaacb has quit IRC | 13:04 | |
| *** lukebrowning has quit IRC | 13:04 | |
| openstackgerrit | Thomas Morin proposed openstack-infra/openstack-zuul-jobs master: Add required projects to a legacy n8g-bgpvpn job https://review.openstack.org/509167 | 13:06 |
| *** jascott1 has joined #openstack-infra | 13:06 | |
| openstackgerrit | Thomas Morin proposed openstack-infra/project-config master: Add projects to required-projects for networking-(bagpipe|bgpvpn) https://review.openstack.org/508825 | 13:06 |
| *** lukebrowning has joined #openstack-infra | 13:06 | |
| *** mriedem has joined #openstack-infra | 13:07 | |
| *** lnxnut has joined #openstack-infra | 13:07 | |
| *** alexchadin has quit IRC | 13:09 | |
| AJaeger | sshnaidm: I think I found the problem, commented on your change | 13:10 |
| *** jascott1 has quit IRC | 13:10 | |
| *** chlong has joined #openstack-infra | 13:11 | |
| *** chlong_ has joined #openstack-infra | 13:11 | |
| *** lukebrowning has quit IRC | 13:11 | |
| *** lukebrowning has joined #openstack-infra | 13:13 | |
| *** martinkopec has joined #openstack-infra | 13:13 | |
| *** alexchadin has joined #openstack-infra | 13:13 | |
| sshnaidm | AJaeger, thanks, will fix | 13:14 |
| AJaeger | sshnaidm: but first double check the base template whether it's correct to inherit from there... | 13:15 |
| *** yamamoto has quit IRC | 13:16 | |
| *** lnxnut has quit IRC | 13:17 | |
| sshnaidm | AJaeger, but I'm gonna change to parent: legacy-puppet-openstack-integration , isn't it? | 13:17 |
| *** lukebrowning has quit IRC | 13:17 | |
| ykarel | AJaeger, i tried cinder-specs locally, the command that runs in job: tox: tox -evenv -vv python setup.py build_sphinx and it creates .tox/venv but fetch-tox-output is trying to look at .tox/docs, .tox/py27, .tox/py35, .tox/pep8, should the commnad be: tox -edocs instead of tox -evenv -- <args> | 13:18 |
| *** eharney has joined #openstack-infra | 13:18 | |
| AJaeger | sshnaidm: yes - and I think that is safe (looking at the definition) | 13:18 |
| *** leyale has joined #openstack-infra | 13:18 | |
| AJaeger | ykarel: tox -e venv is correct. But let's ask mordred whether something is wrong there ^ | 13:19 |
| openstackgerrit | Andrey Kurilin proposed openstack-infra/openstack-zuul-jobs master: Add diskimage-builder to legacy-rally-dsvm-verify jobs https://review.openstack.org/509171 | 13:19 |
| ykarel | AJaeger, Ok | 13:19 |
| fungi | AJaeger: i thought mordred had pushed up some changes to fix publishing to sites on static.o.o (including specs.o.o) in the last 12 hours or so. did those merge already? did they not actually address problems with specs publishing? | 13:20 |
| fungi | thought that was discovered as part of figuring out how to fix publishing from the elections repo to governance.o.o | 13:21 |
| mordred | fungi: yah- I got static ones done - I think specs are a different beast | 13:21 |
| mordred | I'll take specs jobs as today's first task :) | 13:21 |
| fungi | specs is hosted on static | 13:21 |
| *** bnemec has joined #openstack-infra | 13:21 | |
| mordred | yah - the others were jobs that matched 'static-{name}-publish' | 13:22 |
| fungi | oh, got it, and specs publishing jobs were noncomforming (at least in name) | 13:22 |
| *** baoli has joined #openstack-infra | 13:22 | |
| mordred | yah - should be easy to make new specs ones - they just need a different post playbook | 13:23 |
| AJaeger | mordred: great! | 13:23 |
| AJaeger | mordred: we still have a few of your changes in review/merge - but that looks like cleanup. | 13:24 |
| mordred | oh wait - we already havea publish-openstack-specs | 13:24 |
| AJaeger | fungi, mordred : Right now Zuul is very slow ;( | 13:24 |
| *** jpena|lunch is now known as jpena | 13:24 | |
| openstackgerrit | John Trowbridge proposed openstack-infra/tripleo-ci master: WIP: Fix TripleO CI jobs https://review.openstack.org/508660 | 13:26 |
| *** apetrich has quit IRC | 13:26 | |
| *** tmorin has quit IRC | 13:27 | |
| mnaser | i'm seeing really weird failures where ansible seems to have stopped responding and the job times out (and the job times out contacting the node in trusted post as well) | 13:27 |
| AJaeger | mordred: this is all complicated with required-projects, see https://review.openstack.org/508825 | 13:27 |
| *** leyale has quit IRC | 13:27 | |
| mnaser | was there any zuul executor network issues by any chance? http://logs.openstack.org/05/509105/1/check/puppet-openstack-integration-4-scenario002-tempest-centos-7/ae4ddd3/job-output.txt.gz | 13:27 |
| openstackgerrit | John Trowbridge proposed openstack-infra/tripleo-ci master: WIP: Fix TripleO CI jobs https://review.openstack.org/508660 | 13:28 |
| *** apetrich has joined #openstack-infra | 13:28 | |
| *** coolsvap has quit IRC | 13:29 | |
| *** jascott1 has joined #openstack-infra | 13:30 | |
| *** lukebrowning has joined #openstack-infra | 13:31 | |
| *** dizquierdo has joined #openstack-infra | 13:32 | |
| fungi | AJaeger: it ran out of regular memory around 2 hours ago according to cacti | 13:32 |
| openstackgerrit | Eyal Leshem proposed openstack-infra/project-config master: Add neutron to required-projects for dragonflow https://review.openstack.org/508785 | 13:32 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/openstack-zuul-jobs master: Fix puppet jobs by adding integration parent https://review.openstack.org/509173 | 13:33 |
| *** ykarel has quit IRC | 13:34 | |
| AJaeger | fungi: I should have checked ;( Yeah, that would explain it... | 13:34 |
| *** yamamoto has joined #openstack-infra | 13:35 | |
| *** jascott1 has quit IRC | 13:35 | |
| AJaeger | fungi, do you want to restart - or wait for jeblair first? | 13:35 |
| openstackgerrit | Lucas Alvares Gomes proposed openstack-infra/project-config master: Use release-notes-jobs-neutron for networking-ovn https://review.openstack.org/509176 | 13:35 |
| fungi | though it seems not to be using much swap yet (<0.7gib) | 13:35 |
| fungi | AJaeger: i'd rather wait for more of the team to be awake so we can discuss whether there are other patches we want to add when we restart | 13:36 |
| *** lukebrowning has quit IRC | 13:36 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Set default on fetch-tox-output to venv https://review.openstack.org/509177 | 13:36 |
| mordred | AJaeger, fungi: ^^ that should fix the specs jobs | 13:36 |
| AJaeger | fungi, works for me... | 13:37 |
| openstackgerrit | Cedric Savignan proposed openstack-infra/project-config master: networking-bgpvpn: add translation jobs https://review.openstack.org/509178 | 13:37 |
| *** lukebrowning has joined #openstack-infra | 13:37 | |
| AJaeger | mordred: roles/fetch-tox-output/README.rst needs updating for tht change | 13:38 |
| dmsimard | mnaser: I'll set up an elastic recheck query for unreachable tasks, it should never happen and I've seen it a few times now. | 13:38 |
| mordred | AJaeger: thanks - one sec | 13:39 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Set default on fetch-tox-output to venv https://review.openstack.org/509177 | 13:39 |
| *** felipemonteiro has joined #openstack-infra | 13:40 | |
| AJaeger | mordred: one more question, looking at roles/fetch-tox-output/tasks/main.yaml I see several "when: tox_envlist is not defined or tox_envlist == 'all'" - is that logic still correct? | 13:40 |
| AJaeger | (since you change the default away from 'all') | 13:40 |
| *** ijw has joined #openstack-infra | 13:40 | |
| pabelanger | morning | 13:41 |
| AJaeger | good morning, pabelanger | 13:41 |
| *** lbragstad has joined #openstack-infra | 13:41 | |
| pabelanger | anything specific I should be looking at? Was about to check etherpad | 13:41 |
| *** lukebrowning has quit IRC | 13:42 | |
| fungi | pabelanger: that etherpad is a great first place to look. also mordred just pushed up a fix for specs publishing moments ago (509177) | 13:42 |
| AJaeger | pabelanger: see my questions 2 mins ago to mordred - not sure about side effects on that one | 13:43 |
| *** alexchadin has quit IRC | 13:43 | |
| mordred | AJaeger: yah - those logics are still good (although the tox_envlist is not defined part is a little bit of a noop with a default - but the logic is good for all) | 13:43 |
| *** rhallisey has quit IRC | 13:43 | |
| *** alexchadin has joined #openstack-infra | 13:43 | |
| *** rhallisey has joined #openstack-infra | 13:43 | |
| pabelanger | mordred: AJaeger fungi: looks like system-config is still broken. Going to see why | 13:46 |
| AJaeger | fungi, can we promote 508344, please? the top of stack is not running anything currently | 13:46 |
| AJaeger | pabelanger: looking at line 74 of the etherpad, I wonder whether we need to add far more repos to it. | 13:47 |
| fungi | AJaeger: sure. looks like it'll need to be enqueued into the gate first but i can do both | 13:47 |
| AJaeger | pabelanger: http://logs.openstack.org/62/508962/1/check/legacy-openstackci-beaker-ubuntu-trusty/c3a8def/job-output.txt.gz#_2017-10-03_03_46_44_956045 | 13:47 |
| AJaeger | fungi: it's in the gate | 13:47 |
| pabelanger | AJaeger: we do, I've going to make them non-voting first, then we can iterate on them | 13:47 |
| pabelanger | AJaeger: then revert | 13:48 |
| *** kiennt26 has joined #openstack-infra | 13:48 | |
| AJaeger | pabelanger: sounds like a plan | 13:48 |
| dmsimard | pabelanger: what jobs ? | 13:48 |
| AJaeger | pabelanger: that's a long list I referenced - wonder whether we need them all? | 13:48 |
| fungi | AJaeger: possible the status page is broken with my browser or something, but seems to be in check | 13:48 |
| dmsimard | pabelanger: ugh, that again | 13:48 |
| fungi | oh, nevermind that was as a dependency | 13:48 |
| AJaeger | fungi, I see it in gate - integrated queue. 6th item | 13:49 |
| fungi | i've initiated the promote call, but given the state of the events queue it may take a bit of time to take effect | 13:50 |
| AJaeger | mordred: will your fetch-tox change also fix other post jobs? We have not published docs for some days... | 13:50 |
| AJaeger | fungi: yeah ;( | 13:50 |
| AJaeger | thanks, fungi | 13:50 |
| fungi | np | 13:51 |
| openstackgerrit | Paul Belanger proposed openstack-infra/openstack-zuul-jobs master: Make openstackci-beaker-ubuntu-trusty and logstash-filters non-voting https://review.openstack.org/509182 | 13:52 |
| *** ihrachys has joined #openstack-infra | 13:52 | |
| AJaeger | pabelanger, mordred, I could use a review on https://review.openstack.org/509158, please - to make reviewing easier ;) | 13:52 |
| *** jcoufal_ has joined #openstack-infra | 13:52 | |
| AJaeger | https://review.openstack.org/#/c/509178/ is just another change that 509158 should catch | 13:53 |
| evrardjp | should I expect to see an empty page on http://zuulv3.openstack.org/ ? | 13:53 |
| pabelanger | AJaeger: neat job :) | 13:54 |
| pabelanger | evrardjp: no, looking | 13:54 |
| *** baoli_ has joined #openstack-infra | 13:54 | |
| AJaeger | evrardjp: no - but it might be slow | 13:54 |
| pabelanger | zuulv3.o.o is swapping | 13:55 |
| pabelanger | so, things are slower | 13:55 |
| evrardjp | ok | 13:55 |
| *** martinkopec has quit IRC | 13:55 | |
| evrardjp | that might be it | 13:55 |
| fungi | the page is populated by javascript making a call to fetch json from zuul's status api, so if the status api takes a while to respond you'll see no job info until it does | 13:55 |
| *** jcoufal has quit IRC | 13:55 | |
| evrardjp | sorry for having diverted you from your work, I was just worried | 13:55 |
| *** baoli has quit IRC | 13:56 | |
| andreaf | pabelanger: is it possible to use the zuul API to make a more targeted query? i.e. for a specific patch id? | 13:56 |
| AJaeger | pabelanger: thanks. Do you think it will work? This is my first playbook ;) | 13:56 |
| *** verdurin has quit IRC | 13:57 | |
| pabelanger | AJaeger: ya, it looks correct | 13:57 |
| pabelanger | andreaf: i don't believe so | 13:57 |
| fungi | pabelanger: andreaf: there used to be | 13:57 |
| pabelanger | zuul isn't processing anything ATM, deep into something with swap | 13:58 |
| *** sshnaidm is now known as sshnaidm|mtg | 13:58 | |
| *** spectr has quit IRC | 13:58 | |
| fungi | in an effort to support embedding job progress into the gerrit webui we extended the zuul status api to filter by ref... i don't have time to look up the details now though | 13:59 |
| *** spectr has joined #openstack-infra | 13:59 | |
| fungi | something like passing a get variable of change with the change number as the value | 13:59 |
| fungi | maybe | 13:59 |
| andreaf | fungi: no worries - I was just trying to be nice to zuul and query less data - I can check in the code | 14:00 |
| *** lukebrowning has joined #openstack-infra | 14:01 | |
| fungi | swap utilization on zuulv3 is growing pretty rapidly. was 4gib when i checked a few minutes ago, now up to ~7.5gib used | 14:02 |
| openstackgerrit | Monty Taylor proposed openstack-infra/openstack-zuul-jobs master: Replace legacy-requirements with new requirements-check https://review.openstack.org/508898 | 14:02 |
| *** verdurin has joined #openstack-infra | 14:02 | |
| *** efoley has quit IRC | 14:03 | |
| *** efoley_ has joined #openstack-infra | 14:03 | |
| *** leyale has joined #openstack-infra | 14:04 | |
| *** lukebrowning has quit IRC | 14:05 | |
| mordred | AJaeger: https://review.openstack.org/#/c/509158 looks good generally - one issue that I've -1'd | 14:05 |
| AJaeger | thanks, mordred. will fix | 14:06 |
| andreaf | fungi: /status/change/X,Y http://git.openstack.org/cgit/openstack-infra/zuul/tree/zuul/webapp.py?h=feature/zuulv3#n38 | 14:06 |
| *** rkukura has joined #openstack-infra | 14:06 | |
| fungi | andreaf: that looks right, thanks for the details | 14:07 |
| *** ykarel has joined #openstack-infra | 14:07 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Fail if Zuul v2 frozen files are touched https://review.openstack.org/509158 | 14:07 |
| AJaeger | pabelanger, mordred, updated ^ | 14:07 |
| *** nikhil_ has joined #openstack-infra | 14:13 | |
| *** lukebrowning has joined #openstack-infra | 14:13 | |
| *** nikhil_ is now known as Guest61849 | 14:13 | |
| *** Guest61849 is now known as nikhil_k | 14:14 | |
| * AJaeger grabs some waffels - back in a bit... | 14:14 | |
| *** lnxnut has joined #openstack-infra | 14:14 | |
| *** jascott1 has joined #openstack-infra | 14:15 | |
| *** thorst has quit IRC | 14:15 | |
| *** srobert has joined #openstack-infra | 14:16 | |
| *** lukebrowning has quit IRC | 14:18 | |
| *** leyale has quit IRC | 14:18 | |
| *** lukebrowning has joined #openstack-infra | 14:19 | |
| *** jascott1 has quit IRC | 14:20 | |
| *** rbrndt has joined #openstack-infra | 14:20 | |
| *** bobh has joined #openstack-infra | 14:20 | |
| * dhellmann wants waffles | 14:23 | |
| *** lnxnut has quit IRC | 14:24 | |
| *** lukebrowning has quit IRC | 14:24 | |
| *** yboaron_ has quit IRC | 14:25 | |
| *** lukebrowning has joined #openstack-infra | 14:25 | |
| pabelanger | okay, relocating to the bibliotheque. Back online shortly | 14:25 |
| *** yamamoto has quit IRC | 14:26 | |
| *** krtaylor has quit IRC | 14:29 | |
| *** sambetts|afk is now known as sambetts | 14:29 | |
| *** lukebrowning has quit IRC | 14:30 | |
| *** hongbin has joined #openstack-infra | 14:31 | |
| *** chlong has quit IRC | 14:31 | |
| *** chlong_ has quit IRC | 14:31 | |
| *** thorst has joined #openstack-infra | 14:31 | |
| *** lukebrowning has joined #openstack-infra | 14:32 | |
| bnemec | https://www.youtube.com/watch?v=a8jzvh5Wb00 | 14:32 |
| *** chlong has joined #openstack-infra | 14:32 | |
| fungi | http://www.youtube.com/watch?v=d7KbxVtfiUU | 14:34 |
| *** lukebrowning has quit IRC | 14:36 | |
| *** felipemonteiro has quit IRC | 14:37 | |
| *** lukebrowning has joined #openstack-infra | 14:38 | |
| lbragstad | hey folks - we have a change we're trying to get in the gate https://review.openstack.org/#/c/499783/ | 14:39 |
| lbragstad | we've rechecked it and reapplied the +2/+A a couple times, but I'm not seeing it show up in the queue | 14:39 |
| lbragstad | http://zuulv3.openstack.org/ | 14:40 |
| AJaeger | lbragstad: we're restarting zuul soon - let's check afterwards again.... | 14:40 |
| lbragstad | AJaeger: aha - that sounds good, thanks for confirming! | 14:40 |
| fungi | lbragstad: zuul has been at a crawl processing events for the past couple hours. we're discussing a restart in the incident channel currently | 14:40 |
| *** xarses has joined #openstack-infra | 14:41 | |
| openstackgerrit | Rico Lin proposed openstack-infra/project-config master: Remove legacy jobs from heat https://review.openstack.org/509194 | 14:41 |
| lbragstad | perfect, i'll stay out of your hair and look for updates :) | 14:41 |
| *** lukebrowning has quit IRC | 14:43 | |
| *** priteau has quit IRC | 14:44 | |
| *** rpittau_ is now known as rpittau | 14:44 | |
| *** lukebrowning has joined #openstack-infra | 14:44 | |
| *** rpittau has quit IRC | 14:45 | |
| AJaeger | lbragstad: looking at your job: I'll recheck after the restart... | 14:46 |
| lbragstad | AJaeger: thanks again, i appreciate it | 14:47 |
| *** esberglu has quit IRC | 14:49 | |
| *** lukebrowning has quit IRC | 14:49 | |
| *** esberglu has joined #openstack-infra | 14:49 | |
| *** alexchadin has quit IRC | 14:50 | |
| *** jascott1 has joined #openstack-infra | 14:52 | |
| *** ijw has quit IRC | 14:53 | |
| *** esberglu has quit IRC | 14:53 | |
| *** sshnaidm|mtg is now known as sshnaidm | 14:54 | |
| *** eharney has quit IRC | 14:56 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Prevent non-check changes, add infra pipelines https://review.openstack.org/509196 | 14:56 |
| *** jascott1 has quit IRC | 14:57 | |
| *** lukebrowning has joined #openstack-infra | 14:57 | |
| jlvillal | mordred, clarkb: FYI: I tried to test the ironic patch, but both times the grenade job got a POST_FAILURE error: https://review.openstack.org/#/c/509010/ | 14:58 |
| mnaser | hi folks... with zuul v3, how do we handle stable branches... do we keep a zuul.yaml for each repo or how does that work? | 15:01 |
| mnaser | sorry, not each repo, each branch i meant in that sentance | 15:01 |
| *** efoley_ has quit IRC | 15:01 | |
| *** lukebrowning has quit IRC | 15:02 | |
| fungi | mnaser: you can specify branches in your jobs in master, _or_ you can specify jobs in each branch of your project. the idea being that when you branch from master zuul will find your stable branch jobs on your stable branches themselves | 15:02 |
| dhellmann | mnaser : https://docs.openstack.org/infra/manual/zuulv3.html has some info on that | 15:02 |
| mnaser | fungi okay, so we'll have to backport the .zuul.yaml file | 15:03 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Revert "Shift nodepool quota from v2 to v3" https://review.openstack.org/509200 | 15:03 |
| fungi | mnaser: you can do either one | 15:03 |
| mnaser | okay silly question but are .zuul.yaml files matched by branch | 15:03 |
| *** lukebrowning has joined #openstack-infra | 15:03 | |
| mnaser | for example, i define a project-template in stable/pike of puppet-openstack-integration | 15:03 |
| fungi | mnaser: yes, zuul looks at all branches of the repo for zuul configuration files | 15:03 |
| fungi | and limits them to the branches it finds them on if not on master | 15:04 |
| mnaser | okay cool so puppet-nova stable/pike job will use the project-template in stable/pike from puppet-openstack-integration? | 15:05 |
| fungi | so that you can branch your job definitions when you branch your projects | 15:05 |
| *** vponomaryov has quit IRC | 15:05 | |
| *** vponomaryov has joined #openstack-infra | 15:05 | |
| mnaser | cause we have project-templates for puppet modules defined in puppet-openstack-integration repo to avoid duplicating that stuff everywhere | 15:05 |
| fungi | mnaser: yes, that's the idea for the automatic branch matchers. i think that carries across repos | 15:06 |
| mnaser | okay great | 15:06 |
| fungi | i haven't pulled up the docs for that yet, found my browser hung so trying to get it restored to order now | 15:07 |
| *** eharney has joined #openstack-infra | 15:08 | |
| fungi | "...jobs and variants defined in the master branch of a project will apply to all branches, and any further variants defined in other branches get an implied branch matcher of their current branch. This makes it so that we can branch a project from master along with all of its job definitions, and jobs will continue to work as expected." | 15:08 |
| fungi | https://docs.openstack.org/infra/manual/zuulv3.html#how-jobs-are-selected-to-run-in-zuul-v3 | 15:08 |
| *** lukebrowning has quit IRC | 15:08 | |
| *** sshnaidm is now known as sshnaidm|afk | 15:08 | |
| *** lukebrowning has joined #openstack-infra | 15:09 | |
| *** thorst has quit IRC | 15:10 | |
| *** armax has joined #openstack-infra | 15:10 | |
| *** vponomaryov has quit IRC | 15:10 | |
| *** thorst has joined #openstack-infra | 15:11 | |
| *** thorst has quit IRC | 15:11 | |
| *** vponomaryov has joined #openstack-infra | 15:11 | |
| *** rcernin has quit IRC | 15:12 | |
| *** ricky_ has joined #openstack-infra | 15:12 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Prevent non-check changes, add infra pipelines https://review.openstack.org/509196 | 15:13 |
| *** lukebrowning has quit IRC | 15:14 | |
| *** priteau has joined #openstack-infra | 15:15 | |
| mriedem | pabelanger: you know how you'd like to be able to update the keypair in an instance in nova? | 15:16 |
| mriedem | we have competing specs in nova for doing that, and could use some user input | 15:17 |
| *** jascott1 has joined #openstack-infra | 15:17 | |
| *** dave-mccowan has quit IRC | 15:17 | |
| pabelanger | mriedem: right, not an instance but we want to update the keypairs directly. Is there a review? | 15:18 |
| *** apetrich has quit IRC | 15:18 | |
| mriedem | https://review.openstack.org/#/c/375221/ vs https://review.openstack.org/#/c/506552/ | 15:18 |
| mriedem | the former is just providing a new key_name during instance rebuild, | 15:18 |
| mriedem | the latter is updating the server with a new key_name, and then you can refresh the instance in other ways, like reboot possibly, | 15:19 |
| pabelanger | mriedem: k, thanks. I'll take a look and see. | 15:19 |
| mriedem | the tricky thing is whether or not the key is refreshed in the guest is based on cloud-init config in the image | 15:19 |
| *** dave-mccowan has joined #openstack-infra | 15:19 | |
| mriedem | so if you update the keyname in the instance and reboot, it might not actually refresh the guest, so you'd have to then rebuild | 15:19 |
| mriedem | it's not super discoverable by the end user | 15:19 |
| *** esberglu has joined #openstack-infra | 15:20 | |
| sdague | mriedem: yeh, but that's true about initial keypair as well | 15:20 |
| *** pcaruana has quit IRC | 15:20 | |
| *** priteau has quit IRC | 15:20 | |
| mriedem | ? | 15:20 |
| sdague | like if you don't have cloud init (or equivalent) then you don't get those key pairs injected | 15:20 |
| mriedem | true | 15:21 |
| *** lukebrowning has joined #openstack-infra | 15:21 | |
| mordred | mriedem: yah- I think that's not actually the thing we're looking for | 15:21 |
| mriedem | don't know how many openstack public clouds don't have cloud-init in the images | 15:21 |
| pabelanger | mriedem: ya, in our use case, once we launch a server, we'd never change the keypair. We'd like to update the keypair so new servers would use it. So, it would be possible for 2 servers, using the same keypair name, but have different values. Until we deleted the first server | 15:21 |
| *** jascott1 has quit IRC | 15:21 | |
| mordred | mriedem: although I do agree that is a use case tha people have - it's just not one we have ... what pabelanger ^^ said | 15:21 |
| mriedem | ok that's not something we're going to do | 15:21 |
| mriedem | that's like edit flavor | 15:21 |
| *** lnxnut has joined #openstack-infra | 15:21 | |
| fungi | mriedem: lots of public clouds _do_ let users upload their own images, so they're not always stuck using what the provider offers in that regard | 15:22 |
| mordred | awesome. well - it's a key rotation thing and it's painful for users - but it's good to know it's off the table | 15:22 |
| aspiers | clarkb: is this the thing you need updating? http://ctf-dev-environment-vagrant.s3.amazonaws.com/Gerrit-Performance-Tuning-Cheat-Sheet.pdf | 15:22 |
| *** felipemonteiro has joined #openstack-infra | 15:22 | |
| aspiers | or https://www.slideshare.net/JohannesNicolai1/gerrit-performance-tuning-talk-at-google-summit ? | 15:22 |
| clarkb | aspiers: ya I think that is it | 15:23 |
| clarkb | the first | 15:23 |
| mriedem | mordred: so you change your keypair and then what do you, as a user, expect nova to do about all of the instances currently using that keypair? rebuild them? | 15:23 |
| aspiers | I am with the Collabnet guys now | 15:23 |
| sdague | mordred: I think the only way it would be done is if there was sequence numbers added | 15:23 |
| mordred | mriedem: nope. they can stay | 15:23 |
| *** felipemonteiro_ has joined #openstack-infra | 15:23 | |
| mriedem | mordred: then what's the point if you need to rotate the key? | 15:23 |
| sdague | because there is a ton of confusion about X == X except X is not X | 15:23 |
| pabelanger | https://github.com/ansible/ansible/pull/30565 is how we plan on doing it, mordred wrote an ansible patch. But does mean there is a window where we won't be able to launch a server because the keypair doesn't exist | 15:23 |
| sdague | mriedem: slow rotation | 15:23 |
| mordred | mriedem: basically we have a keypair that our automation is configured to use by name when booting new nodes | 15:23 |
| mordred | mriedem: for us to update its contents, we have to delete and re-create it | 15:24 |
| mordred | mriedem: during which time the automation that is creating servers will be broken since the keypair will cease to exist | 15:24 |
| sdague | I could totally see putting rotation in place with a sequence number | 15:24 |
| mordred | yah - again, what pabelanger said :) | 15:24 |
| mordred | sdague: but that wouldn't help us | 15:24 |
| aspiers | clarkb: what's the memory usage on our instance? | 15:24 |
| sdague | mordred: why not? | 15:24 |
| clarkb | aspiers heap is capped to 48GB and we hitbthat and GC periodically (like once a week) | 15:25 |
| mordred | sdague: oh - you're saying make a new keypair, then update our other thing to use the new keypair, then delete the old keypair | 15:25 |
| mordred | sdague: fair - yes, tha would work | 15:25 |
| clarkb | er hit that and have noticeable GCing which degrades service | 15:25 |
| pabelanger | when I rotate my key for openstack-infra, for a period of time, i have old / new keys. Until all our old keypair servers are deleted, then everything will be new keypair and I can finally delete my local old private key | 15:25 |
| mordred | sdague: sorry - I thought you were sugesting nova add a sequence number | 15:25 |
| *** lukebrowning has quit IRC | 15:25 | |
| sdague | mordred: I'm saying if keypair got a monotonically increasing sequence number, so for build you could have the same names, the newest one is always used by default | 15:25 |
| mordred | pabelanger: yah - I think that may just have to be what we do for the other thing too | 15:25 |
| mordred | sdague: AH - yes - that would work GREAT | 15:26 |
| sdague | but, it's kind of moot, because you don't use the keypair api today :) | 15:26 |
| mordred | we don't? | 15:26 |
| aspiers | clarkb: how many daily requests do we get? | 15:26 |
| mordred | we TOTALLY use the keypair api today | 15:26 |
| mordred | that's why this is an issue :) | 15:26 |
| *** yamamoto has joined #openstack-infra | 15:26 | |
| clarkb | aspiers: that I eould have to go ask melody for | 15:26 |
| clarkb | aspiers: I can dig more in a bit | 15:27 |
| sdague | mordred: oh, gotcha, I missunderstood not using cloud init like things | 15:27 |
| mordred | yah- we use glean and use it for keypairs only | 15:27 |
| mordred | we don't use anyof the "run this arbitrary script" cloud-init things | 15:27 |
| *** felipemonteiro has quit IRC | 15:27 | |
| clarkb | aspiers: currently ona call | 15:27 |
| aspiers | clarkb: np | 15:27 |
| sdague | mordred: sure, but that's not all of cloud init | 15:28 |
| mordred | because once you get a keypair on the box, you don't need a daemon running on the box to run scriptsfor you at boot - you can just, youknow, run them normally | 15:28 |
| sdague | also, do you run it on reboot? | 15:28 |
| mordred | sdague: nope. we use instance metadata to bootstrap the servers into being ssh managable | 15:28 |
| aspiers | clarkb: google says they generally catch memory leaks pretty quickly, but maybe we could use MAT to analyse what's using the heap | 15:28 |
| sdague | mordred: sure, so glean is deleted or deactivated after first boot? | 15:28 |
| aspiers | clarkb: Gigamon suggested just following a standard JVM GC tuning guide, e.g. there's one for Jenkins which is good apparently | 15:29 |
| mordred | sdague: yah- basically (or on our non-glean enabled servers, we're disabling/deleting cloud-init) | 15:29 |
| aspiers | clarkb: rule of thumb being bandied about: heap size should be 4x largest repo | 15:29 |
| *** trown is now known as trown|lunch | 15:30 | |
| mordred | once we've gotten ssh to a server once, we consider ownership transferred and we dont' want things injecting files or otherwise modifying contents of things | 15:30 |
| sdague | sure, that's fair | 15:30 |
| mordred | sdague: cloud-init likes to do thing son reboot which break our servers - which is why we disable/delete after first boot (most notably it keeps breaking our hostnames :) ) | 15:30 |
| sdague | I will call you and expert user, and you would accept the fact that you don't get this for reboot | 15:30 |
| jlk | o/ | 15:31 |
| *** camunoz has joined #openstack-infra | 15:31 | |
| *** lnxnut has quit IRC | 15:31 | |
| *** kiennt26 has quit IRC | 15:31 | |
| aspiers | clarkb: potentially upgrading to 2.14 and Java 9 could both help | 15:31 |
| aspiers | according to maintainer from Google | 15:31 |
| *** lukebrowning has joined #openstack-infra | 15:32 | |
| sdague | mordred: ok, well, honestly, proposal for sequence ids on key pairs would be totally in bounds from my perspective | 15:32 |
| sdague | but it is a different thing | 15:32 |
| *** pt_15 has quit IRC | 15:33 | |
| fungi | sdague: seems like it would work for our case. basically we use configuration management to change access keys on running systems, but for bootstrapping initial access we use nova keypairs to communicate an initial keyset and we want to be able to change our bootstrapping credentials without unnecessary downtime | 15:34 |
| *** pblaho has quit IRC | 15:34 | |
| *** yamamoto has quit IRC | 15:34 | |
| sdague | right | 15:34 |
| mordred | sdague: ++ agree | 15:35 |
| fungi | but we're generally distrustful of having some invisible hand reaching into running servers and mucking with configuration uninvited (cloud-init) | 15:36 |
| *** chlong has quit IRC | 15:36 | |
| *** lukebrowning has quit IRC | 15:36 | |
| fungi | so our expectations around what should happen to running systems when a nova keypair gets updated is possibly not common | 15:36 |
| mordred | fungi: also - we could implement 3-step key rotation as described above today, so maybe requesting sequence numbers isn't worht it | 15:36 |
| pabelanger | ya, just means more churn on cloud-launcher and nodepool.yaml, but we can do it | 15:38 |
| *** lukebrowning has joined #openstack-infra | 15:38 | |
| dmsimard | Is it possible to track a bug/query in elastic-recheck without having elastic-recheck comment on the bug ? | 15:39 |
| dmsimard | It seems possible but I don't see how | 15:39 |
| *** kjackal_ has quit IRC | 15:40 | |
| *** itzikb has quit IRC | 15:41 | |
| *** jascott1 has joined #openstack-infra | 15:41 | |
| *** priteau has joined #openstack-infra | 15:42 | |
| *** lukebrowning has quit IRC | 15:42 | |
| smcginnis | So... what's up with zuul? :] | 15:43 |
| *** priteau has quit IRC | 15:43 | |
| *** dave-mcc_ has joined #openstack-infra | 15:44 | |
| fungi | it's reached the point where it's consuming roughly 2x its ram (so ~15gib ram and ~15gib swap) | 15:44 |
| *** psachin has quit IRC | 15:45 | |
| *** martinkopec has joined #openstack-infra | 15:45 | |
| *** rpittau has joined #openstack-infra | 15:46 | |
| *** jascott1 has quit IRC | 15:46 | |
| *** dave-mccowan has quit IRC | 15:46 | |
| smcginnis | There was some talk on the ML of rolling back. Are we looking at that yet? Or is there a chance things can actually start working soon? | 15:46 |
| jdandrea | Zuuuuuul? (Dinsdale?) https://www.youtube.com/watch?v=dLz07TaTDEA | 15:47 |
| smcginnis | :) | 15:47 |
| dmsimard | smcginnis: there are different ongoing discussions, a partial rollback is one of the options on the table and we'll let everyone know once a plan has been agreed on | 15:47 |
| smcginnis | dmsimard: I quick "current status" post to the ML might be good. There's a lot of confusion and frustration out there right now that might be partially mitigated by some communication. | 15:48 |
| dmsimard | smcginnis: soon | 15:49 |
| fungi | smcginnis: that's what we're polishing now | 15:49 |
| smcginnis | ++ | 15:49 |
| *** vsaienk0 has quit IRC | 15:49 | |
| fungi | in the incident channel | 15:49 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Add mysql reporter to the other pipelines https://review.openstack.org/509212 | 15:50 |
| *** lukebrowning has joined #openstack-infra | 15:50 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Revert "Shift nodepool quota from v2 to v3" https://review.openstack.org/509200 | 15:51 |
| pabelanger | mordred: jeblair: clarkb: fungi: ^should give us 80/20 for nodepool | 15:51 |
| dmsimard | something looks off in elastic-recheck, it's complaining about an outdated index http://status.openstack.org/elastic-recheck/ "Delay in Elastic Search: Indexing behind by 59 hours" | 15:51 |
| dmsimard | However logstash.o.o is correctly returning up to date results | 15:51 |
| jeblair | zuul is moving and i sagged gate/check contents | 15:52 |
| jeblair | and it's swapping again | 15:53 |
| *** shardy is now known as shardy_afk | 15:53 | |
| fungi | amusingly, my promote command still hasn't returned control to the shell, now over 2 hours in | 15:54 |
| *** Guest11048 has quit IRC | 15:54 | |
| *** lukebrowning has quit IRC | 15:55 | |
| clarkb | dmsimard: that is because zuulv3 doesn't write console.html files | 15:55 |
| *** baoli_ has quit IRC | 15:58 | |
| *** baoli has joined #openstack-infra | 16:01 | |
| *** lucasagomes is now known as lucas-afk | 16:01 | |
| *** lukebrowning has joined #openstack-infra | 16:02 | |
| *** tosky has quit IRC | 16:04 | |
| *** egonzalez has quit IRC | 16:04 | |
| *** lukebrowning has quit IRC | 16:06 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul feature/zuulv3: Add change ID to NodeRequest https://review.openstack.org/509215 | 16:06 |
| *** lukebrowning has joined #openstack-infra | 16:08 | |
| *** hemna__ has quit IRC | 16:09 | |
| *** dizquierdo has quit IRC | 16:10 | |
| AJaeger | jlk: regarding the translations change: have a look at the screts.yaml file, I think it's wrong... | 16:10 |
| *** srobert has quit IRC | 16:10 | |
| *** hashar is now known as hasharAway | 16:11 | |
| jlk | it's entirely possible. Secrets may have changed out from under me in the week+ since I wrote it | 16:11 |
| AJaeger | yep, might be - I suggest you ask for help here... | 16:12 |
| *** lukebrowning has quit IRC | 16:13 | |
| *** ccamacho has quit IRC | 16:13 | |
| *** dave-mcc_ is now known as dave-mccowan | 16:13 | |
| jlk | I plan to dig in. what's interesting about that error message is that it's complaining about a job that is not in the change. | 16:14 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Revert "Add redirect from status.o.o/zuul to zuulv3.openstack.org" https://review.openstack.org/509220 | 16:14 |
| *** lukebrowning has joined #openstack-infra | 16:14 | |
| pabelanger | mordred: ^ will be needed too | 16:14 |
| *** yamahata has joined #openstack-infra | 16:15 | |
| *** srobert has joined #openstack-infra | 16:19 | |
| *** krtaylor has joined #openstack-infra | 16:19 | |
| *** lukebrowning has quit IRC | 16:19 | |
| *** xyang1 has joined #openstack-infra | 16:19 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Fail if Zuul v2 frozen files are touched https://review.openstack.org/509158 | 16:19 |
| AJaeger | pabelanger, mordred, made this non-voting and changed message ^ | 16:20 |
| SamYaple | are we doing a rollback? (catching up on scrollback) | 16:21 |
| AJaeger | SamYaple: a partial one - mail is just getting send out... | 16:22 |
| SamYaple | ok thanks | 16:22 |
| pabelanger | Yah, ML post should be shortly | 16:22 |
| jlk | AJaeger: actually I don't know why it throws a syntax error. The job is one that isn't touched by my change, and the syntax is correct. It is saying "use the secret named "static_ssh_key" and apply it to a variable named "fileserver". I don't know why zuul was looking for a secret named "fileserver". | 16:24 |
| AJaeger | jlk: I can't help ;( hope mordred or pabelanger can | 16:24 |
| pabelanger | got a log to look at? | 16:25 |
| jlk | pabelanger: see https://review.openstack.org/#/c/502208/9 | 16:25 |
| pabelanger | looking | 16:25 |
| *** yamahata has quit IRC | 16:25 | |
| jlk | I think the YAML parser got confused | 16:25 |
| pabelanger | AJaeger: jlk: I think it is possible zuul didn't fully load configuration again | 16:28 |
| jlk | lets see what happens on recheck, if it ever gets to there | 16:28 |
| AJaeger | jlk: it will eventually ;) | 16:29 |
| *** lnxnut has joined #openstack-infra | 16:29 | |
| pabelanger | lets make a note on etherpad and dig into debug.log | 16:29 |
| pabelanger | clarkb: just a heads up, I might be missing the weekly meeting today (if we have one). I have to run an errand around that time | 16:31 |
| pabelanger | but will catch up on minutes | 16:32 |
| *** lukebrowning has joined #openstack-infra | 16:32 | |
| *** sree has joined #openstack-infra | 16:32 | |
| *** patriciadomin_ is now known as patriciadomin | 16:32 | |
| *** haleyb has joined #openstack-infra | 16:35 | |
| *** trown|lunch is now known as trown | 16:36 | |
| *** lnxnut has quit IRC | 16:36 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Use new post pipelines https://review.openstack.org/509221 | 16:36 |
| jlvillal | Can I assume the zuulv3 status page is down and known about. http://zuulv3.openstack.org/ | 16:36 |
| *** lukebrowning has quit IRC | 16:36 | |
| jeblair | i'm going to restart zuulv3 now and re-enqueue with pipeline contents i saved earlier | 16:37 |
| AJaeger | jlvillal: yes, we're restarting it soon - and rollback partially to v2 | 16:37 |
| pabelanger | jeblair: ack | 16:37 |
| fungi | jlvillal: not so much "down" as zuul is simply too busy to fulfill requests for the status data | 16:37 |
| *** hasharAway has quit IRC | 16:37 | |
| openstackgerrit | yatin proposed openstack-infra/project-config master: Remove legacy magnum jobs from pipeline https://review.openstack.org/508804 | 16:37 |
| jlvillal | fungi: Thanks | 16:37 |
| *** tesseract has quit IRC | 16:37 | |
| jlvillal | AJaeger: Thanks | 16:37 |
| jeblair | fungi: it's down and your promote command still didn't return | 16:38 |
| jeblair | fungi: wanna just c-c that? | 16:38 |
| *** martinkopec has quit IRC | 16:38 | |
| fungi | c-c? | 16:38 |
| AJaeger | control c? | 16:38 |
| AJaeger | kill? | 16:38 |
| fungi | done | 16:38 |
| fungi | sigint sent | 16:38 |
| jeblair | starting | 16:38 |
| jeblair | re-enqueing | 16:42 |
| dansmith | if we rollback, do we have issues with the zuul user holding a -2 on patches that failed the gate in v3? | 16:42 |
| clarkb | dansmith: yes, we'll have to clear those out | 16:42 |
| *** lukebrowning has joined #openstack-infra | 16:42 | |
| fungi | dansmith: we can clear those, just need a query which identifies them | 16:42 |
| clarkb | (we had similar problem with jenkins -2s when zuul was user) | 16:43 |
| dansmith | clarkb: yeah I figure there's a lot more going back than forward | 16:43 |
| jungleboyj | Good note to the ML. Thanks guys! | 16:43 |
| jungleboyj | Is there going to be a follow up as to when we can start rechecking jobs and trying to get things through? | 16:44 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/zuul master: Use new infra pipelines https://review.openstack.org/509223 | 16:44 |
| dansmith | something like this? https://review.openstack.org/#/q/label:verified%253D-2+status:open | 16:44 |
| pabelanger | jungleboyj: we'll likely send out a status update via IRC | 16:45 |
| fungi | well, but filtered to votes left by the zuul user, which is the tricky bit of gerrit query syntax i haven't had time to look up yet | 16:45 |
| dansmith | mordred: does your email mean that v2 and v3 are running check, v3 being advisory only? | 16:45 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/zuul feature/zuulv3: Use new infra pipelines https://review.openstack.org/509224 | 16:45 |
| AJaeger | dansmith: yes | 16:46 |
| dansmith | fungi: I can figure on that for you | 16:46 |
| dansmith | AJaeger: cool, thanks | 16:46 |
| *** hashar has joined #openstack-infra | 16:46 | |
| fungi | thanks dansmith! | 16:46 |
| clarkb | dansmith: fungi I think its something like verified:-2,zuul | 16:46 |
| jungleboyj | pabelanger: Ok. | 16:46 |
| fungi | clarkb: aha, commas | 16:46 |
| *** hashar is now known as hasharAway | 16:46 | |
| dansmith | clarkb: fungi: yep: https://review.openstack.org/#/q/label:verified%253D-2%252Czuul+status:open | 16:47 |
| fungi | dansmith: clarkb: yep, that seems to work | 16:47 |
| fungi | i'll get to work clearing those | 16:47 |
| *** jpich has quit IRC | 16:47 | |
| *** lukebrowning has quit IRC | 16:47 | |
| *** SumitNaiksatam has quit IRC | 16:48 | |
| fungi | though i'll likely need a second pass after the rollback to catch any which might happen in the interim | 16:48 |
| dansmith | fungi: do we need to wait until after v3 is shut down? | 16:48 |
| dansmith | yeah, that | 16:48 |
| fungi | doesn't hurt to get a head start | 16:48 |
| SamYaple | so... i am fully migrated to v3. all the v2 scripts purged. repo structure changed | 16:49 |
| SamYaple | im screwed arent i | 16:49 |
| fungi | SamYaple: well, your v2 jobs still exist, just the translated legacy v3 jobs got purged right? | 16:50 |
| SamYaple | fungi: i mean i dont have teh scripts the v2 jobs called | 16:50 |
| SpamapS | Why can't we keep v3 running, just not with _everything_ in it? | 16:50 |
| fungi | SamYaple: or you mean you've made non-v2-compatible changes in your projects? | 16:50 |
| SamYaple | lots of non-v2-compatible changes | 16:50 |
| fungi | SpamapS: that's what we're doing, advisory check on v3 but still running | 16:51 |
| SamYaple | cherry on top is all of the bash scripts are purged, even if they got restored the repo is different and i would have to rewrite chunks of them :/ | 16:51 |
| SpamapS | fungi: so couldn't we just let SamYaple keep his gate in v3 too? | 16:51 |
| SamYaple | at the very least, can i purge my v2 gates? | 16:51 |
| *** slaweq_ has quit IRC | 16:51 | |
| SpamapS | Seems like a necessary part of the partial rollback | 16:51 |
| fungi | SpamapS: see the announcement mordred sent | 16:52 |
| *** priteau has joined #openstack-infra | 16:52 | |
| ykarel | fungi, should i stop it from being merged: https://review.openstack.org/#/c/508676/, or i should keep as it would automatically be picked up with zuulv3 comes again | 16:52 |
| fungi | not advised because of frequent outages while we have v3 operating in advisory capacity | 16:52 |
| openstackgerrit | Merged openstack-infra/irc-meetings master: Propose OpenStack Valet meeting date/time https://review.openstack.org/508933 | 16:52 |
| SamYaple | ok. lets go another direction. is there anything I can help with for zuulv3? literally anything | 16:53 |
| fungi | SamYaple: i suppose you could switch to no-op jobs in v2 and just pay close attention to zuul -1/+1 in check, but i wouldn't recommend that for quality assurance reasons | 16:53 |
| SamYaple | fungi: meh this project isn't marked stable yet. so thats not a huge concern | 16:54 |
| *** priteau has quit IRC | 16:54 | |
| fungi | SamYaple: https://etherpad.openstack.org/p/zuulv3-issues | 16:54 |
| SamYaple | fungi: thanks. ill see what i can do | 16:54 |
| fungi | feel free to hack on fixes for stuff you see in there | 16:54 |
| fungi | or review proposed fixes for them as well | 16:55 |
| *** lukebrowning has joined #openstack-infra | 16:55 | |
| openstackgerrit | Merged openstack-infra/project-config master: Open read permissions on /opt/cache/files/* https://review.openstack.org/509116 | 16:55 |
| *** derekh has quit IRC | 16:55 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Add openstack-tox-validate job https://review.openstack.org/508996 | 16:56 |
| SamYaple | thanks fungi, ill keep working on the cross-repo stuff for requirements since that made the list | 16:57 |
| fungi | thanks SamYaple! | 16:57 |
| *** esberglu has quit IRC | 16:58 | |
| *** sree has quit IRC | 16:58 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/openstack-zuul-jobs master: Make openstackci-beaker-ubuntu-trusty and logstash-filters non-voting https://review.openstack.org/509182 | 16:58 |
| *** esberglu has joined #openstack-infra | 16:58 | |
| *** yamahata has joined #openstack-infra | 16:58 | |
| *** lukebrowning has quit IRC | 17:00 | |
| *** sree has joined #openstack-infra | 17:00 | |
| *** lukebrowning has joined #openstack-infra | 17:01 | |
| *** baoli has quit IRC | 17:02 | |
| *** baoli has joined #openstack-infra | 17:03 | |
| *** ralonsoh has quit IRC | 17:03 | |
| *** nikhil_k has quit IRC | 17:04 | |
| inc0 | zuul.o.o is a bit slow, fyi | 17:05 |
| *** ijw has joined #openstack-infra | 17:05 | |
| *** pblaho has joined #openstack-infra | 17:05 | |
| SamYaple | inc0: did you see the ML? | 17:06 |
| pabelanger | yes, we are working to fix it | 17:06 |
| *** lukebrowning has quit IRC | 17:06 | |
| *** ykarel has quit IRC | 17:06 | |
| *** gouthamr has joined #openstack-infra | 17:06 | |
| *** sree has quit IRC | 17:07 | |
| inc0 | SamYaple: not yet:( thanks pabelanger | 17:07 |
| *** sree has joined #openstack-infra | 17:07 | |
| *** lukebrowning has joined #openstack-infra | 17:08 | |
| inc0 | soo...since we don't have any zuulv2 gates on and just zuulv3... | 17:09 |
| pabelanger | when we rollback to zuulv2.5, you'll get the gates back | 17:09 |
| inc0 | ok.. | 17:09 |
| clarkb | the entire zuulv2 config is still there in project-config | 17:10 |
| pabelanger | yah | 17:10 |
| *** jpena is now known as jpena|off | 17:10 | |
| inc0 | ah ok | 17:10 |
| dansmith | pabelanger: is that how the math works? you average all the versions of zuul running? | 17:11 |
| dansmith | should be zuulv2.2 then based on the 80/20 split | 17:11 |
| pabelanger | zuulv2.5 is when we replaced jenkins with ansible | 17:11 |
| *** sree has quit IRC | 17:11 | |
| *** jascott1 has joined #openstack-infra | 17:11 | |
| fungi | yeah, technically we'll be running 2.6.0(+) on the v2 system | 17:12 |
| dansmith | oh, heh, okay well that really ruins my joke then | 17:12 |
| *** lukebrowning has quit IRC | 17:12 | |
| *** hemna__ has joined #openstack-infra | 17:12 | |
| fungi | looks like straight 2.6.0 probably since i don't see any newer commits in master than the tag | 17:12 |
| jlk | dansmith: your jokes are bad and you should feel bad. | 17:13 |
| *** Swami has joined #openstack-infra | 17:13 | |
| mordred | dansmith: that's a shame- it was a good math joke | 17:13 |
| smcginnis | :D | 17:13 |
| dansmith | jlk: *cries* | 17:13 |
| SpamapS | seriously, take more care of dansmith's math jokes | 17:14 |
| *** lukebrowning has joined #openstack-infra | 17:14 | |
| *** ociuhandu has quit IRC | 17:15 | |
| inc0 | 80/20 = 4...zuulv4 confirmed? | 17:15 |
| *** SumitNaiksatam has joined #openstack-infra | 17:15 | |
| fungi | yes, on august 0 of 2020 | 17:16 |
| SamYaple | fungi: jokes on you, i already count my days 0 indexed | 17:16 |
| dansmith | august[0] == 1 | 17:16 |
| dansmith | heh right | 17:16 |
| fungi | 8/0/20 | 17:17 |
| SamYaple | fungi: no. 20/8/0 | 17:17 |
| SamYaple | americans. ugh | 17:17 |
| clarkb | but that is a division error | 17:18 |
| *** lukebrowning has quit IRC | 17:18 | |
| inc0 | no, it's august infinity 2020 | 17:18 |
| fungi | SamYaple: yeah, but i couldn't make 80/20 into any sort of sane iso8601 date | 17:18 |
| inc0 | infinity'th | 17:18 |
| SamYaple | haha | 17:19 |
| SamYaple | well this has been fun. now its time for me to cry since i was moments away from my patches hitting post and seeing if it pushed to dockerhub with secrets | 17:19 |
| fungi | well, not for the near future at any rate | 17:19 |
| fungi | the year 8020 is still a ways off | 17:19 |
| fungi | we should have your post jobs working in v3 before then ;) | 17:20 |
| SamYaple | at this rate..... | 17:20 |
| SamYaple | :D | 17:20 |
| *** lukebrowning has joined #openstack-infra | 17:20 | |
| *** kukacz has quit IRC | 17:20 | |
| *** baoli has quit IRC | 17:21 | |
| pabelanger | AJaeger: fungi: clarkb: mordred: could use a +3 on https://review.openstack.org/509182/ to help us land patches in system-config | 17:21 |
| *** baoli has joined #openstack-infra | 17:22 | |
| *** efried has quit IRC | 17:24 | |
| *** slaweq_ has joined #openstack-infra | 17:24 | |
| *** lukebrowning has quit IRC | 17:24 | |
| openstackgerrit | Andrea Frittoli proposed openstack-infra/zuul-jobs master: Add a generic stage-artifacts role https://review.openstack.org/509233 | 17:25 |
| openstackgerrit | Andrea Frittoli proposed openstack-infra/zuul-jobs master: Add compress capabilities to stage artifacts https://review.openstack.org/509234 | 17:25 |
| andreaf | mordred, jeblair, dmsimard: ^^^ a generic role to copy files to a stage folder and compress them | 17:26 |
| *** lukebrowning has joined #openstack-infra | 17:26 | |
| *** kukacz has joined #openstack-infra | 17:27 | |
| dmsimard | andreaf: added to my review queue thanks | 17:27 |
| inc0 | do you have any eta for full zuulv3 migration? as in, Kolla is frozen now until we get our gates back, should I unfreeze it? | 17:27 |
| inc0 | it's days or hours? | 17:27 |
| andreaf | dmsimard: thanks | 17:27 |
| dmsimard | inc0: have you seen http://lists.openstack.org/pipermail/openstack-dev/2017-October/123049.html ? | 17:28 |
| *** slaweq_ has quit IRC | 17:28 | |
| dmsimard | ah nevermind, was catching up with scrollback | 17:28 |
| *** slaweq_ has joined #openstack-infra | 17:28 | |
| openstackgerrit | Andrea Frittoli proposed openstack-infra/zuul-jobs master: Add compress capabilities to stage artifacts https://review.openstack.org/509234 | 17:30 |
| AJaeger | could I get another +2A on a change to move us further with releasenotes, please? https://review.openstack.org/#/c/509145/ | 17:30 |
| AJaeger | sorry, forget - that has +2A already... | 17:30 |
| *** slaweq_ has quit IRC | 17:30 | |
| *** lukebrowning has quit IRC | 17:31 | |
| *** lnxnut has joined #openstack-infra | 17:34 | |
| fungi | inc0: per mordred's status update to the ml, "hopefully in a week or two" | 17:35 |
| * inc0 needs to learn how to read. sorry | 17:35 | |
| *** lukebrowning has joined #openstack-infra | 17:35 | |
| fungi | no apologies needed, just trying to make sure we all stick to a consistent message ;) | 17:35 |
| mordred | fungi, pabelanger: feel like a +3 on https://review.openstack.org/#/c/509177/ ? it just saw a project-config post job break on the same issue | 17:36 |
| *** ijw has quit IRC | 17:36 | |
| inc0 | so, way I understand it is we'll get zuulv3 feedback on all our patches, so we can develop job migration stuff too over this time? | 17:36 |
| AJaeger | inc0: yes | 17:37 |
| inc0 | that's perfect, thank you | 17:37 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Fail if Zuul v2 frozen files are touched https://review.openstack.org/509158 | 17:37 |
| *** rbrndt has quit IRC | 17:37 | |
| fungi | mordred: concerns that it's better suited to openstack-zuul-jobs but not strongly enough to -2 | 17:39 |
| mordred | fungi: yah - I wanna rework it to not be venv-specific like that - that pach is a stopgap - maybe I should put in a TODO note? | 17:40 |
| AJaeger | mordred: sent the note as followup... | 17:40 |
| fungi | if you want to do that in a followup, that would work | 17:40 |
| *** lukebrowning has quit IRC | 17:40 | |
| mordred | fungi: will do | 17:40 |
| fungi | thanks! | 17:40 |
| *** ociuhandu has joined #openstack-infra | 17:41 | |
| *** lukebrowning has joined #openstack-infra | 17:42 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Add TODO note about reworking fetch-tox-output https://review.openstack.org/509237 | 17:42 |
| mordred | fungi: thereyago | 17:42 |
| fungi | +3 | 17:43 |
| mordred | \o/ | 17:43 |
| *** lnxnut has quit IRC | 17:43 | |
| openstackgerrit | Merged openstack-infra/project-config master: Revert "Shift nodepool quota from v2 to v3" https://review.openstack.org/509200 | 17:44 |
| *** lukebrowning has quit IRC | 17:45 | |
| *** lukebrowning has joined #openstack-infra | 17:45 | |
| openstackgerrit | Chandan Kumar proposed openstack-infra/project-config master: Add python-tempestconf project https://review.openstack.org/508502 | 17:45 |
| *** dprince has quit IRC | 17:46 | |
| *** sambetts is now known as sambetts|afk | 17:47 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/elastic-recheck master: Add query for Ansible privilege escalation timeout https://review.openstack.org/505233 | 17:48 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/elastic-recheck master: Add query for UNREACHABLE tasks https://review.openstack.org/509241 | 17:48 |
| clarkb | fungi: not to distract from rollback, but I noticed that for my personal account ovh sends email in english, I wonder if we can get the lang set to english on the infra accounts? | 17:49 |
| clarkb | or we just make pabelanger do all translations :) | 17:49 |
| dmsimard | infra-root: ^ still need reviews on the privilege escalation timeout issue and added a new query for tracking a high number of unreachable tasks | 17:49 |
| pabelanger | clarkb: oui! | 17:49 |
| dmsimard | clarkb: I can also translate :D | 17:49 |
| *** dhajare has quit IRC | 17:50 | |
| andreaf | jeblair. mordred: fyi - I started moving the tempest zuulv3 native job into the tempest repo directly https://review.openstack.org/509242 | 17:50 |
| jeblair | andreaf: cool, thanks! | 17:50 |
| fungi | clarkb: yes, i semi-recently set the default language on our account in ovh to "english" (which required me first changing the location from "france" to something like "united states outlying islands") | 17:51 |
| fungi | clarkb: apparently the account type we have is not allowed for companies in the united states, or something like that, so outlying islands was the closest i could get (i'll make the argument that the island i live on is "outlying") | 17:52 |
| clarkb | ha | 17:52 |
| fungi | however, the e-mails i've been getting are still in french, so i may not have found all the accounts which need correcting | 17:53 |
| *** ekcs has joined #openstack-infra | 17:53 | |
| fungi | infra-root: also citynet just contacted us today asking why we're booting virtual machines in their "nova-local" zone, which is a lower-capacity zone with ssd-backed servers | 17:54 |
| fungi | and has asked us to stop | 17:54 |
| clarkb | I think we boot in the default zone... | 17:55 |
| clarkb | eg wedon't select an az we use what nova gives us | 17:55 |
| fungi | okay, so nodepool doesn't attempt to split requests between availability zones or anything like that | 17:55 |
| pabelanger | ya, I've noticed that too | 17:55 |
| clarkb | fungi: not by default, you can configure it to use different azs | 17:55 |
| pabelanger | Yah | 17:55 |
| clarkb | but default is whatever nova gives you | 17:55 |
| fungi | i'll reply | 17:55 |
| clarkb | fungi: if you ask them what az we should use we can configure it to use that az instead | 17:56 |
| *** bnemec has quit IRC | 17:56 | |
| clarkb | unless os client config is doing something it shouldn't | 17:56 |
| jeblair | is that a novel use of azs? | 17:56 |
| Shrews | i noticed the other day we had 2 az's for them: nova and nova-local. thought that was curious | 17:57 |
| pabelanger | Shrews: ya, seen that a few times too | 17:57 |
| fungi | what's the best way to double-check the default availability zone returned for a given region? | 17:58 |
| clarkb | fungi: I think you boot an instance without specifying then nova show the instance | 17:59 |
| clarkb | fungi: that should give you the az as a key value pair | 17:59 |
| fungi | :( | 17:59 |
| fungi | so there's no show command for nova to just tell you? | 18:00 |
| clarkb | not that I know of | 18:00 |
| clarkb | mriedem: ^ | 18:00 |
| jdandrea | AJaeger https://review.openstack.org/#/c/508971 is merged but a recheck of https://review.openstack.org/#/c/508924 still fails. Do I need to try something different? | 18:00 |
| SamYaple | Shrews: one has local storage perhaps? | 18:00 |
| dansmith | clarkb: nova availability-zone-list ? | 18:01 |
| clarkb | fungi: I looked at oscc and don't see if specifying anything az wise | 18:01 |
| dansmith | oh default... | 18:01 |
| *** bnemec has joined #openstack-infra | 18:01 | |
| *** dave-mccowan has quit IRC | 18:01 | |
| clarkb | dansmith: ya the one nova will use if not specified | 18:01 |
| fungi | dansmith: i'll give that a shot and see if it indicates which is default | 18:01 |
| *** slaweq_ has joined #openstack-infra | 18:01 | |
| fungi | i just don't want to go telling citycloud that they've changed their default az without some proof | 18:02 |
| dansmith | fungi: I don't see anything about the default | 18:02 |
| dansmith | unless you are about azs I'm not sure why you'd be specifying one | 18:02 |
| AJaeger | jdandrea: wait until we partially rolled back to Zuul v2, please | 18:03 |
| fungi | i don't think we are, but can't be 100% sure the libraries we're using aren't picking one at random instead of taking the default | 18:03 |
| dansmith | that would be pretty uncool | 18:03 |
| *** dave-mccowan has joined #openstack-infra | 18:03 | |
| fungi | i agree with that sentiment | 18:03 |
| *** dprince has joined #openstack-infra | 18:04 | |
| jdandrea | AJaeger You got it. | 18:04 |
| dansmith | so I think if you don't specify one, it just picks a host and then your az is whatever that host's az is | 18:04 |
| jdandrea | AJaeger Is there someplace I can keep tabs on status of the rollback? | 18:04 |
| clarkb | in the hpcloud days they could at least specify on the nova side a default az | 18:04 |
| SamYaple | dansmith: there is a defualt_az that deaults to 'nova' in nova.conf, i think it picks that | 18:05 |
| clarkb | which was az2 and made using az2 the worst because everyone landed in it | 18:05 |
| EmilienM | mordred: thanks for sharing the update on openstack-dev, very helpful | 18:05 |
| *** jascott1 has quit IRC | 18:05 | |
| fungi | dansmith: basically, citycloud has asked us why we suddenly started using their "nova-local" az they created for their ssd customers, and wants to know if there's something wrong with nodepool causing it to do that | 18:05 |
| dansmith | SamYaple: hmm, okay, that seems odd to me | 18:05 |
| openstackgerrit | Merged openstack-infra/project-config master: Prevent non-check changes, add infra pipelines https://review.openstack.org/509196 | 18:05 |
| dansmith | fungi: they're doing it wrong if they model that as an az I think | 18:05 |
| fungi | so i don't want to be the openstack person telling them they're openstacking wrong unless i have good reason to think that | 18:05 |
| Shrews | fungi: if the nodepool config specifies azs, one is chose from there at random. failing that, one is chosen at random from what the provider returns to us (equivalent to availability-zone-list), | 18:05 |
| *** baoli has quit IRC | 18:06 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Add some Zuul v2 checking to project-config https://review.openstack.org/509244 | 18:06 |
| dansmith | fungi: because you shouldn't be able to select ssd by az, it should be a flavor thing | 18:06 |
| tobiash | fungi: nodepoolv3 may have changed its behavior as it gets a list of all azs and randomly picks one if not specified in the nodepool config | 18:06 |
| Shrews | fungi: and when we boot new nodes for a request, we boot from the same az for all nodes in the request | 18:06 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Add some Zuul v2 checking to project-config https://review.openstack.org/509244 | 18:06 |
| clarkb | Shrews: wait really? why would we choose one at random instead of using what the cloud defaults to? | 18:06 |
| *** baoli has joined #openstack-infra | 18:06 | |
| clarkb | tobiash: Shrews I think that is a bug and we should let nova decide for us | 18:06 |
| fungi | tobiash: Shrews: thanks, so we really should not pick at random from all the provider's zones. i think that means something _is_ wrong with nodepool causing this | 18:07 |
| clarkb | (by not specifying an az in the request) | 18:07 |
| dansmith | well, that's a reasonable client behavior I think | 18:07 |
| fungi | at any rate, we can update our config to specify the one zone they want us to be using | 18:07 |
| *** lukebrowning has quit IRC | 18:07 | |
| AJaeger | jdandrea: see mordred's email on openstack-dev mailing list - and just listen in here ;) | 18:07 |
| clarkb | I think if you don' tspecify an az then your client shouldn't specify one for you at random | 18:08 |
| tobiash | clarkb, fungi: http://git.openstack.org/cgit/openstack-infra/nodepool/tree/nodepool/driver/openstack/provider.py?h=feature/zuulv3#n335 | 18:08 |
| clarkb | instead it should not specify one in the request and let the cloud sort it out | 18:08 |
| mriedem | heh | 18:09 |
| jdandrea | AJaeger ++ | 18:09 |
| mriedem | yeah, what dan said, | 18:09 |
| Shrews | clarkb: we have to know what AZ is going to be used ahead of time so that we can group all of the nodes together. i don't think it's necessarily a bug to select one from what the provider says is available | 18:09 |
| mriedem | by default you get the az that the host is in on which you land | 18:09 |
| mriedem | randomly | 18:09 |
| *** hrybacki is now known as hrybacki|brb | 18:09 | |
| clarkb | Shrews: I'm not sure I understand | 18:09 |
| Shrews | clarkb: fungi: solution here is to specify the one we want in nodepool.yaml | 18:09 |
| Shrews | clarkb: which part? | 18:10 |
| *** lukebrowning has joined #openstack-infra | 18:10 | |
| clarkb | Shrews: "so that we can group all of the nodes together" | 18:10 |
| tobiash | there is the random choice: http://git.openstack.org/cgit/openstack-infra/nodepool/tree/nodepool/driver/openstack/handler.py?h=feature/zuulv3#n425 | 18:10 |
| clarkb | Shrews: the cloud will do that for you if you don't specify | 18:10 |
| *** dave-mccowan has quit IRC | 18:10 | |
| clarkb | since it will use a default az | 18:10 |
| clarkb | nd we should let the cloud decide what that is rather than relying on chance | 18:10 |
| Shrews | clarkb: if there is a multinode job, we want all nodes for that job to be located in the same AZ | 18:10 |
| clarkb | Shrews: yes and that will happen right? | 18:10 |
| clarkb | nova will use its default az for any reques twithout a specified az | 18:11 |
| clarkb | this is how it worked in hpcloud and vexxhost when I used them at least | 18:11 |
| clarkb | its the setting that SamYaple mentioned above that makes it happen | 18:11 |
| jeblair | this is also the behavior of nodepool before shade (az was only specified to nova if specified in nodepool config) | 18:11 |
| Shrews | clarkb: we've seen the AZ not be consistent if not specified | 18:11 |
| *** dave-mccowan has joined #openstack-infra | 18:12 | |
| *** ijw has joined #openstack-infra | 18:12 | |
| jeblair | on what nova component is the default_az set? | 18:12 |
| Shrews | i forget which cloud (infracloud?), but sometimes it would be blank, othertimes "nova" | 18:12 |
| clarkb | also I would argue forcing people to specify an az in the case where they aren't the same to make multinode slightly better is a better option than randomly choosing az's without anyone knowing | 18:13 |
| clarkb | because multinode should work fine across AZs, impact will be minimal | 18:13 |
| clarkb | but scheduling to the "wrong" AZ randomly is very unexpected | 18:14 |
| mordred | so - it seems like there's 2 related issues - 1 is that, at least for now, we should specify az on citycloud | 18:14 |
| fungi | proposed reply to citycloud: http://paste.openstack.org/show/622573/ | 18:14 |
| mordred | the second is the nodepool behavior which is currently doing one thing to deal with a behavior seen in one cloud which turns out to be the wrong behavior in a different cloud | 18:15 |
| *** dizquierdo has joined #openstack-infra | 18:15 | |
| jeblair | and possibly #3) suggest that a flavor profile may be a better way to implement the citycloud split | 18:15 |
| mordred | for that I think we need to verify the initial behavior we saw (random azs if not specified) and if there are two different behaviors that aren't discoverable, we need to do what we do in all of the places like that, figure out how to expose a flag that indicates what sort of az behavior the cloud in question has | 18:16 |
| *** ijw has quit IRC | 18:16 | |
| mordred | because on a cloud where not specifying an az gives you a random az, then the current logic is needed - but on a cloud where not specifying one gives you a consistent 'default' az, then the current nodepool behavior is incorrect | 18:16 |
| fungi | "on a cloud where the provider will be surprised by you requesting anything besides the default az" | 18:17 |
| mordred | so we may need to be able to characterize cloud behavior - OR - if it's only happening on infracloud, it's possible we have a config bug on infra cloud that we need to sort out | 18:17 |
| mordred | fungi: yah | 18:17 |
| *** hrybacki|brb is now known as hrybacki | 18:17 | |
| *** esberglu has quit IRC | 18:20 | |
| *** esberglu has joined #openstack-infra | 18:21 | |
| *** markmcclain has quit IRC | 18:21 | |
| *** esberglu has quit IRC | 18:21 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Be explicit with availability zones on citycloud https://review.openstack.org/509246 | 18:21 |
| *** esberglu has joined #openstack-infra | 18:22 | |
| clarkb | mordred: should we apply that to the v2 config too? | 18:22 |
| jeblair | mordred, clarkb, fungi, Shrews: here's the behavior change: https://review.openstack.org/450345 | 18:22 |
| mordred | clarkb: v2 has a different behavior | 18:22 |
| jeblair | it looks like it was only a v3 thing | 18:22 |
| clarkb | ah ok | 18:22 |
| *** priteau has joined #openstack-infra | 18:22 | |
| mordred | the patch above should take care of the current issue with citycloud while we learn more about the situation | 18:22 |
| clarkb | and that change hans't merged back to master? | 18:22 |
| jeblair | yeah, master still has the old behavior | 18:23 |
| *** rbrndt has joined #openstack-infra | 18:23 | |
| *** markmcclain has joined #openstack-infra | 18:23 | |
| jeblair | everything about 450345 *looks* like it was solving a problem, but i don't see confirmation of that, and it's surprising this hasn't been a problem in master? | 18:24 |
| clarkb | jeblair: right I think for multinode the biggest thing is being in the same region. | 18:24 |
| clarkb | traffic across AZs shouldn't be the end of the world and so likely just works | 18:24 |
| clarkb | (assuming we end up in that situation on some cloud, sounds like infracloud maybe) | 18:24 |
| jeblair | clarkb, mordred: oh wait -- does it work in master because we launch the node first, then launch subnodes with the master's az? | 18:25 |
| jeblair | so it's possible that's a v3-only solution to a v3-only problem | 18:25 |
| clarkb | I thought we launched them all at the same time, but check for primary toget back first? I'd have to go reread code | 18:25 |
| fungi | i suppose we can add a random-availability-zones option or something to nodepool | 18:26 |
| *** electrofelix has quit IRC | 18:26 | |
| mordred | well - honestly, I just want to verify behavior - because it's possible we're operating off of incorrect informatoin for that patch | 18:27 |
| fungi | 509246 lgtm, should we skip ci on it to get it applied more quickly, or hand-patch it on the servers? | 18:27 |
| mordred | if clouds returning random azs if None is given isn't really a thing, then I think we can just revert that patch and go back to it being an optional list you can give if you want to be explicit | 18:27 |
| fungi | i'll update the text of my reply stating that we're limiting our configuration to only use the "nova" az | 18:28 |
| mordred | ++ | 18:28 |
| jeblair | mordred: yeah, sounds reasonable. | 18:28 |
| Shrews | some context around that az patch: http://eavesdrop.openstack.org/irclogs/%23zuul/%23zuul.2017-03-27.log.html | 18:29 |
| mnaser | the only issue i can imagine happening is you might not be able to land the requested # of vms | 18:30 |
| mnaser | vs if you boot N vms in one single boot request, all of them will land in the same az afaik | 18:30 |
| mnaser | picking random az's might fail on our cloud for example for really old az which we have with customers that refuse to remove their older equipment | 18:31 |
| mnaser | s/equipment/vms/ | 18:32 |
| mordred | "default_schedule_zone = None (StrOpt) Availability zone to use when user doesn't specify one" | 18:32 |
| fungi | updated reply (which i will send once 509246 merges): http://paste.openstack.org/show/622576/ | 18:32 |
| jeblair | mordred: perhaps all of this was based on my misapprehension that nova chose one at random? | 18:33 |
| mordred | https://docs.openstack.org/ocata/config-reference/compute/config-options.html | 18:33 |
| mordred | the default value for default_schedule_zone is None | 18:33 |
| Shrews | jeblair: mordred: better context starting here: http://eavesdrop.openstack.org/irclogs/%23zuul/%23zuul.2017-03-24.log.html#t2017-03-24T15:22:20 | 18:34 |
| mordred | dansmith: do you happen to know off the top of your head what nova's behavior is if default_schedule_zone is not set? | 18:34 |
| mnaser | jeblair i dont think nova picks one at random, i think what happens is that nova will attempt to schedule the VMs and then assign the AZ to the instance of the host it ends up being scheduled on | 18:34 |
| mordred | mnaser: from a consumer POV I'd characterize that as "nova picks one at random" | 18:34 |
| mnaser | mordred agreed | 18:34 |
| mordred | mnaser: althgh that makes sense as a scheduler behavior | 18:34 |
| mnaser | also, a draw back of specifying az is that for operators, things like live migrations become harder because its considered as a constraint in teh scheduler in deciding where to live migrate to | 18:35 |
| mnaser | vs if you dont specify one in booot time, itll live migrate anywhere | 18:35 |
| *** baoli has quit IRC | 18:35 | |
| dansmith | mordred: no, but I can look | 18:35 |
| Shrews | jeblair: mordred: so it's possible that i was remembering that situation (we don't wait long enough on the server to know what AZ it will land in) | 18:35 |
| *** slaweq_ has quit IRC | 18:35 | |
| dansmith | I thought the behavior was what mnaser said | 18:35 |
| *** slaweq_ has joined #openstack-infra | 18:35 | |
| mordred | dansmith: I imagine what he says is correct - it's the thing that makes the most sense | 18:35 |
| dansmith | yes | 18:36 |
| dansmith | ah, well, the conf doc says it: | 18:36 |
| jeblair | i have used the word 'misapprehension' twice regarding this topic, 7 months apart. also i still find "Shrews: no, the wording is good, i just object to the meaning :)" relevant. :) | 18:37 |
| dansmith | if it's None then it's not set, otherwise instances get that as their default if not requesting something else | 18:37 |
| *** baoli has joined #openstack-infra | 18:37 | |
| mordred | jeblair, clarkb, Shrews: so without actually proving it, it sounds like there is a reasonable case to be made that we could have clouds with default_schedule_zone set where not providing a zone explicitly will get us onto the zone that the cloudhas indicated which will end up with our nodes all landing in the same zone for multi-node - and there are also clouds where that is not true | 18:37 |
| jeblair | so, at base, i'm convinced there is something to this. | 18:37 |
| jeblair | mordred: i agree with that summary. | 18:37 |
| clarkb | I also think it is a bug if we have more than one az in infracloud (assming that is where this behavior was observed) | 18:38 |
| mordred | and unless we decide that multi-node jobs being in the same az is not important, we're going to wind up with some form of config | 18:38 |
| dansmith | mordred: right, but also, I don't think that they're using az for a legit thing | 18:38 |
| mordred | dansmith: oh - yah - also there is a bug in the citycloud set up ... | 18:38 |
| dansmith | okay just so that's clear :) | 18:38 |
| mordred | but generally speaking, in places where we've found deployer config bugs that can impact nodepool behavior, we like to at least provide a get-out-of-jail config option for folks | 18:39 |
| clarkb | dansmith: flavors should be used instead? | 18:39 |
| dansmith | clarkb: for something like ssds yeah | 18:39 |
| dansmith | but, | 18:39 |
| dansmith | I think it's legit to say "this az charges you half price because it's old gear" and you choosing to boot into there means they bill you less or something | 18:40 |
| dansmith | but "oh you're consuming our expensive resource because you chose an az" is not a good plan I think | 18:40 |
| dansmith | because your client *should* choose azs at random if you're booting things you want spread across azs | 18:40 |
| fungi | i switched the first and second sentences in my reply | 18:40 |
| dansmith | people do (ab)use AZs for lots of reasons though, | 18:41 |
| dansmith | so an override per cloud definitely makes sense | 18:41 |
| *** lnxnut has joined #openstack-infra | 18:41 | |
| dansmith | like only_azs=nova, or never_azs=nova-local, | 18:41 |
| fungi | yup, that's basically what we're accomplishing with 509246 | 18:41 |
| *** slaweq_ has quit IRC | 18:42 | |
| dansmith | cool | 18:42 |
| openstackgerrit | Merged openstack-infra/project-config master: Be explicit with availability zones on citycloud https://review.openstack.org/509246 | 18:44 |
| mordred | \o/ | 18:45 |
| jeblair | right now apparently | 18:45 |
| mordred | so - is that the thing that was at the root of us having issues booting nodes on citycloud? | 18:45 |
| fungi | i bypassed ci on that change | 18:45 |
| mordred | or was that purely from citycloud being like "hey, why you using expensive things?" | 18:45 |
| fungi | so i can follow up with citycloud without annoying them while we're in zuul v3 rollback limbo | 18:45 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Use new post pipelines https://review.openstack.org/509221 | 18:45 |
| mordred | fungi: cool | 18:45 |
| fungi | mordred: well, the text of their message is quoted in full in my reply in the paste, but my interpretation is that they were getting allocation failures because we were requesting instance in that az and they had insufficient capacity to fulfill | 18:46 |
| mordred | fungi: gotcha | 18:47 |
| fungi | so we very well may have seen boot failures resulting from that | 18:47 |
| clarkb | fungi: you can kick.sh against nl02 to apply quickly too | 18:47 |
| clarkb | it was nl02 right? | 18:47 |
| fungi | is that our only v3 launcher right now? | 18:47 |
| *** priteau has quit IRC | 18:47 | |
| jeblair | i'm restarting zuulv3 in the check-only configuration (to clear out the gate pipeline) | 18:48 |
| mordred | jeblair: kk | 18:48 |
| Shrews | fungi: we also have nl01, but providers are not shared across v3 launchers | 18:48 |
| fungi | thanks Shrews | 18:49 |
| fungi | kicking puppet on nl02 | 18:49 |
| fungi | and it completed. sending reply to citycloud now | 18:50 |
| *** lnxnut has quit IRC | 18:50 | |
| AJaeger | team, is it time for a #status notice? | 18:56 |
| clarkb | AJaeger: lets get the status.o.o/zuul page working the way we want first so that it reflects reality | 18:56 |
| AJaeger | clarkb: ok | 18:58 |
| fungi | clarkb: meeting today? ;) | 19:01 |
| fungi | i spoke too soon | 19:01 |
| *** Goneri has quit IRC | 19:08 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul feature/zuulv3: Explicitely print the result status of each playbook run https://review.openstack.org/509254 | 19:09 |
| *** jtomasek has quit IRC | 19:10 | |
| *** ramishra has quit IRC | 19:11 | |
| *** pt_15 has joined #openstack-infra | 19:11 | |
| *** slaweq_ has joined #openstack-infra | 19:12 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/infra-specs master: The Gerrit 2.13 upgrade is implemented https://review.openstack.org/509257 | 19:12 |
| *** MasterOfBugs has joined #openstack-infra | 19:13 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul feature/zuulv3: Explicitely print the result status of each playbook run https://review.openstack.org/509254 | 19:15 |
| openstackgerrit | Merged openstack-infra/system-config master: Revert "Add redirect from status.o.o/zuul to zuulv3.openstack.org" https://review.openstack.org/509220 | 19:17 |
| fungi | clarkb: ^ | 19:17 |
| fungi | should we kick.sh status.o.o now or let it apply gracefully? | 19:18 |
| fungi | s/gracefully/naturally/ | 19:18 |
| clarkb | kick.sh to get it in fast is probably a good idea | 19:18 |
| fungi | on it | 19:18 |
| clarkb | that way status.o.o/zuul looks right when we send our update momentarily | 19:18 |
| inc0 | qq, how to run this on zuulv3? https://review.openstack.org/#/c/508661/ | 19:18 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Move infra-specs and infra-manual to v3 only https://review.openstack.org/509258 | 19:19 |
| AJaeger | clarkb, team, as just proposed in #openstack-meeting ^ | 19:19 |
| clarkb | inc0: all of the check jobs will run as expected | 19:19 |
| clarkb | inc0: unless we have to restart zuulv3 or similar | 19:19 |
| inc0 | http://zuulv3.openstack.org/ <- should this show job in question? | 19:19 |
| clarkb | inc0: you can see status at http://zuulv3.openstack.org | 19:19 |
| inc0 | right... | 19:20 |
| inc0 | that's why I'm asking;) not there | 19:20 |
| clarkb | inc0: there are 343 unprocessed events in the zuul queue (top of status page) its likely one of those | 19:21 |
| *** wolverineav has quit IRC | 19:21 | |
| inc0 | ok so just wait patiently | 19:21 |
| *** wolverineav has joined #openstack-infra | 19:22 | |
| *** hemna__ has quit IRC | 19:22 | |
| *** slaweq_ has quit IRC | 19:22 | |
| *** slaweq_ has joined #openstack-infra | 19:22 | |
| *** ijw_ has joined #openstack-infra | 19:30 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/puppet-subunit2sql master: Ensure that build_names are unique per project https://review.openstack.org/508258 | 19:34 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/puppet-subunit2sql master: Strip legacy prefix if it's present https://review.openstack.org/508272 | 19:35 |
| fungi | i need to disappear for a little while to start on dinner, but will be back to catch up on things after | 19:36 |
| clarkb | I too need food, will keep an eye on irc | 19:36 |
| jdandrea | AJaeger Much better! https://review.openstack.org/#/c/508924/ (ignoring the Zuul results, right?) | 19:36 |
| jdandrea | (Accidentally posted in -incident) | 19:36 |
| jdandrea | +1 for food, well done all | 19:37 |
| AJaeger | jdandrea: if you get a +1, you're fine ;) | 19:37 |
| jdandrea | AJaeger From Jenkins, but not Zuul. | 19:40 |
| AJaeger | jdandrea: we did a partial rollback - to be able to fix the remaining zuul problems... | 19:41 |
| jdandrea | *nod* | 19:41 |
| *** chlong_ has joined #openstack-infra | 19:43 | |
| *** jcoufal_ has quit IRC | 19:43 | |
| *** chlong has joined #openstack-infra | 19:44 | |
| *** chlong has quit IRC | 19:46 | |
| *** lnxnut has joined #openstack-infra | 19:48 | |
| *** bnemec has quit IRC | 19:49 | |
| mtreinish | clarkb, fungi, andreaf: I'm thinking we should land this before too long: https://review.openstack.org/#/q/status:open+project:openstack-infra/puppet-subunit2sql+branch:master+topic:restore-name-sanity even if we rolled back on v3 for now | 19:54 |
| mtreinish | it just won't be used on v2 | 19:54 |
| andreaf | mtreinish: +1 | 19:55 |
| andreaf | mtreinish: thanks for looking into the bisect tests - I was reading the bisect code last Friday but then I got carried away again | 19:55 |
| andreaf | mtreinish: I will review the tests tomorrow | 19:56 |
| mtreinish | andreaf: ok, no worries | 19:56 |
| mtreinish | andreaf: the thing I was surprised by is that the checks are isolated to a single worker, which means it won't catch isolation issues across a worker boundary | 19:56 |
| andreaf | mtreinish: uhm yeah that's not ideal... | 19:58 |
| mtreinish | unless I was misreading something | 19:58 |
| *** lnxnut has quit IRC | 19:59 | |
| *** srobert has quit IRC | 19:59 | |
| *** jkilpatr_ has quit IRC | 20:00 | |
| *** iyamahat has joined #openstack-infra | 20:03 | |
| inc0 | clarkb: https://review.openstack.org/#/c/508661/46 still nto showing up in zuulv3:( | 20:03 |
| *** dhajare has joined #openstack-infra | 20:04 | |
| clarkb | fungi: http://logs.openstack.org/64/508564/1/check/gate-puppet-openstack_health-puppet-beaker-rspec-ubuntu-xenial/9133aa7/console.html#_2017-10-03_19_32_19_886656 so I tink that chnge may work on trusty and not on xenial. We probably want to update it with a distro version check | 20:05 |
| AJaeger | infra-root, what about #status notice "We rolled the Zuul v3 move partially back - please read http://lists.openstack.org/pipermail/openstack-dev/2017-October/123049.html and http://lists.openstack.org/pipermail/openstack-dev/2017-October/123064.html" | 20:05 |
| clarkb | inc0: the event queue hasn't gone to zero | 20:05 |
| inc0 | right...it actually grew quite a bit | 20:06 |
| *** nunchuck has quit IRC | 20:06 | |
| *** deduped has quit IRC | 20:06 | |
| AJaeger | still growing... | 20:06 |
| inc0 | is there any way to explicitly specify check to run on zuulv3 and not run most of it on zuulv3? I think it will get overloaded quite fast | 20:07 |
| inc0 | or maybe turn off legacy jobs for time being | 20:07 |
| inc0 | and give this time for people to migrate to non-legacy...will free up zuulv3 resources quite a bit | 20:08 |
| *** dizquierdo has quit IRC | 20:08 | |
| clarkb | inc0: right now the problem isn't resources it is zuul itself | 20:09 |
| clarkb | until that gets sorted I don't expect it will make a huge difference either way | 20:09 |
| openstackgerrit | Andrea Frittoli proposed openstack-infra/zuul-jobs master: Add a generic stage-artifacts role https://review.openstack.org/509233 | 20:09 |
| jeblair | inc0: we're trying to use as many zuulv3 resources as possible. we need to run it at scale to find problems. | 20:09 |
| openstackgerrit | Andrea Frittoli proposed openstack-infra/zuul-jobs master: Add compress capabilities to stage artifacts https://review.openstack.org/509234 | 20:09 |
| *** akscram1 has quit IRC | 20:10 | |
| inc0 | ok, I'll stay put then:) | 20:10 |
| *** akscram1 has joined #openstack-infra | 20:10 | |
| jeblair | i'm going to pause the scheduler on completion of the current reconfiguration | 20:14 |
| clarkb | jeblair: pause meaning shutdown? | 20:14 |
| jeblair | clarkb: no; i'm going to put it in queue-only mode. it's a dusty old feature. | 20:15 |
| clarkb | huh | 20:15 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: remove usage of check pipeline from zuul-jobs https://review.openstack.org/509274 | 20:15 |
| AJaeger | jeblair: ^ | 20:15 |
| jeblair | it will help stabilize data structures while i continue to debug | 20:15 |
| AJaeger | what does queue-only mean? It won't start jobs? | 20:16 |
| jeblair | in fact, you can see just how long it's been since it's been used by the current message at the top of the page :) | 20:16 |
| jeblair | AJaeger: yes | 20:16 |
| jeblair | AJaeger: well, at least it won't process any more trigger events | 20:18 |
| AJaeger | infra-root, please keep an eye on https://review.openstack.org/#/q/status:open+++topic:zuulv3-rollback - we need some/all (?) of those in for proper merging of changes. | 20:18 |
| jeblair | it may start builds still | 20:18 |
| * AJaeger waves good night | 20:18 | |
| jeblair | AJaeger: good night, thanks! | 20:19 |
| AJaeger | thanks for explanation, jeblair. Successfull debugging ;) | 20:19 |
| *** jkilpatr has joined #openstack-infra | 20:20 | |
| *** david-lyle has quit IRC | 20:20 | |
| SamYaple | tonyb: https://review.openstack.org/#/c/508791/ successful requirements run! (i know we roled back, but still its good to see green) | 20:21 |
| *** david-lyle has joined #openstack-infra | 20:21 | |
| SamYaple | night AJaeger | 20:22 |
| *** Goneri has joined #openstack-infra | 20:24 | |
| *** david-lyle has quit IRC | 20:25 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Include tenant in pipeline log messages https://review.openstack.org/509279 | 20:29 |
| *** kgiusti has left #openstack-infra | 20:29 | |
| *** eharney has quit IRC | 20:31 | |
| *** dave-mccowan has quit IRC | 20:31 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul feature/zuulv3: Explicitely print the result status of each playbook run https://review.openstack.org/509254 | 20:32 |
| SamYaple | prometheanfire: you may want to review https://review.openstack.org/#/c/508249/ (it looks like its -1, but thats just zuulv3 being borked) | 20:33 |
| *** askb has joined #openstack-infra | 20:35 | |
| *** trown is now known as trown|outtypewww | 20:38 | |
| *** askb has quit IRC | 20:39 | |
| *** askb has joined #openstack-infra | 20:41 | |
| *** laurelm has joined #openstack-infra | 20:43 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul feature/zuulv3: Explicitely print the result status of each playbook run https://review.openstack.org/509254 | 20:46 |
| *** pblaho has quit IRC | 20:46 | |
| clarkb | I've reviewd the changes in topic:zuulv3-rollback would be great if other infra-roots can take a look as able (just so we can get that behind us) | 20:47 |
| dmsimard | stepping away for a bit, be back later -- I have a WIP change for elastic-recheck to support v2 and v3 with the help of https://review.openstack.org/509254 | 20:49 |
| clarkb | dmsimard: thanks, I'll look at it shortly | 20:49 |
| tonyb | SamYaple: Cool. | 20:52 |
| SamYaple | tonyb: im working on figuring out the zuulv3 nova-func job so when we switch back over it wont be an issue | 20:55 |
| tonyb | SamYaple: cool. I need to work out the requirements job | 20:55 |
| ihrachys | the MERGER_FAILURE errors like in https://review.openstack.org/#/c/502564/ , are they worth looking at? seems like no logs for those. | 20:56 |
| clarkb | ihrachys: I think they were/are likely load related | 20:57 |
| ihrachys | is the issue gone now? | 20:57 |
| ihrachys | I know v3 is generally broken but it would be nice to at least get results so that we can dive into fixing them | 20:57 |
| clarkb | ihrachys: no but I think its related to the big memory/swap/general performance issue that jeblair is debugging | 20:58 |
| ihrachys | should we refrain from rechecks, or it's cool with you? | 20:59 |
| clarkb | I think rechecks are fine at this point but they may not report back quickly | 21:00 |
| *** tosky has joined #openstack-infra | 21:00 | |
| ihrachys | ok. thanks a lot. | 21:02 |
| *** chlong_ has quit IRC | 21:03 | |
| *** numans has quit IRC | 21:03 | |
| *** boris_42_ has joined #openstack-infra | 21:05 | |
| boris_42_ | AJaeger: hi there | 21:05 |
| boris_42_ | AJaeger: after reverting to zuul v2 something changed all rally jobs are now failing with the same error http://logs.openstack.org/95/509095/1/check/gate-rally-dsvm-ironic-rally-nv/0af950d/logs/devstack-early.txt.gz#_2017-10-03_20_51_03_242 | 21:06 |
| boris_42_ | AJaeger: did you change version of sed? | 21:06 |
| *** numans has joined #openstack-infra | 21:07 | |
| openstackgerrit | Ihar Hrachyshka proposed openstack-infra/project-config master: Return neutron functional job back to voting for zuulv3 https://review.openstack.org/509287 | 21:07 |
| clarkb | boris_42_: AJaeger has called it a day | 21:07 |
| *** rhallisey has quit IRC | 21:07 | |
| boris_42_ | clarkb: hm?) | 21:08 |
| boris_42_ | clarkb: btw does it make sense to change bugs in Rally jobs for zullv2? | 21:08 |
| openstackgerrit | Ihar Hrachyshka proposed openstack-infra/project-config master: Return neutron functional and api jobs back to voting for zuulv3 https://review.openstack.org/509287 | 21:08 |
| boris_42_ | clarkb: that didn't allow migration script to do it work properly ? | 21:08 |
| boris_42_ | to migrate* | 21:09 |
| clarkb | boris_42_: probably not, we are not planning to rerun the migration tooling instead we are rolling forward with that config as is | 21:09 |
| boris_42_ | clarkb: ok | 21:09 |
| boris_42_ | clarkb: so about this issue, is there any way to fix it? or we should just wait for zuul v3 again? | 21:10 |
| clarkb | I really wish it printed the sed command... | 21:11 |
| *** david-lyle has joined #openstack-infra | 21:11 | |
| clarkb | boris_42_: the odd thing is its the same test nodes in v2 and v3 so should have the same versions of sed | 21:11 |
| clarkb | so I'm skeptical that is related | 21:12 |
| boris_42_ | clarkb: so in all rally jobs with devstack plugins the same is happening | 21:15 |
| *** Goneri has quit IRC | 21:16 | |
| dmsimard | clarkb: the elastic-recheck v3 change isn't in yet, I'll wrap it up after dinner | 21:16 |
| clarkb | looks like devstack is generally working | 21:17 |
| boris_42_ | clarkb: part of job works (that dont' have extra devstack plugins) | 21:19 |
| *** baoli has quit IRC | 21:19 | |
| boris_42_ | work* | 21:19 |
| clarkb | boris_42_: do you know what plugin is failing here? is it ironic? | 21:19 |
| boris_42_ | clarkb: give me second I'll collect all data | 21:20 |
| *** ldnunes has quit IRC | 21:20 | |
| *** dhajare has quit IRC | 21:21 | |
| boris_42_ | clarkb: so http://paste.openstack.org/show/622588/ | 21:22 |
| boris_42_ | clarkb: it seems like iniset function of devstack doesn't work properly in devstack plugins | 21:22 |
| boris_42_ | clarkb: if there is no external plugins everything works well * | 21:23 |
| clarkb | is iniset used in regular devstack? | 21:23 |
| boris_42_ | clarkb: I am not big DevStack expert | 21:24 |
| boris_42_ | clarkb: but I think it is let me check | 21:25 |
| clarkb | boris_42_: looks like it is | 21:25 |
| clarkb | and all over so extr aweird that plugins are failing anot the rest of it | 21:25 |
| boris_42_ | clarkb: yep it is | 21:25 |
| boris_42_ | looks like a magic to me | 21:25 |
| clarkb | boris_42_: our best bet is probably to set +x in iniset in a devstack chnage then depends on that where the jobs are failing and see what command is running that fails | 21:26 |
| clarkb | because right now we can't see that and its difficult to infer | 21:27 |
| clarkb | boris_42_: I'll write the devstack patch now | 21:27 |
| clarkb | boris_42_: these failed changes are against master right? | 21:27 |
| boris_42_ | clarkb: ya I just recently rechecked them | 21:27 |
| boris_42_ | clarkb: patches worked before the migration to zuulv3 | 21:27 |
| *** slaweq_ has quit IRC | 21:27 | |
| *** bobh has quit IRC | 21:28 | |
| boris_42_ | @clarkb by the way https://github.com/openstack-infra/project-config/blob/master/jenkins/jobs/rally.yaml#L851 should we change this to simplify things | 21:28 |
| clarkb | boris_42_: I137b3850fa78dd42bc2bc7557987c484fc211846 is the change id to depends on | 21:29 |
| clarkb | boris_42_: can you epends on where it is failing and we can go from there | 21:29 |
| openstackgerrit | Brian Haley proposed openstack-infra/project-config master: WIP: Fix neutron-failure-rate grafana dashboard https://review.openstack.org/509291 | 21:30 |
| boris_42_ | clarkb: we can use this one https://review.openstack.org/#/c/507276/ | 21:31 |
| boris_42_ | clarkb: it just reported results and it's first patch in queue | 21:31 |
| *** jascott1 has joined #openstack-infra | 21:32 | |
| clarkb | boris_42_: can you add the depeonds on to it? | 21:32 |
| *** camunoz has quit IRC | 21:32 | |
| boris_42_ | clarkb: oh sorry to slow today, let me add empty change and depends on (just to avoid any problems with non working patches) | 21:35 |
| clarkb | boris_42_: sounds good | 21:35 |
| *** jascott1 has quit IRC | 21:36 | |
| boris_42_ | clarkb: done https://review.openstack.org/#/c/509290/ | 21:36 |
| boris_42_ | clarkb: https://review.openstack.org/#/c/486017/ | 21:36 |
| boris_42_ | clarkb: lest's wait for 15-20 min, I'll ping you as soon as we have results | 21:37 |
| *** armax has quit IRC | 21:37 | |
| clarkb | boris_42_: sounds good and thank you and sorry :) | 21:37 |
| *** claudiub has quit IRC | 21:38 | |
| *** jascott1 has joined #openstack-infra | 21:38 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Add jinja filters for manipulating zuul.projects https://review.openstack.org/509294 | 21:39 |
| *** priteau has joined #openstack-infra | 21:41 | |
| *** priteau has quit IRC | 21:41 | |
| *** ijw_ has quit IRC | 21:43 | |
| *** david-lyle has quit IRC | 21:45 | |
| clarkb | boris_42_: reading the existing logs more the iniset function is used earlier and does work | 21:47 |
| clarkb | I'm beginning to think the bug might be local section=/var/cache/ironic/api | 21:47 |
| clarkb | and the /'s areconfusing sed | 21:47 |
| clarkb | the earlire inisets don't have paths as the section (there are paths used in other parts of the function params though) | 21:48 |
| openstackgerrit | Merged openstack-infra/system-config master: Set ZooKeeper purge_interval for on nodepool.o.o https://review.openstack.org/508962 | 21:49 |
| mordred | clarkb: I put this up yesterday https://review.openstack.org/#/c/509007/ ... I tihnk we might still want to land it | 21:50 |
| clarkb | mordred: ++ | 21:50 |
| * clarkb single approves as thats not gating anymore | 21:50 | |
| *** jascott1 has quit IRC | 21:52 | |
| mordred | clarkb: for the infra-manual / infra-specs - you mention maybe wanting to keep infra-manual in v2 - I think Im game for moving it over and sometimes having to wait - but I'll defer to you before I hit +A | 21:52 |
| mordred | oh - nevermind- fungi just got it :) | 21:52 |
| clarkb | mordred: I think its fine as is if others are fine with it | 21:52 |
| clarkb | I still +2'd :) | 21:53 |
| *** esberglu has quit IRC | 21:53 | |
| * clarkb is trying tno to dig into anything too much right now so I can focus on helping boris_42_ et al | 21:53 | |
| *** vgnbkr has quit IRC | 21:53 | |
| *** schang has quit IRC | 21:53 | |
| *** peterstac has quit IRC | 21:54 | |
| *** laurelm has quit IRC | 21:55 | |
| *** dougshelley66 has quit IRC | 21:55 | |
| *** lnxnut has joined #openstack-infra | 21:55 | |
| fungi | mordred: clarkb: yeah, i prefer that we have it as a representative job for afs-based publishing | 21:55 |
| mordred | fungi: ah - good point | 21:56 |
| fungi | and the specs site gets us static.o.o content publishing | 21:56 |
| clarkb | boris_42_: actually did the depends on not get added to that change? | 21:56 |
| *** reed has quit IRC | 21:56 | |
| openstackgerrit | Mike Perez proposed openstack-infra/system-config master: Remove note about only infra projects on StoryBoard https://review.openstack.org/509298 | 21:56 |
| fungi | thanks thingee! | 21:57 |
| *** edmondsw has quit IRC | 21:57 | |
| *** slaweq_ has joined #openstack-infra | 21:58 | |
| *** reed has joined #openstack-infra | 21:59 | |
| clarkb | 509299 pushed to test devstack plugins | 21:59 |
| *** xyang1 has quit IRC | 22:02 | |
| clarkb | the number of jobs a rally change runs is impressive | 22:03 |
| thingee | fungi: np. related to thread on dev with storyboard and new projects :) | 22:03 |
| *** iyamahat_ has joined #openstack-infra | 22:03 | |
| thingee | as I'm trying to create a new project, I have to send patches to other things :) | 22:03 |
| *** iyamahat has quit IRC | 22:04 | |
| fungi | thingee: yep, i figured. i refrained from weighing in because i didn't want to be "that infra person pushing storyboard on everyone" | 22:05 |
| *** lnxnut has quit IRC | 22:07 | |
| mordred | fungi: where does storyboard log? | 22:07 |
| fungi | mordred: #define "log" | 22:08 |
| mordred | fungi: :) | 22:08 |
| mordred | fungi: so - I'm having issues logging in to storyboard andhave been having such for about a week now | 22:08 |
| mordred | fungi: I figured I'm an admin- so maybe I should investigate | 22:08 |
| fungi | mordred: /var/log/apache2/storyboard-ssl-error.log | 22:10 |
| fungi | it's wsgi | 22:10 |
| mordred | [Tue Oct 03 21:57:31.489398 2017] [:error] [pid 3246:tid 139982270277376] [client 47.188.241.202:16830] IOError: failed to write data, referer: https://storyboard.openstack.org/ | 22:10 |
| mordred | \o/ | 22:10 |
| mordred | what data :) | 22:10 |
| *** iyamahat__ has joined #openstack-infra | 22:15 | |
| *** krtaylor has quit IRC | 22:17 | |
| thingee | fungi: lay off my launchpad bro | 22:18 |
| *** iyamahat_ has quit IRC | 22:18 | |
| *** gouthamr has quit IRC | 22:21 | |
| *** claudiub has joined #openstack-infra | 22:23 | |
| clarkb | http://logs.openstack.org/99/509299/1/check/gate-rally-dsvm-manila-multibackend-no-ss/1383329/logs/devstack-early.txt.gz#_2017-10-03_22_26_00_874 | 22:29 |
| clarkb | boris_42_: ^ ya I think its the extra /s | 22:29 |
| clarkb | boris_42_: zeroing in on that I find I5680376e70e74882e9fdb87ee1b95d5f40570ad7 I think basically all those plugins need to stop using the cache dirs | 22:31 |
| clarkb | boris_42_: so this is something that happaned independent of zuulv3 | 22:31 |
| *** slaweq_ has quit IRC | 22:31 | |
| ianw | clarkb: i'm not going to look, but i bet that's the one i commented on it being used far and wide :/ | 22:32 |
| ianw | ok i looked, it is | 22:32 |
| clarkb | :) | 22:32 |
| clarkb | I'm not sure if devstack should revert or force plugins to roll forward on that | 22:32 |
| clarkb | but I am fairly certain we can't blame zuulv3 for it now :) | 22:33 |
| ianw | yeah, this one would be zuul-free | 22:34 |
| *** lbragstad has quit IRC | 22:34 | |
| *** slaweq_ has joined #openstack-infra | 22:40 | |
| ianw | boris_42_: sed 'v asdf' /dev/null <-- i rekon it's a quoting issue | 22:40 |
| boris_42_ | clarkb: I don't want to blame anybody =) I just would like to make everything working again =) | 22:43 |
| clarkb | boris_42_: oh not trying to blame, just making sur ethere isn't some additional step we need to take to make the rollback successful | 22:44 |
| clarkb | boris_42_: but I think ball is in devstack's court now to decide what the best way forward is (so I'm not a ton of help) | 22:44 |
| boris_42_ | ianw: so should I change those things in all the places? | 22:44 |
| ianw | boris_42_: hang on, just looking | 22:45 |
| ianw | iniset /etc/heat/heat.conf /var/cache/heat auth_type password | 22:45 |
| ianw | it's supposed to be [file] [section] [option] [value] | 22:45 |
| ianw | /var/cache/heat does not seem like a correct option | 22:45 |
| *** hasharAway has quit IRC | 22:45 | |
| *** dizquierdo has joined #openstack-infra | 22:46 | |
| ianw | let's take it to #openstack-qa | 22:46 |
| *** rlandy is now known as rlandy|bbl | 22:48 | |
| *** rbrndt has quit IRC | 22:50 | |
| *** gouthamr has joined #openstack-infra | 22:50 | |
| *** jaypipes has quit IRC | 22:50 | |
| *** tpsilva has quit IRC | 22:52 | |
| *** armax has joined #openstack-infra | 22:53 | |
| dmsimard | clarkb: do you think 'tags:"console"' is a good substitute for 'filename:"console.html"' vs 'filename:"job-output.txt"' ? | 22:53 |
| clarkb | dmsimard: probably not as that includes a lot more data but in this case we are just wanting to look at the top level | 22:56 |
| *** sdague has quit IRC | 22:56 | |
| *** apevec has quit IRC | 22:56 | |
| clarkb | I would do (filename:"console.html" OR filename:job-output.txt" | 22:56 |
| *** jascott1 has joined #openstack-infra | 22:57 | |
| dmsimard | yeah, that's what I was doing but writing these queries in line everywhere is painful :p | 22:57 |
| dmsimard | gotta stay pep8 y0 | 22:57 |
| mordred | clarkb: the iniset thing you're discussin with boris_42_ earlier ... | 22:58 |
| mordred | clarkb: I'm getting it in shade tests too | 22:58 |
| mordred | http://logs.openstack.org/56/508156/1/check/gate-shade-functional-py35/0c77316/logs/devstacklog.txt.gz#_2017-10-03_20_16_47_013 | 22:58 |
| ianw | mordred: just debugged it to https://review.openstack.org/#/c/509307/ | 22:58 |
| dmsimard | someone pointed out that he was getting 404's in an ARA report: http://i.share.pho.to/7e653c12_o.png | 22:59 |
| dmsimard | turns out when you look at the job output, the executor ran out of disk space ? | 22:59 |
| dmsimard | http://logs.openstack.org/03/508903/1/check/legacy-releasenotes/46d6430/job-output.txt.gz#_2017-10-02_15_44_38_299976 | 22:59 |
| dmsimard | that was yesterday though ? | 22:59 |
| mordred | ianw: awesome. thnak you! | 22:59 |
| mordred | dmsimard: yah - there was a disk space event yesterday I believe before we switched the executors to rate-limit themselves | 22:59 |
| dmsimard | mordred: ok, thanks | 22:59 |
| mordred | ianw: I mean - I obviously support the intent of the original patch and all | 23:01 |
| *** wolverineav has quit IRC | 23:01 | |
| ianw | mordred: yeah, it should have been caught in reviews. but it raises the point that the devstack gate doesn't run any plugin tests i guess. we should definitely add one | 23:02 |
| thingee | This mentions there are zuul template jobs already defined, but doesn't mention where https://docs.openstack.org/infra/manual/creators.html#use-templates | 23:03 |
| openstackgerrit | Thomas Morin proposed openstack-infra/project-config master: Add projects to required-projects for networking-(bagpipe|bgpvpn) https://review.openstack.org/508825 | 23:04 |
| *** lnxnut has joined #openstack-infra | 23:04 | |
| *** hongbin has quit IRC | 23:05 | |
| *** SumitNaiksatam has quit IRC | 23:06 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Speed configuration building https://review.openstack.org/509309 | 23:06 |
| dmsimard | woah "Together, these changes build an OpenStack-sized configuration in 8% of the time it currently takes." | 23:07 |
| * clarkb reads | 23:07 | |
| *** ijw has joined #openstack-infra | 23:09 | |
| *** apevec has joined #openstack-infra | 23:09 | |
| *** slaweq_ has quit IRC | 23:13 | |
| *** lnxnut has quit IRC | 23:14 | |
| *** claudiub has quit IRC | 23:15 | |
| *** felipemonteiro_ has quit IRC | 23:17 | |
| *** apevec has left #openstack-infra | 23:18 | |
| fungi | niiice! | 23:18 |
| jeblair | i'm going to restart zuulv3 with that now | 23:20 |
| fungi | excellent | 23:21 |
| *** tosky has quit IRC | 23:22 | |
| SamYaple | so should i be generating load on zuulv3? or would infra prefer i remove a few of my tests as to not trigger so many checks until it comes back online in full force | 23:22 |
| clarkb | SamYaple: I think jeblair is wanting us to throw load at it to make sure we are testing it appropriately | 23:23 |
| jeblair | SamYaple: ya, feel free to use it | 23:23 |
| dmsimard | SamYaple: v3 will be getting load naturally because all projects are still on the check queue | 23:23 |
| dmsimard | SamYaple: so don't worry about it | 23:23 |
| SamYaple | oh good. sweet. cause i have a few jobs to add :) | 23:23 |
| *** slaweq_ has joined #openstack-infra | 23:24 | |
| SamYaple | i was so close this morning to publishing an infra-built image to dockerhub using secrets.... mere minutes away from the post job doing its thing when we rolled back to v2.5 | 23:24 |
| SamYaple | but this just means im ready for when it returns | 23:24 |
| *** bnemec has joined #openstack-infra | 23:25 | |
| jeblair | i should have scottied my estimate | 23:27 |
| jeblair | it actually took 7 seconds to build a dynamic config | 23:27 |
| clarkb | jeblair: and old number was 50s? | 23:27 |
| jeblair | yes | 23:27 |
| jeblair | i've rechecked the zuulv3-rollback topic | 23:28 |
| SamYaple | niiice | 23:29 |
| jeblair | there's something 78 jobs running now, and no red on the status page. haven't seen that in a while. | 23:32 |
| jeblair | okay, i need to afk for a while | 23:33 |
| SamYaple | :) thanks ofr the hard work | 23:33 |
| SamYaple | it is very much appreciated | 23:33 |
| mnaser | jeblair the python magician | 23:34 |
| *** xarses has quit IRC | 23:34 | |
| mnaser | woah its so foast | 23:37 |
| *** dizquierdo has quit IRC | 23:38 | |
| *** aeng has joined #openstack-infra | 23:38 | |
| *** mriedem has quit IRC | 23:39 | |
| clarkb | looks like the fetch tox thing is unhappy (but thats not zuul schedulers fault) | 23:40 |
| clarkb | mordred: you'll probably be interested in current jobs on zuulv3 for that reason | 23:40 |
| *** jrist is now known as jrist-afk | 23:42 | |
| *** jrist-afk is now known as jrist | 23:42 | |
| mnaser | is zuulv3 console not working something expected? | 23:43 |
| *** david-lyle has joined #openstack-infra | 23:44 | |
| *** Swami has quit IRC | 23:44 | |
| *** aeng has quit IRC | 23:45 | |
| clarkb | mnaser: it had been having problems for a while | 23:45 |
| clarkb | mnaser: its something that will need debugging | 23:45 |
| mnaser | clarkb i thought it was rollback related that it isn't working | 23:45 |
| clarkb | mnaser: I don't think so | 23:45 |
| clarkb | since everything is on the zuulv3 side for console streaming | 23:46 |
| clarkb | zuulv2 shouldn't touch it | 23:46 |
| *** aeng has joined #openstack-infra | 23:46 | |
| mnaser | weird, okay cool | 23:46 |
| *** dingyichen has joined #openstack-infra | 23:50 | |
| *** slaweq_ has quit IRC | 23:56 | |
| *** markvoelker has quit IRC | 23:59 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!
