| *** jkilpatr has quit IRC | 00:13 | |
| jeblair | pabelanger: can you take a look at http://docs-draft.openstack.org/35/491635/1/check/gate-zuul-sphinx-docs-ubuntu-xenial/bedaf26//doc/build/html/examples.html#roles and tell me what you think? | 00:20 |
|---|---|---|
| jeblair | pabelanger: it's still not possible to make a section heading inside of a role documentation section, but that shows what it would look like if we started using the zuul:var directive for role/job variables | 00:21 |
| jeblair | pabelanger: i made "Role Variables" bold as a sort of pseudo-section-header. | 00:22 |
| pabelanger | jeblair: okay, cool. tomorrow okay? | 00:23 |
| pabelanger | might also try it out | 00:23 |
| *** harlowja has quit IRC | 00:25 | |
| jeblair | pabelanger: oh of course! :) | 00:41 |
| *** fbouliane has quit IRC | 02:17 | |
| *** rbergeron has quit IRC | 02:18 | |
| *** rbergeron has joined #zuul | 02:19 | |
| *** fbouliane has joined #zuul | 02:23 | |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Update the zuul-sphinx extension config https://review.openstack.org/491134 | 03:44 |
| *** harlowja has joined #zuul | 04:35 | |
| *** harlowja has quit IRC | 05:14 | |
| tobiash | \o/ rebased my deployment and it uncovered a spelling error in my project config | 05:44 |
| *** bhavik1 has joined #zuul | 06:05 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool feature/zuulv3: Fix detail headers order for nodepool list https://review.openstack.org/491678 | 06:25 |
| tobiash | Shrews: important bugfix for 'nodepool list --detail'... ;) ^^^ | 06:26 |
| *** bhavik1 has quit IRC | 06:53 | |
| *** amoralej|off is now known as amoralej | 07:36 | |
| *** electrofelix has joined #zuul | 08:46 | |
| *** jkilpatr has joined #zuul | 10:58 | |
| *** amoralej is now known as amoralej|lunch | 11:22 | |
| *** amoralej|lunch is now known as amoralej | 12:49 | |
| *** dkranz_ has joined #zuul | 13:21 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Add migration tool for v2 to v3 conversion https://review.openstack.org/491805 | 13:30 |
| mordred | jeblair, pabelanger (and other folks) ^^ very beginnings of a migration tool | 13:31 |
| mordred | I have learned some things about some topics that are going to be difficult that we should discuss | 13:31 |
| mordred | specifically - project-templates make this hard - there is at least one template that is check: gate-{name}-ubuntu-xenial gate: gate-{name}-ubuntu-xenial | 13:33 |
| mordred | if we expand templates as part of migration, that's not a problem - we can generate a gate-toci-glance-ubuntu-xenial job and add it to the right pipelines of the project and do re-consolidation as a followup | 13:34 |
| mordred | however, if we keep the project-templates, we don't have a great way to deal with situations like that. I mean - we can just add entries for them to the mapping file - but that's going to reuqire MUCH more careful examination by us of the output | 13:39 |
| mordred | hrm. I should say all of that in the infra channel actually | 13:39 |
| *** xinliang has quit IRC | 14:01 | |
| *** xinliang has joined #zuul | 14:13 | |
| jeblair | mordred: i'm having trouble understanding why that template is problematic -- maybe i don't have enough background on the approach you're taking | 14:28 |
| mordred | jeblair: I have sent an email with more words | 14:57 |
| mordred | jeblair: (sorry - I had written the email about an hour ago but didn't manage to hit send) | 14:57 |
| mordred | jeblair: fwiw, I'm leaning towards option 3 in the email the more I poke at it | 14:58 |
| *** dmsimard is now known as dmsimard|afk | 15:00 | |
| *** dmsimard|afk is now known as dmsimard | 16:43 | |
| pabelanger | mordred: mind a review of https://review.openstack.org/490643, simple playbook to role convert. | 16:46 |
| pabelanger | mordred: jeblair: https://review.openstack.org/490576 is also ready to be painted. Our 2 -tarball / -branch-tarball publishing jobs | 16:47 |
| pabelanger | mind loop when into the migration tool above | 16:47 |
| *** electrofelix has quit IRC | 17:24 | |
| *** amoralej is now known as amoralej|off | 18:00 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Add comments about base jobs https://review.openstack.org/491897 | 19:45 |
| jeblair | mordred: have a sec to +3 491610 ? when it lands, i'll start work on rejiggering our base jobs for it | 19:46 |
| mordred | jeblair: done | 19:47 |
| jeblair | mordred: er 491610 is the one that needs the +W :) | 19:58 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Add migration tool for v2 to v3 conversion https://review.openstack.org/491805 | 20:03 |
| mordred | jeblair: whoops :) | 20:04 |
| mordred | well - +3 on the other one too | 20:04 |
| mordred | jeblair, pabelanger: so - for ad-hoc / one-off / simple jobs - 'src/{{ zuul.executor.canonical_name }}' is a bit of a mouthfull | 20:14 |
| mordred | also - I can't even get it right when I type it here, since that's not the variable name | 20:15 |
| mordred | https://review.openstack.org/#/c/491805/2/playbooks/zuul-migrate.yaml for example | 20:15 |
| pabelanger | mordred: ya, it results in a lot of copypasta of zuul_work_dir to playbooks and role vars | 20:16 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: Remove base job https://review.openstack.org/491907 | 20:16 |
| *** jkilpatr has quit IRC | 20:16 | |
| mordred | what if we added a zuul.project.src_dir ? it's not a ton less characters, but it's maybe a little clearer as to what it contains? | 20:17 |
| pabelanger | we could also add it to our base job | 20:19 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Require a base job https://review.openstack.org/491610 | 20:20 |
| jeblair | mordred, pabelanger: both of those sound viable | 20:22 |
| jeblair | i guess the question is, if you put it on the base job, what do the vars sections of the other playbooks/jobs look like? what do they have for a default for zuul_work_dir | 20:23 |
| pabelanger | Agree, no preference. zuul.project.src_dir might be nicer, if we consider adding it to zuul.items | 20:23 |
| mordred | lemme make a quick patch and see what y'all think | 20:24 |
| jeblair | pabelanger: yes, all the "project-like" objects in the variables should stay in sync | 20:24 |
| jeblair | mordred: ^^ | 20:24 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Add zuul.project.src_dir variable https://review.openstack.org/491915 | 20:33 |
| mordred | jeblair, pabelanger: ^^ | 20:33 |
| pabelanger | mordred: left -1 about adding ansible assert, but looks good | 20:36 |
| *** jkilpatr has joined #zuul | 20:37 | |
| jeblair | pabelanger: why does that variable need an assertion? | 20:39 |
| jeblair | pabelanger: do we want to add all of them? | 20:39 |
| jeblair | pabelanger: also you didn't actually leave a -1 | 20:39 |
| pabelanger | jeblair: most of them should be asserted, but I don't think we have all yet. But figured it should be helpful to test that we are actually setting up the variables for ansible properly | 20:40 |
| pabelanger | Ya, didn't actually -1, should have said +0 :) | 20:41 |
| jeblair | pabelanger: you sent a mixed signal, for sure. :) | 20:41 |
| pabelanger | jeblair: Ya, didn't want to block it, so if we don't want to, thats okay. Will add +2 then | 20:42 |
| mordred | I'm happy to add an assert though ... | 20:42 |
| jeblair | mordred: i -1'd for a whitespace nit of all things. sorry. | 20:43 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Add zuul.project.src_dir variable https://review.openstack.org/491915 | 20:43 |
| mordred | ah | 20:43 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Add zuul.project.src_dir variable https://review.openstack.org/491915 | 20:44 |
| mordred | jeblair: fixed. thanks | 20:44 |
| jeblair | lgtm | 20:44 |
| pabelanger | +3 | 20:45 |
| pabelanger | thanks! | 20:45 |
| jeblair | pabelanger, mordred: can you +3 491906? i expect that to break currently running v3 but be required for the next restart | 20:45 |
| jeblair | so i'd like to land it now and do a restart | 20:45 |
| pabelanger | looking | 20:46 |
| mordred | jeblair: wfm | 20:47 |
| pabelanger | +3 | 20:48 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Add zuul.project.src_dir variable https://review.openstack.org/491915 | 20:49 |
| mordred | GAH STAB STAB | 20:49 |
| mordred | sorry | 20:49 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Add zuul.project.src_dir variable https://review.openstack.org/491915 | 20:50 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Add migration tool for v2 to v3 conversion https://review.openstack.org/491805 | 20:50 |
| * mordred had a local rebase problem | 20:50 | |
| jeblair | mordred: neat; the gerrit votes are all intact on 915 | 20:51 |
| pabelanger | 490643 could use a +3, should be straighforward | 20:52 |
| mordred | pabelanger: ++ | 20:57 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Add migration tool for v2 to v3 conversion https://review.openstack.org/491805 | 20:58 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Create fetch-tox-output role https://review.openstack.org/490643 | 20:59 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Add migration tool for v2 to v3 conversion https://review.openstack.org/491805 | 21:01 |
| mordred | sweet. that one should work and you should be able to see the output | 21:04 |
| mordred | jeblair, pabelanger: that has project-templates project pipelines, depedencies and adding variants for node suffixes - next up is actually mapping those up with the jjb jobs themselves, then dealing with the regex/variant mapping, then emitting job content | 21:07 |
| mordred | at which point it should be far enough along that we can tweak it as we find corner cases while working on jobs | 21:07 |
| jeblair | mordred: cool. we'll probably have to wait until the restart to see that run | 21:08 |
| jeblair | apparently puppetmaster ansible is a couple hours behind | 21:09 |
| mordred | jeblair: question ... I defined zuul-migrate job in openstack-infra/zuul with required-projects: project-config ... | 21:09 |
| mordred | jeblair: if I add it to the check pipeline for project-config patches in the project-config repo (so that we see it run when changes are made to the mapping.yaml) | 21:09 |
| mordred | jeblair: I assume it'll do the right thing and get both openstack-infra/zuul and openstack-infra/project-config yeah? | 21:10 |
| pabelanger | mordred: nice | 21:10 |
| jeblair | mordred: it will need zuul listed as a required project | 21:10 |
| jeblair | mordred: (consider: it's not testing zuul, it's using zuul in the job) | 21:11 |
| jeblair | (when used that way) | 21:11 |
| mordred | jeblair: ok. so adding zuul to the required projects in the zuul repo is kosher | 21:12 |
| jeblair | yep | 21:12 |
| jeblair | i think that's the only thing to change you'll need for that to work | 21:12 |
| jeblair | words order wrong, but idea get you | 21:12 |
| mordred | yah | 21:12 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Add migration tool for v2 to v3 conversion https://review.openstack.org/491805 | 21:13 |
| jeblair | okay, zuul has been upgraded, i'm going to restart it now | 21:13 |
| mordred | cool | 21:14 |
| mordred | I found one more thing I need to change | 21:15 |
| jeblair | okay, restarted and loading the ansible config :) | 21:16 |
| pabelanger | thanks! | 21:16 |
| *** dkranz_ has quit IRC | 21:16 | |
| jeblair | something doesn't seem right with status.json | 21:19 |
| pabelanger | agree | 21:19 |
| jeblair | ah | 21:20 |
| jeblair | 2017-08-08 21:16:14,427 ERROR zuul.Scheduler: Error starting Zuul: | 21:20 |
| jeblair | zuul.configloader.ConfigurationSyntaxError: Zuul encountered a syntax error while parsing its configuration in the | 21:21 |
| jeblair | repo openstack-infra/project-config on branch master. The error was: | 21:21 |
| jeblair | expected a list for dictionary value @ data['success']['mysql'] | 21:21 |
| jeblair | we need to update the mysql reporter syntax too | 21:21 |
| mordred | jeblair: I though there was a patch for thatup already | 21:24 |
| jeblair | mordred: i don't see a conflicts-with for it :/ | 21:25 |
| mordred | jeblair: maybe I was just remember the docs update patch - your patch is +3 | 21:25 |
| mordred | jeblair, pabelanger: fwiw - I just did a quick double-check and we do have data in our mysql :) | 21:27 |
| mordred | zuul_buildset: 272 rows in set (0.00 sec) zuul_build: 1030 rows in set (0.01 sec) | 21:28 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Add comments about base jobs https://review.openstack.org/491897 | 21:29 |
| jeblair | mordred: \o/ | 21:30 |
| jeblair | restarting again | 21:50 |
| jeblair | and up | 21:50 |
| jeblair | seems to be working | 21:51 |
| jeblair | i rechecked mordred's change and it's running now | 21:51 |
| mordred | \o/ | 21:52 |
| jeblair | http://zuulv3.openstack.org/static/stream.html?uuid=be4258d38f9d4d1fb4c7b4f725c4ea0e&logfile=console.log | 21:52 |
| mordred | jeblair: there's a bug I have fixed locally - want me to wait til that fails until I push it up? | 21:52 |
| jeblair | mordred: nah go for it | 21:53 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Add migration tool for v2 to v3 conversion https://review.openstack.org/491805 | 21:53 |
| mordred | http://zuulv3.openstack.org/static/stream.html?uuid=37f99614f19b452295adf122caf97e6a&logfile=console.log | 21:54 |
| mordred | jeblair: watching repos copy is not exciting | 21:58 |
| jeblair | mordred: seems to be taking too long | 22:02 |
| jeblair | mordred: like, i'm not sure we're seeing anything past that point | 22:03 |
| jeblair | could there be a behavior change? | 22:03 |
| jeblair | i ran finger too to verify they're stuck at the same spot, so it's not a web thing | 22:04 |
| mordred | jeblair: I agree with you - and also am not sure what's up | 22:05 |
| jeblair | i checked a work dir, job-output.txt is also stuck there, so not a finger thing | 22:05 |
| mordred | yup. I concur | 22:05 |
| jeblair | it looks like the base pre playbook is still running | 22:06 |
| mordred | jeblair: zuul 1061 1 0 21:54 ? 00:00:00 /usr/bin/python2 /tmp/ansible_9kPUu6/ansible_module_zuul_console.py is the only thing running on the node | 22:06 |
| mordred | jeblair: did we maybe only get half of the streaming fix from tobiash deployed? | 22:07 |
| mordred | so there's maybe a handshake going wrong? | 22:07 |
| jeblair | i'm stracing an rsync process which seems to be doing things but slowly | 22:07 |
| mordred | oh. ok | 22:08 |
| mordred | jeblair: oh - you know what | 22:09 |
| mordred | jeblair: our base job doing the repo rsync is ... suboptimal | 22:09 |
| mordred | jeblair: I don't think it's making use of the repos in /opt/git | 22:09 |
| jeblair | mordred: it's not supposed to; we're trying to drop those in v3 | 22:10 |
| mordred | oh - ok. nevermind then :) | 22:10 |
| jeblair | (so we can have smaller images) | 22:11 |
| mordred | yah | 22:11 |
| jeblair | but it is looking like right now the rsync is very very slow | 22:11 |
| jeblair | i don't think it was in the past | 22:11 |
| mordred | maybe infracloud is laggy atm? | 22:12 |
| jeblair | maybe... clarkb and fungi were onlining some hosts there earlier; maybe we should check what compute hosts our nodes landed on | 22:16 |
| jeblair | also, we are now running the disk space watcher thing, but it seems to be be fairly low impact. | 22:17 |
| fungi | vanilla or chocolate? | 22:18 |
| jeblair | looking now | 22:18 |
| fungi | clarkb: did chocolate a few hours ago, i'm poking at vanilla right now | 22:18 |
| jeblair | oh neat, nodepool list doesn't show the ip any more | 22:18 |
| jeblair | we need to add that back :) | 22:19 |
| fungi | i managed to bring one defunct compute node back online in vanilla by rebooting it through ironic | 22:19 |
| jeblair | i need to map 15.184.70.243 to a node | 22:19 |
| fungi | now i'm waiting a bit to see what vm uuids stick around past reasonable job durations | 22:19 |
| jeblair | 2017-08-08 21:54:27,577 DEBUG nodepool.NodeLauncher-0000017006: Node 0000017006 is running [region: RegionOne, az: nova, ip: 15.184.70.243 ipv4: 15.184.70.243, ipv6: ] | 22:19 |
| jeblair | | 0000017006 | infracloud-chocolate | nova | ubuntu-xenial | 2a5f60f1-3f05-4430-9157-2cc038cd7c48 | in-use | 00:00:24:02 | locked | | 22:20 |
| jeblair | there we go, chocolate | 22:20 |
| fungi | and well after clarkb had finished doing things in there | 22:20 |
| clarkb | http://paste.openstack.org/show/617816/ is the list of what I cleaned up | 22:20 |
| clarkb | 2af is not in that list | 22:20 |
| jeblair | let me see if i can find a vanilla node that's slow | 22:20 |
| clarkb | also I cross checked against nova list so should only have gotten nodes nova didn't know about | 22:21 |
| jeblair | | 0000017001 | infracloud-vanilla | nova | ubuntu-xenial | 27f44b59-1448-4a38-bc8a-c95a1667e938 | in-use | 00:00:24:32 | locked | | 22:22 |
| jeblair | that's similarly slow | 22:22 |
| jeblair | so it's both regions | 22:22 |
| clarkb | oh things are slow | 22:22 |
| jeblair | (to be clear, no idea if this is an infracloud or zuul problem) | 22:22 |
| clarkb | we are running at capacity right now and have been all day | 22:23 |
| clarkb | we could just be oversubscribed? | 22:23 |
| clarkb | I would definitely check the hypervisor(s) you are running on and get some performance data there | 22:23 |
| jeblair | clarkb: could be, and of course we have no visibility on our network, so we could be maxing that out | 22:23 |
| jeblair | clarkb: rysincing the git repos from ze01 to the nodes was unusually slow. so either slow disk or network bottleneck on ze01 (unlikely based on cacti graphs) or infracloud. | 22:24 |
| clarkb | I think we know the disk is slow there which is why we have turned on the "we don't care about our data" flag in kvm | 22:25 |
| clarkb | but ya could be network too | 22:25 |
| mordred | jeblair, clarkb: 2017-08-08 21:54:59.848771 | TASK [prepare-workspace : Synchronize src repos to workspace directory.] | 22:25 |
| mordred | 2017-08-08 22:18:32.777985 | ubuntu-xenial | Output suppressed because no_log was given | 22:25 |
| jeblair | it took 24minutes to sync them | 22:25 |
| mordred | yah | 22:25 |
| mordred | and that's just doing zuul and project-config | 22:25 |
| clarkb | you can test disk io fairly easily thlough so maybe start with that | 22:26 |
| clarkb | network is harder without insight to the networking but should be able to draw reasonable conclusions comparing hypervisor to hypervisor against internet to hypervisor (maybe to a ramdisk to avoid slow disk as well) | 22:26 |
| jeblair | a total of about 2.5G of data | 22:26 |
| jeblair | mordred, pabelanger: what do you make of http://logs.openstack.org/05/491805/7/check/tox-py35/07fad41/job-output.txt.gz#_2017-08-08_22_31_10_966898 ? | 22:39 |
| jeblair | mordred: that looks like maybe an error in our local plugin? | 22:39 |
| mordred | jeblair: yes. that looks like that - and it looks like an error in the recent fix we landed - I'm looking at that right now | 22:40 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Don't pass self to a bound method https://review.openstack.org/491946 | 22:43 |
| mordred | jeblair: I'm concerned that that got through testing | 22:43 |
| mordred | jeblair: but I'll put that on the list to sort out as a follow up | 22:43 |
| Shrews | jeblair: nodepool list --detail | 23:23 |
| Shrews | (to get IP and other info) | 23:24 |
| jeblair | Shrews: thx | 23:25 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Don't pass self to a bound method https://review.openstack.org/491946 | 23:30 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!