| *** thorst_afk has joined #openstack-infra | 00:00 | |
| fungi | with 'From .*' initial line, headers for From:, Date:, Subject: | 00:00 |
|---|---|---|
| fungi | show instead gives you commit id for the initial line, with a Date: and Author: header that align the data fields with one another, and indenting the commit message | 00:01 |
| fungi | most of the differences are in the way the commit message and headers are represented rather than the diff itself | 00:02 |
| *** rlandy is now known as rlandy|afk | 00:02 | |
| fungi | i don't think git am actually cares about that version string after the signature separator | 00:03 |
| *** dingyichen has joined #openstack-infra | 00:04 | |
| *** harlowja has quit IRC | 00:04 | |
| fungi | mtreinish: i'm getting lots of those panics from the subscribers actually. seeing if i can track down why | 00:05 |
| jeblair | clarkb: git-am on that worked! thanks | 00:07 |
| ianw | root@ubuntu-xenial-osic-cloud1-s3500-8688880:~# ls | 00:07 |
| ianw | swapfile | 00:07 |
| ianw | is that supposed to be there? | 00:07 |
| *** lamt has joined #openstack-infra | 00:08 | |
| clarkb | ianw: if d-g ran there yes | 00:08 |
| clarkb | ianw: we make a swapfile if no swapfile device and no ephemeral drive to repartition | 00:08 |
| ianw | but in /root? | 00:08 |
| ianw | maybe it was always there and i just never noticed | 00:09 |
| jeblair | clarkb: can you point me at rst docs for '|' ? | 00:09 |
| fungi | jeblair: http://docutils.sourceforge.net/docs/user/rst/quickref.html#substitution-references-and-definitions | 00:10 |
| fungi | i happened to have it up in a tab from a few minutes ago | 00:11 |
| clarkb | jeblair: http://docutils.sourceforge.net/docs/ref/rst/restructuredtext.html#line-blocks | 00:11 |
| jeblair | fungi: different use of '|' i think | 00:11 |
| fungi | oh, the _other_ use for | | 00:11 |
| fungi | yep, also useful ;) | 00:12 |
| fungi | mtreinish: aha! i think i must be misunderstanding how mqtt-bench is supposed to work... if the clients number on the subscriber process exceeds the clients number on the publisher process | 00:12 |
| *** zz_dimtruck is now known as dimtruck | 00:13 | |
| fungi | mtreinish: so you can't use this to test fewer subscribers than publishers. i guess our graph will be triangular? | 00:13 |
| *** dave-mccowan has quit IRC | 00:13 | |
| fungi | mtreinish: that's what seems to have been causing the engrish panics | 00:14 |
| fungi | no, wait, i'm getting it on mismatches the other way too | 00:16 |
| *** lamt has quit IRC | 00:16 | |
| *** lamt has joined #openstack-infra | 00:16 | |
| *** jamesmca_ has joined #openstack-infra | 00:17 | |
| *** mat128 has joined #openstack-infra | 00:18 | |
| *** sbezverk_ has joined #openstack-infra | 00:19 | |
| fungi | yeah, this is turning into an exercise in figuring out what the tool can handle rather than benchmarking anything | 00:21 |
| *** ym200 has joined #openstack-infra | 00:21 | |
| *** sbezverk has quit IRC | 00:22 | |
| *** tonytan4ever has joined #openstack-infra | 00:22 | |
| *** jamesmca_ has quit IRC | 00:22 | |
| *** sbezverk has joined #openstack-infra | 00:23 | |
| fungi | gotta be running into some load/timing issue. if i try to set pubs and subs both to -clients=100 i get that panic on the subscriber too | 00:24 |
| fungi | even 50 pubs and 100 subs | 00:25 |
| *** sbezverk_ has quit IRC | 00:26 | |
| *** ym200 has quit IRC | 00:26 | |
| *** felipemonteiro has joined #openstack-infra | 00:26 | |
| ianw | hmm, journalctl -f has quite a lag between output and when it decides to show it to you | 00:28 |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck master: Add query for UnicodeDecodeError with journal log handler bug 1580728 https://review.openstack.org/461939 | 00:29 |
| openstack | bug 1580728 in OpenStack Compute (nova) "UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 386: ordinal not in range(128) in nova.virt.libvirt.vif:unplug with unicode instance.display_name" [Medium,Triaged] https://launchpad.net/bugs/1580728 | 00:29 |
| *** neiljerram has quit IRC | 00:32 | |
| *** tphummel has quit IRC | 00:35 | |
| *** ijw has quit IRC | 00:35 | |
| *** ym200 has joined #openstack-infra | 00:36 | |
| *** sbezverk_ has joined #openstack-infra | 00:36 | |
| *** ijw has joined #openstack-infra | 00:36 | |
| *** ijw has quit IRC | 00:37 | |
| *** ijw has joined #openstack-infra | 00:37 | |
| *** sbezverk has quit IRC | 00:37 | |
| *** thorst_afk has quit IRC | 00:37 | |
| *** dhinesh has quit IRC | 00:39 | |
| openstackgerrit | Merged openstack-infra/elastic-recheck master: Add query for UnicodeDecodeError with journal log handler bug 1580728 https://review.openstack.org/461939 | 00:39 |
| openstack | bug 1580728 in OpenStack Compute (nova) "UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 386: ordinal not in range(128) in nova.virt.libvirt.vif:unplug with unicode instance.display_name" [Medium,Triaged] https://launchpad.net/bugs/1580728 | 00:39 |
| mtreinish | fungi: hmm, I wonder I wonder if we're hitting a ulimit | 00:40 |
| *** ym200 has quit IRC | 00:40 | |
| *** mriedem has quit IRC | 00:40 | |
| mtreinish | I know when I did my local testing before we set up the firehose back in Aug. I was hitting open fd errors with our for loop testing | 00:40 |
| mtreinish | I can't remember how many clients that was though | 00:40 |
| mtreinish | fungi: you can also turn on guaranteed delivery with qos=1 or qos=2 (on both the pub and sub) | 00:41 |
| *** markvoelker has quit IRC | 00:41 | |
| mtreinish | qos=1 is guaranteed delivery, and qos=2 is guaranteed delivery once | 00:41 |
| *** ijw has quit IRC | 00:41 | |
| mtreinish | obviously there is much more overhead with that too | 00:42 |
| *** markvoelker has joined #openstack-infra | 00:42 | |
| mtreinish | fungi: oh it might also be bandwidth: http://cacti.openstack.org/cacti/graph.php?action=zoom&local_graph_id=4952&rra_id=5&view_type=&graph_start=1493768988&graph_end=1493771861&graph_height=120&graph_width=500&title_font_size=10 | 00:42 |
| mtreinish | 400 mbps seems like a lot for the small vm we've got deployed | 00:43 |
| *** thorst_afk has joined #openstack-infra | 00:44 | |
| fungi | oh, you know what, i forgot the default size on this is 1024 bytes per message | 00:45 |
| *** funzo has joined #openstack-infra | 00:46 | |
| *** thorst_afk has quit IRC | 00:46 | |
| *** markvoelker has quit IRC | 00:46 | |
| *** gildub has quit IRC | 00:47 | |
| *** dave-mccowan has joined #openstack-infra | 00:47 | |
| fungi | nope, dropping the -size to 100 and setting -intervaltime=10 on the pub and with 10 pubs i still get a panic at 100 subs | 00:47 |
| *** gildub has joined #openstack-infra | 00:48 | |
| fungi | feels a lot like we need to get just the right balance of timing and volume parameters to get any useful measurements | 00:50 |
| *** funzo has quit IRC | 00:50 | |
| *** cuongnv has joined #openstack-infra | 00:51 | |
| mtreinish | fungi: heh, script it with nested for loops for both params? | 00:51 |
| mtreinish | find the ones taht don't panic on us | 00:51 |
| *** mat128 has quit IRC | 00:51 | |
| fungi | well, there's a separate (related?) problem which is some combinations i'm trying cause the publisher to just wait indefinitely | 00:52 |
| mtreinish | lol | 00:52 |
| fungi | er, the subscriber i mean | 00:52 |
| mtreinish | I really should just rewrite this in python so we can understand it better | 00:53 |
| mtreinish | lets see if I feel motivated enough after another beer | 00:53 |
| fungi | i'm already trying that solution too, it's not really helping either (but at least it makes it more bearable!) | 00:54 |
| mtreinish | fungi: the beer, or writing your own benchmark tool? :) | 00:55 |
| fungi | the first part of the solution hasn't caused me to want to attempt the second | 00:56 |
| *** yamamoto has joined #openstack-infra | 00:57 | |
| openstackgerrit | Emilien Macchi proposed openstack/os-client-config master: DO NOT MERGE - Revert "Remove the keystoneclient auth fallback" https://review.openstack.org/461945 | 00:57 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: DNM - deploy tripleo with all debug https://review.openstack.org/457037 | 00:57 |
| *** Apoorva_ has joined #openstack-infra | 01:02 | |
| ianw | pabelanger: if around ... if I say "LC_ALL" does that ring any recent bells about fiddling with it on images? | 01:03 |
| *** Goneri has joined #openstack-infra | 01:03 | |
| fungi | not recent, but we've had to deal with setting default locales in the past | 01:04 |
| *** yamamoto has quit IRC | 01:05 | |
| ianw | yeah ... i'm wondering if something has change. I *think* LC_ALL=C.UTF-8 now, which centos doesn't have | 01:05 |
| *** vtapia has quit IRC | 01:05 | |
| openstackgerrit | dongwenjuan proposed openstack-infra/puppet-jenkins master: use 'openjdk-8-jre-headless' for ubuntu-16.04 https://review.openstack.org/461708 | 01:05 |
| *** Apoorva has quit IRC | 01:05 | |
| *** Apoorva_ has quit IRC | 01:07 | |
| *** tiswanso has joined #openstack-infra | 01:07 | |
| fungi | mtreinish: confidence instilling this is https://github.com/takanorig/mqtt-bench/issues/9 | 01:08 |
| *** EricGonczer_ has joined #openstack-infra | 01:09 | |
| mtreinish | fungi: heh, that would be likely | 01:10 |
| mtreinish | fungi: I was assuming the tls code from the go stdlib there was correct | 01:10 |
| mtreinish | but I didn't actually check | 01:10 |
| *** vtapia has joined #openstack-infra | 01:11 | |
| mtreinish | fungi: anyway I started writing something similar in python (where I at least understand all the bits involved) | 01:11 |
| mtreinish | we'll see if I can finish it in a reasonable amount of time | 01:11 |
| fungi | thanks, my patience for trying to figure out the sweet spot in these parameters between what causes the sub process to panic or hang indefinitely has worn thin | 01:11 |
| fungi | oh, even better, sometimes it hangs for a while longer than it should, and then throws that panic | 01:12 |
| *** pvaneck has quit IRC | 01:12 | |
| fungi | anyway, i promised to review some other talks in progress for people so need to shift gears for a bit | 01:12 |
| *** xinliang has quit IRC | 01:13 | |
| jeblair | clarkb: in looking into your patch, i found some other things about line_block that we weren't handling; i think http://git.inaugust.com/cgit/presentty/commit/?id=308a06134d7749638c7ba3afcc4031f31ba09930 may be a more complete solution | 01:15 |
| *** xinliang has joined #openstack-infra | 01:16 | |
| jeblair | clarkb: also note that line blocks should have an empty line after them; otherwise the sphinx parser throws a warning. so if you use them just to force vertical space, i think they should look like: http://paste.openstack.org/show/608638/ | 01:18 |
| *** puiterwijk has quit IRC | 01:19 | |
| *** puiterwijk has joined #openstack-infra | 01:20 | |
| *** puiterwijk has quit IRC | 01:21 | |
| clarkb | jeblair: ah ok and get flowed text is doing ehat my change dud ya? | 01:22 |
| *** puiterwijk has joined #openstack-infra | 01:22 | |
| *** ramishra has joined #openstack-infra | 01:25 | |
| *** yanyanhu has joined #openstack-infra | 01:25 | |
| openstackgerrit | K Jonathan Harker proposed openstack-infra/zuul feature/zuulv3: Add github reporter status_url config option https://review.openstack.org/449794 | 01:27 |
| *** yolkfull__ has joined #openstack-infra | 01:29 | |
| jeblair | clarkb: yep, it has a check for [] that returns '' | 01:29 |
| *** felipemonteiro has quit IRC | 01:31 | |
| *** liujiong has joined #openstack-infra | 01:32 | |
| clarkb | cool I'll give it a try in the moning | 01:33 |
| jeblair | clarkb: btw, if you're curious, 'python presentty/rst.py' will show the sphinx structure for an rst file, so you can see what's going on behind the scenes. | 01:35 |
| jeblair | (that's how i saw how the line_block events work) | 01:35 |
| *** mriedem has joined #openstack-infra | 01:36 | |
| clarkb | oh neat | 01:37 |
| clarkb | I was relying in tracebacks and was abojt to pdb but figured it out | 01:37 |
| clarkb | that looks handy | 01:37 |
| *** gouthamr has quit IRC | 01:37 | |
| clarkb | also looks lioe slides are composed by piles and you just construxt them using a visitor pattern? | 01:41 |
| *** nicolasbock has quit IRC | 01:41 | |
| clarkb | and a pile is a set of urwid rows | 01:45 |
| *** felipemonteiro has joined #openstack-infra | 01:45 | |
| *** thorst_afk has joined #openstack-infra | 01:47 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Set LC_ALL in disk-image-create https://review.openstack.org/461950 | 01:49 |
| ianw | fungi / pabelanger: ^ rather unsatisfying but seems to work | 01:49 |
| ianw | centos doesn't have a great way to strip locales, we regenerate the locale-archive. i'm never sure that it's not already mapped, and if that causes confusion | 01:50 |
| *** zhurong has joined #openstack-infra | 01:50 | |
| jeblair | clarkb: yep, that's about it | 01:51 |
| *** aleph1 has quit IRC | 01:51 | |
| *** thorst_afk has quit IRC | 01:52 | |
| *** nicolasbock has joined #openstack-infra | 01:55 | |
| *** yamahata has joined #openstack-infra | 01:58 | |
| *** jamielennox is now known as jamielennox|away | 02:03 | |
| *** mat128 has joined #openstack-infra | 02:04 | |
| *** zhurong has quit IRC | 02:07 | |
| mtreinish | fungi: I was thinking about it while working on my benchmark script I bet the client counts have to be the same for the mqtt-bench | 02:13 |
| mtreinish | err or pubs >= subs | 02:13 |
| mtreinish | because it's probably doing a topic per worker to avoid duplicate message counts | 02:13 |
| mtreinish | (which is something I'm not caring about) | 02:13 |
| *** nicolasbock has quit IRC | 02:17 | |
| *** jamielennox|away is now known as jamielennox | 02:17 | |
| *** rlandy|afk is now known as rlandy | 02:20 | |
| mtreinish | fungi: I think I've got the script written | 02:22 |
| mtreinish | fungi: let me just setup the repo and I'll push it up | 02:22 |
| *** tonytan4ever has quit IRC | 02:22 | |
| *** ihrachys has quit IRC | 02:29 | |
| mtreinish | fungi: https://github.com/mtreinish/pymqttbench | 02:30 |
| mtreinish | there seems to be a bug where it hangs waiting on timing info with large client counts where it gets stuck waiting on some of the threads | 02:31 |
| mtreinish | not to finish, but it's results from the queue I'm not sure what's causing it | 02:31 |
| mtreinish | but for small counts it seems to work fine | 02:31 |
| mtreinish | but it's definitely not perfect | 02:32 |
| mtreinish | but hopefully better than the go one :) | 02:32 |
| *** nicolasbock has joined #openstack-infra | 02:33 | |
| *** ijw has joined #openstack-infra | 02:33 | |
| mtreinish | fungi: actually just found the bug, just sloppy on my part :) | 02:33 |
| *** Goneri has quit IRC | 02:36 | |
| *** dave-mccowan has quit IRC | 02:36 | |
| *** ijw has quit IRC | 02:38 | |
| mtreinish | fungi: we can add whatever stats and output formats you think will be useful pretty easily on this | 02:38 |
| openstackgerrit | Felipe Monteiro proposed openstack-infra/project-config master: Remove the patrole heat job https://review.openstack.org/461962 | 02:38 |
| *** yamahata has quit IRC | 02:40 | |
| mtreinish | fungi: if you're curious about the results so far, this is what it does: http://paste.openstack.org/show/608657/ | 02:46 |
| *** funzo has joined #openstack-infra | 02:46 | |
| openstackgerrit | Rabi Mishra proposed openstack-infra/project-config master: Change identity-v3-only heat nv job https://review.openstack.org/461968 | 02:47 |
| *** armax has quit IRC | 02:48 | |
| *** thorst_afk has joined #openstack-infra | 02:48 | |
| openstackgerrit | Felipe Monteiro proposed openstack-infra/project-config master: Remove the patrole heat job https://review.openstack.org/461962 | 02:49 |
| *** psachin has joined #openstack-infra | 02:51 | |
| *** funzo has quit IRC | 02:51 | |
| openstackgerrit | Hongbin Lu proposed openstack-infra/project-config master: [Kuryr] Use gate hook in rally jobs https://review.openstack.org/461972 | 02:55 |
| *** hongbin has quit IRC | 02:59 | |
| *** gcb has quit IRC | 02:59 | |
| cloudnull | :q | 03:01 |
| *** gildub has quit IRC | 03:02 | |
| *** armax has joined #openstack-infra | 03:05 | |
| openstackgerrit | Rabi Mishra proposed openstack-infra/project-config master: Change identity-v3-only heat nv job https://review.openstack.org/461968 | 03:06 |
| *** thorst_afk has quit IRC | 03:08 | |
| *** nicolasbock has quit IRC | 03:10 | |
| *** gcb has joined #openstack-infra | 03:13 | |
| *** EricGonczer_ has quit IRC | 03:18 | |
| *** armax has quit IRC | 03:19 | |
| *** winggundamth has joined #openstack-infra | 03:22 | |
| *** felipemonteiro has quit IRC | 03:24 | |
| *** mriedem has quit IRC | 03:25 | |
| *** tphummel has joined #openstack-infra | 03:33 | |
| *** armax has joined #openstack-infra | 03:35 | |
| *** zhurong has joined #openstack-infra | 03:37 | |
| *** ijw has joined #openstack-infra | 03:38 | |
| *** vikrant has joined #openstack-infra | 03:39 | |
| *** zhurong has quit IRC | 03:40 | |
| *** knangia has quit IRC | 03:41 | |
| *** tiswanso has quit IRC | 03:42 | |
| *** ijw has quit IRC | 03:42 | |
| *** masber has quit IRC | 03:46 | |
| *** links has joined #openstack-infra | 03:49 | |
| *** v3i0 has joined #openstack-infra | 03:54 | |
| *** v3i0 has left #openstack-infra | 03:54 | |
| openstackgerrit | Jamie Lennox proposed openstack-infra/zuul feature/zuulv3: Only report to gerrit if the action is from gerrit https://review.openstack.org/461981 | 03:57 |
| *** markvoelker has joined #openstack-infra | 04:00 | |
| *** aeng has quit IRC | 04:03 | |
| *** ym200 has joined #openstack-infra | 04:04 | |
| *** thorst_afk has joined #openstack-infra | 04:04 | |
| *** ym200 has quit IRC | 04:08 | |
| *** thorst_afk has quit IRC | 04:09 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Set LC_ALL in disk-image-create https://review.openstack.org/461950 | 04:16 |
| openstackgerrit | Jamie Lennox proposed openstack-infra/zuul feature/zuulv3: Only report to gerrit if the action is from gerrit https://review.openstack.org/461981 | 04:16 |
| *** salv-orlando has joined #openstack-infra | 04:18 | |
| *** aeng has joined #openstack-infra | 04:20 | |
| *** salv-orlando has quit IRC | 04:23 | |
| *** gyee has quit IRC | 04:23 | |
| *** chlong has joined #openstack-infra | 04:23 | |
| *** armax has quit IRC | 04:27 | |
| *** armax has joined #openstack-infra | 04:28 | |
| *** nmathew has joined #openstack-infra | 04:30 | |
| *** jamesmca_ has joined #openstack-infra | 04:31 | |
| *** krtaylor has joined #openstack-infra | 04:32 | |
| *** jamesmca_ has quit IRC | 04:35 | |
| *** ykarel has joined #openstack-infra | 04:36 | |
| *** dingyichen has quit IRC | 04:36 | |
| *** zhurong has joined #openstack-infra | 04:37 | |
| *** armax has quit IRC | 04:38 | |
| *** dingyichen has joined #openstack-infra | 04:38 | |
| *** yamahata has joined #openstack-infra | 04:41 | |
| *** gildub has joined #openstack-infra | 04:41 | |
| *** namnh has joined #openstack-infra | 04:47 | |
| *** jascott1 has quit IRC | 04:48 | |
| *** salv-orlando has joined #openstack-infra | 04:49 | |
| *** yamahata has quit IRC | 04:55 | |
| *** huanxie has quit IRC | 05:02 | |
| *** huanxie has joined #openstack-infra | 05:03 | |
| *** huanxie has quit IRC | 05:04 | |
| *** huanxie has joined #openstack-infra | 05:05 | |
| *** mat128 has quit IRC | 05:05 | |
| *** pgadiya has joined #openstack-infra | 05:05 | |
| *** aeng has quit IRC | 05:08 | |
| *** gcb has quit IRC | 05:10 | |
| *** Julien-zte has joined #openstack-infra | 05:14 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: TEST: DONT RECHECK or REVIEW: periodic jobs https://review.openstack.org/359215 | 05:18 |
| *** dhajare has joined #openstack-infra | 05:19 | |
| *** erikmwil_ has joined #openstack-infra | 05:20 | |
| *** aeng has joined #openstack-infra | 05:21 | |
| *** ccamacho has left #openstack-infra | 05:22 | |
| *** erikmwilson has quit IRC | 05:23 | |
| *** sree has joined #openstack-infra | 05:25 | |
| *** sree has quit IRC | 05:26 | |
| *** sree has joined #openstack-infra | 05:27 | |
| *** gcb has joined #openstack-infra | 05:27 | |
| openstackgerrit | Andreas Florath proposed openstack/diskimage-builder master: Add refactor of tree-like vs graph https://review.openstack.org/461353 | 05:27 |
| *** cuongnv has quit IRC | 05:30 | |
| *** namnh_ has joined #openstack-infra | 05:32 | |
| *** kjackal_ has joined #openstack-infra | 05:32 | |
| *** Hal has joined #openstack-infra | 05:34 | |
| *** namnh has quit IRC | 05:35 | |
| *** bhavik1 has joined #openstack-infra | 05:37 | |
| *** cuongnv has joined #openstack-infra | 05:38 | |
| *** gouthamr has joined #openstack-infra | 05:44 | |
| *** winggundamth has quit IRC | 05:45 | |
| *** salv-orl_ has joined #openstack-infra | 05:46 | |
| *** ltomasbo|away is now known as ltomasbo | 05:47 | |
| *** gouthamr has quit IRC | 05:48 | |
| *** salv-orlando has quit IRC | 05:49 | |
| *** salv-orl_ has quit IRC | 05:50 | |
| *** masber has joined #openstack-infra | 05:58 | |
| openstackgerrit | Merged openstack-infra/project-config master: Disable EPEL after running Bindep for OSA jobs https://review.openstack.org/461851 | 06:03 |
| *** thorst_afk has joined #openstack-infra | 06:06 | |
| *** Hal has quit IRC | 06:06 | |
| *** prateek has joined #openstack-infra | 06:06 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci master: DO NOT MERGE: Testing LDAP integration https://review.openstack.org/461865 | 06:11 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci master: Deploy FreeIPA-backed LDAP domain for keystone in fakeha-caserver job https://review.openstack.org/461864 | 06:11 |
| *** thorst_afk has quit IRC | 06:11 | |
| *** andreas-f has quit IRC | 06:12 | |
| *** andreas_s has joined #openstack-infra | 06:13 | |
| *** markus_z has joined #openstack-infra | 06:19 | |
| *** jaosorior_away is now known as jaosorior | 06:19 | |
| *** yolanda has quit IRC | 06:20 | |
| *** larainema has quit IRC | 06:20 | |
| *** gongysh has joined #openstack-infra | 06:21 | |
| *** tphummel has quit IRC | 06:21 | |
| *** rwsu has quit IRC | 06:23 | |
| *** pgadiya has quit IRC | 06:25 | |
| *** pcaruana has joined #openstack-infra | 06:27 | |
| *** andreas-f has joined #openstack-infra | 06:34 | |
| *** rwsu has joined #openstack-infra | 06:36 | |
| *** tommylikehu has joined #openstack-infra | 06:44 | |
| *** tie has joined #openstack-infra | 06:45 | |
| *** florianf has joined #openstack-infra | 06:47 | |
| *** funzo has joined #openstack-infra | 06:48 | |
| *** jascott1 has joined #openstack-infra | 06:50 | |
| *** funzo has quit IRC | 06:52 | |
| *** jascott1 has quit IRC | 06:54 | |
| *** eranrom has joined #openstack-infra | 06:56 | |
| *** shardy has joined #openstack-infra | 07:00 | |
| *** martinkopec has joined #openstack-infra | 07:01 | |
| *** kjackal_ has quit IRC | 07:04 | |
| *** ccamacho has joined #openstack-infra | 07:04 | |
| openstackgerrit | Slawek Kaplonski proposed openstack-infra/shade master: Don't get ports info from unavailable neutron service https://review.openstack.org/461903 | 07:04 |
| *** aarefiev has joined #openstack-infra | 07:06 | |
| *** thorst_afk has joined #openstack-infra | 07:06 | |
| *** ociuhandu has quit IRC | 07:08 | |
| *** dizquierdo has joined #openstack-infra | 07:10 | |
| *** namnh has joined #openstack-infra | 07:10 | |
| *** thorst_afk has quit IRC | 07:11 | |
| *** bauwser is now known as bauzas | 07:12 | |
| *** namnh_ has quit IRC | 07:12 | |
| *** Hal has joined #openstack-infra | 07:16 | |
| *** dizquierdo has quit IRC | 07:17 | |
| *** dizquierdo has joined #openstack-infra | 07:17 | |
| openstackgerrit | Oksana Voshchana proposed openstack-infra/devstack-gate master: WIP: Test SSL wsgi https://review.openstack.org/414986 | 07:18 |
| *** aviau has quit IRC | 07:19 | |
| *** aviau has joined #openstack-infra | 07:19 | |
| *** Hal has quit IRC | 07:23 | |
| *** zhurong has quit IRC | 07:24 | |
| *** Hal has joined #openstack-infra | 07:24 | |
| *** ijw has joined #openstack-infra | 07:27 | |
| *** hashar has joined #openstack-infra | 07:28 | |
| *** psachin has quit IRC | 07:29 | |
| *** ssbarnea has joined #openstack-infra | 07:32 | |
| *** ijw has quit IRC | 07:32 | |
| *** jpena|off is now known as jpena | 07:33 | |
| *** ssbarnea has quit IRC | 07:34 | |
| *** ssbarnea has joined #openstack-infra | 07:35 | |
| *** Qiming has quit IRC | 07:36 | |
| *** gaurangt has joined #openstack-infra | 07:38 | |
| *** salv-orlando has joined #openstack-infra | 07:38 | |
| *** dtantsur|afk is now known as dtantsur | 07:40 | |
| *** Qiming has joined #openstack-infra | 07:42 | |
| *** zhurong has joined #openstack-infra | 07:43 | |
| *** adriant has quit IRC | 07:43 | |
| *** salv-orlando has quit IRC | 07:46 | |
| *** gouthamr has joined #openstack-infra | 07:51 | |
| *** oanson has quit IRC | 07:52 | |
| *** oanson has joined #openstack-infra | 07:53 | |
| *** gongysh has quit IRC | 07:55 | |
| *** dizquierdo has quit IRC | 07:56 | |
| *** gouthamr has quit IRC | 07:56 | |
| *** ralonsoh has joined #openstack-infra | 07:56 | |
| *** masber has quit IRC | 07:56 | |
| *** masber has joined #openstack-infra | 07:59 | |
| *** zzzeek has quit IRC | 08:00 | |
| *** zzzeek has joined #openstack-infra | 08:00 | |
| *** rossella_s has joined #openstack-infra | 08:01 | |
| *** neiljerram has joined #openstack-infra | 08:04 | |
| *** ijw has joined #openstack-infra | 08:05 | |
| *** salv-orlando has joined #openstack-infra | 08:05 | |
| *** thorst_afk has joined #openstack-infra | 08:07 | |
| *** ijw has quit IRC | 08:09 | |
| *** alexchadin has joined #openstack-infra | 08:09 | |
| *** lucas-afk is now known as lucasagomes | 08:26 | |
| *** e0ne has joined #openstack-infra | 08:26 | |
| *** thorst_afk has quit IRC | 08:26 | |
| *** electrofelix has joined #openstack-infra | 08:28 | |
| openstackgerrit | Yuval Brik proposed openstack-infra/project-config master: karbor: mark python-karborclient to clone from git https://review.openstack.org/462031 | 08:30 |
| *** gildub has quit IRC | 08:32 | |
| *** markvoelker has quit IRC | 08:32 | |
| *** jamesmca_ has joined #openstack-infra | 08:32 | |
| *** ggnel_t has quit IRC | 08:34 | |
| *** dingyichen has quit IRC | 08:35 | |
| *** sshnaidm|pto is now known as sshnaidm | 08:36 | |
| *** jamesmca_ has quit IRC | 08:37 | |
| *** mikal has quit IRC | 08:39 | |
| *** derekh has joined #openstack-infra | 08:39 | |
| *** mikal has joined #openstack-infra | 08:41 | |
| *** matrohon has joined #openstack-infra | 08:44 | |
| *** zhurong has quit IRC | 08:47 | |
| *** isaacb has joined #openstack-infra | 08:48 | |
| *** jascott1 has joined #openstack-infra | 08:51 | |
| openstackgerrit | Thomas Herve proposed openstack-infra/project-config master: Make Heat AMQP job voting https://review.openstack.org/462041 | 08:54 |
| *** jascott1 has quit IRC | 08:56 | |
| *** salv-orlando has quit IRC | 08:56 | |
| *** namnh_ has joined #openstack-infra | 08:56 | |
| *** namnh has quit IRC | 08:59 | |
| *** namnh has joined #openstack-infra | 08:59 | |
| andymccr | morning infra - i'm running into releasenote gate failures for OSA. The issue was reno's with duplicate hashes - but I've since fixed that, although the releasenote gate is still failing. I'm not sure how i can fix that now. any help would be super appreciated :) | 09:01 |
| *** namnh_ has quit IRC | 09:01 | |
| *** zhurong has joined #openstack-infra | 09:01 | |
| openstackgerrit | dongwenjuan proposed openstack-infra/puppet-jenkins master: use 'openjdk-8-jre-headless' for ubuntu-16.04 https://review.openstack.org/461708 | 09:02 |
| AJaeger | andymccr: ask dhellmann for reno, probably best on #openstack-release | 09:05 |
| *** neiljerram has quit IRC | 09:06 | |
| *** aviau has quit IRC | 09:06 | |
| *** aviau has joined #openstack-infra | 09:07 | |
| *** namnh_ has joined #openstack-infra | 09:07 | |
| *** yamamoto has joined #openstack-infra | 09:07 | |
| *** ggnel_t has joined #openstack-infra | 09:07 | |
| *** hieulq_ has joined #openstack-infra | 09:08 | |
| andymccr | AJaeger: will do - thanks for the pointer! | 09:09 |
| *** ggnel_t has quit IRC | 09:09 | |
| andymccr | probably wait a bit until its a more sensible time for the americas ;) | 09:09 |
| *** namnh has quit IRC | 09:09 | |
| *** yamamoto has quit IRC | 09:11 | |
| *** pbourke has quit IRC | 09:11 | |
| *** hieulq_ has quit IRC | 09:12 | |
| *** hieulq_ has joined #openstack-infra | 09:12 | |
| *** pbourke has joined #openstack-infra | 09:13 | |
| *** namnh has joined #openstack-infra | 09:14 | |
| openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-jenkins master: Fix beaker on xenial https://review.openstack.org/462048 | 09:15 |
| *** namnh_ has quit IRC | 09:16 | |
| odyssey4me | andymccr AJaeger I guess it'd be possible to temporarily disable the job, merge the fix, then enable it again? | 09:18 |
| odyssey4me | maybe use a filter for it on the repo in question | 09:18 |
| andymccr | odyssey4me: the fix is in | 09:19 |
| andymccr | like the duplicates are gone | 09:20 |
| andymccr | and no releases went out with the duplicates in (the tox job runs fine if you just clone it) | 09:20 |
| *** hieulq_ has quit IRC | 09:20 | |
| *** tosky has joined #openstack-infra | 09:21 | |
| *** yee379 has joined #openstack-infra | 09:23 | |
| *** yee37918 has quit IRC | 09:23 | |
| *** cuongnv_ has joined #openstack-infra | 09:23 | |
| odyssey4me | argh, I suspect that this has broken the OSA role jobs https://github.com/openstack-infra/project-config/commit/44ca9dd5f38db60b16512c5dba187edf8e2edb09 | 09:23 |
| odyssey4me | andymccr ^ | 09:23 |
| odyssey4me | see http://logs.openstack.org/38/461938/1/gate/gate-openstack-ansible-repo_build-ansible-func-ubuntu-xenial/d77a1e1/console.html | 09:23 |
| *** cuongnv has quit IRC | 09:24 | |
| *** thorst_afk has joined #openstack-infra | 09:24 | |
| *** vipul- has joined #openstack-infra | 09:25 | |
| *** vipuls has quit IRC | 09:25 | |
| *** Jeffrey4l has quit IRC | 09:25 | |
| *** pots has quit IRC | 09:25 | |
| *** Jeffrey4l has joined #openstack-infra | 09:27 | |
| odyssey4me | it's working for centos, but not for ubuntu it seems | 09:27 |
| andymccr | so the -e /usr/bin/yum isnt working like we expect then | 09:27 |
| odyssey4me | yup | 09:27 |
| andymccr | oh well just delete all ubuntu jobs and move to full time centos7 problem solved | 09:28 |
| odyssey4me | the reason that's been put in is for https://github.com/openstack/openstack-ansible-tests/blob/master/bindep.txt#L52-L53 - I reckon that we just ditch those packages from bindep | 09:28 |
| odyssey4me | I suspect that we don't need them for SNI support | 09:28 |
| *** thorst_afk has quit IRC | 09:28 | |
| odyssey4me | certainly for stuff we use to get ansible started - our role will enable epel and install the package later anyway | 09:29 |
| andymccr | it makes more sense to fix epel install up properly i think - then surely those would work? | 09:29 |
| odyssey4me | sure, although I don't really see why that would be broken | 09:29 |
| *** adreznec has quit IRC | 09:30 | |
| openstackgerrit | Jamie Lennox proposed openstack-infra/zuul feature/zuulv3: Use routes for URL handling in webapp https://review.openstack.org/461331 | 09:30 |
| openstackgerrit | Jamie Lennox proposed openstack-infra/zuul feature/zuulv3: Use tenant_name to look up project key https://review.openstack.org/461332 | 09:30 |
| andymccr | could it be that the task exists 1 because the command fails? | 09:31 |
| *** adreznec has joined #openstack-infra | 09:31 | |
| andymccr | whereas on centos it'll exit 0 | 09:31 |
| odyssey4me | hmm, perhaps - maybe the older method is a better method | 09:32 |
| *** ijw has joined #openstack-infra | 09:32 | |
| odyssey4me | but it should have enabled/disabled | 09:32 |
| andymccr | i'll push a patch in 2 secs | 09:32 |
| *** markvoelker has joined #openstack-infra | 09:33 | |
| *** cuongnv_ is now known as cuongnv | 09:34 | |
| openstackgerrit | Andy McCrae proposed openstack-infra/project-config master: Fix conditional for EPEL Repo enable/disable for OSA https://review.openstack.org/462054 | 09:35 |
| andymccr | odyssey4me: ^ take a look | 09:35 |
| odyssey4me | looks fine to me | 09:36 |
| *** ijw has quit IRC | 09:37 | |
| *** markvoelker has quit IRC | 09:38 | |
| *** Julien-zte has quit IRC | 09:39 | |
| *** dizquierdo has joined #openstack-infra | 09:39 | |
| AJaeger | andymccr: are all your jobs broken? Then I'll +3... | 09:39 |
| *** hieulq_ has joined #openstack-infra | 09:40 | |
| *** lamt has quit IRC | 09:40 | |
| *** zhurong has quit IRC | 09:41 | |
| andymccr | AJaeger: yeah they are | 09:42 |
| *** namnh_ has joined #openstack-infra | 09:46 | |
| *** alexchadin has quit IRC | 09:48 | |
| *** namnh has quit IRC | 09:49 | |
| AJaeger | andymccr: +3 | 09:50 |
| andymccr | thanks a bunch AJaeger | 09:50 |
| *** sdague has joined #openstack-infra | 09:55 | |
| *** tesseract has joined #openstack-infra | 09:56 | |
| *** ym200 has joined #openstack-infra | 09:58 | |
| *** ym200 has quit IRC | 09:58 | |
| openstackgerrit | Alfredo Moralejo proposed openstack-infra/tripleo-ci master: Add subpackages from openstack-tripleo-common from delorean-current https://review.openstack.org/462067 | 09:59 |
| *** ym200 has joined #openstack-infra | 10:00 | |
| openstackgerrit | Merged openstack-infra/project-config master: Fix conditional for EPEL Repo enable/disable for OSA https://review.openstack.org/462054 | 10:00 |
| *** hieulq_ has quit IRC | 10:03 | |
| openstackgerrit | Jamie Lennox proposed openstack-infra/zuul feature/zuulv3: Use routes for URL handling in webapp https://review.openstack.org/461331 | 10:04 |
| openstackgerrit | Jamie Lennox proposed openstack-infra/zuul feature/zuulv3: Use tenant_name to look up project key https://review.openstack.org/461332 | 10:04 |
| *** gcb has quit IRC | 10:04 | |
| *** salv-orlando has joined #openstack-infra | 10:06 | |
| *** liujiong has quit IRC | 10:11 | |
| *** zhurong has joined #openstack-infra | 10:18 | |
| *** oanson has quit IRC | 10:19 | |
| *** oanson has joined #openstack-infra | 10:21 | |
| *** ym200 has quit IRC | 10:21 | |
| *** ym200 has joined #openstack-infra | 10:22 | |
| *** cuongnv has quit IRC | 10:23 | |
| openstackgerrit | Lucas Alvares Gomes proposed openstack-infra/project-config master: Add experimental redfish dsvm job for ironic https://review.openstack.org/462072 | 10:23 |
| openstackgerrit | Lucas Alvares Gomes proposed openstack-infra/project-config master: Add experimental redfish dsvm job for ironic https://review.openstack.org/462072 | 10:26 |
| *** namnh has joined #openstack-infra | 10:27 | |
| *** nicolasbock has joined #openstack-infra | 10:28 | |
| *** namnh_ has quit IRC | 10:30 | |
| *** salv-orlando has quit IRC | 10:30 | |
| *** zhurong has quit IRC | 10:30 | |
| *** jkilpatr has quit IRC | 10:31 | |
| *** ym200 has quit IRC | 10:33 | |
| *** namnh_ has joined #openstack-infra | 10:33 | |
| *** markvoelker has joined #openstack-infra | 10:34 | |
| *** ym200 has joined #openstack-infra | 10:34 | |
| *** neiljerram has joined #openstack-infra | 10:36 | |
| *** namnh has quit IRC | 10:36 | |
| *** hieulq_ has joined #openstack-infra | 10:39 | |
| *** markvoelker has quit IRC | 10:40 | |
| *** namnh has joined #openstack-infra | 10:40 | |
| *** yanyanhu has quit IRC | 10:40 | |
| *** namnh_ has quit IRC | 10:42 | |
| *** namnh_ has joined #openstack-infra | 10:43 | |
| *** ym200 has quit IRC | 10:44 | |
| *** ym200 has joined #openstack-infra | 10:44 | |
| *** namnh has quit IRC | 10:46 | |
| *** funzo has joined #openstack-infra | 10:46 | |
| *** jkilpatr has joined #openstack-infra | 10:49 | |
| *** ym200 has quit IRC | 10:49 | |
| *** funzo has quit IRC | 10:51 | |
| *** bhavik1 has quit IRC | 10:52 | |
| *** jascott1 has joined #openstack-infra | 10:53 | |
| *** hieulq_ has quit IRC | 10:54 | |
| *** jascott1 has quit IRC | 10:57 | |
| openstackgerrit | Lucas Alvares Gomes proposed openstack-infra/project-config master: Add experimental redfish dsvm job for ironic https://review.openstack.org/462072 | 10:58 |
| *** ijw has joined #openstack-infra | 10:59 | |
| openstackgerrit | Tom Barron proposed openstack-infra/project-config master: Experimental py35 dsvm for ceph devstack plugin https://review.openstack.org/454890 | 11:00 |
| *** yolkfull__ has quit IRC | 11:00 | |
| *** alexchadin has joined #openstack-infra | 11:00 | |
| *** sree has quit IRC | 11:02 | |
| *** sree has joined #openstack-infra | 11:02 | |
| *** ijw has quit IRC | 11:04 | |
| *** namnh_ has quit IRC | 11:05 | |
| *** sree has quit IRC | 11:07 | |
| *** sree has joined #openstack-infra | 11:08 | |
| *** darvon has quit IRC | 11:10 | |
| *** ijw has joined #openstack-infra | 11:10 | |
| *** darvon has joined #openstack-infra | 11:11 | |
| *** sree has quit IRC | 11:13 | |
| *** rhallisey has joined #openstack-infra | 11:14 | |
| *** alexchadin has quit IRC | 11:14 | |
| *** alexchadin has joined #openstack-infra | 11:14 | |
| *** ijw has quit IRC | 11:15 | |
| openstackgerrit | Merged openstack-infra/shade master: Don't get ports info from unavailable neutron service https://review.openstack.org/461903 | 11:20 |
| *** dimtruck is now known as zz_dimtruck | 11:20 | |
| *** ijw has joined #openstack-infra | 11:21 | |
| openstackgerrit | Lucas Alvares Gomes proposed openstack-infra/project-config master: Add experimental redfish dsvm job for ironic https://review.openstack.org/462072 | 11:23 |
| *** thorst_afk has joined #openstack-infra | 11:25 | |
| *** ijw has quit IRC | 11:25 | |
| *** lucasagomes is now known as lucas-hungry | 11:28 | |
| *** alexchadin has quit IRC | 11:29 | |
| *** thorst_afk has quit IRC | 11:31 | |
| *** ijw has joined #openstack-infra | 11:31 | |
| openstackgerrit | Slawek Kaplonski proposed openstack-infra/shade master: Replace neutronclient with REST API calls in ports commands https://review.openstack.org/461357 | 11:32 |
| cmurphy | what does POST_FAILURE mean again? is that something the patch could have broken or something wrong with the publisher? | 11:32 |
| openstackgerrit | Slawek Kaplonski proposed openstack-infra/shade master: Replace neutronclient with REST API calls in ports commands https://review.openstack.org/461357 | 11:34 |
| *** alexchadin has joined #openstack-infra | 11:34 | |
| *** tosky has quit IRC | 11:34 | |
| *** ijw has quit IRC | 11:36 | |
| *** markvoelker has joined #openstack-infra | 11:36 | |
| *** nmathew has quit IRC | 11:39 | |
| openstackgerrit | Andrea Frittoli proposed openstack-infra/project-config master: Remove the tempest layer4 job https://review.openstack.org/461867 | 11:40 |
| *** markvoelker has quit IRC | 11:40 | |
| *** ldnunes has joined #openstack-infra | 11:40 | |
| sdague | cmurphy: my understanding is it's nearly always the system breaking down, not the patch | 11:41 |
| *** pradk has joined #openstack-infra | 11:43 | |
| *** pradk has quit IRC | 11:43 | |
| cmurphy | hmm | 11:44 |
| *** rfolco has joined #openstack-infra | 11:44 | |
| *** tosky has joined #openstack-infra | 11:45 | |
| *** gordc has joined #openstack-infra | 11:46 | |
| *** hashar is now known as hasharLunch | 11:48 | |
| *** roflik has joined #openstack-infra | 11:51 | |
| *** pcaruana has quit IRC | 11:53 | |
| *** thorst_afk has joined #openstack-infra | 11:55 | |
| *** srobert_ has joined #openstack-infra | 11:56 | |
| *** srobert has quit IRC | 11:56 | |
| AJaeger | cmurphy: documented in the infra-manual at https://docs.openstack.org/infra/manual/developers.html#automated-testing - basically POST_FAILURE means that we could not collect any log files. It's often a CI problem | 11:57 |
| *** pcaruana has joined #openstack-infra | 11:57 | |
| AJaeger | cmurphy: but it could be also your patch breaking the system very badly, e.g. with losing network | 11:58 |
| roflik | is this the right place for questions about jenkins-job-builder? | 11:58 |
| *** wolverineav has joined #openstack-infra | 11:58 | |
| *** links has quit IRC | 12:00 | |
| *** isaacb has quit IRC | 12:01 | |
| *** zhurong has joined #openstack-infra | 12:01 | |
| *** ijw has joined #openstack-infra | 12:02 | |
| *** isaacb has joined #openstack-infra | 12:03 | |
| cmurphy | AJaeger: thanks, I was looking for that | 12:05 |
| *** Qiming has quit IRC | 12:06 | |
| openstackgerrit | Monty Taylor proposed openstack/os-client-config master: WIP Braindump thoughts and a tenantive plan for v2 https://review.openstack.org/462092 | 12:06 |
| *** watersoul_ has quit IRC | 12:06 | |
| *** watersoul has joined #openstack-infra | 12:07 | |
| *** huanxie has quit IRC | 12:07 | |
| *** rlandy has joined #openstack-infra | 12:08 | |
| *** ijw has quit IRC | 12:08 | |
| openstackgerrit | Merged openstack-infra/shade master: Move REST error_messages to error_message argument https://review.openstack.org/461290 | 12:09 |
| openstackgerrit | Merged openstack-infra/shade master: Replace neutronclient with REST API calls in quotas commands https://review.openstack.org/461362 | 12:09 |
| *** dprince has joined #openstack-infra | 12:09 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack-infra/shade master: Updated from global requirements https://review.openstack.org/461375 | 12:11 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/project-config master: Exclude undercloud containers from gates https://review.openstack.org/462112 | 12:17 |
| *** hasharLunch is now known as hashar | 12:19 | |
| *** Qiming has joined #openstack-infra | 12:20 | |
| *** rhallisey has quit IRC | 12:22 | |
| *** rhallisey has joined #openstack-infra | 12:22 | |
| *** isaacb has quit IRC | 12:23 | |
| *** ijw has joined #openstack-infra | 12:23 | |
| EmilienM | AJaeger: hey, good afternoon. Any chance to get https://review.openstack.org/#/c/462112 ? our gate is now blocked on this. https://review.openstack.org/#/c/446940/ merged without my +1 and the job was not ready | 12:24 |
| *** askb has quit IRC | 12:24 | |
| *** srobert_ has quit IRC | 12:24 | |
| *** srobert has joined #openstack-infra | 12:25 | |
| *** ijw has quit IRC | 12:28 | |
| openstackgerrit | Merged openstack-infra/project-config master: Remove the tempest layer4 job https://review.openstack.org/461867 | 12:29 |
| *** mat128 has joined #openstack-infra | 12:31 | |
| *** ijw has joined #openstack-infra | 12:33 | |
| *** nicolasbock has quit IRC | 12:34 | |
| *** alexchadin has quit IRC | 12:34 | |
| *** markvoelker has joined #openstack-infra | 12:37 | |
| *** sree has joined #openstack-infra | 12:37 | |
| *** ggnel_t has joined #openstack-infra | 12:38 | |
| *** jpena is now known as jpena|lunch | 12:40 | |
| *** baoli has joined #openstack-infra | 12:41 | |
| *** links has joined #openstack-infra | 12:41 | |
| *** markvoelker has quit IRC | 12:41 | |
| EmilienM | AJaeger, fungi: thank you | 12:41 |
| *** lucas-hungry is now known as lucasagomes | 12:42 | |
| openstackgerrit | Merged openstack-infra/project-config master: Exclude undercloud containers from gates https://review.openstack.org/462112 | 12:42 |
| *** ijw has quit IRC | 12:42 | |
| *** dprince has quit IRC | 12:43 | |
| *** eharney has joined #openstack-infra | 12:44 | |
| *** nicolasbock has joined #openstack-infra | 12:47 | |
| openstackgerrit | Martin André proposed openstack-infra/project-config master: Add undercloud container to t-h-t check https://review.openstack.org/462141 | 12:48 |
| *** felipemonteiro has joined #openstack-infra | 12:49 | |
| pabelanger | morning | 12:49 |
| *** mriedem has joined #openstack-infra | 12:51 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/project-config master: Enable EPEL for openstack-ansible CentOS bindeps packages https://review.openstack.org/459913 | 12:54 |
| *** gcb has joined #openstack-infra | 12:56 | |
| *** nmathew has joined #openstack-infra | 12:56 | |
| *** kgiusti has joined #openstack-infra | 12:57 | |
| *** vikrant has quit IRC | 12:58 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Create openstack-zuul-jobs / openstack-zuul-roles projects https://review.openstack.org/461897 | 12:58 |
| AJaeger | fungi, the change above needs infra PTL +1 ^ | 12:58 |
| *** mdrabe has joined #openstack-infra | 12:59 | |
| AJaeger | pabelanger: I think you have an unrelated change in it ^ | 12:59 |
| pabelanger | AJaeger: Ah,yes | 12:59 |
| pabelanger | thanks | 12:59 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Create openstack-zuul-jobs / openstack-zuul-roles projects https://review.openstack.org/461897 | 13:00 |
| pabelanger | forgot to git stash my current working directory | 13:00 |
| *** masber has quit IRC | 13:01 | |
| pabelanger | yonglihe: cmurphy: do you mind reviewing: https://review.openstack.org/#/q/topic:puppet-lint too? some fixes for puppet lint failures | 13:01 |
| *** claudiub has joined #openstack-infra | 13:02 | |
| *** mpaolino has joined #openstack-infra | 13:03 | |
| pabelanger | mordred: any chance on vhd-util patch today or any objections if I work on it? | 13:03 |
| *** nicolasbock has quit IRC | 13:05 | |
| *** gouthamr has joined #openstack-infra | 13:05 | |
| *** alexchadin has joined #openstack-infra | 13:06 | |
| *** nmathew has quit IRC | 13:06 | |
| *** felipemonteiro_ has joined #openstack-infra | 13:08 | |
| *** jamesdenton has joined #openstack-infra | 13:08 | |
| *** lamt has joined #openstack-infra | 13:08 | |
| *** felipemonteiro has quit IRC | 13:11 | |
| openstackgerrit | Jordan Pittier proposed openstack/os-client-config master: Revert "Use interface not endpoint_type for keystoneclient" https://review.openstack.org/462144 | 13:11 |
| *** eharney has quit IRC | 13:11 | |
| openstackgerrit | Jordan Pittier proposed openstack/os-client-config master: Revert "Use interface not endpoint_type for keystoneclient" https://review.openstack.org/462144 | 13:11 |
| *** tosky has quit IRC | 13:11 | |
| *** felipemonteiro__ has joined #openstack-infra | 13:13 | |
| *** felipemonteiro_ has quit IRC | 13:13 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Add reprepro for docker https://review.openstack.org/453304 | 13:13 |
| *** sree has quit IRC | 13:14 | |
| *** sree has joined #openstack-infra | 13:15 | |
| *** askb has joined #openstack-infra | 13:15 | |
| openstackgerrit | Jordan Pittier proposed openstack/os-client-config master: Revert "Use interface not endpoint_type for keystoneclient" https://review.openstack.org/462144 | 13:16 |
| *** jaypipes has joined #openstack-infra | 13:18 | |
| fungi | pabelanger: one thing which i'm still unclear on (and i didn't pick up from the meeting discussion unfortunately) is the reason for splitting untrusted shared jobs and untrusted shared roles into different repos | 13:19 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci master: DO NOT MERGE: Testing LDAP integration https://review.openstack.org/461865 | 13:19 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci master: Deploy FreeIPA-backed LDAP domain for keystone in fakeha-caserver job https://review.openstack.org/461864 | 13:19 |
| fungi | pabelanger: i mean, rather than one shared repo to contain the untrusted jobs and untrusted roles together | 13:19 |
| *** sree has quit IRC | 13:19 | |
| *** tiswanso has joined #openstack-infra | 13:20 | |
| fungi | is it a design constraint or just an aesthetic/cosmetic thing? | 13:20 |
| *** mat128 has quit IRC | 13:21 | |
| *** trown|outtypewww is now known as trown | 13:21 | |
| *** jamesmca_ has joined #openstack-infra | 13:21 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Add reprepro for docker https://review.openstack.org/453304 | 13:21 |
| *** isaacb has joined #openstack-infra | 13:21 | |
| *** nicolasbock has joined #openstack-infra | 13:22 | |
| *** sshnaidm is now known as sshnaidm|afk | 13:22 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci master: DO NOT MERGE: Testing LDAP integration https://review.openstack.org/461865 | 13:23 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci master: Deploy FreeIPA-backed LDAP domain for keystone in fakeha-caserver job https://review.openstack.org/461864 | 13:23 |
| pabelanger | fungi: I think design, if we get to a point where we want to publish openstack-zuul-roles for people (maybe 3rd party CI as a example) it would make sense to break them away from jobs (playbooks). However, it could make job a little harder to write, since changes could me over 2 repos now | 13:24 |
| pabelanger | fungi: I initially was going to add roles into openstack-zuul-jobs first, then work to break out later. But opted to also create openstack-zuul-roles at the same time | 13:25 |
| *** salv-orlando has joined #openstack-infra | 13:25 | |
| pabelanger | However, I don't have a strong preference TBH. Single repo maybe be fine for our use case too. | 13:26 |
| *** eharney has joined #openstack-infra | 13:26 | |
| fungi | okay, it just seems similar to (using our jjb configs as an analogy) putting the macros.yaml file in a different repo from the job templates and jobs | 13:26 |
| *** tongli has joined #openstack-infra | 13:26 | |
| pabelanger | Ya, thats a good way to look at it | 13:26 |
| fungi | though considering that in-repo jobs for other projects may reuse those roles, maybe not dragging in the shared job definitions repo makes sense, i don't really know | 13:27 |
| fungi | anyway, not an objection, just trying to understand the logic there | 13:28 |
| *** zhurong has quit IRC | 13:28 | |
| pabelanger | Ya, I think there is a little bit of experimenting here too. We may find it too much work with 2 repos and just decide to move back to a single | 13:28 |
| *** chlong has quit IRC | 13:28 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci master: DO NOT MERGE: Testing LDAP integration https://review.openstack.org/461865 | 13:28 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci master: Deploy FreeIPA-backed LDAP domain for keystone in fakeha-caserver job https://review.openstack.org/461864 | 13:28 |
| pabelanger | fungi: do you mind +3 on https://review.openstack.org/#/q/topic:puppet-lint change, help get some of our puppet jobs back in working order | 13:29 |
| *** liusheng has quit IRC | 13:32 | |
| *** liusheng has joined #openstack-infra | 13:33 | |
| *** ltomasbo is now known as ltomasbo|away | 13:33 | |
| *** arxcruz has quit IRC | 13:33 | |
| *** jamesmca_ has quit IRC | 13:34 | |
| *** arxcruz has joined #openstack-infra | 13:34 | |
| *** EricGonczer_ has joined #openstack-infra | 13:34 | |
| *** dtantsur is now known as dtantsur|brb | 13:35 | |
| openstackgerrit | Merged openstack-infra/system-config master: Add reprepro for docker https://review.openstack.org/453304 | 13:35 |
| *** sbezverk_ is now known as sbezverk | 13:35 | |
| *** ltomasbo|away is now known as ltomasbo | 13:36 | |
| *** lamt has quit IRC | 13:37 | |
| *** gcb has quit IRC | 13:40 | |
| *** gcb has joined #openstack-infra | 13:40 | |
| *** jaosorior is now known as jaosorior_away | 13:41 | |
| *** jpena|lunch is now known as jpena | 13:41 | |
| *** xyang1 has joined #openstack-infra | 13:41 | |
| *** tosky has joined #openstack-infra | 13:42 | |
| *** gcb has quit IRC | 13:43 | |
| openstackgerrit | Merged openstack-infra/puppet-infracloud master: Fix puppet-lint job https://review.openstack.org/461785 | 13:44 |
| *** gcb has joined #openstack-infra | 13:44 | |
| *** ggnel_t has quit IRC | 13:44 | |
| openstackgerrit | Merged openstack-infra/puppet-infracloud master: Add bindep.txt file https://review.openstack.org/461571 | 13:44 |
| *** psachin has joined #openstack-infra | 13:45 | |
| openstackgerrit | Merged openstack-infra/puppet-germqtt master: Fix puppet-lint job https://review.openstack.org/461787 | 13:45 |
| openstackgerrit | Merged openstack-infra/puppet-germqtt master: Add bindep.txt file https://review.openstack.org/461561 | 13:45 |
| openstackgerrit | Merged openstack-infra/puppet-apps_site master: Fix puppet-lint job https://review.openstack.org/461788 | 13:46 |
| openstackgerrit | Merged openstack-infra/puppet-apps_site master: Add bindep.txt file https://review.openstack.org/461541 | 13:46 |
| *** mriedem has quit IRC | 13:48 | |
| *** dprince has joined #openstack-infra | 13:49 | |
| openstackgerrit | Rosario Di Somma proposed openstack-infra/shade master: Use REST API for volume attach and volume backup calls https://review.openstack.org/461781 | 13:49 |
| *** Julien-zte has joined #openstack-infra | 13:50 | |
| *** mriedem has joined #openstack-infra | 13:51 | |
| *** armax has joined #openstack-infra | 13:51 | |
| pabelanger | fungi: mind +3 on https://review.openstack.org/#/c/461517 removes cache-bindep element and move pip install of bindep into infra-package-needs. Cosmetic change | 13:52 |
| *** Goneri has joined #openstack-infra | 13:52 | |
| pabelanger | https://review.openstack.org/#/c/461525/ too if you have the time | 13:53 |
| *** felipemonteiro_ has joined #openstack-infra | 13:53 | |
| *** Swami has joined #openstack-infra | 13:54 | |
| *** jascott1 has joined #openstack-infra | 13:56 | |
| *** felipemonteiro__ has quit IRC | 13:57 | |
| *** markvoelker has joined #openstack-infra | 13:57 | |
| *** jamesmca_ has joined #openstack-infra | 13:57 | |
| *** links has quit IRC | 13:58 | |
| *** Goneri has quit IRC | 13:58 | |
| *** jascott1 has quit IRC | 14:00 | |
| openstackgerrit | Merged openstack-infra/puppet-ansible master: Fix beaker on xenial https://review.openstack.org/461811 | 14:00 |
| *** zhurong has joined #openstack-infra | 14:02 | |
| *** srobert has quit IRC | 14:04 | |
| openstackgerrit | Merged openstack-infra/project-config master: Create openstack-zuul-jobs / openstack-zuul-roles projects https://review.openstack.org/461897 | 14:04 |
| *** xarses has quit IRC | 14:08 | |
| *** pradk has joined #openstack-infra | 14:12 | |
| fungi | pabelanger: sorry, i must be going blind. approved! | 14:12 |
| *** zz_dimtruck is now known as dimtruck | 14:15 | |
| fungi | pabelanger: i think i see a potential major simplification for 461525 | 14:16 |
| *** isaacb has quit IRC | 14:17 | |
| fungi | pip can already install from a git repo url, no need for git clone and cleanup | 14:17 |
| *** funzo has joined #openstack-infra | 14:18 | |
| *** rbrndt has joined #openstack-infra | 14:18 | |
| pabelanger | fungi: sure, will look shortly | 14:18 |
| *** openstackgerrit has quit IRC | 14:18 | |
| *** isaacb has joined #openstack-infra | 14:18 | |
| *** rossella_s has quit IRC | 14:20 | |
| *** rossella_s has joined #openstack-infra | 14:21 | |
| *** funzo has quit IRC | 14:23 | |
| *** openstackgerrit has joined #openstack-infra | 14:23 | |
| openstackgerrit | Merged openstack-infra/project-config master: Remove cache-bindep element https://review.openstack.org/461517 | 14:23 |
| *** mdrabe_ has joined #openstack-infra | 14:23 | |
| *** isaacb has quit IRC | 14:24 | |
| *** mdrabe has quit IRC | 14:24 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Clone zuul from git.openstack.org for zuul-env https://review.openstack.org/461525 | 14:25 |
| *** dave-mccowan has joined #openstack-infra | 14:25 | |
| *** amotoki has joined #openstack-infra | 14:26 | |
| *** ihrachys has joined #openstack-infra | 14:26 | |
| *** jamesmca_ has quit IRC | 14:29 | |
| *** jamesmca_ has joined #openstack-infra | 14:30 | |
| openstackgerrit | Jiri Stransky proposed openstack-infra/tripleo-ci master: Use mixed release for container upgrade https://review.openstack.org/460061 | 14:30 |
| openstackgerrit | Jiri Stransky proposed openstack-infra/tripleo-ci master: Use mixed release for container upgrade https://review.openstack.org/460061 | 14:31 |
| *** srobert has joined #openstack-infra | 14:32 | |
| *** askb has quit IRC | 14:36 | |
| *** marst has quit IRC | 14:38 | |
| *** prateek has quit IRC | 14:39 | |
| *** mat128 has joined #openstack-infra | 14:39 | |
| dmsimard | clarkb, fungi, jeblair: do you happen to know if it's possible to set up nodepool to use boot from volume ? | 14:40 |
| *** mat128 has quit IRC | 14:40 | |
| *** lamt has joined #openstack-infra | 14:40 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/infra-manual master: New subsection on voting third-party CI permission https://review.openstack.org/462176 | 14:41 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Fixes needed for reprepro docker mirror https://review.openstack.org/462178 | 14:42 |
| *** jordanP has joined #openstack-infra | 14:44 | |
| fungi | dmsimard: maybe mordred's played with it? as long as shade supports it, probably not hard to implement but at the moment i'm pretty sure it expects boot from image instead | 14:44 |
| dmsimard | fungi: ack, thanks | 14:45 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/infra-manual master: New subsection on voting third-party CI permission https://review.openstack.org/462176 | 14:47 |
| AJaeger | fungi, fixed your wording ^ | 14:47 |
| *** EricGonczer_ has quit IRC | 14:48 | |
| *** marst has joined #openstack-infra | 14:50 | |
| *** zhurong has quit IRC | 14:52 | |
| fungi | AJaeger: thanks! i was probably not paying close attention to what i wrote there ;) | 14:52 |
| *** salv-orl_ has joined #openstack-infra | 14:53 | |
| *** dtantsur|brb is now known as dtantsur | 14:56 | |
| *** salv-orlando has quit IRC | 14:56 | |
| *** xarses has joined #openstack-infra | 14:58 | |
| *** xarses_ has joined #openstack-infra | 14:58 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: Fixed group event serialization issue https://review.openstack.org/462184 | 14:58 |
| *** erlon has joined #openstack-infra | 14:58 | |
| *** rkukura has joined #openstack-infra | 14:59 | |
| *** annegentle has joined #openstack-infra | 14:59 | |
| *** andreww has joined #openstack-infra | 15:00 | |
| *** Goneri has joined #openstack-infra | 15:01 | |
| *** gcb has quit IRC | 15:02 | |
| *** xarses has quit IRC | 15:02 | |
| *** xarses has joined #openstack-infra | 15:03 | |
| *** gcb has joined #openstack-infra | 15:03 | |
| *** andreww has quit IRC | 15:05 | |
| *** dougwig has joined #openstack-infra | 15:05 | |
| *** andreww has joined #openstack-infra | 15:06 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Increase file permissions around generate keys https://review.openstack.org/461218 | 15:06 |
| *** andreww has quit IRC | 15:07 | |
| clarkb | dmsimard: you can attach metadata to images if that allows you to boot from volume. But otherwise I think it would require code changes | 15:07 |
| *** andreww has joined #openstack-infra | 15:07 | |
| *** xarses has quit IRC | 15:07 | |
| clarkb | apparently image metadata can be used to instruct generic volume attachments, but not sure about boot from volume | 15:07 |
| *** Goneri has quit IRC | 15:10 | |
| *** gcb has quit IRC | 15:10 | |
| *** Goneri has joined #openstack-infra | 15:11 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci master: Deploy FreeIPA-backed LDAP domain for keystone in fakeha-caserver job https://review.openstack.org/461864 | 15:12 |
| openstackgerrit | Merged openstack-infra/system-config master: Fixes needed for reprepro docker mirror https://review.openstack.org/462178 | 15:12 |
| *** tjones has joined #openstack-infra | 15:12 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci master: DO NOT MERGE: Testing LDAP integration https://review.openstack.org/461865 | 15:13 |
| *** jascott1 has joined #openstack-infra | 15:13 | |
| *** hongbin has joined #openstack-infra | 15:13 | |
| openstackgerrit | Merged openstack-infra/project-config master: Clone zuul from git.openstack.org for zuul-env https://review.openstack.org/461525 | 15:14 |
| openstackgerrit | Merged openstack-infra/openstackid-resources master: Fixed group event serialization issue https://review.openstack.org/462184 | 15:15 |
| *** chlong has joined #openstack-infra | 15:15 | |
| clarkb | devstack-gate reviewers https://review.openstack.org/#/c/461883/ review on that would be much appreciated as I'm hoping to have something in place to debug all these cinder fails before we all disappear. Hopefully gather enough info to fix either during summit or after | 15:16 |
| *** rcarrillocruz has quit IRC | 15:16 | |
| *** gcb has joined #openstack-infra | 15:17 | |
| *** salv-orl_ has quit IRC | 15:18 | |
| *** rcarrillocruz has joined #openstack-infra | 15:18 | |
| *** eharney has quit IRC | 15:18 | |
| *** salv-orlando has joined #openstack-infra | 15:18 | |
| amrith | hi infra folks, is keystone v2 auth disabled in the CI? something appears to have changed in the last day that has caused all trove CI jobs to fail with keystone auth v2 failures. | 15:22 |
| sdague | clarkb: http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Setting%20instance%20vm_state%20to%20ERROR%5C%22%20AND%20message%3A%5C%22_unplug_vifs%5C%22%20AND%20message%3A%5C%22UnicodeDecodeError%5C%22%20AND%20tags%3A%5C%22screen-n-cpu.txt%5C%22%20AND%20voting%3A1 | 15:22 |
| sdague | grok is definitely not parsing things right | 15:22 |
| amrith | sdague ^^ you've made some changes to keystone settings in the CI in the past couple of weeks, do you have any guidance on things that may have changed in the past 24-36 hours in this regard? | 15:23 |
| *** alexchadin has quit IRC | 15:24 | |
| sdague | amrith: yes, keystone v2 admin api is no longer running | 15:24 |
| amrith | oh? | 15:24 |
| sdague | auth still is | 15:24 |
| amrith | when did this change go in, was it in the past 36 hours? | 15:24 |
| sdague | I thought it was longer ago than that | 15:24 |
| sdague | but in the last week | 15:25 |
| amrith | yes | 15:25 |
| amrith | I saw your changes | 15:25 |
| amrith | they were 13 days back | 15:25 |
| amrith | the failure I'm seeing is in the past day | 15:25 |
| amrith | like this | 15:25 |
| amrith | http://logs.openstack.org/90/461590/3/check/gate-trove-functional-dsvm-mysql-ubuntu-xenial/1795e58/console.html#_2017-05-02_11_10_19_123775 | 15:25 |
| sdague | in the last 36 hours is systemd | 15:25 |
| amrith | well, systemd shouldn't have caused this | 15:25 |
| amrith | it is a definitive error saying that the server is temporarily unable to do stuff for me, so take an asprin and call in the morning | 15:25 |
| *** martinkopec has quit IRC | 15:26 | |
| sdague | amrith: oh... you know what | 15:26 |
| amrith | the service itself is running | 15:26 |
| amrith | v3 is responding just fine | 15:26 |
| sdague | I think one thing might have been missed | 15:26 |
| amrith | no, did I win the lottery? | 15:26 |
| amrith | again? | 15:26 |
| amrith | can I retire? | 15:26 |
| amrith | I'm guessing I didn't. :( | 15:27 |
| *** aarefiev is now known as aarefiev_afk | 15:27 | |
| amrith | what was it sdague | 15:28 |
| amrith | and was it something in the past 36 or so hours? | 15:28 |
| clarkb | sdague: I see the issue in multiline rule. patch in a sec | 15:28 |
| amrith | because this all worked on may 1 | 15:28 |
| sdague | amrith: where are you getting identity_admin from ? | 15:28 |
| openstackgerrit | Rosario Di Somma proposed openstack-infra/shade master: Use REST API for volume attach and volume backup calls https://review.openstack.org/461781 | 15:29 |
| amrith | lmc | 15:29 |
| sdague | so, we should have effectively scrubbed all of that | 15:30 |
| amrith | trove_auth_url is set in trove's config files as http://<host>/identity_admin/v2.0/tokens | 15:30 |
| amrith | we set it in the devstack plugin thusly: https://github.com/openstack/trove/blob/master/devstack/plugin.sh#L219 | 15:31 |
| sdague | http://logs.openstack.org/90/461590/3/check/gate-trove-functional-dsvm-mysql-ubuntu-xenial/1795e58/console.html#_2017-05-02_11_10_00_112182 | 15:31 |
| *** trown is now known as trown|brb | 15:31 | |
| amrith | yes, that's right | 15:31 |
| sdague | where is that test.conf built? | 15:31 |
| amrith | good q, let me check | 15:32 |
| amrith | for the tests you are looking at, the devstack plugin link I sent you is the thing that gets used | 15:32 |
| pabelanger | clarkb: jeblair: fungi: https://review.openstack.org/#/c/461905 should be ready now. Adds openstack-zuul-jobs / openstack-zuul-roles into zuulv3-dev.o.o. | 15:33 |
| pabelanger | I could rebase to see about removing -1 from zuul | 15:33 |
| *** lamt has quit IRC | 15:33 | |
| sdague | amrith: ok, KEYSTONE_AUTH_URI should be with /identity not /identity_admin | 15:33 |
| amrith | sdague, that is set: https://github.com/openstack/trove/blob/master/devstack/plugin.sh#L197 | 15:33 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Add openstack-zuul-jobs / openstack-zuul-roles to zuul.yaml https://review.openstack.org/461905 | 15:33 |
| *** eharney has joined #openstack-infra | 15:34 | |
| amrith | sdague, did the admin service stop running in the past day or so? | 15:34 |
| sdague | https://github.com/openstack-dev/devstack/blob/8b8441f3becbae2e704932569bff384dcc5c6713/lib/keystone#L116-L122 | 15:34 |
| sdague | amrith: yes | 15:34 |
| amrith | OK, thanks. then I'll find out where we get that KEYSTONE_AUTH_URI and change that and try again | 15:34 |
| *** trown|brb is now known as trown | 15:35 | |
| sdague | amrith: https://review.openstack.org/#/c/458226/ | 15:35 |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Handle journald timestamps in multiline logstash https://review.openstack.org/462191 | 15:35 |
| clarkb | sdague: ^ that should fix it | 15:35 |
| amrith | sdague, the timing is about right | 15:35 |
| amrith | thank you so much, will go try with that. | 15:35 |
| sdague | amrith: so that should have landed all the required elements at once | 15:35 |
| sdague | amrith: can you try a recheck on that patch? | 15:36 |
| amrith | I've pushed up recent patches that have failed with this | 15:36 |
| sdague | because the real mystery is where it's getting the old KEYSTONE_AUTH_URI value | 15:36 |
| clarkb | pabelanger: and once that merges you'll have to restart zuulv3 right? | 15:36 |
| amrith | so maybe trove is doing something wrong (that would be a first so I shudder to suggest the idea) | 15:36 |
| sdague | amrith: well, the real trick is figuring out where that reference to idenity_admin is coming from | 15:37 |
| sdague | because it really should have been fully purged | 15:37 |
| amrith | i think it is in project-config in the job definition | 15:37 |
| amrith | let me look | 15:37 |
| pabelanger | clarkb: yes, I had to manual copy file onto zuulv3.o.o, then reload zuul-scheduler | 15:37 |
| pabelanger | I see new private keys get created too! | 15:37 |
| amrith | nope, not there | 15:37 |
| clarkb | pabelanger: gotcha so its already in place then? | 15:38 |
| jeblair | clarkb, pabelanger: no restart should be required for 461905. a reconfiguration or restart is (was?) required for the change that added them to zuul.yaml. | 15:38 |
| pabelanger | clarkb: Yup, if not zuul would -1 again I think | 15:38 |
| jeblair | clarkb, pabelanger: no restart should be required for 461905. a reconfiguration or restart is (was?) required for the change that added them to main.yaml. <-- correction | 15:38 |
| clarkb | I was going to say that is zuul.yaml :) thanks for clarifying | 15:38 |
| jeblair | pabelanger: right, if the necessary reconfiguration has happened, zuulv3 should +1 that change, otherwise -1 it. | 15:39 |
| pabelanger | agree, main.yaml was the reload | 15:39 |
| jeblair | and it will be effective as soon as it lands | 15:39 |
| pabelanger | jeblair: if the reason zuul didn't +1 because we are lacking a noop-job? | 15:39 |
| jeblair | pabelanger: no, zuul should report on that change, it's a change to a repo it's configured to act on | 15:40 |
| *** Goneri has quit IRC | 15:40 | |
| *** lamt has joined #openstack-infra | 15:40 | |
| *** jamesmca_ has quit IRC | 15:41 | |
| pabelanger | Hmm, I don't think that happened | 15:41 |
| clarkb | where is zuul configured to act on project-config? | 15:41 |
| jeblair | pabelanger: oh, that's not true, we are not set up to report on project-config | 15:41 |
| clarkb | right that | 15:41 |
| pabelanger | Oh, right | 15:41 |
| *** ykarel has quit IRC | 15:41 | |
| *** Goneri has joined #openstack-infra | 15:41 | |
| jeblair | neat. :) | 15:41 |
| jeblair | it reports config errors, but not anything else | 15:42 |
| *** Rockyg has joined #openstack-infra | 15:42 | |
| clarkb | pabelanger: I'm behind on private keys, did we do things related to yesterday's stuff? | 15:42 |
| *** jamesmca_ has joined #openstack-infra | 15:42 | |
| clarkb | re pabelanger | I see new private keys get created too! | 15:42 |
| jeblair | i think that's zuul secrets private keys | 15:43 |
| pabelanger | clarkb: Oh, that was zuulv3 creating private keys for secrets, if our jobs will use them | 15:43 |
| clarkb | ah | 15:43 |
| *** felipemonteiro_ has quit IRC | 15:43 | |
| clarkb | thanks | 15:43 |
| *** ltomasbo is now known as ltomasbo|away | 15:43 | |
| amrith | sdague, I'll see what we get invoked in the plugin (https://review.openstack.org/#/c/462165/2/devstack/plugin.sh) | 15:43 |
| amrith | sdague, thanks | 15:47 |
| *** martinkopec has joined #openstack-infra | 15:47 | |
| sdague | yeh, it's definitely weird, we need to figure out where that is coming from | 15:47 |
| sdague | because I can't see anything that would set it as such | 15:48 |
| clarkb | infra-root can I also get review on https://review.openstack.org/#/c/462191/1 to fix logstash parsing issue sdague pointed out? | 15:49 |
| pabelanger | looking | 15:49 |
| pabelanger | +2 | 15:50 |
| *** felipemonteiro has joined #openstack-infra | 15:50 | |
| *** isaacb has joined #openstack-infra | 15:51 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add openstack-zuul-jobs / openstack-zuul-roles to zuul.yaml https://review.openstack.org/461905 | 15:51 |
| openstackgerrit | Paul Belanger proposed openstack-infra/openstack-zuul-jobs master: Initial commit https://review.openstack.org/462195 | 15:51 |
| AJaeger | pabelanger: you should add some team members to https://review.openstack.org/#/admin/groups/1759,members for your change to merge ;) | 15:52 |
| openstackgerrit | Paul Belanger proposed openstack-infra/openstack-zuul-jobs master: Initial commit https://review.openstack.org/462195 | 15:53 |
| *** mpaolino has quit IRC | 15:53 | |
| pabelanger | AJaeger: Ha, ya guess we should | 15:53 |
| *** felipemonteiro_ has joined #openstack-infra | 15:53 | |
| pabelanger | cool, zuul noop noop that change ^ :) | 15:54 |
| *** Apoorva has joined #openstack-infra | 15:56 | |
| *** felipemonteiro has quit IRC | 15:57 | |
| *** tongli has quit IRC | 15:59 | |
| openstackgerrit | Merged openstack-infra/system-config master: Handle journald timestamps in multiline logstash https://review.openstack.org/462191 | 15:59 |
| pabelanger | mordred: we now have docker mirror for packaging: http://mirror.dfw.rax.openstack.org/deb-docker/ | 16:01 |
| *** felipemonteiro_ has quit IRC | 16:02 | |
| *** knangia has joined #openstack-infra | 16:02 | |
| *** felipemonteiro_ has joined #openstack-infra | 16:02 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/openstack-zuul-jobs master: WIP: Add tox-linters job https://review.openstack.org/462202 | 16:05 |
| *** ccamacho has quit IRC | 16:07 | |
| *** rossella_s has quit IRC | 16:08 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/devstack-gate master: Do not merge: Test ARA 0.13.0rc1 https://review.openstack.org/439966 | 16:10 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/devstack-gate master: Do not merge: Test ARA 0.13.0rc1 https://review.openstack.org/439966 | 16:10 |
| *** martinkopec has quit IRC | 16:12 | |
| *** rlandy is now known as rlandy|brb | 16:16 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: Source ci mirrors file if it exists only https://review.openstack.org/460022 | 16:18 |
| *** felipemonteiro__ has joined #openstack-infra | 16:18 | |
| *** matrohon has quit IRC | 16:20 | |
| *** jordanP has quit IRC | 16:21 | |
| *** felipemonteiro_ has quit IRC | 16:21 | |
| *** tphummel has joined #openstack-infra | 16:21 | |
| *** lucasagomes is now known as lucas-afk | 16:21 | |
| *** martinkopec has joined #openstack-infra | 16:24 | |
| *** pcaruana has quit IRC | 16:24 | |
| clarkb | jeblair: pabelanger fungi thanks for th ereviews | 16:24 |
| *** andreas_s has quit IRC | 16:25 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/openstack-zuul-jobs master: WIP: Add tox-linters job https://review.openstack.org/462202 | 16:25 |
| *** shardy has quit IRC | 16:28 | |
| *** martinkopec has quit IRC | 16:28 | |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/infra-specs master: Revise security spec to discuss SSH keys https://review.openstack.org/462207 | 16:29 |
| *** erikmwil_ has quit IRC | 16:31 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: WIP: Move playbook out of zuul https://review.openstack.org/462210 | 16:31 |
| openstackgerrit | Paul Belanger proposed openstack-infra/openstack-zuul-jobs master: WIP: Import playbooks from feature/zuulv3 development https://review.openstack.org/462202 | 16:32 |
| *** vhosakot has joined #openstack-infra | 16:33 | |
| *** mhickey has joined #openstack-infra | 16:34 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: WIP: Add jobs back to .zuul.yaml https://review.openstack.org/462212 | 16:35 |
| *** isaacb_ has joined #openstack-infra | 16:35 | |
| *** Goneri has quit IRC | 16:36 | |
| mtreinish | fungi: so a fun thing I was hitting with https://github.com/mtreinish/pymqttbench is that switching it to use threading instead of multiprocessing DOSes mosquitto (it also has much worse performance) | 16:37 |
| mtreinish | but when I tested that the mosquitto memory usage just grows until it gets OOMKilled | 16:37 |
| *** rlandy|brb is now known as rlandy | 16:37 | |
| mtreinish | I haven't figured out why though | 16:37 |
| *** trown is now known as trown|lunch | 16:38 | |
| *** tesseract has quit IRC | 16:38 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/openstack-zuul-jobs master: WIP: Import playbooks from feature/zuulv3 development https://review.openstack.org/462202 | 16:38 |
| fungi | mtreinish: publisher socket not closing properly maybe? | 16:38 |
| *** felipemonteiro__ has quit IRC | 16:38 | |
| fungi | or are you reusing a socket there? | 16:38 |
| *** felipemonteiro__ has joined #openstack-infra | 16:39 | |
| *** isaacb has quit IRC | 16:39 | |
| *** isaacb_ has quit IRC | 16:39 | |
| mtreinish | fungi: it should close the socket after the publish | 16:40 |
| mtreinish | it's baked into the paho library when use use the publish module | 16:40 |
| *** dtantsur is now known as dtantsur|afk | 16:40 | |
| fungi | ahh | 16:41 |
| mtreinish | fungi: https://github.com/eclipse/paho.mqtt.python/blob/master/src/paho/mqtt/publish.py#L65-L70 | 16:41 |
| fungi | strange! | 16:41 |
| *** armax has quit IRC | 16:43 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/openstack-zuul-jobs master: WIP: Allow openstack-infra/zuul access our jobs https://review.openstack.org/462214 | 16:44 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: WIP: Add jobs back to .zuul.yaml https://review.openstack.org/462212 | 16:44 |
| mtreinish | fungi: oh actually it doesn't the disconnect method doesn't explicitly call close() on the socket. It just sends the disconnect mqtt command | 16:44 |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck master: Add query for rescue instance sec groups test fail bug 1688038 https://review.openstack.org/462215 | 16:44 |
| openstack | bug 1688038 in OpenStack Compute (nova) "test_rescued_vm_add_remove_security_group fails with "InstanceNotRescuable: Instance 4869b462-c3cf-4437-8c94-1d0dcd5fff8b cannot be rescued: Driver Error: failed to connect to monitor socket: No such process"" [Undecided,New] https://launchpad.net/bugs/1688038 | 16:44 |
| *** dkehn has quit IRC | 16:44 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Initial commit https://review.openstack.org/462195 | 16:45 |
| *** gyee has joined #openstack-infra | 16:45 | |
| mtreinish | which means the publisher client is relying on either the broker to close the socket or python to do it automatically | 16:45 |
| fungi | maybe that's lagging and causing a pileup? | 16:45 |
| * fungi needs to disappear for a while. pre-travel errands | 16:45 | |
| fungi | back in an hour or two | 16:46 |
| *** harlowja has joined #openstack-infra | 16:48 | |
| *** markus_z has quit IRC | 16:48 | |
| *** dprince has quit IRC | 16:48 | |
| *** jordanP has joined #openstack-infra | 16:49 | |
| *** hashar has quit IRC | 16:51 | |
| *** armax has joined #openstack-infra | 16:51 | |
| *** EricGonczer_ has joined #openstack-infra | 16:52 | |
| openstackgerrit | Ken Dreyer proposed openstack/python-jenkins master: doc: add get_job_config() example https://review.openstack.org/462219 | 16:56 |
| openstackgerrit | Merged openstack-infra/elastic-recheck master: Add query for rescue instance sec groups test fail bug 1688038 https://review.openstack.org/462215 | 16:56 |
| openstack | bug 1688038 in OpenStack Compute (nova) "test_rescued_vm_add_remove_security_group fails with "InstanceNotRescuable: Instance 4869b462-c3cf-4437-8c94-1d0dcd5fff8b cannot be rescued: Driver Error: failed to connect to monitor socket: No such process"" [Undecided,New] https://launchpad.net/bugs/1688038 | 16:56 |
| *** e0ne has quit IRC | 16:57 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: WIP: Add jobs back to .zuul.yaml https://review.openstack.org/462212 | 16:58 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: WIP: Add jobs back to .zuul.yaml https://review.openstack.org/462212 | 16:59 |
| *** ralonsoh has quit IRC | 17:00 | |
| *** dizquierdo has quit IRC | 17:01 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: WIP: Add jobs back to .zuul.yaml https://review.openstack.org/462212 | 17:01 |
| *** otherwiseguy has joined #openstack-infra | 17:01 | |
| morgan | seeing a spike in 502s on gerrit and very slow respoonses when commenting. | 17:01 |
| morgan | not sure if its something on my end though | 17:02 |
| *** clenimar has quit IRC | 17:02 | |
| clarkb | melody says GCing has picked up | 17:02 |
| *** derekh has quit IRC | 17:02 | |
| *** EricGonczer_ has quit IRC | 17:04 | |
| *** dprince has joined #openstack-infra | 17:04 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: Readded empty group events https://review.openstack.org/462226 | 17:04 |
| *** psachin has quit IRC | 17:05 | |
| *** jpena is now known as jpena|off | 17:06 | |
| *** EricGonczer_ has joined #openstack-infra | 17:07 | |
| openstackgerrit | Merged openstack-infra/openstackid-resources master: Readded empty group events https://review.openstack.org/462226 | 17:07 |
| *** EricGonczer_ has quit IRC | 17:10 | |
| *** Swami has quit IRC | 17:10 | |
| openstackgerrit | Sean Dague proposed openstack-infra/devstack-gate master: Make syslog.txt be more constrained https://review.openstack.org/462231 | 17:18 |
| sdague | clarkb: ^^^ solves the 12M syslog.txt that crashes browsers | 17:18 |
| *** rkukura has quit IRC | 17:19 | |
| clarkb | sdague: will review thanks | 17:20 |
| *** tosky has quit IRC | 17:21 | |
| *** tie has quit IRC | 17:22 | |
| *** trown|lunch is now known as trown | 17:22 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Add empty bindep.txt file https://review.openstack.org/462232 | 17:22 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: WIP: Add jobs back to .zuul.yaml https://review.openstack.org/462212 | 17:24 |
| *** annegentle has quit IRC | 17:24 | |
| *** zul has quit IRC | 17:26 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Add simple tox.ini https://review.openstack.org/462234 | 17:27 |
| AJaeger | pabelanger: your zuul-jobs changes need a tox.ini file, I've started one ^ | 17:27 |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/zuul feature/zuulv3: Add support for bwrap https://review.openstack.org/453851 | 17:27 |
| *** Rockyg has quit IRC | 17:27 | |
| pabelanger | AJaeger: Ya, we're just working on testing zuulv3 right now. But agree | 17:27 |
| AJaeger | pabelanger: your zuulv3 changes need that - the linters job will not run without... | 17:28 |
| pabelanger | AJaeger: agree | 17:28 |
| AJaeger | pabelanger: feel free to abandon them if these are too early... | 17:28 |
| *** zul has joined #openstack-infra | 17:29 | |
| pabelanger | AJaeger: sure, we're just working on getting zuul jobs working again right now | 17:29 |
| pabelanger | but I can review shortly | 17:29 |
| *** vhosakot has quit IRC | 17:29 | |
| mtreinish | sdague: btw, does this address your comments from a couple weeks ago about documenting the firehose interfaces: https://docs.openstack.org/infra/system-config/firehose_schema.html ? | 17:30 |
| sdague | mtreinish: looks mostly like it does | 17:31 |
| mtreinish | sdague: ok, cool | 17:32 |
| *** mat128 has joined #openstack-infra | 17:33 | |
| *** thorst_afk is now known as thorst | 17:34 | |
| *** mhickey has quit IRC | 17:35 | |
| *** Hal has quit IRC | 17:36 | |
| *** mat128 has quit IRC | 17:36 | |
| *** rbrndt has quit IRC | 17:40 | |
| clarkb | sdague: btw I'm working on the real fix for logstash rules. Turns out that indexer.conf.erb file isn't used anymore... | 17:41 |
| *** mat128 has joined #openstack-infra | 17:41 | |
| pabelanger | AJaeger: if you stacked your patches on 462202, you should see jobs running | 17:42 |
| pabelanger | AJaeger: other then noop | 17:42 |
| openstackgerrit | Clark Boylan proposed openstack-infra/logstash-filters master: Handle the new journald based logs https://review.openstack.org/462238 | 17:43 |
| clarkb | sdague: ^ can you look at that? thats where we actually source the rules from. Working on cleaning up the file that confused us now | 17:43 |
| clarkb | pabelanger: fungi ^ you reviewed the other change(s) too sorry for having to go back and forth on this | 17:44 |
| *** mtreinish has quit IRC | 17:46 | |
| AJaeger | pabelanger: I can do it - but guess we need to approve in different order. Let's see, worth the test | 17:47 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Add empty bindep.txt file https://review.openstack.org/462232 | 17:48 |
| pabelanger | AJaeger: right, we don't gate on zuulv3-dev.o.o today. So, we'll be able to merge the changes | 17:48 |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Remove no longer used logstash config https://review.openstack.org/462240 | 17:49 |
| clarkb | pabelanger: fungi ^ thats cleanup to make this less confusing in the future | 17:49 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Add simple tox.ini https://review.openstack.org/462234 | 17:49 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Add untrusted-projects ansible test https://review.openstack.org/461881 | 17:50 |
| clarkb | jesusaur: ^ FYI the logstash things may interest you | 17:51 |
| fungi | clarkb: had the same thing happen to me with the logserver cleanup cronjob. it got halfway migrated to a different repo and never cleaned up | 17:51 |
| fungi | so i ended up making emergency changes to the copy we weren't using. man that was confusing | 17:52 |
| clarkb | ya I was really confused why the configs weren't updating | 17:52 |
| *** mat128 has quit IRC | 17:52 | |
| *** felipemonteiro_ has joined #openstack-infra | 17:53 | |
| clarkb | sdague: we may also need to update the date parser at the end of that file but want to get all of this in before I keep trying to track down bugs | 17:53 |
| openstackgerrit | Merged openstack-infra/logstash-filters master: Handle the new journald based logs https://review.openstack.org/462238 | 17:53 |
| *** hongbin has quit IRC | 17:55 | |
| *** annegentle has joined #openstack-infra | 17:55 | |
| *** felipemonteiro__ has quit IRC | 17:56 | |
| *** baoli has quit IRC | 17:57 | |
| *** e0ne has joined #openstack-infra | 17:59 | |
| *** baoli has joined #openstack-infra | 17:59 | |
| *** annegent_ has joined #openstack-infra | 17:59 | |
| pabelanger | AJaeger: http://zuulv3-dev.openstack.org/logs/b2b8dbe30fcf42f6a9e862630a65bd2c/console.log | 18:00 |
| *** rkukura has joined #openstack-infra | 18:01 | |
| *** annegentle has quit IRC | 18:02 | |
| *** mtreinish has joined #openstack-infra | 18:05 | |
| *** jamesmca_ has quit IRC | 18:07 | |
| openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-jenkins master: Fix beaker on xenial https://review.openstack.org/462048 | 18:08 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Add simple tox.ini https://review.openstack.org/462234 | 18:11 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Add empty bindep.txt file https://review.openstack.org/462232 | 18:12 |
| *** salv-orlando has quit IRC | 18:12 | |
| *** krtaylor has quit IRC | 18:12 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/openstack-zuul-roles master: Initial commit https://review.openstack.org/462244 | 18:14 |
| *** otherwiseguy has quit IRC | 18:14 | |
| AJaeger | pabelanger: https://review.openstack.org/#/c/462234 passes now - so, time to revert dependencies? | 18:15 |
| pabelanger | AJaeger: not just yet. We are still debugging some failure | 18:16 |
| AJaeger | pabelanger: ok. Feel free to do it yourselves if I'm not around... | 18:16 |
| pabelanger | ack, we also need to move things into 462244 | 18:17 |
| *** jordanP has quit IRC | 18:17 | |
| AJaeger | let me copy over tox.ini, bindep, and requirements to zuul-roles | 18:17 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-cgit master: Fix beaker https://review.openstack.org/461815 | 18:18 |
| *** mriedem is now known as mriedem_away | 18:19 | |
| pabelanger | AJaeger: ya, we might be able to land that sooner there. | 18:19 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Add empty bindep.txt file https://review.openstack.org/462247 | 18:20 |
| AJaeger | argh, wrong window ;( | 18:20 |
| openstackgerrit | Paul Belanger proposed openstack-infra/openstack-zuul-roles master: WIP: Initial import of feature/zuulv3 ansible roles https://review.openstack.org/462249 | 18:21 |
| pabelanger | AJaeger: if you want to add bindep.txt to 462244, I don't object | 18:23 |
| AJaeger | pabelanger: one by one... | 18:24 |
| *** david-lyle has quit IRC | 18:24 | |
| *** dklyle has joined #openstack-infra | 18:24 | |
| *** jascott1 has quit IRC | 18:25 | |
| *** jascott1 has joined #openstack-infra | 18:26 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-roles master: Add tox.ini https://review.openstack.org/462253 | 18:26 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-roles master: Add empty bindep.txt https://review.openstack.org/462254 | 18:26 |
| *** srobert has quit IRC | 18:27 | |
| *** rbrndt has joined #openstack-infra | 18:27 | |
| *** jascott1 has quit IRC | 18:27 | |
| clarkb | ok I think logstash rules should be happy | 18:27 |
| clarkb | I had to manually restart the daemons too as puppet doesn't do that automatically | 18:27 |
| openstackgerrit | Merged openstack-infra/devstack-gate master: Collect lvmdump info in logs https://review.openstack.org/461883 | 18:30 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-jenkins master: Fix beaker on xenial https://review.openstack.org/462048 | 18:30 |
| openstackgerrit | Clark Boylan proposed openstack-infra/logstash-filters master: Handle journald timestamps with microseconds https://review.openstack.org/462257 | 18:34 |
| clarkb | confirmed ^ is necessary after getting everything working | 18:34 |
| clarkb | pabelanger: fungi one more review please? :) | 18:34 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-cgit master: Fix beaker https://review.openstack.org/461815 | 18:34 |
| openstackgerrit | Sean Dague proposed openstack-infra/project-config master: move neutron-full-multinode jobs to experimental https://review.openstack.org/462258 | 18:34 |
| sdague | clarkb / AJaeger also ^^ | 18:35 |
| clarkb | sdague: hrm I thought those jobs had runtime about the same as single node | 18:36 |
| clarkb | the extra devstack run is added time but we get it back with tempest running quicker | 18:36 |
| sdague | not the "full" job | 18:36 |
| sdague | that's running all the scenarios | 18:36 |
| sdague | in serial | 18:37 |
| clarkb | oh | 18:37 |
| clarkb | fun | 18:37 |
| *** xarses_ has quit IRC | 18:37 | |
| *** xarses_ has joined #openstack-infra | 18:37 | |
| *** mat128 has joined #openstack-infra | 18:38 | |
| sdague | it's the long pole in the tent. In an ideal world the -nv jobs would post in async so that long running -nv jobs don't hold up important fixes | 18:38 |
| clarkb | we could probably add that pipeline to zuul | 18:39 |
| clarkb | similar to tripleo check | 18:39 |
| *** annegent_ has quit IRC | 18:40 | |
| *** trown is now known as trown|brb | 18:40 | |
| *** annegentle has joined #openstack-infra | 18:41 | |
| *** chem is now known as chem_gone | 18:41 | |
| *** jamesmca_ has joined #openstack-infra | 18:42 | |
| openstackgerrit | Eric Harney proposed openstack-infra/elastic-recheck master: Add Cinder query for bug 1580728 https://review.openstack.org/462265 | 18:44 |
| openstack | bug 1580728 in OpenStack Compute (nova) "UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 386: ordinal not in range(128) in nova.virt.libvirt.vif:unplug with unicode instance.display_name" [Medium,Triaged] https://launchpad.net/bugs/1580728 | 18:44 |
| *** trown|brb is now known as trown | 18:46 | |
| openstackgerrit | Merged openstack-infra/puppet-bandersnatch master: Fix beaker on xenial https://review.openstack.org/461805 | 18:46 |
| sdague | sad slow gerrit is sad and slow :( | 18:47 |
| clarkb | oh ya that was what I was going to do before I had to go eat breakfast | 18:47 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Be less agressive with implied branch matchers https://review.openstack.org/462268 | 18:48 |
| clarkb | I'm going to restart gerrit due to the memory leak | 18:48 |
| clarkb | now is it $status notice or alert that I want? | 18:48 |
| clarkb | er #status | 18:49 |
| jeblair | clarkb: status notice | 18:50 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Be less agressive with implied branch matchers https://review.openstack.org/462268 | 18:50 |
| *** felipemonteiro_ has quit IRC | 18:50 | |
| clarkb | alert is the one that updates channel topics right? | 18:50 |
| *** mat128 has quit IRC | 18:51 | |
| *** felipemonteiro_ has joined #openstack-infra | 18:51 | |
| clarkb | #status notice Gerrit on review.openstack.org is being restarted to accomodate a memory leak in Gerrit. Service should return shortly. | 18:51 |
| openstackstatus | clarkb: sending notice | 18:51 |
| *** annegentle has quit IRC | 18:51 | |
| jeblair | clarkb: right | 18:51 |
| jeblair | clarkb: think of it like "#status red alert" :) | 18:51 |
| clarkb | jeblair: there is an open source remake of those games | 18:51 |
| clarkb | I just discovered it last week | 18:52 |
| -openstackstatus- NOTICE: Gerrit on review.openstack.org is being restarted to accomodate a memory leak in Gerrit. Service should return shortly. | 18:52 | |
| *** slagle has quit IRC | 18:53 | |
| *** krtaylor has joined #openstack-infra | 18:54 | |
| *** annegentle has joined #openstack-infra | 18:54 | |
| *** Sihan has joined #openstack-infra | 18:54 | |
| openstackstatus | clarkb: finished sending notice | 18:54 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Be less agressive with implied branch matchers https://review.openstack.org/462268 | 18:54 |
| *** jamesmca_ has quit IRC | 18:54 | |
| *** electrofelix has quit IRC | 18:55 | |
| Sihan | Hi Guys, in zuul, how could i config some projects vote and some project not vote? | 18:55 |
| openstackgerrit | Merged openstack-infra/project-config master: Add specs and validation jobs for service-types-authority https://review.openstack.org/461208 | 18:55 |
| *** felipemonteiro_ has quit IRC | 18:55 | |
| Sihan | Not sure, could I have two check pipeline? one for voting and one for not voting | 18:56 |
| jeblair | Sihan: yes, that's exactly how | 18:56 |
| jeblair | Sihan: just give them different names, and add the projects you want to vote to one, and the ones you don't want to vote to the other | 18:56 |
| *** jordanP has joined #openstack-infra | 18:56 | |
| *** baoli has quit IRC | 18:57 | |
| jeblair | Sihan: for example, in openstack we have several pre-merge pipelines: check (votes), experimental (leaves a message but does not vote), and silent (does not vote or leave a message). | 18:57 |
| *** baoli has joined #openstack-infra | 18:57 | |
| *** hongbin has joined #openstack-infra | 18:59 | |
| *** florianf has quit IRC | 19:00 | |
| *** gordc has quit IRC | 19:04 | |
| *** hongbin has quit IRC | 19:06 | |
| *** hongbin has joined #openstack-infra | 19:07 | |
| *** otherwiseguy has joined #openstack-infra | 19:09 | |
| *** baoli has quit IRC | 19:10 | |
| *** baoli has joined #openstack-infra | 19:11 | |
| openstackgerrit | Merged openstack-infra/logstash-filters master: Handle journald timestamps with microseconds https://review.openstack.org/462257 | 19:11 |
| jeblair | fungi: have you seen this? telnet mapscii.me | 19:13 |
| *** Sihan has quit IRC | 19:13 | |
| mtreinish | jeblair: that is awesome | 19:14 |
| fungi | Sorry, you reached a full server, please try again. | 19:14 |
| *** Hal has joined #openstack-infra | 19:14 | |
| fungi | i guess they're getting /.ed | 19:14 |
| fungi | got through... either it's very slow or depends on codepoints/colors my xterm is not rendering | 19:15 |
| mtreinish | jeblair: heh, it's a node.js app: https://github.com/rastapasta/mapscii I find something funny about using js for a telnet service :) | 19:16 |
| fungi | yup. does leave you wondering | 19:16 |
| jeblair | mtreinish: yeah... apparently it's cause fetching and manipulating osm vector map tiles is easy in nodejs | 19:17 |
| mtreinish | hmm, I could see that | 19:18 |
| fungi | oho, it's using braille codepoints which either aren't rendering visibly for me or i'm completely lacking a font covering them | 19:18 |
| SotK | that is a wonderful creation | 19:22 |
| mtreinish | pabelanger, clarkb, jeblair: if you get a sec can you take a look at: https://review.openstack.org/449294 and https://review.openstack.org/460730 to add logstash worker events to the firehose | 19:25 |
| *** hashar has joined #openstack-infra | 19:26 | |
| fungi | mtreinish: at this point that's basically identical to what you've gotten working on the subunit-worker systems, right? | 19:27 |
| *** baoli has quit IRC | 19:27 | |
| fungi | so in theory, no new bugs ;) | 19:27 |
| mtreinish | fungi: yeah, it should be | 19:27 |
| *** baoli has joined #openstack-infra | 19:27 | |
| mtreinish | it's not a straight copy and paste though, so I might have screwed something up in the process (that's probably pretty likely) | 19:27 |
| mtreinish | but I think I fixed all the bugs we found doing the subunit-worker | 19:27 |
| clarkb | mtreinish: I will take a look in a bit, currently fixing things for journald formatted logs | 19:28 |
| clarkb | I think the change I am restarting logstash for will be the last | 19:29 |
| clarkb | also I wish syslog/journald used iso8601 timestamps | 19:29 |
| clarkb | so much better | 19:29 |
| mtreinish | clarkb: I don't know how to do that in journald, but rsyslog at least lets you set the template format and you could use iso8601 timestamps in that | 19:31 |
| mtreinish | clarkb: but yeah, it would make life a lot easier | 19:31 |
| clarkb | mtreinish: I read the manpage and couldnt determine a way to do it | 19:31 |
| fungi | journalctl has --output=short-iso | 19:32 |
| clarkb | ooh | 19:33 |
| clarkb | maybe thats what we want | 19:33 |
| mtreinish | clarkb: http://www.rsyslog.com/doc/v8-stable/configuration/templates.html#standard-template-for-writing-to-files | 19:33 |
| clarkb | sdague: ^ thoughts on making that switch? we should be able to handle both at this point | 19:33 |
| mtreinish | there is a dateFormat there | 19:33 |
| clarkb | sdague: I personally like iso8601 as its extremly clear | 19:34 |
| mtreinish | clarkb: also I think RSYSLOG_TraditionalFileFormat is iso8601 timestamps | 19:34 |
| mtreinish | but it's been a long time since I've had to configure rsyslog | 19:34 |
| *** mat128 has joined #openstack-infra | 19:34 | |
| mgagne | pabelanger: we will be performing a maintenance on internap-mtl01 in 30 minutes where Neutron will be updated. Almost all Neutron and Nova API calls will stop working during the maintenance window. | 19:34 |
| clarkb | mgagne: that should be fine if you are ok with us making requests that will fail | 19:35 |
| fungi | popping out to grab late lunch/early dinner before i start packing for boston, but can review logging fixes when i get back in an hour or so | 19:35 |
| mtreinish | clarkb: but yeah doing it directly with journalctl would be better (per fungi's suggestion) | 19:35 |
| mgagne | clarkb: no problem with failures =) | 19:35 |
| mtreinish | fungi: are we out of time for doing more firehose benchmarking (now that we've goot a script that works)? | 19:36 |
| clarkb | mgagne: our garbage collector should swing around and clean up any resources that failed to delete after the fact | 19:36 |
| clarkb | mtreinish: are you really wanting a billion messages a day from logstash processing? | 19:40 |
| mtreinish | clarkb: it is a firehose | 19:40 |
| mtreinish | if it ends up being too much we can always turn it off | 19:40 |
| *** salv-orlando has joined #openstack-infra | 19:41 | |
| clarkb | mtreinish: reviewed the puppet change | 19:46 |
| clarkb | and the processor change | 19:48 |
| mtreinish | clarkb: do you want me to respin the puppet-log_processor patch to pass the cert file creation up a level to system-config? | 19:48 |
| clarkb | mtreinish: the problem is right now its being down by logstash::indexer where we are removing support for mqtt | 19:49 |
| *** mdrabe_ is now known as mdrabe | 19:49 | |
| clarkb | mtreinish: I'm fine if we want to push it down into log_processor instead or just set it in the system-config change logstash_worker.pp | 19:49 |
| clarkb | mtreinish: its really a feature of the config is system-config so may be best to set it there? | 19:49 |
| *** mriedem_away is now known as mriedem | 19:49 | |
| mtreinish | clarkb: ok, I can do that | 19:50 |
| mtreinish | I was conflating puppet-log_processor and logstash::indexer, but I can do the file creation in system-config and push a follow up change removing that option from logstash::indexer | 19:50 |
| *** markvoelker has quit IRC | 19:51 | |
| clarkb | cool | 19:52 |
| *** mat128 has quit IRC | 19:55 | |
| *** gyee has quit IRC | 19:55 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/system-config master: Enable mqtt notifications from logstash gearman worker https://review.openstack.org/460730 | 19:57 |
| mtreinish | clarkb: ^^^ there's the system-config side | 19:58 |
| *** markvoelker has joined #openstack-infra | 19:58 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/puppet-logstash master: Remove the mqtt options from logstash::indexer https://review.openstack.org/462279 | 20:00 |
| mtreinish | clarkb: ^^^ and the puppet-logstash | 20:00 |
| *** jascott1 has joined #openstack-infra | 20:01 | |
| *** ijw has joined #openstack-infra | 20:10 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci master: Add subpackages from openstack-tripleo-common from delorean-current https://review.openstack.org/462067 | 20:10 |
| *** otherwiseguy has quit IRC | 20:17 | |
| *** dprince has quit IRC | 20:19 | |
| *** jordanP has quit IRC | 20:20 | |
| *** e0ne has quit IRC | 20:25 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Add untrusted-projects ansible test https://review.openstack.org/461881 | 20:25 |
| *** mat128 has joined #openstack-infra | 20:27 | |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci master: Support heterogeneous OVB environments https://review.openstack.org/458651 | 20:30 |
| *** tiswanso has quit IRC | 20:32 | |
| *** mat128 has quit IRC | 20:32 | |
| *** ramishra has quit IRC | 20:34 | |
| *** trown is now known as trown|outtypewww | 20:34 | |
| *** tiswanso has joined #openstack-infra | 20:36 | |
| *** krtaylor has quit IRC | 20:39 | |
| *** tiswanso has quit IRC | 20:40 | |
| *** dave-mccowan has quit IRC | 20:41 | |
| *** dhajare has quit IRC | 20:45 | |
| *** Goneri has joined #openstack-infra | 20:45 | |
| fungi | mtreinish: i can still run benchmarks and collect data if you want to build graphs | 20:48 |
| *** dhinesh has joined #openstack-infra | 20:49 | |
| *** dimtruck is now known as zz_dimtruck | 20:49 | |
| mtreinish | fungi: ok cool | 20:49 |
| mtreinish | fungi: yeah I think graphs would be good for the slides | 20:49 |
| fungi | i'm actually not doing much else this week other than summit prep, packing, flying and trying to pitch in when stuff is horribly broken | 20:50 |
| openstackgerrit | Sean McGinnis proposed openstack-infra/project-config master: Clean up stable jobs due to driverfixes filters https://review.openstack.org/462287 | 20:50 |
| smcginnis | clarkb: Hopefully that's right. ^^ | 20:50 |
| clarkb | smcginnis: thanks also caught one with the lvmdump http://logs.openstack.org/76/450276/3/gate/gate-tempest-dsvm-neutron-full-ubuntu-xenial/5f59209/logs/screen-c-vol.txt.gz?level=ERROR digging through that now | 20:52 |
| mtreinish | fungi: ok sure, I'm in a similar mode | 20:56 |
| mtreinish | except s/when stuff is horribly broken/I can't look at slides anymore/g | 20:56 |
| *** salv-orl_ has joined #openstack-infra | 20:56 | |
| clarkb | smcginnis: in that example the cinder service thinks that the volume was deactivated properly. Then lvextend fails because lvextend says it wasn't | 20:57 |
| fungi | mtreinish: heh, nice ;) | 20:57 |
| smcginnis | clarkb: I saw one of those. Looked like the retries on deactivate were fine, but something else failed with the extend. | 20:58 |
| fungi | mtreinish: anyway, i'll give your pymqttbench thing a shot here in a bit. still have my test vm handy | 20:58 |
| mtreinish | fungi: ok cool | 20:58 |
| clarkb | smcginnis: Snapshot origin volumes can be resized only while inactive so lvextend thinks its active not inactive | 20:58 |
| clarkb | smcginnis: definitely something weird going on, especially since its lvextend and cinder both catching this situation | 20:58 |
| mtreinish | fungi: in about 20min I'm gonna step away and run some errands, but I should be back online after that | 20:59 |
| fungi | mtreinish: take your time. it's python, so i'm sure i can figure it out if i need to tweak something | 20:59 |
| smcginnis | clarkb: Looks like the state did change to deactivated. But I wonder if there was some internal state that still wasn't complete. | 20:59 |
| *** tiswanso has joined #openstack-infra | 20:59 | |
| *** salv-orlando has quit IRC | 20:59 | |
| clarkb | smcginnis: the lvmdump pv status shows it is read write visible the entire time | 21:00 |
| clarkb | smcginnis: I'm guessing that at the very least an inactive pv shouldn't be writeable | 21:00 |
| *** mdrabe has quit IRC | 21:00 | |
| smcginnis | clarkb: Let me check what I get when I do that. | 21:00 |
| *** Apoorva_ has joined #openstack-infra | 21:00 | |
| mtreinish | fungi: heh, that's a big part of why I wrote a new one. The go was making it really annoying to debug and iterate on | 21:01 |
| *** askb has joined #openstack-infra | 21:01 | |
| smcginnis | clarkb: Oh wait.... pv status? The lv gets deactivated. The pv should stay rw. | 21:02 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Add untrusted-projects ansible test https://review.openstack.org/461881 | 21:02 |
| clarkb | smcginnis: sorry its the lv | 21:03 |
| clarkb | smcginnis: its the section with the same name as the volume uuid | 21:03 |
| smcginnis | ok | 21:03 |
| *** Apoorva has quit IRC | 21:04 | |
| *** tiswanso has quit IRC | 21:04 | |
| *** zz_dimtruck is now known as dimtruck | 21:04 | |
| clarkb | smcginnis: if you grab the lvmdump file untarball it then grep you'll find it | 21:04 |
| *** eharney has quit IRC | 21:05 | |
| *** Goneri has quit IRC | 21:06 | |
| clarkb | smcginnis: replied to project-config change, one small regex thing otherwise lgtm | 21:06 |
| *** rfolco has quit IRC | 21:09 | |
| openstackgerrit | Sean McGinnis proposed openstack-infra/project-config master: Clean up stable jobs due to driverfixes filters https://review.openstack.org/462287 | 21:09 |
| clarkb | smcginnis: ^ one last thing, have to balance parens | 21:11 |
| clarkb | smcginnis: also double check em on that | 21:11 |
| smcginnis | clarkb: Dang it, I thought I checked that. | 21:11 |
| *** ldnunes has quit IRC | 21:12 | |
| smcginnis | clarkb: Wait, isn't it right? | 21:12 |
| clarkb | smcginnis: let me look at it again | 21:12 |
| *** tphummel has quit IRC | 21:12 | |
| *** hongbin has quit IRC | 21:12 | |
| clarkb | oh yup silly me | 21:12 |
| clarkb | this is why I have people double check me :) | 21:12 |
| smcginnis | whew | 21:12 |
| *** hongbin has joined #openstack-infra | 21:12 | |
| smcginnis | ;) | 21:13 |
| *** thorst has quit IRC | 21:13 | |
| fungi | 2x +2. it's a thing (which i often ignore because we need more reviewers) | 21:13 |
| *** dougwig has quit IRC | 21:15 | |
| clarkb | fungi: can you review https://review.openstack.org/462287 will fix some stable branch job stuff | 21:15 |
| smcginnis | clarkb: Local test after deactivating an lv: status = ["READ", "WRITE", "VISIBLE"] | 21:16 |
| smcginnis | clarkb: Is that what you were seeing? Need to pull down that tgz yet. | 21:16 |
| clarkb | smcginnis: yes thats what I see | 21:16 |
| clarkb | seems odd it would have status write if not active | 21:17 |
| smcginnis | yeah... | 21:17 |
| *** chlong has quit IRC | 21:17 | |
| *** jsavak has joined #openstack-infra | 21:18 | |
| *** krtaylor has joined #openstack-infra | 21:19 | |
| *** gyee has joined #openstack-infra | 21:25 | |
| *** Sukhdev has joined #openstack-infra | 21:25 | |
| *** mat128 has joined #openstack-infra | 21:26 | |
| *** otherwiseguy has joined #openstack-infra | 21:28 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Be less agressive with implied branch matchers https://review.openstack.org/462268 | 21:28 |
| *** mat128 has quit IRC | 21:31 | |
| *** jascott1 has quit IRC | 21:32 | |
| openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-jenkins master: Fix beaker on xenial https://review.openstack.org/462048 | 21:32 |
| clarkb | also lvchange doesn't seem to trigger archival | 21:33 |
| smcginnis | Return code 3 is EINVALID_CMD_LINE. Looking through the code now to see what conditions would get to that. | 21:33 |
| *** esberglu has quit IRC | 21:34 | |
| clarkb | smcginnis: stderr says "Stderr: u" Snapshot origin volumes can be resized only while inactive" so thats why I was going after that | 21:34 |
| smcginnis | clarkb: Oh right. | 21:35 |
| *** rlandy is now known as rlandy|afk | 21:35 | |
| smcginnis | clarkb: So somehow it doesn't think it has actually been deactivated, even though we return from checking that it's deactivated. | 21:35 |
| openstackgerrit | Merged openstack-infra/project-config master: Clean up stable jobs due to driverfixes filters https://review.openstack.org/462287 | 21:35 |
| clarkb | ya | 21:36 |
| *** thorst has joined #openstack-infra | 21:38 | |
| fungi | is that following an lvchange -a n? | 21:39 |
| *** tphummel has joined #openstack-infra | 21:39 | |
| smcginnis | fungi: Yup | 21:39 |
| smcginnis | fungi: We had a batch of issues with it timing out waiting for the volume to actually deactivate. | 21:40 |
| fungi | that i've experienced first-hand | 21:40 |
| smcginnis | fungi: Now it appears to comlpete, but the extend call fails saying it is not. | 21:40 |
| clarkb | from what I can tell cinder does the lvchange -a n, then checks status until it flips to inactive, then extends | 21:40 |
| fungi | that one is definitely new to me though | 21:40 |
| clarkb | in some cases the status check never shows it as inactive and that fails | 21:40 |
| clarkb | in others it says its good and then the extend fails on it | 21:40 |
| clarkb | but guessing its same udnerlying issue in both situations | 21:41 |
| smcginnis | clarkb: See seem to be getting around timing out waiting now. But still wish I knew why it suddenly seems to take so much longer. | 21:41 |
| fungi | maybe we time out waiting for it to go inactive but move on anyway? | 21:41 |
| clarkb | and this lvmdump isn't as hlpeful as I had hoped | 21:41 |
| smcginnis | fungi: That was my though, but at least in this case we do not exhaust our retries. | 21:41 |
| fungi | bizarre | 21:42 |
| smcginnis | fungi: And we would get an error stating if that was the case. | 21:42 |
| smcginnis | clarkb: It's at least data. | 21:42 |
| fungi | wouldn't expect there's some other process in parallel switching it back to active behind its back | 21:42 |
| smcginnis | fungi: I sure hope not. | 21:42 |
| fungi | also, possible there's a kernel bug i guess | 21:43 |
| fungi | (where it could misreport volume state) | 21:43 |
| *** thorst has quit IRC | 21:43 | |
| smcginnis | clarkb: Didn't you say there was a kernel bump? | 21:43 |
| fungi | all of these possibilities are terrible | 21:43 |
| clarkb | smcginnis: there was but it appears to have happened a day or two before we noticed the problem | 21:44 |
| clarkb | thats based on nodepool logs not ubuntus | 21:45 |
| *** jamesmca_ has joined #openstack-infra | 21:46 | |
| cmurphy | this puppet job I'm trying to fix https://review.openstack.org/#/c/462048/ consistently fails with POST_FAILURE, but I've run it locally and confirmed it doesn't take out the test node, so I'm really confused | 21:46 |
| aeng | fedora openid is down? | 21:47 |
| clarkb | cmurphy: one trick is to recheck it then connect via telnet to the console log port and watch at what step it dies | 21:48 |
| cmurphy | clarkb: oh good idea | 21:49 |
| *** jsavak has quit IRC | 21:49 | |
| smcginnis | About how long after a project-config change merges until that gets picked up by zuul? | 21:51 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Report job shadow errors to users https://review.openstack.org/462300 | 21:51 |
| clarkb | smcginnis: 15 minutes to 45 minutes depending where inthe ansible puppet run things are | 21:52 |
| smcginnis | clarkb: thanks | 21:52 |
| clarkb | smcginnis: you'll see the zuul status page update its last reconfigured timestamp at the bottom of the page | 21:52 |
| smcginnis | clarkb: Oh right, forgot about that. | 21:52 |
| aeng | sorry, wrong channel..lol | 21:53 |
| *** jascott1 has joined #openstack-infra | 21:54 | |
| *** Julien-zte has quit IRC | 21:54 | |
| cmurphy | clarkb: hmm it ends successfully after gzipping logs | 21:59 |
| cmurphy | and yet | 21:59 |
| clarkb | cmurphy: and then post failure? | 21:59 |
| cmurphy | yep | 21:59 |
| clarkb | so its likely related to copying the logs | 21:59 |
| clarkb | cmurphy: do you have the ip address you telnet'd to ? I can check zuul execturos | 22:00 |
| cmurphy | clarkb: 2001:4800:1ae1:18:f816:3eff:fef9:433 19885 | 22:00 |
| cmurphy | clarkb: http://paste.openstack.org/show/608761/ | 22:00 |
| clarkb | cmurphy: http://paste.openstack.org/show/608763/ its permissions | 22:04 |
| clarkb | I think tripleo ran into this too | 22:04 |
| clarkb | cmurphy: you need to make sure the jenkins user can read the logs source? | 22:04 |
| *** jkilpatr has quit IRC | 22:05 | |
| *** ijw has quit IRC | 22:06 | |
| *** ijw has joined #openstack-infra | 22:06 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: TEST: DONT RECHECK or REVIEW: periodic jobs https://review.openstack.org/359215 | 22:06 |
| cmurphy | ooh because the module is messing with the jenkins user | 22:07 |
| cmurphy | thanks clarkb | 22:07 |
| *** gouthamr has quit IRC | 22:07 | |
| *** wolverineav has quit IRC | 22:07 | |
| *** jamesdenton has quit IRC | 22:08 | |
| *** wolverineav has joined #openstack-infra | 22:08 | |
| *** baoli has quit IRC | 22:09 | |
| *** jamesmca_ has quit IRC | 22:09 | |
| *** Apoorva_ has quit IRC | 22:10 | |
| *** Apoorva has joined #openstack-infra | 22:10 | |
| *** ijw has quit IRC | 22:11 | |
| *** ijw has joined #openstack-infra | 22:11 | |
| *** marst has quit IRC | 22:12 | |
| *** jamesmca_ has joined #openstack-infra | 22:14 | |
| *** wolverineav has quit IRC | 22:15 | |
| clarkb | jeblair: patch for | in presentty works great, thanks | 22:18 |
| jeblair | clarkb: yay! | 22:18 |
| clarkb | jeblair: ther eisn't a good way to pop out of a presentation then rsume at the slide you left off is there? | 22:21 |
| *** knangia has quit IRC | 22:21 | |
| clarkb | not a big deal can skip ahead | 22:21 |
| jeblair | clarkb: no; i start up the presenters console and use it to jump | 22:22 |
| clarkb | jeblair: I think I can actually just switch terminals | 22:22 |
| jeblair | clarkb: oh, like for a demo? yeah that's what i usually do | 22:22 |
| *** warpc_ has joined #openstack-infra | 22:23 | |
| clarkb | yup | 22:23 |
| *** cmurphy_ has joined #openstack-infra | 22:23 | |
| clarkb | and this looks like it will work great | 22:23 |
| *** jamesmca_ has quit IRC | 22:23 | |
| *** Sukhdev has quit IRC | 22:23 | |
| *** jascott1_ has joined #openstack-infra | 22:23 | |
| *** ediardo_ has joined #openstack-infra | 22:23 | |
| *** mgkwill_ has joined #openstack-infra | 22:24 | |
| *** mnencia has quit IRC | 22:24 | |
| *** EricGonczer_ has joined #openstack-infra | 22:25 | |
| *** abadger2000 has joined #openstack-infra | 22:25 | |
| *** jgrassle1 has joined #openstack-infra | 22:25 | |
| *** mikal_ has joined #openstack-infra | 22:26 | |
| *** hashar_ has joined #openstack-infra | 22:26 | |
| *** mnencia has joined #openstack-infra | 22:26 | |
| *** SotK_ has joined #openstack-infra | 22:26 | |
| *** cmurphy has quit IRC | 22:26 | |
| *** cmurphy_ is now known as cmurphy | 22:26 | |
| *** apetrich_ has joined #openstack-infra | 22:27 | |
| *** alex_xu_ has joined #openstack-infra | 22:27 | |
| *** verdurin_ has joined #openstack-infra | 22:27 | |
| *** nibz has joined #openstack-infra | 22:27 | |
| *** mariojv_ has joined #openstack-infra | 22:27 | |
| *** rodrigod` has joined #openstack-infra | 22:27 | |
| *** slagle has joined #openstack-infra | 22:27 | |
| *** tdasilva- has joined #openstack-infra | 22:28 | |
| *** bexelbie- has joined #openstack-infra | 22:28 | |
| *** erlon has quit IRC | 22:28 | |
| *** alex_xu has quit IRC | 22:28 | |
| *** mgkwill has quit IRC | 22:28 | |
| *** ediardo has quit IRC | 22:28 | |
| *** bexelbie has quit IRC | 22:28 | |
| *** rodrigods has quit IRC | 22:28 | |
| *** jgrassler has quit IRC | 22:28 | |
| *** jascott1 has quit IRC | 22:28 | |
| *** mikal has quit IRC | 22:28 | |
| *** apetrich has quit IRC | 22:28 | |
| *** verdurin has quit IRC | 22:28 | |
| *** mariojv has quit IRC | 22:28 | |
| *** jesusaur has quit IRC | 22:28 | |
| *** SotK has quit IRC | 22:28 | |
| *** puiterwijk|cld has quit IRC | 22:28 | |
| *** warpc has quit IRC | 22:28 | |
| *** nibalizer has quit IRC | 22:28 | |
| *** hashar has quit IRC | 22:28 | |
| *** harlowja has quit IRC | 22:28 | |
| *** tdasilva has quit IRC | 22:28 | |
| *** xgerman has quit IRC | 22:28 | |
| *** abadger1999 has quit IRC | 22:28 | |
| *** d0ugal has quit IRC | 22:28 | |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck master: Remove stale queries for invalid or fixed bugs https://review.openstack.org/462308 | 22:28 |
| *** mgkwill_ is now known as mgkwill | 22:28 | |
| *** bexelbie- is now known as bexelbie | 22:29 | |
| *** warpc_ is now known as warpc | 22:29 | |
| *** jesusaur has joined #openstack-infra | 22:29 | |
| *** ediardo_ is now known as ediardo | 22:29 | |
| clarkb | also friendly reminder to double check you aren't vulnerable to amt mess before conferencing | 22:30 |
| * clarkb is about to reboot as soon as packge updates are done t ofigure that out | 22:30 | |
| *** gouthamr has joined #openstack-infra | 22:30 | |
| *** nibz is now known as nibalizer | 22:31 | |
| *** salv-orl_ has quit IRC | 22:32 | |
| fungi | cmurphy: clarkb: if this permissions issue persists in zuul v3 i think we should probably take steps to blanket normalize permissions at the end of jobs or something | 22:36 |
| *** jkilpatr has joined #openstack-infra | 22:39 | |
| * mtreinish has returned | 22:42 | |
| *** baoli has joined #openstack-infra | 22:43 | |
| *** baoli has quit IRC | 22:43 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: CI test - never merge https://review.openstack.org/461001 | 22:44 |
| *** baoli has joined #openstack-infra | 22:44 | |
| *** xyang1 has quit IRC | 22:47 | |
| *** rbrndt has quit IRC | 22:51 | |
| openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-jenkins master: Fix beaker on xenial https://review.openstack.org/462048 | 22:51 |
| *** annegentle has quit IRC | 22:51 | |
| *** rbrndt has joined #openstack-infra | 22:51 | |
| *** rbrndt has quit IRC | 22:51 | |
| cmurphy | fungi: I think in this case it was just the puppet-jenkins module changing the jenkins user's authorized_keys | 22:52 |
| cmurphy | it wouldn't generally be a problem for anything else | 22:52 |
| *** d0ugal has joined #openstack-infra | 22:53 | |
| fungi | cmurphy: i would be inclined to agree _if_ it were the first time we've seen this | 22:53 |
| fungi | but there have been other jobs which made files unreadable by the user we're copying them as, resulting in "mysterious" POST_FAILURE job states | 22:53 |
| fungi | i think it counts as a sharp edge we should file down (if it's still around come zuul v3) | 22:54 |
| cmurphy | fungi: it wasn't an unreadable file, the file's contents were literally set to the string 'sshkey' | 22:54 |
| fungi | oh, hah. so it broke ssh access entirely? | 22:55 |
| cmurphy | yep | 22:55 |
| fungi | that's an ice cream of a different flavor entirely | 22:55 |
| cmurphy | i would hope no other project has a good reason to do something like that | 22:55 |
| fungi | changing my opinion, we might want some other means of signaling "zuul couldn't connect to the worker to retrieve the logs" distinct from other general failures at that stage | 22:56 |
| openstackgerrit | Merged openstack-infra/elastic-recheck master: Remove stale queries for invalid or fixed bugs https://review.openstack.org/462308 | 22:58 |
| openstackgerrit | Hongbin Lu proposed openstack-infra/project-config master: [Kuryr] Make fullstack jobs voting https://review.openstack.org/461911 | 23:00 |
| *** pradk has quit IRC | 23:00 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Store initial repo state in the merger https://review.openstack.org/461176 | 23:01 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Use previously stored repo state on executor https://review.openstack.org/461177 | 23:01 |
| *** sdague has quit IRC | 23:02 | |
| *** lamt has quit IRC | 23:02 | |
| *** lamt has joined #openstack-infra | 23:02 | |
| *** xarses has joined #openstack-infra | 23:03 | |
| *** xarses_ has quit IRC | 23:03 | |
| *** andreww has quit IRC | 23:03 | |
| *** csomerville has joined #openstack-infra | 23:04 | |
| *** hashar_ has quit IRC | 23:05 | |
| *** lamt has quit IRC | 23:05 | |
| *** lamt has joined #openstack-infra | 23:06 | |
| *** lamt has quit IRC | 23:07 | |
| *** hongbin has quit IRC | 23:08 | |
| *** cody-somerville has quit IRC | 23:09 | |
| *** larainema has joined #openstack-infra | 23:10 | |
| *** ym200 has joined #openstack-infra | 23:11 | |
| *** annegentle has joined #openstack-infra | 23:11 | |
| *** EricGonc_ has joined #openstack-infra | 23:11 | |
| *** EricGonczer_ has quit IRC | 23:11 | |
| *** annegentle has quit IRC | 23:12 | |
| *** EricGonc_ has quit IRC | 23:13 | |
| openstackgerrit | Merged openstack-infra/devstack-gate master: Only hardlink files from cache_dir if it exists https://review.openstack.org/459763 | 23:14 |
| *** EricGonczer_ has joined #openstack-infra | 23:14 | |
| *** ramishra has joined #openstack-infra | 23:15 | |
| *** rkukura has quit IRC | 23:16 | |
| *** jamesdenton has joined #openstack-infra | 23:16 | |
| *** funzo has joined #openstack-infra | 23:21 | |
| *** dklyle has quit IRC | 23:23 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/nodepool master: Remove timestamp in devstack plugin https://review.openstack.org/462319 | 23:23 |
| *** ym200 has quit IRC | 23:23 | |
| *** rfolco has joined #openstack-infra | 23:24 | |
| *** rfolco has quit IRC | 23:24 | |
| *** jamesdenton has quit IRC | 23:25 | |
| *** gildub has joined #openstack-infra | 23:25 | |
| *** funzo has quit IRC | 23:26 | |
| *** jamesdenton has joined #openstack-infra | 23:26 | |
| *** benj_ has quit IRC | 23:28 | |
| *** harlowja has joined #openstack-infra | 23:31 | |
| *** benj_ has joined #openstack-infra | 23:31 | |
| *** lamt has joined #openstack-infra | 23:34 | |
| *** Apoorva_ has joined #openstack-infra | 23:37 | |
| *** Apoorva__ has joined #openstack-infra | 23:40 | |
| *** Apoorva_ has quit IRC | 23:40 | |
| *** Apoorva has quit IRC | 23:41 | |
| *** gcb has quit IRC | 23:42 | |
| *** ramishra has quit IRC | 23:46 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/nodepool master: Turn of time stamps in dib log https://review.openstack.org/462321 | 23:52 |
| *** ijw has quit IRC | 23:52 | |
| *** ijw has joined #openstack-infra | 23:52 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Report job shadow errors to users https://review.openstack.org/462300 | 23:52 |
| *** lamt has quit IRC | 23:54 | |
| *** ijw has quit IRC | 23:55 | |
| *** lamt has joined #openstack-infra | 23:55 | |
| *** ijw has joined #openstack-infra | 23:57 | |
| *** ijw has joined #openstack-infra | 23:58 | |
| *** thorst has joined #openstack-infra | 23:58 | |
| *** thorst has quit IRC | 23:58 | |
| jeblair | ianw: do you have a link to a problem logfile? | 23:58 |
| *** masber has joined #openstack-infra | 23:58 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!