| jeblair | https://jenkins.openstack.org/job/wheel-build-ubuntu-trusty-amd64/1/console | 00:00 |
|---|---|---|
| jeblair | wheel builds are not fast | 00:00 |
| *** amitgandhinz has joined #openstack-infra | 00:00 | |
| Sam-I-Am | fungi: apparently i'm good at trying to submit review comments when backups kick off | 00:01 |
| prometheanfire | Sam-I-Am: you +w it kthnx | 00:01 |
| clarkb | greghaynes: jeblair http://paste.openstack.org/show/485905/ found it | 00:03 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/release-tools: remove the announce flag from the release wrapper scripts https://review.openstack.org/275986 | 00:03 |
| greghaynes | clarkb: that is post-disconnect but confirms that it disconnected | 00:03 |
| clarkb | yes I don't tink we actually see the disconnect in our logs due to the log level | 00:03 |
| clarkb | we only see the side effect | 00:04 |
| openstackgerrit | Ian Wienand proposed openstack-infra/puppet-nodepool: Add periodic cleanup of log files https://review.openstack.org/275968 | 00:04 |
| greghaynes | Last time I looked at the code for that I want to say gearman client would bubble up the exception | 00:04 |
| *** rbrndt_ has quit IRC | 00:04 | |
| jeblair | there ought to be a failed status command, but nodepool may gracefully handle that | 00:05 |
| openstackgerrit | Merged openstack-infra/git-restack: Remove argparse from requirements https://review.openstack.org/270411 | 00:05 |
| greghaynes | Yea, and maybe we fail to log it :( | 00:05 |
| openstackgerrit | Merged openstack-infra/git-restack: Correct metadata URLs https://review.openstack.org/275444 | 00:05 |
| clarkb | I definitely haven't been able to find it if we do | 00:05 |
| jeblair | self.__log.exception("Exception while listing functions") | 00:05 |
| jeblair | clarkb: check for that ^ | 00:06 |
| clarkb | http://paste.openstack.org/show/485906/ happens after that first error I found | 00:06 |
| greghaynes | Silly threads ;) | 00:07 |
| *** amitgandhinz has quit IRC | 00:07 | |
| *** rguillebert has quit IRC | 00:07 | |
| clarkb | but I think that is a nother symptom | 00:07 |
| clarkb | maybe? | 00:07 |
| jeblair | greghaynes: i'm not sure that's silly threads | 00:07 |
| greghaynes | It could be, depends if gear client knows to error on no servers for that method... | 00:08 |
| jeblair | greghaynes: we should at least not assume that didn't actually happen in the reported order | 00:08 |
| greghaynes | Yep, it could have | 00:08 |
| *** yuanying has quit IRC | 00:09 | |
| openstackgerrit | Khai Do proposed openstack-infra/puppet-gerrit: Allow configuration of Gerrit httpd.maxQueued option https://review.openstack.org/275988 | 00:10 |
| clarkb | I can start a tcpdump now and see how fast it grows | 00:10 |
| clarkb | and stop it in an hour if it is too large | 00:10 |
| clarkb | looks like you can post rotate gzip | 00:11 |
| greghaynes | Iirc there is an arg for max file size | 00:11 |
| clarkb | yup or time based | 00:12 |
| *** sridhar_ram1 has joined #openstack-infra | 00:12 | |
| clarkb | time based is nice isnce that is what we get from the logs | 00:12 |
| jeblair | greghaynes, clarkb: 1) nodepool main thread -> geard "status"; 2) disconnection happens; 3) geard notices broken pipe; 4) nodepool build thread -> geard "submit_job" -- returns error 5) nodepool build thread marks connection as failed; 6) nodepool build thread tries to submit to next available connection, there is none, it raises NoConnectedServersError; 7) nodepool admin thread finally times out waiting for a response from its ... | 00:13 |
| jeblair | ... long-dead connection. | 00:13 |
| openstackgerrit | Merged openstack/requirements: Updated from generate-constraints https://review.openstack.org/273907 | 00:13 |
| jeblair | greghaynes, clarkb: i think that explains the order | 00:13 |
| openstackgerrit | Merged openstack-infra/project-config: Rename .log to .log.txt in run-tox https://review.openstack.org/203099 | 00:14 |
| *** sridhar_ram has quit IRC | 00:15 | |
| openstackgerrit | Merged openstack-infra/nodepool: Optimize "open" method with context manager https://review.openstack.org/250658 | 00:15 |
| clarkb | tcpdump -i eth0 -s 1500 -w 'trace_%Y-%m-%d_%H:%M:%S.pcap' -W 24 -G 3600 port 4730 | 00:15 |
| clarkb | does that look good ? | 00:15 |
| clarkb | maybe add a -z gzip | 00:15 |
| *** yuanying has joined #openstack-infra | 00:16 | |
| clarkb | I can do that in /opt to prevent hosing the service too bad if it grows out of control | 00:16 |
| cody-somerville | jeblair: re: mutex, isn't there (or wasn't there) already a way to say only a single instance of this job should run at a time? (used to be used for publish jobs but I don't see any evidence of that now) | 00:17 |
| * greghaynes lacks man pages on phone but seems ok | 00:17 | |
| * clarkb tries it and can stop it in an hour if it looks terrible | 00:17 | |
| greghaynes | Ya, if it gets too unwieldy we could lower the -S too | 00:18 |
| jeblair | cody-somerville: no; i've suggested we need a new pipeline manager that drops duplicate items (for doc publishing) | 00:18 |
| jeblair | cody-somerville: has not been written yet | 00:18 |
| *** tzn has joined #openstack-infra | 00:19 | |
| *** PsionTheory has quit IRC | 00:19 | |
| *** yuanying has quit IRC | 00:19 | |
| cody-somerville | Was there something that would serialize certain jobs or something? I distinctly remember something along that line because I think it would make people wonder why jobs weren't running. | 00:19 |
| cody-somerville | I see hold-following-changes option in the docs but that doesn't feel like what I'm trying to remember. | 00:21 |
| *** yuanying has joined #openstack-infra | 00:22 | |
| *** mtanino has quit IRC | 00:22 | |
| clarkb | we are up to 15 MB | 00:22 |
| clarkb | thats for about 5 minutes | 00:22 |
| * clarkb maths | 00:22 | |
| jeblair | cody-somerville: there's that; there's the job trees; and there's the implicit serialization if jenkins only has one node of a type | 00:22 |
| clarkb | 180MB per hour? that should be doable | 00:23 |
| *** r-daneel has joined #openstack-infra | 00:23 | |
| openstackgerrit | Thanh Ha proposed openstack-infra/jenkins-job-builder: Fix delete-all command to include all job types https://review.openstack.org/275993 | 00:25 |
| jlvillal | greghaynes: Is there documentation about network_info.json file format, other than: https://github.com/openstack-infra/glean/blob/master/glean/cmd.py#L365-L400 ? | 00:25 |
| jlvillal | I saw you listed as a core on the glean project. | 00:26 |
| clarkb | jlvillal: yes rackspace/novad document it | 00:26 |
| openstackgerrit | Thanh Ha proposed openstack-infra/jenkins-job-builder: Fix delete-all command to include all job types https://review.openstack.org/275993 | 00:26 |
| jlvillal | clarkb: Ah great. Let me look for that. Thanks :) | 00:26 |
| clarkb | jlvillal: it gre wout of rackspace's ironic deployment iirc but was add ed to nova | 00:27 |
| clarkb | jlvillal: glean is just a cnsumer | 00:27 |
| * jlvillal tries to parse. Thinks clarkb is on a phone. | 00:27 | |
| *** achanda has joined #openstack-infra | 00:27 | |
| clarkb | sorry actually have baby on my lap | 00:27 |
| jlvillal | :D | 00:28 |
| clarkb | it grew out of rackspaces ironic deployment | 00:28 |
| clarkb | but the code was added to nova | 00:28 |
| jlvillal | Ah! | 00:28 |
| clarkb | glean is just the consumer | 00:28 |
| cody-somerville | jeblair: oh, I think I'm thinking of concurrent: false in the jjb file but I suppose that was removed since it probably doesn't work with multi-master setup eh? | 00:28 |
| jeblair | cody-somerville: yep | 00:28 |
| jeblair | clarkb, greghaynes: i have a theory. it's the same as the last theory i had, but it's still valid. we're seeing the results of thread starvation. | 00:30 |
| jeblair | clarkb, greghaynes: annotated log lines: http://paste.openstack.org/show/485909/ | 00:30 |
| openstackgerrit | Khai Do proposed openstack-infra/system-config: Increase the HTTP incoming connection queue https://review.openstack.org/275995 | 00:30 |
| jeblair | the submit delete job happens _before_ the admin request, yet geard is handling the admin request during the disconnect | 00:30 |
| zaro | clarkb, jeblair: gerrit tuning ^ | 00:30 |
| jeblair | that suggests to me that the delete job submission _was_ handled | 00:31 |
| jeblair | yet nodepool timed out on it | 00:31 |
| clarkb | jeblair: it didn't provide enough cycles t oget those bytes out? | 00:31 |
| jeblair | which suggests that nodepool didn't actually process the data in that thread within the timeout window | 00:31 |
| jeblair | clarkb: yeah, and the timeout actually woke up the thread which, if it had managed to process the data queued for it, would have been satisfied | 00:32 |
| jlvillal | clarkb: jroll was nice enough to point me to a spec which seemed to have decent info: http://specs.openstack.org/openstack/nova-specs/specs/liberty/implemented/metadata-service-network-info.html#rest-api-impact | 00:35 |
| jlvillal | clarkb: Thanks | 00:35 |
| *** tzn has quit IRC | 00:35 | |
| *** sdake has joined #openstack-infra | 00:35 | |
| cody-somerville | Where are patch dependencies in gerrit shown these days? | 00:35 |
| jlvillal | cody-somerville: On the right, upper part | 00:36 |
| jlvillal | I believe | 00:36 |
| clarkb | cody-somerville: on the right as related changes | 00:36 |
| clarkb | order is top is child bottom is parent | 00:36 |
| cody-somerville | so related changes *are* dependencies, it doesn't possibly list "related" some other way (such as being mentioned in commit message or something)? | 00:37 |
| clarkb | cody-somerville: it does same topic too iirc | 00:37 |
| clarkb | its definitely not as useable as the old screen | 00:37 |
| clarkb | and if there is a tree you just see a linear list thing which is ocnfusing | 00:38 |
| clarkb | it needs to be a graph | 00:38 |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool: Increase gearman timeouts https://review.openstack.org/275997 | 00:38 |
| ianw | clarkb: has the builder restarted with the new logging config file? i'm struggling to see anything wrong | 00:38 |
| clarkb | ianw: I believe it has but maybe not | 00:39 |
| ianw | file looks ok, daemon args look ok | 00:39 |
| clarkb | nodepool builder started on january 29 | 00:39 |
| *** cburgess_ has quit IRC | 00:39 | |
| clarkb | I wonder if we just have the keys wrong for the names of loggers? | 00:39 |
| clarkb | which may have happened in the builer renamed/moved things | 00:40 |
| clarkb | oh ya qualname for the image log | 00:40 |
| jeblair | clarkb, greghaynes: https://review.openstack.org/275997 | 00:40 |
| *** Sukhdev has quit IRC | 00:40 | |
| *** chlong has joined #openstack-infra | 00:40 | |
| jeblair | krotscheck is gone: https://jenkins.openstack.org/job/wheel-build-ubuntu-trusty-amd64/1/console | 00:40 |
| *** cburgess has joined #openstack-infra | 00:42 | |
| ianw | clarkb: hmm, a -builder in there now? https://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/templates/nodepool/nodepool-builder.logging.conf.erb#n280 | 00:42 |
| clarkb | ianw: I am oduble checking | 00:42 |
| ianw | clarkb: we have "log = logging.getLogger("nodepool.image.build.%s" ..." | 00:43 |
| ianw | in the builder | 00:43 |
| ianw | hmm, also, the latest update was merged 2016-02-01 21:01:23 (GMT) , which is after jan 29 | 00:45 |
| *** fitoduarte has quit IRC | 00:46 | |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config: Fix typo in wheel-index script https://review.openstack.org/275999 | 00:46 |
| *** daemontool has joined #openstack-infra | 00:47 | |
| *** daemontool_ has quit IRC | 00:48 | |
| jeblair | fungi, krotscheck: i manually released the wheel mirror -- http://mirror.nyj01.internap.openstack.org/wheel/ubuntu-14.04-x86_64/a/ | 00:48 |
| jeblair | the full release time was 3.5 minutes, the noop release time is 18s | 00:48 |
| clarkb | ianw: so ya may just need to restart the service | 00:48 |
| clarkb | ianw: I can do that now | 00:48 |
| ianw | clarkb: ok, there's a cleanup in https://review.openstack.org/#/c/275982/ but waiting on ci | 00:49 |
| jeblair | fungi, krotscheck: i've approved the vos release job for the mirror | 00:49 |
| jeblair | fungi, krotscheck: we need to land https://review.openstack.org/275999 and then we can see if the wheel mirrors actually work | 00:50 |
| *** sdake_ has joined #openstack-infra | 00:50 | |
| openstackgerrit | venkatamahesh proposed openstack-infra/system-config: Fix spell typos https://review.openstack.org/276000 | 00:51 |
| *** jpr has joined #openstack-infra | 00:52 | |
| clarkb | ianw: restarted | 00:54 |
| *** sdake has quit IRC | 00:54 | |
| *** pavel_bondar has quit IRC | 00:54 | |
| clarkb | we are up to 115MB on that pcap file | 00:54 |
| *** Swami has quit IRC | 00:55 | |
| *** rockyg has quit IRC | 00:56 | |
| openstackgerrit | Merged openstack-infra/project-config: Added wheel-release job https://review.openstack.org/273549 | 00:57 |
| *** Kennan is now known as Kennan_Vacation | 00:59 | |
| openstackgerrit | John L. Villalovos proposed openstack-infra/glean: Add link to documentation on network_info.json https://review.openstack.org/276003 | 00:59 |
| *** bpokorny has quit IRC | 01:00 | |
| *** jamielennox is now known as jamielennox|away | 01:01 | |
| clarkb | I am going to try and clear the backlog of branch deletion requests on the infra list now | 01:02 |
| *** eil397 has quit IRC | 01:04 | |
| *** pavel_bondar has joined #openstack-infra | 01:07 | |
| ianw | if i ever submit a spelling correction change, i'm going to make sure i subtly misspell something in the change description just to troll | 01:07 |
| prometheanfire | ianw: hi :P | 01:08 |
| *** shashank_hegde has quit IRC | 01:08 | |
| clarkb | ianw: mispelling is my specailty | 01:08 |
| prometheanfire | ianw: reworkflowplease https://review.openstack.org/#/c/273790/ | 01:08 |
| *** sflanigan has quit IRC | 01:09 | |
| *** ybathia has quit IRC | 01:10 | |
| jeblair | clarkb: letting the babies type for you? | 01:11 |
| clarkb | jeblair: that is one of my methods | 01:11 |
| *** yuanying has quit IRC | 01:11 | |
| fungi | jeblair: awesome. that's a reasonable amount of time for releasing the updates to that volume, i guess | 01:11 |
| clarkb | ok I got through the first of the branch delete requets, am being pulled away before I can get to the others | 01:12 |
| clarkb | will try to follow up in the morning | 01:12 |
| *** gibi has quit IRC | 01:12 | |
| *** yuanying has joined #openstack-infra | 01:12 | |
| *** sridhar_ram1 has quit IRC | 01:12 | |
| fungi | clarkb: oh, thanks for working on branch deletions. i've actually been accumulating a list of all the ones which have been requested but haven't found time to actually do them | 01:12 |
| *** pvaneck has quit IRC | 01:13 | |
| openstackgerrit | Apoorva Deshpande proposed openstack-infra/ciwatch: Ability to select specific CI results https://review.openstack.org/274412 | 01:13 |
| Sam-I-Am | dhellmann: ping | 01:14 |
| clarkb | fungi: well I thought I had time now but family has distracted me | 01:15 |
| clarkb | will likely be able to continue tomorrow though | 01:15 |
| clarkb | and pcap isn't going too crazy so I think we can leave it overnight | 01:15 |
| *** sputnik13 has quit IRC | 01:15 | |
| *** gibi has joined #openstack-infra | 01:16 | |
| fungi | yeah, that's a much slower growth rate than i was concerned it might be | 01:17 |
| clarkb | fungi: did ytou see my response about muranoclient stable/juno? they don't have a tag on that commit so I left it in place | 01:18 |
| fungi | no, hadn't seen it yet. also there are a few from post-juno-eol that i was going to tag | 01:19 |
| clarkb | the others don't have juno-eol tags but have tags so I suggested possibly adding juno-eol tags | 01:20 |
| clarkb | but went ahead and dleted the branches as we don't need to worry about GC remove things | 01:20 |
| *** gibi has quit IRC | 01:20 | |
| fungi | openstack-manuals is one | 01:20 |
| fungi | oh, and also openstack/requirements still had a stable/icehouse that needed eol-tagging and deletion | 01:20 |
| *** FallenPegasus has quit IRC | 01:21 | |
| *** yuanying has quit IRC | 01:21 | |
| *** nikhil has quit IRC | 01:21 | |
| *** gibi has joined #openstack-infra | 01:22 | |
| *** piet has joined #openstack-infra | 01:22 | |
| *** Jeffrey4l has joined #openstack-infra | 01:22 | |
| *** yuanying has joined #openstack-infra | 01:23 | |
| jpr | so I got my ci toolchain operational today. (nodepool builds images, registers and launches them, jenkins talks to them). yay! now I'm working with the openstack-dev/sandbox so I can observe a commit flowing through the pipeline | 01:23 |
| *** kzaitsev_mb has quit IRC | 01:23 | |
| *** gildub_ has joined #openstack-infra | 01:24 | |
| clarkb | jpr: woot | 01:24 |
| *** kzaitsev_mb has joined #openstack-infra | 01:25 | |
| jpr | clarkb: thanks. nice to see the parts talking happy. couple of issue i'll report up but generally seems smooth. :) | 01:27 |
| *** gildub has quit IRC | 01:27 | |
| *** gildub_ has quit IRC | 01:27 | |
| *** gildub has joined #openstack-infra | 01:27 | |
| *** bmjen has quit IRC | 01:29 | |
| *** crinkle_ has joined #openstack-infra | 01:30 | |
| *** crinkle has quit IRC | 01:31 | |
| *** Hunner has quit IRC | 01:31 | |
| *** crinkle_ is now known as crinkle | 01:31 | |
| *** sdake_ has quit IRC | 01:33 | |
| *** piet has quit IRC | 01:35 | |
| *** bmjen has joined #openstack-infra | 01:35 | |
| *** kzaitsev_mb has quit IRC | 01:35 | |
| *** piet has joined #openstack-infra | 01:35 | |
| *** piet has quit IRC | 01:35 | |
| *** nikhil has joined #openstack-infra | 01:35 | |
| *** piet has joined #openstack-infra | 01:36 | |
| *** Hunner has joined #openstack-infra | 01:36 | |
| *** Hunner has quit IRC | 01:36 | |
| *** Hunner has joined #openstack-infra | 01:36 | |
| *** tiswanso has joined #openstack-infra | 01:36 | |
| openstackgerrit | Masahito Muroi proposed openstack-infra/project-config: Adds a devstack test job using Congress new architecture https://review.openstack.org/275514 | 01:38 |
| openstackgerrit | Masahito Muroi proposed openstack-infra/project-config: Adds a tox test job for Congress new architecture https://review.openstack.org/275515 | 01:38 |
| *** jpr has quit IRC | 01:38 | |
| *** weshay_xchat has joined #openstack-infra | 01:38 | |
| *** Jeffrey4l has quit IRC | 01:39 | |
| *** Daisy has joined #openstack-infra | 01:40 | |
| *** r-daneel has quit IRC | 01:41 | |
| *** tiswanso has quit IRC | 01:43 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config: add a job to send automated announcements of releases https://review.openstack.org/272767 | 01:43 |
| *** tiswanso has joined #openstack-infra | 01:43 | |
| jeblair | fungi: i want to add conflicts-with to gertty, and i have 2 choices on implementation -- | 01:43 |
| fungi | dhellmann: jeblair: ^ rebased to deal with a merge conflict on master, but there's another patchset coming with fixes for the job failure | 01:44 |
| jeblair | fungi: a) the same way gerrit does it -- after syncing a change, perform a gerrit query to ask which changes conflict with it | 01:44 |
| jeblair | fungi: this adds an extra network request to each change synced | 01:44 |
| jeblair | fungi: b) after syncing a change, do local git operations against each open change to see which changes it conflicts with | 01:45 |
| *** diana_clarke has quit IRC | 01:45 | |
| jeblair | fungi: this can be done without modifying the local working tree, so should be _relatively_ lightweight | 01:45 |
| *** diana_clarke has joined #openstack-infra | 01:46 | |
| jeblair | fungi: but it's still a sort of n^2 problem | 01:46 |
| jeblair | fungi: so it may end up eating considerable local cpu | 01:47 |
| fungi | yeah, that's an interesting tradeoff | 01:47 |
| fungi | calculate conflicts locally, or offload calculation on gerrit | 01:47 |
| jeblair | fungi: and would probably necessitate creating a second processing queue (since i wouldn't want to hold up network requests for a bunch of local work) | 01:47 |
| jeblair | yep | 01:47 |
| fungi | in theory gerrit's doing it centrally and caching the result | 01:47 |
| jeblair | so basically, more network traffic or more local work | 01:47 |
| jeblair | (i usually like having gertty do local work, but this seems like it could be a lot) | 01:48 |
| fungi | i guess if you're offline, you're not going to be triggering checks for conflicts anyway | 01:48 |
| *** yamamoto_ has joined #openstack-infra | 01:48 | |
| *** thiagop has quit IRC | 01:48 | |
| *** weshay_xchat is now known as weshay | 01:48 | |
| jeblair | yeah, they'd only be triggered by change syncs | 01:49 |
| fungi | i lean toward the extra network over extra cpu, but if it's way easier to implement local checking then... | 01:49 |
| jeblair | i think easier implementation is probably actually the network. i'm just worried about people being concerned about slow syncing | 01:50 |
| *** ericksonsantos has quit IRC | 01:50 | |
| fungi | relying on gerrit for it decreases the overall amount of conflict checking which would otherwise be happening redundantly across all gertty users' systems who were following the same repos | 01:50 |
| jeblair | there's that too :) | 01:50 |
| fungi | especially since gerrit's going to be calculating it for us anyway whether or not we query it | 01:51 |
| fungi | one potential performance boost would be to have a changes query option in gerrit which includes the conflicts list | 01:52 |
| fungi | assuming that doesn't already exist | 01:52 |
| fungi | would save the extra query, right? | 01:52 |
| jeblair | fungi: yeah, i don't think it does. you're right though, that would be nice | 01:52 |
| jeblair | doesn't seem like it should be hard | 01:52 |
| jeblair | and gerrit itself could make use of it | 01:52 |
| fungi | they do similar stuff for other extended details as query options | 01:52 |
| *** piet has quit IRC | 01:52 | |
| fungi | so it's in keeping with their overall api model | 01:53 |
| jeblair | (but maybe there's a reason why they don't and they do the extra query) | 01:53 |
| fungi | right, i wonder the same | 01:53 |
| *** angdraug has quit IRC | 01:53 | |
| jeblair | but i'd like something sooner than that anyway, so that's back-burner | 01:53 |
| openstackgerrit | Matthew Kassawara proposed openstack-infra/irc-meetings: Remove APAC networking guide team meeting https://review.openstack.org/276012 | 01:55 |
| *** amrith is now known as _amrith_ | 01:55 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config: add a job to send automated announcements of releases https://review.openstack.org/272767 | 01:55 |
| fungi | dhellmann: jeblair: ^ failed the cross-check, turns out a handful of projects should have probably just been using the python-jobs group so i've updated them to suit | 01:55 |
| fungi | jeblair: sure, i was thinking more than gertty could take advantage of that for a performance boost once it's in gerrit, if it can detect it | 01:56 |
| jeblair | fungi, krotscheck: a (software) release triggered the wheel build (yay!), so i manually (vos) released again; this time 32 seconds with the content change. | 01:56 |
| jeblair | fungi: yep | 01:56 |
| fungi | jeblair: also if it's really burdensome on some connections, could always make it a toggleable feature in config | 01:56 |
| jeblair | fungi: true | 01:56 |
| fungi | which is the default mode depending on how generally burdensome it turns out to be i guess | 01:57 |
| jeblair | fungi: okay, i _think_ you've talked me into doing the network mode, but before i make a final call (well, as final as anything gets), i think i may hack out a quick script to simulate what the local work would actually look like on my database... | 01:57 |
| jeblair | fungi: and i'll reconsider if it turns out to be super cheap | 01:57 |
| openstackgerrit | Matthew Kassawara proposed openstack-infra/irc-meetings: Remove APAC installation guide team meeting https://review.openstack.org/276013 | 01:58 |
| fungi | jeblair: awesome. seems like a great feature either way | 01:58 |
| openstackgerrit | Merged openstack-infra/system-config: Add iftop to all Ubuntu servers https://review.openstack.org/275308 | 01:58 |
| *** ericksonsantos has joined #openstack-infra | 01:59 | |
| *** thiagop has joined #openstack-infra | 01:59 | |
| *** alivigni has joined #openstack-infra | 01:59 | |
| fungi | jeblair: i think one other benefit to the network solution might be if you're not syncing all changes for a repo (think repos you don't follow but you've submitted to, so they show up in your owned changes list) then you still get conflicts identified | 01:59 |
| *** yamamoto_ has quit IRC | 01:59 | |
| *** yamamoto_ has joined #openstack-infra | 02:00 | |
| jeblair | fungi: oh yes, that's true | 02:00 |
| jeblair | fungi: if i go that way, i should remember to sync those changes :) | 02:00 |
| fungi | heh | 02:00 |
| *** alivigni has quit IRC | 02:00 | |
| *** baoli has joined #openstack-infra | 02:02 | |
| fungi | jeblair: if you're still okay with 272767 after my update, i'm still around to babysit a manual application on all the jenkins masters this evening | 02:03 |
| fungi | otherwise we can try again tomorrow | 02:03 |
| jeblair | fungi: looking | 02:03 |
| *** sflanigan has joined #openstack-infra | 02:03 | |
| fungi | i split the rebase patchset from the make-the-job-pass patchset for clarity | 02:07 |
| jeblair | fungi: +3 | 02:08 |
| fungi | it got lucky and the change to add the mirror builder job landed just before it, so conflicted in context with it in the layout.yaml | 02:08 |
| jeblair | s/builder/release/ but yeah | 02:08 |
| jeblair | it's been a really lucky change | 02:09 |
| fungi | er, yep that one | 02:09 |
| fungi | so, so lucky | 02:09 |
| *** yamamoto_ has quit IRC | 02:10 | |
| *** yamamoto_ has joined #openstack-infra | 02:10 | |
| *** zz_dimtruck is now known as dimtruck | 02:10 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config: Use diskimage-builder from git for nodepool https://review.openstack.org/275006 | 02:11 |
| jeblair | fungi: speaking of lucky -- this change (and its dependency) would let us make sure that we don't vos-release in the middle of a mirror build job (since they are running on different hosts) | 02:11 |
| *** Sukhdev has joined #openstack-infra | 02:11 | |
| tonyb | lifeless: Silly random question ... Is there an acceptable way to remove an item from pip's wheel cache? | 02:11 |
| tonyb | lifeless: say a system library has changed and now the built wheel wont work | 02:12 |
| *** Sukhdev has quit IRC | 02:12 | |
| *** yamamoto_ has quit IRC | 02:13 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/puppet-nodepool: Add periodic cleanup of log files https://review.openstack.org/275968 | 02:13 |
| fungi | jeblair: this which? | 02:14 |
| jeblair | fungi: heh https://review.openstack.org/275975 | 02:14 |
| fungi | my mind reading just isn't what it used to be | 02:15 |
| jeblair | fungi: i knew that | 02:15 |
| fungi | indeed | 02:15 |
| fungi | so the job failure there is expected? | 02:16 |
| fungi | right, the dependency | 02:16 |
| openstackgerrit | Merged openstack-infra/project-config: Fix typo in wheel-index script https://review.openstack.org/275999 | 02:16 |
| jeblair | fungi: yeah, i'm pretty sure we don't use zuul from git in that test | 02:16 |
| fungi | it's not an integration test job | 02:16 |
| *** zhurong has joined #openstack-infra | 02:18 | |
| fungi | at least from a zuul codebase perspective | 02:18 |
| prometheanfire | ianw: thanks | 02:19 |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config: Atomically replace the wheel index file https://review.openstack.org/276015 | 02:20 |
| openstackgerrit | Merged openstack-infra/zuul: Pass ZUUL_TEST_ROOT through tox https://review.openstack.org/275973 | 02:21 |
| jeblair | fungi, krotscheck: while we're on the subject of atoms: https://review.openstack.org/276015 | 02:21 |
| *** piet has joined #openstack-infra | 02:21 | |
| fungi | jeblair: zuul change lgtm | 02:22 |
| fungi | i was thinking jhesketh might also want to review this time of day, but now realize he's more likely conferencing | 02:22 |
| jhesketh | fungi: which change? | 02:24 |
| jeblair | i don't think any of our wheel mirror building is actually subject to racing, but it's nice not to have to worry about it | 02:24 |
| *** nikhil_k has joined #openstack-infra | 02:24 | |
| jeblair | jhesketh: https://review.openstack.org/275974 see https://review.openstack.org/275975 for it's proposed use | 02:25 |
| jeblair | jhesketh: i broke my vow of silence on zuulv2, but i proposed the same change to zuulv3 too in penance | 02:26 |
| jhesketh | Heh okay. I'll take a look :-) | 02:26 |
| jeblair | it's not too crazy :) | 02:27 |
| *** nikhil has quit IRC | 02:27 | |
| fungi | seemed pretty straightforward to me, but i trust jhesketh to have more nuanced grasp of the codebase than i | 02:28 |
| *** abitha has quit IRC | 02:31 | |
| *** jamielennox|away is now known as jamielennox | 02:32 | |
| *** diana_clarke has quit IRC | 02:32 | |
| *** jed56 has quit IRC | 02:33 | |
| lifeless | tonyb: yes, just delete it | 02:34 |
| *** piet has quit IRC | 02:35 | |
| tonyb | lifeless: is there a nice way to locate it (other than gnu find)? | 02:35 |
| lifeless | tonyb: not today | 02:37 |
| tonyb | lifeless: okay thanks | 02:38 |
| *** yamahata has quit IRC | 02:39 | |
| fungi | grrr, the layout job on 272767 is still failing? | 02:40 |
| fungi | aha, duplicate job definition | 02:42 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config: add a job to send automated announcements of releases https://review.openstack.org/272767 | 02:42 |
| fungi | dhellmann: jeblair: that ^ should address it | 02:42 |
| jhesketh | jeblair: just one nit but otherwise lgtm | 02:42 |
| jeblair | fungi: +3 | 02:42 |
| * jhesketh is surprised it already wasn't automated | 02:44 | |
| jeblair | jhesketh: replied, thanks! | 02:45 |
| openstackgerrit | Jamie Lennox proposed openstack-infra/project-config: Run the identity v3 only job in additional services https://review.openstack.org/271128 | 02:45 |
| jeblair | jhesketh: i'll probably do the cleanup as a followup; i'm just not sure where the best place to do that is | 02:46 |
| jhesketh | jeblair: the mutex cleanup? | 02:46 |
| jeblair | ya | 02:46 |
| jhesketh | yeah I don't think it's a real concern | 02:47 |
| jhesketh | presumably anything with a mutex gets run somewhat regularly | 02:47 |
| jhesketh | and if not, it's not much memory | 02:47 |
| jeblair | jhesketh: maybe in onBuildCompleteEvent | 02:48 |
| jeblair | er | 02:48 |
| jeblair | onBuildCompleted | 02:48 |
| jeblair | or even _doBuildCompletedEvent | 02:48 |
| jeblair | see? :) | 02:48 |
| *** bpokorny has joined #openstack-infra | 02:48 | |
| jhesketh | heh, yep, somewhere like that | 02:48 |
| jeblair | no actually onBuildCompleted :) | 02:49 |
| jeblair | in the manager | 02:50 |
| jeblair | as that's where it's guaranteed to be the current build | 02:50 |
| *** eliqiao_ has joined #openstack-infra | 02:51 | |
| *** thorst has joined #openstack-infra | 02:52 | |
| *** eliqiao has quit IRC | 02:54 | |
| *** FallenPegasus has joined #openstack-infra | 02:57 | |
| *** apoorvad has quit IRC | 02:59 | |
| *** sdake has joined #openstack-infra | 03:01 | |
| openstackgerrit | Merged openstack-infra/zuul: Pass ZUUL_TEST_ROOT through tox https://review.openstack.org/275976 | 03:03 |
| *** jamielennox is now known as jamielennox|away | 03:06 | |
| *** tphummel has quit IRC | 03:06 | |
| *** unicell has quit IRC | 03:06 | |
| *** weshay has quit IRC | 03:08 | |
| *** yamamoto_ has joined #openstack-infra | 03:09 | |
| *** baoli has quit IRC | 03:10 | |
| *** ccrouch has quit IRC | 03:11 | |
| *** jamielennox|away is now known as jamielennox | 03:11 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Add job mutex support https://review.openstack.org/275974 | 03:12 |
| jeblair | fungi, jhesketh: that exercise caught a logic error (in findJobsToRun) that i think would have been harmless, but it's much cleaner this way. | 03:13 |
| jeblair | so i pushed a new PS rather than a followup | 03:14 |
| openstackgerrit | Ian Wienand proposed openstack-infra/puppet-nodepool: Add periodic cleanup of log files https://review.openstack.org/275968 | 03:15 |
| *** yuanying has quit IRC | 03:15 | |
| *** thorst has quit IRC | 03:17 | |
| *** thorst has joined #openstack-infra | 03:17 | |
| *** weshay_xchat has joined #openstack-infra | 03:18 | |
| *** ccrouch has joined #openstack-infra | 03:19 | |
| *** yuanying has joined #openstack-infra | 03:19 | |
| *** ccrouch has quit IRC | 03:22 | |
| *** diana_clarke has joined #openstack-infra | 03:23 | |
| *** yuanying has quit IRC | 03:24 | |
| *** sridhar_ram has joined #openstack-infra | 03:25 | |
| *** baoli has joined #openstack-infra | 03:26 | |
| *** woodster_ has quit IRC | 03:26 | |
| *** thorst has quit IRC | 03:27 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config: add a job to send automated announcements of releases https://review.openstack.org/272767 | 03:27 |
| fungi | dhellmann: jeblair: argh! that wasn't the only duplicate job definition, but since jjb's parser short-circuits on the first error, i didn't realize it | 03:28 |
| *** baoli has quit IRC | 03:32 | |
| *** eliqiao has joined #openstack-infra | 03:33 | |
| *** yuanying has joined #openstack-infra | 03:35 | |
| *** eliqiao_ has quit IRC | 03:35 | |
| *** dims has joined #openstack-infra | 03:37 | |
| *** Daisy has quit IRC | 03:37 | |
| jhesketh | jeblair: cool, reviewed | 03:38 |
| *** ccrouch has joined #openstack-infra | 03:38 | |
| *** ashleighfarnham has quit IRC | 03:39 | |
| *** piet has joined #openstack-infra | 03:41 | |
| *** baoli has joined #openstack-infra | 03:41 | |
| fungi | finally 272767 is passing the layout job | 03:41 |
| *** weshay_xchat has quit IRC | 03:41 | |
| *** yuanying_ has joined #openstack-infra | 03:41 | |
| *** Jeffrey4l has joined #openstack-infra | 03:42 | |
| *** yuanying has quit IRC | 03:44 | |
| *** woodster_ has joined #openstack-infra | 03:44 | |
| *** sdake has quit IRC | 03:45 | |
| *** daemontool has quit IRC | 03:46 | |
| *** shashank_hegde has joined #openstack-infra | 03:46 | |
| *** daemontool has joined #openstack-infra | 03:46 | |
| *** links has joined #openstack-infra | 03:49 | |
| *** Sukhdev has joined #openstack-infra | 03:50 | |
| *** harlowja_at_home has joined #openstack-infra | 03:51 | |
| *** amotoki has joined #openstack-infra | 03:52 | |
| *** vivekd has joined #openstack-infra | 03:52 | |
| *** sdake has joined #openstack-infra | 03:52 | |
| *** dims has quit IRC | 03:53 | |
| *** salv-orl_ has joined #openstack-infra | 03:54 | |
| *** esker has quit IRC | 03:55 | |
| *** FallenPegasus has quit IRC | 03:56 | |
| *** salv-orlando has quit IRC | 03:58 | |
| *** Sukhdev has quit IRC | 03:58 | |
| jeblair | fungi: do you know the lyrics to the x-files theme? | 03:58 |
| *** vivekd has quit IRC | 03:59 | |
| fungi | i can... whistle it? | 04:03 |
| fungi | there was some narrative sorts of voice samples in the background for at least some seasons, if my memory isn't playing tricks on me | 04:04 |
| fungi | but i don't recall any lyrics, per se | 04:04 |
| *** shashank_hegde has quit IRC | 04:04 | |
| *** bpokorny_ has joined #openstack-infra | 04:04 | |
| *** shashank_hegde has joined #openstack-infra | 04:05 | |
| fungi | anyway, the truth is out there (i'm certain) | 04:05 |
| openstackgerrit | Merged openstack-infra/project-config: add a job to send automated announcements of releases https://review.openstack.org/272767 | 04:05 |
| *** esker has joined #openstack-infra | 04:05 | |
| *** r1chardj0n3s has joined #openstack-infra | 04:05 | |
| fungi | and _now_ i can pull that on the jenkins masters and start updating | 04:06 |
| r1chardj0n3s | hello, anyone about to help poke at a gate issue please? | 04:06 |
| *** piet has quit IRC | 04:06 | |
| *** yuanying_ has quit IRC | 04:06 | |
| *** sabeen has quit IRC | 04:07 | |
| *** yuanying has joined #openstack-infra | 04:07 | |
| *** yuanying has quit IRC | 04:08 | |
| *** bpokorny has quit IRC | 04:08 | |
| *** yuanying has joined #openstack-infra | 04:08 | |
| *** bpokorny_ has quit IRC | 04:08 | |
| *** ccrouch1 has joined #openstack-infra | 04:09 | |
| openstackgerrit | Merged openstack-infra/devstack-gate: Try vxlan instead of gre for overlays https://review.openstack.org/274821 | 04:09 |
| r1chardj0n3s | also, the captcha thing on paste.openstack.org doesn't work :-( | 04:09 |
| *** Sukhdev has joined #openstack-infra | 04:09 | |
| fungi | r1chardj0n3s: what sort of gate issue? also captcha thing is not working in what way? | 04:10 |
| r1chardj0n3s | sorry for the two-fer :-) | 04:10 |
| *** sc68cal has quit IRC | 04:10 | |
| *** sc68cal has joined #openstack-infra | 04:10 | |
| r1chardj0n3s | I just tried to paste https://gist.github.com/r1chardj0n3s/498f859256bf77d8983e into paste.openstack.org and was challenged for spam, I typed in the captcha and hit paste again, and was just redirected to the url "data:," | 04:10 |
| r1chardj0n3s | I'll happily file a bug if there's a tracker I can poke at? | 04:11 |
| *** boris-42 has quit IRC | 04:11 | |
| *** mwhahaha has quit IRC | 04:11 | |
| *** dmellado has quit IRC | 04:11 | |
| *** hrubi_ has quit IRC | 04:11 | |
| *** Ng has quit IRC | 04:11 | |
| *** ryanpetrello has quit IRC | 04:11 | |
| *** ashtokolov has quit IRC | 04:11 | |
| *** briancurtin has quit IRC | 04:11 | |
| *** SlickNik has quit IRC | 04:11 | |
| fungi | strange. i put in a made-up url, got the captcha prompt, entered the text it displayed, and it worked | 04:12 |
| *** ccrouch has quit IRC | 04:12 | |
| *** samyewell has quit IRC | 04:12 | |
| *** vaidy_ril has quit IRC | 04:12 | |
| *** isviridov_away has quit IRC | 04:12 | |
| *** dtroyer has quit IRC | 04:12 | |
| *** evgenyl has quit IRC | 04:12 | |
| *** kozhukalov has quit IRC | 04:12 | |
| *** ams_ has quit IRC | 04:12 | |
| *** fmccrthy has quit IRC | 04:12 | |
| *** phschwartz has quit IRC | 04:12 | |
| *** phschwartz_ has joined #openstack-infra | 04:12 | |
| *** SlickNik has joined #openstack-infra | 04:12 | |
| r1chardj0n3s | *shrug* | 04:12 |
| fungi | took me to http://paste.openstack.org/show/485915/ | 04:12 |
| *** jamielennox is now known as jamielennox|away | 04:12 | |
| *** trown|outttypeww has quit IRC | 04:12 | |
| fungi | i'll test your url | 04:12 |
| *** comstud has quit IRC | 04:12 | |
| *** ikalnitsky has quit IRC | 04:12 | |
| *** mtanino__ has quit IRC | 04:12 | |
| *** tpeoples has quit IRC | 04:12 | |
| *** jbryce has quit IRC | 04:12 | |
| *** ganso has quit IRC | 04:12 | |
| *** edwarnicke has quit IRC | 04:12 | |
| *** auggy has quit IRC | 04:12 | |
| *** mhayden has quit IRC | 04:12 | |
| *** mnencia has quit IRC | 04:12 | |
| r1chardj0n3s | might be the huge content I was pasting? | 04:12 |
| *** khappone_ has joined #openstack-infra | 04:12 | |
| *** Nakato has quit IRC | 04:12 | |
| *** kozhukalov has joined #openstack-infra | 04:12 | |
| *** trown has joined #openstack-infra | 04:12 | |
| *** ikalnitsky has joined #openstack-infra | 04:13 | |
| *** evgenyl has joined #openstack-infra | 04:13 | |
| *** bryan_att has quit IRC | 04:13 | |
| *** eglute has quit IRC | 04:13 | |
| *** sigmavirus24_awa has quit IRC | 04:13 | |
| *** bastelfreak has quit IRC | 04:13 | |
| *** tbarron has quit IRC | 04:13 | |
| *** mcarden has quit IRC | 04:13 | |
| *** jroll has quit IRC | 04:13 | |
| *** Apsu has quit IRC | 04:13 | |
| *** khappone has quit IRC | 04:13 | |
| *** dtroyer has joined #openstack-infra | 04:13 | |
| *** gus has quit IRC | 04:13 | |
| r1chardj0n3s | sorry, wasn't clear: I was pasting the *content* at that gist URL, not the gist URL itself | 04:13 |
| *** sdake has quit IRC | 04:13 | |
| *** mcarden has joined #openstack-infra | 04:13 | |
| *** davidlenwell_ has joined #openstack-infra | 04:13 | |
| *** davidlenwell has quit IRC | 04:13 | |
| fungi | ohhh | 04:13 |
| *** chlong has quit IRC | 04:13 | |
| *** calebb has quit IRC | 04:13 | |
| *** Nakato has joined #openstack-infra | 04:13 | |
| *** jpeeler has quit IRC | 04:13 | |
| *** sdague has quit IRC | 04:13 | |
| *** elmiko has quit IRC | 04:13 | |
| *** ryanpetrello has joined #openstack-infra | 04:14 | |
| *** bastelfreak has joined #openstack-infra | 04:14 | |
| *** jbryce has joined #openstack-infra | 04:14 | |
| *** calebb has joined #openstack-infra | 04:14 | |
| *** Apsu has joined #openstack-infra | 04:14 | |
| *** Zara has quit IRC | 04:14 | |
| *** dosaboy_ has quit IRC | 04:14 | |
| *** elmiko has joined #openstack-infra | 04:14 | |
| *** comstud has joined #openstack-infra | 04:14 | |
| fungi | usually it only prompts with a captcha if the content includes urls, but if it's a job result then it likely does | 04:14 |
| *** sdague has joined #openstack-infra | 04:14 | |
| *** tbarron has joined #openstack-infra | 04:14 | |
| *** gus has joined #openstack-infra | 04:14 | |
| *** shashank_hegde has quit IRC | 04:14 | |
| r1chardj0n3s | yep, lots of URLs | 04:14 |
| *** jroll has joined #openstack-infra | 04:14 | |
| *** eglute has joined #openstack-infra | 04:14 | |
| *** Zara has joined #openstack-infra | 04:15 | |
| *** dosaboy has joined #openstack-infra | 04:15 | |
| *** mhayden has joined #openstack-infra | 04:15 | |
| *** ganso has joined #openstack-infra | 04:15 | |
| *** mwhahaha has joined #openstack-infra | 04:15 | |
| *** mtanino__ has joined #openstack-infra | 04:16 | |
| *** mnencia has joined #openstack-infra | 04:16 | |
| *** samyewell has joined #openstack-infra | 04:16 | |
| *** bryan_att has joined #openstack-infra | 04:16 | |
| *** edwarnicke has joined #openstack-infra | 04:16 | |
| r1chardj0n3s | anyhoo, the gate issue we're seeing in Horizon land: https://review.openstack.org/#/c/274377/ is a good example where the integration tests fail intermittently. I had a look at the two run's consoles, that's the diff at the gist above, and the two environments aren't the same | 04:16 |
| *** fmccrthy has joined #openstack-infra | 04:16 | |
| *** sigmavirus24_awa has joined #openstack-infra | 04:16 | |
| *** tpeoples has joined #openstack-infra | 04:16 | |
| *** hrubi has joined #openstack-infra | 04:17 | |
| *** xek_ has joined #openstack-infra | 04:18 | |
| *** dmellado has joined #openstack-infra | 04:18 | |
| *** harlowja_at_home has quit IRC | 04:18 | |
| *** ams_ has joined #openstack-infra | 04:18 | |
| *** boris-42 has joined #openstack-infra | 04:18 | |
| *** ashtokolov has joined #openstack-infra | 04:18 | |
| *** isviridov_away has joined #openstack-infra | 04:19 | |
| *** timrc has quit IRC | 04:19 | |
| *** pasquier-s has quit IRC | 04:19 | |
| *** auggy has joined #openstack-infra | 04:19 | |
| *** thiagop has quit IRC | 04:19 | |
| *** Ryan_Lane has quit IRC | 04:19 | |
| *** amit213 has quit IRC | 04:19 | |
| *** bradjones has quit IRC | 04:19 | |
| *** wfoster has quit IRC | 04:19 | |
| *** abramley has quit IRC | 04:19 | |
| *** vaidy_ril has joined #openstack-infra | 04:19 | |
| fungi | so anyway, this is a diff of two job logs? | 04:19 |
| fungi | the paste service is running lodgeit, which i think may not have an upstream any longer (or that might also mean we're de facto upstream?) | 04:19 |
| fungi | lodgeit also seems to bug out on really huge paste content anyway (though usually it just truncates to whatever the database max field size is i think) | 04:19 |
| *** SergeyLukjanov has quit IRC | 04:19 | |
| *** auggy has quit IRC | 04:19 | |
| *** auggy has joined #openstack-infra | 04:19 | |
| *** Ryan_Lane_ has joined #openstack-infra | 04:19 | |
| *** HeOS_ has quit IRC | 04:19 | |
| *** mriedem_afk has quit IRC | 04:19 | |
| *** pblaho has quit IRC | 04:19 | |
| *** tristanC has quit IRC | 04:19 | |
| *** cgross has quit IRC | 04:19 | |
| *** smcginnis has quit IRC | 04:19 | |
| *** edunham has quit IRC | 04:19 | |
| *** Daviey has quit IRC | 04:19 | |
| *** HeOS_ has joined #openstack-infra | 04:19 | |
| *** tristanC has joined #openstack-infra | 04:19 | |
| *** Daviey has joined #openstack-infra | 04:19 | |
| *** pblaho has joined #openstack-infra | 04:19 | |
| r1chardj0n3s | yep, it's cool, I can just switch to gist if it doesn't like me :-) | 04:19 |
| *** xek has quit IRC | 04:19 | |
| *** tlbr has quit IRC | 04:19 | |
| *** loquacities has quit IRC | 04:19 | |
| *** russellb has quit IRC | 04:19 | |
| *** abramley has joined #openstack-infra | 04:19 | |
| *** tlbr has joined #openstack-infra | 04:19 | |
| *** Ng has joined #openstack-infra | 04:20 | |
| *** SergeyLukjanov has joined #openstack-infra | 04:20 | |
| *** wfoster has joined #openstack-infra | 04:20 | |
| *** russellb has joined #openstack-infra | 04:20 | |
| *** pasquier-s has joined #openstack-infra | 04:20 | |
| *** edunham has joined #openstack-infra | 04:20 | |
| r1chardj0n3s | and yeah, the paste is the diff of the passing, and then failing test runs for https://review.openstack.org/#/c/274377/ | 04:20 |
| *** thiagop has joined #openstack-infra | 04:20 | |
| *** sridhar_ram has quit IRC | 04:20 | |
| *** cgross has joined #openstack-infra | 04:20 | |
| *** timrc has joined #openstack-infra | 04:20 | |
| *** briancurtin has joined #openstack-infra | 04:20 | |
| *** loquacities has joined #openstack-infra | 04:20 | |
| fungi | so the biggest difference there is that devstack-trusty images in rax-iad (your "-" lines) are built from updated snapshots of rackspace-provided base images | 04:20 |
| *** smcginnis has joined #openstack-infra | 04:20 | |
| *** jpeeler has joined #openstack-infra | 04:20 | |
| r1chardj0n3s | one of the test nodes is rax (passes) and the other is internap? | 04:21 |
| *** bradjones has joined #openstack-infra | 04:21 | |
| *** bradjones has quit IRC | 04:21 | |
| *** bradjones has joined #openstack-infra | 04:21 | |
| fungi | whereas the devstack-trusty images in internap-nyj01 are built with dib from official ubuntu cloud base images | 04:21 |
| r1chardj0n3s | ah, right | 04:21 |
| fungi | so a good possibility is that the rackspace base images include some extra packages preinstalled which the official ubuntu cloud images do not | 04:22 |
| fungi | and if that job has somehow grown a dependence on one of those additional packages, you'd expect it to cease passing anywhere besides rackspace | 04:22 |
| r1chardj0n3s | hmm, I'm not sure where we go from here then :/ | 04:23 |
| jeblair | fungi: "the x-files is a show / with music by mark snow" :) | 04:24 |
| fungi | jeblair: ha! | 04:24 |
| fungi | r1chardj0n3s: selenium.common.exceptions.WebDriverException: Message: Element is not clickable at point (944, 0.98333740234375). Other element would receive the click: <html webdriver="true"></html> | 04:24 |
| fungi | that seems to be the actual failure i guess? | 04:24 |
| r1chardj0n3s | I guess if I can reproduce the internap-nyj01 environment, and thus the specific error we're seeing, I can try to get some sort of work-around until internap updates | 04:24 |
| r1chardj0n3s | yes, that's the failure. we have absolutely no idea what could cause it :-( | 04:24 |
| r1chardj0n3s | hence we're desperately trying to figure how to reproduce the error, which is proving difficult (my cloud instances are all rax) | 04:25 |
| fungi | r1chardj0n3s: are you happening to see this error in any other providers (ovh, bluebox)? | 04:25 |
| fungi | or only on internap nodes? | 04:26 |
| r1chardj0n3s | fungi: don't know, this is the first I've examined for this specific detail | 04:26 |
| r1chardj0n3s | I'll have a look at some others | 04:26 |
| r1chardj0n3s | this particular run was useful because it contained a pass and a fail, making the diff simpler | 04:26 |
| *** thorst has joined #openstack-infra | 04:27 | |
| *** chlong has joined #openstack-infra | 04:28 | |
| *** ccrouch1 has quit IRC | 04:31 | |
| r1chardj0n3s | fungi: ovh also errors | 04:32 |
| *** thorst has quit IRC | 04:32 | |
| *** burgerk has joined #openstack-infra | 04:34 | |
| *** vivekd has joined #openstack-infra | 04:35 | |
| openstackgerrit | Akihiro Motoki proposed openstack-infra/project-config: Add pep8 job to i18n repo https://review.openstack.org/276036 | 04:36 |
| r1chardj0n3s | fungi: I've not been able to find a bluebox run that passed or failed | 04:38 |
| *** ashleighfarnham has joined #openstack-infra | 04:40 | |
| jeblair | r1chardj0n3s: i don't see any in logstash, though i do see some successes on ovh | 04:40 |
| r1chardj0n3s | jeblair: here's a patch chain that has an integration test fail, fail, success over nodes in internap, ovh and then rax https://review.openstack.org/#/c/256140/22 | 04:41 |
| r1chardj0n3s | noting that the code changes in question won't affect the integration tests in any way, so there's effectively the same tests being run in each three | 04:41 |
| *** dimtruck is now known as zz_dimtruck | 04:41 | |
| r1chardj0n3s | http://logs.openstack.org/40/256140/22/check/gate-horizon-dsvm-integration/5f9f312/console.html and http://logs.openstack.org/69/258769/20/check/gate-horizon-dsvm-integration/33f1c65/console.html and http://logs.openstack.org/31/259231/23/check/gate-horizon-dsvm-integration/480195e/console.html | 04:42 |
| *** shashank_hegde has joined #openstack-infra | 04:44 | |
| r1chardj0n3s | jeblair: sorry, I should have asked, if you have a success in ovh I'd like to see it please | 04:44 |
| *** baoli has quit IRC | 04:44 | |
| r1chardj0n3s | (I've never used logstash, and am about to for the first time ;-) | 04:45 |
| *** baoli has joined #openstack-infra | 04:45 | |
| jeblair | r1chardj0n3s: i see 30% pass rate in ovh-gra1, 25% pass rate in ovh-bhs1, 20% pass in nyj01, no runs in blue box, 50% in rax-dfw (sample size 2), 69% rax-iad, 57% rax-ord | 04:46 |
| r1chardj0n3s | over what period? we only just fixed a 100% failure rate issue yesterday | 04:46 |
| jeblair | r1chardj0n3s: i don't think i can share all of the things i clicked to do that, but you can start here: http://logstash.openstack.org/#/dashboard/file/logstash.json?query=build_name:gate-horizon-dsvm-integration | 04:47 |
| jeblair | r1chardj0n3s: oh, that was over 10 days | 04:47 |
| openstackgerrit | Merged openstack/diskimage-builder: Print unparsable file in pkg-map https://review.openstack.org/267305 | 04:47 |
| r1chardj0n3s | right, so limiting it to much more recent would be necessary | 04:47 |
| r1chardj0n3s | the xvfbwrapper release knocked us out for a couple of days | 04:47 |
| jeblair | r1chardj0n3s: node_provider and build_status are the cols you're interested in | 04:48 |
| jeblair | r1chardj0n3s: and if you click on a row, it has a link to the full logs, change, etc | 04:48 |
| *** vivekd has quit IRC | 04:48 | |
| jeblair | r1chardj0n3s: top right has a time range selector | 04:48 |
| r1chardj0n3s | thanks | 04:48 |
| *** sridhar_ram has joined #openstack-infra | 04:49 | |
| *** baoli has quit IRC | 04:49 | |
| jeblair | r1chardj0n3s: i do still see an ovh success (very) recently | 04:50 |
| jeblair | like 40 mins ago | 04:50 |
| jeblair | r1chardj0n3s: http://logs.openstack.org/48/271248/5/gate/gate-horizon-dsvm-integration/6435112/ | 04:50 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Fix tar listing in functional tests https://review.openstack.org/276037 | 04:51 |
| ianw | greghayens / SpamapS: ^ bit of a "doh" one ... hopefully it doesn't break something we are tacitly relying on | 04:51 |
| jeblair | r1chardj0n3s: it's late here; gotta run | 04:51 |
| r1chardj0n3s | jeblair: thanks!! | 04:51 |
| ianw | greghaynes, even | 04:51 |
| openstackgerrit | Merged openstack/os-testr: correct typo https://review.openstack.org/275008 | 04:52 |
| openstackgerrit | Merged openstack/os-testr: remove python 2.6 trove classifier https://review.openstack.org/260822 | 05:00 |
| *** jtomasek_ has joined #openstack-infra | 05:02 | |
| *** camunoz has quit IRC | 05:02 | |
| *** davidlenwell_ has quit IRC | 05:02 | |
| *** sflanigan has quit IRC | 05:03 | |
| *** eliqiao_ has joined #openstack-infra | 05:05 | |
| *** davidlenwell has joined #openstack-infra | 05:05 | |
| *** burgerk has quit IRC | 05:08 | |
| *** eliqiao has quit IRC | 05:08 | |
| *** DericHorn-HP has joined #openstack-infra | 05:10 | |
| *** vivekd has joined #openstack-infra | 05:11 | |
| *** boris-42 has quit IRC | 05:13 | |
| *** camunoz has joined #openstack-infra | 05:15 | |
| *** DericHorn-HP has quit IRC | 05:17 | |
| *** curl_e has joined #openstack-infra | 05:17 | |
| *** piet has joined #openstack-infra | 05:18 | |
| *** sflanigan has joined #openstack-infra | 05:20 | |
| *** ashleighfarnham has quit IRC | 05:23 | |
| prometheanfire | ianw: still doesn't seem to want to workflow 273790 | 05:26 |
| *** shashank_hegde has quit IRC | 05:26 | |
| *** sridhar_ram1 has joined #openstack-infra | 05:26 | |
| *** sridhar_ram has quit IRC | 05:27 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/shade: Fix heat create_stack and delete_stack https://review.openstack.org/276045 | 05:28 |
| *** yumapath has joined #openstack-infra | 05:29 | |
| *** FallenPegasus has joined #openstack-infra | 05:30 | |
| *** thorst has joined #openstack-infra | 05:31 | |
| *** ashleighfarnham has joined #openstack-infra | 05:31 | |
| *** Daisy has joined #openstack-infra | 05:32 | |
| openstackgerrit | IWAMOTO Toshihiro proposed openstack/requirements: Bump ryu to 3.30 https://review.openstack.org/264026 | 05:32 |
| *** markvoelker_ has quit IRC | 05:34 | |
| *** zhurong has quit IRC | 05:36 | |
| *** thorst has quit IRC | 05:37 | |
| *** sdake has joined #openstack-infra | 05:40 | |
| *** krtaylor has quit IRC | 05:44 | |
| *** sdake_ has joined #openstack-infra | 05:45 | |
| *** sdake has quit IRC | 05:45 | |
| *** haleyb has quit IRC | 05:48 | |
| openstackgerrit | Khai Do proposed openstack-infra/puppet-gerrit: Allow configuration of Gerrit httpd.maxQueued option https://review.openstack.org/275988 | 05:49 |
| *** sridhar_ram1 has quit IRC | 05:50 | |
| *** vgridnev has joined #openstack-infra | 05:51 | |
| openstackgerrit | Khai Do proposed openstack-infra/system-config: Increase the HTTP incoming connection queue https://review.openstack.org/275995 | 05:51 |
| *** FallenPegasus has quit IRC | 05:52 | |
| *** krtaylor has joined #openstack-infra | 05:56 | |
| *** mrmartin has joined #openstack-infra | 05:58 | |
| *** piet has quit IRC | 06:00 | |
| *** krtaylor has quit IRC | 06:02 | |
| *** rossella_s has quit IRC | 06:02 | |
| *** rossella_s has joined #openstack-infra | 06:03 | |
| *** mrmartin has quit IRC | 06:09 | |
| *** mrmartin has joined #openstack-infra | 06:09 | |
| *** hdd has quit IRC | 06:10 | |
| *** baoli has joined #openstack-infra | 06:11 | |
| *** vivekd_ has joined #openstack-infra | 06:11 | |
| *** haleyb has joined #openstack-infra | 06:12 | |
| *** vivekd has quit IRC | 06:12 | |
| *** sflanigan has quit IRC | 06:12 | |
| *** vivekd_ is now known as vivekd | 06:12 | |
| *** sdake_ has quit IRC | 06:13 | |
| *** sdake has joined #openstack-infra | 06:13 | |
| *** mrmartin has quit IRC | 06:14 | |
| *** exploreshaifali has joined #openstack-infra | 06:14 | |
| *** krtaylor has joined #openstack-infra | 06:15 | |
| *** baoli has quit IRC | 06:16 | |
| *** tiswanso has quit IRC | 06:16 | |
| *** abregman has joined #openstack-infra | 06:19 | |
| *** armax has quit IRC | 06:20 | |
| *** jamespage has quit IRC | 06:28 | |
| *** gnuoy has quit IRC | 06:28 | |
| *** gnuoy has joined #openstack-infra | 06:28 | |
| *** jamespage has joined #openstack-infra | 06:29 | |
| *** tongli has quit IRC | 06:30 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/requirements: Updated from generate-constraints https://review.openstack.org/276058 | 06:31 |
| *** r1chardj0n3s has left #openstack-infra | 06:32 | |
| *** markvoelker has joined #openstack-infra | 06:35 | |
| *** FallenPegasus has joined #openstack-infra | 06:36 | |
| *** woodster_ has quit IRC | 06:36 | |
| *** henrynash has joined #openstack-infra | 06:39 | |
| *** markvoelker has quit IRC | 06:39 | |
| *** vivekd has quit IRC | 06:40 | |
| *** jtomasek_ has quit IRC | 06:40 | |
| *** zhurong has joined #openstack-infra | 06:45 | |
| *** ashleighfarnham has quit IRC | 06:49 | |
| *** camunoz has quit IRC | 06:50 | |
| *** krtaylor has quit IRC | 06:54 | |
| openstackgerrit | Lana Brindley proposed openstack-infra/project-config: Modifying docimpact group https://review.openstack.org/276065 | 06:56 |
| *** tdasilva has quit IRC | 07:00 | |
| *** frickler has quit IRC | 07:00 | |
| *** Mmike has quit IRC | 07:00 | |
| *** Swami has joined #openstack-infra | 07:02 | |
| *** sflanigan has joined #openstack-infra | 07:03 | |
| *** sflanigan has joined #openstack-infra | 07:03 | |
| *** camunoz has joined #openstack-infra | 07:04 | |
| *** FallenPegasus has quit IRC | 07:04 | |
| *** FallenPegasus has joined #openstack-infra | 07:05 | |
| *** FallenPegasus has quit IRC | 07:06 | |
| *** krtaylor has joined #openstack-infra | 07:06 | |
| *** Daisy has quit IRC | 07:06 | |
| *** Sukhdev has quit IRC | 07:06 | |
| *** markus_zz has joined #openstack-infra | 07:06 | |
| *** markus_zz has quit IRC | 07:07 | |
| *** Mmike has joined #openstack-infra | 07:08 | |
| *** salv-orl_ has quit IRC | 07:13 | |
| *** Jeffrey4l has quit IRC | 07:19 | |
| ianw | prometheanfire: ok, seems to be going now | 07:21 |
| *** tdasilva has joined #openstack-infra | 07:24 | |
| AJaeger | amotoki: could you review https://review.openstack.org/#/c/274402/2 - shouldn't this be django.pot files? | 07:25 |
| amotoki | AJaeger: yeah, I think it should be django/djangojs.po. | 07:27 |
| amotoki | AJaeger: i wonder what happens. we need to investigate more. I have no time next two or three hours. | 07:28 |
| AJaeger | amotoki: ok, I'll check myself now thanks | 07:28 |
| AJaeger | It's wrong in zanata, did I rename it wrongly? I'm on it.. | 07:29 |
| *** ianw has quit IRC | 07:30 | |
| *** nmagnezi has joined #openstack-infra | 07:30 | |
| *** ianw has joined #openstack-infra | 07:30 | |
| *** Jeffrey4l has joined #openstack-infra | 07:32 | |
| AJaeger | amotoki: looks I screwed up in zanata... | 07:34 |
| *** infra-red has joined #openstack-infra | 07:35 | |
| *** markvoelker has joined #openstack-infra | 07:36 | |
| *** unicell has joined #openstack-infra | 07:37 | |
| openstackgerrit | Spencer Krum proposed openstack-infra/system-config: Add omfracloud to nodepool https://review.openstack.org/275485 | 07:38 |
| openstackgerrit | Spencer Krum proposed openstack-infra/system-config: Add OmfraCloud to puppetmaster_clouds https://review.openstack.org/275477 | 07:38 |
| *** Daisy has joined #openstack-infra | 07:41 | |
| *** markvoelker has quit IRC | 07:41 | |
| *** infra-red has quit IRC | 07:41 | |
| openstackgerrit | Spencer Krum proposed openstack-infra/project-config: Use omfracloud in nodepool https://review.openstack.org/275491 | 07:41 |
| *** thorst has joined #openstack-infra | 07:43 | |
| nibalizer | infra-root taking the hiera lock | 07:43 |
| *** _nadya_ has joined #openstack-infra | 07:43 | |
| nibalizer | done with the lock | 07:45 |
| *** infra-red has joined #openstack-infra | 07:47 | |
| *** thorst has quit IRC | 07:48 | |
| *** eliqiao_ is now known as eliqiao | 07:49 | |
| openstackgerrit | ice4o@hotmail.com proposed openstack-infra/jenkins-job-builder: Add HockeyApp Plugin support. https://review.openstack.org/275304 | 07:49 |
| *** infra-red has quit IRC | 07:49 | |
| *** ihrachys has joined #openstack-infra | 07:51 | |
| *** gyee has quit IRC | 07:53 | |
| *** infra-red has joined #openstack-infra | 07:56 | |
| *** sdake has quit IRC | 07:56 | |
| *** frickler has joined #openstack-infra | 07:57 | |
| *** exploreshaifali has quit IRC | 07:57 | |
| *** infra-re_ has joined #openstack-infra | 07:58 | |
| *** abregman has quit IRC | 07:58 | |
| *** Mmike has quit IRC | 07:59 | |
| *** Mmike has joined #openstack-infra | 07:59 | |
| openstackgerrit | Merged openstack/diskimage-builder: Add support for gentoo to simple-init https://review.openstack.org/273790 | 08:00 |
| *** infra-red has quit IRC | 08:01 | |
| *** bnemec has joined #openstack-infra | 08:02 | |
| *** diana_clarke has quit IRC | 08:03 | |
| *** abregman has joined #openstack-infra | 08:04 | |
| *** dkaigarodsev_ has joined #openstack-infra | 08:08 | |
| *** vgridnev has quit IRC | 08:08 | |
| dkaigarodsev_ | hi to all, please take a look on https://review.openstack.org/#/c/275780/ | 08:09 |
| *** sflanigan has quit IRC | 08:09 | |
| *** fedexo has joined #openstack-infra | 08:11 | |
| yolanda | morning | 08:13 |
| *** fedexo_ has joined #openstack-infra | 08:16 | |
| *** mikelk has joined #openstack-infra | 08:22 | |
| *** mrmartin has joined #openstack-infra | 08:22 | |
| *** ihrachys has quit IRC | 08:25 | |
| *** ihrachys has joined #openstack-infra | 08:25 | |
| *** vgridnev has joined #openstack-infra | 08:25 | |
| *** sdake has joined #openstack-infra | 08:26 | |
| *** jistr has joined #openstack-infra | 08:29 | |
| *** zeih has joined #openstack-infra | 08:30 | |
| *** ifarkas has joined #openstack-infra | 08:30 | |
| *** jcoufal has joined #openstack-infra | 08:30 | |
| *** jaosorior has joined #openstack-infra | 08:31 | |
| *** samuelBartel has joined #openstack-infra | 08:32 | |
| *** fedexo has quit IRC | 08:33 | |
| *** fedexo_ has quit IRC | 08:33 | |
| *** zeih has quit IRC | 08:34 | |
| *** zeih has joined #openstack-infra | 08:34 | |
| *** andymaier has joined #openstack-infra | 08:34 | |
| *** jlanoux has joined #openstack-infra | 08:35 | |
| *** infra-re_ has quit IRC | 08:36 | |
| *** dingyichen has quit IRC | 08:40 | |
| *** matrohon has joined #openstack-infra | 08:41 | |
| openstackgerrit | Merged openstack-infra/project-config: Add pypi publish job to tempest https://review.openstack.org/275958 | 08:42 |
| *** jtomasek__ has joined #openstack-infra | 08:43 | |
| *** achanda has quit IRC | 08:45 | |
| *** thorst has joined #openstack-infra | 08:46 | |
| *** vgridnev has quit IRC | 08:46 | |
| openstackgerrit | Merged openstack-infra/project-config: Run all Rally jobs against Keystone API v3. Add job for v2. https://review.openstack.org/274668 | 08:48 |
| openstackgerrit | Merged openstack-infra/project-config: Add pep8 job to i18n repo https://review.openstack.org/276036 | 08:48 |
| openstackgerrit | Merged openstack-infra/project-config: Remove ML reporting for periodic-qa jobs https://review.openstack.org/275954 | 08:48 |
| *** achanda has joined #openstack-infra | 08:48 | |
| *** dizquierdo has joined #openstack-infra | 08:48 | |
| *** jpr has joined #openstack-infra | 08:49 | |
| *** yaume has joined #openstack-infra | 08:49 | |
| *** jordanP has joined #openstack-infra | 08:52 | |
| *** thorst has quit IRC | 08:52 | |
| openstackgerrit | Merged openstack-infra/system-config: Fix spell typos https://review.openstack.org/276000 | 08:52 |
| *** hashar has joined #openstack-infra | 08:53 | |
| *** jcoufal has quit IRC | 08:54 | |
| *** alivigni has joined #openstack-infra | 08:58 | |
| *** dguitarbite has joined #openstack-infra | 08:59 | |
| openstackgerrit | Merged openstack-infra/puppet-jenkins: Use 'ruby' instead of 'ruby1.9.1' for Debian https://review.openstack.org/275329 | 08:59 |
| amotoki | AJaeger: I tested the translation script on designate-dashboard in my local env, but django/djangojs.po were retrieved as expected. | 08:59 |
| amotoki | AJaeger: I really wonder what happened? | 08:59 |
| openstackgerrit | Merged openstack-infra/project-config: Retire icehouse and juno for networking-midonet https://review.openstack.org/272005 | 09:02 |
| openstackgerrit | Merged openstack-infra/project-config: networking-midonet: Add experimental rally job https://review.openstack.org/268859 | 09:03 |
| *** arxcruz has joined #openstack-infra | 09:03 | |
| openstackgerrit | Merged openstack-infra/project-config: Add new experimental job for manila-image-elements project https://review.openstack.org/275682 | 09:04 |
| openstackgerrit | Merged openstack-infra/project-config: Add a non-voting inspector job for ironic-python-agent https://review.openstack.org/275637 | 09:04 |
| *** rhallisey has joined #openstack-infra | 09:04 | |
| openstackgerrit | Merged openstack-infra/project-config: Watcher Dashboard project creation https://review.openstack.org/275167 | 09:07 |
| *** infra-red has joined #openstack-infra | 09:12 | |
| openstackgerrit | Marton Kiss proposed openstack-infra/groups: Disable leaflet_more_maps module https://review.openstack.org/276105 | 09:13 |
| openstackgerrit | Jan Klare proposed openstack-infra/project-config: removed unused job gate-openstack-chef-repo https://review.openstack.org/276106 | 09:13 |
| *** abregman has quit IRC | 09:14 | |
| *** infra-re_ has joined #openstack-infra | 09:16 | |
| *** jistr has quit IRC | 09:16 | |
| *** openstackgerrit has quit IRC | 09:17 | |
| *** abregman has joined #openstack-infra | 09:17 | |
| *** openstackgerrit has joined #openstack-infra | 09:17 | |
| *** infra-red has quit IRC | 09:18 | |
| *** kzaitsev_mb has joined #openstack-infra | 09:19 | |
| openstackgerrit | Julien Danjou proposed openstack/requirements: Add overtest to global-req https://review.openstack.org/274348 | 09:22 |
| openstackgerrit | Merged openstack-infra/groups: Disable leaflet_more_maps module https://review.openstack.org/276105 | 09:25 |
| EmilienM | AJaeger: can you check https://review.openstack.org/#/c/273812/ when you have 2 min please? | 09:25 |
| *** Swami_ has joined #openstack-infra | 09:26 | |
| *** Swami has quit IRC | 09:30 | |
| *** kzaitsev_mb has quit IRC | 09:31 | |
| *** jlanoux has quit IRC | 09:33 | |
| *** achanda has quit IRC | 09:33 | |
| *** jlanoux has joined #openstack-infra | 09:33 | |
| therve | Is there something going on with node speed? | 09:35 |
| therve | It takes about 1h to just setup a build for heat integration tests, and then it timeouts running them | 09:35 |
| *** gildub has quit IRC | 09:36 | |
| *** markvoelker has joined #openstack-infra | 09:37 | |
| *** aysyd has joined #openstack-infra | 09:39 | |
| *** Daisy has quit IRC | 09:41 | |
| *** electrofelix has joined #openstack-infra | 09:41 | |
| *** aysyd has quit IRC | 09:41 | |
| *** markvoelker has quit IRC | 09:41 | |
| *** isaacb has joined #openstack-infra | 09:42 | |
| *** Jeffrey4l has quit IRC | 09:43 | |
| *** tdurakov_ is now known as tdurakov | 09:44 | |
| *** tzn has joined #openstack-infra | 09:45 | |
| *** ociuhandu has joined #openstack-infra | 09:45 | |
| *** zhurong has quit IRC | 09:48 | |
| *** sfinucan has joined #openstack-infra | 09:48 | |
| *** ociuhandu has quit IRC | 09:49 | |
| *** trown has quit IRC | 09:49 | |
| *** ociuhandu has joined #openstack-infra | 09:49 | |
| *** thorst has joined #openstack-infra | 09:51 | |
| AJaeger | EmilienM: the change does not what you expect it to do - and I'm not sure whether this is the right approach at all. I suggest you discuss with rest of infra team, I'd like to have fungi, clark, or jeblair comment on 273812. Best to come back once those are awake and discuss with them... | 09:52 |
| EmilienM | AJaeger: cool, thanks! | 09:53 |
| AJaeger | therve: is there a regression in OpenStack itself - or a node problem? Please check a bit more | 09:53 |
| *** dalgaaf has joined #openstack-infra | 09:53 | |
| AJaeger | therve: sorry, can't help further... | 09:53 |
| openstackgerrit | Masahito Muroi proposed openstack-infra/project-config: Adds a tox test job for Congress new architecture https://review.openstack.org/275515 | 09:54 |
| therve | AJaeger, Yeah no worries. It says the usual "this is a slow node" | 09:55 |
| samuelBartel | hi all | 09:55 |
| samuelBartel | I have a question about the project creator guide | 09:56 |
| *** thorst has quit IRC | 09:57 | |
| samuelBartel | for the pypi part after registering the project and giving the appropriate right to the openstackci user is there anything else to do about pypi? | 09:57 |
| samuelBartel | AJaeger: for this review https://review.openstack.org/#/c/267806/ were you refering to that or something ese? | 09:59 |
| *** Jeffrey4l has joined #openstack-infra | 10:01 | |
| *** rossella_s has quit IRC | 10:02 | |
| *** rossella_s has joined #openstack-infra | 10:03 | |
| *** rlandy has quit IRC | 10:04 | |
| *** Swami_ has quit IRC | 10:04 | |
| *** ildikov has quit IRC | 10:07 | |
| *** infra-red has joined #openstack-infra | 10:07 | |
| *** rlandy has joined #openstack-infra | 10:08 | |
| AJaeger | samuelBartel: that should be all about pypi, let me check 267806... | 10:08 |
| AJaeger | samuelBartel: https://pypi.python.org/pypi/puppet-ui does not exist | 10:11 |
| *** infra-re_ has quit IRC | 10:11 | |
| AJaeger | samuelBartel: https://pypi.python.org/pypi/poppy-ui does ;) | 10:11 |
| *** mrmartin has quit IRC | 10:12 | |
| *** jlanoux_ has joined #openstack-infra | 10:14 | |
| *** infra-re_ has joined #openstack-infra | 10:14 | |
| *** e0ne has joined #openstack-infra | 10:14 | |
| openstackgerrit | Samuel BARTEL proposed openstack-infra/project-config: add project poppy-ui a Horizon Plugin to work with Poppy's CDN API https://review.openstack.org/267806 | 10:15 |
| samuelBartel | AJaeger, ok thank you | 10:15 |
| *** infra-red has quit IRC | 10:16 | |
| AJaeger | samuelBartel: +2 | 10:17 |
| *** jlanoux has quit IRC | 10:17 | |
| samuelBartel | AJaeger, Thanks! | 10:17 |
| *** notnownikki has joined #openstack-infra | 10:18 | |
| *** NobodyCa1 has joined #openstack-infra | 10:18 | |
| *** tcammann_ has joined #openstack-infra | 10:18 | |
| *** tdurakov has quit IRC | 10:18 | |
| *** tcammann has joined #openstack-infra | 10:19 | |
| *** esp has joined #openstack-infra | 10:20 | |
| dkaigarodsev_ | folks, please review https://review.openstack.org/#/c/275780/ | 10:23 |
| *** tcammann has quit IRC | 10:24 | |
| *** dizquierdo has quit IRC | 10:24 | |
| *** esp has quit IRC | 10:24 | |
| *** tcammann_ has quit IRC | 10:25 | |
| *** NobodyCa1 has quit IRC | 10:26 | |
| *** mrmartin has joined #openstack-infra | 10:26 | |
| *** Jeffrey4l has quit IRC | 10:29 | |
| *** markmcd1 has quit IRC | 10:32 | |
| *** esikachev has joined #openstack-infra | 10:35 | |
| *** markmcd has joined #openstack-infra | 10:36 | |
| *** jmccrory has quit IRC | 10:39 | |
| *** kmartin__ has joined #openstack-infra | 10:39 | |
| *** mugsie has quit IRC | 10:39 | |
| *** mrodden has quit IRC | 10:40 | |
| *** kmartin_ has quit IRC | 10:40 | |
| *** mugsie has joined #openstack-infra | 10:40 | |
| *** bnemec has quit IRC | 10:40 | |
| *** mrodden has joined #openstack-infra | 10:41 | |
| *** infra-red has joined #openstack-infra | 10:41 | |
| *** jmccrory has joined #openstack-infra | 10:42 | |
| *** bnemec has joined #openstack-infra | 10:42 | |
| *** sdake has quit IRC | 10:44 | |
| *** ldnunes has joined #openstack-infra | 10:44 | |
| *** infra-re_ has quit IRC | 10:44 | |
| *** infra-r__ has joined #openstack-infra | 10:45 | |
| *** infra-red has quit IRC | 10:47 | |
| *** tzn has quit IRC | 10:48 | |
| *** mrmartin has quit IRC | 10:48 | |
| *** ociuhandu has quit IRC | 10:50 | |
| *** mrmartin has joined #openstack-infra | 10:50 | |
| *** esikachev has quit IRC | 10:52 | |
| *** esikachev has joined #openstack-infra | 10:53 | |
| *** salv-orlando has joined #openstack-infra | 10:54 | |
| *** thorst has joined #openstack-infra | 10:55 | |
| *** jaypipes has joined #openstack-infra | 10:57 | |
| *** yumapath has quit IRC | 10:58 | |
| *** kzaitsev_mb has joined #openstack-infra | 10:59 | |
| *** sfinucan has quit IRC | 11:00 | |
| *** bnemec has quit IRC | 11:01 | |
| *** sfinucan has joined #openstack-infra | 11:01 | |
| *** jaosorior has quit IRC | 11:01 | |
| *** thorst has quit IRC | 11:03 | |
| *** sc68cal has quit IRC | 11:08 | |
| *** mrmartin has quit IRC | 11:10 | |
| *** dabukalam has joined #openstack-infra | 11:10 | |
| *** yamahata has joined #openstack-infra | 11:10 | |
| *** FallenPegasus has joined #openstack-infra | 11:11 | |
| *** rfolco has joined #openstack-infra | 11:12 | |
| *** dims has joined #openstack-infra | 11:12 | |
| *** infra-red has joined #openstack-infra | 11:13 | |
| *** infra-r__ has quit IRC | 11:17 | |
| openstackgerrit | Lucas Alvares Gomes proposed openstack-infra/project-config: Add docs for ironic-staging-drivers https://review.openstack.org/276163 | 11:18 |
| openstackgerrit | Costin Galan proposed openstack/requirements: commit 996a5923c5d6b1d8775c591d3c9875c8d52fcbb1 https://review.openstack.org/275905 | 11:24 |
| *** _nadya_ has quit IRC | 11:25 | |
| *** yamamoto_ has quit IRC | 11:29 | |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack/requirements: Updated from generate-constraints https://review.openstack.org/276058 | 11:32 |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack/requirements: Updated from generate-constraints https://review.openstack.org/276058 | 11:33 |
| *** infra-re_ has joined #openstack-infra | 11:33 | |
| *** erlon has joined #openstack-infra | 11:33 | |
| *** Daisy has joined #openstack-infra | 11:35 | |
| pavel_bondar | Hi, I am looking someone from infra team, who could answer if it is possible to supply custom script to grenade tests, that would be called on neutron upgrade | 11:36 |
| *** bnemec has joined #openstack-infra | 11:36 | |
| *** infra-red has quit IRC | 11:36 | |
| *** markvoelker has joined #openstack-infra | 11:38 | |
| AJaeger | pavel_bondar: better ask qa team - #openstack-qa | 11:39 |
| pavel_bondar | AJaeger: Thanks! | 11:39 |
| openstackgerrit | Sam Betts proposed openstack-infra/project-config: Add a tinyIPA non-voting test to Ironic https://review.openstack.org/269684 | 11:42 |
| *** markvoelker has quit IRC | 11:43 | |
| *** mrmartin has joined #openstack-infra | 11:49 | |
| *** kzaitsev_mb has quit IRC | 11:51 | |
| *** jaosorior has joined #openstack-infra | 11:55 | |
| *** jistr has joined #openstack-infra | 11:56 | |
| *** ihrachys has quit IRC | 11:57 | |
| *** resker has joined #openstack-infra | 11:58 | |
| *** esker has quit IRC | 12:01 | |
| *** rvba has quit IRC | 12:01 | |
| *** rvba has joined #openstack-infra | 12:03 | |
| *** rvba has quit IRC | 12:04 | |
| *** rvba has joined #openstack-infra | 12:04 | |
| *** sc68cal has joined #openstack-infra | 12:04 | |
| openstackgerrit | Evgeny Sikachev proposed openstack-infra/project-config: Rename sahara-scenario -> sahara-tests https://review.openstack.org/269815 | 12:05 |
| *** BobBall has joined #openstack-infra | 12:07 | |
| *** _nadya_ has joined #openstack-infra | 12:08 | |
| *** xek_ is now known as xek | 12:08 | |
| *** ildikov has joined #openstack-infra | 12:09 | |
| *** sc68cal has quit IRC | 12:10 | |
| *** vgridnev has joined #openstack-infra | 12:13 | |
| *** aysyd has joined #openstack-infra | 12:17 | |
| *** Daisy has quit IRC | 12:22 | |
| *** Daisy has joined #openstack-infra | 12:22 | |
| openstackgerrit | Sam Betts proposed openstack-infra/project-config: Add a tinyIPA non-voting test to Ironic https://review.openstack.org/269684 | 12:23 |
| *** dizquierdo has joined #openstack-infra | 12:24 | |
| *** dkliban has joined #openstack-infra | 12:25 | |
| *** krotscheck1 has joined #openstack-infra | 12:26 | |
| *** jaypipes_ has joined #openstack-infra | 12:28 | |
| *** jaypipes_ has quit IRC | 12:28 | |
| *** _amrith_ is now known as amrith | 12:28 | |
| *** salv-orlando has quit IRC | 12:32 | |
| openstackgerrit | Evgeny Sikachev proposed openstack-infra/project-config: Rename sahara-scenario -> sahara-tests https://review.openstack.org/269815 | 12:34 |
| *** kzaitsev_mb has joined #openstack-infra | 12:34 | |
| *** yamahata has quit IRC | 12:38 | |
| *** thorst has joined #openstack-infra | 12:38 | |
| openstackgerrit | Giulio Fidente proposed openstack-infra/tripleo-ci: Add support for --overcloud-update and use it for HA job https://review.openstack.org/260466 | 12:39 |
| *** weshay_xchat has joined #openstack-infra | 12:44 | |
| *** weshay_xchat is now known as weshay | 12:44 | |
| *** lucasagomes is now known as lucas-hungry | 12:46 | |
| *** salv-orlando has joined #openstack-infra | 12:46 | |
| openstackgerrit | Vasyl Saienko proposed openstack/requirements: Add netmiko to requirements https://review.openstack.org/276200 | 12:48 |
| *** baoli has joined #openstack-infra | 12:48 | |
| *** cgross has quit IRC | 12:48 | |
| *** daemontool_ has joined #openstack-infra | 12:48 | |
| *** mkoderer has quit IRC | 12:49 | |
| *** erlon_ has joined #openstack-infra | 12:50 | |
| *** gibi_ has joined #openstack-infra | 12:52 | |
| *** erlon has quit IRC | 12:52 | |
| *** erlon_ is now known as erlon | 12:52 | |
| *** rossella_ has joined #openstack-infra | 12:52 | |
| *** mhayden has quit IRC | 12:53 | |
| *** bastelfreak has quit IRC | 12:53 | |
| *** daemontool has quit IRC | 12:53 | |
| *** gibi has quit IRC | 12:53 | |
| *** mrodden has quit IRC | 12:53 | |
| *** rossella_s has quit IRC | 12:53 | |
| *** markvoelker has joined #openstack-infra | 12:53 | |
| *** bastelfreak has joined #openstack-infra | 12:55 | |
| *** mrodden has joined #openstack-infra | 12:55 | |
| *** mhayden has joined #openstack-infra | 12:57 | |
| *** dkliban has left #openstack-infra | 12:57 | |
| *** daemontool_ has quit IRC | 12:58 | |
| *** markvoelker has quit IRC | 12:58 | |
| *** dtantsur|afk is now known as dtantsur | 12:58 | |
| *** ihrachys has joined #openstack-infra | 13:01 | |
| *** mkoderer has joined #openstack-infra | 13:03 | |
| *** cgross has joined #openstack-infra | 13:04 | |
| *** EricGonczer_ has joined #openstack-infra | 13:04 | |
| *** sc68cal has joined #openstack-infra | 13:06 | |
| *** EricGonczer_ has quit IRC | 13:08 | |
| flip214 | anyone knows about "ValueError: No closing quotation" in bashate? http://logs.openstack.org/38/276138/1/check/gate-drbd-devstack-bashate/b076b2f/console.html#_2016-02-04_10_56_15_233 | 13:09 |
| *** zz_dimtruck is now known as dimtruck | 13:10 | |
| *** markvoelker has joined #openstack-infra | 13:11 | |
| *** sc68cal has quit IRC | 13:12 | |
| *** yamahata has joined #openstack-infra | 13:12 | |
| *** ociuhandu has joined #openstack-infra | 13:15 | |
| *** bexelbie has quit IRC | 13:22 | |
| *** bexelbie has joined #openstack-infra | 13:22 | |
| *** gordc has joined #openstack-infra | 13:22 | |
| *** yamahata has quit IRC | 13:24 | |
| *** trown has joined #openstack-infra | 13:26 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/shade: Fix heat create_stack and delete_stack https://review.openstack.org/276045 | 13:27 |
| *** denisra has quit IRC | 13:28 | |
| *** mrmartin has quit IRC | 13:29 | |
| *** yamahata has joined #openstack-infra | 13:30 | |
| *** yamamoto has joined #openstack-infra | 13:31 | |
| *** edmondsw has joined #openstack-infra | 13:31 | |
| *** kgiusti has joined #openstack-infra | 13:32 | |
| *** EricGonczer_ has joined #openstack-infra | 13:32 | |
| *** ildikov has quit IRC | 13:34 | |
| *** yamahata has quit IRC | 13:36 | |
| *** david-lyle has quit IRC | 13:38 | |
| *** kushal has joined #openstack-infra | 13:38 | |
| *** pradk has joined #openstack-infra | 13:42 | |
| *** Guest89472 has quit IRC | 13:42 | |
| *** yamamoto has quit IRC | 13:43 | |
| *** burgerk has joined #openstack-infra | 13:43 | |
| *** dkranz has joined #openstack-infra | 13:43 | |
| *** mase_x200 has joined #openstack-infra | 13:46 | |
| *** mase_x200 has quit IRC | 13:46 | |
| *** EricGonczer_ has quit IRC | 13:46 | |
| *** lucas-hungry is now known as lucasagomes | 13:48 | |
| *** Jeffrey4l has joined #openstack-infra | 13:48 | |
| *** jpr has quit IRC | 13:49 | |
| *** sdake has joined #openstack-infra | 13:50 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid: Member Email Verification https://review.openstack.org/276239 | 13:51 |
| *** sigmavirus24_awa is now known as sigmavirus24 | 13:51 | |
| *** sigmavirus24 is now known as sigmavirus24_awa | 13:52 | |
| *** _nadya_ has quit IRC | 13:53 | |
| *** alivigni has quit IRC | 13:56 | |
| *** links has quit IRC | 13:56 | |
| *** krotscheck1 has quit IRC | 13:58 | |
| *** Hal has joined #openstack-infra | 14:00 | |
| *** baoli has quit IRC | 14:00 | |
| *** amrith is now known as _amrith_ | 14:00 | |
| *** Hal is now known as Guest82710 | 14:01 | |
| *** ayoung has quit IRC | 14:01 | |
| *** ildikov has joined #openstack-infra | 14:01 | |
| *** baoli has joined #openstack-infra | 14:01 | |
| *** jlanoux_ has quit IRC | 14:02 | |
| *** diana_clarke has joined #openstack-infra | 14:05 | |
| *** julim has joined #openstack-infra | 14:07 | |
| *** sc68cal has joined #openstack-infra | 14:08 | |
| *** daemontool has joined #openstack-infra | 14:09 | |
| *** jcoufal has joined #openstack-infra | 14:10 | |
| *** dims_ has joined #openstack-infra | 14:10 | |
| *** jcoufal has quit IRC | 14:10 | |
| *** jcoufal has joined #openstack-infra | 14:11 | |
| *** dims has quit IRC | 14:12 | |
| *** zeih_ has joined #openstack-infra | 14:12 | |
| *** sc68cal has quit IRC | 14:13 | |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder: Fix delete-all command to include all job types https://review.openstack.org/275993 | 14:14 |
| *** sc68cal has joined #openstack-infra | 14:16 | |
| *** zeih has quit IRC | 14:16 | |
| *** burgerk_ has joined #openstack-infra | 14:16 | |
| *** burgerk has quit IRC | 14:16 | |
| *** ikhudoshyn has quit IRC | 14:18 | |
| *** ikhudoshyn has joined #openstack-infra | 14:18 | |
| fungi | prometheanfire: 273790 needs a new (or replaced) workflow +1. it was last approved by ianw more than 7 hours before its depends-on change merged, so it wasn't enqueued at the time | 14:19 |
| *** sabeen1 has joined #openstack-infra | 14:19 | |
| *** tzn has joined #openstack-infra | 14:20 | |
| *** salv-orlando has quit IRC | 14:20 | |
| Shrews | fungi: cache looking ok this morning? | 14:21 |
| *** tiswanso has joined #openstack-infra | 14:21 | |
| *** ihrachys has quit IRC | 14:23 | |
| *** sabeen2 has joined #openstack-infra | 14:23 | |
| *** salv-orlando has joined #openstack-infra | 14:24 | |
| *** tzn has quit IRC | 14:24 | |
| *** nmagnezi_ has joined #openstack-infra | 14:25 | |
| *** sabeen1 has quit IRC | 14:26 | |
| *** tzn has joined #openstack-infra | 14:26 | |
| *** baoli has quit IRC | 14:28 | |
| *** nmagnezi has quit IRC | 14:28 | |
| *** baoli has joined #openstack-infra | 14:28 | |
| *** flepied1 has joined #openstack-infra | 14:28 | |
| fungi | EmilienM: as AJaeger points out, that job doesn't check what you're wanting, but also it was going to be removed because the docs team has gone a different direction with docimpact bug triage after it was pointed out that at least some projects are pushing for us to skip ci entirely (and so preserve any existing verify vote) on changes which only update the commit message | 14:29 |
| *** wshao has joined #openstack-infra | 14:29 | |
| *** flepied has quit IRC | 14:30 | |
| fungi | therve: are you seeing this one-hour setup time consistently? or did you just see it in one or a few jobs? where does it seem to be spending the most time? do you maybe see any really long gaps between timestamps in the logs? | 14:30 |
| therve | fungi, Not consistently, frequently enough. | 14:30 |
| *** jlanoux has joined #openstack-infra | 14:31 | |
| EmilienM | fungi: cool, np. I'll abandon it. | 14:32 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Remove git-commit-message check https://review.openstack.org/276267 | 14:32 |
| AJaeger | fungi, EmilienM: Let's cleanup ^ | 14:33 |
| *** tiswanso has quit IRC | 14:33 | |
| *** tiswanso has joined #openstack-infra | 14:34 | |
| *** dimtruck is now known as zz_dimtruck | 14:34 | |
| *** tzn has quit IRC | 14:35 | |
| *** jsavak has joined #openstack-infra | 14:35 | |
| *** burgerk_ has quit IRC | 14:36 | |
| *** resker has quit IRC | 14:36 | |
| fungi | Shrews: i'll check, thanks for the reminder | 14:37 |
| *** annegentle has joined #openstack-infra | 14:37 | |
| *** ihrachys has joined #openstack-infra | 14:38 | |
| *** moravec has quit IRC | 14:39 | |
| fungi | Shrews: looks like we still have a fair amount of weirdness at http://puppetboard.openstack.org/ but it's possible that's puppetdb still confused from earlier. checking the actual ansible inventory next | 14:39 |
| *** moravec has joined #openstack-infra | 14:40 | |
| fungi | Shrews: yep, the cache is back to being all uuids instead of hostnames for rackspace dfw | 14:41 |
| fungi | other providers/regions are fine | 14:41 |
| fungi | so looks like it's still intermittently deciding not to use hostnames in just that one cloud | 14:43 |
| fungi | or possibly not getting any hostnames from that one cloud | 14:43 |
| *** mrmartin has joined #openstack-infra | 14:43 | |
| *** sridhar_ram has joined #openstack-infra | 14:46 | |
| *** xyang1 has joined #openstack-infra | 14:47 | |
| *** ayoung has joined #openstack-infra | 14:47 | |
| *** piet has joined #openstack-infra | 14:47 | |
| Shrews | fungi: if you could run this: http://paste.openstack.org/show/485963/ | 14:48 |
| Shrews | fungi: replacing 'RAX' with the entry from clouds.yaml where we see the problem | 14:49 |
| *** _nadya_ has joined #openstack-infra | 14:49 | |
| fungi | doing | 14:50 |
| mrmartin | fungi: who can disable the puppet runs on ask-staging.o.o ? | 14:51 |
| *** baoli has quit IRC | 14:52 | |
| *** baoli has joined #openstack-infra | 14:52 | |
| fungi | Shrews: entries look like this right now: id=44dbb9fb-dce7-4694-9365-91e25f97eb1f, name=jenkins02.openstack.org | 14:53 |
| openstackgerrit | Ihar Hrachyshka proposed openstack/requirements: Cap testtools to <2.0.0 https://review.openstack.org/276275 | 14:53 |
| fungi | Shrews: but as i saw yesterday, removing the cache and letting it get regenerated it came back fine, so seems to be intermittent behavior maybe? | 14:53 |
| Shrews | fungi: oh, you may need to provide region_name after the cloud name | 14:54 |
| anteaya | morning AJaeger what is on your important list today? | 14:54 |
| *** dkranz_ has joined #openstack-infra | 14:54 | |
| fungi | mrmartin: it can be done with a configuration change now: http://docs.openstack.org/infra/system-config/puppet.html#disabling-puppet-on-nodes | 14:54 |
| *** zz_dimtruck is now known as dimtruck | 14:55 | |
| *** trown is now known as trown|mtg | 14:55 | |
| fungi | Shrews: same result, just a few fewer entries since it's limited to one region now | 14:55 |
| *** Daisy has quit IRC | 14:55 | |
| openstackgerrit | Lucas Alvares Gomes proposed openstack/diskimage-builder: Install IPA in a virtual enviroment https://review.openstack.org/235843 | 14:55 |
| Shrews | fungi: hrm, ok | 14:56 |
| *** pradk has quit IRC | 14:56 | |
| *** pradk has joined #openstack-infra | 14:56 | |
| anteaya | jeblair: zuul memory usage took a bump yesterday, going from a steady 10G to about 15G now, it may still be rising: http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=392&rra_id=all | 14:57 |
| *** _amrith_ is now known as amrith | 14:57 | |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Source undercloud environment variable from a file https://review.openstack.org/275667 | 14:57 |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Split the deploy script into its own file https://review.openstack.org/275668 | 14:57 |
| anteaya | jeblair: did you want to restart now that 275483 has merged? | 14:58 |
| *** dkranz has quit IRC | 14:58 | |
| fungi | Shrews: probably we need some sort of debug logging bubbled up from ansible so that when the cache gets updated we collect whatever additional detail would help and then timestamp it so we can map it back to a particular cache update | 14:58 |
| *** dkranz_ has quit IRC | 14:58 | |
| *** dkranz has joined #openstack-infra | 14:59 | |
| *** ccrouch has joined #openstack-infra | 14:59 | |
| *** Daisy_ has joined #openstack-infra | 14:59 | |
| *** dimtruck is now known as zz_dimtruck | 14:59 | |
| *** eharney has joined #openstack-infra | 15:00 | |
| openstackgerrit | Costin Galan proposed openstack/requirements: Fix Windows rename operation https://review.openstack.org/275905 | 15:00 |
| AJaeger | anteaya: celebrating the first working part of translation setup! | 15:01 |
| *** ihrachys has quit IRC | 15:01 | |
| *** esp has joined #openstack-infra | 15:01 | |
| anteaya | AJaeger: yay | 15:01 |
| AJaeger | anteaya: and then documenting it - so going over https://review.openstack.org/273759 and getting it ready for merging ;) | 15:01 |
| fungi | Shrews: an alternative is that i could run that test in a loop with a pause in there, and trap for an incorrect result | 15:01 |
| anteaya | it is working! | 15:01 |
| AJaeger | anteaya: thanks for asking! | 15:01 |
| anteaya | woooo for working things | 15:01 |
| anteaya | :) | 15:01 |
| openstackgerrit | Marton Kiss proposed openstack-infra/system-config: Disable ask-staging.openstack.org puppet run https://review.openstack.org/276277 | 15:01 |
| *** tcammann has joined #openstack-infra | 15:01 | |
| AJaeger | anteaya: and then take it slow ;) | 15:02 |
| anteaya | AJaeger: well asking you is just easier for me than waiting, so thought I'd ask | 15:02 |
| mrmartin | fungi: patch done ^^^ for ask-staging puppet run | 15:02 |
| AJaeger | anteaya: have to leave now. BYe | 15:02 |
| anteaya | AJaeger: thanks, see you | 15:02 |
| AJaeger | anteaya: very much appreciated! | 15:02 |
| anteaya | :) | 15:02 |
| *** AJaeger has quit IRC | 15:02 | |
| *** hashar has quit IRC | 15:03 | |
| openstackgerrit | Merged openstack-infra/storyboard-webclient: Index tags in search https://review.openstack.org/274723 | 15:03 |
| *** tcammann_ has joined #openstack-infra | 15:03 | |
| *** doug-fish has joined #openstack-infra | 15:03 | |
| *** burgerk has joined #openstack-infra | 15:04 | |
| *** amitgandhinz has joined #openstack-infra | 15:04 | |
| *** samueldmq has joined #openstack-infra | 15:05 | |
| *** EricGonczer_ has joined #openstack-infra | 15:05 | |
| *** sigmavirus24_awa is now known as sigmavirus24 | 15:06 | |
| openstackgerrit | Vasyl Saienko proposed openstack/requirements: Add netmiko to requirements https://review.openstack.org/276200 | 15:06 |
| fungi | i'm taking a position that for puppet disabling changes like 276277 we only need one infra-core reviewer to approve it (also self-approving these should be fine for infra-core members) | 15:07 |
| openstackgerrit | Lucas Alvares Gomes proposed openstack-infra/project-config: Add docs-on-rtfd hook for ironic-staging-drivers project https://review.openstack.org/276283 | 15:07 |
| *** nikhil_k is now known as nikhil | 15:08 | |
| fungi | i love that now we have a record of what was disabled, when, and why | 15:08 |
| *** tcammann has quit IRC | 15:08 | |
| *** tcammann_ has quit IRC | 15:08 | |
| *** nmagnezi_ has quit IRC | 15:09 | |
| *** esp has quit IRC | 15:10 | |
| *** samueldmq has quit IRC | 15:10 | |
| mrmartin | yolanda: if you have 10secs, please review this ask-staging puppet disable patch: https://review.openstack.org/#/c/276277/ thnx. | 15:11 |
| *** dguitarbite has quit IRC | 15:11 | |
| yolanda | sure | 15:11 |
| *** wshao has quit IRC | 15:11 | |
| Shrews | fungi: sooo... looking at your clouds.yaml, you don't actually set the cache class, which means it *should* be defaulting to memory cache, not file ???? | 15:11 |
| openstackgerrit | Timur Sufiev proposed openstack/requirements: Remove version exclusions for xvfbwrapper and selenium https://review.openstack.org/276288 | 15:11 |
| *** dguitarbite has joined #openstack-infra | 15:11 | |
| Shrews | fungi: so i am confuzzled | 15:12 |
| *** moravec has quit IRC | 15:12 | |
| yolanda | mrmartin, too late, fungi had alerady approved it | 15:12 |
| *** rbrndt_ has joined #openstack-infra | 15:12 | |
| mrmartin | oh ok | 15:13 |
| mrmartin | really | 15:13 |
| tristanC | matbu: hey, should we have a talk about fixing shade to manage heat stack ? I just realise you submitted a fix too :-) | 15:15 |
| Shrews | fungi: oh, i see. the ansible inv does it directly | 15:17 |
| Shrews | tristanC: yes, please work with matbu. i think you guys are fixing similar things, which is why i poked him earlier | 15:18 |
| tristanC | Shrews: mea culpa, i started coding without checking open reviews first heh | 15:18 |
| openstackgerrit | yolanda.robla proposed openstack-infra/puppet-infracloud: Add element to bring up bridge on infra cloud https://review.openstack.org/276292 | 15:19 |
| *** abregman has quit IRC | 15:20 | |
| *** phschwartz_ is now known as phschwartz | 15:20 | |
| *** nmagnezi_ has joined #openstack-infra | 15:20 | |
| *** zeih_ has quit IRC | 15:20 | |
| *** zz_dimtruck is now known as dimtruck | 15:24 | |
| *** arxcruz has quit IRC | 15:26 | |
| *** zeih has joined #openstack-infra | 15:27 | |
| dhellmann | fungi, jeblair : our fancy new announce job failed on a zuul-cloner error: https://jenkins02.openstack.org/job/python-keystoneclient-announce-release/1/console | 15:28 |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config: Do not offline wheel build workers https://review.openstack.org/276300 | 15:28 |
| fungi | Shrews: makes sense that ansible inventory is instructing it to use a cache file in a location it expects | 15:29 |
| jeblair | dhellmann: any idea what the command line actually was? | 15:29 |
| *** woodster_ has joined #openstack-infra | 15:29 | |
| *** e0ne has quit IRC | 15:29 | |
| jeblair | we may want to switch to -x while setting that up | 15:29 |
| dhellmann | jeblair : yeah, I should have put that in to start | 15:29 |
| *** achanda has joined #openstack-infra | 15:30 | |
| *** e0ne has joined #openstack-infra | 15:30 | |
| dhellmann | jeblair : http://paste.openstack.org/show/485972/ | 15:30 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard: Don't add non-existent items when resolving lists https://review.openstack.org/276303 | 15:31 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config: run announce-release job with verbose flag https://review.openstack.org/276304 | 15:31 |
| *** angdraug has joined #openstack-infra | 15:31 | |
| *** esker has joined #openstack-infra | 15:32 | |
| openstackgerrit | Giulio Fidente proposed openstack-infra/tripleo-ci: Deploy using network isolation the HA job https://review.openstack.org/273424 | 15:32 |
| dhellmann | jeblair : the only thing I can think is that TOOLS_TMP is coming up empty, and throwing off the alignment | 15:32 |
| dhellmann | jeblair : is there a way to retrigger that job after adding the -x? | 15:32 |
| *** vgridnev has quit IRC | 15:32 | |
| jeblair | dhellmann: i see it | 15:33 |
| dhellmann | ah, missing backslash | 15:33 |
| *** julim has quit IRC | 15:33 | |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config: Add backslash to release tools job https://review.openstack.org/276305 | 15:34 |
| jeblair | dhellmann: ^ | 15:34 |
| anteaya | so 276305 over 276304? | 15:34 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config: fix announce-release use of zuul-cloner https://review.openstack.org/276304 | 15:35 |
| * anteaya waits | 15:35 | |
| *** julim has joined #openstack-infra | 15:35 | |
| dhellmann | jeblair : ah, my connection died just as I spotted it, so we both submitted the fix | 15:36 |
| dhellmann | jeblair, anteaya: if we take mine, you can both vote on it ;-) | 15:36 |
| anteaya | I'll vote on both since you both have teh same patch now and let the two of you decide | 15:37 |
| dhellmann | anteaya : my second version should do the same thing as jeblair's | 15:37 |
| anteaya | I'm not picking sides | 15:37 |
| anteaya | dhellmann: it does | 15:37 |
| * dhellmann is suffering from a recovering irc client replaying 1000s of lines of backlog | 15:37 | |
| anteaya | there suit yourselves | 15:38 |
| anteaya | dhellmann: nice | 15:38 |
| dhellmann | I think I need a new router :-( | 15:38 |
| dhellmann | maybe I can get one of the fancy new software-defined ones | 15:38 |
| *** krotscheck1 has joined #openstack-infra | 15:38 | |
| jeblair | dhellmann: they will certainly make sure you don't get too many packets at once | 15:39 |
| jeblair | at least, this has been my experience | 15:39 |
| anteaya | ha ha ha | 15:39 |
| dhellmann | jeblair : do you know how to connect it to my cable modem? | 15:39 |
| *** kzaitsev_mb has quit IRC | 15:39 | |
| jeblair | dhellmann: cloud | 15:39 |
| * dhellmann googles "cloud adapter cable" | 15:39 | |
| openstackgerrit | Giulio Fidente proposed openstack-infra/tripleo-ci: Deploy using network isolation the HA job https://review.openstack.org/273424 | 15:39 |
| jeblair | dhellmann: get the ones with the gold plated tips, they sound better | 15:40 |
| dhellmann | jeblair : oh, of course, always go gold plated | 15:40 |
| dhellmann | jeblair : after that change merges, can you retrigger the job? I can run that announcement by hand, but since we have one for testing already... | 15:41 |
| anteaya | +1 for gold tips | 15:41 |
| jeblair | dhellmann: yes, but i'm hoping another infra-root will :) | 15:42 |
| dhellmann | jeblair : ok :-) | 15:42 |
| openstackgerrit | Merged openstack-infra/devstack-gate: Use BUILD_TIMEOUT https://review.openstack.org/275908 | 15:42 |
| *** achanda has quit IRC | 15:42 | |
| jeblair | why do we have a periodic-neutron pipeline? | 15:43 |
| fungi | also i'm curious to see zuul-cloner actually work in the release pipeline. i thought jesusaur had identified lack of a ref envvar in release as what was tripping up z-c previously? | 15:43 |
| jeblair | (and for that matter, why do we have a periodic-qa pipeline any more?) | 15:43 |
| anteaya | I'm a big fan of thesauraus.com | 15:43 |
| dhellmann | fungi : in this case I'm unsetting all of those variables anyway because I want master. I could just run git, but it seemed fine to use the local cache | 15:43 |
| anteaya | where do folks think I found abide? | 15:43 |
| fungi | dhellmann: oh, right, you're supplying them in the script | 15:43 |
| fungi | so should work | 15:43 |
| dhellmann | fungi : right | 15:43 |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack-infra/project-config: [WIP] Trying to run Nova py27/py34 with oslo master as experimental jobs https://review.openstack.org/276313 | 15:44 |
| anteaya | jeblair: neutron wanted a periodic pipeline | 15:44 |
| anteaya | this was during nova mid-cycle if I recall | 15:44 |
| jeblair | for what purpose? | 15:44 |
| anteaya | I don't know as we have guidelines on who can create a new pipeline | 15:44 |
| *** hughsaunders has quit IRC | 15:44 | |
| jeblair | i would think 'does something unique that can not be done by an existing one' would be at least one of those guidelines :) | 15:45 |
| *** hughsaunders has joined #openstack-infra | 15:45 | |
| * anteaya is looking | 15:45 | |
| *** arxcruz has joined #openstack-infra | 15:45 | |
| *** annegentle has quit IRC | 15:46 | |
| anteaya | I can only say no to andreas so many times, I was distracted | 15:46 |
| anteaya | granted I made a bad choice | 15:46 |
| anteaya | looking for the patch | 15:46 |
| *** tzn has joined #openstack-infra | 15:46 | |
| * krotscheck1 prefers wingtips to gold tips. Of course, gold wingtips trump everything. | 15:46 | |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config: Remove periodic-neutron pipeline in favor of periodic https://review.openstack.org/276317 | 15:47 |
| dhellmann | krotscheck1 : cloud adapter cables with wingtips would make a good april fool's joke | 15:47 |
| krotscheck1 | dhellmann: We have to get our infrastructure ready for the roarin' 20's after all. | 15:47 |
| dhellmann | krotscheck1 : wings, clouds, you know | 15:48 |
| krotscheck1 | jeblair: I've been keeping up with your patches from last night. I don't see a whole lot of things that I can grab/maintain right now, is there anything that caught your attention? | 15:48 |
| krotscheck1 | I'll do a quick manual test of the wheel mirrors to make sure they work. | 15:48 |
| jeblair | krotscheck1: i don't think the job has run with the correct index generation, has it? | 15:48 |
| fungi | jeblair: i think periodic-qa used to report to the qa ml, but yeah could have been rolled into periodic once that was dropped | 15:49 |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config: Remove periodic-qa pipeline in favor of periodic https://review.openstack.org/276318 | 15:49 |
| jeblair | fungi: that is correct | 15:49 |
| krotscheck1 | jeblair: It hasn't, but that's not necessary for a test with a properly spelled package. | 15:49 |
| *** hughsaunders has quit IRC | 15:49 | |
| fungi | and i don't think periodic-neutron ever did anything different from periodic, but would have to check git history to be certain | 15:49 |
| *** hughsaunders has joined #openstack-infra | 15:49 | |
| *** kzaitsev_mb has joined #openstack-infra | 15:50 | |
| jeblair | fungi: from what i can see it git-history, that's the case. i think they just got misdirected | 15:50 |
| matbu | tristanC: yep | 15:50 |
| matbu | tristanC: we can work together | 15:50 |
| jeblair | krotscheck1: roger. i just kicked off a manual run of the job so we should have indexes soon anyway. (well, soonish) | 15:50 |
| *** hughsaunders has quit IRC | 15:50 | |
| matbu | tristanC: ther eis only a fix for cinder, iirc | 15:50 |
| jeblair | it does take 25 mins | 15:51 |
| anteaya | jeblair: oh I didn't approve it I thought I had, I saw it though: https://review.openstack.org/#/c/272880/ | 15:51 |
| krotscheck1 | jeblair: I was actually really happy to see the wheels show up on AFS :) | 15:51 |
| matbu | tristanC: feel free to update the review | 15:51 |
| anteaya | I can't remember anymore | 15:51 |
| jeblair | krotscheck1: me too, and vos release was quick | 15:51 |
| jeblair | krotscheck1: btw, even though you drop off of irc, i leave you status updates that you can check for in eavesdrop. | 15:51 |
| *** annegentle has joined #openstack-infra | 15:51 | |
| *** hughsaunders has joined #openstack-infra | 15:51 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources: Summit Application API https://review.openstack.org/221964 | 15:51 |
| anteaya | fungi: http://git.openstack.org/cgit/openstack-infra/project-config/log/?qt=grep&q=periodic-neutron | 15:52 |
| *** annegentle has quit IRC | 15:52 | |
| *** hashar has joined #openstack-infra | 15:52 | |
| tristanC | matbu: I'll have a look then after, did you fixed the invalid 'cache' parameter in stack_create ? | 15:53 |
| jeblair | yolanda: see https://review.openstack.org/276317 (related to https://review.openstack.org/#/c/272880/ ) | 15:53 |
| matbu | tristanC: i don't think so | 15:53 |
| krotscheck1 | jeblair: Good to know, thanks :) | 15:54 |
| yolanda | jeblair looking | 15:54 |
| * krotscheck1 really wishes that the HPE CYA-fest finishes up so he can have his bouncer back. | 15:54 | |
| *** ikhudoshyn has left #openstack-infra | 15:54 | |
| tristanC | matbu: alright, if you don't mind, I'll update your patch and rebase my stack_create fix on top of it | 15:54 |
| fungi | anteaya: jeblair: yeah, without any commit message indicating why they made it a separate pipeline, i'm not even sure why that got approved | 15:55 |
| anteaya | fungi: fair comment | 15:55 |
| matbu | tristanC: yes that's fine, my review is pending for a while and didn't take the time to take care of it in the last couple of weeks | 15:55 |
| yolanda | jeblair can you clarify a bit more? i see other periodic pipelines such as qa or stable | 15:55 |
| *** piet has quit IRC | 15:55 | |
| yolanda | what's criteria for allowing orn ot? | 15:55 |
| anteaya | I had thought I was in on the review of that, turns out I must have just seen it fly by | 15:56 |
| *** annegentle has joined #openstack-infra | 15:56 | |
| matbu | tristanC: and it's a blocker for my ansible V2 heat module | 15:56 |
| jeblair | yolanda: stable sends failure emails to stable maintainers, so it has a different behavior than periodic | 15:56 |
| jeblair | yolanda: -qa used to do the same thing, but to a different email address. it does not anymore, so i'm also removing it in https://review.openstack.org/276318 | 15:57 |
| *** tzn has quit IRC | 15:57 | |
| *** hashar has quit IRC | 15:57 | |
| yolanda | jeblair so general rule will be to only have one periodic for all projects? | 15:58 |
| *** AJaeger has joined #openstack-infra | 15:58 | |
| fungi | yolanda: one periodic pipeline per report destination, so all periodic jobs without a separate reporter should just share a generic periodic pipeline | 15:59 |
| *** ddieterly has joined #openstack-infra | 16:00 | |
| yolanda | ok understood | 16:00 |
| fungi | the only downside to moving periodic-qa jobs to periodic is that the log base path for them is going to change now, so the qa team would likely benefit from a heads-up | 16:00 |
| *** _nadya_ has quit IRC | 16:00 | |
| fungi | neutron less so, as that pipeline has only existed for a couple days | 16:00 |
| jeblair | mtreinish: heads-up on https://review.openstack.org/276318 :) | 16:00 |
| jesusaur | fungi: is z-c working correctly in release? that's very curious | 16:01 |
| jeblair | i believe the qa team is happily using the health db now to navigate that anyway | 16:01 |
| *** esikachev has quit IRC | 16:01 | |
| *** zeih has quit IRC | 16:01 | |
| fungi | jesusaur: turns out no, this particular job is mucking with zuul envvars to set the ones z-c wants on its own | 16:01 |
| AJaeger | speaking about periodic, I need some help reviewing this one - the change for master looks strange to me: https://review.openstack.org/274930 | 16:01 |
| krotscheck1 | jeblair: Downloading from URL http://mirror.dfw.rax.openstack.org/wheel/ubuntu-14.04-x86_64/alembic/alembic-0.8.4-py2.py3-none-any.whl | 16:01 |
| krotscheck1 | Dead easy | 16:01 |
| * AJaeger will now read backscroll on eavesdrop | 16:01 | |
| *** piet has joined #openstack-infra | 16:01 | |
| jesusaur | fungi: ahh | 16:01 |
| jeblair | AJaeger: see https://review.openstack.org/276317 | 16:01 |
| *** Daisy_ has quit IRC | 16:02 | |
| *** mtanino has joined #openstack-infra | 16:02 | |
| krotscheck1 | jeblair: It also grabbed Mako, MarkupSafe (where it actively discarded two pypi links), and a few others. | 16:02 |
| AJaeger | jeblair: when the networking queue was introduced, the idea was to easy select them in healthcheck | 16:02 |
| krotscheck1 | It couldn't find python-editor though | 16:03 |
| *** Daisy has joined #openstack-infra | 16:04 | |
| * krotscheck1 thinks we're good | 16:04 | |
| fungi | AJaeger: also worth noting, periodic job names usually start with periodic- | 16:04 |
| *** r-daneel has joined #openstack-infra | 16:04 | |
| fungi | AJaeger: note that their job-template name starts with a {pipeline} parameter which isn't even really being used | 16:05 |
| jeblair | AJaeger: a pipeline seems like a very heavyweight way to group projects; simply using a naming convention seems easier. | 16:05 |
| clarkb | krotscheck1: http://mirror.dfw.rax.openstack.org/wheel/ubuntu-14.04-x86_64/ looks empty | 16:05 |
| clarkb | not sure if we need an index there for stuff to work properly | 16:05 |
| jeblair | clarkb: index page generation was broken | 16:05 |
| *** hdd has joined #openstack-infra | 16:05 | |
| fungi | AJaeger: healthcheck groups on pipeline? | 16:06 |
| krotscheck1 | clarkb: try http://mirror.dfw.rax.openstack.org/wheel/ubuntu-14.04-x86_64/alembic | 16:06 |
| clarkb | krotscheck1: ya that has things, was just pointing out the index isn't quite right. Sounds like this is a known problem | 16:06 |
| krotscheck1 | Yeah, typo on my part. jeblair fixed it, we're rebuilding the index. | 16:06 |
| fungi | AJaeger: that rationale would have at least been useful to mention in the commit message for the change introducing the pipeline too (not that i think it's necessarily a good reason, but there was no reason mentioned at all) | 16:06 |
| jeblair | krotscheck1: release failed https://jenkins.openstack.org/job/wheel-release-trustyx64/1/console | 16:07 |
| krotscheck1 | Checking | 16:07 |
| *** trown|mtg is now known as trown | 16:07 | |
| jeblair | krotscheck1: i think we forgot to add afs to release slave :) | 16:07 |
| *** Daisy has quit IRC | 16:07 | |
| fungi | oh, whoops | 16:07 |
| krotscheck1 | .... | 16:08 |
| krotscheck1 | ..... | 16:08 |
| krotscheck1 | ....... | 16:08 |
| sdague | jeblair / fungi / et all - https://review.openstack.org/#/c/274887/ | 16:08 |
| krotscheck1 | where's the beer? | 16:08 |
| jeblair | krotscheck1: you want to grab that while i eat my wheaties? | 16:08 |
| sdague | we got a -2 merge conflict on that patch | 16:08 |
| krotscheck1 | jeblair: On it | 16:08 |
| sdague | for completely non understood reasons | 16:08 |
| *** sridhar_ram has quit IRC | 16:09 | |
| jeblair | sdague: i'll look into it post-wheaties :) | 16:09 |
| *** jed56 has joined #openstack-infra | 16:09 | |
| sdague | jeblair: ok, cool | 16:09 |
| fungi | sdague: so the change it depends on didn't fail to merge or get updated? | 16:09 |
| sdague | correct | 16:09 |
| sdague | and was a direct parent | 16:09 |
| fungi | hrm, its parent did fail to merge. digging down further | 16:09 |
| clarkb | the parent ya that | 16:10 |
| anteaya | dougwig: can you look at https://review.openstack.org/#/c/276317/1 when you get a moment? | 16:10 |
| krotscheck1 | jeblair, fungi: Urm..... this says AFS is there. http://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/manifests/release_slave.pp#n36 | 16:10 |
| AJaeger | fungi: agree, it was mentioned only in discussion in the related https://review.openstack.org/#/c/270595 | 16:10 |
| sdague | dansmith: can you explain what you saw on that patch? | 16:10 |
| *** annegentle has quit IRC | 16:11 | |
| *** moravec has joined #openstack-infra | 16:11 | |
| *** annegentle has joined #openstack-infra | 16:11 | |
| AJaeger | fungi, jeblair: So, why is a separate queue bad? Should we discuss creation here next time? | 16:11 |
| dansmith | sdague: I had a stack of like five patches, all +2+W.. Several were in gate, and when one of them merged, the others above it were kicked out as "the patch in front of this failed to merge" -- even though it did merge | 16:11 |
| fungi | er, wait, maybe i'm reading the "related changes" list upside down again. going back to comparing commit ids | 16:12 |
| krotscheck1 | jeblair, fungi: Puppet hasn't run on the release slave in 2 days | 16:12 |
| sdague | fungi: right, parents are on the bottom | 16:12 |
| AJaeger | fungi: mtreinish said "http://status.openstack.org/openstack-health/#/g/build_queue/periodic-qa" | 16:12 |
| fungi | dansmith: sdague: that sounds like the n-way merge bug in jgit may still be present | 16:12 |
| sdague | https://review.openstack.org/gitweb?p=openstack/nova.git;a=commitdiff;h=7a9942c5924d7ccfe321c54d0b5e70cfb3aede57 is the parent | 16:12 |
| AJaeger | "What I really think we should do is add a periodic-neutron queue though and just not have a email reporter and rely on openstack-health to do the job tracking. There have been a number of jobs like this and I think it'll be better to make a seperate queue for this." | 16:12 |
| *** hughsaunders has quit IRC | 16:13 | |
| dansmith | fungi: sdague: since it was not a merge conflict, I just rechecked it instead of rebasing | 16:13 |
| AJaeger | jeblair: https://review.openstack.org/#/c/276317/ is based on an abandoned change, let's rebase it... | 16:13 |
| fungi | sdague: dansmith: or maybe somehow zuul thought that change which merged didn't (stray error reported back by gerrit? timeout trying to check the state?). i'll look in zuul logs | 16:13 |
| dansmith | fungi: sdague: this is definitely not the first time I've seen that happen, and a recheck usually works fine | 16:13 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Remove periodic-neutron pipeline in favor of periodic https://review.openstack.org/276317 | 16:13 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Remove periodic-qa pipeline in favor of periodic https://review.openstack.org/276318 | 16:14 |
| sdague | fungi: actually, with all the gerrit 502 errors, how resilient is zuul to those? | 16:14 |
| clarkb | fungi: that seems plausible, though the -2 is reported at 2:22pm PST but the parent that merged doesn't merge until almost an hour later | 16:14 |
| fungi | sdague: likely not very | 16:14 |
| fungi | sdague: oh, though i think zuul is still using ssh api | 16:14 |
| clarkb | sdague: fungi except that 502s and ssh api are separate | 16:14 |
| openstackgerrit | Athlan-Guyot sofer proposed openstack-infra/project-config: puppet: initiate testing for puppet-openstack-cookicutter https://review.openstack.org/272156 | 16:14 |
| fungi | right, what clarkb said | 16:14 |
| sdague | clarkb: ok, so then very | 16:14 |
| openstackgerrit | yolanda.robla proposed openstack-infra/puppet-infracloud: Add element to bring up bridge on infra cloud https://review.openstack.org/276292 | 16:15 |
| *** xyang1 has quit IRC | 16:15 | |
| fungi | okay, so when https://review.openstack.org/274870 merged, zuul claimed its child depended on a change which failed to merge | 16:16 |
| *** hughsaunders has joined #openstack-infra | 16:16 | |
| AJaeger | fungi, I'll soon send a change to rename the gate names in the periodic pipeline... | 16:16 |
| fungi | just making sure i'm referring to the right change before i dive into logs | 16:16 |
| fungi | zuul debug logs are soooo huuuuuge | 16:18 |
| fungi | decompressing them takes a while | 16:18 |
| clarkb | jeblair: https://review.openstack.org/#/c/275995/2 is a gerrit tuning change, I have +2'd but know you were interested in these changes so haven't approved | 16:18 |
| *** regXboi has joined #openstack-infra | 16:19 | |
| *** piet has quit IRC | 16:19 | |
| greghaynes | clarkb: jeblair https://review.openstack.org/#/c/275997/ LGTM if we want to try it out | 16:20 |
| AJaeger | krotscheck1: I see a failure on https://jenkins.openstack.org/job/wheel-release-trustyx64/ - was this mentioned here already? | 16:21 |
| clarkb | greghaynes: I should review the child of that one | 16:21 |
| greghaynes | oh, oops | 16:21 |
| greghaynes | wrong change | 16:21 |
| greghaynes | I meant the gearman timeout change | 16:21 |
| greghaynes | oh, that is the correct one (ECOFFEE) | 16:22 |
| clarkb | fungi: 274870 itself had a merge fail | 16:22 |
| *** nmagnezi_ has quit IRC | 16:22 | |
| *** Daisy has joined #openstack-infra | 16:22 | |
| greghaynes | it just depends on my disconnect one, I'll fix that up | 16:22 |
| clarkb | fungi: so I think the problem is between 274870 and its parent whcih was then transitive to 274870's child | 16:22 |
| fungi | clarkb: yeah, i think i'm getting all twisted around trying to navigate this in the terrible gerrit webui | 16:22 |
| clarkb | fungi: indeed | 16:23 |
| *** esker has quit IRC | 16:23 | |
| fungi | i need to rewind and start back at the beginning again | 16:23 |
| *** yamamoto has joined #openstack-infra | 16:23 | |
| *** ihrachys has joined #openstack-infra | 16:23 | |
| fungi | i'm also uncertain gerrit is returning the things i'm asking for when i query | 16:23 |
| clarkb | fungi: my new trick is I got to a change and copy pasta the sha1 for the parent then paste that in the search bar | 16:23 |
| clarkb | because related changes is useless for figuring this out | 16:23 |
| fungi | right, did that, but then i think the resulting change's sha didn't match what i asked for | 16:24 |
| *** david-lyle has joined #openstack-infra | 16:24 | |
| clarkb | I don't think I have seen that | 16:24 |
| *** esker has joined #openstack-infra | 16:24 | |
| fungi | so 274887 which got kicked back with a dependent change failed to merge error has a parent of 7a9942c5924d7ccfe321c54d0b5e70cfb3aede57 | 16:24 |
| openstackgerrit | Merged openstack-infra/puppet-gerrit: Allow configuration of Gerrit httpd.maxQueued option https://review.openstack.org/275988 | 16:25 |
| Sam-I-Am | dhellmann: looks like tox is happy now, thx | 16:26 |
| fungi | okay, the search is pulling that one, but the merge times don't make sense for what sdague and dansmith are describing | 16:26 |
| gordc | hi, can someone with permissions please help me add lianhao lu to ceilometer-stable-maint https://review.openstack.org/#/admin/groups/533,members | 16:26 |
| clarkb | https://review.openstack.org/#/c/274869/ <- https://review.openstack.org/#/c/274870/ <- https://review.openstack.org/#/c/274887/ is what I have | 16:26 |
| clarkb | fungi: ^ | 16:26 |
| fungi | dansmith: sdague: on the "recheck because jenkins is stupid" problem, it looks like the parent change didn't merge until 45 minutes ago. the changes you rechecked were kicked back when that change failed to merge many hours ago | 16:27 |
| *** Daisy has quit IRC | 16:27 | |
| dansmith | fungi: oh, I think sdague is confused about which ones I'm talking about | 16:27 |
| fungi | dansmith: sdague: are you certain your timeline for this is what you think it is? | 16:27 |
| *** yamamoto has quit IRC | 16:27 | |
| dansmith | fungi: the one that caused them all to be kicked out was the one before 274870 | 16:27 |
| clarkb | ya 274896 is the base change | 16:28 |
| dansmith | fungi: I only rechecked 274870 last night, and when it was happy through check, I rechecked the rest this morning | 16:28 |
| fungi | okay, so 274869 merged yesterday at 23:07 utc | 16:28 |
| clarkb | yup and it was ~45 minutes before that when 274870 (its child) decided it couldn't merge because it depended on something fialing to merge | 16:29 |
| fungi | 274870 fell out of the gate at 22:22 yesterday | 16:29 |
| fungi | right | 16:29 |
| *** shardy has joined #openstack-infra | 16:30 | |
| fungi | so looking into what zuul logged about 274870 at 22:22 | 16:30 |
| clarkb | and I think 274887 just picked up whatever 274870 ran into transitively | 16:30 |
| *** sridhar_ram has joined #openstack-infra | 16:31 | |
| fungi | right, but i'm mostly curious what it was that 274870 ran into | 16:31 |
| *** esikachev has joined #openstack-infra | 16:32 | |
| dansmith | I figured it was that thing that sdague added | 16:32 |
| dansmith | if owner == "dansmith" && random.randint() < 5: fake_conflict() | 16:32 |
| fungi | you're not supposed to know about that! ;) | 16:33 |
| dansmith | well, I didn't .. UNTIL NOW | 16:33 |
| fungi | *mindwipe* | 16:33 |
| dansmith | lol | 16:33 |
| *** piet has joined #openstack-infra | 16:33 | |
| *** dizquierdo has quit IRC | 16:34 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/puppet-infracloud: Add element to bring up bridge on infra cloud https://review.openstack.org/276292 | 16:34 |
| *** harlowja_at_home has joined #openstack-infra | 16:34 | |
| openstackgerrit | Volodymyr Samotiy proposed openstack-infra/project-config: Adding broadview-lib project and jobs. https://review.openstack.org/275873 | 16:34 |
| *** infra-re_ has quit IRC | 16:36 | |
| krotscheck1 | AJaeger: Yes | 16:36 |
| *** salv-orlando has quit IRC | 16:37 | |
| krotscheck1 | AJaeger: puppet apparently hasn't run on the release slave recently | 16:37 |
| dhellmann | Sam-I-Am : good! | 16:37 |
| Sam-I-Am | dhellmann: well, i think so at least | 16:37 |
| Sam-I-Am | s/think/hope | 16:37 |
| fungi | 2016-02-03 22:22:13,512 DEBUG zuul.Scheduler: Re-enqueueing changes for pipeline gate | 16:38 |
| Sam-I-Am | dhellmann: i have no idea why that would suddenly break | 16:38 |
| fungi | it looks like we may have had a reconfiguration happen right when that change got kicked out | 16:38 |
| *** krotscheck1_ has joined #openstack-infra | 16:39 | |
| *** krotscheck1 has quit IRC | 16:39 | |
| *** krotscheck1_ is now known as krotscheck1 | 16:39 | |
| *** esikachev has quit IRC | 16:40 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Switch to dnf for fedora https://review.openstack.org/276339 | 16:40 |
| openstackgerrit | Merged openstack-infra/system-config: Disable ask-staging.openstack.org puppet run https://review.openstack.org/276277 | 16:41 |
| pabelanger | fungi: first steps for bindep on fedora ^ | 16:41 |
| jeblair | Feb 4 16:36:31 release puppet-user[8246]: Could not find data item afsadmin_keytab in any Hiera data file and no default supplied at /opt/system-config/production/manifests/site.pp:898 on node release.slave.openstack.org | 16:41 |
| pabelanger | but looks like bindep-cache is missing, checking that now | 16:41 |
| jeblair | krotscheck1: oops, need to muck with the hiera groups | 16:41 |
| krotscheck1 | jeblair: Oops. | 16:42 |
| *** Daisy has joined #openstack-infra | 16:42 | |
| *** vgridnev has joined #openstack-infra | 16:43 | |
| jeblair | clarkb, zaro: re https://review.openstack.org/275995 i can accept "match upstream default" as a reason to do that -- but there's also some commentary about "clarkb confirmed we are seeing the same error" that i would like to know more about | 16:44 |
| *** mrmartin has quit IRC | 16:46 | |
| *** armax has joined #openstack-infra | 16:46 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/puppet-infracloud: Add element to bring up bridge on infra cloud https://review.openstack.org/276292 | 16:47 |
| *** Daisy has quit IRC | 16:47 | |
| clarkb | jeblair: in the upstream change they linked 2 ml threads. On the first one is the representative traceback. That shows up in our error logs as well | 16:48 |
| jeblair | nibalizer: can a host belong to more than one group for hiera purposes? | 16:48 |
| *** amaretskiy has quit IRC | 16:48 | |
| fungi | sdague: dansmith: clarkb: so it looks like we started a reconfiguration of zuul at 2016-02-03 22:22:08,078, in the course of that 274870,2 was reenqueued into the gate pipeline at 22:22:13,523. at 22:22:33,403 its needed changes was checked and 274869,1 was identified as needed. but here's the kicker: | 16:49 |
| fungi | 2016-02-03 22:22:33,403 DEBUG zuul.DependentPipelineManager: Change <Change 0x7fc9649fc890 274869,1> in project openstack/nova does not share a change queue with <Change 0x7fca11d7ba50 274870,2> in project openstack/nova | 16:49 |
| *** mikelk has quit IRC | 16:49 | |
| jeblair | clarkb: are you suspecting that vm time is slowing processing which is pushing more connections into the queue, so we shuld increase the queue to alter service from rejected to delayed under those circumstances? | 16:49 |
| fungi | how do two changes for the same project not share a change queue? | 16:49 |
| dansmith | fungi: orly | 16:49 |
| fungi | so this seems like a reconfigure-time bug in zuul at first glimpse | 16:50 |
| fungi | jeblair: your take on that ^? | 16:50 |
| *** rhallisey has quit IRC | 16:50 | |
| dansmith | fungi: smells like sweet vindication to me :) | 16:50 |
| jeblair | fungi: i wonder if the cache leak jhesketh fixed (but is not in production) is related | 16:50 |
| fungi | oh, perhaps | 16:51 |
| clarkb | jeblair: I think when it gets really bad and the GCing is frequent the queues grow because gerrit is spending more time looking for memory. In that situation the increased queue length is likely to not help tremendously. But we also see the 500s errors less frequently when gerrit otherwise has memory and in those situations I think we either have more requests coming in than can be hanlded (lots of | 16:51 |
| jeblair | fungi: can you paste a good chunk of logs around the reconfiguration? | 16:51 |
| clarkb | clients) or perhaps the VM is starved | 16:51 |
| *** sdake has quit IRC | 16:51 | |
| *** sdake has joined #openstack-infra | 16:52 | |
| *** shardy has quit IRC | 16:52 | |
| *** jaosorior has quit IRC | 16:52 | |
| jeblair | clarkb: ok. i'll buy that increasing the queue to compensate for temorary gc problems will be good; i am worried that if we see that significantly during the rest of the time that means bigger problems | 16:52 |
| fungi | jeblair: i can try... the debug output during reconfiguration includes a dump of the entire parsed layout and also messages about reenqueuing every single active changeish's state, so is massive (and paste truncates at a pretty small content length) | 16:53 |
| fungi | but i can put together a trimmed-down summary perhaps | 16:54 |
| jeblair | fungi: i just need the bits about re-enquing things around that particular change | 16:54 |
| fungi | ahh, can do | 16:54 |
| jeblair | def do not need parsed state :) | 16:54 |
| fungi | putting something together now | 16:54 |
| *** jpr has joined #openstack-infra | 16:57 | |
| *** amrith is now known as _amrith_ | 16:57 | |
| *** jcoufal has quit IRC | 16:58 | |
| *** jtomasek__ has quit IRC | 16:58 | |
| *** _amrith_ is now known as amrith | 16:58 | |
| *** jgriffith is now known as jgriffith_away | 16:58 | |
| jeblair | remote: https://review.openstack.org/276354 Create afs-admin hiera group | 16:58 |
| jeblair | openstackgerrit: ^ | 16:58 |
| *** abregman has joined #openstack-infra | 16:59 | |
| *** e0ne has quit IRC | 16:59 | |
| jeblair | fungi: was 274869,1 merged at the time or in the queue? (related log lines there may be helpful) | 17:01 |
| krotscheck1 | jeblair: Question: If I overwrite a file with another file in AFS, and the two files are identical, vos release _will_ update because the timestamp changed, yes? | 17:01 |
| fungi | jeblair: it was still in the queue running jobs | 17:01 |
| fungi | still trimming up a summary here | 17:01 |
| *** bnemec has quit IRC | 17:02 | |
| fungi | 274869,1 eventually merged ~45 minutes later | 17:02 |
| *** samuelBartel has quit IRC | 17:02 | |
| *** gyee has joined #openstack-infra | 17:02 | |
| *** jistr has quit IRC | 17:02 | |
| *** Daisy has joined #openstack-infra | 17:02 | |
| *** ihrachys has quit IRC | 17:03 | |
| jeblair | krotscheck1: yep, but apparently just the metadata -- vos release with no changes takes 18s, vos release with a full second run of the mirror build takes 35s | 17:03 |
| *** annegentle has quit IRC | 17:03 | |
| *** annegentle has joined #openstack-infra | 17:03 | |
| jeblair | krotscheck1: (vos release for the first run took 2.5 minutes) | 17:04 |
| krotscheck1 | jeblair: Ok, so updating the index regularly isn't that much of a concern. | 17:04 |
| *** annegentle has quit IRC | 17:04 | |
| *** annegentle has joined #openstack-infra | 17:05 | |
| *** jordanP has quit IRC | 17:06 | |
| jeblair | krotscheck1: i don't think so. it might be nice to only update things with changed content, but without stable timestamps on the wheel builder (we would need to keep the builds around rather than doing clean builds each time), we'd just be switching direction -- reading all the data for rsync to do a hash compare vs writing all the data. | 17:06 |
| jeblair | krotscheck1: and the actual index file is just one more file out of the whole tree that's like that. | 17:06 |
| *** Daisy has quit IRC | 17:07 | |
| jeblair | krotscheck1: i will say that if we could avoid doing vos releases at all when nothing changes, that would be a win though | 17:07 |
| jeblair | krotscheck1: because that means the clients don't need to go back to the server and check for updates | 17:08 |
| *** abregman has quit IRC | 17:08 | |
| fungi | jeblair: hrm... my very trimmed-down summary is still >3k lines | 17:08 |
| fungi | i need to see if i can distill this further | 17:08 |
| *** jaosorior_ has joined #openstack-infra | 17:08 | |
| jeblair | krotscheck1: (in summary -- we're okay where we are, however, the less we can update, the better) | 17:08 |
| jeblair | fungi: i don't really need much beyond where those changes are either | 17:09 |
| *** jaosorior_ is now known as jaosorior | 17:09 | |
| *** annegentle has quit IRC | 17:09 | |
| *** annegentle has joined #openstack-infra | 17:09 | |
| *** _nadya_ has joined #openstack-infra | 17:09 | |
| fungi | yeah, i was just trying to capture the order in which changes were reenqueued into the gate, and what changes got ejected, and some details on the oparent change which was not ejected | 17:09 |
| *** _nadya_ has quit IRC | 17:10 | |
| jeblair | whether any interesting changes had failed jobs is good info | 17:10 |
| fungi | but there were... lots of changes in the gate | 17:10 |
| fungi | i'll skip things further down the gate pipeline than this dependent series | 17:10 |
| jeblair | (interesting meaning the ones directly connected to the ones we're looking at) | 17:10 |
| *** rockyg has joined #openstack-infra | 17:11 | |
| fungi | right | 17:11 |
| *** krotscheck1_ has joined #openstack-infra | 17:12 | |
| *** unicell has quit IRC | 17:12 | |
| *** krotscheck1 has quit IRC | 17:13 | |
| *** tphummel has joined #openstack-infra | 17:13 | |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/project-config: Only replace wheel index file if it has changed https://review.openstack.org/276358 | 17:14 |
| *** yamahata has joined #openstack-infra | 17:14 | |
| clarkb | AJaeger: for the openstack-manuals juno eol we just need to tag tip of stable/juno with juno-eol then delete the branch right? | 17:16 |
| *** jistr has joined #openstack-infra | 17:16 | |
| *** harlowja_at_home has quit IRC | 17:17 | |
| *** infra-red has joined #openstack-infra | 17:17 | |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/project-config: Only replace wheel index file if it has changed https://review.openstack.org/276358 | 17:17 |
| *** krotscheck1_ has quit IRC | 17:17 | |
| *** placko has joined #openstack-infra | 17:18 | |
| *** yaume has quit IRC | 17:18 | |
| *** jistr has quit IRC | 17:18 | |
| *** dtantsur is now known as dtantsur|afk | 17:18 | |
| *** infra-re_ has joined #openstack-infra | 17:18 | |
| *** diana_clarke has quit IRC | 17:19 | |
| *** infra-red has quit IRC | 17:22 | |
| *** Daisy has joined #openstack-infra | 17:23 | |
| *** infra-red has joined #openstack-infra | 17:23 | |
| *** krotscheck1 has joined #openstack-infra | 17:24 | |
| * krotscheck1 wishes that his client would just stay connected. | 17:24 | |
| *** flepied has joined #openstack-infra | 17:24 | |
| *** shashank_hegde has joined #openstack-infra | 17:25 | |
| *** mrmartin has joined #openstack-infra | 17:25 | |
| *** infra-re_ has quit IRC | 17:25 | |
| *** sabeen2 has quit IRC | 17:26 | |
| *** jpr has quit IRC | 17:26 | |
| clarkb | we hvaen't had any gearman disconnects yet | 17:26 |
| *** flepied1 has quit IRC | 17:26 | |
| *** jpr has joined #openstack-infra | 17:26 | |
| clarkb | I will keep tcpdump running | 17:27 |
| *** Daisy has quit IRC | 17:27 | |
| *** jpr1 has joined #openstack-infra | 17:28 | |
| fungi | heisenbug | 17:28 |
| clarkb | we have ~3.2GB of 200MB files | 17:28 |
| *** ccrouch1 has joined #openstack-infra | 17:28 | |
| clarkb | which is sustainable on /opt for now | 17:29 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Rename periodic networking jobs https://review.openstack.org/276367 | 17:30 |
| *** jpr has quit IRC | 17:30 | |
| fungi | jeblair: clarkb: finally managed to trim this down short enough for paste.o.o to not have a fit http://paste.openstack.org/show/485998 | 17:31 |
| *** tiswanso has quit IRC | 17:32 | |
| *** moravec has quit IRC | 17:32 | |
| *** tiswanso has joined #openstack-infra | 17:32 | |
| *** ccrouch has quit IRC | 17:32 | |
| *** jsavak has quit IRC | 17:33 | |
| *** Daisy has joined #openstack-infra | 17:33 | |
| *** moravec has joined #openstack-infra | 17:33 | |
| jeblair | fungi: ah, thanks! | 17:35 |
| *** trown is now known as trown|lunch | 17:35 | |
| *** shashank_hegde has quit IRC | 17:36 | |
| *** stevelle has joined #openstack-infra | 17:36 | |
| *** Swami has joined #openstack-infra | 17:37 | |
| *** Swami_ has joined #openstack-infra | 17:37 | |
| *** esker has quit IRC | 17:37 | |
| *** Swami_ has quit IRC | 17:38 | |
| *** esker has joined #openstack-infra | 17:38 | |
| *** annegentle has quit IRC | 17:39 | |
| *** salv-orlando has joined #openstack-infra | 17:39 | |
| *** Daisy has quit IRC | 17:39 | |
| openstackgerrit | Steve Martinelli proposed openstack/requirements: bump keystoneclient in u-c to 2.1.2 https://review.openstack.org/276372 | 17:39 |
| openstackgerrit | Steve Martinelli proposed openstack/requirements: bump openstackclient in u-c to 2.1.0 https://review.openstack.org/276373 | 17:41 |
| *** krtaylor has quit IRC | 17:41 | |
| *** lucasagomes is now known as lucas-dinner | 17:44 | |
| *** moravec has quit IRC | 17:44 | |
| *** SumitNaiksatam has joined #openstack-infra | 17:44 | |
| *** salv-orlando has quit IRC | 17:44 | |
| *** jsavak has joined #openstack-infra | 17:45 | |
| jeblair | fungi: based on the distance between them, i'm guessing there may have been a reconfiguration _between_ wheen 869 and 870 were enqueued... | 17:48 |
| *** unicell has joined #openstack-infra | 17:48 | |
| jeblair | (originally) | 17:48 |
| *** alivigni has joined #openstack-infra | 17:49 | |
| *** moravec has joined #openstack-infra | 17:49 | |
| *** sfinucan has quit IRC | 17:50 | |
| *** matrohon has quit IRC | 17:50 | |
| fungi | but prior to the reconfiguration at 22:22 it thought they shared a change queue. immediately after it did not | 17:50 |
| *** mrmartin has quit IRC | 17:50 | |
| *** _nadya_ has joined #openstack-infra | 17:50 | |
| *** esikachev has joined #openstack-infra | 17:51 | |
| *** smarcet has joined #openstack-infra | 17:51 | |
| krotscheck1 | fungi: jeblair's mutex change needs one more core https://review.openstack.org/#/c/275974/ | 17:52 |
| *** slogan621 has joined #openstack-infra | 17:53 | |
| *** kzaitsev_mb has quit IRC | 17:53 | |
| slogan621 | AJaeger: regarding https://review.openstack.org/#/c/275873/ | 17:54 |
| slogan621 | It's an independent, unofficial project | 17:54 |
| openstackgerrit | Merged openstack-infra/project-config: fix announce-release use of zuul-cloner https://review.openstack.org/276304 | 17:54 |
| jeblair | 2016-02-03 20:53:27,898 DEBUG zuul.DependentPipelineManager: Re-enqueing change <Change 0x7fcad096d110 274869,1> in queue <ChangeQueue gate: integrated> | 17:54 |
| jeblair | 2016-02-03 20:55:47,439 DEBUG zuul.DependentPipelineManager: Adding change <Change 0x7fca11d7ba50 274870,2> to queue <ChangeQueue gate: integrated> | 17:55 |
| *** amit213 has joined #openstack-infra | 17:55 | |
| slogan621 | we will have 3 projects, in succession - broadview-lib provides access to the metric data, broadview-collector (to follow in a few weeks) will be an agent that reads this data and publishes to Monasca API. The final project will be broadview-ui, which is a horizon panel that allows you to configure all of this. | 17:55 |
| jeblair | fungi: yeah, so 869 was already in the queue, then it was reconfigured, then 870 was added, then reconfigured again | 17:55 |
| jeblair | fungi: there is no test case for that :) | 17:55 |
| slogan621 | We have a proposal in to talk with the monasca team on this in Austin, doing the code now :-) | 17:55 |
| *** amit213 has quit IRC | 17:55 | |
| *** krtaylor has joined #openstack-infra | 17:55 | |
| jeblair | fungi: and i think i have created a failing test for it :) | 17:55 |
| fungi | jeblair: aha! | 17:56 |
| *** amit213 has joined #openstack-infra | 17:56 | |
| *** erlon has quit IRC | 17:56 | |
| *** ayoung has quit IRC | 17:58 | |
| krotscheck1 | sla | 17:58 |
| krotscheck1 | ok | 17:58 |
| krotscheck1 | Argh | 17:58 |
| krotscheck1 | Wrong window | 17:58 |
| *** ddieterly has quit IRC | 17:59 | |
| slogan621 | AJaeger: I can add the above in the code review, should I click on something, like your question, and use Reply? I'm sorta new to gerrit.... | 17:59 |
| fungi | krotscheck1: you are entitled to nine fives of uptime under your current sla | 17:59 |
| krotscheck1 | fungi: 55.5555555% | 17:59 |
| anteaya | slogan621: yes use Reply | 18:00 |
| * krotscheck1 wishes he could sleep the rest of that. | 18:00 | |
| slogan621 | anteaya: thanks. | 18:00 |
| anteaya | slogan621: welcome | 18:00 |
| *** jlanoux has quit IRC | 18:00 | |
| *** dizquierdo has joined #openstack-infra | 18:02 | |
| *** rossella_ has quit IRC | 18:02 | |
| *** rossella_ has joined #openstack-infra | 18:03 | |
| *** jsavak has quit IRC | 18:04 | |
| *** esker has quit IRC | 18:04 | |
| jeblair | fungi, jhesketh: the test case passes pre-connections :( | 18:04 |
| fungi | so regression | 18:04 |
| fungi | this did seem like sorta newish behavior | 18:05 |
| openstackgerrit | Khai Do proposed openstack-infra/system-config: Increase the HTTP incoming connection queue https://review.openstack.org/275995 | 18:05 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-infracloud: Use SSL for rabbitmq traffic https://review.openstack.org/276375 | 18:07 |
| dhellmann | jeblair : that announce job fix has merged; can I talk you into rerunning the job for python-keystoneclient? | 18:09 |
| fungi | dhellmann: i can give it a shot shortly, though i guess we shouldn't reenqueue the ref since uploading to pypi a second time will fail | 18:10 |
| fungi | so i'll have to directly retrigger the job | 18:10 |
| dhellmann | fungi : yeah, just the announce job failed and that was at the end of the list of jobs | 18:10 |
| dhellmann | fungi : what info can I pull together to have it ready for you? version? sha? | 18:11 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config: Add Infra Cloud controller node https://review.openstack.org/209698 | 18:11 |
| fungi | i'll just grab the parameters off the previous jenkins run | 18:11 |
| dhellmann | fungi : ok, cool | 18:11 |
| *** sdake has quit IRC | 18:12 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/infra-manual: Add information about enabling translations https://review.openstack.org/273759 | 18:12 |
| *** mrmartin has joined #openstack-infra | 18:12 | |
| *** Swami has quit IRC | 18:12 | |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config: Add Infra Cloud compute node definition https://review.openstack.org/213980 | 18:13 |
| *** alivigni has quit IRC | 18:15 | |
| *** Daisy has joined #openstack-infra | 18:16 | |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config: Add Infra Cloud controller node https://review.openstack.org/209698 | 18:16 |
| AJaeger | clarkb, yes, tag the tip and delete the branch, please | 18:16 |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack-infra/project-config: [WIP] Trying to run Nova py27/py34 with oslo master as experimental jobs https://review.openstack.org/276313 | 18:16 |
| *** ifarkas has quit IRC | 18:17 | |
| fungi | clarkb: oh, i didn't notice you mentioning in here that you were taking the juno-eol tag request for openstack-manuals. i just did it a few minutes agi | 18:18 |
| fungi | ago | 18:18 |
| fungi | sorry about that | 18:18 |
| clarkb | fungi: oh its fine | 18:18 |
| clarkb | I hadn't gotten around to it, but was following up after deleting the branches in those other projects | 18:18 |
| fungi | also any idea who removed the stable/icehouse branch from requirements? | 18:18 |
| anteaya | I have no idea | 18:19 |
| clarkb | I do not | 18:19 |
| clarkb | I only did the ones I responded to on the mailing list | 18:19 |
| jeblair | not i | 18:19 |
| fungi | it's happened sometime in the last 1.5 months but no idea when exactly | 18:19 |
| jeblair | oh, well, if it's 1.5 months, at least, i don't _think_ it was i :) | 18:20 |
| fungi | nevermind--i'm blind. it's not deleted | 18:20 |
| anteaya | hahaha | 18:20 |
| jeblair | it wasn't me! | 18:20 |
| anteaya | trick question | 18:20 |
| fungi | i'll follow up to my own e-mail :/ | 18:20 |
| *** Daisy has quit IRC | 18:20 | |
| AJaeger | fungi, thanks for removing the branch! | 18:21 |
| *** ociuhandu has quit IRC | 18:22 | |
| fungi | AJaeger: my pleasure. sorry it's taken me so long to get around to is | 18:23 |
| fungi | it | 18:23 |
| AJaeger | fungi: no problem - just didn't want it forgotten ;) | 18:23 |
| AJaeger | one question, why does git branch --remote locally still show gerrit/stable/juno and origin/stable/juno | 18:23 |
| fungi | it wasn't forgotten, just deep down a pile of other stuff :/ | 18:23 |
| fungi | AJaeger: git remote update | 18:24 |
| *** salv-orlando has joined #openstack-infra | 18:24 | |
| AJaeger | still there ;( | 18:24 |
| *** trown|lunch is now known as trown | 18:25 | |
| fungi | interesting. maybe it hasn't propagated yet? https://review.openstack.org/#/admin/projects/openstack/openstack-manuals,branches doesn't list it any longer | 18:25 |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid: Added User Validation https://review.openstack.org/276380 | 18:25 |
| AJaeger | git.openstack.org and github don't have it anymore either | 18:26 |
| fungi | i also don't see it at http://git.openstack.org/cgit/openstack/openstack-manuals any longer | 18:26 |
| fungi | yeah | 18:26 |
| *** Daisy has joined #openstack-infra | 18:26 | |
| dhellmann | AJaeger : did you use the purge option when you updated? | 18:26 |
| *** apoorvad has joined #openstack-infra | 18:26 | |
| AJaeger | dhellmann: no | 18:26 |
| fungi | oh, right, try adding --prune | 18:26 |
| dhellmann | right, I always think it's purge but it's prune | 18:27 |
| AJaeger | dhellmann: yes, that works! Thanks! | 18:27 |
| AJaeger | " x [deleted] (none) -> origin/stable/juno" ;) | 18:27 |
| anteaya | dried plums to the rescue | 18:27 |
| *** Swami has joined #openstack-infra | 18:28 | |
| * AJaeger is happy now, thanks for the help | 18:28 | |
| fungi | okay, so the two repos (openstack-manuals and requirements) that needed our help tagging are taken care of and the corresponding branch deletions completed | 18:28 |
| *** salv-orlando has quit IRC | 18:28 | |
| fungi | back to dhellmann's job rerun | 18:28 |
| *** e0ne has joined #openstack-infra | 18:28 | |
| * dhellmann stands ready | 18:28 | |
| AJaeger | dougwig: the qa pipeline is not sending out emails anymore - and thus not needed anymore either... | 18:29 |
| fungi | and then i'll hopefully get back to figuring out why our ansible inventory keeps going screwy | 18:29 |
| dougwig | AJaeger: ahh, then that queue is now redundant, indeed. | 18:29 |
| AJaeger | dougwig: yes | 18:29 |
| fungi | and will be cleaned up in short order as well | 18:29 |
| AJaeger | anteaya: did you want to review the periodic-qa pipeline removal as well? Or do you wait for others to review? | 18:30 |
| *** Daisy has quit IRC | 18:30 | |
| *** piet has quit IRC | 18:31 | |
| *** placko has quit IRC | 18:31 | |
| *** piet has joined #openstack-infra | 18:31 | |
| anteaya | AJaeger: I can review the -qa removal, I would like to see mtreinish and sdague at least state their acknowledement of the change prior to it merging | 18:31 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/infra-manual: Add information about enabling translations https://review.openstack.org/273759 | 18:31 |
| *** tongli has joined #openstack-infra | 18:31 | |
| cbader | clarkb, sorry to bother you but have you ever seen an error for diskimage-builder where it fails during the MySQL-python/setup.py complaining about mysql_config not found? | 18:31 |
| anteaya | AJaeger: on one of the patches you linked were this was discussed mtreinish felt he still had need of the pipeline though openstack-health was able to do some of the work, I'm not sure what his evaluation of current status is | 18:32 |
| clarkb | cbader: that means you are missing the mysqlclient dev header package with icnludes the config thing | 18:32 |
| AJaeger | anteaya: good idea | 18:32 |
| AJaeger | sdague, mtreinish: Could you review https://review.openstack.org/#/c/276318 , please? | 18:32 |
| jeblair | anteaya: which patch was that? | 18:33 |
| *** trown is now known as trown|mtg | 18:33 | |
| AJaeger | jeblair: my comment from earlier | 18:33 |
| fungi | dhellmann: waiting for a a7fb2d0,0 directory to appear at http://logs.openstack.org/7b/7bced8a36a82415f9075f3730897fab834fbeb4b/release/python-keystoneclient-announce-release/ now | 18:34 |
| AJaeger | jeblair: in 270595 | 18:34 |
| *** kushal has quit IRC | 18:35 | |
| *** Jeffrey4l has quit IRC | 18:35 | |
| dhellmann | fungi : polling... | 18:35 |
| anteaya | jeblair: https://review.openstack.org/#/c/270595 | 18:35 |
| jeblair | i see | 18:35 |
| anteaya | jeblair: added a comment linking that patch to 276318 | 18:36 |
| *** sputnik13 has joined #openstack-infra | 18:36 | |
| AJaeger | anteaya: thanks for carefully reviewing my plurals ;) | 18:36 |
| anteaya | AJaeger: you were making many changes, it was easy to miss :) | 18:37 |
| dims_ | AJaeger : general question about ci jobs.... builders get into job templates those get into job groups and then the entry in project. right? | 18:37 |
| *** ccrouch1 has quit IRC | 18:37 | |
| anteaya | jeblair: I added a comment to 276318 that adds a link to 270595 | 18:37 |
| AJaeger | dims_: For jenkins/* : Yes, that's one way of doing it. You can add builders to simple jobs as well | 18:38 |
| *** ybathia has joined #openstack-infra | 18:38 | |
| AJaeger | And you can add templates to projects.yaml as well directly | 18:38 |
| anteaya | dims_: or you can have a builder for a single job template | 18:38 |
| AJaeger | Or not use a separate builder at all ;) | 18:38 |
| anteaya | dims_: it doesn't have to have a job group, but if it does have a job group that is the heirarchy | 18:38 |
| AJaeger | dims_: best show us some code and ask | 18:38 |
| cbader | clarkb, thanks looks like I didn't have libmysqlclient-dev three other identical systems did, oh well thanks. | 18:39 |
| AJaeger | If it's a single template, you do not need a job-group, I consider it extra overhead in that case | 18:39 |
| jeblair | anteaya, AJaeger: i suspect he is just using pipelines to group jobs together; if so, let's group them together in some other way, perhaps by name, or by configuration of openstack-health. i think that the same grouping information may already even be available in the form of pipeline+project (since all of the openstack-qa periodic jobs are run on tempest). | 18:39 |
| anteaya | jeblair: that is a fair assessment and a reasonable way forward | 18:39 |
| *** annegentle has joined #openstack-infra | 18:40 | |
| AJaeger | jeblair: yes, agreed. the pipelines was perhaps the quickest way ;) | 18:40 |
| *** yamahata has quit IRC | 18:40 | |
| anteaya | things tend to get used for many things which may include the original intent | 18:40 |
| jeblair | i've asked in #openstack-qa if there's anyone who can participate in this conversation | 18:41 |
| jeblair | crickets | 18:41 |
| *** shashank_hegde has joined #openstack-infra | 18:41 | |
| anteaya | well mtreinish is at lca | 18:41 |
| anteaya | so I'm going with he's asleep at present | 18:41 |
| jeblair | which is why i was asking if there's anyone else | 18:41 |
| dims_ | anteaya : AJaeger : https://review.openstack.org/#/c/276313/ :) | 18:41 |
| *** mrmartin has quit IRC | 18:41 | |
| *** _nadya_ has quit IRC | 18:42 | |
| anteaya | dims_: we aren't creating check jobs anymore | 18:42 |
| AJaeger | dims_: that should work on the builders/job-template/job-group in general | 18:42 |
| anteaya | dims_: jobs in check and gate and experimental are all called gate | 18:43 |
| AJaeger | dims_: now let me review the fine print ;) | 18:43 |
| anteaya | dims_: jobs in periodic are called periodic- | 18:43 |
| *** eil397 has joined #openstack-infra | 18:43 | |
| dims_ | anteaya AJaeger : ack on the naming | 18:43 |
| *** amotoki has quit IRC | 18:43 | |
| dims_ | so i want to test it using experimental first and then then move it to periodic | 18:43 |
| jeblair | dims_: those pipelines have completely different characteristics | 18:44 |
| *** annegentle has quit IRC | 18:44 | |
| jeblair | dims_: one is driven by changes, the other is not | 18:44 |
| dims_ | AJaeger : anteaya : i have a travis setup https://travis-ci.org/dims/ that i use right now and i want to get that into periodic jobs | 18:44 |
| krotscheck1 | offline, back in 30 | 18:45 |
| jeblair | dims_: a job that works one way (or at all) in one is not guaranteed to work the same way (or at all) in the other | 18:45 |
| *** krotscheck1 has quit IRC | 18:45 | |
| AJaeger | dims_: on which projects do you want to run it? Right now adding it to python-jobs sounds like overkill since that means that nearly *every* of our 200ish repos will get it. | 18:45 |
| dims_ | jeblair : so let me put it another way, i'd like a periodic job that i can trigger to see if it works and debug it :) | 18:45 |
| jeblair | AJaeger: 420 | 18:45 |
| *** unicell has quit IRC | 18:46 | |
| dims_ | AJaeger : nova/ceilometer/keystone/cinder/glance/heat/ironic | 18:46 |
| *** unicell has joined #openstack-infra | 18:46 | |
| jeblair | dims_: can you do your interactive development of the job locally and then push the result to project-config? | 18:46 |
| *** annegentle has joined #openstack-infra | 18:46 | |
| fungi | boot times in ovh are looking somewhat painful... 30+ minutes now (some have been building for >40) | 18:46 |
| dims_ | jeblair : i have tested the scripts in that review by hand yes | 18:47 |
| AJaeger | dims_: create a separate job-group and add it only to those repos that need it. compare 14 jobs vs 840 jobs... | 18:47 |
| anteaya | fungi: :( | 18:47 |
| dims_ | AJaeger : yep. that's why i was asking how they are laid out :) | 18:47 |
| AJaeger | Ah ;) | 18:48 |
| AJaeger | dims_: I added further comments... | 18:48 |
| * dims_ gets lost | 18:48 | |
| dims_ | thanks AJaeger | 18:48 |
| anteaya | dims_: you are in good company | 18:48 |
| *** kushal has joined #openstack-infra | 18:48 | |
| fungi | looks like ~2.5 hours ago the "time to ready" for gra1 went off the chart at http://grafana.openstack.org/dashboard/db/nodepool-ovh | 18:49 |
| fungi | not entirely sure how to read that | 18:50 |
| fungi | pabelanger: is that supposed to dynamically size the y axis? | 18:50 |
| openstackgerrit | Merged openstack/requirements: Bump Bandit from 0.13.2 to 0.17.3 https://review.openstack.org/274939 | 18:50 |
| jeblair | fungi: i don't think it went off the chart, i think it stopped getting data | 18:50 |
| fungi | oh. ew | 18:51 |
| jeblair | (because no nodes are becoming ready?) | 18:51 |
| fungi | i guess that's an artifact of trying to chart durations? | 18:51 |
| *** jsavak has joined #openstack-infra | 18:51 | |
| jeblair | yep | 18:51 |
| fungi | but yeah, if we're going straight from building->delete then that would make sense | 18:52 |
| prometheanfire | anyone able to get one more +2? https://review.openstack.org/#/c/273772/ dib | 18:52 |
| jeblair | fungi: and ready node launch attempts is 0 | 18:52 |
| *** rkukura_ has joined #openstack-infra | 18:52 | |
| fungi | right | 18:52 |
| *** mrmartin has joined #openstack-infra | 18:52 | |
| jeblair | so yeah, i think we just lost all of gra1 | 18:52 |
| *** rkukura has quit IRC | 18:52 | |
| *** rkukura_ is now known as rkukura | 18:52 | |
| fungi | so i guess ovh-gra1 is completely out to lunch for us over the past 2.5 hours | 18:53 |
| fungi | right, that | 18:53 |
| *** dizquierdo has quit IRC | 18:53 | |
| fungi | they have a status site, but i don't know what http://status.ovh.com/vms/index_gra1.html means | 18:53 |
| *** esker has joined #openstack-infra | 18:54 | |
| fungi | is vert tres mal in french? | 18:54 |
| *** Sukhdev has joined #openstack-infra | 18:54 | |
| anteaya | fungi: I haven't seen that kind of layout for a status page before | 18:54 |
| anteaya | G104, A15 has more than 10 servers down now | 18:56 |
| anteaya | I refreshed | 18:56 |
| fungi | i see a few activities "in progress" at http://status.ovh.com/ | 18:56 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Add reconfigure test case. https://review.openstack.org/276395 | 18:56 |
| jeblair | fungi, sdague, dansmith, clarkb, jhesketh: ^ that's the test case for the observed failure | 18:57 |
| pabelanger | fungi: not actually, sure. need to see what some of jeblair code does | 18:57 |
| dansmith | jeblair: nice | 18:57 |
| fungi | pabelanger: jeblair had a good explanation | 18:58 |
| *** sridhar_ram has quit IRC | 18:58 | |
| *** trown|mtg is now known as trown | 18:58 | |
| *** ccrouch has joined #openstack-infra | 18:59 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Add reconfigure test case https://review.openstack.org/276395 | 18:59 |
| dhellmann | fungi : that job shouldn't take very long to run, but I don't see the output directory, yet. Is it queued up somewhere (I don't see anything in the zuul status, but maybe it's just in jenkins)? | 18:59 |
| fungi | dhellmann: my trigger-job.py run is still blocking so i don't think it's been picked up out of the gearman queue yet | 19:00 |
| *** kzaitsev_mb has joined #openstack-infra | 19:00 | |
| anteaya | jeblair: thank you that was the sentence I was struggling to understand | 19:00 |
| jeblair | anteaya: it was missing a | 19:00 |
| fungi | dhellmann: it may be a while. i don't think this gets to take advantage of the queue priorities that zuul sets on jobs in pipelines like release | 19:00 |
| dhellmann | fungi : ack, I'm going to step away to go to the bank then | 19:01 |
| *** ybathia has quit IRC | 19:01 | |
| fungi | dhellmann: yeah, once this script returns i'll check in on it and give you a heads up | 19:01 |
| anteaya | jeblair: yes, glad you put the indicate in there | 19:01 |
| anteaya | makes sense now | 19:01 |
| jeblair | word | 19:01 |
| jeblair | fungi: which script are you using? | 19:02 |
| anteaya | ha ha ha | 19:02 |
| *** tiswanso has quit IRC | 19:02 | |
| *** ccrouch1 has joined #openstack-infra | 19:02 | |
| fungi | jeblair: /opt/zuul/tools/trigger-job.py (we can't zuul enqueue-ref because the job we want to rerun depends on success of other jobs which are not idempotent, particularly the pypi upload) | 19:02 |
| *** esikachev has quit IRC | 19:03 | |
| *** ccrouch has quit IRC | 19:03 | |
| *** jsavak has quit IRC | 19:03 | |
| dhellmann | fungi: thanks, I'll check back in a bit | 19:03 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Use high precedence when manually submitting jobs https://review.openstack.org/276406 | 19:05 |
| jeblair | fungi: for next time | 19:05 |
| *** tiswanso has joined #openstack-infra | 19:05 | |
| *** sdake has joined #openstack-infra | 19:05 | |
| *** tiswanso has quit IRC | 19:05 | |
| fungi | ooh! | 19:05 |
| *** tiswanso has joined #openstack-infra | 19:05 | |
| fungi | great idea. we pretty much always want them prioritized if we're doing that | 19:05 |
| *** dansmith has quit IRC | 19:05 | |
| *** dansmith has joined #openstack-infra | 19:05 | |
| jeblair | fungi: good news, it's not low! so you basically just went to the back of the check queue | 19:05 |
| jeblair | fungi: yup | 19:05 |
| harlowja | mordred jeblair when u guys want to come to yahoo to do a infra talk :) | 19:09 |
| harlowja | maybe in same meeting/discussion can talk about yahoo donating resources | 19:09 |
| anteaya | jeblair: so zuul's memory did a really interesting thing here not long ago: http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=392&rra_id=all | 19:10 |
| *** david-lyle has quit IRC | 19:11 | |
| anteaya | looks like about 3 hours ago | 19:11 |
| anteaya | did you want to restart to pick up jhesketh's patch? | 19:11 |
| *** ayoung has joined #openstack-infra | 19:13 | |
| jeblair | harlowja: i'm available in the next 2 weeks or 2nd week of march; mordred is traveling, so we'll have to ask him later this week | 19:13 |
| harlowja | damn, ibm got u busy | 19:13 |
| fungi | harlowja: well, our infra-cloud sprint is in three weeks which is why i assume he's only available for the next two | 19:14 |
| harlowja | :) | 19:14 |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack-infra/project-config: [WIP] Trying to run Nova py27/py34 with oslo master as experimental jobs https://review.openstack.org/276313 | 19:14 |
| harlowja | jeblair fungi ok let me see if i can find monty | 19:14 |
| harlowja | then we'll decide | 19:14 |
| anteaya | if I were in your seat I'd proceed with jeblair | 19:15 |
| pabelanger | ianw: have you looked at cache-bindep on fedora recently? | 19:15 |
| anteaya | but that's me | 19:15 |
| fungi | i wonder if our pstn for pbx.o.o listed at https://wiki.openstack.org/wiki/Infrastructure/Conferencing#Calling has expired? anybody have details on that? i can't seem to dial it (my cell phone acts like the other end hung up before it ever rings) | 19:15 |
| *** krotscheck1 has joined #openstack-infra | 19:15 | |
| jeblair | fungi: oh drat, let me see | 19:16 |
| *** mrmartin has quit IRC | 19:16 | |
| *** Daisy has joined #openstack-infra | 19:16 | |
| pabelanger | jeblair: it is still registered to voip.ms | 19:16 |
| jeblair | we still have $2.62 on the account :) | 19:16 |
| pabelanger | asterisk -rx "sip show registry" | 19:16 |
| jeblair | voip.ms claims it's registered too | 19:16 |
| fungi | clarkb: you logged into pbx.o.o a few minutes before i said anything. were you looking into it too? | 19:17 |
| pabelanger | testing DID now | 19:17 |
| clarkb | fungi: not that specific thing | 19:17 |
| fungi | ahh | 19:17 |
| clarkb | was just checking if the service was up | 19:17 |
| fungi | yeah, same | 19:17 |
| fungi | i checked to see if it was up by trying to dial into it, which is why i ended up asking | 19:18 |
| jeblair | wrong ip address | 19:18 |
| jeblair | do we still have the old server up? | 19:18 |
| jeblair | the centos6 one? | 19:18 |
| pabelanger | I thought we deleted it | 19:18 |
| fungi | checking | 19:18 |
| *** mrmartin has joined #openstack-infra | 19:18 | |
| pabelanger | that would do it however | 19:18 |
| fungi | it's still up | 19:19 |
| fungi | 198.61.194.179, 2001:4800:780e:0510:3bc3:d7f6:ff04:5899 | 19:19 |
| pabelanger | that will do it, if asterisk is still running | 19:19 |
| *** jaosorior has quit IRC | 19:19 | |
| jeblair | yeah, voip.ms sees a registration from 198.61.194.179 | 19:19 |
| clarkb | hrm I thoguht I had deleted it but I guess not | 19:19 |
| *** ybathia has joined #openstack-infra | 19:19 | |
| fungi | up 337 days so hasn't rebooted and autostarted anything recently | 19:20 |
| fungi | however if we're puppeting based on ansible inventory now, maybe puppet assured services running again? | 19:20 |
| pabelanger | easy to check, 1 sec | 19:20 |
| *** Daisy has quit IRC | 19:20 | |
| fungi | since now ansible connects even if the machine is no longer in dns | 19:20 |
| pabelanger | yup | 19:21 |
| pabelanger | asterisk running on old system | 19:21 |
| pabelanger | which is holding registration | 19:21 |
| *** sabeen1 has joined #openstack-infra | 19:21 | |
| fungi | pretty sure the sequence here was 1. downed services and disabled puppet agent on old server, 2. switched to ansible using nova list to figure out what to puppet, 3. services started running again | 19:22 |
| *** electrofelix has quit IRC | 19:22 | |
| jeblair | do we want to delete the old server, or perhaps power it off? | 19:22 |
| fungi | i'll power it down for a start | 19:22 |
| jeblair | (though i wonder if that will make ansible hang) | 19:22 |
| pabelanger | something fishy with pbx.o.o too | 19:22 |
| fungi | it's going down now | 19:22 |
| pabelanger | dallas.voip.ms:5060 N 159094 105 Registered Sun, 20 Dec 2015 23:57:32 | 19:23 |
| pabelanger | registration is stale | 19:23 |
| pabelanger | by 6 weeks | 19:23 |
| pabelanger | it should have bounced between both of the servers | 19:23 |
| pabelanger | so, we should restart asterisk on pbx.o.o once old server is stopped | 19:23 |
| *** sabeen3 has joined #openstack-infra | 19:23 | |
| fungi | it's thoroughly stopped | 19:23 |
| fungi | doesn't respond to ping now | 19:24 |
| pabelanger | okay, let me recycle pbx.o.o | 19:24 |
| *** Sukhdev has quit IRC | 19:24 | |
| AJaeger | dims_: have a look at http://paste.openstack.org/show/486007/ | 19:24 |
| jeblair | voip.ms shows correct ip now | 19:24 |
| AJaeger | dims_: that avoids a lot of copy & paste you did for your change... | 19:24 |
| pabelanger | okay, restarted | 19:24 |
| pabelanger | pstn working | 19:25 |
| *** amitgandhinz has quit IRC | 19:25 | |
| jeblair | and sip | 19:25 |
| fungi | confirmed, seems good | 19:25 |
| *** sabeen1 has quit IRC | 19:26 | |
| pabelanger | time to add asterisk to grafana :) | 19:26 |
| *** mrmartin has quit IRC | 19:26 | |
| *** sridhar_ram has joined #openstack-infra | 19:26 | |
| pabelanger | fun friday project | 19:26 |
| *** amitgandhinz has joined #openstack-infra | 19:26 | |
| *** hashar has joined #openstack-infra | 19:27 | |
| *** _nadya_ has joined #openstack-infra | 19:27 | |
| *** piet has quit IRC | 19:30 | |
| *** ybathia has quit IRC | 19:32 | |
| *** ybathia has joined #openstack-infra | 19:32 | |
| *** piet has joined #openstack-infra | 19:33 | |
| jeblair | fungi: topic:wheel_mirror could use some +As | 19:33 |
| jeblair | clarkb: can you look at https://review.openstack.org/276354 | 19:34 |
| dims_ | AJaeger : is it "python:" or "envlist:"? | 19:34 |
| clarkb | jeblair: sure, let me refresh my memory on what the group thing does | 19:34 |
| fungi | jeblair: ooh, more wheel_mirror changes | 19:35 |
| AJaeger | dims_: the variable is "{python}" | 19:35 |
| AJaeger | So, it's "python:" | 19:35 |
| *** pblaho has quit IRC | 19:36 | |
| AJaeger | dims_: there's a colon missing after line 28 - it should be "- 'check-{name}-{python}-with-oslo-master':" | 19:36 |
| clarkb | found the docs | 19:36 |
| *** Daisy has joined #openstack-infra | 19:36 | |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack-infra/project-config: [WIP] Trying to run Nova py27/py34 with oslo master as experimental jobs https://review.openstack.org/276313 | 19:36 |
| *** sc68cal has quit IRC | 19:36 | |
| clarkb | looks like that provides the afsadmin.yaml hiera data to these machines which muts be where the keytab originates | 19:36 |
| dims_ | AJaeger : right. also fixed the name check->periodic | 19:37 |
| AJaeger | dims_: thanks, now let's see whether jenkins is happy ;) | 19:37 |
| *** jsavak has joined #openstack-infra | 19:38 | |
| dims_ | AJaeger : ack thanks muchly! | 19:38 |
| dims_ | AJaeger : so this same job, if i add it to experimental queue, that should work too right? | 19:39 |
| AJaeger | dims_: it should... | 19:40 |
| *** Daisy has quit IRC | 19:41 | |
| fungi | dims_: note that there are plenty of zuul envvar differences between periodic and experimental, so depending on what you're relying on behavior may be different | 19:41 |
| *** Sukhdev has joined #openstack-infra | 19:42 | |
| fungi | for example, you may be checking out a proposed change on an experimental run vs a branch tip on a periodic run | 19:42 |
| dims_ | fungi : ack. will stick to what i really want which is periodic | 19:42 |
| dims_ | just wish if there was a way to trigger it so i can test more than once a day :) | 19:42 |
| fungi | i suspect that's waiting for zuul to have an api trigger, so that we could have a means of firing periodic jobs without having to do hackish things like leave comments on changes in gerrit which have nothing to do with the ref which the job is going to test | 19:44 |
| fungi | s/firing periodic jobs/firing periodic jobs on demand/ | 19:45 |
| dims_ | fungi : ++++++ | 19:45 |
| AJaeger | fungi, dims_ : For this change, I think it's fine to add it to experimental - it will then run the change where the "check experimental" is issued. But that should help dims_ to test it... | 19:45 |
| fungi | yeah | 19:46 |
| *** jsavak has quit IRC | 19:46 | |
| dims_ | AJaeger : k when i update the commit message i'll add that too | 19:46 |
| AJaeger | ok | 19:47 |
| * AJaeger signs off for today - bye! | 19:47 | |
| dims_ | good night AJaeger | 19:47 |
| *** jsavak has joined #openstack-infra | 19:47 | |
| openstackgerrit | Merged openstack-infra/project-config: Atomically replace the wheel index file https://review.openstack.org/276015 | 19:48 |
| fungi | jeblair: on 276354 i believe there's a second place that has to be added for now? unless that changed very recently. looking back at what i did for the last host group i added | 19:48 |
| openstackgerrit | Merged openstack-infra/project-config: Do not offline wheel build workers https://review.openstack.org/276300 | 19:48 |
| krotscheck1 | jeblair: How critical is that mutex piece? | 19:49 |
| jeblair | krotscheck1: not critical | 19:49 |
| *** notnownikki has quit IRC | 19:49 | |
| krotscheck1 | jeblair: Oh good. | 19:50 |
| krotscheck1 | jeblair: I'm guesing the afs-admin group patch is more critical ;) | 19:50 |
| jeblair | krotscheck1: yes, that's needed for vos release to work | 19:51 |
| jeblair | krotscheck1: (though actually, the mutex is only not critical as long as the atomic index replace lands) | 19:52 |
| *** nikhil_k has joined #openstack-infra | 19:52 | |
| fungi | jeblair: left comment on 276354, you want to add them to the corresponding line in modules/openstack_project/files/puppetmaster/groups.txt too | 19:52 |
| krotscheck1 | fungi: It looks like that one landed? | 19:53 |
| jeblair | oh | 19:54 |
| jeblair | fungi: what does $group="foo" actually do then? | 19:54 |
| *** salv-orlando has joined #openstack-infra | 19:54 | |
| clarkb | jeblair: it affects which hiera files are copied | 19:55 |
| clarkb | that means also copy the foo.yaml file to the host | 19:55 |
| jeblair | then what does groups.txt do? | 19:55 |
| fungi | jeblair: yeah, that's the hiera piece, distinct from the ansible piece (though mordred has plans to combine them) | 19:55 |
| clarkb | groups.txt I do not know | 19:55 |
| *** nikhil has quit IRC | 19:55 | |
| fungi | groups.txt is the ansible group definition, the group variable in puppet is for hiera | 19:55 |
| fungi | aiui | 19:56 |
| *** stevemar has quit IRC | 19:56 | |
| jeblair | i still don't understand what causes what to happen :( | 19:56 |
| clarkb | looks like groups.txt is to disable hosts | 19:56 |
| clarkb | I believe the change as written is correct | 19:56 |
| *** Daisy has joined #openstack-infra | 19:56 | |
| fungi | oh, when i added the wheel-mirror group mordred indicated it needed to be done in both places | 19:57 |
| clarkb | (I am basing this off of the docs) | 19:57 |
| clarkb | groups.txt apperas to only be mentioned around disabling hosts | 19:57 |
| openstackgerrit | Merged openstack-infra/project-config: Only replace wheel index file if it has changed https://review.openstack.org/276358 | 19:57 |
| fungi | http://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/files/puppetmaster/expand-groups.sh is what seems to use it | 19:58 |
| krotscheck1 | nice | 19:58 |
| *** aysyd has quit IRC | 19:58 | |
| fungi | launch/launch-node.py and modules/openstack_project/manifests/puppetmaster.pp both call that | 19:58 |
| openstackgerrit | Fausto Marzi proposed openstack-infra/irc-meetings: Add Freezer team meeting https://review.openstack.org/241039 | 19:59 |
| *** EricGonc_ has joined #openstack-infra | 20:00 | |
| fungi | i don't see that generated-groups file being consumed by any of our code, so presumably this is something ansible uses itself | 20:01 |
| *** Daisy has quit IRC | 20:01 | |
| sdague | fyi, all neutron jobs are failing on py34 constraints | 20:01 |
| sdague | No matching distribution found for oslo.messaging===4.1.0 (from -c /home/jenkins/workspace/gate-neutron-python34-constraints/upper-constraints.txt (line 213)) | 20:01 |
| sdague | I assume mirror related | 20:02 |
| krotscheck1 | Very likely | 20:02 |
| *** EricGonczer_ has quit IRC | 20:02 | |
| sdague | https://jenkins01.openstack.org/job/gate-neutron-python34-constraints/1452/console | 20:02 |
| jeblair | fungi: it would make sense to me if we were trying to do a new ansible group for orchestration purposes... but... | 20:02 |
| fungi | sdague: krotscheck1: no, release problem related | 20:02 |
| fungi | https://pypi.python.org/pypi/oslo.messaging | 20:02 |
| *** sdake_ has joined #openstack-infra | 20:02 | |
| krotscheck1 | Ahhhh, no py34 artifact | 20:03 |
| jeblair | fungi: i guess we could cargo-cult it for now... | 20:03 |
| fungi | looks like it only ended up with a wheel uploaded (and it's not a universal wheel) | 20:03 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Rename periodic networking jobs https://review.openstack.org/276367 | 20:03 |
| sdague | fungi: oh, gotcha | 20:03 |
| *** sdake has quit IRC | 20:03 | |
| fungi | sdague: first i've heard of this, so presumably the release team is also unaware | 20:03 |
| sdague | could be, I just took a glance at zuul, saw the christmas tree, and got this far | 20:04 |
| *** ayoung has quit IRC | 20:04 | |
| clarkb | case of 500 error during upload maybe | 20:04 |
| fungi | jeblair: right, it's not clear to me that there needs to be a 1:1 correspondence but it would be helpful for mordred to explain the reasoning once he's around again | 20:04 |
| fungi | clarkb: most likely, yes | 20:04 |
| *** sdake has joined #openstack-infra | 20:04 | |
| *** rfolco has quit IRC | 20:04 | |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config: Create afs-admin hiera group https://review.openstack.org/276354 | 20:04 |
| jeblair | fungi, clarkb: with cargo culting ^ | 20:05 |
| *** gyee has quit IRC | 20:05 | |
| *** diana_clarke has joined #openstack-infra | 20:06 | |
| clarkb | proper SPAM worship has been applied | 20:06 |
| jeblair | fungi: did you want to review my update to https://review.openstack.org/275974 or should i carry over your +2 as moral support? :) | 20:06 |
| fungi | jeblair: oh, i missed the new patchset | 20:07 |
| *** ccrouch1 has quit IRC | 20:07 | |
| * jeblair tries not to look at clarkb | 20:07 | |
| *** rfolco has joined #openstack-infra | 20:08 | |
| *** sdake_ has quit IRC | 20:08 | |
| openstackgerrit | Anne Gentle proposed openstack-infra/project-config: Adds swagger migrate and copy job for api-site repo https://review.openstack.org/269809 | 20:09 |
| fungi | jeblair: why is the held mutex released because the holding job is complete an error condition? i'm probably not understanding | 20:09 |
| *** ccrouch has joined #openstack-infra | 20:09 | |
| jeblair | fungi: because now i expect it to be released when the job itself actually completes | 20:09 |
| fungi | oh | 20:10 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/shade: Add normalize stack function for heat stack_list https://review.openstack.org/254337 | 20:10 |
| fungi | so should be an unreachable condition | 20:10 |
| jeblair | fungi: if it hits that case, it will proceed without deadlocking but it's ... yeah | 20:10 |
| fungi | ahh, yep, i should have looked further down that file before asking ;) | 20:10 |
| jeblair | fungi: (it wasn't before -- in the previous version that was the only way it got 'unlocked') | 20:10 |
| fungi | yep, hence my confusion | 20:11 |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config: Mirror release: Add verbose flag https://review.openstack.org/276424 | 20:11 |
| tristanC | matbu: Shrews: I've updated matbu works on heat api of shade. let me know if it works for you | 20:11 |
| *** vgridnev has quit IRC | 20:11 | |
| clarkb | jeblair I think the mutex acquire is racey | 20:11 |
| clarkb | between if not m, the log debug, and setting the dict | 20:12 |
| fungi | oh, removing my workflow +1 since clarkb is reviewing | 20:12 |
| jeblair | clarkb: scheduler is single-threaded | 20:12 |
| clarkb | well its rough onphone reviewing | 20:12 |
| *** ZZelle_ has joined #openstack-infra | 20:12 | |
| jeblair | clarkb: so the actual code isn't reentrant at all | 20:13 |
| *** ZZelle_ has quit IRC | 20:13 | |
| *** andymaier has quit IRC | 20:13 | |
| clarkb | ok, it does seem odd to have mutex code that isnt thread safe | 20:14 |
| jeblair | clarkb: it's not a programming construct, it's a job scheduling construct that borrows a term from programming -- the purpose is to avoid having two jobs run at the same time | 20:14 |
| *** ldnunes has quit IRC | 20:15 | |
| *** xyang1 has joined #openstack-infra | 20:16 | |
| jeblair | clarkb: this is part of the scheduling algorithm which is designed to be single-threaded -- its interactions with other threads are well defined, but you definitely can't run two schedulers at once. | 20:17 |
| *** kzaitsev_mb has quit IRC | 20:17 | |
| *** Daisy has joined #openstack-infra | 20:17 | |
| clarkb | ya Iam mostly just worried about readability and maintainability | 20:18 |
| jeblair | (or put another way -- it is thread safe, this just all happens in one thread :) | 20:18 |
| clarkb | stuff like thid is easy yo trip over in the future if implicit expectations are not met | 20:18 |
| *** jgriffith_away is now known as jgriffith | 20:19 | |
| jeblair | clarkb: i think if a future programmer wanted a threading mutex, they should probably use threading.Lock rather the zuul.scheduler.MutexHandler | 20:20 |
| clarkb | why not use that here? | 20:20 |
| clarkb | the config load could assign a lock object to the jobs based on name then use that? | 20:21 |
| sdake | hey folks | 20:21 |
| *** ccrouch has quit IRC | 20:21 | |
| sdake | i didn't get an opportunity to vote on the naming for openstack | 20:21 |
| sdake | are my voting rights damaged in some way? | 20:21 |
| clarkb | I guess tricky bit there is reloads | 20:21 |
| *** Daisy has quit IRC | 20:21 | |
| sdake | I also didn't get an opportunity to vote for the board changes | 20:21 |
| clarkb | (as a lock must span a reload) | 20:22 |
| *** rfolco has quit IRC | 20:22 | |
| *** ccrouch has joined #openstack-infra | 20:22 | |
| clarkb | sdake are you an active foundation member? and if so is your membership at least 6 months old? I think that is the rough criteria (active is voted in one of last 2 board elections if membership was old enough to do so) | 20:23 |
| fungi | sdake: the foundation member system keys on the "I don't mind occasionally receiving updates and communications from the OpenStack Foundation." option in your profile https://www.openstack.org/profile/ | 20:23 |
| fungi | if that's not checked, you wouldn't have received notification of the board election, nor ballot e-mails for the civs elections for release names, i believe | 20:24 |
| sdake | fungi that is checked | 20:24 |
| sdake | my membership is several years old clarkb :) | 20:24 |
| sdake | i voted in the ptl and tc elections | 20:24 |
| fungi | sdake: and your primary e-mail address listed there is correct? | 20:25 |
| clarkb | sdake: those dont count, has to be board iirc | 20:25 |
| sdake | let me check | 20:25 |
| fungi | sdake: ptl and tc elections are completely unrelated | 20:25 |
| sdake | you know, I voted for the board actually | 20:25 |
| sdake | just not the release names | 20:25 |
| openstackgerrit | Merged openstack/requirements: Updated from generate-constraints https://review.openstack.org/276058 | 20:25 |
| sdake | sorry for the misinformation | 20:25 |
| fungi | sdake: ptls and tc are elected by active technical contributors. the at-large board seats and release names are selected by foundation membership | 20:25 |
| *** sputnik13 has quit IRC | 20:25 | |
| jeblair | clarkb: i believe that patch is now waiting on you; i'm late for lunch | 20:26 |
| sdake | definition of atc is commit to the repo correct? | 20:26 |
| fungi | sdake: mordred generated the ballots for the release names, so he would probably have to dig deeper to see who was in the list he used | 20:26 |
| Shrews | tristanC: kk | 20:26 |
| sdake | fungi i'll ping mordred whwne he is about | 20:26 |
| sdake | I think my wife mentioned he is stuck traveling in an airplane atm | 20:26 |
| openstackgerrit | Merged openstack-infra/ciwatch: Ability to select specific CI results https://review.openstack.org/274412 | 20:27 |
| sdake | or moving from ny to austin | 20:27 |
| sdake | yup thats the one :) | 20:27 |
| fungi | sdake: yeah, atc is interpreted for most cases as owner of a gerrit change which merged to an officially-governed team's repository within the ~calendar year prior | 20:27 |
| fungi | er, elapsed year | 20:27 |
| *** EricGonc_ has quit IRC | 20:27 | |
| sdake | fungi i thought it was moved to every 6 months? | 20:27 |
| anteaya | sdake: your wife is tracking monty? | 20:27 |
| *** sputnik13 has joined #openstack-infra | 20:27 | |
| sdake | anteaya she tracks everyone | 20:27 |
| anteaya | sdake: hopefully not me | 20:28 |
| anteaya | I'm very boring | 20:28 |
| sdake | anteaya my wife is robyn bergeron, perhaps you know her? | 20:28 |
| anteaya | I don't know as I do | 20:28 |
| sdake | shes about quite a bit | 20:28 |
| fungi | sdake: no, atc is an official criteria for the electorate. the summit organizers decide who to give discount codes for registration and lately that's been contributors to the current release | 20:28 |
| sdake | 6 months of traveal a year | 20:28 |
| clarkb | jeblair: fungi dont wsit further on me I too must lunch then have appointment in an hour | 20:28 |
| anteaya | if we've met and I've forgotten then I'm sorry | 20:28 |
| sdake | fungi thanks for explaining the rules | 20:29 |
| sdake | a wiki apge would be good ;) | 20:29 |
| anteaya | sdake: wow, that's a lot | 20:29 |
| *** flepied has quit IRC | 20:29 | |
| sdake | ya she is illing me slowly | 20:29 |
| anteaya | :( | 20:29 |
| fungi | sdake: i try to be very careful and not refer to the summit discount registration codes as "atc" anything. they're related but not the same | 20:29 |
| sdake | not getting fed, having to deal with two teenages | 20:29 |
| anteaya | ah | 20:29 |
| sdake | fungi well i wont make a wiki page, but i understand | 20:29 |
| anteaya | single parent waiting on spouse to come home, yup my mom was that one | 20:29 |
| fungi | no idea where on the wiki i'd put all these random mostly-unrelated facts that they'd be at all discoverable | 20:30 |
| sdake | so atc is contribution in last year to a repo, discount code is contribution in last 6 months | 20:30 |
| anteaya | and they'd often be out of date | 20:30 |
| fungi | right (less than 6 months really, it's since the last release's release date) | 20:30 |
| anteaya | sdake: current definitions yes | 20:30 |
| sdake | agree about not knowing where to put them | 20:30 |
| anteaya | sdake: atc definition is in the by-laws | 20:30 |
| sdake | got it | 20:30 |
| sdake | ok thanks i'll track down modred maybe he didn't like my vote for the M cycle | 20:31 |
| sdake | I chose the longest name on the list | 20:31 |
| sdake | it was like 13 characters ;-) | 20:31 |
| *** jgriffith is now known as jgriffith_away | 20:31 | |
| sdake | thanks I dont mind not voting for the name of openstack, i was more concerned about tc, ptl, and board | 20:32 |
| sdake | sounds like nothing is busted there | 20:32 |
| *** flepied has joined #openstack-infra | 20:32 | |
| ianw | clarkb: so what's the situation with image uploads atm? I've tried running a devstck/f23 job, but it seems like it still didn't grow the root partition ... which suggests it is an image without the proper growroot support, which would mean it was old-ish | 20:32 |
| anteaya | sdake: did you check your spam and trash folders in your email client? | 20:32 |
| sdake | anteaya ya but i get alot of mail i probably just missed it | 20:33 |
| *** bhunter71 has joined #openstack-infra | 20:33 | |
| crinkle | nibalizer: yolanda I added bullet points for infra-cloud to the meeting agenda if you want to review or add more | 20:34 |
| anteaya | sdake: it happens | 20:34 |
| *** TheJulia has quit IRC | 20:34 | |
| openstackgerrit | Merged openstack-infra/openstackid: Added User Validation https://review.openstack.org/276380 | 20:35 |
| anteaya | sdake: subject was: Poll: OpenStack N Release Naming | 20:35 |
| *** infra-re_ has joined #openstack-infra | 20:35 | |
| sdake | ok sorry for the noise thanks folks | 20:35 |
| anteaya | sdake: and the same for O | 20:35 |
| *** TheJulia has joined #openstack-infra | 20:35 | |
| anteaya | sdake: no problem, we take elections and polls seriously | 20:35 |
| anteaya | it is important folks get access to voting | 20:36 |
| sdake | yup i got em | 20:36 |
| sdake | thanks | 20:36 |
| sdake | just missed them | 20:36 |
| anteaya | welcome | 20:36 |
| anteaya | okay | 20:36 |
| *** stevemar has joined #openstack-infra | 20:36 | |
| *** Daisy has joined #openstack-infra | 20:37 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/shade: Fix heat create_stack and delete_stack https://review.openstack.org/276045 | 20:37 |
| *** eil397 has quit IRC | 20:38 | |
| *** lezbar has quit IRC | 20:38 | |
| *** infra-red has quit IRC | 20:38 | |
| tristanC | Shrews: and the follow-up to fix shade stack_create and stack_delete is now rebased | 20:38 |
| *** amitgandhinz has quit IRC | 20:39 | |
| fungi | looks like ovh-gra1 finally started working again about 1.5 hours ago | 20:39 |
| openstackgerrit | Fausto Marzi proposed openstack-infra/project-config: Add python-jobs and python3-jobs to freezer* https://review.openstack.org/269138 | 20:40 |
| *** russellb_ has joined #openstack-infra | 20:40 | |
| *** trown is now known as trown|outttypeww | 20:41 | |
| *** EricGonczer_ has joined #openstack-infra | 20:41 | |
| fungi | dhellmann: jeblair: the announcement job rerun seems to have gotten past the z-c phase at least http://logs.openstack.org/7b/7bced8a36a82415f9075f3730897fab834fbeb4b/release/python-keystoneclient-announce-release/a7fb2d0,0/console.html | 20:41 |
| *** jed56_ has joined #openstack-infra | 20:41 | |
| *** wolsen_ has joined #openstack-infra | 20:41 | |
| *** rvba` has joined #openstack-infra | 20:42 | |
| *** Daisy has quit IRC | 20:42 | |
| dhellmann | sigh | 20:42 |
| *** smarcet has quit IRC | 20:42 | |
| dhellmann | oh, where did "master" come from? | 20:42 |
| *** SlickN1k has joined #openstack-infra | 20:42 | |
| clarkb | ianw thr gearman client disconnects break the whole upload process | 20:42 |
| clarkb | ianw patch to address that is in review | 20:42 |
| dhellmann | fungi, jeblair: ah, it's getting "master" from ZUUL_REF | 20:43 |
| dhellmann | I expected that to include the tag | 20:43 |
| fungi | dhellmann: oh! i think that may be my fault | 20:44 |
| *** infra-red has joined #openstack-infra | 20:44 | |
| fungi | i thought i copied all the parameters from https://jenkins02.openstack.org/job/python-keystoneclient-announce-release/1/parameters/ accurately but i see i have --refname master as a copy/paste error | 20:44 |
| *** timrc_ has joined #openstack-infra | 20:44 | |
| fungi | time to try again with the right refname, sorry! | 20:45 |
| *** sshnaidm_ has joined #openstack-infra | 20:45 | |
| dhellmann | fungi : for this one I expected ref/tags/2.1.2 | 20:45 |
| *** nelsnels_ has joined #openstack-infra | 20:45 | |
| fungi | i'll use jeblair's priority tweak this time so it should be quicker | 20:45 |
| dhellmann | fungi : np, as long as it's not another round of changes to the job itself :-) | 20:45 |
| *** MCoLo__ has joined #openstack-infra | 20:45 | |
| *** vgridnev has joined #openstack-infra | 20:45 | |
| fungi | well, that might be step after next. who knows | 20:45 |
| *** briancurtin_ has joined #openstack-infra | 20:45 | |
| dhellmann | fungi : I'm more confident of the script it's trying to run, we haven't changed that in a while | 20:46 |
| * dhellmann knocks wood | 20:46 | |
| fungi | cool | 20:46 |
| *** mattymo_ has joined #openstack-infra | 20:46 | |
| openstackgerrit | Ben Swartzlander proposed openstack/requirements: Raise manilaclient upper to 1.7.0 https://review.openstack.org/274983 | 20:46 |
| *** infra-re_ has quit IRC | 20:47 | |
| *** fmccrthy_ has joined #openstack-infra | 20:47 | |
| *** moravec has quit IRC | 20:47 | |
| *** bastelfreak1 has joined #openstack-infra | 20:47 | |
| *** piet has quit IRC | 20:47 | |
| *** sridhar_ram has quit IRC | 20:47 | |
| *** jpr1 has quit IRC | 20:47 | |
| *** jed56 has quit IRC | 20:47 | |
| *** angdraug has quit IRC | 20:47 | |
| *** bastelfreak has quit IRC | 20:47 | |
| *** rvba has quit IRC | 20:47 | |
| *** isaacb has quit IRC | 20:47 | |
| *** tdasilva has quit IRC | 20:47 | |
| *** briancurtin has quit IRC | 20:47 | |
| *** timrc has quit IRC | 20:47 | |
| *** russellb has quit IRC | 20:47 | |
| *** fmccrthy has quit IRC | 20:47 | |
| *** ganso has quit IRC | 20:47 | |
| *** SlickNik has quit IRC | 20:47 | |
| *** crinkle has quit IRC | 20:47 | |
| *** sshnaidm has quit IRC | 20:47 | |
| *** lennyb has quit IRC | 20:47 | |
| *** pkoniszewski has quit IRC | 20:47 | |
| *** MCoLo has quit IRC | 20:47 | |
| *** nelsnelson has quit IRC | 20:47 | |
| *** ttx has quit IRC | 20:47 | |
| *** Sam-I-Am has quit IRC | 20:47 | |
| *** dulek has quit IRC | 20:47 | |
| *** wolsen has quit IRC | 20:47 | |
| *** mattymo has quit IRC | 20:47 | |
| *** SergK has quit IRC | 20:47 | |
| *** ctlaugh has quit IRC | 20:47 | |
| *** clif_h has quit IRC | 20:47 | |
| *** SlickN1k is now known as SlickNik | 20:47 | |
| *** jed56_ is now known as jed56 | 20:48 | |
| *** fmccrthy_ is now known as fmccrthy | 20:48 | |
| *** lennyb has joined #openstack-infra | 20:48 | |
| *** briancurtin_ is now known as briancurtin | 20:48 | |
| *** pkoniszewski has joined #openstack-infra | 20:48 | |
| *** ganso has joined #openstack-infra | 20:49 | |
| *** tdasilva has joined #openstack-infra | 20:49 | |
| *** sridhar_ram has joined #openstack-infra | 20:49 | |
| *** piet has joined #openstack-infra | 20:49 | |
| *** jpr1 has joined #openstack-infra | 20:49 | |
| *** angdraug has joined #openstack-infra | 20:49 | |
| *** crinkle has joined #openstack-infra | 20:49 | |
| *** clif_h has joined #openstack-infra | 20:49 | |
| *** ttx has joined #openstack-infra | 20:49 | |
| *** Sam-I-Am has joined #openstack-infra | 20:49 | |
| *** dulek has joined #openstack-infra | 20:49 | |
| *** Sukhdev has quit IRC | 20:49 | |
| *** jpr1 has quit IRC | 20:49 | |
| *** tdasilva has quit IRC | 20:49 | |
| *** tdasilva has joined #openstack-infra | 20:49 | |
| *** piet has quit IRC | 20:49 | |
| *** crinkle has quit IRC | 20:49 | |
| *** austin81 has joined #openstack-infra | 20:49 | |
| *** dulek has quit IRC | 20:50 | |
| *** sridhar_ram has quit IRC | 20:50 | |
| *** angdraug has quit IRC | 20:50 | |
| *** Sam-I-Am has quit IRC | 20:50 | |
| openstackgerrit | Mark Vanderwiel proposed openstack-infra/project-config: Adding integration test support for app catalog horizon plugin https://review.openstack.org/276440 | 20:51 |
| *** bastelfreak1 is now known as bastelfreak | 20:51 | |
| *** bswartz has joined #openstack-infra | 20:51 | |
| *** krotscheck1 is now known as krotscheck_dcm | 20:51 | |
| *** stevemar has quit IRC | 20:52 | |
| *** moravec has joined #openstack-infra | 20:52 | |
| *** gordc has quit IRC | 20:52 | |
| *** sridhar_ram has joined #openstack-infra | 20:53 | |
| *** arxcruz has quit IRC | 20:54 | |
| *** moravec has quit IRC | 20:55 | |
| *** SergK has joined #openstack-infra | 20:56 | |
| openstackgerrit | Armando Migliaccio proposed openstack-infra/project-config: Restrict neutron-dsvm-api to running only during the check phase https://review.openstack.org/274182 | 20:56 |
| ianw | clarkb: ok, wasn't sure if some had been pushed through | 20:58 |
| *** isaacb has joined #openstack-infra | 20:59 | |
| fungi | dhellmann: so, now we're getting "fatal: Not a valid object name tags^" http://logs.openstack.org/7b/7bced8a36a82415f9075f3730897fab834fbeb4b/release/python-keystoneclient-announce-release/a7fb2d0,1/console.html | 21:00 |
| *** apoorvad has quit IRC | 21:00 | |
| *** bgmccollum has joined #openstack-infra | 21:00 | |
| fungi | dhellmann: any clue what command is being run at that stage? | 21:01 |
| dhellmann | fungi : oh, my ref parsing is bad | 21:01 |
| dhellmann | need to cut -f3 not -f2 | 21:01 |
| fungi | ahh, yep, you were probably testing with tags/foo rather than refs/tags/foo | 21:01 |
| fungi | which git will treat similarly | 21:01 |
| *** ctlaugh has joined #openstack-infra | 21:02 | |
| dhellmann | no, I keep forgetting whether cut is 0-based or 1-based indexing | 21:02 |
| fungi | oh, heh | 21:03 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config: fix ZUUL_REF parsing in release announce job https://review.openstack.org/276446 | 21:03 |
| dhellmann | fungi : ^^ | 21:03 |
| *** e0ne has quit IRC | 21:04 | |
| *** hdd has quit IRC | 21:06 | |
| *** jpr has joined #openstack-infra | 21:06 | |
| *** EricGonczer_ has quit IRC | 21:06 | |
| *** amrith is now known as _amrith_ | 21:06 | |
| *** NobodyCa1 has joined #openstack-infra | 21:07 | |
| *** EricGonczer_ has joined #openstack-infra | 21:09 | |
| *** moravec has joined #openstack-infra | 21:10 | |
| *** esp_ has joined #openstack-infra | 21:10 | |
| *** armax has quit IRC | 21:12 | |
| *** esp_ has quit IRC | 21:14 | |
| *** NobodyCa1 has quit IRC | 21:14 | |
| *** _nadya_ has quit IRC | 21:16 | |
| mgagne | thinking about adding github enterprise support to jeepyb. shouldn't be hard. my challenge is with our current setup. the organization name doesn't match the prefix found in gerrit. :D | 21:18 |
| *** infra-red has quit IRC | 21:18 | |
| matbu | tristanC: I will test it tomorow, thanks | 21:19 |
| *** arxcruz has joined #openstack-infra | 21:20 | |
| *** kzaitsev_mb has joined #openstack-infra | 21:21 | |
| *** EricGonczer_ has quit IRC | 21:22 | |
| *** EricGonczer_ has joined #openstack-infra | 21:23 | |
| *** ybathia has quit IRC | 21:27 | |
| *** kgiusti has left #openstack-infra | 21:28 | |
| *** weshay has quit IRC | 21:29 | |
| ianw | greghaynes: made some comments on deletion change -> https://review.openstack.org/#/c/273703 . i think i understand it, but the whole thing is growing increasingly complex. sorry to sound a bit grouchy :) | 21:32 |
| *** Sam-I-Am has joined #openstack-infra | 21:35 | |
| *** slogan621 has quit IRC | 21:36 | |
| *** Daisy has joined #openstack-infra | 21:38 | |
| *** Sukhdev has joined #openstack-infra | 21:41 | |
| *** zul has joined #openstack-infra | 21:42 | |
| *** Daisy has quit IRC | 21:42 | |
| *** tiswanso has quit IRC | 21:44 | |
| *** EricGonczer_ has quit IRC | 21:44 | |
| *** fmccrthy has quit IRC | 21:44 | |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack-infra/project-config: [WIP] Trying to run Nova py27/py34 with oslo master as experimental jobs https://review.openstack.org/276313 | 21:45 |
| openstackgerrit | Anne Gentle proposed openstack-infra/project-config: Adds swagger migrate and copy job for api-site repo https://review.openstack.org/269809 | 21:45 |
| *** ybathia has joined #openstack-infra | 21:45 | |
| *** pkoniszewski has quit IRC | 21:45 | |
| *** bhunter71 has quit IRC | 21:45 | |
| fungi | clarkb: w00t! osic provisioned our tenants/accounts! | 21:46 |
| *** bhunter71 has joined #openstack-infra | 21:46 | |
| *** pradk has quit IRC | 21:46 | |
| *** baoli has quit IRC | 21:46 | |
| jeblair | fungi: do we have changes prepped for that? | 21:46 |
| fungi | jeblair: not to my knowledge, no | 21:46 |
| *** fmccrthy has joined #openstack-infra | 21:47 | |
| jeblair | what's the status of the infra-cloud nodepool changes... | 21:47 |
| *** mattymo_ has quit IRC | 21:47 | |
| fungi | we weren't really sure of the osic details (and still aren't, they're being sent to clarkb shortly it sounds like) | 21:47 |
| fungi | as far as api endpoint and other bits we'll want to put in the clouds.yaml at least | 21:48 |
| *** pradk has joined #openstack-infra | 21:48 | |
| jeblair | oh, so we need to wait for clarkb to finish his appointment | 21:48 |
| fungi | yeah | 21:48 |
| fungi | nibalizer was still tweaking hiera earlier today on infra-cloud nodepool things i think | 21:48 |
| fungi | but i don't know where that's at | 21:48 |
| anteaya | yay! | 21:49 |
| *** mattymo has joined #openstack-infra | 21:49 | |
| *** sdake has quit IRC | 21:49 | |
| *** gordc has joined #openstack-infra | 21:49 | |
| *** EricGonczer_ has joined #openstack-infra | 21:49 | |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack-infra/project-config: Experimental periodic jobs for testing against oslo.* from master https://review.openstack.org/276313 | 21:50 |
| openstackgerrit | Mark Vanderwiel proposed openstack-infra/project-config: Adding integration test support for app catalog horizon plugin https://review.openstack.org/276440 | 21:50 |
| *** FallenPegasus has quit IRC | 21:51 | |
| openstackgerrit | Mark Vanderwiel proposed openstack-infra/project-config: Adding integration test support for lbaas v2 dashboard https://review.openstack.org/275394 | 21:51 |
| *** pkoniszewski has joined #openstack-infra | 21:51 | |
| *** apoorvad has joined #openstack-infra | 21:52 | |
| *** armax has joined #openstack-infra | 21:53 | |
| jeblair | i have some comments for nibalizer, and yolanda had a question... so i don't think they're quite ready to merge yet. though it would be nice to do so soon to avoid conflicts | 21:53 |
| *** FallenPegasus has joined #openstack-infra | 21:53 | |
| jeblair | fungi: you may want to give the nodepool related changes in topic:infra-cloud a quick look | 21:53 |
| *** nwkarsten has joined #openstack-infra | 21:54 | |
| fungi | agreed, probably we should line up the osic nodepool patches behind the omfra-cloud nodepool patches to minimize conflicts/contention | 21:54 |
| fungi | thanks, i just saw a couple of your review comments fly by my e-mail for those too | 21:54 |
| *** julim has quit IRC | 21:55 | |
| *** julim has joined #openstack-infra | 21:55 | |
| *** savihou has joined #openstack-infra | 21:57 | |
| *** russellb_ is now known as russellb | 21:57 | |
| *** russellb has joined #openstack-infra | 21:57 | |
| *** Daisy has joined #openstack-infra | 21:58 | |
| *** FallenPegasus has quit IRC | 21:59 | |
| savihou | Howdy, anyone onboard who is running a ci system on top of aws ec2 ? | 22:00 |
| openstackgerrit | Frank Kloeker proposed openstack-infra/system-config: Manifest for configuration translation checksite https://review.openstack.org/276466 | 22:00 |
| fungi | savihou: not to my knowledge. since people are developing for openstack they generally have openstack-based clouds on which to put their workloads as well | 22:00 |
| anteaya | savihou: most of our third party ci operators are testing propiratary drivers with openstack | 22:01 |
| openstackgerrit | Merged openstack-infra/system-config: Create afs-admin hiera group https://review.openstack.org/276354 | 22:01 |
| *** FallenPegasus has joined #openstack-infra | 22:01 | |
| fungi | jeblair: saw your recheck on 275974. is tests.test_scheduler.TestScheduler.test_idle racy? | 22:01 |
| anteaya | savihou: here is our list of accounts with links to pages with operator contact and setup information: https://wiki.openstack.org/wiki/ThirdPartySystems | 22:01 |
| *** julim has quit IRC | 22:02 | |
| jeblair | fungi: i'm starting to think a lot of stuff may be racy after connections | 22:02 |
| openstackgerrit | Anne Gentle proposed openstack-infra/project-config: Adds swagger migrate and copy job for api-site repo https://review.openstack.org/269809 | 22:02 |
| fungi | ugh | 22:02 |
| *** rossella_ has quit IRC | 22:02 | |
| *** diana_clarke has quit IRC | 22:03 | |
| *** Daisy has quit IRC | 22:03 | |
| *** jed56 has quit IRC | 22:03 | |
| *** rossella_ has joined #openstack-infra | 22:03 | |
| *** fitoduarte has joined #openstack-infra | 22:03 | |
| *** eharney has quit IRC | 22:03 | |
| *** thorst has quit IRC | 22:04 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/shade: Fix heat create_stack and delete_stack https://review.openstack.org/276045 | 22:05 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/shade: Add normalize stack function for heat stack_list https://review.openstack.org/254337 | 22:05 |
| nibalizer | jeblair: fungi I aam back in PDX as of this afternoon | 22:05 |
| savihou | fungi: thanks for the info. | 22:05 |
| nibalizer | so there will be less time-shift lag between us | 22:05 |
| nibalizer | for hiera, I don't have the hiera lock right now | 22:05 |
| fungi | nibalizer: ooh! get some rest | 22:05 |
| nibalizer | all my changes are committed | 22:05 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/release-tools: remove notable changes option https://review.openstack.org/276468 | 22:05 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/release-tools: process pypi flag for release announcements https://review.openstack.org/276469 | 22:05 |
| jeblair | nibalizer: yay! sorry i left a -1 on your omfra changes that i should have caught earlier | 22:05 |
| nibalizer | jeblair: no worriees | 22:05 |
| nibalizer | I will check that out | 22:06 |
| annegentle | anteaya: do you have a minute? I'm bouncing an idea around in my head and would like to get an infra perspective | 22:06 |
| fungi | nibalizer: also i could use your help debugging some ansible inventory wonkiness i keep not having enough time+brains to figure out, once you're rested up | 22:06 |
| * anteaya has time to listen to annegentle | 22:06 | |
| anteaya | annegentle: go ahead | 22:06 |
| annegentle | ha! Well thanks | 22:06 |
| anteaya | :) | 22:06 |
| *** krtaylor has quit IRC | 22:06 | |
| nibalizer | fungi: sure | 22:06 |
| nibalizer | was it the stuff from yesterdaay w/ uuids instead of hostnames? | 22:07 |
| annegentle | anteaya: ok, so it's a follow-on to Tokyo when me, you and Russell Sim were talking about running a server to serve developer/api content. | 22:07 |
| fungi | nibalizer: yeah, it's still happening apparently | 22:07 |
| fungi | nibalizer: but intermittent, seems like | 22:07 |
| annegentle | anteaya: Russell originally was gonna write a spec, but I think it need to fall to me. | 22:07 |
| fungi | nibalizer: and only to rax-dfw that i've seen so far | 22:07 |
| fungi | nibalizer: some times our cached inventory has only uuids for all the hosts in dfw, but is fine for all other clouds/regions | 22:07 |
| nibalizer | heh | 22:08 |
| anteaya | annegentle: okay | 22:08 |
| fungi | nibalizer: and removing the cache and letting it repopulate seems to fix it | 22:08 |
| annegentle | anteaya: and I think you pointed to your spec template, which is pretty straightforward, http://git.openstack.org/cgit/openstack-infra/infra-specs/tree/template.rst | 22:08 |
| nibalizer | well our plan was always to put the rube-goldberg machine in place, and learn about it as we go | 22:08 |
| anteaya | annegentle: I'm so glad I did that | 22:08 |
| *** FallenPegasus has quit IRC | 22:09 | |
| savihou | anteaya: as our product is cloud based storage, this shouldn't be a storage provider issue:) | 22:09 |
| annegentle | anteaya: and now I'm thinking of dependencies... is it puppet scripting that would be an implementation detail for "let's run this doc generator, swagger-ui" | 22:09 |
| *** Guest82710 has quit IRC | 22:09 | |
| *** jamielennox|away is now known as jamielennox | 22:10 | |
| anteaya | annegentle: I think fill in the broad strokes with what you would like to see and what you think would be a good idea to get you there | 22:10 |
| annegentle | anteaya: is that enough to get the discussion going? | 22:10 |
| annegentle | anteaya: or should I do more homework? | 22:10 |
| anteaya | annegentle: let the reviewers fill in the blanks about how to puppet/ansible the thing | 22:10 |
| annegentle | anteaya: I guess that's my "stuck" part | 22:10 |
| anteaya | annegentle: yeah don't stay stuck | 22:10 |
| anteaya | annegentle: you have enough to offer a first draft | 22:10 |
| annegentle | anteaya: ok | 22:10 |
| anteaya | annegentle: I have no idea what our current puppet/ansible landscape is at this moment and it keeps changing | 22:11 |
| *** henrynash has quit IRC | 22:11 | |
| *** dkranz has quit IRC | 22:11 | |
| annegentle | anteaya: is the process, draft spec, get it posted, discuss at -infra team meeting? | 22:11 |
| *** FallenPegasus has joined #openstack-infra | 22:11 | |
| anteaya | annegentle: don't hold the expectation for yourself that you need to have that to offer the spec | 22:11 |
| annegentle | anteaya: ok good! Cuz I don't :) | 22:11 |
| *** nmagnezi_ has joined #openstack-infra | 22:11 | |
| anteaya | draft spec, submit to gerrit, gather reviewers | 22:11 |
| *** moravec has quit IRC | 22:12 | |
| anteaya | when reviewers have a fleshed out spec and are in agreement then to the infra meeting | 22:12 |
| anteaya | if you get stuck for no reviewers come back and bug in channel | 22:12 |
| *** amitgandhinz has joined #openstack-infra | 22:12 | |
| fungi | really, it's fine to fill in the use case, put tbd for everything else and push that into gerrit, then solicit some collaboration/comments on it too | 22:12 |
| openstackgerrit | Merged openstack-infra/project-config: Mirror release: Add verbose flag https://review.openstack.org/276424 | 22:12 |
| anteaya | it should be read to pass by the time it goes to infra meeting | 22:12 |
| anteaya | annegentle: yeah what fungi said | 22:12 |
| *** nwkarsten has quit IRC | 22:13 | |
| *** bhunter71 has quit IRC | 22:13 | |
| annegentle | ok thanks fungi and anteaya, super helpful. I'm soooo stuck on getting RST+Swagger framework that we need to be looking at alternatives. | 22:13 |
| anteaya | annegentle: yup, paint the broad strokes so folks can at least understand what you want | 22:14 |
| fungi | we won't bring it up for a council vote until the various sections are fleshed out, but best place to start is with the detailed description of what you want, and worry less about how it gets implemented | 22:14 |
| *** isaacb has quit IRC | 22:14 | |
| *** bhunter71 has joined #openstack-infra | 22:14 | |
| anteaya | savihou: sounds like you want to set up a ci system for yourself, have you read http://docs.openstack.org/infra/system-config/third_party.html | 22:14 |
| jeblair | i keep typing berkeley instead of kerberos | 22:15 |
| jeblair | that's weird | 22:15 |
| anteaya | ha ha ha | 22:15 |
| anteaya | heck of a typo | 22:15 |
| fungi | what an odd association | 22:15 |
| anteaya | its the k | 22:15 |
| jeblair | i know, it's totally an mit thing | 22:15 |
| fungi | lots of the same letters | 22:16 |
| annegentle | fungi: my dependency on implementors for the other is why I'm looking for an alternative, but yeah... resources for the middle. | 22:16 |
| fungi | wow, we have 20 open changes under topic:infra-cloud | 22:16 |
| *** FallenPegasus has quit IRC | 22:16 | |
| * anteaya has money on many of those patches being owned by crinkle | 22:16 | |
| jeblair | bout half | 22:17 |
| jeblair | maybe 2/3 | 22:17 |
| fungi | lots, yes | 22:17 |
| *** vgridnev has quit IRC | 22:17 | |
| fungi | i'd love to get these burned down in the next couple of weeks if we can | 22:17 |
| savihou | anteaya: I did :) I'm setting (trying) a devstack based ci system using ec2 resources .. | 22:17 |
| fungi | so that we're as caught up as possible when we meet in colorado | 22:17 |
| jeblair | fungi: i'm thinking that by the end the month we should have 0 :) | 22:17 |
| fungi | couldn't have said it better myself | 22:17 |
| anteaya | savihou: okay, I hope that is going well for you | 22:18 |
| anteaya | savihou: many newcomers are using puppet-openstackci to get set up | 22:18 |
| *** FallenPegasus has joined #openstack-infra | 22:18 | |
| *** Daisy has joined #openstack-infra | 22:18 | |
| savihou | anteaya: Intersting, not familiar with this one, will look for it. Actually, I'm not sure this is the place to talk about run time issus.. is it ? | 22:19 |
| *** krtaylor has joined #openstack-infra | 22:20 | |
| anteaya | SpamapS: when you have a minute this is an aging patch with some review commentary on it: https://review.openstack.org/#/c/213933/ | 22:20 |
| *** _amrith_ is now known as amrith | 22:20 | |
| anteaya | savihou: what kind of run time issues do you want to talk about? | 22:20 |
| anteaya | we do our best work with details | 22:20 |
| *** nwkarsten has joined #openstack-infra | 22:21 | |
| jlvillal | We would like to have the OpenStack bot automatically generate requirements.txt changes for openstack/bifrost Submitted this patch a little over a week ago: https://review.openstack.org/#/c/272334/ | 22:22 |
| *** davidlenwell has quit IRC | 22:22 | |
| jlvillal | If anyone has time to review it :) | 22:22 |
| anteaya | savihou: and all of our repos are here: http://git.openstack.org/cgit | 22:22 |
| anteaya | savihou: puppet-openstackci is here: http://git.openstack.org/cgit/openstack-infra/puppet-openstackci/tree/ | 22:22 |
| *** Daisy has quit IRC | 22:22 | |
| *** davidlenwell has joined #openstack-infra | 22:23 | |
| anteaya | jlvillal: these are the folks you need to talk to: https://review.openstack.org/#/admin/groups/131,members | 22:23 |
| anteaya | jlvillal: sometimes they are in this channel but not always | 22:23 |
| jlvillal | anteaya: Thanks. I noticed that infra-core was in the list. So I thought here might be good. | 22:24 |
| jlvillal | anteaya: But I will try over in -qa land :) | 22:24 |
| *** mase_x200 has joined #openstack-infra | 22:25 | |
| *** crinkle has joined #openstack-infra | 22:25 | |
| *** diana_clarke has joined #openstack-infra | 22:25 | |
| anteaya | jlvillal: infra does have +2 on it but usually only uses it when it is needed to unwedge the gate | 22:26 |
| *** kzaitsev_mb has quit IRC | 22:26 | |
| savihou | anteaya: ok, so here goes, I'm trying to have my dsvm-tempest-my-cinder-driver jenkins task run, but this gets hung at ...http://pastebin.com/eiLRMJy6 | 22:26 |
| jlvillal | anteaya: Ah, good to know! Thank you. | 22:26 |
| anteaya | jlvillal: welcome | 22:27 |
| *** kzaitsev_mb has joined #openstack-infra | 22:27 | |
| savihou | anteaya: that's the core ci system im using :) | 22:29 |
| *** thorst has joined #openstack-infra | 22:29 | |
| anteaya | savihou: how much time have you spend as a developer using our system and going through sucessful build logs? | 22:30 |
| anteaya | for instance a build log for a cinder dsvm tempest test run on our system | 22:31 |
| openstackgerrit | Ronald Bradford proposed openstack-infra/project-config: Add grenade testing for oslo.log https://review.openstack.org/276476 | 22:32 |
| *** rockyg has quit IRC | 22:32 | |
| savihou | anteaya: I've tried that before reaching out.. | 22:32 |
| anteaya | that == reading a cinder dsvm tempest build log run on our system? | 22:33 |
| *** kzaitsev_mb has quit IRC | 22:33 | |
| anteaya | it may be time to reach ou | 22:33 |
| anteaya | t | 22:33 |
| *** FallenPegasus has quit IRC | 22:33 | |
| anteaya | join me in #openstack-cinder | 22:33 |
| *** thorst has quit IRC | 22:34 | |
| *** tzn has joined #openstack-infra | 22:34 | |
| *** sabeen3 has quit IRC | 22:34 | |
| *** sabeen1 has joined #openstack-infra | 22:34 | |
| *** kzaitsev_mb has joined #openstack-infra | 22:34 | |
| *** EricGonczer_ has quit IRC | 22:36 | |
| *** FallenPegasus has joined #openstack-infra | 22:36 | |
| *** bgmccollum has left #openstack-infra | 22:36 | |
| openstackgerrit | Antoine Musso proposed openstack-infra/zuul: Status: apply default style by default https://review.openstack.org/276480 | 22:38 |
| openstackgerrit | James E. Blair proposed openstack-infra/infra-specs: Update docs-publishing spec to catch up to reality https://review.openstack.org/276481 | 22:38 |
| openstackgerrit | James E. Blair proposed openstack-infra/infra-specs: Update doc publishing spec for AFS https://review.openstack.org/276482 | 22:38 |
| *** tzn has quit IRC | 22:38 | |
| jeblair | annegentle, AJaeger, fungi: ^ | 22:38 |
| annegentle | woo jeblair! | 22:39 |
| *** Daisy has joined #openstack-infra | 22:39 | |
| *** salv-orl_ has joined #openstack-infra | 22:39 | |
| jeblair | that's two stages so we can make sure we know where the ground has shifted, and then separately, what to do if we use afs | 22:39 |
| openstackgerrit | Merged openstack-infra/project-config: fix ZUUL_REF parsing in release announce job https://review.openstack.org/276446 | 22:40 |
| *** sdake has joined #openstack-infra | 22:41 | |
| *** salv-orlando has quit IRC | 22:42 | |
| *** alivigni has joined #openstack-infra | 22:42 | |
| *** alivigni has quit IRC | 22:42 | |
| crinkle | does AFS require cinder? does that change the requirements for infra-cloud? | 22:43 |
| fungi | dhellmann: i'll retry that ^ again in a bit | 22:43 |
| *** Daisy has quit IRC | 22:43 | |
| openstackgerrit | Anne Gentle proposed openstack-infra/infra-specs: Adds specification for virtual server to run swagger-ui https://review.openstack.org/276484 | 22:43 |
| jeblair | crinkle: no it doesn't -- and it's actually the mirror that would drive that requirement more... | 22:44 |
| jeblair | crinkle: for the new mirrors, on clouds with cinder, i'm using a smaller node for the mirror and attaching a cinder volume | 22:44 |
| fungi | the non-afs-based mirrors would have benefitted more from cinder than the afs-based design, in fact | 22:44 |
| jeblair | crinkle: on clouds without cinder, i'm using a larger node size with a larger root partition | 22:44 |
| fungi | yeah, it's mostly a question of having enough space to contain the warm cache | 22:45 |
| jeblair | crinkle: on bluebox, where we control the flavors but do not have cinder, i'm using a small node with a large root partition | 22:45 |
| *** mase_x200 has quit IRC | 22:45 | |
| fungi | which, as you would expect, takes a lot less room that a full copy of the mirror contents | 22:45 |
| jeblair | fungi: even less than i guessed as it turns out | 22:45 |
| jeblair | like a few gb | 22:46 |
| fungi | er, than a full copy | 22:46 |
| jeblair | but that may change as we put more things in local mirrors | 22:46 |
| crinkle | jeblair: fungi: okay, sounds like I don't need to worry about it then, at least right now? | 22:46 |
| anteaya | annegentle: this is provide a server for development tools for the doc team? | 22:46 |
| jeblair | crinkle: we did not have a cinder requirement before, right? | 22:46 |
| crinkle | jeblair: correct | 22:46 |
| fungi | crinkle: yeah, this shouldn't change any requirements for infra-cloud | 22:47 |
| crinkle | jeblair: i wanted to make sure the requirements hadn't changed | 22:47 |
| jeblair | crinkle: then yeah, we're definitely not adding one :) | 22:47 |
| crinkle | \o/ | 22:47 |
| annegentle | anteaya: a server that shows the API operations in OpenStack. Here's their example website: http://petstore.swagger.io/ | 22:47 |
| fungi | but thanks for checking! | 22:47 |
| *** david-lyle has joined #openstack-infra | 22:48 | |
| jeblair | crinkle: to round things out -- our afs _servers_ do use cinder, and will probably use quite a lot of it -- but we only need a few of those and can put them anywhere in the world we want | 22:48 |
| jeblair | we do not need an afs server per-cloud or anything like that | 22:48 |
| anteaya | annegentle: okay I'm hearing that the topic is the docs team needs a server for dev tools, we favour swagger but fairy slipper exists too | 22:48 |
| *** gyee has joined #openstack-infra | 22:48 | |
| fungi | (well, anywhere in the world that has computers and network connectivity anyway) | 22:48 |
| crinkle | jeblair: okay that clears things up a bit | 22:48 |
| *** ybathia has quit IRC | 22:48 | |
| jeblair | crinkle: i said something about afs that cleared things up a bit! | 22:49 |
| * jeblair does happy dance | 22:49 | |
| *** dimtruck is now known as zz_dimtruck | 22:49 | |
| *** Daisy has joined #openstack-infra | 22:49 | |
| fungi | i vote for relocating them to a secret bunker, deep beneath the earth's crust | 22:49 |
| annegentle | anteaya: yeah, swagger-ui has a huge community around it, fairy-slipper has me, Russell Sim, Karen Hawkey, and elmiko (who is working to get up to speed) | 22:49 |
| crinkle | fungi: that will keep the data extra-safe | 22:49 |
| jeblair | fungi: i vote we test your "necessary connectivity" theory and put them on a remote beach. | 22:50 |
| annegentle | anteaya: so somewhat it buys time while serving the audience what they need | 22:50 |
| * fungi has a remote beach. we need a _remoter_ meach | 22:50 | |
| fungi | beach | 22:50 |
| anteaya | annegentle: right so just reframing the topic a bit in my mind, for me having docs devs have the tools they need is very important | 22:50 |
| anteaya | annegentle: so for me that's the topic | 22:50 |
| anteaya | annegentle: what the tool is is secondary to the fact that there should be one | 22:51 |
| anteaya | if that makes sense | 22:51 |
| annegentle | anteaya: ok, cool. Here's another site using swagger, embedded more with other content https://forgeapi.puppetlabs.com/ | 22:51 |
| annegentle | anteaya: oh, yeah, those are separate I suppose, although a good "five whys" analysis never hurts anyone | 22:51 |
| anteaya | annegentle: so for me the title of the spec is Enable serving REST API documentation | 22:52 |
| anteaya | swagger is an implimentation detail | 22:52 |
| *** david-lyle has quit IRC | 22:52 | |
| anteaya | an important one to be sure | 22:52 |
| annegentle | anteaya: sure, I struggled with the title. We do already serve REST API docs, it's just that they're FTPed and not "served" | 22:52 |
| annegentle | anteaya: is that distinction important for reviewers? The serving vs. file copying? | 22:52 |
| anteaya | fair enough | 22:52 |
| anteaya | well tools for doc devs is a more compeling title of a spec for me | 22:53 |
| anteaya | browsing specs | 22:53 |
| annegentle | anteaya: ok yeah! | 22:53 |
| anteaya | thanks | 22:54 |
| *** ybathia has joined #openstack-infra | 22:54 | |
| jeblair | Feb 4 22:34:58 release puppet-user[23578]: Could not find data item afsadmin_keytab in any Hiera data file and no default supplied at /opt/system-config/production/manifests/site.pp:902 on node relea | 22:54 |
| jeblair | se.slave.openstack.org | 22:54 |
| jeblair | fungi: ^ it copied the group file there... | 22:54 |
| *** Daisy has quit IRC | 22:54 | |
| elmiko | annegentle: hey | 22:54 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/shade: Add get_stack_outputs method https://review.openstack.org/276487 | 22:55 |
| jeblair | fungi: oh... i think i see it | 22:55 |
| *** hashar has quit IRC | 22:56 | |
| anteaya | crinkle: taking an initial pass on the infra-cloud patches, have you seen yolanda's question on https://review.openstack.org/#/c/234535/ | 22:56 |
| *** armax has quit IRC | 22:57 | |
| dhellmann | fungi : thanks, I'll check for logs in a while | 22:57 |
| fungi | jeblair: that makes... one of us | 22:57 |
| jeblair | Debug: hiera(): Looking for data source hieradata/production/group/afsadmin | 22:58 |
| jeblair | Debug: hiera(): Cannot find datafile /opt/system-config/hieradata/production/group/afsadmin.yaml, skipping | 22:58 |
| jeblair | fungi: ^ i named the file in hirea afs-admin, but put named the group in the manifest afsadmin | 22:58 |
| fungi | got it | 22:58 |
| openstackgerrit | Anne Gentle proposed openstack-infra/infra-specs: Adds specification for serving REST API info on developer.openstack.org https://review.openstack.org/276484 | 22:58 |
| jeblair | fungi: _but_ i also named the group in groups.txt afs-admin. i still suspect that's a noop. | 22:59 |
| annegentle | elmiko: hey | 22:59 |
| annegentle | elmiko: working on a spec, see what you think ^^ | 22:59 |
| jeblair | fungi: i've changed hiera but not the manifest, so we'll see shortly :) | 22:59 |
| fungi | jeblair: i don't doubt it. would love to hear what mordred says that's for besides disabling groups of servers (if anything) | 22:59 |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack-infra/project-config: Experimental periodic jobs for testing against oslo.* from master https://review.openstack.org/276313 | 22:59 |
| fungi | jeblair: or possibly for applying group roles | 23:00 |
| *** amitgandhinz has quit IRC | 23:00 | |
| *** daemontool has quit IRC | 23:01 | |
| crinkle | anteaya: I seem to recall yolanda was working with nibalizer to get that fixed elsewhere, I'll respond on the patch | 23:01 |
| *** ayoung has joined #openstack-infra | 23:02 | |
| *** r1chardj0n3s has joined #openstack-infra | 23:02 | |
| *** xyang1 has quit IRC | 23:02 | |
| r1chardj0n3s | hi, is there any chance of getting access to https://jenkins01.openstack.org/job/gate-horizon-dsvm-integration/2056/console with a login? | 23:02 |
| anteaya | crinkle: awesome thank you | 23:02 |
| anteaya | annegentle: nicely done | 23:02 |
| *** ayoung_ has joined #openstack-infra | 23:03 | |
| anteaya | r1chardj0n3s: it is done | 23:03 |
| anteaya | r1chardj0n3s: it will be scheduled for deletion, what were you looking for? | 23:04 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config: Add infracloud playbook https://review.openstack.org/234535 | 23:04 |
| *** eil397 has joined #openstack-infra | 23:07 | |
| *** nmagnezi_ has quit IRC | 23:08 | |
| r1chardj0n3s | anteaya: hi, so we're having a lot of trouble figuring out why our tests are failing on ovh & internap nodes, but passing on rax nodes. We can't reproduce the failures locally, so I need either a) access directly to an ovh/internap node (and that test I linked is an ovh node) *or* b) some instructions for how to create a VM which has exactly the same setup as those nodes | 23:08 |
| r1chardj0n3s | (sorry for the Wall Of Text there :-) | 23:08 |
| anteaya | wall of text is fine | 23:08 |
| anteaya | you had a long chat in infra channel not long ago I believe, was this the topic? | 23:09 |
| *** Daisy has joined #openstack-infra | 23:09 | |
| r1chardj0n3s | yep, it was that chat that narrowed the issue down to the node provider being a consistent factor in the failure | 23:09 |
| *** pahuang has joined #openstack-infra | 23:09 | |
| *** annegentle has quit IRC | 23:10 | |
| anteaya | okay | 23:10 |
| anteaya | I remember seeing your name in backscroll | 23:10 |
| *** yamahata has joined #openstack-infra | 23:10 | |
| anteaya | was there an agreed upon way forward? | 23:11 |
| r1chardj0n3s | so I think it was fungi who mentioned in passing the source of the ovh/internap images, but it was a little too abstract a reference for me to follow | 23:11 |
| *** kzaitsev_mb has quit IRC | 23:11 | |
| anteaya | okay fair enough | 23:11 |
| r1chardj0n3s | yep, it was | 23:11 |
| anteaya | so for us to hold a node we need enough time in advace of the job finished to hold it | 23:11 |
| r1chardj0n3s | so, fungi: could you be more specific about how I could reproduce "the devstack-trusty images in internap-nyj01 are built with dib from official ubuntu cloud base images" in a local VM? | 23:12 |
| anteaya | if it finishes we can't hold it, it has moved to the delete pile | 23:12 |
| r1chardj0n3s | (please) | 23:12 |
| r1chardj0n3s | ok, thanks anteaya | 23:12 |
| r1chardj0n3s | I think reproducing locally will be simpler | 23:12 |
| anteaya | *in advance of the job finishing | 23:12 |
| anteaya | reproducing locally is our prefered method to be sure | 23:12 |
| anteaya | I can't help you there, as you have figured out | 23:12 |
| r1chardj0n3s | definitely my preferred too :-) | 23:12 |
| anteaya | :) | 23:13 |
| anteaya | ah yes okay so rax is not using dib images | 23:13 |
| r1chardj0n3s | I'm googling "dib image" at the moment :-) | 23:13 |
| anteaya | so here is our log: http://nodepool.openstack.org/ | 23:13 |
| mordred | fungi, jeblair: groups in groups.txt are for being able to do groups in hiera | 23:13 |
| anteaya | dib == diskimage builder | 23:13 |
| *** Daisy has quit IRC | 23:13 | |
| r1chardj0n3s | ahh, thanks | 23:13 |
| anteaya | mordred: hello there | 23:13 |
| mordred | so, you can put things in hiera/groups/afs-admin.yaml and all of the machines in that group will receive copies of those secrets | 23:14 |
| anteaya | r1chardj0n3s: so the thing is that when your job is run using an env created by an image built by disk image builder, it fails | 23:14 |
| mordred | anteaya: heya | 23:14 |
| anteaya | r1chardj0n3s: rax can't upload our dib images so doesn't use them | 23:14 |
| r1chardj0n3s | ahhh! | 23:14 |
| anteaya | r1chardj0n3s: nodepool.o.o lists the disk image built images | 23:14 |
| fungi | mordred: then what are the group variables in the global site manifest for? | 23:15 |
| anteaya | hopefully it is a simple as clicking and downloading the one you want | 23:15 |
| r1chardj0n3s | ok, so I need to somehow create a VM using the dib, yep | 23:15 |
| fungi | mordred: also hey there! | 23:15 |
| r1chardj0n3s | yep | 23:15 |
| r1chardj0n3s | thanks for that ref anteaya | 23:15 |
| anteaya | r1chardj0n3s: I hope that it is useful and helps you | 23:15 |
| anteaya | r1chardj0n3s: I'm surprised I was useful | 23:16 |
| jpr | asselin_: in the single_node_ci.pp, what resolves the value of {build.parameters[LOG_PATH]} for the zuul_scheduler url_pattern? https://github.com/openstack-infra/puppet-openstackci/blob/master/manifests/single_node_ci.pp#L215 | 23:16 |
| r1chardj0n3s | :-) | 23:16 |
| anteaya | :) | 23:16 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config: Fix typo in afs admin group https://review.openstack.org/276494 | 23:16 |
| jeblair | fungi, mordred: ^ | 23:16 |
| ianw | r1chardj0n3s: 1) create virtualenv, pip install diskimage_builder 2) git clone git.openstack.org/openstack-infra/project-config 3) run ./tools/build-image.sh 4) ask more questions when something doesn't work :) | 23:16 |
| fungi | r1chardj0n3s: to be a little more precise, we have trouble with getting our images into swift for rackspace's task-based glance uploads | 23:17 |
| *** annegentle has joined #openstack-infra | 23:17 | |
| fungi | an outstanding bug we're working through | 23:17 |
| anteaya | ianw fungi: ah thank you, appreciate the details | 23:17 |
| jpr | asselin_: my zuul pipeline complains about: KeyError: 'LOG_PATH' which I think stems from this value | 23:17 |
| jpr | 23:17 | |
| r1chardj0n3s | ianw, fungi: thanks! | 23:17 |
| *** regXboi has quit IRC | 23:19 | |
| r1chardj0n3s | ianw: so there's not a place I can just download the dib that the ovh/internap nodes use? | 23:19 |
| jeblair | fungi, mordred: let me take a stab at this: groups.txt causes group files of the same name to be copied to the hosts in those groups. $group="foo" causes a host to know to look in the file that was copied. | 23:20 |
| ianw | r1chardj0n3s: no, in a word | 23:20 |
| jeblair | r1chardj0n3s, ianw: this is something we would like to do | 23:20 |
| r1chardj0n3s | ok, I'm just concerned I won't have exactly the same environment, but I'll give it a shot | 23:20 |
| *** angdraug has joined #openstack-infra | 23:21 | |
| ianw | jeblair: yes, well with the big reduction in the types of images it might be moving into the realm of practical | 23:21 |
| ianw | r1chardj0n3s : i'm not sure about those providers, but i have managed to get the image shared with me on RAX ... they have an interface for letting others get to snapshots. but you'd need an account, and one of the admins would need to log in | 23:22 |
| *** SumitNaiksatam has quit IRC | 23:22 | |
| *** SumitNaiksatam has joined #openstack-infra | 23:23 | |
| *** bhunter71 has quit IRC | 23:23 | |
| ianw | i think you mentioned holding a node ... if there's on in flight one of the root's can do that if poking around will help | 23:23 |
| fungi | jeblair: oh, yes the hosts need to know what group file they're looking for. clicking now | 23:23 |
| r1chardj0n3s | ianw: unfortunately I don't need the rax image, since I know they're OK :-) | 23:23 |
| r1chardj0n3s | ianw: yep, anteaya mentioned I need to ask while in flight. I'm going to try to reproduce locally before asking though. | 23:24 |
| r1chardj0n3s | now I know about dib and stuff | 23:24 |
| fungi | r1chardj0n3s: main reason we don't publish the images we're using is that they've got our access baked into them, and the easiest way to customize them so that you can log in is to build them again | 23:24 |
| r1chardj0n3s | ah, ok, thanks | 23:25 |
| nibalizer | jeblair: lgtm | 23:25 |
| anteaya | fungi: I didn't know that, thank you | 23:25 |
| fungi | r1chardj0n3s: since their only purpose, currently, is for our ci to run jobs on, we're not introducing the additional complications from making them post-build configurable to bootstrap new access credentials | 23:25 |
| anteaya | r1chardj0n3s: if you have to ask us to hold a node ask with the link to a paste with your public key | 23:25 |
| anteaya | or be ready to pm or email it | 23:26 |
| *** bhunter71 has joined #openstack-infra | 23:26 | |
| fungi | yeah, i'm happy to place a worker on hold if it's running a job you think is going to fail, and then authorize your ssh key to log into the jenkins user on it | 23:26 |
| jeblair | fungi: glean will support key injection, so we're basically just missing a qick howto or simple script describing how to boot an image locally with a configdrive setup. | 23:27 |
| r1chardj0n3s | hmm, there's some dependencies I need to install for ./tools/build-image.sh | 23:27 |
| fungi | jeblair: good point | 23:27 |
| *** sputnik13 has quit IRC | 23:27 | |
| r1chardj0n3s | are there docs for dependency installs for this tool? | 23:27 |
| fungi | jeblair: i guess we're using that with ironic? | 23:27 |
| *** gordc has quit IRC | 23:28 | |
| *** pradk has quit IRC | 23:28 | |
| jeblair | fungi: i think if we just start publishing them, we can convince mordred to do that | 23:28 |
| fungi | r1chardj0n3s: if there aren't comments in the script as to basic dependencies, then we should probably add some (or a readme) | 23:28 |
| jeblair | fungi: i don't know? | 23:28 |
| nibalizer | I think publishing the images would be so cool | 23:29 |
| fungi | r1chardj0n3s: you're at least the third person we've pointed at that script this week, so it's becoming popular | 23:29 |
| r1chardj0n3s | fungi: afaict no script comment and readme points to http://docs.openstack.org/infra/system-config/nodepool.html which doesn't have the dependencies either afaict | 23:29 |
| r1chardj0n3s | tho there's a puppet link from there, which might be helpful if I knew puppet :-) | 23:29 |
| *** Daisy has joined #openstack-infra | 23:29 | |
| *** kzaitsev_mb has joined #openstack-infra | 23:30 | |
| *** chlong has quit IRC | 23:30 | |
| ianw | r1chardj0n3s: in dib, try tests/install_test_deps.sh script, that should get it | 23:31 |
| r1chardj0n3s | ah! thanks | 23:31 |
| *** dims_ has quit IRC | 23:31 | |
| *** jamielennox is now known as jamielennox|away | 23:31 | |
| clarkb | dib documents its deps | 23:31 |
| jpr | asselin_: well I see it now its actually meant for zuul gearman.py to consume : http://git.openstack.org/cgit/openstack-infra/zuul/tree/zuul/launcher/gearman.py#n343 | 23:31 |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack-infra/project-config: Experimental periodic jobs for testing against oslo.* from master https://review.openstack.org/276313 | 23:31 |
| clarkb | I am almost homr and will retrieve cloud stuff for osic | 23:31 |
| fungi | r1chardj0n3s: also, we're using it (for nodepool) under ubuntu 14.04 lts (trusty tahr) right now, so if you've chosen a different base distro to run dib on, i'm not sure what that might affect | 23:32 |
| r1chardj0n3s | oh | 23:32 |
| fungi | in theory dib works on multiple platforms | 23:32 |
| *** dims has joined #openstack-infra | 23:32 | |
| fungi | though you're likely to get a little better help with it in the tripleo channel, since they're the authors | 23:33 |
| *** sputnik13 has joined #openstack-infra | 23:33 | |
| anteaya | clarkb: travel safely | 23:33 |
| r1chardj0n3s | so yeah, I'm running it on my 15.04 desktop... | 23:33 |
| fungi | that may be close enough | 23:33 |
| fungi | we can mostly tell you how we've used it and what we've seen, but our experience is mostly limited (in production) to running it on 14.04 | 23:33 |
| fungi | i expect that newer ubuntu would work just as well | 23:33 |
| nibalizer | jeblair: re 275477 is there really a benefit to doing it that way? | 23:34 |
| jeblair | nibalizer: less typing? | 23:34 |
| openstackgerrit | Tim Burke proposed openstack-dev/hacking: Run all flake8 checks https://review.openstack.org/276500 | 23:34 |
| nibalizer | seems like we save one less password but we do so at the cost of special-casing infra | 23:34 |
| nibalizer | special-casing infra clodu | 23:34 |
| nibalizer | it seems to me like the different IC regions are going to be up and down and reloaded at different times, so administratively flattening their auth doesn't make a lot of sense | 23:34 |
| *** Daisy has quit IRC | 23:34 | |
| *** sigmavirus24 is now known as sigmavirus24_awa | 23:35 | |
| nibalizer | if they shared a keystone then it wouldn't even be an issue | 23:35 |
| jeblair | nibalizer: well, i guess it's sort of special -- but we're also walking a line with the fact that our two "regions" are really "two clouds" | 23:35 |
| jeblair | nibalizer: but i think ultimately we have discussed that it would be nice to have a central keystone | 23:35 |
| jeblair | nibalizer: so it's partially thinking ahead to that, and partially just that i'm not sure what having different creds gets us. | 23:36 |
| *** sputnik13 has quit IRC | 23:36 | |
| *** dingyichen has joined #openstack-infra | 23:37 | |
| *** dingyichen has quit IRC | 23:39 | |
| *** bhunter71 has quit IRC | 23:39 | |
| mordred | jeblair: yes to your stab on groups | 23:39 |
| r1chardj0n3s | ianw: I'm not sure what you mean by "in dib" in your "in dib, try tests/install_test_deps.sh script" (I can't find that in the project-config git clone or the venv I installed diskimage_builder into) | 23:39 |
| ianw | r1chardj0n3s: ahh, yeah, ok, that script is probably only if you git checked it out | 23:40 |
| mordred | jeblair: however, in the future, I tink $group = 'foo' can go away and we can just have the host read anything in its hiera hierarchy | 23:40 |
| mordred | jeblair: and let the copying do the segregation | 23:40 |
| *** FallenPegasus has quit IRC | 23:40 | |
| mordred | jeblair: but that's optimization for later | 23:40 |
| ianw | r1chardj0n3s: apt-get install -y debootstrap inetutils-ping kpartx qemu-utils | 23:40 |
| ianw | should get you mostly there | 23:41 |
| r1chardj0n3s | cheers | 23:41 |
| r1chardj0n3s | ah, found the install_deps script | 23:41 |
| ianw | i think it checks early for qemu conversion tools, that's probably where you failed | 23:41 |
| *** dingyichen has joined #openstack-infra | 23:42 | |
| *** rkukura has quit IRC | 23:42 | |
| r1chardj0n3s | lol "curl: command not found" :-) | 23:42 |
| jeblair | mordred: ah that would be good -- that also gets to another question i had -- can you list a host in more than one group in groups.txt? | 23:42 |
| *** FallenPegasus has joined #openstack-infra | 23:42 | |
| r1chardj0n3s | ok, script is progressing now :-) | 23:43 |
| jeblair | mordred: (because if you drop the need for $group="foo", then groups.txt is the only place where multiple membership would matter) | 23:43 |
| mordred | jeblair: yes, you can in groups.txt | 23:43 |
| mordred | jeblair: but you cannot list more than one group in $group='foo' | 23:43 |
| jeblair | mordred: cool, so that'll be a win | 23:44 |
| mordred | jeblair: which is where making $group= go away becomes handy. yah | 23:44 |
| jeblair | nibalizer: i'm not set on having one set of creds -- if it helps to have 2 that's fine -- i just thought it wasn't necessary and would be simpler to have 1 set. | 23:45 |
| jeblair | nibalizer: but that's also why i asked fungi to check out the changes too -- we could use more opinions | 23:45 |
| nibalizer | jeblair: I think as long as the backend has the possibility of two creds then our config mgmt should support different passwords | 23:45 |
| *** denisra has joined #openstack-infra | 23:45 | |
| nibalizer | for instance you can't even auth to east right now, its just not up, so right now the passwords are not the same | 23:46 |
| fungi | yeah, they're on my queue for tonight or tomorrow to start going through | 23:46 |
| nibalizer | cool | 23:46 |
| fungi | i'm happy to provide (terrible, terrible) opinions | 23:46 |
| nibalizer | fungi: i mean we do have a krb5 server.. | 23:47 |
| *** rkukura has joined #openstack-infra | 23:47 | |
| fungi | this is true | 23:47 |
| fungi | isn't kerberos really the keystone v4 api? | 23:47 |
| jeblair | nibalizer: yeah, i'm just saying, let's not make it hard for ourselves and use the same password both places :) | 23:48 |
| nibalizer | jeblair: well but then why special case infracloud? Why not make all cloud passworrds the same? | 23:49 |
| jeblair | nibalizer: because they are different trust domains -- i don't trust rax with our ovh password. but i do trust the infra-cloud-east admins with our infra-cloud-west password | 23:50 |
| *** sputnik13 has joined #openstack-infra | 23:50 | |
| jeblair | nibalizer: i mean, i trust the infra-cloud-east admins _almost_ as much as i trust the west ones ;) | 23:51 |
| clarkb | I am taking the lock on the passwords file | 23:52 |
| clarkb | I will be adding osic details | 23:52 |
| nibalizer | kk | 23:52 |
| clarkb | nibalizer: you odn't already have it do you? | 23:52 |
| fungi | i trust the infra-cloud-east admins more than i trust myself. though that's probably not saying a lot | 23:52 |
| nibalizer | clarkb: no i relased it hours ago | 23:52 |
| clarkb | and if I remember my emacs its ^C^S ^C^X when I am done | 23:52 |
| nibalizer | clarkb: i use vim... | 23:53 |
| * krotscheck_dcm reads backscroll. | 23:53 | |
| jeblair | also if you use emacs, there's an actual lock. | 23:53 |
| fungi | there sort of is with vim too | 23:53 |
| clarkb | jeblair: oh thats neat more raisins to use emacs for this particular task | 23:53 |
| nibalizer | oh the passwords file | 23:53 |
| nibalizer | oh thats differen than the hiera | 23:53 |
| jeblair | nibalizer: okay, i am convinced you understand my position. if you still feel strongly, i will go with your suggestion. | 23:54 |
| krotscheck_dcm | oookaaaay, so, remote beaches? | 23:54 |
| nibalizer | jeblair: I like the idea of letting fungi break our tie :) | 23:55 |
| nibalizer | sounds like we can all amuse ourselves with getting osci up today anyways | 23:55 |
| jeblair | nibalizer: do you want to land the omfra patches first or osci? | 23:55 |
| jeblair | they will conflict so we should decide ordering | 23:55 |
| krotscheck_dcm | jeblair: Looks like the job passed. are we waiting on vos? | 23:56 |
| nibalizer | lets do osci then I can rebase the omfra patches | 23:56 |
| nibalizer | clarkb: are those under a topic? | 23:56 |
| clarkb | nibalizer: I literally only just got credentials 2 minutes ago | 23:56 |
| jeblair | krotscheck_dcm: sorry, forgot to topic a change -- waiting on https://review.openstack.org/276494 | 23:56 |
| krotscheck_dcm | Oh, wheel-release is still broken | 23:56 |
| clarkb | ^X^S not ^C^S | 23:56 |
| krotscheck_dcm | jeblair: But, like, it's there. It's working. | 23:57 |
| *** Sukhdev has quit IRC | 23:57 | |
| fungi | yeah, i figured osci second since infra-cloud was already in progress | 23:57 |
| krotscheck_dcm | Hrm. The index is still not there. | 23:57 |
| fungi | but if you're worried infra-cloud will still take longer then we can go the other way | 23:57 |
| *** jpr has quit IRC | 23:58 | |
| clarkb | ok I no longer have the write lock on passwords file | 23:58 |
| *** jsavak has quit IRC | 23:58 | |
| *** Sukhdev has joined #openstack-infra | 23:58 | |
| jeblair | krotscheck_dcm: i'm guessing it's in the RW but not RO volumes | 23:58 |
| krotscheck_dcm | got it | 23:58 |
| jeblair | krotscheck_dcm: oh hrm | 23:58 |
| *** avarner has joined #openstack-infra | 23:59 | |
| jeblair | krotscheck_dcm: you should look at it -- do you have an afs client handy? | 23:59 |
| krotscheck_dcm | not anymore. | 23:59 |
| avarner | Is openstack infra group considered the administrator for etherpad.openstack.org? | 23:59 |
| krotscheck_dcm | jeblair: Is the error pasteable? | 23:59 |
| jeblair | krotscheck_dcm: this is what's in RW: http://paste.openstack.org/show/486032/ | 23:59 |
| krotscheck_dcm | Oh. | 23:59 |
| openstackgerrit | Sahdev Zala proposed openstack/requirements: Update to recent release of tosca-parser 0.4.0 https://review.openstack.org/276511 | 23:59 |
| krotscheck_dcm | Well. | 23:59 |
| krotscheck_dcm | That's. | 23:59 |
| krotscheck_dcm | concise | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!