| mordred | which is where these get built | 00:00 |
|---|---|---|
| marun | clarkb: I'm afraid the refactored testr artifact processing has uncovered an assumption that testr and subunit are installed system-wide. | 00:00 |
| clarkb | mordred: no it was exploding locally too | 00:00 |
| mordred | clarkb: yeah? | 00:00 |
| clarkb | mordred: so this is some progress if not all the progress | 00:00 |
| mordred | clarkb, SpamapS woot! | 00:00 |
| clarkb | mordred: the mysql on nodepool will be an issue too | 00:00 |
| SpamapS | the answer there is to -hlocalhost isn't it? | 00:00 |
| marun | clarkb: since the neutron functional job only installs in a venv, this assumption is violated and the test artifacts aren't saved. | 00:00 |
| SpamapS | just make sure the socket is used. | 00:00 |
| clarkb | mordred: SpamapS the thing to check is boot off that image and see that the openstack_citest or whatever the user name is user is on the server | 00:00 |
| mordred | SpamapS: probably | 00:00 |
| clarkb | SpamapS: mordred if you can convince puppet-mysql to do that | 00:01 |
| clarkb | marun: doesn't it know to run out of a vnev? | 00:01 |
| *** bswartz has joined #openstack-infra | 00:01 | |
| marun | clarkb: devstack-gate doesn't, but I could fix that | 00:01 |
| *** bknudson has joined #openstack-infra | 00:01 | |
| SpamapS | /dev/loop0p1 9.0G 4.8G 3.8G 56% /tmp/image.mHQiUpcS/mnt | 00:02 |
| SpamapS | rather large image | 00:02 |
| marun | clarkb: there is currently no concept of 'venv' in d-g, though. | 00:02 |
| mordred | SpamapS: yes | 00:02 |
| mordred | it's a very large image | 00:03 |
| mordred | SpamapS: it contains a cache of everyhting you might ever need in any gate job | 00:03 |
| marun | clarkb: Can you think of another way to fix that than adding YACV (yet another configuration variable) to point to it? | 00:03 |
| *** melwitt has quit IRC | 00:03 | |
| SpamapS | solid HDD light for the last 4 minutes | 00:03 |
| SpamapS | mordred: cacheallthethings.qcow2 | 00:03 |
| clarkb | marun: I am trying to figure out how it worked for tempest first | 00:03 |
| clarkb | marun: my guess is do something like what tempest did | 00:03 |
| *** dimssum__ has joined #openstack-infra | 00:04 | |
| *** dimssum__ is now known as dimsum__ | 00:04 | |
| marun | clarkb: tempest relies on testr and subunit being installed system-wide | 00:04 |
| clarkb | oh I see | 00:04 |
| marun | https://github.com/openstack-infra/devstack-gate/blob/master/functions.sh#L536 | 00:04 |
| *** dimsum__ has quit IRC | 00:04 | |
| marun | https://github.com/openstack-infra/devstack-gate/blob/master/functions.sh#L532 | 00:04 |
| openstackgerrit | Merged openstack-infra/zuul: Check liveness of changes before removing https://review.openstack.org/153813 | 00:05 |
| marun | clarkb: so, would you prefer that d-g can be informed of which venv to use? | 00:05 |
| clarkb | marun: and devstack must install them for us when tempest is used but not during your tests? | 00:05 |
| marun | clarkb: or just install testr/subunit automatically? | 00:05 |
| marun | clarkb: correct | 00:05 |
| clarkb | marun: no I think likely the best thing to do is just install testr globally since that is what tempest expects | 00:05 |
| marun | clarkb: ok, I'll submit a patch shortly. | 00:05 |
| marun | clarkb: danke | 00:05 |
| *** lnxnut_ has quit IRC | 00:06 | |
| *** gyee has quit IRC | 00:06 | |
| *** markvoelker has quit IRC | 00:07 | |
| mordred | clarkb: what was the thing you wanted me to review? | 00:07 |
| *** Sukhdev has joined #openstack-infra | 00:07 | |
| clarkb | mordred: 154188 its the "simple" fix for spec_helper | 00:07 |
| mordred | 154188? | 00:07 |
| mordred | yah | 00:07 |
| mordred | looking | 00:07 |
| *** chlong has joined #openstack-infra | 00:07 | |
| *** nelsnelson has joined #openstack-infra | 00:08 | |
| *** lnxnut_ has joined #openstack-infra | 00:08 | |
| mordred | clarkb: +# | 00:08 |
| mordred | clarkb: +3 | 00:08 |
| clarkb | mordred: thanks | 00:08 |
| * mordred is at his favorite part of the process now- uploading the image | 00:09 | |
| *** crc32 has quit IRC | 00:09 | |
| dougwig | hi infra. stackforge/octavia was rolling along until last week, when things won't merge anymore. is there something wrong with the project config for that? example review: https://review.openstack.org/#/c/149789/ | 00:10 |
| openstackgerrit | Richard Jones proposed openstack-infra/infra-specs: Add initial draft of bower spec https://review.openstack.org/154297 | 00:11 |
| r1chardj0n3s | mordred: ^^ | 00:12 |
| mordred | oh my | 00:12 |
| r1chardj0n3s | ;) | 00:12 |
| *** lnxnut_ has quit IRC | 00:12 | |
| *** r-daneel has quit IRC | 00:12 | |
| jeblair | dougwig: 149789 depends on 151496 which is WIP | 00:13 |
| dougwig | d'oh, i totally missed that. thanks. | 00:13 |
| * dougwig gets more coffee. | 00:13 | |
| jeblair | np. just call me 'zuul' | 00:14 |
| *** dims__ has joined #openstack-infra | 00:14 | |
| openstackgerrit | Merged openstack-infra/project-config: Fix rspec installs during image builds https://review.openstack.org/154188 | 00:14 |
| *** gyee has joined #openstack-infra | 00:14 | |
| *** carl_baldwin has quit IRC | 00:15 | |
| mordred | O M G could someone at rackspace please update the SSL certs??? the python warning is driving me batty | 00:15 |
| mordred | jroll: you fixed the config-drive thing - next task - fix all the SSL certs | 00:16 |
| * mordred hands jroll a fluffy sea otter | 00:16 | |
| openstackgerrit | Maru Newby proposed openstack-infra/devstack-gate: Install testr system-wide for artifact processing https://review.openstack.org/154299 | 00:18 |
| marun | clarkb: ^^ | 00:18 |
| jhesketh | mordred: re r1chardj0n3s's bower.js stuff, I've been talking to him about it and in my opinion AFS for a shared cache is overkill. I think we should start with the simpler implementation (as outlined in the spec) and go from there | 00:18 |
| jhesketh | mordred: if the failure rate at that point turns out to be worse than we'd like, we can look at building more complicated solutions | 00:18 |
| *** salv-orlando has quit IRC | 00:19 | |
| SpamapS | mordred: working now? | 00:20 |
| mordred | jhesketh: I do like the general idea of starting with things that are smaller | 00:20 |
| *** salv-orlando has joined #openstack-infra | 00:20 | |
| mordred | SpamapS: dunno - still polling on glance task-show | 00:20 |
| openstackgerrit | Clark Boylan proposed openstack-infra/puppet-project_config: Provide way to specify repo ref https://review.openstack.org/154300 | 00:21 |
| mordred | oh - clarkb is doing it | 00:22 |
| jeblair | r1chardj0n3s: thanks! i've left you some feedback | 00:22 |
| clarkb | mordred: ya I figured I would | 00:22 |
| mordred | clarkb: woot | 00:22 |
| clarkb | mordred: working on the second half now | 00:22 |
| jeblair | clarkb: aha! i see what you are doing! what problems is that solving? | 00:23 |
| mordred | jeblair: race condition | 00:23 |
| jeblair | mordred: that much i got | 00:24 |
| mordred | jeblair: because the git pull is happening on each host | 00:24 |
| jeblair | mordred: what is the race breaking? | 00:24 |
| clarkb | jeblair: earlier today I merged a change to add a project | 00:24 |
| jeblair | clarkb: is this causing the empty git.o.o repos? | 00:24 |
| clarkb | jeblair: puppet ran on gitXX with project_config prior to that merge and review.o.o ran after | 00:24 |
| mordred | yah | 00:24 |
| clarkb | jeblair: yes | 00:24 |
| *** salv-orlando has quit IRC | 00:24 | |
| jeblair | awesome! | 00:24 |
| clarkb | so next step is to have ansible check what HEAD is at the beginning of puppet_run_all then feed that specific ref to all the puppet runs | 00:24 |
| mordred | so we really want the orchestration end to be the one that knows "hey, all y'all - how bout you pull ref X" | 00:24 |
| jeblair | teamwork! | 00:25 |
| mordred | SpamapS: I'm booting a server ... let's see what breaks this time! | 00:25 |
| jroll | mordred: I've tried, I promise :x | 00:26 |
| jogo | clarkb: \o/ thanks | 00:27 |
| r1chardj0n3s | cheers jeblair! | 00:27 |
| * r1chardj0n3s AFKs for a bit now | 00:27 | |
| * mordred hands jroll a box of fluffy sea otters | 00:27 | |
| *** salv-orlando has joined #openstack-infra | 00:27 | |
| jhesketh | jeblair: I've responded to your cache query on r1chardj0n3s's spec. Richard will need to check what I've said is true, but I think it's in line with his thinking | 00:27 |
| *** r1chardj0n3s is now known as r1chardj0n3s_afk | 00:27 | |
| clarkb | mordred: except I have to figure out how to pass an env var from env calling ansible into the env ansible runs commands in | 00:27 |
| mordred | clarkb: it's possible we want to extend our puppet module | 00:28 |
| mordred | hangon | 00:28 |
| mordred | way too much overload | 00:28 |
| openstackgerrit | Merged openstack-infra/zuul: Avoid leaking dynamic change queues https://review.openstack.org/153814 | 00:28 |
| *** sabeen has quit IRC | 00:29 | |
| mordred | clarkb: it's possible we might want to extend our ansible module that calls puppet with a facility to pass in arbitrary key-value things to pass to the puppet invocation | 00:29 |
| *** tmcpeak has joined #openstack-infra | 00:30 | |
| *** tmcpeak has quit IRC | 00:32 | |
| jeblair | jhesketh, r1chardj0n3s_afk: thanks. i've responded to your responses. :) | 00:32 |
| clarkb | mordred: `git ls-remote --heads https://git.openstack.org/openstack-infra/project-config master | cut -f 1` is the magic to run to get the ref | 00:33 |
| clarkb | don't need the --heads even | 00:33 |
| *** sarob has quit IRC | 00:35 | |
| *** ashleighfarnham has quit IRC | 00:35 | |
| *** ZZelle has quit IRC | 00:36 | |
| *** signed8bit is now known as signed8bit_ZZZzz | 00:37 | |
| *** dangers is now known as dangers_away | 00:38 | |
| clarkb | mordred: I think I may have it | 00:39 |
| mordred | woohoo! | 00:39 |
| clarkb | mordred: http://docs.ansible.com/faq.html#how-do-i-access-shell-environment-variables | 00:39 |
| jhesketh | jeblair: and more responses to your responses :-) | 00:39 |
| jhesketh | *to my responses | 00:39 |
| mordred | SpamapS, clarkb, jeblair: BOOM! I have networking | 00:41 |
| *** yamamoto has joined #openstack-infra | 00:42 | |
| mordred | ok. so - I have booted ubuntu on rax | 00:42 |
| mordred | I need to verify that centos/systemd work ... | 00:42 |
| mordred | and I need to verify that this all works as expected on HP | 00:42 |
| mordred | but I think we're close to this task being done-ish | 00:42 |
| *** Ng has joined #openstack-infra | 00:42 | |
| openstackgerrit | Maru Newby proposed openstack-infra/zuul: Add support for a skip-if filter on jobs https://review.openstack.org/146762 | 00:43 |
| clarkb | mordred: how are roles and tasks different? | 00:44 |
| mordred | clarkb: a role is a named collection of tasks | 00:44 |
| clarkb | mordred: and if I attach environemnt to a role should I expect that to work properly see http://docs.ansible.com/playbooks_environment.html | 00:44 |
| openstackgerrit | Merged openstack-infra/zuul: Add ignore-dependencies option https://review.openstack.org/154274 | 00:44 |
| marun | jhesketh: I've added back test_model at your request ^^ | 00:45 |
| jhesketh | marun: shiny :-) | 00:45 |
| mordred | clarkb: no - I think you'd still need the task to consume it too | 00:45 |
| fungi | nikhil_k: i've added you to https://launchpad.net/~glance-coresec now and set you as an administrator for it. please adjust the membership on that group to reflect glance core reviewers who should be engaged in confirming/debunking embargoed glance security bugs and vetting the non-public patches attached to them | 00:45 |
| clarkb | mordred: so fwiw this is the clunkiest bit of ansible | 00:46 |
| clarkb | mordred: you have to do far too much plumbing for something as simple as make this env var appear over there happen | 00:46 |
| jeblair | jhesketh: i've responded, but that's about the max length conversation i can have in a review | 00:47 |
| mordred | clarkb: yes, I agree | 00:47 |
| mordred | I think we need to re-convene on how we unclunky it | 00:47 |
| marun | jeblair: Your approval would be much appreciated on https://review.openstack.org/#/c/146762/ No changes other than adding test_model.py back at jhesketh's request. | 00:47 |
| *** Guest46399 has quit IRC | 00:47 | |
| jeblair | jhesketh, r1chardj0n3s_afk: i have two points that i'm very firm on: (1) we can not require that packages be published to github, and (2) we can not require that the gate access github for tests | 00:47 |
| mordred | clarkb: I was mostly waiting to make it betterer until we'd gotten angry at it so that the conversation would be fuller | 00:47 |
| mordred | jhesketh, r1chardj0n3s_afk: I concur with jeblair on those two points | 00:48 |
| marun | jhesketh: thank you! | 00:48 |
| * mordred dinners | 00:48 | |
| jhesketh | jeblair: I thought we rebuilt the nodepool images nightly? | 00:48 |
| jeblair | jhesketh: we _attempt_ to do so nightly | 00:49 |
| jeblair | jhesketh: they often fail for weeks on end | 00:49 |
| nikhil_k | fungi: awesome, thanks! | 00:49 |
| jeblair | jhesketh: because things like github failing, or some puppet thing breaking, or any number of things going wrong | 00:49 |
| r1chardj0n3s_afk | jeblair: bower supports many schemes that no-one uses... | 00:49 |
| jhesketh | jeblair: okay, noted | 00:49 |
| jhesketh | fair enough, you've convinced me :-) | 00:49 |
| jeblair | jhesketh: as i mentioned (and was discussed in scrollback), our current images are more than 170 hours old | 00:49 |
| *** sarob has joined #openstack-infra | 00:49 | |
| * jhesketh fails to keep up with scrollback | 00:50 | |
| r1chardj0n3s_afk | jeblair: we cab look into supporting other git repos, but I'm not sure even the bower tool itself does (it doesn't implement all of its own spec ;) | 00:50 |
| r1chardj0n3s_afk | jeblair: I will investigate though | 00:50 |
| marun | sdague: regarding https://review.openstack.org/#/c/153365/, am I just confused as to the utility of grep/egrep? | 00:51 |
| jeblair | r1chardj0n3s_afk: thanks. if that's the case, that's kind of terrible (the spec seems halfway sensible) | 00:51 |
| fungi | sounds like the spec was wishful thinking. rfc it and they will come | 00:52 |
| openstackgerrit | Clark Boylan proposed openstack-infra/puppet-ansible: Pass PUPPET_CONFIG_REF through to the puppet run https://review.openstack.org/154304 | 00:54 |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config: Set PROJECT_CONFIG_REF for ansible https://review.openstack.org/154305 | 00:56 |
| clarkb | mordred: ^ that is probably all wrong, feel free to update as necessary | 00:56 |
| *** Longgeek has joined #openstack-infra | 00:56 | |
| *** MarkAtwood has quit IRC | 00:57 | |
| *** ildikov has quit IRC | 01:01 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/puppet-ansible: Pass PROJECT_CONFIG_REF through to the puppet run https://review.openstack.org/154304 | 01:04 |
| *** shashankhegde has quit IRC | 01:06 | |
| *** zhiwei has joined #openstack-infra | 01:07 | |
| jogo | clarkb: very excited about aiopcpu patches landing, thanks | 01:07 |
| clarkb | jogo: hopefully they get in ;) | 01:08 |
| *** hdd has quit IRC | 01:12 | |
| *** gokrokve has quit IRC | 01:14 | |
| *** Longgeek has quit IRC | 01:15 | |
| *** ildikov has joined #openstack-infra | 01:15 | |
| *** achanda has quit IRC | 01:16 | |
| *** zz_dimtruck is now known as dimtruck | 01:18 | |
| *** ddieterly has joined #openstack-infra | 01:21 | |
| *** gokrokve has joined #openstack-infra | 01:22 | |
| *** tmcpeak has joined #openstack-infra | 01:25 | |
| *** otter768 has joined #openstack-infra | 01:27 | |
| openstackgerrit | Merged openstack-infra/elastic-recheck: Add query for ironic bug 1418833 https://review.openstack.org/154272 | 01:27 |
| openstack | bug 1418833 in Ironic "check-tempest-dsvm-ironic-agent_ssh fails due to ConfigDriveWriteError" [Critical,Fix committed] https://launchpad.net/bugs/1418833 - Assigned to Jay Faulkner (jason-oldos) | 01:27 |
| *** marun has quit IRC | 01:28 | |
| jhesketh | jeblair: btw, did you see my change for zuul to disable a pipeline after too many consecutive failures? | 01:29 |
| *** woodster_ has quit IRC | 01:30 | |
| *** tiswanso has quit IRC | 01:31 | |
| *** ChuckC_ has quit IRC | 01:32 | |
| *** tiswanso has joined #openstack-infra | 01:32 | |
| *** ChuckC_ has joined #openstack-infra | 01:32 | |
| *** ChuckC_ is now known as ChuckC | 01:32 | |
| *** signed8bit_ZZZzz is now known as signed8bit | 01:33 | |
| *** otter768 has quit IRC | 01:33 | |
| *** ChuckC has quit IRC | 01:35 | |
| *** ChuckC has joined #openstack-infra | 01:36 | |
| *** hdd has joined #openstack-infra | 01:38 | |
| *** shashankhegde has joined #openstack-infra | 01:38 | |
| *** Sukhdev has quit IRC | 01:39 | |
| *** ddieterl_ has joined #openstack-infra | 01:43 | |
| *** ddieterly has quit IRC | 01:43 | |
| openstackgerrit | Darragh Bailey proposed stackforge/python-jenkins: Provider helper classes for plugin data management https://review.openstack.org/154316 | 01:47 |
| dstufft | Does Openstack rely on apt-get installed python at all in debstack or the CI or anything? | 01:50 |
| sdake | how many physical machines compose the jenkins gate? | 01:50 |
| jeblair | jhesketh: i think i saw it briefly but haven't reviewed it yet. seems useful even if we don't end up wanting to use it upstream | 01:52 |
| jeblair | sdake: around 1000 virtual machines on public cloud providers. no idea how much that translates to physically. | 01:53 |
| jhesketh | jeblair: cool. It's mostly targeted at 3rd parties. Even if upstream infra (ie 1st parties) breaks terribly I suspect we want to keep voting | 01:53 |
| sdake | jeblair cool thanks | 01:53 |
| clarkb | dstufft yes all python installations are from system packages | 01:53 |
| jeblair | sdake: (plus some physical hardware in use by tripleo, but that probably wasn't your question) | 01:53 |
| clarkb | dstufft we dont custom python | 01:53 |
| sdake | say I want to run docker in the gate | 01:54 |
| dstufft | clarkb: er, I meant python-pip sorry | 01:54 |
| sdake | would that be possible? | 01:54 |
| *** gokrokve has quit IRC | 01:54 | |
| jeblair | sdake: are you asking in reference to a particular project? | 01:54 |
| clarkb | yes its already done by the ipa image builds | 01:54 |
| sdake | what I'd like to do is run docker with some specific arguments (could be in a shell script) and then run tempest after | 01:54 |
| clarkb | dstufft no thats all get pip now iirc | 01:54 |
| sdake | sweet got a pointer to how they did it? | 01:54 |
| sdake | (github link?) | 01:54 |
| sdake | jeblair for kolla | 01:55 |
| dstufft | clarkb: ok, then you won't be affected by the patch that Ubuntu landed into vivid ~5 hours ago | 01:55 |
| sdake | I want to add functional pre-commit testing | 01:55 |
| *** ChuckC_ has joined #openstack-infra | 01:55 | |
| jhesketh | jeblair: however my initial implementation was to simply set a cut-off on the window. That is, once the window falls below X we stop voting. This solution turns out not to work because of the way independant pipelines have a change queue for each project change. What do you think about just counting failures at a pipeline level? Or possibly a more complicated solution of having a pipeline wide window | 01:55 |
| jeblair | sdake: basically you have root on those machines so you can do whatever in the tests :) | 01:56 |
| sdake | sweet | 01:56 |
| sdake | what starts the test? some specific command? | 01:56 |
| *** bhuvan has quit IRC | 01:56 | |
| sdake | liek I know there is a job pep8 which runs testr pep8 iirc | 01:57 |
| clarkb | sdake https://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/ironic-python-agent-jobs.yaml | 01:57 |
| sdake | but how could I get a specific shell script ran? | 01:57 |
| sdake | jeblair line 10 is from the python-agent repo? | 01:58 |
| *** ChuckC has quit IRC | 01:58 | |
| jeblair | sdake: yep | 01:58 |
| sdake | and that returns 0 if the test cases pass? | 01:58 |
| jeblair | sdake: yep | 01:59 |
| *** dmorita has quit IRC | 01:59 | |
| sdake | am I permitted to create jobs for non-integrated projects (i.e. stackforge project) | 01:59 |
| jeblair | sdake: sure thing. stick 'em in their own file (kolla.yaml) | 01:59 |
| jeblair | jhesketh: oops, our changes are colliding (since i just added that for crd). so we could do those, or instead of dynamically creating them, we could statically create one for each project and then work out how to get the correct one for the 'live' change that is being enqueued | 02:00 |
| sdake | jeblair I hate to sound like a noob, but I dont totally understand the structure of that file - do you have a link to appropraite documentation? | 02:00 |
| jeblair | jhesketh: but that still may not be what you want because then each project would have to independently hit the min | 02:01 |
| jhesketh | yeah I saw something fly by for crd, but it was already merged so I haven't read the code yet | 02:01 |
| jeblair | jhesketh: so probably something on the pipeline is what you want. | 02:01 |
| jhesketh | certainly I think on the pipeline is easier to do | 02:02 |
| jeblair | jhesketh: i did write it in december :) | 02:02 |
| jhesketh | oh, I must have missed one of the changes to review then | 02:02 |
| sdake | jeblair is there a fedora 21 node tyep? | 02:02 |
| *** ChuckC has joined #openstack-infra | 02:02 | |
| * jhesketh tries to stay on top of zuul changes | 02:02 | |
| *** tmcpeak has quit IRC | 02:02 | |
| *** rwsu is now known as rwsu-afk | 02:02 | |
| jeblair | sdake: http://docs.openstack.org/infra/manual/drivers.html#running-jobs-with-zuul | 02:02 |
| jhesketh | jeblair: to rephrase the question, can you see an easy way, or a useful case, for having a pipeline window? | 02:03 |
| *** nelsnelson has quit IRC | 02:04 | |
| *** ChuckC_ has quit IRC | 02:04 | |
| *** nelsnelson has joined #openstack-infra | 02:04 | |
| jeblair | jhesketh: i feel like the main reason to do so is for this, and given that, it's worth considering an alternate implementation. | 02:04 |
| jhesketh | I guess I was asking if there was another reason to do it. But otherwise I agree | 02:05 |
| *** shashankhegde has quit IRC | 02:05 | |
| jeblair | sdake: i believe there is. note that fedora nodes come and go faster than other node types because of their support period. you may need to keep your scripts up to date as that changes. | 02:05 |
| jhesketh | jeblair: thanks, I should be able to roll something this afternoon :-) | 02:06 |
| *** stevemar has quit IRC | 02:06 | |
| sdake | I see- reason I ask is I need a newer version of docker then is in 14.04 | 02:06 |
| jeblair | sdake: (i'm not sure what the overlap or notification is going to look like, but in general, i don't expect it to be long) | 02:06 |
| *** stevemar has joined #openstack-infra | 02:06 | |
| sdake | and I don't know how to package for ubuntu | 02:06 |
| jeblair | sdake: it's fine to use, just making sure you understand that they are a bit more 'best-effort' as far as the support period goes. | 02:07 |
| sdake | understood | 02:07 |
| sdake | I'll try ubuntu then if I can figure out how to package a new version of docker | 02:07 |
| sdake | I assume we can install stuff from the network with the gate? | 02:07 |
| jeblair | sdake: for a stackforge project, yeah, you should be able to add a ppa and install that way. | 02:08 |
| jeblair | (we wouldn't do that for an openstack project for policy reasons) | 02:08 |
| sdake | ya, this is more proof of concept prototype work, Im sure the new version of ubuntu will have the proper version of docker | 02:09 |
| sdake | by the time we are done, ubuntu should have the proper version | 02:09 |
| clarkb | sdake we stick to ubuntu lts for the most part fwiw | 02:10 |
| clarkb | ubuntu intermediate releases are only supported for 9 months | 02:10 |
| sdake | I can work with that, I just need to find an ubuntu packager in the short term :) | 02:10 |
| *** Longgeek has joined #openstack-infra | 02:12 | |
| sdake | jeblair is it possible for a test case to consume multiple machines? | 02:13 |
| sdake | vms rather | 02:13 |
| *** salv-orlando has quit IRC | 02:15 | |
| jeblair | sdake: yes, but it's pretty difficult and we're still just getting started with openstack. i'd recommend against it for now. | 02:18 |
| *** spzala has quit IRC | 02:18 | |
| sdake | bummer so single node only | 02:18 |
| jeblair | you can do a lot with a node :) | 02:18 |
| *** masayukig has quit IRC | 02:19 | |
| sdake | I want to test HA which requires 3 nodes | 02:19 |
| *** masayukig has joined #openstack-infra | 02:19 | |
| sdake | specifically i have a rabbitmq container | 02:19 |
| sdake | and I want to run it on 3 nodes and run test cases against it | 02:19 |
| jeblair | sdake: can you run 3x on one node? then shoot one of them to test ha? | 02:20 |
| sdake | I'm asking around atm about that exact question | 02:20 |
| *** nelsnelson has quit IRC | 02:22 | |
| *** woodster_ has joined #openstack-infra | 02:23 | |
| *** dims__ has quit IRC | 02:23 | |
| sdake | still asking :) | 02:23 |
| mordred | clarkb: I'll take a look at those when I wake up | 02:26 |
| *** lnxnut has joined #openstack-infra | 02:26 | |
| *** gokrokve has joined #openstack-infra | 02:27 | |
| *** sdake has left #openstack-infra | 02:32 | |
| *** sdake has joined #openstack-infra | 02:32 | |
| *** bhunter71 has quit IRC | 02:35 | |
| *** Longgeek has quit IRC | 02:37 | |
| sdake | jeblair last Q then I'll leave you alone :) is it possible to do nested virt - in other words could I start 3 vms from within 1 vm to run the test case? | 02:38 |
| *** gyee has quit IRC | 02:38 | |
| *** rfolco has joined #openstack-infra | 02:39 | |
| *** lnxnut has quit IRC | 02:39 | |
| jeblair | sdake: possible, but it's so slow as to be unusable (we tried with devstack) | 02:39 |
| *** lnxnut has joined #openstack-infra | 02:40 | |
| sdake | container launch for me is much tidier then devstack | 02:40 |
| jeblair | sdake: so the nested virt is suitable for running something like cirros (which is what the devstack jobs do), but not real workloads nested | 02:40 |
| sdake | devstack takes 20 minutes on my high end hardwware | 02:40 |
| sdake | so I couldn't do a docker pull docker run in a nested virt then | 02:40 |
| mordred | our cloud providers do not support accelerated passthrough | 02:41 |
| mordred | so nested virt is legit nested | 02:41 |
| mordred | and not usuable speed-wise | 02:41 |
| *** Liu_xinguo has joined #openstack-infra | 02:44 | |
| *** rackertom has joined #openstack-infra | 02:45 | |
| *** markvoelker has joined #openstack-infra | 02:47 | |
| *** gokrokve_ has joined #openstack-infra | 02:49 | |
| *** yamahata has quit IRC | 02:49 | |
| *** unicell has quit IRC | 02:50 | |
| *** gokrokve_ has quit IRC | 02:50 | |
| *** wuhg has joined #openstack-infra | 02:50 | |
| *** gokrokve_ has joined #openstack-infra | 02:51 | |
| *** cinerama has quit IRC | 02:51 | |
| *** gokrokve has quit IRC | 02:52 | |
| *** tqtran has quit IRC | 02:54 | |
| openstackgerrit | Merged openstack-infra/devstack-gate: Separate public and flat interfaces for multinode https://review.openstack.org/153093 | 02:54 |
| *** cinerama has joined #openstack-infra | 02:54 | |
| openstackgerrit | Merged openstack-infra/devstack-gate: Diagram the nova net multinode network setup https://review.openstack.org/153672 | 02:54 |
| openstackgerrit | Merged openstack-infra/devstack-gate: Enable nova multi host on aiopcpu tests https://review.openstack.org/131604 | 02:55 |
| openstackgerrit | Merged openstack-infra/devstack-gate: Enable nova-api-metadata for multi host nova-network https://review.openstack.org/136179 | 02:55 |
| sdake | thanks jeblair I'm off | 02:55 |
| sdake | you rock as usual :) | 02:55 |
| sdake | mordred cool thanks, I guess I'll nix that idea of nested virt then | 02:56 |
| *** otter768 has joined #openstack-infra | 03:01 | |
| *** sdake has quit IRC | 03:03 | |
| *** r1chardj0n3s_afk is now known as r1chardj0n3s | 03:03 | |
| *** dims__ has joined #openstack-infra | 03:03 | |
| *** dims__ has quit IRC | 03:07 | |
| *** dims__ has joined #openstack-infra | 03:07 | |
| *** dimtruck is now known as zz_dimtruck | 03:07 | |
| *** ddieterl_ has quit IRC | 03:08 | |
| *** ddieterly has joined #openstack-infra | 03:09 | |
| *** ddieterly has quit IRC | 03:09 | |
| mordred | sdague: it would be cool if it worked ... | 03:09 |
| *** ddieterly has joined #openstack-infra | 03:10 | |
| *** jerryz has joined #openstack-infra | 03:10 | |
| *** tqtran has joined #openstack-infra | 03:11 | |
| clarkb | so if using containeds anyway why not test it that way | 03:12 |
| clarkb | should even be testable running 3 daemons without any virt | 03:12 |
| clarkb | also protip rabbitmq ha recovery is bonghits | 03:12 |
| clarkb | any messages on the losing partition go poof into the ether | 03:13 |
| *** jyuso has quit IRC | 03:14 | |
| *** ddieterly has quit IRC | 03:14 | |
| mordred | clarkb: from what I can tell, rabbitmq ha is not ha by any definitions I've ever had of that term | 03:15 |
| clarkb | yes pretty much | 03:15 |
| *** jyuso1 has joined #openstack-infra | 03:15 | |
| clarkb | it just detects partitions and one side loses badly | 03:15 |
| mordred | well, that's better than not detecting partitions | 03:15 |
| openstackgerrit | Steve Baker proposed openstack-infra/project-config: check-heat-dsvm-functional-mysql raise timeout https://review.openstack.org/154325 | 03:15 |
| jhesketh | clarkb: so... next steps for swift logs. Should we switch over some more jobs? | 03:15 |
| * mordred loves swift logs | 03:16 | |
| clarkb | jhesketh maybe? also possibly concurrent uploads | 03:16 |
| *** salv-orlando has joined #openstack-infra | 03:16 | |
| jhesketh | clarkb: I was thinking about that.. we are already uploading 10 files at a time as it turns out | 03:16 |
| clarkb | but I am pretty confident in it actually working at this point that more jobs sounds good | 03:16 |
| jhesketh | due to the way the formpost works | 03:16 |
| clarkb | oh thats probably good enough | 03:16 |
| jhesketh | maybe we need another way of looking into the high processing times | 03:17 |
| jhesketh | eg making sure it's actually a bandwidth issue and not something else? | 03:17 |
| *** lnxnut has quit IRC | 03:17 | |
| clarkb | possibly | 03:17 |
| *** salv-orlando has quit IRC | 03:21 | |
| *** boris-42 has quit IRC | 03:22 | |
| jhesketh | clarkb: maybe the upload script should print out what it is doing so we can get the timestamps in the jenkins logs (we'll never see them in the actual swift logs as the uploading all happens after the console is fetched) | 03:22 |
| *** spzala has joined #openstack-infra | 03:22 | |
| clarkb | jhesketh that seems reasonable | 03:22 |
| *** baoli has quit IRC | 03:23 | |
| jhesketh | okay, gimme a sec :-) | 03:23 |
| *** gokrokve_ has quit IRC | 03:25 | |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/project-config: Add extra information to zuul swift uploading https://review.openstack.org/154326 | 03:28 |
| jhesketh | clarkb: ^ pretty crude but should give us an idea of what's slow | 03:29 |
| *** david-lyle is now known as david-lyle_afk | 03:29 | |
| *** lnxnut has joined #openstack-infra | 03:29 | |
| *** baoli has joined #openstack-infra | 03:29 | |
| *** BharatK has joined #openstack-infra | 03:29 | |
| *** nelsnelson has joined #openstack-infra | 03:30 | |
| *** dmsimard_away is now known as dmsimard | 03:30 | |
| *** dmsimard is now known as dmsimard_away | 03:32 | |
| *** ayoung is now known as ayoung_sleep | 03:34 | |
| *** amrith is now known as _amrith_ | 03:34 | |
| *** nelsnelson has quit IRC | 03:35 | |
| *** salv-orlando has joined #openstack-infra | 03:35 | |
| *** ashleighfarnham has joined #openstack-infra | 03:36 | |
| *** Longgeek has joined #openstack-infra | 03:37 | |
| *** salv-orlando has quit IRC | 03:37 | |
| *** ashleighfarnham has quit IRC | 03:40 | |
| *** gokrokve has joined #openstack-infra | 03:42 | |
| *** salv-orlando has joined #openstack-infra | 03:44 | |
| *** unicell has joined #openstack-infra | 03:44 | |
| *** _amrith_ is now known as amrith | 03:45 | |
| *** achanda has joined #openstack-infra | 03:45 | |
| *** Ryan_Lane has quit IRC | 03:46 | |
| *** koolhead17 has joined #openstack-infra | 03:46 | |
| *** dims_ has joined #openstack-infra | 03:47 | |
| *** dims__ has quit IRC | 03:49 | |
| *** hdd has quit IRC | 03:50 | |
| greghaynes | clarkb: thers settings for what rmq does when partitions happen | 03:51 |
| greghaynes | clarkb: really where you lose data is when things rejoin the cluster | 03:51 |
| greghaynes | whoever is the joiner just drops all its data on the floor, which may or may not be an issue | 03:52 |
| clarkb | ya there are manual options but iirc even they drop data on the floor | 03:52 |
| greghaynes | yep. You can set it to require quorum to accept a message, which is kind of the best you can get in terms of HA | 03:52 |
| greghaynes | its not totally reliable even so though | 03:53 |
| greghaynes | clarkb: We had a lot of input on the tripleo rmq ha configs btw, if youre looking for reference | 03:54 |
| clarkb | greghaynes: nah, I was just commenting on sdake's interest in testing rabbitmq ha | 03:54 |
| clarkb | the big thing being one node should be sufficient you just need 3 daemons | 03:55 |
| greghaynes | ah yep | 03:55 |
| *** tqtran has quit IRC | 03:55 | |
| *** carl_baldwin has joined #openstack-infra | 03:58 | |
| *** tmcpeak has joined #openstack-infra | 03:58 | |
| greghaynes | running a build using the openstack-repos element really makes me want to parellelize source-repositories... | 03:58 |
| mordred | greghaynes: I think we'd like you if you did | 03:59 |
| *** salv-orlando has quit IRC | 04:00 | |
| *** yamahata has joined #openstack-infra | 04:00 | |
| *** tmcpeak has quit IRC | 04:00 | |
| clarkb | greghaynes: its not that bad, though you have liekly never built a snapshot image so don't have that to compare against | 04:00 |
| *** vigneshvar has joined #openstack-infra | 04:00 | |
| greghaynes | clarkb: yea, im spoiled | 04:00 |
| greghaynes | i was building images in a couple mins | 04:00 |
| *** sdake has joined #openstack-infra | 04:02 | |
| clarkb | greghaynes: for me the longest step with a relatively up to date cache is the file copy | 04:04 |
| clarkb | greghaynes: it is why I hate the file copy and wish dib would mount the image and chroot on top of it | 04:04 |
| openstackgerrit | Merged openstack-infra/nodepool: Fix nodepool --version https://review.openstack.org/145930 | 04:04 |
| clarkb | (I understand the tmpfs case but you can switch based on tmpfs I just haven't had the time to actually change dib to do that yet) | 04:05 |
| greghaynes | I had big plans at one point to use lvm and be able to lvm snapshot a base image and chroot into it | 04:05 |
| clarkb | greghaynes: we spend all this time extracting a qcow2 image into a tarball, exploding the tarball, chrooting into that, mounting a filesystem, then copying everything from the chroot into that | 04:05 |
| *** salv-orlando has joined #openstack-infra | 04:05 | |
| clarkb | greghaynes: when all we need to do is take the source qcow2, mount a copy, and write into it, done | 04:06 |
| clarkb | greghaynes: also if we did that precise could build centos7 images | 04:06 |
| *** achanda has quit IRC | 04:07 | |
| greghaynes | I thought you either had to nbd or convert to !qcow2 to edit it? | 04:07 |
| *** mriedem has quit IRC | 04:07 | |
| clarkb | maybe? oh especially since you want raw to convert to other outputs | 04:07 |
| clarkb | but the copying of files at the end should be avoidable at lesat | 04:08 |
| greghaynes | yea, thats the other problem, but you could in theory optimize the same in and out format case | 04:08 |
| greghaynes | sure, and really we should at a minimum be able to make a source cache | 04:08 |
| *** achanda has joined #openstack-infra | 04:08 | |
| clarkb | but ya looks like nbd is required | 04:08 |
| greghaynes | where we dont rexplode the tarball unless the source file isnt in the cache | 04:08 |
| greghaynes | er, dont reconvert | 04:09 |
| clarkb | mordred: did SpamapS say if he had to change anything to make mysql work or if he checked the resulting image? | 04:09 |
| *** ddieterly has joined #openstack-infra | 04:10 | |
| *** baoli has quit IRC | 04:10 | |
| greghaynes | IIRC he ran into the missing versionedobjects repo issue when he was trying out the in_chroot setting | 04:11 |
| clarkb | then I fixed it and I thought he said he got mysql working | 04:11 |
| greghaynes | o | 04:12 |
| *** salv-orlando has quit IRC | 04:12 | |
| clarkb | well "fixed" the proper race killing fix is in progress | 04:12 |
| *** salv-orlando has joined #openstack-infra | 04:14 | |
| *** Longgeek has quit IRC | 04:14 | |
| *** ddieterly has quit IRC | 04:15 | |
| *** salv-orl_ has joined #openstack-infra | 04:18 | |
| *** salv-orlando has quit IRC | 04:19 | |
| *** tiswanso has quit IRC | 04:22 | |
| *** salv-orl_ has quit IRC | 04:23 | |
| *** mtanino has quit IRC | 04:23 | |
| mordred | clarkb: I do not know if he's booted the image | 04:26 |
| *** salv-orlando has joined #openstack-infra | 04:27 | |
| *** spzala has quit IRC | 04:27 | |
| *** dims_ has quit IRC | 04:32 | |
| *** salv-orlando has quit IRC | 04:33 | |
| *** jdg is now known as jgriffith | 04:34 | |
| *** harlowja is now known as harlowja_away | 04:34 | |
| *** jgriffith is now known as Guest3282 | 04:34 | |
| *** vigneshvar has quit IRC | 04:35 | |
| *** Guest3282 is now known as jgriffith | 04:36 | |
| *** dumbhead has left #openstack-infra | 04:43 | |
| *** Sukhdev has joined #openstack-infra | 04:48 | |
| openstackgerrit | greghaynes proposed openstack-infra/project-config: set -x in cache-testrepository install https://review.openstack.org/154337 | 04:49 |
| *** lnxnut has quit IRC | 04:50 | |
| *** lnxnut has joined #openstack-infra | 04:51 | |
| *** coolsvap_ is now known as coolsvap | 04:52 | |
| *** otter768 has quit IRC | 04:53 | |
| *** lnxnut has quit IRC | 04:56 | |
| *** gokrokve_ has joined #openstack-infra | 04:56 | |
| *** gokrokve has quit IRC | 04:59 | |
| *** gokrokve_ has quit IRC | 05:00 | |
| *** unicell1 has joined #openstack-infra | 05:04 | |
| *** unicell has quit IRC | 05:04 | |
| *** wuhg has quit IRC | 05:09 | |
| *** garyh has quit IRC | 05:09 | |
| *** ddieterly has joined #openstack-infra | 05:11 | |
| *** nikil89_ has joined #openstack-infra | 05:15 | |
| *** ddieterly has quit IRC | 05:16 | |
| *** nikil89_ is now known as naggappan | 05:18 | |
| *** ivar-laz_ has joined #openstack-infra | 05:19 | |
| *** achanda has quit IRC | 05:19 | |
| *** salv-orlando has joined #openstack-infra | 05:21 | |
| *** lnxnut has joined #openstack-infra | 05:21 | |
| *** ivar-lazzaro has quit IRC | 05:22 | |
| *** harlowja_at_home has joined #openstack-infra | 05:25 | |
| *** salv-orlando has quit IRC | 05:27 | |
| openstackgerrit | Richard Jones proposed openstack-infra/infra-specs: Enable bower for managing Horizon components https://review.openstack.org/154297 | 05:29 |
| *** abhirc has quit IRC | 05:37 | |
| *** lnxnut has quit IRC | 05:38 | |
| *** daneyon has quit IRC | 05:40 | |
| *** BharatK has quit IRC | 05:41 | |
| *** garyh has joined #openstack-infra | 05:41 | |
| *** unicell has joined #openstack-infra | 05:42 | |
| *** salv-orlando has joined #openstack-infra | 05:42 | |
| *** unicell1 has quit IRC | 05:43 | |
| *** salv-orlando has quit IRC | 05:45 | |
| *** camunoz has quit IRC | 05:45 | |
| *** hdd has joined #openstack-infra | 05:45 | |
| *** carl_baldwin has quit IRC | 05:46 | |
| *** zz_dimtruck is now known as dimtruck | 05:47 | |
| *** ildikov has quit IRC | 05:49 | |
| *** oomichi has quit IRC | 05:50 | |
| *** harlowja_at_home has quit IRC | 05:51 | |
| *** mrmartin has joined #openstack-infra | 05:53 | |
| *** enikanorov_ has joined #openstack-infra | 05:53 | |
| *** salv-orlando has joined #openstack-infra | 05:55 | |
| *** BharatK has joined #openstack-infra | 05:58 | |
| *** salv-orlando has quit IRC | 06:00 | |
| *** harlowja_away has quit IRC | 06:01 | |
| *** jyuso1 has quit IRC | 06:09 | |
| *** jyuso1 has joined #openstack-infra | 06:09 | |
| *** mrmartin has quit IRC | 06:09 | |
| *** Ryan_Lane has joined #openstack-infra | 06:10 | |
| *** ddieterly has joined #openstack-infra | 06:12 | |
| *** ddieterly has quit IRC | 06:17 | |
| *** salv-orlando has joined #openstack-infra | 06:18 | |
| *** achanda has joined #openstack-infra | 06:20 | |
| *** salv-orlando has quit IRC | 06:21 | |
| *** r1chardj0n3s is now known as r1chardj0n3s_afk | 06:22 | |
| *** ivar-laz_ has quit IRC | 06:22 | |
| *** imcsk8 has quit IRC | 06:25 | |
| *** imcsk8 has joined #openstack-infra | 06:25 | |
| *** ashleighfarnham has joined #openstack-infra | 06:26 | |
| *** signed8bit has quit IRC | 06:29 | |
| openstackgerrit | YAMAMOTO Takashi proposed openstack/requirements: Add ryu>=3.18 https://review.openstack.org/154354 | 06:29 |
| *** camunoz has joined #openstack-infra | 06:34 | |
| *** Ryan_Lane has quit IRC | 06:35 | |
| *** unicell has quit IRC | 06:37 | |
| *** unicell has joined #openstack-infra | 06:37 | |
| openstackgerrit | Steve Martinelli proposed openstack/requirements: Add oslo.policy to global requirements https://review.openstack.org/148629 | 06:37 |
| *** hdd has quit IRC | 06:44 | |
| *** ildikov has joined #openstack-infra | 06:44 | |
| *** Sukhdev has quit IRC | 06:46 | |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/infra-specs: Enable bower for managing Horizon components https://review.openstack.org/154297 | 06:46 |
| *** vigneshvar has joined #openstack-infra | 06:46 | |
| *** yamahata has quit IRC | 06:49 | |
| openstackgerrit | Steve Martinelli proposed openstack/requirements: Add oslo.policy to global requirements https://review.openstack.org/148629 | 06:49 |
| *** mrunge has joined #openstack-infra | 06:50 | |
| *** abhi1892 has joined #openstack-infra | 06:50 | |
| *** lnxnut has joined #openstack-infra | 06:51 | |
| *** shashankhegde has joined #openstack-infra | 06:52 | |
| *** otter768 has joined #openstack-infra | 06:54 | |
| *** salv-orlando has joined #openstack-infra | 06:56 | |
| *** lnxnut has quit IRC | 06:56 | |
| *** salv-orlando has quit IRC | 06:58 | |
| *** otter768 has quit IRC | 06:59 | |
| *** abhi1892 has quit IRC | 06:59 | |
| *** camunoz is now known as camunoz_gone | 06:59 | |
| *** achanda has quit IRC | 07:02 | |
| *** Ryan_Lane has joined #openstack-infra | 07:03 | |
| *** Ryan_Lane has quit IRC | 07:04 | |
| *** achanda has joined #openstack-infra | 07:07 | |
| *** scheuran has joined #openstack-infra | 07:08 | |
| *** salv-orlando has joined #openstack-infra | 07:09 | |
| *** ashleighfarnham has quit IRC | 07:10 | |
| *** ddieterly has joined #openstack-infra | 07:13 | |
| *** Longgeek has joined #openstack-infra | 07:15 | |
| *** salv-orlando has quit IRC | 07:16 | |
| *** Longgeek has quit IRC | 07:16 | |
| *** ddieterly has quit IRC | 07:17 | |
| *** sdake has quit IRC | 07:29 | |
| *** doude has joined #openstack-infra | 07:32 | |
| *** avozza is now known as zz_avozza | 07:32 | |
| *** HeOS has quit IRC | 07:32 | |
| *** dims__ has joined #openstack-infra | 07:33 | |
| *** achanda has quit IRC | 07:33 | |
| *** markvoelker has quit IRC | 07:33 | |
| *** markvoelker has joined #openstack-infra | 07:34 | |
| *** armax has quit IRC | 07:36 | |
| *** dims__ has quit IRC | 07:37 | |
| *** markvoelker has quit IRC | 07:38 | |
| *** achanda has joined #openstack-infra | 07:39 | |
| *** jyuso1 has quit IRC | 07:42 | |
| *** yfried_ is now known as yfried|afk | 07:43 | |
| *** jyuso1 has joined #openstack-infra | 07:44 | |
| *** belmoreira has joined #openstack-infra | 07:45 | |
| *** Ryan_Lane has joined #openstack-infra | 07:47 | |
| *** yfried|afk is now known as yfried_ | 07:48 | |
| *** achanda has quit IRC | 07:48 | |
| *** sputnik13 has joined #openstack-infra | 07:50 | |
| sputnik13 | are there instances where tempest failure is an anomaly and just rerunning it makes it succeed? | 07:51 |
| *** unicell has quit IRC | 07:51 | |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/zuul: Add support for disabling bad pipelines https://review.openstack.org/154367 | 07:51 |
| *** lnxnut has joined #openstack-infra | 07:52 | |
| *** unicell has joined #openstack-infra | 07:52 | |
| jhesketh | sputnik13: yes, a lot of these can be categorised by bugs that elastic-recheck is tracking | 07:52 |
| jhesketh | if you get a comment from elastic-recheck it probably means you hit a known race condition | 07:53 |
| sputnik13 | mmm, where would I see such a thing? | 07:53 |
| sputnik13 | https://review.openstack.org/#/c/152357/ | 07:53 |
| sputnik13 | sorry, I'm still relatively new | 07:53 |
| sputnik13 | I can't see anything in the failure log that makes me think it's the code that I checked in as the unit tests for it work, and it's not a terribly complicated piece of code | 07:54 |
| *** chlong has quit IRC | 07:54 | |
| sputnik13 | is there an easy way to force a re-run of the tests, short of submitting a new patch? | 07:55 |
| *** mrunge_ has joined #openstack-infra | 07:56 | |
| *** lnxnut has quit IRC | 07:57 | |
| *** mrunge has quit IRC | 07:57 | |
| *** jcoufal has joined #openstack-infra | 07:58 | |
| jhesketh | sputnik13: elastic-recheck is a bot that will leave a comment on your patch with the first line being 'recheck' | 08:00 |
| jhesketh | anybody leaving a comment with the first line being 'recheck' will actually re-run the tests (to answer your second question) | 08:00 |
| jhesketh | so you'd see the bot in the comments section, but I'm guessing since you haven't it's not a tracked bug | 08:00 |
| jhesketh | a recheck (by leaving a comment yourself) is probably a good next step if you don't think the failure is due to your code | 08:00 |
| jhesketh | sputnik13: somehow that test segfaulted with no seemingly useful info! | 08:02 |
| jhesketh | curious t o see if it happens again | 08:03 |
| sputnik13 | jhesketh: me too... do I have to essentially submit a new patch? | 08:03 |
| *** frobware_ has joined #openstack-infra | 08:03 | |
| *** fifieldt has joined #openstack-infra | 08:03 | |
| jhesketh | sputnik13: nope, just leave a comment with nothing more than the text 'recheck' | 08:04 |
| jhesketh | then you'll see it appear on http://status.openstack.org/zuul/ (you can use the filter to filter by your change number 152357) | 08:04 |
| sputnik13 | oh, neat | 08:04 |
| sputnik13 | thank you | 08:04 |
| *** markvoelker has joined #openstack-infra | 08:04 | |
| jhesketh | no worries :-) | 08:07 |
| *** yamamoto has quit IRC | 08:07 | |
| *** mrmartin has joined #openstack-infra | 08:08 | |
| jhesketh | sputnik13: jobs a re-running. The job that failed previously is coming through here https://jenkins06.openstack.org/job/gate-tempest-dsvm-neutron-src-oslo.utils/29/console | 08:08 |
| sputnik13 | oh that's nice | 08:09 |
| *** markvoelker has quit IRC | 08:09 | |
| *** shashankhegde has quit IRC | 08:11 | |
| *** ddieterly has joined #openstack-infra | 08:13 | |
| *** stevemar has quit IRC | 08:13 | |
| *** Longgeek has joined #openstack-infra | 08:14 | |
| *** luqas has joined #openstack-infra | 08:16 | |
| *** dtantsur|afk is now known as dtantsur | 08:17 | |
| *** ddieterly has quit IRC | 08:18 | |
| ttx | jeblair: +1 | 08:19 |
| *** woodster_ has quit IRC | 08:20 | |
| *** mpaolino has joined #openstack-infra | 08:22 | |
| *** oomichi_ has joined #openstack-infra | 08:23 | |
| *** sc68cal has quit IRC | 08:23 | |
| *** frobware_ has quit IRC | 08:24 | |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/zuul: Reorder settled check in tests https://review.openstack.org/153799 | 08:24 |
| *** zz_avozza is now known as avozza | 08:25 | |
| *** arxcruz has joined #openstack-infra | 08:26 | |
| openstackgerrit | Merged openstack-infra/project-config: Updates Kolla Project Description https://review.openstack.org/154261 | 08:28 |
| jhesketh | jeblair: just been catching up on all your zuul changes that went in over the weekend, nice stuff! | 08:29 |
| jhesketh | jeblair: I think we need to restart zuul on the server to get https://review.openstack.org/#/c/153801/3 to reload the webrpc stuff. At the moment http://zuul.openstack.org/ is showing all grey dots because the json feed doesn't have the new 'live' attribute | 08:30 |
| *** sc68cal has joined #openstack-infra | 08:30 | |
| *** jgallard__ has joined #openstack-infra | 08:31 | |
| *** achuprin_ has quit IRC | 08:32 | |
| openstackgerrit | Merged openstack-infra/zuul: Reorder settled check in tests https://review.openstack.org/153799 | 08:33 |
| *** yamamoto has joined #openstack-infra | 08:37 | |
| *** yamamoto has quit IRC | 08:38 | |
| openstackgerrit | David Pursehouse proposed openstack-infra/jenkins-job-builder: Add support to customize trigger-builds block thresholds https://review.openstack.org/154379 | 08:41 |
| *** e0ne has joined #openstack-infra | 08:43 | |
| *** andreykurilin_ has joined #openstack-infra | 08:43 | |
| *** gokrokve has joined #openstack-infra | 08:43 | |
| *** jlibosva has joined #openstack-infra | 08:44 | |
| *** achuprin_ has joined #openstack-infra | 08:45 | |
| *** Ryan_Lane has quit IRC | 08:45 | |
| *** jistr has joined #openstack-infra | 08:47 | |
| *** MaxV has joined #openstack-infra | 08:48 | |
| *** afazekas has joined #openstack-infra | 08:52 | |
| *** gokrokve has quit IRC | 08:54 | |
| *** gokrokve has joined #openstack-infra | 08:55 | |
| *** otter768 has joined #openstack-infra | 08:55 | |
| *** jcoufal_ has joined #openstack-infra | 08:57 | |
| *** HeOS has joined #openstack-infra | 08:59 | |
| *** jcoufal has quit IRC | 09:00 | |
| *** gokrokve has quit IRC | 09:00 | |
| *** otter768 has quit IRC | 09:00 | |
| *** doude has quit IRC | 09:01 | |
| *** jpich has joined #openstack-infra | 09:02 | |
| *** markus_z has joined #openstack-infra | 09:03 | |
| *** markvoelker has joined #openstack-infra | 09:05 | |
| *** mrunge_ has quit IRC | 09:07 | |
| *** sarob has quit IRC | 09:09 | |
| *** markvoelker has quit IRC | 09:10 | |
| *** Guest82511 has joined #openstack-infra | 09:11 | |
| *** mrunge has joined #openstack-infra | 09:11 | |
| openstackgerrit | Andrey Pavlov proposed openstack/requirements: Add botocore to requirements https://review.openstack.org/154386 | 09:11 |
| *** vigneshvar has quit IRC | 09:13 | |
| *** ddieterly has joined #openstack-infra | 09:14 | |
| *** derekh has joined #openstack-infra | 09:17 | |
| *** hashar has joined #openstack-infra | 09:17 | |
| *** sputnik13 has quit IRC | 09:17 | |
| *** ddieterly has quit IRC | 09:18 | |
| *** vigneshvar has joined #openstack-infra | 09:21 | |
| *** e0ne is now known as e0ne_ | 09:24 | |
| *** jcoufal_ has quit IRC | 09:27 | |
| *** salv-orlando has joined #openstack-infra | 09:28 | |
| *** koolhead17 has quit IRC | 09:32 | |
| *** dizquierdo has joined #openstack-infra | 09:33 | |
| *** e0ne_ has quit IRC | 09:35 | |
| *** salv-orlando has quit IRC | 09:36 | |
| *** salv-orlando has joined #openstack-infra | 09:40 | |
| *** andreykurilin_ has quit IRC | 09:41 | |
| *** hichihara has quit IRC | 09:41 | |
| *** salv-orlando has quit IRC | 09:47 | |
| *** ihrachyshka has joined #openstack-infra | 09:48 | |
| *** e0ne has joined #openstack-infra | 09:52 | |
| ihrachyshka | jogo, hey | 09:56 |
| openstackgerrit | YAMAMOTO Takashi proposed openstack/requirements: README.rst: Fix a typo https://review.openstack.org/154394 | 09:56 |
| *** ssam2 has joined #openstack-infra | 09:57 | |
| *** salv-orlando has joined #openstack-infra | 10:02 | |
| *** salv-orlando has quit IRC | 10:04 | |
| openstackgerrit | Mehdi Abaakouk proposed openstack-infra/project-config: gnocchi: install the git repo https://review.openstack.org/154398 | 10:06 |
| *** markvoelker has joined #openstack-infra | 10:06 | |
| *** bradjones has quit IRC | 10:07 | |
| *** rlandy has joined #openstack-infra | 10:09 | |
| sileht | Hi, any idea why http://pypi.iad.openstack.org/simple/python-qpid-proton/ is not up2date, 0.8 exists in pypi.python.org: https://pypi.python.org/pypi/python-qpid-proton/0.8-0 | 10:09 |
| *** sarob has joined #openstack-infra | 10:10 | |
| *** bradjones has joined #openstack-infra | 10:10 | |
| *** ghostpl has joined #openstack-infra | 10:10 | |
| *** yamahata has joined #openstack-infra | 10:11 | |
| *** markvoelker has quit IRC | 10:11 | |
| *** salv-orlando has joined #openstack-infra | 10:12 | |
| *** BobBall_AWOL is now known as BobBall | 10:12 | |
| *** jcoufal has joined #openstack-infra | 10:15 | |
| *** ddieterly has joined #openstack-infra | 10:15 | |
| *** sarob has quit IRC | 10:16 | |
| *** ddieterly has quit IRC | 10:20 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/storyboard: Fix notifications tests docstrings referencing map_resource method https://review.openstack.org/154402 | 10:21 |
| *** jp_at_hp has joined #openstack-infra | 10:22 | |
| *** salv-orlando has quit IRC | 10:28 | |
| *** boris-42 has joined #openstack-infra | 10:29 | |
| *** pelix has joined #openstack-infra | 10:29 | |
| *** yamahata has quit IRC | 10:31 | |
| *** zhiwei has quit IRC | 10:31 | |
| *** Longgeek has quit IRC | 10:37 | |
| *** Guest82511 has quit IRC | 10:38 | |
| *** ldnunes has joined #openstack-infra | 10:39 | |
| *** cdent has joined #openstack-infra | 10:44 | |
| *** Longgeek has joined #openstack-infra | 10:45 | |
| *** andreaf_ has quit IRC | 10:47 | |
| *** yfried_ is now known as yfried|afk | 10:55 | |
| *** yfried|afk is now known as yfried_ | 10:56 | |
| *** otter768 has joined #openstack-infra | 10:56 | |
| ihrachyshka | chmouel, dtroyer, ianw, sdague, we need to push the following devstack patches to merge to start fixing stable gates: https://review.openstack.org/#/c/154252/ , https://review.openstack.org/#/c/154217/ | 10:58 |
| openstackgerrit | Igor Belikov proposed openstack-infra/jenkins-job-builder: Add 'last-completed' to copyartifact 'which-build' values https://review.openstack.org/154418 | 10:58 |
| *** dprince has joined #openstack-infra | 10:58 | |
| *** koolhead17 has joined #openstack-infra | 10:59 | |
| *** otter768 has quit IRC | 11:01 | |
| *** doude has joined #openstack-infra | 11:05 | |
| *** bradjones has quit IRC | 11:07 | |
| *** markvoelker has joined #openstack-infra | 11:07 | |
| *** salv-orlando has joined #openstack-infra | 11:07 | |
| *** bradjones has joined #openstack-infra | 11:09 | |
| *** salv-orlando has quit IRC | 11:10 | |
| *** sarob has joined #openstack-infra | 11:11 | |
| *** markvoelker has quit IRC | 11:12 | |
| *** ddieterly has joined #openstack-infra | 11:16 | |
| *** sarob has quit IRC | 11:16 | |
| openstackgerrit | Mateusz Matuszkowiak proposed openstack-infra/project-config: Add Fuel Elastic Search/Kibana plugin https://review.openstack.org/154422 | 11:16 |
| *** ddieterly has quit IRC | 11:20 | |
| *** salv-orlando has joined #openstack-infra | 11:20 | |
| *** EmilienM|afk is now known as EmilienM | 11:21 | |
| *** doude has quit IRC | 11:21 | |
| *** yfried_ is now known as yfried|afk | 11:22 | |
| *** yfried|afk is now known as yfried_ | 11:22 | |
| *** ihrachyshka has quit IRC | 11:23 | |
| *** ihrachyshka has joined #openstack-infra | 11:23 | |
| *** dims__ has joined #openstack-infra | 11:23 | |
| *** salv-orlando has quit IRC | 11:24 | |
| *** aysyd has joined #openstack-infra | 11:27 | |
| *** jkt has quit IRC | 11:30 | |
| *** jkt has joined #openstack-infra | 11:30 | |
| *** mrmartin has quit IRC | 11:31 | |
| openstackgerrit | Antoine Musso proposed openstack-infra/zuul: Merger: ensure_cloned() now looks for '.git' https://review.openstack.org/123437 | 11:37 |
| *** jgallard__ has quit IRC | 11:43 | |
| *** pc_m has joined #openstack-infra | 11:44 | |
| openstackgerrit | Antoine Musso proposed openstack-infra/zuul: Merger: ensure_cloned() now looks for '.git' https://review.openstack.org/123437 | 11:45 |
| *** _nadya_ has joined #openstack-infra | 11:45 | |
| *** salv-orlando has joined #openstack-infra | 11:48 | |
| *** hashar has quit IRC | 11:54 | |
| *** claudiub_ has joined #openstack-infra | 11:57 | |
| *** chlong has joined #openstack-infra | 11:59 | |
| *** dimtruck is now known as zz_dimtruck | 12:03 | |
| *** e0ne is now known as e0ne_ | 12:08 | |
| *** _nadya_ has quit IRC | 12:08 | |
| *** markvoelker has joined #openstack-infra | 12:08 | |
| *** avozza is now known as zz_avozza | 12:10 | |
| *** _nadya_ has joined #openstack-infra | 12:11 | |
| *** sarob has joined #openstack-infra | 12:12 | |
| *** markvoelker has quit IRC | 12:13 | |
| *** Longgeek has quit IRC | 12:13 | |
| *** ddieterly has joined #openstack-infra | 12:16 | |
| *** sarob has quit IRC | 12:17 | |
| *** e0ne_ has quit IRC | 12:18 | |
| *** ddieterly has quit IRC | 12:21 | |
| *** miqui has quit IRC | 12:22 | |
| *** dprince has quit IRC | 12:22 | |
| *** mpaolino has quit IRC | 12:24 | |
| *** belmoreira has quit IRC | 12:24 | |
| openstackgerrit | Ilia Meerovich proposed openstack-infra/jenkins-job-builder: BugFix: pollscm trigger docs & unittests https://review.openstack.org/147555 | 12:25 |
| *** changbl has quit IRC | 12:29 | |
| *** dizquierdo has quit IRC | 12:29 | |
| *** tmcpeak has joined #openstack-infra | 12:33 | |
| *** zz_dimtruck is now known as dimtruck | 12:33 | |
| *** yfried_ is now known as yfried|afk | 12:35 | |
| *** yfried|afk is now known as yfried_ | 12:38 | |
| *** oomichi_ has quit IRC | 12:42 | |
| *** dimtruck is now known as zz_dimtruck | 12:43 | |
| *** dprince has joined #openstack-infra | 12:45 | |
| *** koolhead17 has quit IRC | 12:48 | |
| *** ilyashakhat has quit IRC | 12:51 | |
| openstackgerrit | Mehdi Abaakouk proposed openstack-infra/project-config: gnocchi: install the git repo https://review.openstack.org/154398 | 12:51 |
| *** weshay has joined #openstack-infra | 12:52 | |
| *** lnxnut has joined #openstack-infra | 12:53 | |
| openstackgerrit | Mateusz Matuszkowiak proposed openstack-infra/project-config: Add Fuel Elastic Search/Kibana plugin https://review.openstack.org/154422 | 12:53 |
| *** radez_g0n3 is now known as radez | 12:55 | |
| *** denis_makogon has quit IRC | 12:57 | |
| *** otter768 has joined #openstack-infra | 12:57 | |
| *** lnxnut has quit IRC | 12:58 | |
| *** ildikov has quit IRC | 12:59 | |
| *** hashar has joined #openstack-infra | 13:00 | |
| *** otter768 has quit IRC | 13:02 | |
| *** zz_avozza is now known as avozza | 13:05 | |
| *** markvoelker has joined #openstack-infra | 13:05 | |
| *** ilyashakhat has joined #openstack-infra | 13:10 | |
| *** sarob has joined #openstack-infra | 13:14 | |
| *** ildikov has joined #openstack-infra | 13:15 | |
| *** ddieterly has joined #openstack-infra | 13:17 | |
| *** doude has joined #openstack-infra | 13:18 | |
| *** sarob has quit IRC | 13:18 | |
| *** bhunter71 has joined #openstack-infra | 13:19 | |
| *** BharatK has quit IRC | 13:20 | |
| *** skolekonov has joined #openstack-infra | 13:21 | |
| *** ddieterly has quit IRC | 13:22 | |
| *** bswartz has quit IRC | 13:22 | |
| *** _nadya_ has quit IRC | 13:23 | |
| *** Guest82511 has joined #openstack-infra | 13:24 | |
| *** spzala has joined #openstack-infra | 13:24 | |
| *** e0ne has joined #openstack-infra | 13:25 | |
| *** ZZelle has joined #openstack-infra | 13:26 | |
| *** mpaolino has joined #openstack-infra | 13:26 | |
| *** woodster_ has joined #openstack-infra | 13:36 | |
| fungi | sileht: what you _should_ be asking is why https://pypi.python.org/pypi/python-qpid-proton/0.8-0 doesn't have a files section and why the corresponding version is also not present at https://pypi.python.org/simple/python-qpid-proton/ | 13:38 |
| openstackgerrit | Merged openstack-infra/storyboard: Fix exception in stories db api https://review.openstack.org/153968 | 13:38 |
| sileht | fungi, oh ok, I will see with the maintainer, thanks | 13:39 |
| dhellmann | clarkb (and others): thanks for fixing up the oslo.versionedobjects repo yesterday | 13:39 |
| *** bradjones has quit IRC | 13:40 | |
| *** ihrachyshka has quit IRC | 13:40 | |
| *** bknudson has quit IRC | 13:41 | |
| *** vigneshvar has quit IRC | 13:41 | |
| *** mfink_ has joined #openstack-infra | 13:42 | |
| *** ihrachyshka has joined #openstack-infra | 13:42 | |
| *** salv-orlando has quit IRC | 13:42 | |
| *** _nadya_ has joined #openstack-infra | 13:44 | |
| *** armax has joined #openstack-infra | 13:47 | |
| mordred | fungi: my god - what the heck version number is 0.8-0? | 13:47 |
| fungi | mordred: not one of mine ;) | 13:49 |
| openstackgerrit | Ihar Hrachyshka proposed openstack-infra/system-config: Collect #openstack-stable logs https://review.openstack.org/154474 | 13:49 |
| fungi | mordred: their previous version is 0.7-0 so this seems consistent (for them anyway) but maybe it predates pep 440 enforcement for uploads and so their latest upload failed? | 13:50 |
| pc_m | clarkb: fungi: ping | 13:50 |
| fungi | mordred: yeah, 0.7-0 was uploaded back in july | 13:50 |
| fungi | sileht: mordred: yeah and https://pypi.python.org/pypi/python-qpid-proton/0.8-0/json just has an empty list for the new release key in the releases dict, so clearly something is very wrong there | 13:52 |
| fungi | dstufft: ^ if you're awake already | 13:53 |
| fungi | pc_m: what else broke? | 13:53 |
| pc_m | fungi: I had VPN repo functional tests working in experimental queue, and then made them non-voting in check queue with https://review.openstack.org/#/c/152602/ | 13:53 |
| *** carl_baldwin has joined #openstack-infra | 13:53 | |
| sileht | fungi, I have ping kgiusti about that too | 13:54 |
| pc_m | fungi: However, recent reviews don't seem to be showing the check-neutron-vpnaas-dsvm-functional running | 13:54 |
| pc_m | fungi: Any idea what may be wrong? | 13:54 |
| mordred | fungi: every day brings me new joy | 13:54 |
| *** dmsimard_away is now known as dmsimard | 13:55 | |
| fungi | pc_m: have an example openstack/neutron-vpnaas change where you would have expected to see it appear? | 13:55 |
| pc_m | fungi: Here's one that was updated 2/7 (two days after the merge) https://review.openstack.org/145005 | 13:56 |
| pc_m | fungi: patch set 12 | 13:57 |
| *** marun has joined #openstack-infra | 13:57 | |
| *** emagana has joined #openstack-infra | 13:58 | |
| *** radez is now known as radez_g0n3 | 13:58 | |
| *** nicodemos has joined #openstack-infra | 13:59 | |
| *** yamahata has joined #openstack-infra | 13:59 | |
| *** mrmartin has joined #openstack-infra | 14:00 | |
| *** bknudson has joined #openstack-infra | 14:00 | |
| dstufft | fungi: I'm aware now! | 14:01 |
| dstufft | awake* | 14:01 |
| *** salv-orlando has joined #openstack-infra | 14:01 | |
| *** mwagner_lap has quit IRC | 14:02 | |
| *** fifieldt has quit IRC | 14:02 | |
| fungi | pc_m: blame dougwig. looks like he undid your change, probably in a rebase accident, via https://review.openstack.org/153373 | 14:02 |
| fungi | dstufft: well, that make one of us | 14:03 |
| * fungi still hasn't grabbed coffee yet | 14:03 | |
| dstufft | fungi: well physically awake anyways ;) not mentally | 14:03 |
| fungi | new rule: coffee first then irc | 14:03 |
| mordred | fungi: my usual route is "start water boiling for coffee, pick up laptop and start reading scrollback, pour coffee in mug, start typing" | 14:04 |
| pc_m | fungi: Thanks! I was going nuts... Do I push a new review to restore? | 14:04 |
| fungi | pc_m: yeah you'll need a new change to fix that | 14:04 |
| pc_m | fungi: Also, I think I'm seeing pylint failures in VPN now... need to check... wondering if the change broke that too. | 14:05 |
| dstufft | https://pypi.python.org/pypi/python-qpid-proton/0.8-0 doesn't seem to be have had any files uploaded to PyPI | 14:05 |
| dstufft | that | 14:05 |
| dstufft | that's why the file list is empty | 14:05 |
| dstufft | not a pypi bug, just the maintainer didn't upload | 14:05 |
| *** bcourt has joined #openstack-infra | 14:05 | |
| *** avozza is now known as zz_avozza | 14:05 | |
| *** annegentle has joined #openstack-infra | 14:06 | |
| *** bswartz has joined #openstack-infra | 14:06 | |
| *** elliottk_ has joined #openstack-infra | 14:07 | |
| *** david-lyle_afk is now known as david-lyle | 14:07 | |
| pc_m | fungi: Actually, looks like pylint failed for FW, I did a recheck and it passed. | 14:07 |
| *** elliottk has quit IRC | 14:07 | |
| fungi | dstufft: thanks. sileht is asking the maintainer for details | 14:08 |
| anteaya | pc_m: FW? | 14:08 |
| anteaya | firewall | 14:08 |
| anteaya | forced water | 14:08 |
| mordred | fungi: testing centos dib image on rax now | 14:08 |
| *** zz_avozza is now known as avozza | 14:08 | |
| anteaya | functional wawa | 14:08 |
| mordred | fungi: please cross your fingers - if this one fails I'll have to debug systemd ... | 14:09 |
| *** elliottk_ has left #openstack-infra | 14:09 | |
| *** mjturek1 has joined #openstack-infra | 14:09 | |
| fungi | pc_m: not sure, though it looked like dougwig's change added those jobs for the first time, so possibly it was already broken | 14:09 |
| *** elliottk_ has joined #openstack-infra | 14:10 | |
| *** reed has joined #openstack-infra | 14:10 | |
| fungi | anteaya: presumably firewall (neutron-fwaas specifically since we were previously speaking in context of the neutron-vpnaas project) | 14:10 |
| *** elliottk has joined #openstack-infra | 14:10 | |
| anteaya | okay, surprisingly my first guess has backing | 14:11 |
| anteaya | I was hoping for functional wawa | 14:11 |
| anteaya | it just rolls off the tongue | 14:11 |
| *** mestery has quit IRC | 14:11 | |
| fungi | mordred: yeah, i've recently been forced to learn all sorts of things i didn't want to know about systemd, since i chose to accept the new default in debian/jessie | 14:12 |
| *** elliottk_ has left #openstack-infra | 14:14 | |
| *** sarob has joined #openstack-infra | 14:15 | |
| *** GheRivero has quit IRC | 14:15 | |
| anteaya | wznoinsk: hello | 14:17 |
| *** lnxnut has joined #openstack-infra | 14:17 | |
| *** GheRivero has joined #openstack-infra | 14:17 | |
| anteaya | wznoinsk: turns out you are the intel nfv ci rep I have been looking for | 14:17 |
| openstackgerrit | Paul Michali proposed openstack-infra/project-config: Restore lost changes to neutron-vpnaas https://review.openstack.org/154487 | 14:18 |
| *** sarob has quit IRC | 14:19 | |
| pc_m | fungi: can you and others review ^^^ | 14:19 |
| *** dustins has joined #openstack-infra | 14:21 | |
| *** dmsimard is now known as dmsimard_away | 14:22 | |
| *** lnxnut has quit IRC | 14:22 | |
| *** radez_g0n3 is now known as radez | 14:23 | |
| *** amotoki has joined #openstack-infra | 14:23 | |
| *** mriedem has joined #openstack-infra | 14:25 | |
| hashar | hey guys :-] | 14:27 |
| sdague | fungi: https://review.openstack.org/#/c/154398 - I'd like to encourage the external plugin migrations, so if you could take a look at that | 14:27 |
| hashar | is anyone familiar with nodepool here? I am wondering why it needs to maintain a database connection for each node :/ | 14:27 |
| mordred | hashar: that seems like a lot of database connections... | 14:28 |
| anteaya | jaypipes: so I'm told wznoinsk is the intel nfv ci rep, just found that out. He has been attending meetings so should be around if you have questions. I know you were looking for the intel nfv rep as well | 14:28 |
| hashar | mordred: taken from http://ci.openstack.org/nodepool/installation.html#database | 14:28 |
| hashar | " When adding or deleting nodes, Nodepool will hold open a database connection for each node. " | 14:29 |
| hashar | seems that is just when creating and deleting them :-] | 14:29 |
| mordred | hrm. yes - I can verify this to be the case in our nodepool | 14:29 |
| openstackgerrit | Merged openstack-infra/project-config: Remove stable-compat jobs from pycadf https://review.openstack.org/154268 | 14:29 |
| jaypipes | anteaya: ty much! | 14:30 |
| mordred | hashar: I'd have to wait for jeblair for a why | 14:30 |
| anteaya | jaypipes: np, keep me informed if you could | 14:30 |
| kgiusti | fungi: sileht: pong - seems I've caused you guys quite a bit of grief - sorry. What can I do to help? | 14:31 |
| jaypipes | anteaya: will do. | 14:31 |
| anteaya | jaypipes: thanks | 14:31 |
| hashar | mordred: I guess I can survive; will probably start with 50 VM or 100 connections which is not unreasonable :-] | 14:31 |
| sileht | kgiusti, the 0.8-0 file is missing on pypi | 14:31 |
| wznoinsk | hi anteaya, nfv ci is in my timeline for this/next week not yet started | 14:31 |
| anteaya | wznoinsk: ah okay | 14:31 |
| sileht | kgiusti, the entry have been created but the file is missing | 14:32 |
| anteaya | wznoinsk: jaypipes here he is | 14:32 |
| anteaya | wznoinsk: so here is some background | 14:32 |
| anteaya | wznoinsk: at paris summit I agreed to work with whoever adrian sent me for nfv testing | 14:32 |
| openstackgerrit | Merged openstack-infra/project-config: Add docs jobs for diskimage-builder https://review.openstack.org/153810 | 14:32 |
| kgiusti | sileht: ah, so much for "learn by doing" - I obviously screwed something up there. | 14:32 |
| anteaya | wznoinsk: so then I was given the names of some folks in china, who are great people | 14:32 |
| *** ddieterly has joined #openstack-infra | 14:33 | |
| anteaya | wznoinsk: but as it turns out they aren't working on nfv testing, they are working on the inital foundation pci and sr-iov | 14:33 |
| kgiusti | sileht: I'm on the package management page now - it shows both 0.8-0 and 0.7-0 in the release list. | 14:33 |
| anteaya | wznoinsk: my understanding is pci is being tested and they are working on sr-iov | 14:33 |
| wznoinsk | yes, things has shifted here a bit but the nfv ci is coming very soon | 14:33 |
| kgiusti | sileht: is the package bad? or just not visible to you? | 14:33 |
| wznoinsk | yes, Yongli is SRIOV CI | 14:34 |
| anteaya | wznoinsk: however that doesn't help us as we had initiall asked for nfv, which is what I had agreed to support | 14:34 |
| anteaya | wznoinsk: right and they are doing great work | 14:34 |
| sileht | kgiusti, 0.7 have a file attached to the package but not 0.8 | 14:34 |
| anteaya | wznoinsk: and jaypipes who is with nova, who wants to know the status of testing nfv is also interested | 14:34 |
| anteaya | wznoinsk: so intel has been doing some great work and is very diligent and that is good | 14:35 |
| anteaya | wznoinsk: but my initial agreement was to support nfv testing | 14:35 |
| anteaya | so I need to see some work happening on nfv testing | 14:35 |
| wznoinsk | I'm just finishing on our other Networking CI which started to comment last night, NFV to come next | 14:35 |
| anteaya | have a presented my position accurately | 14:35 |
| anteaya | wznoinsk: here is the thing | 14:35 |
| anteaya | I have put a lot of work in to helping intel ci reps | 14:36 |
| anteaya | including being up in the middle of the night for meetings that suit them | 14:36 |
| anteaya | I need to see some of what I said I would be getting | 14:36 |
| anteaya | I have been working hard for months | 14:36 |
| *** abhirc has joined #openstack-infra | 14:36 | |
| *** baoli has joined #openstack-infra | 14:36 | |
| anteaya | since paris summit | 14:36 |
| anteaya | I need to feel that that dedication is understood apprcicated respected valued and will turn into the results we asked for in the first place | 14:37 |
| anteaya | which is nfv testing | 14:37 |
| sileht | kgiusti, perhaps you just have to reuploaded the version | 14:38 |
| anteaya | wznoinsk: so if things shifted around a bit and nfv got a back burner, perhaps they need to shift back | 14:38 |
| anteaya | wznoinsk: since I had hoped by this time we would at least have some sort of prototype | 14:39 |
| wznoinsk | Adrian mentioned your dedication and support many times, I believe getting the CI working will not be a hard to accomplish task, is there any priority you'd like to see on top of it? any particular functions tested etc? | 14:39 |
| anteaya | I would like to hear it is a priority for you | 14:39 |
| anteaya | you think it will not be a hard to accomplish task | 14:39 |
| anteaya | I disagree | 14:39 |
| anteaya | noone else is doing it | 14:39 |
| anteaya | and it is entirely new territory | 14:39 |
| anteaya | so I would like to feel someone besides me and nova addresses it with a sense of urgency | 14:40 |
| wznoinsk | I mean from an infrastructure point of view (to get the CI working), what tests we want to run inside and how to solve them that's a neverending story | 14:40 |
| anteaya | well it sure is now | 14:40 |
| anteaya | since it isn't even started | 14:40 |
| anteaya | so let's start shall we? | 14:40 |
| *** baoli has quit IRC | 14:41 | |
| wznoinsk | sure | 14:41 |
| anteaya | wznoinsk: great | 14:41 |
| *** baoli has joined #openstack-infra | 14:41 | |
| anteaya | jaypipes: do let me know when you have time to discuss initial testing steps with wznoinsk | 14:41 |
| anteaya | or point me to the channel log after the discussion | 14:41 |
| anteaya | wznoinsk: please co-ordinate with jaypipes | 14:42 |
| anteaya | I'd like to see you had an initial discussion by next week | 14:42 |
| anteaya | at the very least | 14:42 |
| kgiusti | sileht: probably, but I'd like to know what I did wrong in the first place - this is my first pypi package, and it's obvious I don't know what I'm doing :( | 14:42 |
| *** pradk has joined #openstack-infra | 14:43 | |
| *** sabeen1 has joined #openstack-infra | 14:43 | |
| *** dims__ has quit IRC | 14:43 | |
| jaypipes | anteaya: will try my best. | 14:43 |
| *** amrith is now known as _amrith_ | 14:43 | |
| anteaya | jaypipes: thank you | 14:43 |
| *** dims__ has joined #openstack-infra | 14:43 | |
| *** dims__ has quit IRC | 14:43 | |
| mordred | kgiusti: the release has been registered, but there are no files there | 14:44 |
| sileht | kgiusti, I'm not familiar with pypi too, I just known that openstack-infra use https://pypi.python.org/pypi/twine | 14:44 |
| wznoinsk | jaypipes: I need to synch up with Adrian on latest discussion about nfv ci here but we can start talking already, what you're upto exactly? | 14:44 |
| kgiusti | sileht: I can't even be sure the setup.py is correct - the version #'s aren't right | 14:44 |
| *** pradk has quit IRC | 14:44 | |
| *** joesavak has joined #openstack-infra | 14:44 | |
| *** dims__ has joined #openstack-infra | 14:44 | |
| mordred | kgiusti: when you did the upload, did you run "python setup.py upload" or did you run "python setup.py sdist upload" (the first form will happily upload 0 files for you) | 14:44 |
| kgiusti | mordred: I did the second - 'sdist upload' | 14:45 |
| anteaya | wznoinsk: do share the channel log in your chat with adrian: http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2015-02-10.log | 14:45 |
| mordred | kgiusti: hrm. weird | 14:45 |
| *** esmute has quit IRC | 14:45 | |
| kgiusti | mordred: or at least that's the way I remember it. But I can't remember what I had for breakfast this morning, so... | 14:45 |
| *** radez is now known as radez_g0n3 | 14:45 | |
| mordred | :) | 14:46 |
| *** juice has quit IRC | 14:46 | |
| mordred | me either | 14:46 |
| mordred | and I'm still having it :) | 14:46 |
| kgiusti | mordred: let me try that again. But before I do, do you know how I can increment the '-0' portion of the version #? | 14:46 |
| mordred | you can upload by hand here: https://pypi.python.org/pypi?:action=files&name=python-qpid-proton&version=0.8-0 | 14:47 |
| mordred | kgiusti: where is your source code and I'll take a look? | 14:47 |
| kgiusti | "Hi, my name's Kenneth, and I'm a C programmer." | 14:47 |
| mordred | hehe | 14:47 |
| *** esmute has joined #openstack-infra | 14:47 | |
| mordred | kgiusti: before we started openstack, I was hacking on the internals of a database in C++ with jaypipes - I hear you :) | 14:48 |
| *** juice has joined #openstack-infra | 14:48 | |
| *** amitgandhinz has joined #openstack-infra | 14:49 | |
| *** tiswanso has joined #openstack-infra | 14:49 | |
| kgiusti | mordred: be careful of jaypipes - I heard he owns a pug. Pug owners aren't quite normal. | 14:49 |
| jaypipes | I have 3 pugs actually :) | 14:49 |
| kgiusti | mordred: https://github.com/apache/qpid-proton/blob/0.8/proton-c/bindings/python/setup.py.in | 14:49 |
| kgiusti | jaypipes: you have me beat - I only have 2. Primary and backup. | 14:49 |
| jaypipes | :) | 14:50 |
| anteaya | jaypipes also has a pug stroller | 14:50 |
| anteaya | that still kills me | 14:50 |
| fungi | sdague: no concerns with that job cloning something from github which it could just as easily pull from our git mirror? | 14:50 |
| jaypipes | haters! :) | 14:50 |
| anteaya | it makes me laugh every time | 14:50 |
| kgiusti | jaypipes: pug stroller: +1! | 14:50 |
| fungi | sdague: or does the plugin layer have sufficient smarts to clone from the local copy on the worker (in which case it's just fetching updates from github, so not quite as bad)? | 14:51 |
| mordred | kgiusti: so - you want to set: https://github.com/apache/qpid-proton/blob/0.8/proton-c/bindings/python/setup.py.in#L28 if you want to bump the final version ... however, first let me make another suggestion (and I'll follow up with a PR if you want) | 14:51 |
| mordred | kgiusti: which is to change the - here https://github.com/apache/qpid-proton/blob/0.8/proton-c/bindings/python/setup.py.in#L93 to a . | 14:52 |
| mordred | kgiusti: at which point your versions will be compatible with normal python version semantics | 14:52 |
| mordred | kgiusti: happy to send a patch - but it's a one-character change, so I don't know if it's worth me doing it | 14:53 |
| sdague | fungi: by adding to PROJECTS it will get updated by zuul | 14:53 |
| kgiusti | mordred: man I owe you one big $(BEVERAGE) of Vancouver's finest. | 14:54 |
| *** koolhead17 has joined #openstack-infra | 14:54 | |
| fungi | sdague: cool, i'll read the source for details on exactly what 'enable_plugin gnocchi https://github.com/stackforge/gnocchi | 14:54 |
| mordred | kgiusti: mmm. I like $(BEVERAGE) ! | 14:54 |
| fungi | masterenable_plugin gnocchi https://github.com/stackforge/gnocchi | 14:54 |
| fungi | master' does | 14:54 |
| fungi | gah, my paste has gone off the rails | 14:54 |
| kgiusti | mordred: thought so (just a lucky guess) :) | 14:54 |
| sdague | fungi: ERROR_ON_CLONE is still true the gate runs, so I think it's actually fine | 14:55 |
| fungi | sdague: interesting... so the second parameter to enable_plugin is basically a no-op in this case (and could be replaced with garbage)? | 14:56 |
| *** yfried_ is now known as yfried|afk | 14:56 | |
| *** otter768 has joined #openstack-infra | 14:58 | |
| fungi | sdague: aha, yep. it's just adding to the GITREPO array which ends up being unused in our ci | 14:59 |
| sdague | fungi: yeh | 14:59 |
| fungi | still would be nice to avoid future cargo cultists thinking this parameter is doing something real in the job | 14:59 |
| *** jgallard__ has joined #openstack-infra | 14:59 | |
| sdague | so in the ec2 job definition I expressed that fact with a comment | 14:59 |
| *** r-daneel has joined #openstack-infra | 15:00 | |
| sdague | once we've got a few more of these out there, I'm going to write a more complete guide | 15:00 |
| *** e0ne is now known as e0ne_ | 15:02 | |
| *** otter768 has quit IRC | 15:02 | |
| *** sabeen2 has joined #openstack-infra | 15:03 | |
| fungi | would something like UNUSED_ARGUMENT in place of the url work fine with devstack? | 15:03 |
| *** sabeen1 has quit IRC | 15:04 | |
| *** esmute has quit IRC | 15:04 | |
| openstackgerrit | Merged openstack-infra/project-config: Add devstack-plugin-sheepdog project into stackforge https://review.openstack.org/154199 | 15:04 |
| *** radez_g0n3 is now known as radez | 15:05 | |
| openstackgerrit | Merged openstack-infra/project-config: Add puppet tripleo check tests https://review.openstack.org/154152 | 15:05 |
| *** BadCub has quit IRC | 15:05 | |
| *** vipul has quit IRC | 15:05 | |
| *** radez is now known as radez_g0n3 | 15:05 | |
| *** mwagner_lap has joined #openstack-infra | 15:06 | |
| *** esmute has joined #openstack-infra | 15:06 | |
| *** BadCub has joined #openstack-infra | 15:07 | |
| *** vipul has joined #openstack-infra | 15:07 | |
| *** e0ne_ is now known as e0ne | 15:07 | |
| *** yfried|afk is now known as yfried_ | 15:07 | |
| anteaya | so neutron is discussing creating a neutron lib to address tagging issues with the *aas repos | 15:07 |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck: Add query for cells volume exercise bug 1420322 https://review.openstack.org/154508 | 15:07 |
| openstack | bug 1420322 in OpenStack Compute (nova) "gate-devstack-dsvm-cells fails in volumes exercise with "Server ex-vol-inst not deleted"" [Medium,Triaged] https://launchpad.net/bugs/1420322 | 15:07 |
| anteaya | if anyone wants to join -neutron and offer an opinion | 15:08 |
| *** dangers_away is now known as dangers | 15:08 | |
| openstackgerrit | Merged openstack-infra/project-config: move from macro install-chefdk to chef-bootstrap https://review.openstack.org/154207 | 15:09 |
| *** emagana has quit IRC | 15:09 | |
| openstackgerrit | Merged openstack-infra/project-config: Exporting PROJECTS variable for glusterfs job https://review.openstack.org/154227 | 15:09 |
| openstackgerrit | Merged openstack-infra/project-config: set -x in cache-testrepository install https://review.openstack.org/154337 | 15:09 |
| *** emagana has joined #openstack-infra | 15:09 | |
| *** nelsnelson has joined #openstack-infra | 15:10 | |
| *** zz_jgrimm is now known as jgrimm | 15:10 | |
| *** thedodd has joined #openstack-infra | 15:11 | |
| *** nelsnelson has quit IRC | 15:11 | |
| sileht | fungi, for your comment here: https://review.openstack.org/#/c/154398/2/jenkins/jobs/gnocchi.yaml, if I use 'enable_plugin gnocchi /opt/git/stackforge/gnocchi' like the glusterfs plugin does, it's OK ? | 15:13 |
| *** Longgeek has joined #openstack-infra | 15:14 | |
| fungi | sileht: yeah, that looks like it should be just fine | 15:14 |
| sileht | fungi, thanks | 15:15 |
| openstackgerrit | Mehdi Abaakouk proposed openstack-infra/project-config: gnocchi: install the git repo https://review.openstack.org/154398 | 15:15 |
| *** dmsimard_away is now known as dmsimard | 15:15 | |
| *** emagana has quit IRC | 15:15 | |
| fungi | sileht: alternatively i think we can put something there to make it more obvious it's not referenced at all by the job, but probably not as critical for now | 15:15 |
| *** Longgeek has quit IRC | 15:16 | |
| *** sarob has joined #openstack-infra | 15:16 | |
| fungi | sileht: the enable_plugin function just expects something in that parameter, but the array it's added to shouldn't actually be getting values passed to git calls so will never be validated | 15:16 |
| openstackgerrit | Merged openstack-infra/project-config: Zuul: Ignore dependencies in merge-check https://review.openstack.org/154278 | 15:16 |
| jeblair | i will restart zuul now for that ^ | 15:17 |
| *** coolsvap is now known as coolsvap_ | 15:17 | |
| *** nelsnelson has joined #openstack-infra | 15:18 | |
| *** cnesa has joined #openstack-infra | 15:18 | |
| fungi | cool, thanks | 15:18 |
| jeblair | fungi: can you look at 154232? | 15:19 |
| fungi | that'll also address jhesketh's comments about all the grey dots on the zuul.openstack.org sitye | 15:19 |
| fungi | the restart will i mean, not 154232 | 15:20 |
| *** sigmavirus24_awa is now known as sigmavirus24 | 15:20 | |
| *** sarob has quit IRC | 15:20 | |
| jeblair | ya | 15:20 |
| *** gokrokve has joined #openstack-infra | 15:21 | |
| jeblair | also, anyone have any depends-on: changes handy? | 15:22 |
| fungi | i have some i think. looking | 15:22 |
| *** juice has quit IRC | 15:22 | |
| vponomaryov | Hello. Does anybody know is there requirement to support SQLite in OpenStack projects? | 15:23 |
| anteaya | vponomaryov: not that I am aware | 15:23 |
| fungi | hrm, maybe not any of mine now... looks like the ones they depend on have merged | 15:23 |
| *** esmute has quit IRC | 15:23 | |
| anteaya | vponomaryov: the db of choice is mysql to the best of my knowledge | 15:23 |
| *** NobodyCam has quit IRC | 15:24 | |
| *** BadCub has quit IRC | 15:24 | |
| *** vipul has quit IRC | 15:24 | |
| *** jesusaurus has quit IRC | 15:24 | |
| fungi | vponomaryov: sqlite support has mostly been to provide an easy test target so that devs don't need to install a database daemon (though i agree that's a poor excuse for a proper mock) | 15:25 |
| *** jedimike has joined #openstack-infra | 15:25 | |
| *** esmute has joined #openstack-infra | 15:25 | |
| *** NobodyCam has joined #openstack-infra | 15:26 | |
| anteaya | the middle of the night meetings followed by the every other week early morning neutron meeting is not my favourite combination | 15:26 |
| anteaya | I'm going to see if I can get some more sleep | 15:26 |
| *** juice has joined #openstack-infra | 15:26 | |
| *** BadCub has joined #openstack-infra | 15:26 | |
| *** vipul has joined #openstack-infra | 15:27 | |
| openstackgerrit | Bharat Kumar Kobagana proposed openstack-infra/project-config: Minimizing the plugin specific configuration in job definition https://review.openstack.org/154518 | 15:27 |
| fungi | gerrit message searches don't do well with depends-on | 15:27 |
| vponomaryov | fungi, anteya: thanks | 15:27 |
| *** jesusaurus has joined #openstack-infra | 15:27 | |
| jeblair | fungi: yeah, i was afraid of that (that's going to make finding reverse dependencies difficult) | 15:28 |
| jeblair | i'll cook up something synthetic | 15:28 |
| fungi | jeblair: aha! https://review.openstack.org/140043 looks like a great test | 15:29 |
| fungi | jeblair: it has two depends-on headers, one of which references an abandoned change. not sure what that might break ;) | 15:29 |
| jeblair | fungi: the second references a change in the same repo, which is unecessary | 15:29 |
| *** lnxnut has joined #openstack-infra | 15:29 | |
| *** lnxnut has quit IRC | 15:29 | |
| fungi | ahh, yep | 15:30 |
| jeblair | (but should still work as long as it does not create a cycle | 15:30 |
| fungi | also there's a semi-circular dep | 15:30 |
| * mordred throws angry bulls at everyone around | 15:30 | |
| jeblair | fungi: how? | 15:30 |
| * mordred throws angry bulls at self | 15:30 | |
| *** timcline has joined #openstack-infra | 15:30 | |
| *** ghostpl has quit IRC | 15:30 | |
| *** carl_baldwin has quit IRC | 15:30 | |
| *** lnxnut has joined #openstack-infra | 15:30 | |
| mordred | sigh. I just spent 2 hours trying to troubleshoot why the systemd stuff wasn't working - and the real problem was that I'd run the wrong image buidl script and had not actually included it in the image at all | 15:30 |
| *** wenlock has joined #openstack-infra | 15:31 | |
| fungi | jeblair: 140043 depends on (abandoned)151910 and 151912, then 151912 depends on (abandoned)151910 too | 15:31 |
| *** koolhead17 has quit IRC | 15:31 | |
| fungi | so maybe not circular, just redundant | 15:31 |
| fungi | mordred: you should totally blame lennart for that | 15:32 |
| mordred | fungi: it's clearly his fault | 15:32 |
| fungi | it's the fashionable choice, after all | 15:32 |
| sdague | mordred: wait... you actually concluded that systemd was not the source of some problem you had? ;) | 15:32 |
| *** tmcpeak has quit IRC | 15:32 | |
| mordred | sdague: right? | 15:32 |
| mordred | sdague: I'm as shocked as the next person | 15:33 |
| *** koolhead17 has joined #openstack-infra | 15:33 | |
| jeblair | fungi: i'll wait a few mins for a bare-precise to register, then recheck it | 15:33 |
| mordred | also - if this does work - I will quickly need to say that the systemd answer to this problem is much clearer than the upstart answer was | 15:33 |
| mordred | although both are bonghits amounts of worse than sysvinit woudl be to solve the exact same problem | 15:33 |
| *** dustins_ has joined #openstack-infra | 15:34 | |
| *** andreykurilin_ has joined #openstack-infra | 15:34 | |
| jeblair | when i quit firefox, it spends a couple of minutes in iowait. :( | 15:34 |
| *** zz_dimtruck is now known as dimtruck | 15:34 | |
| fungi | jeblair: 144167 and 152508 would make good (and somewhat more typical) examples | 15:34 |
| *** achanda has joined #openstack-infra | 15:34 | |
| fungi | jeblair: one has a single dep, one has two deps one of which is merged and the other is not | 15:35 |
| jeblair | fungi: ah 152508 may actually change test results based on it! | 15:35 |
| fungi | indeed | 15:36 |
| *** tsg has joined #openstack-infra | 15:36 | |
| *** dustins has quit IRC | 15:37 | |
| clarkb | you all are awake far too early in the morning | 15:39 |
| clarkb | mordred how was my ansible stuff? | 15:39 |
| openstackgerrit | Maru Newby proposed openstack-infra/project-config: Add conditional skips for neutron jobs https://review.openstack.org/152298 | 15:39 |
| jeblair | clarkb: i woke up the moment the zuul config change was approved | 15:39 |
| jeblair | bare-precise | building | 0.73 | 15:40 |
| jeblair | jeez ^ | 15:40 |
| fungi | gate-system-config-puppet-syntax NOT_REGISTERED | 15:41 |
| fungi | saw that | 15:41 |
| jeblair | fungi: ^^ | 15:41 |
| fungi | yeah | 15:41 |
| fungi | jeblair: is that in rax? | 15:41 |
| jeblair | yup | 15:41 |
| jeblair | all 3 of the ones building are | 15:41 |
| fungi | it's "waiting" for a floating ip to become available and get assigned i think | 15:41 |
| fungi | er, not a floating ip | 15:41 |
| jroll | floating ip in rax, huh | 15:41 |
| fungi | but an ip address | 15:41 |
| marun | sdague: I'm happy to ensure that testr is installed as part of nodepool prep. Any pointers as to how to do that? | 15:42 |
| jeblair | i kind of wish that if nova did not have the resources to complete an operation it would just say so | 15:42 |
| fungi | at least we've been told in the past when nova boot calls don't complete for up to an hour in rackspace it's because nova didn't have a global ipv4 address to turn over and assign to the instance | 15:42 |
| mordred | clarkb: sorry - haven't looked yet - I dove straight back in to nodepool-dib | 15:42 |
| mordred | clarkb: let me look now | 15:42 |
| jeblair | marun: why do you want to install testr in nodepool? | 15:43 |
| *** sdake has joined #openstack-infra | 15:43 | |
| openstackgerrit | Sergey Skripnick proposed openstack-infra/project-config: Add rally-ci group with label-Verified permission https://review.openstack.org/154532 | 15:43 |
| marun | jeblair: this - https://review.openstack.org/#/c/154299/ | 15:43 |
| *** achanda has quit IRC | 15:43 | |
| sdague | jeblair: I suggested it become part of the base image, instead of installing on the fly | 15:44 |
| clarkb | it should happen as part of the job. we dont install any other python libs globally | 15:44 |
| marun | jeblair: sdague suggested that installing testr on-demand for jobs that don't have devstack install it system-wide was a bad idea. | 15:44 |
| jeblair | sdague: yeah, make sense... but i wonder, why not run it out of the venv? what needs to use testr? | 15:44 |
| *** pradk has joined #openstack-infra | 15:44 | |
| jeblair | marun: ^ | 15:44 |
| kgiusti | sileht: mordred: fyi - looks like python-qpid-proton-0.8.2.tar.gz should be available on pypi. I'll try running the amqp1 functional tests against it now - fingers (toes, eyes) crossed... | 15:44 |
| mtreinish | jeblair: it's the subunit artifact processing | 15:44 |
| mordred | kgiusti: woohoo! | 15:44 |
| marun | jeblair: clarkb suggested that it would be preferable to do it system-wide | 15:44 |
| marun | jeblair: if we wanted to do it out of a venv if it wasn't installed globally, we'd have to add a config option to d-g so that the d-g wrapper would know which venv to target | 15:45 |
| clarkb | jeblair tempest expects it globally so rather than a bunch of logic to find testr I figured use the expectation it is global | 15:45 |
| jeblair | mtreinish: what artifact processing? | 15:45 |
| marun | e.g. /top/stack/new/neutron/.tox/dsvm-functional | 15:46 |
| marun | jeblair: https://github.com/openstack-infra/devstack-gate/blob/master/functions.sh#L514 | 15:46 |
| sileht | kgiusti, cool thanks | 15:46 |
| marun | jeblair: post-processing of testr results for display as html | 15:46 |
| marun | jeblair: tempest already did it, I'm working on allowing non-tempest jobs to use the same code | 15:46 |
| marun | jeblair: we were previously copy-n-pasting into hook scripts | 15:47 |
| mtreinish | jeblair: pulling the most recent subunit stream out of the testrepository, to both store it on the log server and make the html page everyone is fond of | 15:47 |
| marun | jeblair: the catch is that tempest installs testr system-wide, but non-tempest jobs running out of venv's do not | 15:47 |
| kgiusti | sileht: mordred: no - thank you folks, I'm the one who messed it up to begin with. :P I'll let you know how the functional tests go. | 15:47 |
| *** sdake has quit IRC | 15:47 | |
| *** sdake has joined #openstack-infra | 15:48 | |
| *** stevemar has joined #openstack-infra | 15:48 | |
| *** hdd has joined #openstack-infra | 15:49 | |
| jeblair | marun, sdague, mtreinish, clarkb: so we probably want it in a system-wide venv, like zuul-cloner and subunit2sql, right? | 15:49 |
| sdague | jeblair: seems reasonable to me | 15:49 |
| *** skolekonov has quit IRC | 15:50 | |
| mordred | I feel like AFS should be able to solve this, but for the life of me I can't think how | 15:50 |
| marun | jeblair: any pointers as to which repo to propose a change to, to make that happen? | 15:50 |
| mtreinish | jeblair: sure, although we kill the subunit2sql venv after it runs in nodepool. We could just remove the rm and reuse it for this | 15:51 |
| clarkb | jeblair I think that over complicates it but ok | 15:51 |
| *** naggappan has quit IRC | 15:51 | |
| jeblair | clarkb: but it's a python program, it might have dependencies, etc; shouldn't we isolate it from the global environment? | 15:52 |
| *** rwsu-afk is now known as rwsu | 15:52 | |
| clarkb | its been installed by jobs globally for a long time, and that apparently works | 15:52 |
| jeblair | clarkb: which jobs? | 15:52 |
| clarkb | tempest jobs | 15:52 |
| jeblair | clarkb: right, that's not globally though | 15:52 |
| clarkb | yes it is or the current code doesnt work | 15:53 |
| clarkb | mtreinish ^ | 15:53 |
| jeblair | clarkb: i mean it's not all jobs | 15:53 |
| *** dhritishikhar_ has joined #openstack-infra | 15:53 | |
| marun | jeblair: all dsvm jobs? | 15:53 |
| clarkb | no just those expecting to run the testr processing | 15:53 |
| jeblair | clarkb: i do agree it would be nice if the job itself did this | 15:54 |
| *** tsg has quit IRC | 15:54 | |
| *** yfried_ is now known as yfried|afk | 15:54 | |
| mtreinish | clarkb: yeah it's being installed by devstack. Although jogo has a patch up to stop that | 15:54 |
| jeblair | (which it already does) | 15:54 |
| *** ssam2 has quit IRC | 15:54 | |
| jeblair | but how to handle the different places it might be installed? | 15:54 |
| *** _amrith_ is now known as amrith | 15:54 | |
| *** cnesa has quit IRC | 15:55 | |
| marun | jeblair: as previously suggested, a configuration variable indicating a venv to target is one option | 15:55 |
| jeblair | i was initially confused because to date, this code has only run if we're running tempest in d-g | 15:55 |
| jeblair | but we're talking about other jobs re-using the framework | 15:56 |
| marun | correct. re-use instead of copy-n-paste | 15:56 |
| jeblair | marun: presumably you have a hook function that's something like "cd /path/to/neutron/; tox" right? | 15:56 |
| marun | jeblair: correct | 15:56 |
| jeblair | marun, clarkb: so that could be followed by "process_results /path/to/neutron/.tox/something" ? | 15:57 |
| clarkb | you could pass in the env bin path yes | 15:57 |
| clarkb | you need testr and python to be run out of that | 15:58 |
| marun | jeblair: we are already doing something similar in our copy/paste solution | 15:58 |
| marun | https://github.com/openstack/neutron/blob/master/neutron/tests/contrib/post_test_hook.sh#L61 | 15:58 |
| *** ociuhandu has joined #openstack-infra | 15:58 | |
| *** ghostpl has joined #openstack-infra | 15:58 | |
| marun | so if you'd prefer, I could remove the DEVSTACK_GATE_TESTR_ARTIFACT_TARGET option and require that hook scripts call the function to process as suggested | 15:59 |
| jeblair | that seems fairly reasonable; any reason not to do that? | 15:59 |
| marun | there is 1 | 16:00 |
| jeblair | (wow, nodepool gave up on the bare-precise nodes) | 16:00 |
| marun | if we want to change the function, we'd have to coordinate landing the change across an ever-increasing number of jobs | 16:00 |
| marun | if it's localized to d-g, then maintenance would be much simpler | 16:01 |
| marun | I'm not sure how much it will change, though. | 16:01 |
| jeblair | marun: oh, i was still thinking the function would be in d-g, but would just take some args | 16:01 |
| *** jaypipes has quit IRC | 16:01 | |
| jeblair | marun: or is that what you meant too, but you meant changing the function signature/api? | 16:01 |
| marun | jeblair: Sure, I'm just saying that we'd have to enforce a hard contract on that function signature | 16:01 |
| marun | jeblair: yeah | 16:02 |
| jeblair | gotcha | 16:02 |
| marun | jeblair: I'm not staying that's a deal-breaker, just that it's something to consider. | 16:02 |
| fungi | jeblair: we just grew a ready bare-precise in rax-iad at least | 16:02 |
| jeblair | yay | 16:02 |
| fungi | just one, but that's enough to get jobs registered | 16:02 |
| *** signed8bit has joined #openstack-infra | 16:03 | |
| openstackgerrit | Merged openstack-infra/elastic-recheck: Add query for cells volume exercise bug 1420322 https://review.openstack.org/154508 | 16:03 |
| openstack | bug 1420322 in OpenStack Compute (nova) "gate-devstack-dsvm-cells fails in volumes exercise with "Server ex-vol-inst not deleted"" [Medium,Triaged] https://launchpad.net/bugs/1420322 | 16:03 |
| fungi | wind warning here today. hopefully the power will hold up | 16:03 |
| marun | fungi: whereabouts? | 16:03 |
| fungi | marun: islands off the coast of north carolina | 16:04 |
| marun | fungi: ah, nice. you live on an island? | 16:04 |
| jeblair | clarkb, sdague: what do you think about that? | 16:04 |
| *** mtanino has joined #openstack-infra | 16:04 | |
| fungi | just finished strapping down or bringing in everything which might go airborne at categoty 1 hurricane windspeeds (though this is a noreaster, so it's just gusts not sustained) | 16:04 |
| *** esmute has quit IRC | 16:04 | |
| fungi | marun: yep | 16:04 |
| marun | fungi: Bridge or ferry/boat to the mainland? | 16:05 |
| fungi | marun: we have a couple bridges... it's only a few miles to the mainland | 16:05 |
| jeblair | fungi: enqueued 140043 | 16:05 |
| clarkb | jeblair: being explicit about the python bin path is fine too | 16:05 |
| fungi | still, enough wind to send your trashcan a few blocks away | 16:06 |
| jeblair | fungi: it pulled in both changes it depended on (including the abandoned one) | 16:06 |
| *** BadCub has quit IRC | 16:06 | |
| fungi | jeblair: i see that--looking good! | 16:06 |
| *** BadCub has joined #openstack-infra | 16:06 | |
| *** NobodyCam has quit IRC | 16:07 | |
| marun | fungi: I grew up on a peninsula whose access to civilization was a 40m ferry ride. I like living near water. But it was sheltered ocean - no hurricanes! | 16:07 |
| *** esmute has joined #openstack-infra | 16:07 | |
| *** NobodyCam has joined #openstack-infra | 16:07 | |
| jeblair | fungi: enqueued 152508 and it pulled in the drupal change too | 16:07 |
| *** achanda has joined #openstack-infra | 16:07 | |
| *** yfried|afk is now known as yfried_ | 16:07 | |
| *** che-arne has joined #openstack-infra | 16:08 | |
| *** achanda has quit IRC | 16:08 | |
| *** enikanorov__ has joined #openstack-infra | 16:09 | |
| fungi | i suppose we could turn off the click to expand on the non-live changes, though maybe that's unnecessary complication in the js | 16:09 |
| *** enikanorov_ has quit IRC | 16:09 | |
| fungi | the grey subway dots will make it pretty obvious anyway i think | 16:10 |
| fungi | also the eta is sort of meaningless for them | 16:10 |
| marun | jeblair, clarkb, sdague: so, our choices are a) set a bin path to testr b) require that hook scripts explicitly call a function? | 16:10 |
| *** crc32 has joined #openstack-infra | 16:10 | |
| clarkb | marun: and I think a is a subset of b right? | 16:10 |
| marun | clarkb: hmmm, I hadn't thought so. I was thinking that a) was automatic post-processing as we're already doing but setting a bin path via a configuration option | 16:11 |
| *** jkraj has joined #openstack-infra | 16:11 | |
| jeblair | marun: what's doing the post-processing? | 16:12 |
| marun | jeblair: currently, d-g cleanup. | 16:12 |
| jeblair | oh i thought it only did it for tempest | 16:12 |
| marun | jeblair: or, do you mean executable? | 16:12 |
| marun | jeblair: it did, until I refactored :) | 16:12 |
| marun | https://review.openstack.org/#/c/153677/ | 16:12 |
| marun | jeblair: but it's just a function call, so whether it happens in the post-hook or in cleanup doesn't really matter | 16:13 |
| jeblair | this is when you learn that my mental model of openstack-infra is kept up to date entirely by code reviews :) | 16:13 |
| marun | jeblair: heh | 16:13 |
| fungi | marun: worth noting we explicitly call testr from an existing (tox-created) venv for post-processing of tox-based jobs, but outside of tox's execution... so running it directly from a venv isn't too unusual for us in other contexts | 16:13 |
| marun | fungi: is that run-tox.sh in project-config? | 16:13 |
| *** jerryz has quit IRC | 16:14 | |
| clarkb | fungi: is ATC based only on changes made to master? | 16:14 |
| marun | fungi: looks like it, ok | 16:14 |
| clarkb | fungi: looking at the EOL icehouse thread and supposedly you don't get atc for stable branch maintenance? | 16:14 |
| jeblair | marun: ok, so the choices you presented were basically between two ways of providing info to that function -- a) variables b) arguments ? | 16:14 |
| fungi | marun: yep | 16:14 |
| marun | jeblair: hmmm... maybe we should be using run-tox.sh instead? | 16:15 |
| marun | jeblair: it already runs out of a venv | 16:15 |
| jeblair | marun: if that would work, then maybe so. | 16:15 |
| marun | https://github.com/openstack-infra/project-config/blob/master/jenkins/scripts/run-tox.sh#L14 | 16:15 |
| marun | oh, I remember the sticking point | 16:15 |
| *** ssam2 has joined #openstack-infra | 16:15 | |
| marun | https://github.com/openstack-infra/project-config/blob/master/jenkins/scripts/run-tox.sh#L76 | 16:15 |
| clarkb | marun: that would be a c) option :) the only thing that would have to be understood there is log scping | 16:15 |
| *** dustins_ has quit IRC | 16:15 | |
| fungi | clarkb: it's based on gerrit owners of changes merged in a specific time period to projects counted in the official set | 16:15 |
| clarkb | oh sudo checking too | 16:15 |
| marun | we'd have to make that configurale | 16:16 |
| marun | so that the sudo-requiring functional jobs wouldn't trip that check | 16:16 |
| clarkb | fungi: so any change within the window regardless of branch (feature/, master, stable/) should be included right? | 16:16 |
| fungi | clarkb: yep see https://git.openstack.org/cgit/openstack-infra/system-config/tree/tools/atc/email_stats.py | 16:16 |
| *** sarob has joined #openstack-infra | 16:16 | |
| fungi | clarkb: the challenge for stable branch maintenance is that the people may be 100% reviewer and 0% change owner | 16:17 |
| marun | clarkb: also, the permissions that tox runs as would need to be configurable iirc | 16:17 |
| fungi | clarkb: hence adding zul to the extra-atcs list in openstack/governance recently | 16:17 |
| marun | I'm not sure anymore why we need to run as different users though :/ | 16:17 |
| clarkb | fungi: right because they are backports so author isn't reset | 16:17 |
| fungi | clarkb: well, the stable branch managers are likely not doing the backports | 16:18 |
| jeblair | marun: in devstack it's so that we can revoke sudo from the user after setup (to ensure rootwrap, etc, are correctly tested) | 16:18 |
| clarkb | marun: to exercise root wrap irc | 16:18 |
| fungi | clarkb: gerrit owner is set based on who uploads the change i think, but if you're only reviewing backports and not uploading backports, you don't show up in the set of change owners | 16:18 |
| jeblair | (but retain sudo for the devstack-gate script itself) | 16:19 |
| clarkb | fungi: http://lists.openstack.org/pipermail/openstack-dev/2015-February/056457.html is the specific thread entry | 16:19 |
| marun | ah, ok. that makes sense. so run-tox.sh would have to able to change the user | 16:19 |
| fungi | clarkb: thanks, been behind on e-mail this morning | 16:19 |
| clarkb | fungi: in this particular case the claim appears to be "I have pushed backports but get no recognition" | 16:19 |
| clarkb | fungi: which may be true to stackalytics but should't be for ATC rolls | 16:19 |
| marun | jeblair, clarkb: Ok, I'll propose a patch for project-config that updates run-tox.sh to support what we need for functional testing. | 16:19 |
| clarkb | I have also read reed's ask.o.o answer and don't know how that conclusion was reached | 16:19 |
| clarkb | marun: kk | 16:20 |
| *** mpaolino has quit IRC | 16:20 | |
| marun | and if that gets accepted I'll revert the d-g processing to target only tempest | 16:20 |
| marun | clarkb, fungi, jeblair: thank you! | 16:20 |
| * reed on a call ... | 16:20 | |
| jeblair | marun, clarkb: perhaps what we really need is a refactor of the run-tox script to use functions for these tasks; so we could have unit test and functional test variants of the scripts | 16:20 |
| clarkb | reed: ya its fine we can discuss after | 16:20 |
| *** ChuckC has quit IRC | 16:21 | |
| jeblair | marun: (these scripts are basically ancient; most of the code dates from when we had 6 projects) | 16:21 |
| jeblair | maybe 4 | 16:21 |
| *** jgallard__ has quit IRC | 16:21 | |
| *** ChuckC has joined #openstack-infra | 16:21 | |
| *** sarob has quit IRC | 16:21 | |
| *** jgallard__ has joined #openstack-infra | 16:21 | |
| *** crc32 has quit IRC | 16:22 | |
| jeblair | marun: and btw, sorry for muddling things, but thank you very much for digging into this :) | 16:22 |
| *** esmute has quit IRC | 16:23 | |
| *** yfried_ is now known as yfried|afk | 16:24 | |
| *** NobodyCam has quit IRC | 16:24 | |
| *** juice has quit IRC | 16:24 | |
| *** carl_baldwin has joined #openstack-infra | 16:24 | |
| *** abhirc_ has joined #openstack-infra | 16:24 | |
| *** BadCub has quit IRC | 16:24 | |
| *** hashar has quit IRC | 16:24 | |
| *** vipul has quit IRC | 16:25 | |
| *** NobodyCam has joined #openstack-infra | 16:25 | |
| *** juice has joined #openstack-infra | 16:26 | |
| *** esmute has joined #openstack-infra | 16:26 | |
| jeblair | fungi: the precise apply job in 152508 has succeeded (it failed in previous runs without crd) | 16:26 |
| *** vipul has joined #openstack-infra | 16:26 | |
| *** ChuckC_ has joined #openstack-infra | 16:26 | |
| *** BadCub has joined #openstack-infra | 16:26 | |
| *** hashar has joined #openstack-infra | 16:26 | |
| *** yfried|afk is now known as yfried_ | 16:26 | |
| fungi | i think that's a good sign it's working as designed | 16:27 |
| jeblair | yaay! | 16:27 |
| *** abhirc has quit IRC | 16:27 | |
| *** HeOS has quit IRC | 16:28 | |
| clarkb | fungi: jeblair: did that depend on zuul cloner to get everything set up properly? | 16:28 |
| clarkb | eg g-g-p wouldn't have worked right? | 16:28 |
| marun | jeblair: I'll be happy to refactor as part of the work. :) | 16:28 |
| fungi | clarkb: well, it depends on having working integration test jobs | 16:28 |
| fungi | clarkb: so devstack-gate or zuul-cloner based currently i think | 16:29 |
| clarkb | fungi: ah right its the integration not git set up that really matters | 16:29 |
| *** chlong has quit IRC | 16:30 | |
| *** yamahata has quit IRC | 16:30 | |
| *** ChuckC has quit IRC | 16:30 | |
| jeblair | fungi: right | 16:31 |
| *** che-arne has quit IRC | 16:33 | |
| *** scheuran has quit IRC | 16:34 | |
| SpamapS | clarkb: reading backscroll.. no I have not booted the image I made just yet | 16:34 |
| *** koolhead17 has quit IRC | 16:35 | |
| *** unicell1 has joined #openstack-infra | 16:36 | |
| *** e0ne is now known as e0ne_ | 16:36 | |
| *** hashar has quit IRC | 16:37 | |
| *** cdent_ has joined #openstack-infra | 16:37 | |
| *** unicell has quit IRC | 16:37 | |
| *** unicell1 has quit IRC | 16:39 | |
| *** e0ne_ is now known as e0ne | 16:39 | |
| *** cdent has quit IRC | 16:40 | |
| *** cdent_ has quit IRC | 16:40 | |
| *** cdent has joined #openstack-infra | 16:41 | |
| *** ashleighfarnham has joined #openstack-infra | 16:42 | |
| *** juice has quit IRC | 16:43 | |
| *** NobodyCam has quit IRC | 16:43 | |
| *** jesusaurus has quit IRC | 16:43 | |
| *** esmute has quit IRC | 16:43 | |
| *** dtantsur is now known as dtantsur|afk | 16:43 | |
| *** yfried_ is now known as yfried|afk | 16:43 | |
| *** ayoung_sleep is now known as ayoung | 16:43 | |
| *** BadCub has quit IRC | 16:44 | |
| *** vipul has quit IRC | 16:44 | |
| clarkb | reed: to recap there is a claim (on a thread about EOLing icehouse) that one potential reason there is little interest in stable branch maintenance is that doing so does not grant you ATC status | 16:44 |
| reed | I saw that thread, I need to gather some details | 16:44 |
| clarkb | reed: fungi has refuted this here and my reading of the ATC generation script seems to confirm. Probably best if you or fungi respond to the list with a definitive answer | 16:44 |
| *** esmute has joined #openstack-infra | 16:45 | |
| reed | clarkb, that's my thought too, that he's wrong and fungi is right | 16:45 |
| *** BadCub has joined #openstack-infra | 16:45 | |
| *** NobodyCam has joined #openstack-infra | 16:45 | |
| clarkb | reed: I mentioned you because our ask.o.o answer about atc stuff was linked as evidence that only master counts but I cannot get that out of the ask thread when I read it | 16:45 |
| *** juice has joined #openstack-infra | 16:45 | |
| reed | clarkb, thanks, I thought my English failed there | 16:45 |
| clarkb | s/our/your/ | 16:45 |
| fungi | reed: clarkb: i've just replied on that thread and will reply on the ask.o.o entry as well | 16:46 |
| *** vipul has joined #openstack-infra | 16:46 | |
| clarkb | fungi: awesome thanks a bunch | 16:46 |
| reed | the other thing he says wrong is that analytics looks at trunk only... not sure about stackalytics but activity.o.o does all activity in the whole repo | 16:46 |
| *** jesusaurus has joined #openstack-infra | 16:46 | |
| reed | will reply in a bit | 16:46 |
| clarkb | reed: I think the analytics thing is a slight issue. basically when you backport a change the original author retains authorship | 16:46 |
| * reed stuck in some administrativia still | 16:46 | |
| clarkb | reed: so the backporter isn't credited as author. They are the change owner and committer which is different and if analytics only focuses on the author that will happen | 16:47 |
| reed | clarkb, that's a more interesting issue indeed, like counting co-author | 16:47 |
| *** doude has quit IRC | 16:47 | |
| clarkb | yup | 16:47 |
| *** shashankhegde has joined #openstack-infra | 16:48 | |
| *** bdpayne has joined #openstack-infra | 16:49 | |
| fungi | it's annoying that askbot comments are limited and you can't follow up to your comment with a second comment to increase that limit | 16:49 |
| *** ssam2 has quit IRC | 16:49 | |
| *** tiswanso has quit IRC | 16:49 | |
| openstackgerrit | Merged openstack/requirements: Update gabbi to 0.6.0 https://review.openstack.org/153301 | 16:50 |
| fungi | length-limited i mean | 16:50 |
| *** gyee has joined #openstack-infra | 16:51 | |
| *** tqtran has joined #openstack-infra | 16:51 | |
| reed | fungi, the idea is that you provide an answer instead or (with enough karma) fix someone else's answer | 16:51 |
| *** tiswanso_ has joined #openstack-infra | 16:51 | |
| reed | comments are useless | 16:51 |
| pc_m | clarkb: Can you review https://review.openstack.org/#/c/154487/? This is to restore an accidentally reverted change. | 16:52 |
| fungi | reed: agreed, except when people misinterpret comments into other contexts and there's not enough space to provide clear rebuttals to them | 16:52 |
| reed | let's clarify the answer then | 16:52 |
| openstackgerrit | Derek Higgins proposed openstack-infra/system-config: Set hp1 max-node to 0 https://review.openstack.org/154561 | 16:53 |
| fungi | reed: i think the answer is perfectly clear, and the comment thread which ensued after it is where misunderstandings are arising from | 16:53 |
| reed | exactly my point: comments are useless :) | 16:53 |
| reed | delete them :) | 16:53 |
| openstackgerrit | Derek Higgins proposed openstack-infra/system-config: Set hp1 max-servers: to 0 https://review.openstack.org/154561 | 16:53 |
| *** yfried|afk is now known as yfried_ | 16:53 | |
| *** bswartz has quit IRC | 16:53 | |
| clarkb | pc_m: done. How did those changes not conflict? or did they and the rebase was not proper? | 16:54 |
| fungi | reed: i'm pretty sure that all of kevin's misunderstandings from that are not based at all on the answer, but instead on the commentary dialogue (no doubt he googled until he found some discussion which sounded vague enough to support his assumption) | 16:54 |
| derekh | Hi, could somebody take a look at https://review.openstack.org/154561 to stop using hp1 while its failing tests | 16:54 |
| *** shashankhegde has quit IRC | 16:55 | |
| *** arxcruz has quit IRC | 16:55 | |
| clarkb | yolanda: still around? I left a couple comments on 137401 | 16:55 |
| *** ihrachyshka has quit IRC | 16:55 | |
| pc_m | clarkb: Thanks! Not sure. | 16:55 |
| pc_m | dougwig: ^^ | 16:55 |
| fungi | clarkb: i think the rebase reverted the conflict rather than accommodate it | 16:55 |
| clarkb | fungi: gotcha | 16:56 |
| *** afazekas has quit IRC | 16:56 | |
| *** coolsvap_ is now known as coolsvap | 16:56 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/puppet-project_config: Provide way to specify repo ref https://review.openstack.org/154300 | 16:58 |
| clarkb | jeblair: ^ that should make puppet lint happy | 16:58 |
| jeblair | fungi, mrmartin: 152508 passes tests :) | 16:59 |
| reed | fungi, I edited the answer to include your comment | 16:59 |
| *** otter768 has joined #openstack-infra | 16:59 | |
| *** luqas has quit IRC | 16:59 | |
| jd__ | knock knock! hello there – could we fill this https://review.openstack.org/#/admin/groups/600,members group with oslo-core at least? cc dims__ dhellmann | 17:00 |
| *** sarob has joined #openstack-infra | 17:00 | |
| fungi | reed: wfm. hopefully that will help | 17:00 |
| dims__ | you beat me jd__ | 17:00 |
| jd__ | :-) | 17:00 |
| fungi | jd__: in the pas we've just added dhellmann and let him set it up however he wants from there | 17:00 |
| fungi | jd__: dims__: which i've now done | 17:01 |
| fungi | s/pas/past | 17:01 |
| *** bswartz has joined #openstack-infra | 17:01 | |
| *** ssam2 has joined #openstack-infra | 17:02 | |
| clarkb | mordred: https://review.openstack.org/#/c/154304/ https://review.openstack.org/#/c/154305/ and https://review.openstack.org/154300 :) | 17:02 |
| jd__ | fungi: thanks | 17:03 |
| *** otter768 has quit IRC | 17:03 | |
| clarkb | mordred: the change to the puppet-ansible repo is the one I am least sure about | 17:03 |
| *** koolhead17 has joined #openstack-infra | 17:03 | |
| mordred | clarkb: yah - I have some thoughts but am in a meeting - I promise - responding is next up | 17:04 |
| dims__ | fungi: thanks! | 17:05 |
| *** e0ne has quit IRC | 17:06 | |
| *** nijaba_ is now known as nijaba | 17:06 | |
| *** yamahata has joined #openstack-infra | 17:06 | |
| openstackgerrit | Merged openstack-infra/project-config: Restore lost changes to neutron-vpnaas https://review.openstack.org/154487 | 17:06 |
| clarkb | nibalizer: can you explain how the thing in 153823 makes testing harder for non rooters? | 17:06 |
| *** jistr has quit IRC | 17:07 | |
| *** jcoufal has quit IRC | 17:07 | |
| clarkb | nibalizer: is it because puppetmaster is set? When I test locally I always use puppet apply | 17:07 |
| clarkb | nibalizer: so not sure I understand the issue there | 17:07 |
| clarkb | derekh: +2 | 17:08 |
| derekh | clarkb, thanks | 17:08 |
| *** yamahata has quit IRC | 17:09 | |
| *** gokrokve_ has joined #openstack-infra | 17:09 | |
| nibalizer | clarkb: it creates a lock between site.pp and that file | 17:09 |
| openstackgerrit | Merged openstack-infra/system-config: Zuul status: Use a grey icon for non-live changes https://review.openstack.org/154232 | 17:09 |
| nibalizer | its better now that its certname instead of fqdn | 17:09 |
| clarkb | nibalizer: I still do not follow | 17:09 |
| clarkb | nibalizer: a lock? | 17:09 |
| *** yamahata has joined #openstack-infra | 17:09 | |
| *** MaxV has quit IRC | 17:09 | |
| clarkb | nibalizer: I don't doubt there is something that makes live difficult I just have no understanding of it with my use of puppet apply | 17:09 |
| mordred | just so that everyone is aware - the only thing worse than systemd is a system where SOME things are in systemd and SOME things are not | 17:10 |
| nibalizer | yea puppet apply works | 17:10 |
| *** ChuckC has joined #openstack-infra | 17:10 | |
| *** MaxV has joined #openstack-infra | 17:10 | |
| nibalizer | i prefer to test with a master | 17:10 |
| nibalizer | because then i can have my master set up and massage code there, then spin up and down clients as I want | 17:11 |
| clarkb | assuming you have a local master how does certname make life difficult? you can use the same certnames that we use | 17:11 |
| clarkb | then it should just work (tm) right? | 17:11 |
| nibalizer | ya i think you're right, there might not actually be a problem now | 17:12 |
| nibalizer | my problems are all from when that was fqdn | 17:12 |
| *** gokrokve has quit IRC | 17:12 | |
| nibalizer | i honestly haven't spun up my infra testing environment in some time | 17:13 |
| nibalizer | been busy with other stuff | 17:13 |
| *** Chuck__ has joined #openstack-infra | 17:13 | |
| *** ChuckC_ has quit IRC | 17:14 | |
| nibalizer | also that line is ... weird | 17:14 |
| nibalizer | i guess its okay, but its asserting what is already true, if that makes sense? | 17:14 |
| *** MaxV has quit IRC | 17:15 | |
| dhellmann | fungi: could you add me to oslo-versionedobjects-release so I can update that one, too, please? https://review.openstack.org/#/admin/groups/601,members | 17:15 |
| *** dustins has joined #openstack-infra | 17:15 | |
| fungi | dhellmann: done | 17:15 |
| *** jkraj has quit IRC | 17:15 | |
| dhellmann | fungi: thanks! | 17:16 |
| *** ChuckC has quit IRC | 17:17 | |
| *** kgiusti has quit IRC | 17:19 | |
| *** hashar has joined #openstack-infra | 17:19 | |
| openstackgerrit | Sergey Skripnick proposed openstack-infra/project-config: Add rally-ci group with label-Verified permission https://review.openstack.org/154532 | 17:20 |
| *** amotoki has quit IRC | 17:21 | |
| *** jpich has quit IRC | 17:22 | |
| *** unicell has joined #openstack-infra | 17:22 | |
| *** wznoinsk_ has joined #openstack-infra | 17:22 | |
| *** coolsvap is now known as coolsvap_ | 17:23 | |
| *** juice has quit IRC | 17:23 | |
| *** wznoinsk is now known as wznoinski__ | 17:25 | |
| *** esmute has quit IRC | 17:25 | |
| *** wznoinski__ is now known as wznoinsk__ | 17:25 | |
| *** esmute has joined #openstack-infra | 17:25 | |
| asselin_ | anyone seen this? not able to start nodepool: http://paste.openstack.org/show/170829/ | 17:25 |
| *** bswartz has quit IRC | 17:25 | |
| *** juice has joined #openstack-infra | 17:25 | |
| zaro | morning | 17:26 |
| *** wznoinsk_ is now known as wznoinsk | 17:27 | |
| clarkb | nibalizer ya the first time we run we pass the certname as an arg iirc | 17:28 |
| clarkb | so its true but not otherwise configured. this configures it | 17:28 |
| clarkb | asselin_ upgrade your version of lockfile | 17:29 |
| *** mpaolino has joined #openstack-infra | 17:29 | |
| *** wznoinsk has quit IRC | 17:29 | |
| *** jedimike has quit IRC | 17:29 | |
| mrmartin | jeblair: 152508 is testing together with dependent patch? | 17:30 |
| pleia2 | good morning | 17:30 |
| openstackgerrit | Merged openstack-infra/storyboard: Fix notifications tests docstrings referencing map_resource method https://review.openstack.org/154402 | 17:31 |
| *** wznoinsk has joined #openstack-infra | 17:31 | |
| *** annegentle has quit IRC | 17:32 | |
| asselin_ | clarkb, thanks | 17:33 |
| clarkb | asselin_ fix is in review. we were depending on lockfile instead of having pytjon-daemon pull it in | 17:33 |
| clarkb | this confused pip so fix is remove lockfile from our dep list | 17:33 |
| *** tiswanso_ has quit IRC | 17:33 | |
| asselin_ | ok i see | 17:34 |
| *** sputnik13 has joined #openstack-infra | 17:34 | |
| *** wznoinsk has quit IRC | 17:34 | |
| *** ildikov has quit IRC | 17:34 | |
| *** wznoinsk has joined #openstack-infra | 17:34 | |
| *** tiswanso has joined #openstack-infra | 17:34 | |
| *** yfried_ is now known as yfried|afk | 17:34 | |
| *** jlibosva has quit IRC | 17:35 | |
| jeblair | mrmartin: yep! | 17:35 |
| mrmartin | awesome! | 17:35 |
| *** ianw has quit IRC | 17:39 | |
| *** emagana has joined #openstack-infra | 17:40 | |
| fungi | just to bring over the discussion from #openstack-qa, it looks like we will need to manually merge stable/juno global requirements pin 151770 since it won't pass grenade jobs without stable/juno grenade fix 153702 (which in turn depends on the reqs pin being in place) | 17:41 |
| fungi | mtreinish: jogo: ^ | 17:41 |
| fungi | any objection to me exercising gerrit superpowers to merge 151770 now? | 17:41 |
| *** lnxnut_ has joined #openstack-infra | 17:41 | |
| *** emagana_ has joined #openstack-infra | 17:41 | |
| fungi | adam_g: ^ | 17:41 |
| *** bswartz has joined #openstack-infra | 17:42 | |
| *** mpaolino has quit IRC | 17:42 | |
| *** esmute has quit IRC | 17:42 | |
| *** BadCub has quit IRC | 17:43 | |
| *** vipul has quit IRC | 17:43 | |
| *** juice has quit IRC | 17:43 | |
| *** jesusaurus has quit IRC | 17:44 | |
| mtreinish | fungi: I support you excercising you powers | 17:44 |
| *** esmute has joined #openstack-infra | 17:44 | |
| * mtreinish shields his eyes | 17:44 | |
| *** lnxnut has quit IRC | 17:44 | |
| *** shashankhegde has joined #openstack-infra | 17:44 | |
| *** juice has joined #openstack-infra | 17:44 | |
| *** emagana has quit IRC | 17:45 | |
| *** BadCub has joined #openstack-infra | 17:45 | |
| * fungi tries to use his powers only for good, less often for personal amusement | 17:45 | |
| *** vipul has joined #openstack-infra | 17:45 | |
| *** koolhead_ has joined #openstack-infra | 17:46 | |
| *** ghostpl has quit IRC | 17:46 | |
| *** jesusaurus has joined #openstack-infra | 17:46 | |
| *** Ryan_Lane has joined #openstack-infra | 17:46 | |
| *** ashleighfarnham has quit IRC | 17:46 | |
| *** Chuck__ is now known as ChuckC | 17:48 | |
| *** koolhead17 has quit IRC | 17:48 | |
| *** ashleighfarnham has joined #openstack-infra | 17:49 | |
| clarkb | fungi the superhero we deserve | 17:49 |
| clarkb | or is it we dont deserve? english is hard | 17:50 |
| *** e0ne has joined #openstack-infra | 17:51 | |
| openstackgerrit | Merged openstack-infra/project-config: gnocchi: install the git repo https://review.openstack.org/154398 | 17:52 |
| *** melwitt has joined #openstack-infra | 17:52 | |
| fungi | hearing no objections, i don the cape | 17:53 |
| *** avozza is now known as zz_avozza | 17:54 | |
| fungi | mtreinish: jogo: adam_g: done. should be safe to put 153702 through normally now | 17:54 |
| jogo | fungi: thanks | 17:55 |
| *** dhritishikhar_ has quit IRC | 17:55 | |
| *** MaxV has joined #openstack-infra | 17:56 | |
| *** bhuvan has joined #openstack-infra | 17:56 | |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/storyboard: Removed unsupported python versions from setup.cfg https://review.openstack.org/154582 | 17:58 |
| *** changbl has joined #openstack-infra | 17:58 | |
| *** derekh has quit IRC | 17:59 | |
| *** wznoinsk_mobl has joined #openstack-infra | 18:01 | |
| *** dannywilson has joined #openstack-infra | 18:01 | |
| *** dannywilson has quit IRC | 18:02 | |
| *** signed8bit has quit IRC | 18:02 | |
| *** dannywilson has joined #openstack-infra | 18:02 | |
| *** _nadya_ has quit IRC | 18:03 | |
| david-lyle | jeblair, clarkb or fungi, some time ago the previous owner of the openstack-horizon channel added a bunch of flags on that room, I was not so lucky, can one of you please give me the operator status at least | 18:04 |
| openstackgerrit | Merged openstack-infra/project-config: Adds networking-hyperv in StackForge https://review.openstack.org/153691 | 18:04 |
| david-lyle | *flags for you on that room | 18:04 |
| *** yfried|afk is now known as yfried_ | 18:04 | |
| *** sdake has quit IRC | 18:04 | |
| *** wznoinsk has quit IRC | 18:04 | |
| *** ashleighfarnham has quit IRC | 18:05 | |
| fungi | david-lyle: sure, freenode confirms you're logged in, so i'm pretty sure i'm talking to the real you ;) | 18:05 |
| *** shashankhegde has quit IRC | 18:05 | |
| david-lyle | fungi: most of the time | 18:06 |
| *** patrickeast has joined #openstack-infra | 18:06 | |
| david-lyle | fungi: thank you | 18:06 |
| *** gokrokve_ has quit IRC | 18:07 | |
| fungi | david-lyle: i added the same flags for you that our general irc operators have on that channel | 18:08 |
| fungi | david-lyle: though you had a couple of other flags already (O and V) which i left on as well | 18:09 |
| openstackgerrit | Lance Bragstad proposed openstack/requirements: Add python-keyczar to global-requirements.py https://review.openstack.org/154590 | 18:09 |
| david-lyle | fungi, perfect. 3 releases in, I can change the channel topic, progress as promised | 18:09 |
| *** harlowja has joined #openstack-infra | 18:10 | |
| fungi | coming along nicely! ;) | 18:10 |
| *** kgiusti has joined #openstack-infra | 18:11 | |
| *** wznoinsk_mobl has quit IRC | 18:11 | |
| *** hashar has quit IRC | 18:11 | |
| *** andreykurilin_ has quit IRC | 18:12 | |
| *** thedodd has quit IRC | 18:13 | |
| *** abhirc_ has quit IRC | 18:14 | |
| *** abhirc has joined #openstack-infra | 18:14 | |
| *** tsg has joined #openstack-infra | 18:14 | |
| *** jaypipes has joined #openstack-infra | 18:15 | |
| *** ihrachyshka has joined #openstack-infra | 18:15 | |
| *** dalgaaf has quit IRC | 18:17 | |
| *** achanda has joined #openstack-infra | 18:18 | |
| *** vigneshvar has joined #openstack-infra | 18:19 | |
| *** thedodd has joined #openstack-infra | 18:19 | |
| clarkb | fungi k9 exploded on one of your messages to the eol threads. so weird | 18:20 |
| fungi | asploded how? | 18:20 |
| * fungi tries to stick to plain text e-mail... i don't even think i've crypto-signed any messages to that thread | 18:21 | |
| clarkb | (no subject) and no content but apparently enough header to associate with the thread | 18:21 |
| *** openstackgerrit has quit IRC | 18:22 | |
| *** openstackgerrit has joined #openstack-infra | 18:22 | |
| *** esmute has quit IRC | 18:23 | |
| *** thedodd has quit IRC | 18:24 | |
| *** juice has quit IRC | 18:24 | |
| *** yfried_ is now known as yfried|afk | 18:24 | |
| *** thedodd has joined #openstack-infra | 18:25 | |
| *** Ala has quit IRC | 18:26 | |
| mordred | jeblair, clarkb: ok. I have ubuntu working on rackspace but can't get centos working. for nodepool-dib that's obviously not a big deal | 18:26 |
| *** pelix has quit IRC | 18:26 | |
| mordred | HOWEVER - the reason I can't get it working is systemd | 18:26 |
| mordred | so there is a problem to solve at some point if we want to add fedora or debian nodes | 18:26 |
| mordred | that said - centos7 has a weird hybrid thing going on | 18:26 |
| *** esmute has joined #openstack-infra | 18:26 | |
| mordred | so it's quite possible that it'll be less bonghits on debian/fedora | 18:26 |
| *** juice has joined #openstack-infra | 18:26 | |
| mordred | so I might try them next | 18:27 |
| mordred | (I don't want to declare that it's usable knowing that there is a looming broken thing) | 18:27 |
| greghaynes | I know some of the tripleo rhers were doing centos7 work on dib... | 18:27 |
| greghaynes | so in theory it should work | 18:27 |
| *** hdd has quit IRC | 18:28 | |
| *** jp_at_hp has quit IRC | 18:29 | |
| SpamapS | I think it's more that centos7/rhel7's networking works very differently than fedora/debian's networking with systemd. | 18:29 |
| greghaynes | aye, fun | 18:29 |
| mordred | greghaynes: this is also about getting network info from the rax config-drive ... so it's an extra special beasty | 18:30 |
| mordred | centos7 itself is working just fine | 18:30 |
| mordred | also - centos7 does not have "ifdown -a; ifup -a" | 18:30 |
| mordred | you have to specify interfaces | 18:30 |
| *** crc32 has joined #openstack-infra | 18:31 | |
| greghaynes | Ive been running into an error when making nodepool trusty images where prepare_tempest_testrepository.py fails with chown: cannot access ‘/opt/git/openstack/tempest/preseed-streams’: No such file or directory | 18:31 |
| greghaynes | AIUI preseed streams is a testr thing... theres a severe lack of debug info though | 18:31 |
| *** shashankhegde has joined #openstack-infra | 18:32 | |
| SpamapS | mordred: 'service networking restart' doesn't work? | 18:32 |
| SpamapS | that used to be a thing anyway | 18:32 |
| *** jgallard__ has quit IRC | 18:33 | |
| *** tsg has quit IRC | 18:35 | |
| openstackgerrit | MORITA Kazutaka proposed openstack-infra/project-config: Add test job for sheepdog devstack plugin https://review.openstack.org/154605 | 18:36 |
| *** crc32 has quit IRC | 18:38 | |
| openstackgerrit | gordon chung proposed openstack/requirements: add ceilometermiddleware https://review.openstack.org/154607 | 18:38 |
| clarkb | greghaynes mtreinish | 18:39 |
| pleia2 | clarkb: so I'm back to trying to figure out what the intent is for all these devstack-gate jobs, what is the goal here, vote on the -icehouse but not voting on -juno? back in september we were voting on icehouse but not havana: https://review.openstack.org/#/c/122885/ (see Jenkins' check build on patchset 2) | 18:40 |
| *** yfried|afk is now known as yfried_ | 18:40 | |
| fungi | clarkb: the three messages i sent related to those threads are http://lists.openstack.org/pipermail/openstack-dev/2015-February/056465.html 056472.html and 056476.html any idea which one of those is broken in k9? | 18:40 |
| * greghaynes summons an mtreinish | 18:40 | |
| clarkb | pleia2 I think tempest wants them voting and d-g does not | 18:40 |
| clarkb | pleia2 so likely we need 2 sets of jobs | 18:41 |
| pleia2 | clarkb: for both icehouse and juno, or? | 18:41 |
| clarkb | fungi the one about stable tempest helping defcore | 18:41 |
| clarkb | pleia2 ya for both branches | 18:41 |
| pleia2 | ok, thanks, I'll see where this goes and change it again if we learn differently | 18:42 |
| fungi | greghaynes: make sure you didn't accidentally run it once as root and pollute the ownership of stuff in your cache | 18:42 |
| *** ssam2 has quit IRC | 18:43 | |
| *** BadCub has quit IRC | 18:43 | |
| fungi | greghaynes: we were getting errors which looked basically like that some of the time which turned out to be local cache ownership issues | 18:43 |
| greghaynes | fungi: this is as the result of a nodepool dib image build, so I hope tats not an issue ;) | 18:43 |
| greghaynes | oh | 18:43 |
| *** vipul has quit IRC | 18:43 | |
| *** NobodyCam has quit IRC | 18:43 | |
| *** BadCub has joined #openstack-infra | 18:44 | |
| fungi | greghaynes: if someone previously did a 'sudo nodepool image-create foo' then parts of the cache could be root-owned and inaccessible to the nodepool user which nodepool is trying that as when run from its periodic scheduler | 18:44 |
| *** jesusaurus has quit IRC | 18:44 | |
| *** esmute has quit IRC | 18:44 | |
| greghaynes | aye :( | 18:44 |
| *** vipul has joined #openstack-infra | 18:44 | |
| *** NobodyCam has joined #openstack-infra | 18:44 | |
| fungi | greghaynes: try to chown -R nodepool:nodepool /your/cache/dir | 18:44 |
| *** Sukhdev has joined #openstack-infra | 18:45 | |
| fungi | and see if that fixes it | 18:45 |
| pc_m | clarkb: ping | 18:45 |
| *** esmute has joined #openstack-infra | 18:45 | |
| clarkb | pong | 18:45 |
| greghaynes | wow, there is definitely things in the cache not owned by me | 18:45 |
| greghaynes | thats a bad sign | 18:45 |
| pc_m | clarkb: Had a question on coverage testing... Is there a way to collect coverage info for dsvm-functional modules too? | 18:45 |
| *** mgariepy has left #openstack-infra | 18:46 | |
| *** gokrokve has joined #openstack-infra | 18:47 | |
| pc_m | clarkb: I ran "tox -e cover" and don't have an omit clause. The functional modules all show 0 lines coverage. | 18:47 |
| clarkb | pc_m run your tests and services under coverage | 18:47 |
| clarkb | tox -ecover is purely for whatever is run by tox -epy27 | 18:48 |
| pc_m | clarkb: Any pointers on how that is done? link? | 18:48 |
| clarkb | pc_m http://nedbatchelder.com/code/coverage/ is the coverage docs | 18:49 |
| pc_m | clarkb: thanks! | 18:49 |
| clarkb | but basically instead of python somescript you run coverage somescript | 18:49 |
| fungi | pc_m: you might want to engage #openstack-qa but in general the topic of trying to get coverage from functional/integration tests has come up many times, several solutions have been tried with a variety of trade-offs, and is generally non-trivial | 18:49 |
| pc_m | fungi: Ah, good to know. I was wondering how to run the tests for those functional modules. | 18:50 |
| *** markus_z has quit IRC | 18:51 | |
| pc_m | fungi: Is there any documentation on the methods that you know of? | 18:51 |
| clarkb | I am going to miss first bit of meeting looks like. errand is running long | 18:52 |
| fungi | pc_m: for unit tests it's easier because execution happens in a sort of rarified environment, but threading is a major challenge | 18:52 |
| fungi | pc_m: looks like there may be some helpful hints in https://bitbucket.org/ned/coveragepy/issue/149 | 18:54 |
| pc_m | fungi: cool | 18:55 |
| fungi | pc_m: anyway my memory's fuzzy but i know the qa team have discussed it in prior summit sessions and some of them likely recall the specific challenges | 18:56 |
| *** dmsimard has quit IRC | 18:56 | |
| jhesketh | Morning | 18:56 |
| anteaya | morning jhesketh | 18:57 |
| mtreinish | greghaynes: what's up? | 18:58 |
| fungi | pleia2: have a change proposed yet for adding your user to openstack_project::base and our global site manifest? | 18:59 |
| *** tsg has joined #openstack-infra | 18:59 | |
| *** _nadya_ has joined #openstack-infra | 18:59 | |
| morganfainberg | anteaya, FYI our meeting ran short - i know you use us as a reminder for -infra | 18:59 |
| *** ianw has joined #openstack-infra | 18:59 | |
| morganfainberg | anteaya, ;) | 18:59 |
| fungi | meeting time | 18:59 |
| fungi | in 10 secs | 18:59 |
| anteaya | morganfainberg: :) I saw you earlier | 19:00 |
| *** otter768 has joined #openstack-infra | 19:00 | |
| *** emagana_ has quit IRC | 19:00 | |
| anteaya | morganfainberg: and thanks | 19:00 |
| *** wznoinsk_mobl has joined #openstack-infra | 19:00 | |
| pleia2 | fungi: nope! | 19:00 |
| *** gyee has quit IRC | 19:00 | |
| openstackgerrit | Khai Do proposed openstack-infra/system-config: Add the luanchpadlib OS package to review-dev.o.o https://review.openstack.org/154612 | 19:00 |
| *** esmute has quit IRC | 19:01 | |
| fungi | pleia2: i guess that's the next item for your to do list ;) | 19:01 |
| *** tiswanso has quit IRC | 19:01 | |
| pleia2 | fungi: sure thing :) | 19:01 |
| mtreinish | greghaynes: oh, you're hitting issues with the preseed-testrepository script in nodepool | 19:01 |
| mtreinish | do you have a stack trace | 19:01 |
| greghaynes | mtreinish: hey, yep | 19:02 |
| greghaynes | mtreinish: no stack trace | 19:02 |
| *** vipul has quit IRC | 19:02 | |
| *** NobodyCam has quit IRC | 19:02 | |
| *** BadCub has quit IRC | 19:02 | |
| *** tiswanso_ has joined #openstack-infra | 19:02 | |
| greghaynes | mtreinish: it fails on chowning that dir because it doesnt exist, the prepare_tempest_repository script I assume is what should make it? | 19:02 |
| *** dims__ has quit IRC | 19:02 | |
| *** dims__ has joined #openstack-infra | 19:03 | |
| *** ashleighfarnham has joined #openstack-infra | 19:03 | |
| mtreinish | greghaynes: yeah the script makes it: http://git.openstack.org/cgit/openstack-infra/project-config/tree/nodepool/scripts/prepare_tempest_testrepository.py#n52 | 19:03 |
| *** esmute has joined #openstack-infra | 19:03 | |
| *** NobodyCam has joined #openstack-infra | 19:03 | |
| *** ivar-lazzaro has joined #openstack-infra | 19:03 | |
| *** dims_ has joined #openstack-infra | 19:04 | |
| greghaynes | ah! I notice there is not checking of return code there | 19:04 |
| * greghaynes adds this | 19:04 | |
| *** BadCub has joined #openstack-infra | 19:04 | |
| *** bradjones has joined #openstack-infra | 19:04 | |
| *** ashleighfarnham has quit IRC | 19:04 | |
| *** otter768 has quit IRC | 19:04 | |
| *** vipul has joined #openstack-infra | 19:05 | |
| *** ivar-lazzaro has quit IRC | 19:05 | |
| *** tobiash_ has joined #openstack-infra | 19:06 | |
| *** ivar-lazzaro has joined #openstack-infra | 19:06 | |
| *** tobiash has quit IRC | 19:06 | |
| fungi | psh, error handling... who needs that?!? | 19:06 |
| mordred | fungi: NOBODY | 19:06 |
| *** wznoinsk has joined #openstack-infra | 19:07 | |
| *** dims__ has quit IRC | 19:07 | |
| mtreinish | greghaynes: is there a return code for os.mkdir, I thought it just raised an IOError | 19:08 |
| greghaynes | you may be right, is that all that is happening with that call? | 19:08 |
| *** dmorita has joined #openstack-infra | 19:09 | |
| greghaynes | anyhow, well find out momentarially when my build fails :) | 19:09 |
| openstackgerrit | Steve Martinelli proposed openstack-infra/devstack-gate: Add oslo.policy to devstack-vm-gate-wrap.sh https://review.openstack.org/154615 | 19:09 |
| *** _nadya_ has quit IRC | 19:09 | |
| *** ashleighfarnham has joined #openstack-infra | 19:09 | |
| *** wznoinsk_mobl has quit IRC | 19:10 | |
| sdague | jeblair: so I a funny thing happens with the depends on - http://status.openstack.org/zuul/ | 19:10 |
| sdague | see 154216,3 and 154575,2 in that list | 19:11 |
| *** EmilienM is now known as EmilienM|afk | 19:11 | |
| *** jesusaurus has joined #openstack-infra | 19:11 | |
| fungi | sdague: i think https://review.openstack.org/154232 hasn't taken effect yet to make that a little more obvious | 19:12 |
| yolanda | clarkb, i answered you as well | 19:12 |
| yolanda | i want to talk with you about the testing | 19:12 |
| *** tiswanso_ has quit IRC | 19:12 | |
| *** shashankhegde has quit IRC | 19:13 | |
| sdague | fungi: so, not quite. 154575,2 shows up as both a live and non-live change | 19:13 |
| fungi | sdague: oh, actually after i force-reloaded my cache of the javascript for that updated | 19:13 |
| fungi | sdague: yeah, it's both a live and non-live change | 19:14 |
| sdague | shouldn't it only be one? | 19:14 |
| fungi | sdague: it's being tested itself as a live change, but also appears as a non-live change applied while testing 154216,3 | 19:14 |
| sdague | ok, I guess I assumed that it would only show up once | 19:15 |
| fungi | sdague: i don't think the goal was to collapse those, but maybe that's a good improvement to consider | 19:15 |
| jeblair | sdague: yeah, that might be slightly confusing from ux, but is sort of comforting when you think about what might happen if the parent fails; at least this makes it clear that no matter the outcome, the child will still be tested | 19:15 |
| jhesketh | sdague: http://logs.openstack.org/92/154292/1/check/check-dg-tempest-dsvm-full/ddceb0d/index.html | 19:16 |
| jhesketh | sdague: so I think the most obvious thing that is missing is the documentation | 19:16 |
| sdague | jeblair: yep, I get it now. I might try to mock up some different visual metaphor for this so that it doesn't confuse people | 19:16 |
| *** tiswanso_ has joined #openstack-infra | 19:16 | |
| sdague | jhesketh: are all those changes coming from swift? | 19:17 |
| sdague | like all the logs? | 19:17 |
| jeblair | yeah. also, it looks like grey dots are in now, so if you don't have those, you should reload | 19:17 |
| jhesketh | sdague: the link I sent you is since index.html isn't stored locally. Once you browse to a log, you'll probably need to append ?source=swift | 19:17 |
| greghaynes | mtreinish: success! now I have a traceback :) | 19:17 |
| greghaynes | tyty | 19:17 |
| openstackgerrit | Ian Wienand proposed openstack-infra/nodepool: Add generate-log-config command https://review.openstack.org/153904 | 19:17 |
| mtreinish | greghaynes: what's it say? | 19:18 |
| *** dims__ has joined #openstack-infra | 19:18 | |
| *** shashankhegde has joined #openstack-infra | 19:18 | |
| *** mrunge has quit IRC | 19:18 | |
| greghaynes | mtreinish: http://paste.ubuntu.com/10162632/ | 19:19 |
| sdague | jhesketh: is the index static or or dynamically generated, I forget? | 19:20 |
| jhesketh | sdague: static | 19:20 |
| *** elliottk has quit IRC | 19:20 | |
| sdague | do you have an instance of - http://logs.openstack.org/92/154292/1/check/check-dg-tempest-dsvm-full/ddceb0d/logs/ that's off swift? | 19:21 |
| *** dims_ has quit IRC | 19:21 | |
| greghaynes | mtreinish: gonna need some debugging, but I at least know where to look now | 19:22 |
| mtreinish | greghaynes: isn't that because of the paren around stats, out? if it was a tuple response you wouldn't need them | 19:22 |
| mtreinish | also that looks like the script is out of date | 19:22 |
| mtreinish | because it's trying to run testr as part of the image create (which doesn't work for a number of reasons) | 19:23 |
| greghaynes | ah | 19:23 |
| jhesketh | sdague: yes, append index.html: http://logs.openstack.org/92/154292/1/check/check-dg-tempest-dsvm-full/ddceb0d/logs/index.html | 19:23 |
| *** esmute has quit IRC | 19:23 | |
| mtreinish | greghaynes: https://review.openstack.org/#/c/151732/ | 19:23 |
| greghaynes | you are correct, I am out of date | 19:24 |
| jhesketh | sdague: which will happen automatically if the local copy isn't there. Also will be slightly faster if you could please see my comments on https://review.openstack.org/#/c/141260/\ | 19:24 |
| clarkb | yolanda we should build images in tests the same way we build in prod | 19:24 |
| patrickeast | has anyone run into problems with python-daemon 2.0.5? i updated my zuul and nodepool nodes today and ran into errors like http://paste.openstack.org/show/170892/, i saw this change https://github.com/openstack-infra/zuul/commit/7e11bd727606afa5d55cddfa07439ae646e0f330 and tried to roll it back to 2.0.4 and things work better now | 19:25 |
| patrickeast | is this a known issue? | 19:25 |
| yolanda | clarkb, what do you mean? about the image testing spec? | 19:25 |
| *** esmute has joined #openstack-infra | 19:26 | |
| sdague | jhesketh: ok, so feedback. I find the icons extremely helpful in the apache view, the text content type just feels busy. K / M vs. KiB MiB would be nice to fix. And the size column needs right justification because it's a list of numbers. | 19:27 |
| clarkb | yolanda just that we shouldnt special case the fake image builds that way | 19:27 |
| asselin__ | patrickeast, <clarkb> asselin_ upgrade your version of lockfile | 19:28 |
| *** andreykurilin_ has joined #openstack-infra | 19:28 | |
| asselin__ | patrickeast, see the scrollback | 19:28 |
| clarkb | yolanda can you have the tests poll the db? | 19:28 |
| yolanda | ah, clarkb, i know, i added a comment, because if i remove builds from the taskmanager, then tests fail | 19:28 |
| yolanda | didn't want to update the testing behaviour | 19:28 |
| yolanda | so that's why i wanted feedback | 19:28 |
| jhesketh | sdague: okay, that shouldn't be too difficult | 19:28 |
| patrickeast | asselin__: awesome, thanks! | 19:29 |
| yolanda | so it's fine for me to poll database, but then it will be just an special case for it? or shall we update tests generally? | 19:29 |
| clarkb | yes tests will have to be updated | 19:29 |
| sdague | jhesketh: the help text, might be nice to add, though honestly I don't know how many folks are really using it | 19:29 |
| sdague | so I'd say let people complain | 19:29 |
| fungi | patrickeast: asselin__: proposed fix for that is https://review.openstack.org/148252 | 19:30 |
| fungi | which should cause daemon to pull in an appropriately new lockfile release on its own | 19:30 |
| sdague | or... can we enable server side includes and just pull it in that way? | 19:30 |
| yolanda | clarkb, ok.. so hall we create a different change for that? not sure if we want to mess it | 19:30 |
| yolanda | first update tests to poll database instead of relying on taskmanager in an independent change | 19:30 |
| patrickeast | fungi: oo nice | 19:30 |
| clarkb | no I think it should be part of the change that changes image building | 19:31 |
| jhesketh | sdague: I think it makes more sense to include the help text staticly so that as the logs/docs change the old text is still relevent | 19:31 |
| mtreinish | sdague: ah, all that time I spent decoding the filenames is going to disappear... | 19:31 |
| clarkb | since they are couples | 19:31 |
| asselin__ | fungi, wierd, my issue was with nodepool...perhaps another similar fix is needed? | 19:31 |
| *** andreykurilin_ has quit IRC | 19:31 | |
| patrickeast | i saw the same problem with nodepool and zuul | 19:31 |
| yolanda | clarkb , ok, i'm fine with it.. i try to send changes in smallest commits as possible, but this is related | 19:31 |
| fungi | asselin__: yep. also jenkins-job-builder. i think we patched it in jjb already but not sure about nodepool | 19:31 |
| sdague | jhesketh: I guess, part of the reason it was nice to have it dynamic is that it let us be helpful with older logs as well | 19:31 |
| yolanda | let's see if i have some time this week to work on it | 19:31 |
| sdague | when we realized it was less clear | 19:31 |
| asselin__ | fungi, we're going to get a merge conflict fyi. not sure the status of this: https://review.openstack.org/#/c/149336/ | 19:32 |
| *** Guest92220 has joined #openstack-infra | 19:32 | |
| jhesketh | sdague: sure, but the older logs have it now and will continue to since they'll still be fetched from disk. Newer logs will all have the help text if we turn it on soon enough | 19:32 |
| sdague | right, but that does get updated, and typically when it does, we want to post date it | 19:33 |
| sdague | also, it means we can use the same text files as on the static server | 19:33 |
| sdague | unless we have a mechanism to bulk rebuild the index files | 19:33 |
| fungi | asselin__: indeed. one or the other can rebase when the time comes | 19:34 |
| *** samueldmq is now known as samueldmq-away | 19:34 | |
| jhesketh | sdague: what if the log format changes and we want to update the help text in a way that doesn't apply to older logs? | 19:35 |
| jhesketh | my feeling is that by the time we update the help text people aren't interested in older logs anyway | 19:35 |
| *** maxbit has joined #openstack-infra | 19:40 | |
| openstackgerrit | Matt Riedemann proposed openstack-infra/project-config: Make gate-ceilometer-python34 only run on master branch https://review.openstack.org/154631 | 19:41 |
| mriedem | fungi: mtreinish: ^ | 19:41 |
| mtreinish | mriedem: +1 | 19:42 |
| clarkb | yolanda was this a problem with earlier patchsets? | 19:42 |
| yolanda | no, saw it in the latest one | 19:43 |
| *** NobodyCam has quit IRC | 19:43 | |
| sdague | jhesketh: honestly, that documentation tends to lag by months behind file changes | 19:43 |
| sdague | so the opposite has always been true | 19:43 |
| *** BadCub has quit IRC | 19:44 | |
| yolanda | it was waiting forever for image to be created, and if i moved it to task manager it was finding it properly | 19:44 |
| *** NobodyCam has joined #openstack-infra | 19:44 | |
| clarkb | yolanda maybe we should sort that out? perhaps that indicates older patchsets might work better? | 19:44 |
| *** BadCub has joined #openstack-infra | 19:44 | |
| yolanda | clarkb, didn't did so much, lack of time so i did a small rebase... but this was quite a significant change, a new method added, so it may affect test behaviour in some way | 19:45 |
| *** VijayTripathi has joined #openstack-infra | 19:45 | |
| jhesketh | sdague: fair, but the real fix there would be not to lag so much? (I know, much easier said than done) | 19:46 |
| yolanda | but you see, tests passed in previous changes, and i didn't update tests | 19:46 |
| mtreinish | jhesketh: yeah I mean jogo and I only went through and updated things after a ML thread about how the help was wrong | 19:46 |
| mtreinish | it doesn't really get updated otherwise | 19:46 |
| *** dprince has quit IRC | 19:47 | |
| clarkb | yolanda right thats why I am wondering if older versions might be better? I will run some local testing in a bit | 19:47 |
| yolanda | clarkb, nice, sorry to don't help so much but this week is impossible for me | 19:48 |
| clarkb | its ok just making sure I understand what is going on :) | 19:48 |
| yolanda | i'd like to be more involved in nodepool, so please share what you get | 19:48 |
| clarkb | will do | 19:48 |
| yolanda | cool | 19:49 |
| sdague | so is this just a red herring - http://logs.openstack.org/94/152994/6/check/check-devstack-dsvm-updown/eaba588/console.html#_2015-02-10_17_16_36_476 ? | 19:52 |
| openstackgerrit | Merged openstack-infra/reviewstats: Add tchaypo as core https://review.openstack.org/150332 | 19:53 |
| mtreinish | sdague: the job passed? was that not supposed to happen? | 19:54 |
| openstackgerrit | Merged openstack-infra/reviewstats: Update Heat's core team https://review.openstack.org/150750 | 19:54 |
| sdague | mtreinish: the job passed | 19:54 |
| sdague | but that output looks like it would be an error | 19:55 |
| sdague | I'm curious if we just need to fix a d-g thing | 19:55 |
| *** baoli has quit IRC | 19:57 | |
| tchaypo | ironic. | 19:58 |
| openstackgerrit | Andrey Pavlov proposed openstack-infra/project-config: Ajust gating jobs for ec2-api project https://review.openstack.org/154640 | 19:59 |
| *** Guest92220 is now known as annegentle | 20:01 | |
| tristanC | Hello folks, I wanted to talk during the meeting about that 'include: layout-file' feature presented here: https://review.openstack.org/#/c/152290/. What is the way forward to get that feature accepted ? | 20:01 |
| *** vipul has quit IRC | 20:01 | |
| jeblair | tristanC: wait for someone to review it | 20:01 |
| jhesketh | tristanC: I just had a look and will review it shortly | 20:01 |
| tristanC | oh alright, great :) | 20:01 |
| *** NobodyCam has quit IRC | 20:01 | |
| jhesketh | tristanC: quick question though (possibly also directed at jeblair), would it be better to have a conf.d type directory rather than having to specify includes? | 20:01 |
| *** jesusaurus has quit IRC | 20:01 | |
| *** BadCub has quit IRC | 20:02 | |
| *** esmute has quit IRC | 20:02 | |
| anteaya | no tc meeting? | 20:02 |
| clarkb | also there is a spec for that | 20:02 |
| clarkb | so the change should hopefully implement the spec | 20:02 |
| jhesketh | clarkb: a spec for splitting zuul's conf? | 20:02 |
| clarkb | jhesketh: yes | 20:02 |
| clarkb | let me get a link | 20:03 |
| jhesketh | oh, I must have missed that one | 20:03 |
| jhesketh | thanks | 20:03 |
| tristanC | clarkb: oups, I didn't knew that... | 20:03 |
| *** dprince has joined #openstack-infra | 20:03 | |
| fungi | http://specs.openstack.org/openstack-infra/infra-specs/specs/zuul_split.html | 20:03 |
| tristanC | jhesketh: yes sure, I just went the naive approach and extended the 'include' directive... | 20:03 |
| fungi | is the spec | 20:03 |
| *** esmute has joined #openstack-infra | 20:03 | |
| *** BadCub has joined #openstack-infra | 20:03 | |
| jhesketh | speaking of specs, there is a new version of bower for horizon sutff https://review.openstack.org/#/c/154297/ | 20:04 |
| *** NobodyCam has joined #openstack-infra | 20:04 | |
| *** bhuvan has quit IRC | 20:04 | |
| clarkb | https://review.openstack.org/#/c/133249/ | 20:04 |
| *** vipul has joined #openstack-infra | 20:04 | |
| clarkb | its merged so it just needs to be implemented | 20:04 |
| *** jesusaurus has joined #openstack-infra | 20:04 | |
| jhesketh | yep and has my preference for a conf dir rather than specific files | 20:04 |
| *** rfolco has quit IRC | 20:05 | |
| *** aysyd has quit IRC | 20:05 | |
| *** yfried_ is now known as yfried|afk | 20:05 | |
| tristanC | fungi: thanks for the link, good to know there is an infra-specs repository :) | 20:06 |
| fungi | tristanC: we're not as disorganized and impulsive as some people might claim ;) | 20:06 |
| fungi | well, i am, but the rest are sensible folk | 20:07 |
| *** Rockyg has joined #openstack-infra | 20:07 | |
| tristanC | alright, I can rework the proposed review to accomodate a conf.d like collection of files | 20:07 |
| tristanC | though, while trying to refactor the layout.yaml of project-config, I found out a lot of dependencies, e.g. triple-o project adding a pipeline that is used in most project | 20:09 |
| fungi | clarkb: jeblair: mordred: zaro: looking at http://cacti.openstack.org/cacti/graph_view.php?action=tree&tree_id=1&leaf_id=9 before i go firing up the trusty replacement for review.o.o, i'm thinking we're good for sticking with the 30g ram flavor we're already using, but if we want to resize now is the time to discuss it | 20:09 |
| clarkb | fungi: yes I agree, don't forget the ssd volume for git repos too | 20:09 |
| clarkb | though maybe we can just detach, then reattach to new node? | 20:10 |
| clarkb | I personally like rsync for simplicity though | 20:10 |
| * sc68cal scratches his head at http://paste.openstack.org/show/170926/ | 20:10 | |
| sc68cal | odd failure installing tempest via devstack.. | 20:10 |
| *** bhuvan has joined #openstack-infra | 20:12 | |
| clarkb | sc68cal: maybe you don't have sudo rights with that user? | 20:12 |
| sc68cal | nope, it's a devstack run so the stack user has the sudo privs | 20:13 |
| *** shashankhegde has quit IRC | 20:13 | |
| krtaylor | fungi, did we reach a conclusion on review.o.o upgrade date? | 20:14 |
| *** bswartz has quit IRC | 20:14 | |
| clarkb | sc68cal: devstack-gate removes the stack user's sudo rights | 20:14 |
| clarkb | sc68cal: so if running d-g twice it would fail like that iirc. Just pointing out there are scenarios where this likely won't work | 20:14 |
| sc68cal | ah you know what - I think it's related to I78d51f04ed01da4ce8aa0e127be028f969d3b4f8 just merged | 20:14 |
| *** ldnunes has quit IRC | 20:15 | |
| jeblair | asselin__: what's your best guess for how long the rule update would take at hp? | 20:15 |
| *** achanda has quit IRC | 20:15 | |
| asselin__ | jeblair, 2 months | 20:15 |
| jeblair | asselin__: (it seemed like you were saying the 2 month figure involved an error that had to be corrected last time) | 20:15 |
| asselin__ | jeblair, cody-somerville would know best | 20:15 |
| pleia2 | is it incidental or intentional that root admins are also storyboard admins? | 20:15 |
| jeblair | pleia2: intentional | 20:15 |
| *** wznoinsk has quit IRC | 20:16 | |
| fungi | asselin__: make cody-somerville personally change the firewall rules ;) | 20:16 |
| jeblair | pleia2: they are also gerrit admins, we'll fix that soon | 20:16 |
| *** wznoinsk_mobl has joined #openstack-infra | 20:16 | |
| pleia2 | jeblair: thanks | 20:16 |
| asselin__ | jeblair, it my case it was 2 months + 1 month to fix the fat fingered mistake | 20:16 |
| jeblair | cody-somerville: ? | 20:16 |
| cody-somerville | Heya. | 20:16 |
| cody-somerville | Whats needed? :) | 20:16 |
| asselin__ | that only affected by site (wrong ip) | 20:16 |
| clarkb | yolanda: so my local tests don't seem to be returning. I assume this is the behavior your saw with images not going ready | 20:17 |
| jeblair | cody-somerville: gerrit's ip will be changing. what's the minimum turnaround time for hp to upgrade firewall egress rules? | 20:17 |
| yolanda | yes, same one | 20:17 |
| *** wznoinsk_mobl has quit IRC | 20:17 | |
| *** cnesa has joined #openstack-infra | 20:17 | |
| yolanda | when you test the change with testr directly you see the issue, that is just watiing for getimage | 20:17 |
| *** ashleighfarnham has quit IRC | 20:17 | |
| cody-somerville | minimum? It'll depend on queue length. And they have scheduled times for deploys. So I'd say it'll probably take a month to two months. | 20:18 |
| clarkb | yolanda: I wonder if that incidicates that the actual daemon won't see the image go ready either | 20:18 |
| *** bswartz has joined #openstack-infra | 20:18 | |
| yolanda | clarkb, but the actual daemon will do real tests | 20:19 |
| clarkb | yolanda: ya I need to read up on that portion of the code | 20:19 |
| *** esmute has quit IRC | 20:19 | |
| yolanda | real checks, i mean | 20:19 |
| *** _nadya_ has joined #openstack-infra | 20:19 | |
| *** jesusaurus has quit IRC | 20:21 | |
| *** achanda has joined #openstack-infra | 20:21 | |
| *** vipul has quit IRC | 20:21 | |
| *** BadCub has quit IRC | 20:21 | |
| *** NobodyCam has quit IRC | 20:21 | |
| *** changbl has quit IRC | 20:21 | |
| jeblair | clarkb, fungi, mordred: what do you think about 1 month, so schedule for mar 21? | 20:22 |
| clarkb | yolanda: I think this doesn't work because UploadImageTask needs to have its main() method called | 20:22 |
| clarkb | jeblair: that weekend also works for me | 20:22 |
| clarkb | jeblair: It is a bit frustrating that our ability to progress is impeded by arbitrary corporate rules in $locations though | 20:22 |
| jeblair | i could be convinced that 1 month is a good notice period. honestly, i still feel like 2 months is us bending backwards to enable hp's dysfunction. | 20:22 |
| *** NobodyCam has joined #openstack-infra | 20:22 | |
| clarkb | jeblair: yes I would -1 2 months | 20:22 |
| clarkb | 1 month seems to be reasonable ish | 20:23 |
| *** zz_avozza is now known as avozza | 20:23 | |
| *** esmute has joined #openstack-infra | 20:23 | |
| *** BadCub has joined #openstack-infra | 20:23 | |
| *** vipul has joined #openstack-infra | 20:23 | |
| clarkb | yolanda: I think we should go back to the patchset 5 type change | 20:23 |
| mordred | I'm fine with that - and just for the record, I agree - I think it's crazy that this is a conversation | 20:23 |
| clarkb | yolanda: since that doesn't treat the image upload as a Task object which requires the task managers | 20:23 |
| yolanda | clarkb, looking | 20:24 |
| mordred | I mainly just want to make sure that the folks who work at places as broken as HP is don't get screwed in the process | 20:24 |
| fungi | jeblair: looks like that's right on the heels of feature freeze, so may be quiet-ish | 20:24 |
| clarkb | mordred: and the majority of them won't be. https works fine for pushing code and reviewing code | 20:24 |
| fungi | jeblair: possibly the best choice that close to release | 20:24 |
| *** jesusaurus has joined #openstack-infra | 20:25 | |
| fungi | jeblair: and i'm around that weekend as well | 20:25 |
| mordred | clarkb: k | 20:25 |
| clarkb | mordred: the only people that need to do crazyness are folks like asselin__ that either have to do silly proxying or wait for firewall updates (or both) to run third party CIs with zuul | 20:25 |
| clarkb | mordred: we know this works because it is how you get around China's firewall | 20:25 |
| jeblair | yeah, i feel like 1 month is a perfectly defensible time frame for that kind of update | 20:25 |
| krtaylor | ++ | 20:25 |
| jeblair | so it's a little painful for us, and a little painful for third parties | 20:25 |
| jeblair | which is the nature of all good compromises :) | 20:25 |
| asselin__ | correct, only impacts thirdparty who use zuul | 20:26 |
| fungi | as long as feature freeze doesn't get delayed for some reason we should be fine with that date i think | 20:26 |
| fungi | asselin__: only impacts third parties who have their zuul server stuck behind obnoxious egress filters they can't control | 20:26 |
| yolanda | clarkb, so you mean getting rid of the uploadimagetask object? | 20:26 |
| *** r1chardj0n3s_afk is now known as r1chardj0n3s | 20:26 | |
| anteaya | I should be around march 21 | 20:26 |
| yolanda | this was created on future patches | 20:27 |
| *** signed8bit has joined #openstack-infra | 20:27 | |
| asselin__ | fungi, correct. | 20:27 |
| jeblair | also, we should be pretty clear that an ip change can happen at any time | 20:27 |
| jeblair | if there is some problem with the server, we may have to move to a new ip with little or no notice | 20:28 |
| mordred | yah | 20:28 |
| clarkb | yolanda: yup | 20:28 |
| mordred | it IS a cloud, after all | 20:28 |
| asselin__ | for everyone else on our team, we used corkscrew | 20:28 |
| clarkb | yolanda: I commented on the change | 20:28 |
| anteaya | jeblair: agreed folks should know that | 20:28 |
| jeblair | so, happy to do some work to avoid breaking people with scheduled work, but we absolutely can't promise that there will always be notice | 20:28 |
| yolanda | clarkb , nice, i like your proposal | 20:29 |
| mordred | wenlock: ^^ you guys have magical pony powers - the above conversation might be of interest | 20:29 |
| mordred | wenlock: maybe someone could get 29418 blanket-outbound approved ... | 20:29 |
| *** sparkycollier has joined #openstack-infra | 20:29 | |
| wenlock | mordred, was just reading that, not sure which stream of conv though, trying to find the begining | 20:29 |
| wenlock | can you summarize? | 20:29 |
| asselin__ | this is what we used for 'normal' gerrit usage: http://paste.openstack.org/show/170931/ | 20:30 |
| mordred | wenlock: we're moving the gerrit server so it's IP is going to change | 20:30 |
| mordred | wenlock: and HP has stupid firewall rules | 20:30 |
| asselin__ | if we can find a working solution for zuul as clarkb says exists, that would be fine with me | 20:30 |
| *** maxbit has quit IRC | 20:30 | |
| mordred | wenlock: I think that sums it up | 20:30 |
| mordred | :) | 20:30 |
| wenlock | ahh yes,mordred, i know someone that knows someone that can help us fix that in our network | 20:30 |
| wenlock | we did it before | 20:30 |
| clarkb | asselin__: with https that shouldnt be necessary, you just need an http proxy and a different gerrit remote (one that uses https) | 20:30 |
| yolanda | clarkb, i'll file a change tomororw, first thing in the morning :) | 20:30 |
| yolanda | i want to have this updated | 20:30 |
| mordred | wenlock: awesome - I'll be sure to ping you when we have the new IP | 20:30 |
| clarkb | yolanda: sounds good, thank you | 20:31 |
| wenlock | bottom line, its just an ncs request, osanchez had set that up for us | 20:31 |
| wenlock | mordred sure, i'll forward to osanchez to help us kick that | 20:31 |
| *** amrith is now known as _amrith_ | 20:31 | |
| asselin__ | clarkb, sorry, not following | 20:31 |
| yolanda | oh, see that caching of floating ips got merged, i didn't realize :) | 20:31 |
| yolanda | i need to sync nodepool this week , want to have these patches | 20:31 |
| clarkb | asselin__: gerrit 2.8 accepts changes over https | 20:31 |
| clarkb | asselin__: so you don't need corkscrew and all that. You just need to be able to https out which is super common in corp envs | 20:31 |
| wenlock | ohh thats nice, clarkb, is git-review smart enough to make that switch? | 20:32 |
| clarkb | wenlock: if you change your gerrit remote | 20:32 |
| asselin__ | clarkb, ok nice. | 20:32 |
| asselin__ | clarkb, and for zuul, we can connect vis https? | 20:32 |
| clarkb | asselin__: no zuul requires ssh stream events | 20:32 |
| wenlock | clarkb not in the .gitreview file? | 20:32 |
| clarkb | wenlock: no because that would require changing all .gitreview files and using https across the board which I will personally -2 in projects I have the power to do that in :) | 20:33 |
| wenlock | ic, so it stays as a personal choice | 20:33 |
| clarkb | I don't want to have to deal with gerrit https passwords | 20:33 |
| clarkb | I think ssh as the default is best | 20:34 |
| clarkb | https is then useable for those that it is easier for | 20:34 |
| wenlock | ahh, i thought it would use pub keys, eek if it doesn't | 20:34 |
| clarkb | it uses the gerrit http password which you can get in your settings pages | 20:35 |
| clarkb | same thing used for REST api access | 20:35 |
| *** EmilienM|afk is now known as EmilienM | 20:35 | |
| fungi | yeah, if gerrit https supported x.509 client certs for auth it would make some of this saner | 20:35 |
| wenlock | yep | 20:35 |
| clarkb | fungi: not sure I agree, I would have a hard enough time deal with x.509 certs for my self | 20:35 |
| clarkb | I can only imagine how much support that would require for the general use bas | 20:36 |
| clarkb | keystone devs and fungi would be the only group with reliable push access :) | 20:36 |
| wenlock | clarkb ive been seeing that alot in other api's using https:// protocols though.. | 20:36 |
| fungi | well, it's no more complicated than pubkey auth for ssh, just not as well-known | 20:36 |
| wenlock | clarkb, docker api example | 20:36 |
| clarkb | fungi: it is in that openssl is the worst set of tools for a user ever | 20:36 |
| clarkb | fungi: even worse than socat | 20:36 |
| morganfainberg | hm. keystone. | 20:36 |
| clarkb | fungi: ssh-keygen on the other hand is simple and easy | 20:37 |
| stevemar | clarkb, keystone you say | 20:37 |
| morganfainberg | clarkb, i heard keystone | 20:37 |
| r1chardj0n3s | morning | 20:37 |
| sc68cal | Yeah i'm still trying to figure out why tempest is failing to install (http://paste.openstack.org/show/170926/) - it's a brand new VM, never run devstack before on it | 20:37 |
| clarkb | ha I highlighted all of keystone :) | 20:37 |
| morganfainberg | stevemar, ^^ | 20:37 |
| r1chardj0n3s | jhesketh: it seems things are busy in infra-land ;) | 20:37 |
| morganfainberg | client certs are probably not fun for most people to deal with | 20:38 |
| clarkb | sc68cal: can you try running that command directly? | 20:38 |
| clarkb | morganfainberg: exactly | 20:38 |
| clarkb | I always end up reading docs, then getting sad that the docs are terrible, then reading more docs, eventually giving in and just running commands until something works, but it doesn't work so I go back to the terrible docs | 20:39 |
| jhesketh | r1chardj0n3s: things are always busy in infra-land | 20:39 |
| clarkb | then heartbleed happens and it all makes sense | 20:39 |
| r1chardj0n3s | :) | 20:39 |
| fungi | new ip addresses for trusty review.o.o replacement: 104.130.159.134 2001:4800:7818:102:be76:4eff:fe05:9b12 | 20:39 |
| *** ashleighfarnham has joined #openstack-infra | 20:39 | |
| fungi | has someone already started a draft announcement? | 20:39 |
| sc68cal | clarkb: yeah, it's the "tox -e venv" command inside the tempest git dir | 20:40 |
| clarkb | fungi: I had not, maybe I should do that but also lunch | 20:40 |
| *** tsg_ has joined #openstack-infra | 20:40 | |
| fungi | i can get it going | 20:40 |
| sc68cal | run by devstack near the end | 20:40 |
| clarkb | sc68cal: that doesn't look like a tox comamdn, it is installing globally not in a tox env | 20:40 |
| *** ildikov has joined #openstack-infra | 20:40 | |
| clarkb | oh I see tox in the command that is weird | 20:40 |
| wenlock | fungi, so review.openstack.org will be 104.130.159.134, any others on 29418 ??? | 20:40 |
| anteaya | fungi: they should be able to access review-dev should they not? | 20:41 |
| wenlock | fungi btw, what was the old ip? | 20:41 |
| fungi | wenlock: we have a review-dev which is useful for testing things on if you want, 104.239.132.36 2001:4800:7818:104:be76:4eff:fe05:a763 | 20:41 |
| clarkb | sc68cal: I do not know why a tox venv python would be updating /usr/local | 20:41 |
| wenlock | fungi, perfect, ty | 20:42 |
| clarkb | wenlock: host review.openstack.org :) | 20:42 |
| fungi | wenlock: old (current) review.openstack.org is 23.253.232.87 2001:4800:7815:104:3bc3:d7f6:ff03:bf5d | 20:42 |
| fungi | also in dns | 20:42 |
| sc68cal | clarkb: neither do i | 20:42 |
| wenlock | fungi , clarkb ty, i'll see if i can dig up those old request | 20:42 |
| *** tsg has quit IRC | 20:42 | |
| wenlock | and have things updated on our end | 20:42 |
| *** MaxV has quit IRC | 20:42 | |
| *** VijayTripathi has quit IRC | 20:42 | |
| *** MaxV has joined #openstack-infra | 20:43 | |
| *** achanda has quit IRC | 20:43 | |
| wenlock | mordred, no email needed if that info is good | 20:43 |
| *** claudiub_ has quit IRC | 20:43 | |
| *** Sukhdev has quit IRC | 20:44 | |
| *** timo has joined #openstack-infra | 20:45 | |
| clarkb | wenlock: we have to send email for everyone else | 20:45 |
| *** timo is now known as Guest68417 | 20:45 | |
| wenlock | clarkb, yep, i was referring to the message he was sending me. | 20:46 |
| clarkb | oh | 20:46 |
| wenlock | on the info fungi just relayed | 20:46 |
| *** gokrokve has quit IRC | 20:46 | |
| *** gokrokve has joined #openstack-infra | 20:46 | |
| sc68cal | clarkb: reverting 9f1612e7afd9de56bc70d5c5b1c63ad5181d84ab fixes the issue - https://review.openstack.org/#/c/150928/ | 20:47 |
| clarkb | sc68cal: so that was tested, interesting that its breaking for you | 20:48 |
| *** MaxV has quit IRC | 20:48 | |
| clarkb | sc68cal: and this is a fresh host right? | 20:48 |
| sc68cal | yup | 20:48 |
| sc68cal | clarkb: https://bugs.launchpad.net/devstack/+bug/1420480 | 20:48 |
| openstack | Launchpad bug 1420480 in devstack "DevStack fails to install tempest" [Undecided,New] | 20:48 |
| clarkb | sc68cal: its a tempest bug if reverting that change fixes it | 20:48 |
| *** mestery has joined #openstack-infra | 20:49 | |
| wenlock | fungi, do you have dates yet for when those will cut over? | 20:49 |
| sc68cal | clarkb: ok - going to do a full run with the revert in my VM and see if it completes the run | 20:50 |
| wenlock | fungi, the new ip's that is | 20:50 |
| sc68cal | clarkb: believe me, I nuked the VM 3 times and ran devstack and got this error each time | 20:50 |
| clarkb | sc68cal: sure just pointing out you filed the bug in devstack but it would be tempest if tempest broke it | 20:50 |
| sc68cal | it's possible that some other change is triggering it | 20:50 |
| *** gokrokve has quit IRC | 20:51 | |
| clarkb | sc68cal: also what version of pip and setuptools are ou using? devstack should update both to latest but maybe that doesn't happen for some reason? | 20:51 |
| *** mestery has quit IRC | 20:51 | |
| sc68cal | clarkb: pip 6.0.8 from /usr/local/lib/python2.7/dist-packages (python 2.7) | 20:51 |
| *** Guest68417 has quit IRC | 20:52 | |
| clarkb | ya thats up to date so no problem there | 20:52 |
| sc68cal | I had clean runs this morning :-\ | 20:52 |
| sc68cal | err... that sounds wrong | 20:52 |
| clarkb | heh | 20:53 |
| anteaya | wenlock: well march 21 is being offered as an option with no clear acceptance that I have seen | 20:53 |
| anteaya | wenlock: so dates to be determined | 20:53 |
| clarkb | sc68cal: also ubuntu 14.04? fedora 21? I am assuming a debuntu version based ont he /usr/local stuff | 20:53 |
| *** _amrith_ is now known as amrith | 20:53 | |
| sc68cal | 14.04 | 20:53 |
| fungi | wenlock: i'm presently drafting the announcement so we can reach a consensus on the schedule | 20:53 |
| anteaya | wenlock: part of the date selection process was understanding how long big corps needed to update | 20:54 |
| *** shashankhegde has joined #openstack-infra | 20:54 | |
| *** nicodemos has quit IRC | 20:54 | |
| wenlock | anteaya last time we did it it took us 3 months, but i anticipate updates to take less than 30 days | 20:55 |
| anteaya | was/is | 20:55 |
| *** crc32 has joined #openstack-infra | 20:55 | |
| *** CIBuilds has joined #openstack-infra | 20:55 | |
| anteaya | wenlock: that is useful information to incorprate into out date selection, thank you | 20:55 |
| wenlock | anteaya i need to get the request going though to find out what the window will be, let me see if i can find a specific SLA they provide | 20:55 |
| anteaya | out/our | 20:55 |
| anteaya | wenlock: awesome | 20:55 |
| fungi | draft announcement is at https://etherpad.openstack.org/p/nMchefon2n | 20:58 |
| fungi | i had some stuff in there about using the gerrit rest api with git-review but ripped it back out because i don't know of any concise documentation we can point people at for the current release | 20:58 |
| *** mriedem has quit IRC | 20:59 | |
| *** BadCub has quit IRC | 20:59 | |
| *** esmute has quit IRC | 20:59 | |
| *** jesusaurus has quit IRC | 20:59 | |
| *** vipul has quit IRC | 20:59 | |
| *** NobodyCam has quit IRC | 20:59 | |
| clarkb | fungi: +2 I have a mail list thread on the https stuff | 20:59 |
| clarkb | fungi: let me get a link to that and you can just refer to it | 21:00 |
| fungi | clarkb: awesome--thanks! | 21:00 |
| clarkb | fungi: http://lists.openstack.org/pipermail/openstack-dev/2014-September/045383.html thread starts there | 21:00 |
| *** otter768 has joined #openstack-infra | 21:01 | |
| *** BobBall is now known as BobBall_AWOL | 21:01 | |
| clarkb | http://lists.openstack.org/pipermail/openstack-dev/2014-September/045385.html has my directions | 21:01 |
| clarkb | and ZZelle has info on using a git credential manager | 21:01 |
| *** VijayTripathi has joined #openstack-infra | 21:01 | |
| *** mriedem has joined #openstack-infra | 21:02 | |
| clarkb | sc68cal: fwiw I can run `tox -evenv -- ls` in an up to date tempest repo just fine | 21:02 |
| clarkb | sc68cal: so possibly that devstack is doing something funny | 21:03 |
| sc68cal | dunno - it's totally not touched | 21:03 |
| sc68cal | 5fc64f1fcba9a971e3c7f9e6100fd84d10678425 is the sha of devstack | 21:03 |
| clarkb | and I really need to go get lunch now. so back in a bit | 21:03 |
| sc68cal | clarkb: no problem | 21:04 |
| *** sdake has joined #openstack-infra | 21:04 | |
| openstackgerrit | Khai Do proposed openstack-infra/project-config: Setup project to run jobs on jenkins-dev.o.o https://review.openstack.org/153739 | 21:05 |
| *** otter768 has quit IRC | 21:05 | |
| jeblair | fungi: made suggested addition | 21:06 |
| jeblair | fungi: lgtm | 21:06 |
| *** baoli has joined #openstack-infra | 21:07 | |
| jeblair | and i do think we pretty much have consensus on mar 21 at this point | 21:07 |
| fungi | jeblair: thanks--that helps | 21:07 |
| *** esmute has joined #openstack-infra | 21:08 | |
| wenlock | anteaya, i found the answer in our faq from network connectivity services. Normal request are 90 days, expedited request are 30 days. | 21:08 |
| *** Alexandra_ has joined #openstack-infra | 21:10 | |
| jeblair | so, the good (?) news is that we have sufficient time to file an expedited request. | 21:10 |
| wenlock | jeblair, hehe yep | 21:10 |
| jeblair | the bad news is that if we ever had a problem with the gerrit server that resulted in an immediate ip change, hp would get a one month vacation from having to work on openstack | 21:10 |
| wenlock | it also helps if we can use the same request number as we did the first time, so i have to get that from my team who filed it | 21:10 |
| fungi | okay, if nobody else has any feedback on the announcement at https://etherpad.openstack.org/p/nMchefon2n i'll go ahead and send it | 21:10 |
| *** CIBuilds is now known as Tim_O | 21:11 | |
| wenlock | jeblair, would be nice if it was a pool of 5 ip's or something or 1 ip in each zone that it could possibly live in | 21:11 |
| *** vipul has joined #openstack-infra | 21:11 | |
| wenlock | fungi, wondering if there were backup ip's included, would be nice for future issues. | 21:11 |
| jeblair | wenlock: yes it would, but rax doesn't have floating ips, and we can't use their load-balancer product with non-http things :( | 21:12 |
| fungi | wenlock: lots of things would be nice, but our service providers assign ip addresses when we build a new instance | 21:12 |
| *** BadCub has joined #openstack-infra | 21:12 | |
| openstackgerrit | Khai Do proposed openstack-infra/project-config: Setup a project to test zuul-dev.o.o periodic pipeline https://review.openstack.org/154667 | 21:12 |
| wenlock | fungi, jeblair, gotcha | 21:12 |
| *** NobodyCam has joined #openstack-infra | 21:12 | |
| *** esmute has quit IRC | 21:12 | |
| fungi | wenlock: it's this new-fangled "cloud" concept you might have heard of already ;) | 21:12 |
| *** jesusaurus has joined #openstack-infra | 21:12 | |
| wenlock | haha | 21:12 |
| wenlock | fungi, im starting to get familiar with it....funny enough that was the same issue raised on other teams we're doing pilots with | 21:13 |
| fungi | wenlock: great to know there's some consensus around pain points at least | 21:13 |
| wenlock | old networks suck | 21:13 |
| *** amrith is now known as _amrith_ | 21:14 | |
| *** esmute has joined #openstack-infra | 21:14 | |
| openstackgerrit | Khai Do proposed openstack-infra/project-config: Setup a project to test zuul-dev.o.o periodic pipeline https://review.openstack.org/154667 | 21:14 |
| *** stevemar has quit IRC | 21:14 | |
| *** sparkycollier has quit IRC | 21:15 | |
| *** marun has quit IRC | 21:16 | |
| fungi | okay, announcement sent | 21:18 |
| *** radez_g0n3 is now known as radez | 21:20 | |
| *** Hefeweizen has quit IRC | 21:21 | |
| *** achanda has joined #openstack-infra | 21:21 | |
| anteaya | fungi: thanks | 21:21 |
| *** bswartz has quit IRC | 21:28 | |
| openstackgerrit | Elizabeth K. Joseph proposed openstack-infra/system-config: Add Elizabeth K. Joseph to servers and storyboard https://review.openstack.org/154672 | 21:30 |
| *** emagana has joined #openstack-infra | 21:32 | |
| zaro | jeblair: was wondering, where is the gerrit trove db? rax or hp? | 21:32 |
| fungi | zaro: rackspace dfw region | 21:33 |
| *** rlandy has quit IRC | 21:33 | |
| fungi | zaro: same provider and region as the server connecting to it | 21:33 |
| openstackgerrit | Elizabeth K. Joseph proposed openstack-infra/system-config: Add Elizabeth K. Joseph to servers and storyboard https://review.openstack.org/154672 | 21:33 |
| *** Alexandra_ has quit IRC | 21:33 | |
| *** jsavak has joined #openstack-infra | 21:35 | |
| zaro | I just asked SlickNik about the connection timeout and he says that rax might be using something in front of the trove db, repose?, and there's a timeout setting there as well. | 21:36 |
| *** dmsimard_away has joined #openstack-infra | 21:37 | |
| *** dmsimard_away is now known as dmsimard | 21:37 | |
| *** joesavak has quit IRC | 21:38 | |
| *** mestery has joined #openstack-infra | 21:39 | |
| *** baoli has quit IRC | 21:40 | |
| *** baoli has joined #openstack-infra | 21:40 | |
| *** _nadya_ has quit IRC | 21:41 | |
| fungi | zaro: ooh, interesting | 21:42 |
| fungi | that sounds entirely possible | 21:42 |
| zaro | know anybody we can ask about that? | 21:43 |
| anteaya | would JayF be someone we could ask? | 21:44 |
| jeblair | https://etherpad.openstack.org/p/Kukz1pmRkY | 21:44 |
| fungi | jroll: phschwartz: do either of you happen to know why can tell us about possible timeouts for established sockets to trove databases? | 21:44 |
| jeblair | fungi, clarkb, sdague: https://etherpad.openstack.org/p/Kukz1pmRkY | 21:44 |
| jeblair | anteaya: ^ | 21:44 |
| *** dannywilson has quit IRC | 21:44 | |
| jeblair | lemme know if that looks like it's ready for the -dev list. | 21:45 |
| *** dannywilson has joined #openstack-infra | 21:45 | |
| jroll | fungi: uhhhhh | 21:45 |
| JayF | I don't know things about trove? | 21:45 |
| *** Sukhdev has joined #openstack-infra | 21:45 | |
| jroll | fungi: can you clarify? I can try to round someone up | 21:45 |
| mtreinish | jeblair: does it detect if you did a circular dependency to know not to do that? | 21:46 |
| jroll | fungi: is this a problem or a question you have? | 21:46 |
| jeblair | mtreinish: see section "Cycles" :) | 21:46 |
| SlickNik | zaro / fungi: I'm trying to see if I can reach out to some of the RAX folks who help operate trove their public cloud to see if I can find out some more about it as well. | 21:47 |
| fungi | jroll: we have server instances in dfw using cloud databases in dfw and see the connections to them time out in a matter of a few minutes | 21:47 |
| *** ociuhandu has quit IRC | 21:47 | |
| mtreinish | jeblair: heh, didn't get that far | 21:47 |
| anteaya | jeblair: yes was reading, nicely communicated | 21:47 |
| jeblair | mtreinish: i did also inclued "don't create a cycle" in the tldr :) | 21:47 |
| jroll | fungi: lemme ask around, can you put in a support ticket while you're at it? | 21:48 |
| jeblair | mtreinish: i should put it in the tltldrdr | 21:48 |
| mtreinish | jeblair: that's why I asked. You said don't do it, but I was curious why | 21:48 |
| anteaya | jeblair: in the case of a patch series that is tested together in check and then one of the dependent patches is removed, what happens? | 21:48 |
| mtreinish | err curios what would happen if I didn't listen | 21:48 |
| jeblair | oh good, hopefully it will encourage people to keep reading then :) | 21:48 |
| anteaya | jeblair: dependencies is removed | 21:48 |
| fungi | jroll: i think i tried previously, but was unable to reproduce the behavior using the command-line mysqlclient | 21:49 |
| jeblair | anteaya: still asking about in check? so like if one is abandoned while something depending on it is testing? | 21:49 |
| *** bswartz has joined #openstack-infra | 21:49 | |
| jeblair | anteaya: or were you asking what happens in gate? | 21:49 |
| fungi | jroll: though we've had a couple database-using applications written in entirely unrelated frameworks display similar behavior | 21:50 |
| jroll | fungi: hrm, weird. is there traffic moving over it when it times out? it does connect at all, correct? | 21:50 |
| zaro | fungi: remember it only repros for us when connectionpool=true | 21:50 |
| jroll | oh | 21:50 |
| jeblair | anteaya: (regardless, the answer is "nothing happens in check" and "it is removed in gate") | 21:50 |
| jroll | fungi: zaro: maybe a connection limit thing? | 21:50 |
| jeblair | zaro: connectionpool!=true means make a new connection each time | 21:50 |
| *** gyee has joined #openstack-infra | 21:51 | |
| fungi | jroll: yeah, it behaves like an idle timeout, but mysqlclient reports that the trove instance is configured for default (many hours) idle timeout | 21:51 |
| jeblair | jroll: there are about 4 connections involved | 21:51 |
| zaro | jroll: it only disconnect on after idling for about 5 mins. | 21:51 |
| jroll | fungi: right, I'm wondering if there's network gear killing it | 21:51 |
| jroll | I've seen this with F5s | 21:51 |
| fungi | jroll: i'm assuming there probably is | 21:51 |
| fungi | jroll: such as the f5 big-ip default 5-minute socket timeout, yeah | 21:51 |
| jroll | yeah... idk if this is something I can get folks to fix but I can try | 21:51 |
| jroll | yeah | 21:51 |
| fungi | used to manage dozens of them myself | 21:51 |
| fungi | that was one of the things we always had to bump for customers | 21:52 |
| jroll | fungi: right, that's what this looks like, I'll see if I can do something though it will likely need a ticket to get traction | 21:52 |
| *** asselin_ has quit IRC | 21:52 | |
| anteaya | jeblair: was asking about the jenkins comment | 21:52 |
| *** dannywilson has quit IRC | 21:53 | |
| fungi | anyway, i can try to write something up in a ticket, but i'm not sure how to describe a concise case to reproduce it | 21:53 |
| *** dannywilson has joined #openstack-infra | 21:53 | |
| jroll | fungi: just give what you have and we'll see how it goes | 21:53 |
| anteaya | jeblair: to know if some patch was tested with another patch in one set of tests and if the dependency relationship is removed or changed to recheck it to test both together again | 21:53 |
| anteaya | jeblair: and yes, figured it wouldn't work in gate but might be helpful at some point to have a comment with the tests to know if the testing environment (which patches it was tested wtih) has changed | 21:54 |
| anteaya | if I am making sense | 21:54 |
| *** dkliban is now known as dkliban_afk | 21:54 | |
| jroll | fungi: to be clear, pass me the ticket id when you have it | 21:55 |
| jeblair | anteaya: to remove or change the dependency relationship, you would have to upload a new version of the patch | 21:55 |
| anteaya | jeblair: that is true | 21:55 |
| jeblair | anteaya: so that would be obvious. if there's a transitive dependency relationship and something in the middle changes, then, yeah, the results of the thing on the end are now invalid | 21:56 |
| *** iccha has joined #openstack-infra | 21:56 | |
| iccha | hey I hear some trouble with instances in dfw with mysql | 21:56 |
| iccha | anyone got an instance id? | 21:56 |
| anteaya | iccha: you are here | 21:56 |
| anteaya | iccha: I'm so glad | 21:56 |
| jeblair | anteaya: but then gerrit will say that it depends on an outdated patch. so that should be apparent too. | 21:56 |
| anteaya | okay | 21:57 |
| anteaya | I'll look for that | 21:57 |
| pleia2 | welcome iccha, zaro was looking into some trove issues | 21:57 |
| anteaya | jeblair: I added a question to line 108 in the etherpad | 21:57 |
| pleia2 | I think he'd be the one to chat with | 21:57 |
| fungi | iccha: i can get you the nova and trove instance uuids for one known to be causing a problem anyway | 21:57 |
| anteaya | or asked you a question | 21:57 |
| jeblair | anteaya: yeah, working out the timestamp now | 21:57 |
| anteaya | jeblair: cool, thanks | 21:57 |
| iccha | thanks fungi that ll be useful | 21:57 |
| jroll | iccha: you got this then? | 21:57 |
| iccha | ya i pinged our ops and they have not heard about anything else in dfw, so having instance id might help | 22:00 |
| fungi | iccha: we're seeing it, for example, with established mysql sockets originating from nova instance 9d3ff96d-920d-4d4f-854e-7057d268795c to trove instance f8bc89f6-adfb-4e89-9312-ea38b8c63fb0 (both in dfw) | 22:00 |
| fungi | iccha: this has been going on for months, maybe a year, probably for at least as long as we've been using rackspace trove instances | 22:00 |
| fungi | we already moved one service off of a trove instance and back onto a local mysql database on the server due to similar behavior | 22:01 |
| *** crc32 has quit IRC | 22:01 | |
| fungi | i'm assuming this is something inherent in the design of the deployment | 22:02 |
| ttx | fun part about Liberty is... I ran a test poll with yesterday's ballot set and Lizard won | 22:02 |
| *** dimtruck is now known as zz_dimtruck | 22:03 | |
| mtreinish | ttx: did some one write a bot for last min voting? | 22:03 |
| ttx | mtreinish: I checked for that | 22:03 |
| fungi | iccha: basically the common thread between the cases we've seen is that the client end doesn't know that the server has dropped its established socket, and then throws an error and has to reconnect. and this seems to happen after only a few (~3) minutes of inactivity on the socket | 22:03 |
| ttx | mtreinish: nothing obvious in the last line | 22:03 |
| fungi | ttx: mtreinish: people who love liberty also love procrastinating? | 22:04 |
| ttx | mtreinish: here was yesterday's test: http://civs.cs.cornell.edu/cgi-bin/results.pl?id=E_7d805614c0655c5d | 22:04 |
| *** Hefeweizen has joined #openstack-infra | 22:04 | |
| sdague | fungi: sounds pretty 'merican | 22:04 |
| mtreinish | fungi: heh, probably | 22:04 |
| *** e0ne has quit IRC | 22:05 | |
| mtreinish | ttx: yeah comparing that with the final set doesn't reall show anything obvious | 22:05 |
| ttx | lookign at trends on the poll, all I can say is that Lizard started strong but Liberty caught up | 22:05 |
| * mtreinish inner conspiracy nut is disappointed | 22:06 | |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/zuul: Add support for disabling bad pipelines https://review.openstack.org/154367 | 22:06 |
| sdague | mordred: http://logs.openstack.org/37/151037/2/check/check-requirements-integration-dsvm/087d1d7/console.html#_2015-02-10_21_46_35_337 ??? | 22:06 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/zuul: Refactor sources out of triggers https://review.openstack.org/118993 | 22:06 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/zuul: Add gerrit reviews into patchset approvals https://review.openstack.org/97390 | 22:06 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/zuul: Add support for negative requirements https://review.openstack.org/102726 | 22:06 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/zuul: Configure triggers dynamically https://review.openstack.org/119534 | 22:06 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/zuul: Add support for 'connection' concept https://review.openstack.org/121528 | 22:06 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/zuul: Call driver methods more dynamically https://review.openstack.org/119533 | 22:06 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/zuul: Add base class for triggers https://review.openstack.org/119532 | 22:06 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/zuul: Add base class for sources https://review.openstack.org/119531 | 22:06 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/zuul: Add base class for reporters https://review.openstack.org/119530 | 22:06 |
| sdague | error in setup command: Error parsing /tmp/tmp.A8NOB41seo/projects/installceilometer/setup.cfg: ValueError: git history requires a target version of pbr.version.SemanticVersion(2014.2.3), but target version is pbr.version.SemanticVersion(2014.2.2) | 22:07 |
| jhesketh | sdague: just wondering if you were able to respond to my comments on https://review.openstack.org/#/c/141260/ please | 22:07 |
| mtreinish | sdague: that's because after the most recent release no one bumped the version string on stable's setup.cfg | 22:07 |
| mtreinish | mriedem: has patches to fix it everywhere | 22:07 |
| mtreinish | fwiw, it's on that tracking etherpad as #2 | 22:07 |
| sdague | mtreinish: oh, so all of stable juno is bust? | 22:07 |
| sdague | for a new reason | 22:07 |
| mtreinish | sdague: yeah pretty much | 22:07 |
| mtreinish | which is part of what we were raising on the list... | 22:08 |
| *** dustins has quit IRC | 22:08 | |
| clarkb | though pbr enforcement of that is unreleased | 22:09 |
| clarkb | I think | 22:09 |
| *** joesavak has joined #openstack-infra | 22:09 | |
| mtreinish | clarkb: oh right it was reverted at one point | 22:10 |
| *** crainte has joined #openstack-infra | 22:10 | |
| jeblair | anyone have any further thoughts on the CRD email? or shall i fire it off? | 22:10 |
| crainte | iccha '-')/ | 22:10 |
| *** bswartz has quit IRC | 22:11 | |
| clarkb | jeblair I just realized it will break sdagues hopes for no tests on only commit changes | 22:11 |
| clarkb | but I dont think that needs to be in the email | 22:11 |
| jeblair | clarkb: ? | 22:11 |
| *** vipul has quit IRC | 22:11 | |
| sdague | clarkb: 0.10.7 right? | 22:11 |
| fungi | jeblair: crd announcement lgtm | 22:11 |
| fungi | sorry, slow reader and keep getting sucked into other broken things | 22:12 |
| clarkb | jeblair sdague wants/wanted to not test patches that only update the commit message | 22:12 |
| iccha | fungi: crainte is our ops engineer who is looking currently into your issue | 22:12 |
| anteaya | jeblair: lgtm too | 22:12 |
| *** NobodyCam has quit IRC | 22:12 | |
| clarkb | sdague no thats off a featyre branch without semver | 22:12 |
| anteaya | jeblair: mostly this will just create a flood of questions but the email is clear enough that now I think I know what is going on | 22:12 |
| sdague | clarkb: here is the freeze - http://logs.openstack.org/37/151037/2/check/check-requirements-integration-dsvm/087d1d7/logs/pip-freeze.txt.gz | 22:12 |
| *** VijayTripathi1 has joined #openstack-infra | 22:12 | |
| *** jsavak has quit IRC | 22:12 | |
| *** MaxV has joined #openstack-infra | 22:13 | |
| fungi | iccha: crainte: thanks. as much as anything we were curious whether this was an intentional part of the design and if so why such a short timeout was chosen | 22:13 |
| anteaya | so when they pop in here hopefully I can say something coherent and useful | 22:13 |
| *** VijayTripathi has quit IRC | 22:13 | |
| sdague | clarkb: so I think the no tests on commit message change will just need to be smarter.... possibly zuul smarter | 22:13 |
| crainte | fungi not that i'm aware of, typically you see that kind of timeout when you put the server behind a load balancer. That doesn't seem to be the case here though | 22:13 |
| jeblair | clarkb: i don't think it ends all hope of that. but that isn't straightforward to start with. | 22:13 |
| *** mrmartin has quit IRC | 22:13 | |
| sdague | jeblair: agreed | 22:13 |
| zaro | jeblair: does depends-on require the full change id? | 22:13 |
| jeblair | zaro: yes | 22:14 |
| jeblair | zaro: probably | 22:14 |
| jeblair | :) | 22:14 |
| jeblair | zaro: maybe not but let's say it does | 22:14 |
| *** NobodyCam has joined #openstack-infra | 22:14 | |
| sdague | let's all pretend | 22:14 |
| anteaya | it does for me | 22:14 |
| fungi | crainte: would a packet capture help at all? | 22:14 |
| jeblair | i'll put that in the email and send it off. | 22:14 |
| crainte | fungi i'm watching traffic on the database instance now but there is a big lack of mysql traffic. trying to find your cloud server and capture traffic on that end as well | 22:15 |
| crainte | fungi if you have anything to generate mysql traffic to the instance please do | 22:15 |
| *** vipul has joined #openstack-infra | 22:15 | |
| fungi | crainte: right, one of the reasons we suspect an idle timeout is because this is a fairly idle system. i can cause it to do things now | 22:15 |
| *** VijayTripathi has joined #openstack-infra | 22:15 | |
| jroll | fungi: are you getting a tcp reset? | 22:16 |
| *** tiswanso_ has quit IRC | 22:16 | |
| jroll | crainte: is there an F5 in front of the dbaas stuff? | 22:16 |
| fungi | jroll: i'm not sure in this particular case. i haven't fired up tcpdump to see if i can get it to happen while i'm watching | 22:16 |
| *** thedodd has quit IRC | 22:17 | |
| *** VijayTripathi1 has quit IRC | 22:17 | |
| zaro | jeblair: ok. lgtm | 22:17 |
| iccha | jroll: no f5 | 22:17 |
| jroll | oh, interesting | 22:18 |
| sdague | clarkb: oh, the install pbr from repo parts are still in there | 22:18 |
| ekarlso | hey guys, I have a git requirement in requirements.txt, how can I make devstack honor my requirement there and not the one in global reqs ? | 22:18 |
| *** claudiub has joined #openstack-infra | 22:18 | |
| zaro | fungi: which trove db are we looking at? review.o.o or review-dev.o,o? | 22:18 |
| fungi | zaro: review-dev | 22:18 |
| zaro | fungi: i set the connectionpool to false so it's not gonna timeout anymore. | 22:19 |
| fungi | zaro: oh | 22:19 |
| ekarlso | I've even added LIBS_FROM_GIT=kazoo | 22:19 |
| zaro | shall i change it back to true | 22:19 |
| openstackgerrit | Sean Dague proposed openstack/requirements: don't install pbr from upstream source https://review.openstack.org/154687 | 22:19 |
| zaro | ? | 22:19 |
| ekarlso | without it being honored :( | 22:19 |
| fungi | zaro: how easy would it be to put it back so that rackspace can troubleshoot? | 22:19 |
| *** mriedem has quit IRC | 22:19 | |
| zaro | easy, doing now | 22:20 |
| *** tiswanso has joined #openstack-infra | 22:20 | |
| fungi | zaro: thanks, i'll start tcpdump while it's up and running and then try to catch a timeout | 22:20 |
| *** mriedem has joined #openstack-infra | 22:20 | |
| mriedem | mtreinish: sdague: https://review.openstack.org/#/q/Ib8a29258d99de75b49a9b19aef36bb99bc5fcac0,n,z | 22:20 |
| sdague | ekarlso: LIBS_FROM_GIT only works for libraries that have git install paths in devstack | 22:21 |
| mriedem | mtreinish: sdague: the ceilometer one depends on https://review.openstack.org/#/c/154631/ | 22:21 |
| zaro | ok, done. it should timeout in about 3-5 mins | 22:21 |
| ekarlso | sdague: :'( | 22:21 |
| sdague | mriedem: so we could just fix the tests | 22:21 |
| ekarlso | sdague: so no way to pull a lib from git =? | 22:21 |
| sdague | ekarlso: make it an external devstack plugin | 22:21 |
| fungi | zaro: thanks, i'll start the tcpdump in a couple minutes and then try to interact with the site and see what we catch | 22:21 |
| ekarlso | sdague: eh, really ? :| | 22:21 |
| *** unicell has quit IRC | 22:21 | |
| ekarlso | sounds like a bit of crap atm | 22:22 |
| crainte | fungi iccha zaro https://gist.github.com/crainte/abf7b44ad74da0bac342 :P | 22:22 |
| anteaya | ekarlso: let's ensure our language supports a positive work environment | 22:23 |
| fungi | crainte: is that 120 seconds timeout for inactive sessions? | 22:23 |
| zaro | crainte: 120 was exactly what we were seeing. | 22:23 |
| ekarlso | anteaya: ok, I meant it's really sad that I can't specify a requirement from git and have it installed.. | 22:23 |
| fungi | crainte: when queried the mysqld running there claims to have default session timeouts (which i think are something like 12 hours) | 22:23 |
| ekarlso | withou | 22:23 |
| ekarlso | without having to do semi-hacks to get it in there | 22:23 |
| anteaya | ekarlso: and your clearly explained statement does so wonderfully | 22:24 |
| *** crc32 has joined #openstack-infra | 22:24 | |
| anteaya | ekarlso: and yes requirements are difficult | 22:24 |
| fungi | ekarlso: semi-hacks like releasing software> | 22:24 |
| *** pc_m has quit IRC | 22:24 | |
| sdague | the external devstack plugins also aren't semi-hacks | 22:24 |
| *** tiswanso has quit IRC | 22:25 | |
| crainte | fungi zaro updated that gist | 22:25 |
| zaro | fungi: i just saw the timeout | 22:25 |
| fungi | zaro: yep, i'm looking at the capture now | 22:25 |
| crainte | the show variables does return 28800 but the show global returns the configured 120 | 22:25 |
| sdague | mriedem: this might fix it as well - https://review.openstack.org/#/c/154688/ | 22:26 |
| sdague | anyway, time to call it a night, see you folks in the morning | 22:26 |
| fungi | crainte: yep, tcp reset coming back from the database server/port | 22:26 |
| *** stevemar has joined #openstack-infra | 22:26 | |
| fungi | crainte: thanks--that explains our confusion and the behavior we're seeing | 22:27 |
| fungi | jeblair: ^ https://gist.github.com/crainte/abf7b44ad74da0bac342 | 22:27 |
| jeblair | fungi: following | 22:27 |
| jeblair | not sure i understand the difference between 'show variables' and 'show global variables' | 22:27 |
| *** mriedem has quit IRC | 22:27 | |
| fungi | crainte: is the global wait_timeout configurable from the client side or is that immutable and set by rackspace? | 22:28 |
| crainte | looking it up before i make that suggestion :P | 22:29 |
| *** dprince has quit IRC | 22:29 | |
| jeblair | (or why the global scope would override the session scope) | 22:29 |
| crainte | fungi looks like it is configurable via control panel/api | 22:30 |
| jeblair | OOOH | 22:30 |
| jeblair | i think that if we had a mysql lib client query that value it would show 120 in both places | 22:31 |
| *** elliottk_ has joined #openstack-infra | 22:31 | |
| *** arosen has joined #openstack-infra | 22:31 | |
| jeblair | but the _interactive_ sessions get the default set from interactive_timout, so that's what we see as the session (non-global) value | 22:31 |
| arosen | Hi, I keep hitting these error in my CI runs pretty frequently: ReadTimeoutError: HTTPConnectionPool(host='pypi.dfw.openstack.org', port=80): Read timed out. | 22:31 |
| jeblair | non-interactive (eg, gerrit) sessions get their value initialized from global wait_timout | 22:31 |
| *** cnesa has quit IRC | 22:31 | |
| arosen | I was wondering if there was anything obvious I could do. Is there a better mirror or ..? | 22:32 |
| fungi | this explains why i also wasn't able to reproduce the idle timeout behavior with an idle mysqlclient cli session | 22:32 |
| jeblair | arosen: this is for your own third-party ci? | 22:32 |
| arosen | http://logs2.aaronorosen.com/88/153088/8/check/dsvm-tempest-full-congress-nodepool/cd8fd02/logs/devstacklog.txt.gz#_2015-02-10_22_29_15_134 <-- it works for a while | 22:32 |
| arosen | jeblair: yup | 22:32 |
| fungi | arosen: use pypi.python.org instead | 22:32 |
| jeblair | arosen, fungi: ++ | 22:33 |
| arosen | if I set that in /etc/nodepool/scripts/configure_mirrors.sh do i need to rebuild the image? | 22:33 |
| dansmith | jeblair: thanks for the extra awesome | 22:33 |
| jeblair | dansmith: you're welcome! | 22:34 |
| fungi | crainte: so i'm looking in the dashboard and it's not obvious where you would adjust these values, but presumably they can be set via the cli/api? | 22:34 |
| mordred | jeblair: yes to your above statement | 22:34 |
| mordred | sorry I wasn't here to chime in | 22:34 |
| crainte | fungi correct, in the dashboard it's Databases > Configurations | 22:35 |
| crainte | you basically build a set of default configurations and then apply that set to an instance | 22:35 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/zuul: Update the description for grey icons on status pg https://review.openstack.org/154690 | 22:35 |
| fungi | crainte: ooh! i was looking at instances | 22:35 |
| jeblair | mordred: np; sounds like we understand the problem and hopefully ^^ might be able to fix it! | 22:35 |
| fungi | crainte: aha, so we can create a new db config and then apply that. thanks | 22:35 |
| crainte | correct | 22:35 |
| mordred | jeblair: yay! | 22:36 |
| jeblair | mordred: i am, however, going to blame mysql for being slightly insane in this regard. | 22:36 |
| mordred | jeblair: well ... there is actually a good reason for it - but I agree it's not unconfusing | 22:36 |
| *** NobodyCam has quit IRC | 22:37 | |
| elliottk_ | hi all. i could use some help from someone familiar with the JJB email-ext module. is there anyone here who is? | 22:37 |
| *** NobodyCam has joined #openstack-infra | 22:38 | |
| zaro | sure, ask away | 22:38 |
| elliottk_ | thanks, zaro | 22:38 |
| fungi | jeblair: mordred: clarkb: zaro: so anyway, we probably want to create a custom config with the wait_timeout cranked up to a more reasonable value and then apply that named configuration to all our trove instances | 22:38 |
| clarkb | 8 hours | 22:39 |
| elliottk_ | i'm trying to configure my email-ext so that the failure trigger and success trigger sections have, perhaps, different recipients as well as different subject lines, for example | 22:40 |
| mordred | oh - wait - hang on - I may have been missing the context on the why here | 22:40 |
| jeblair | mordred: which? initializing one value from an identically named other value? or selectively initializing it from a third, different value in certain circumstances? :) | 22:40 |
| dims__ | jeblair, folks, very cool. ("Depends-On") | 22:40 |
| *** elliottk_ has left #openstack-infra | 22:40 | |
| mordred | fungi: what was the base problem? | 22:40 |
| *** elliottk_ has joined #openstack-infra | 22:40 | |
| mordred | jeblair: how else are you going to justify highly-paid consultants? | 22:40 |
| mordred | fungi: this is for the trove connection from ... gerrit? | 22:40 |
| jeblair | mordred: you're right, there was a good reason :) | 22:40 |
| fungi | mordred: yep | 22:41 |
| jeblair | fungi, clarkb: 8 hours (mysql default) sounds good. | 22:41 |
| fungi | mordred: rackspace default for new trove instances seems to be 2 minutes idle timeout | 22:41 |
| mordred | ok. it does not bother me as a setting for gerrit | 22:41 |
| jeblair | mordred: would it bother you for something else? | 22:41 |
| jeblair | eg, storyboard? | 22:42 |
| *** mwagner_lap has quit IRC | 22:42 | |
| zaro | i'm noticing that this little bit of puppet doesn't seem to work. http://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/manifests/gerrit.pp#n274 | 22:42 |
| crainte | fungi this might just be a mysql 5.1 default (for rackspace) I'll try to get it updated to a higher value. Our 5.6 instances seem to have 3600 | 22:42 |
| mordred | it would - often times in higher-volume (potentially higher numbers of connections) because of the way thread memory works - you want to reap and re-create rather than hold open for ages | 22:42 |
| mordred | for instance, for web properties, having a wait_timeout of over 1 is almost never correct | 22:43 |
| fungi | crainte: interesting. i wonder if that changed in response to customer feedback ;) | 22:43 |
| openstackgerrit | Joe Gordon proposed openstack-infra/devstack-gate: Disable install tempest by default https://review.openstack.org/154694 | 22:43 |
| jeblair | http://dev.mysql.com/doc/refman/5.1/en/server-system-variables.html#sysvar_wait_timeout | 22:43 |
| zaro | the source is a soft link, not an actual file. does anybody know if that should work? | 22:43 |
| mordred | BUT - as with all tuning, it depends on usage | 22:43 |
| jeblair | fungi, crainte, mordred: i thought that mysql doc was saying the default was 8 hours | 22:43 |
| mordred | and the way gerrit is using this I would imagine it's fine | 22:43 |
| mordred | yes | 22:43 |
| mordred | the default is 8 hours | 22:43 |
| mordred | mysql is notorious for having bad defaults | 22:43 |
| jeblair | mordred: do you think we should have a lower value for storyboard? | 22:44 |
| mordred | the real thing is that we noticed at all - gerrit shoudl be transparently reconnecting | 22:44 |
| crainte | jeblair mysql default, but rackspace configs override it. we currently have 120s for mysql 5.1 and 3600s for mysql 5.6 | 22:44 |
| jeblair | crainte: gotcha | 22:44 |
| fungi | mordred: fwiw i strongly suspect this is what was happening to paste/lodgeit as well. it was using sqlalchemy (purportedly) and should have been automagically detecting and reconnecting via mysql ping functionality | 22:45 |
| mordred | jeblair: apps _should_ be designed such that a network interruption to the db does not cause a failed operation, but merely a reconnect - if they want to appropriately deal with HA systems and whatnot | 22:45 |
| clarkb | mordred we thought that about lodgeit too | 22:45 |
| mordred | yah - clearly neither is actually designed to dtrt here - and they're not high traffic enough - so I thin kthe wait_timeout is fine | 22:45 |
| fungi | mordred: we were also seeing it from jeepyb's gerritdb module until i added an explicit mysql ping call to it | 22:46 |
| mordred | I'd like to think about it more for storyboard - but not right now - it's not an urgent setting | 22:46 |
| elliottk_ | zaro: what i'm trying to do is use the email-ext module to allow me to have different recipients and different subjects, depending on if the job fails or succeeds. i have not been able to figure out how to do this | 22:46 |
| jeblair | mordred: yeah, i'm not sure how forcing a disconnect rather than just expecting clients to cycle their own connections is preferable. i'm looking forward to hearing war stories over beer that explain all. :) | 22:46 |
| elliottk_ | the config files of jenkins-ui-generated jobs have a deeper level of email notification than i have been able to create using jjb | 22:47 |
| mordred | jeblair: it's actually to deal with poorly-behaving clients :) | 22:47 |
| elliottk_ | i can paste a couple config files if that would help | 22:47 |
| *** mfink_ has quit IRC | 22:47 | |
| *** Rockyg has quit IRC | 22:47 | |
| mordred | jeblair: but poorly behaving clients in a runaway web app that's getting slammed can up the memory pressure inside of mysql, cause all queries to get slower and death by thundering herd | 22:48 |
| elliottk_ | zaro: or some screenshots, if that's even better | 22:48 |
| mordred | we have no web apps that see that kind of traffic | 22:48 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/os-loganalyze: Assume trailing slashes want to load indexes https://review.openstack.org/141260 | 22:48 |
| mordred | jeblair: the other thing is that you want to make sure that your app handles disconnects properly so that it can handle db HA motions properly | 22:49 |
| fungi | mordred: jeblair: it's entirely likely that the rackspace default difference between mysql 5.1 and 5.6 (2 minutes vs an hour) is why it's only the things we put onto trove early seemed to be doing this but things we added later weren't (because they got 5.6) | 22:49 |
| *** shashankhegde has quit IRC | 22:49 | |
| mordred | then - if your app _Does_ ... because MySQL connection cost is so low, but because the internal thread scheduling is so bad | 22:49 |
| jeblair | fungi: yeah | 22:49 |
| mordred | you may want to trigger disconnects while idle _anyway_ so that the mysql thread scheduler can do a better job | 22:49 |
| *** shashankhegde has joined #openstack-infra | 22:49 | |
| mordred | it's one of those places where you really want all of the parts of the system to be resilient to get it to _really_ scream | 22:50 |
| zaro | elliottk_: the jjb email-ext module might need to be updated | 22:50 |
| zaro | elliottk_: are there any existing tests in that for it? | 22:50 |
| elliottk_ | zaro: i'm not sure what you mean by "any existing tests" | 22:51 |
| *** Tim_O has quit IRC | 22:51 | |
| elliottk_ | zaro: are you saying the jjb email-ext module may not current do what i need? | 22:51 |
| zaro | elliottk_: http://git.openstack.org/cgit/openstack-infra/jenkins-job-builder/tree/tests/publishers/fixtures/email-ext001.yaml | 22:51 |
| *** achanda has quit IRC | 22:52 | |
| *** esmute has quit IRC | 22:52 | |
| fungi | zaro: i've applied a 28800 second wait_timeout config to the review-dev db | 22:52 |
| elliottk_ | zaro: yes, this is the same as the page i'm using: http://ci.openstack.org/jenkins-job-builder/publishers.html | 22:52 |
| *** hdd has joined #openstack-infra | 22:52 | |
| elliottk_ | but from what i'm seeing, it does not currently support having different info for different email triggers | 22:53 |
| zaro | fungi: cool. i assume we don't need to restart Gerrit. | 22:53 |
| fungi | zaro: doesn't seem so | 22:53 |
| *** timcline has quit IRC | 22:53 | |
| zaro | fungi: i'm using review-dev.o.o so will let you know if it makes a difference. | 22:53 |
| fungi | zaro: i've used review-dev's ui again to get it to reconnect, so i'll check it again in 5-10 minutes and a few more times over the evening too | 22:54 |
| *** niska has quit IRC | 22:54 | |
| *** camunoz_gone has quit IRC | 22:55 | |
| *** BadCub has quit IRC | 22:55 | |
| *** vipul has quit IRC | 22:55 | |
| jogo | mordred: what does this mean http://logs.openstack.org/80/154680/1/check/check-requirements-integration-dsvm/c983c3a/console.html#_2015-02-10_22_26_13_283 | 22:55 |
| *** NobodyCam has quit IRC | 22:56 | |
| *** achanda has joined #openstack-infra | 22:56 | |
| zaro | elliottk_: different info? what type of other info were you expecting? i see failure, success, and aborted | 22:56 |
| jogo | mordred: ohh never mind, I sdague fixued those | 22:56 |
| jogo | just need to poke a few patches | 22:56 |
| jogo | ihrachyshka: https://review.openstack.org/#/q/Ib8a29258d99de75b49a9b19aef36bb99bc5fcac0,n,z are ready Ithink | 22:57 |
| *** niska has joined #openstack-infra | 22:57 | |
| *** camunoz_gone has joined #openstack-infra | 22:57 | |
| *** vipul has joined #openstack-infra | 22:57 | |
| *** esmute has joined #openstack-infra | 22:58 | |
| *** NobodyCam has joined #openstack-infra | 22:58 | |
| SlickNik | iccha / crainte: Thanks for following up with the wait_timeout matter! | 22:59 |
| *** dannywilson has quit IRC | 22:59 | |
| zaro | elliottk_: i think you just need to have multiple '- email-ext' definitions. | 23:00 |
| fungi | SlickNik: iccha: crainte: definitely--huge thanks. didn't realize the global values could override the reported session values, nor that we could set them ourselves. this ought to help tremendously | 23:00 |
| *** dannywilson has joined #openstack-infra | 23:01 | |
| *** dannywilson has quit IRC | 23:01 | |
| elliottk_ | zaro: oh! i hadn't thought of that. let me try | 23:01 |
| zaro | yeah thanks a bunch. that timeout was annoying the bleepers out of me. | 23:01 |
| *** BadCub has joined #openstack-infra | 23:01 | |
| *** otter768 has joined #openstack-infra | 23:01 | |
| *** bknudson has quit IRC | 23:01 | |
| *** dmorita has quit IRC | 23:01 | |
| crainte | fungi SlickNik np. i have a PR in now to make the change when new instances are created, so hopefully this won't creep up again | 23:02 |
| *** patrickeast_ has joined #openstack-infra | 23:02 | |
| *** patrickeast has quit IRC | 23:02 | |
| *** patrickeast_ is now known as patrickeast | 23:02 | |
| anteaya | as a note, I'm afk for tomorrow running errands | 23:03 |
| *** dannywil_ has joined #openstack-infra | 23:03 | |
| mordred | kk | 23:03 |
| *** dannywil_ has quit IRC | 23:04 | |
| *** dannywilson has joined #openstack-infra | 23:04 | |
| fungi | ooh, there's a devstack change cross-repo-depending on a devstack-gate change in the check pipeline right now | 23:05 |
| mordred | fungi: NEAT | 23:06 |
| *** otter768 has quit IRC | 23:06 | |
| dansmith | I'm about to push up a 23 patch set across four projects | 23:06 |
| *** reed has quit IRC | 23:06 | |
| dansmith | (no not really) | 23:06 |
| fungi | dansmith: i was about to not be at all surprised | 23:07 |
| jeblair | by pure coincidence, i'm about to leave for the day! | 23:07 |
| *** Alexandra_ has joined #openstack-infra | 23:07 | |
| dansmith | heh | 23:07 |
| jeblair | (oh, then, uh, not really) | 23:07 |
| dansmith | heh | 23:07 |
| jogo | mordred: so I think this patch is old enough https://review.openstack.org/#/c/121251/1 | 23:07 |
| *** kgiusti has quit IRC | 23:07 | |
| jogo | mordred: actually we may want to revisit what belongs in openstack/openstack | 23:08 |
| *** hdd has quit IRC | 23:08 | |
| *** armax has quit IRC | 23:08 | |
| fungi | yep, jogo seems to be taking crd for a spin with devstack change 154689 so that he can actually see whether it works with the devstack-gate change 154694 it's depending on | 23:08 |
| jogo | now that we have decoupled a lot of projects from the gate | 23:08 |
| *** alexpilotti has joined #openstack-infra | 23:09 | |
| openstackgerrit | K Jonathan Harker proposed openstack-infra/jenkins-job-builder: The email publisher needs a list of recipients https://review.openstack.org/153432 | 23:10 |
| *** alexpilotti has quit IRC | 23:10 | |
| *** baoli has quit IRC | 23:11 | |
| mordred | jogo: uhm. yeah | 23:11 |
| *** doug-fish has left #openstack-infra | 23:11 | |
| *** baoli has joined #openstack-infra | 23:11 | |
| elliottk_ | zaro: i don't believe the recommendation of having multiple email-ext definitions is going to solve my problem | 23:11 |
| jogo | mordred: maybe make openstack/openstack everything under openstack | 23:12 |
| jogo | and not just 'integrated gate' openstack | 23:12 |
| elliottk_ | zaro: i need one definition with, for example, a different subject for a success trigger and a different one for a failure trigger | 23:12 |
| *** joesavak has quit IRC | 23:12 | |
| mordred | jogo: well ... in the new world order - is there any benefit in that repo anymore? | 23:12 |
| mordred | jogo: (just asking - I don't know if I have an answer) | 23:12 |
| lifeless | does it have a definition even? | 23:12 |
| mordred | lifeless: right now it's everything that's in the integrated gate | 23:13 |
| jogo | mordred: I think so, for tracking purposes | 23:13 |
| elliottk_ | zaro: i'm not sure how to make it clear the problem i'm having. i can upload some screenshots or config.xml snippets if that could help | 23:13 |
| lifeless | mordred: I know that | 23:13 |
| *** alexpilotti has joined #openstack-infra | 23:13 | |
| jogo | mordred: to say opentack landed x patches this week | 23:13 |
| lifeless | mordred: the context was new world order... | 23:13 |
| mordred | lifeless: then why did you ask? :) | 23:13 |
| * mordred trolls lifeless | 23:13 | |
| * lifeless trouts mordred | 23:13 | |
| mordred | jogo: nod | 23:13 |
| * mordred llamas lifeless | 23:13 | |
| jogo | mordred: because I think this graph is neato https://github.com/openstack/openstack/graphs/commit-activity | 23:13 |
| *** BadCub has quit IRC | 23:14 | |
| *** esmute has quit IRC | 23:14 | |
| mordred | jogo: what if we just made you a graph like that out of graphite? | 23:14 |
| jogo | mordred: having a single git tree is still useful for debugging things | 23:14 |
| mtreinish | mordred: would it be as pretty? | 23:14 |
| mordred | mtreinish: it would be prettier, as it wouldn't be tainted by that terrible URL :) | 23:14 |
| *** vipul has quit IRC | 23:14 | |
| *** NobodyCam has quit IRC | 23:14 | |
| jogo | even if some were release based things and other are install from source | 23:14 |
| mtreinish | heh, fair enough :) | 23:15 |
| jogo | mordred: but I would settle for a graphite graph | 23:15 |
| *** NobodyCam has joined #openstack-infra | 23:16 | |
| *** esmute has joined #openstack-infra | 23:17 | |
| openstackgerrit | Akihiro Motoki proposed openstack-infra/project-config: Add networking-nec project to StackForge https://review.openstack.org/154705 | 23:17 |
| openstackgerrit | Elizabeth K. Joseph proposed openstack-infra/project-config: Make devstack-gate stable release tests non-voting https://review.openstack.org/154706 | 23:18 |
| *** vipul has joined #openstack-infra | 23:19 | |
| clarkb | mordred: its a handy way to track numbers if nothing else | 23:19 |
| clarkb | gah I was scrolled up | 23:19 |
| *** shashankhegde has quit IRC | 23:20 | |
| * mordred hands clarkb a llama | 23:20 | |
| jogo | mordred: I'll propose a change to openstack/openstack and we can discuss it there | 23:21 |
| clarkb | mordred: I can report that slingtv espn seems to work pretty well. so there is no reason to have cable anymore | 23:21 |
| mordred | jogo: kk | 23:21 |
| mordred | clarkb: woot | 23:21 |
| *** BadCub has joined #openstack-infra | 23:21 | |
| zaro | jeblair: was wondering why ZUUL_PIPELINE param is sent for silent pipeline but not for periodic pipeline? is that something that's configurable? | 23:22 |
| *** hdd has joined #openstack-infra | 23:22 | |
| jeblair | zaro: it should always be sent | 23:22 |
| zaro | *i mean sent to jenkins | 23:22 |
| zaro | jeblair: i don't see it in some of the pipelines.. https://jenkins-dev.openstack.org/job/gate-noop/ | 23:23 |
| zaro | jeblair: ohh wait, i do see it. it's just not in the jenkins desription for the job. | 23:24 |
| *** crainte has quit IRC | 23:24 | |
| zaro | jeblair: jenkins build info is set by zuul as well right? i guess that's the problem. why is it different for different jobs? | 23:26 |
| *** amitgandhinz has quit IRC | 23:27 | |
| openstackgerrit | Merged openstack-infra/project-config: Make gate-ceilometer-python34 only run on master branch https://review.openstack.org/154631 | 23:27 |
| *** gyee has quit IRC | 23:28 | |
| *** iccha has left #openstack-infra | 23:28 | |
| *** tsg_ has quit IRC | 23:29 | |
| zaro | fungi: good sign, i haven't seen a timeout yet. | 23:30 |
| fungi | zaro: same here | 23:31 |
| *** esmute has quit IRC | 23:31 | |
| *** mestery has quit IRC | 23:31 | |
| *** BadCub has quit IRC | 23:33 | |
| *** NobodyCam has quit IRC | 23:33 | |
| *** vipul has quit IRC | 23:34 | |
| *** elliottk_ has left #openstack-infra | 23:34 | |
| *** mestery has joined #openstack-infra | 23:36 | |
| *** esmute has joined #openstack-infra | 23:37 | |
| *** marun has joined #openstack-infra | 23:37 | |
| *** vipul has joined #openstack-infra | 23:38 | |
| *** ihrachyshka has quit IRC | 23:38 | |
| *** chlong has joined #openstack-infra | 23:39 | |
| *** BadCub has joined #openstack-infra | 23:40 | |
| *** NobodyCam has joined #openstack-infra | 23:41 | |
| *** vigneshvar has quit IRC | 23:42 | |
| *** ociuhandu has joined #openstack-infra | 23:44 | |
| *** unicell has joined #openstack-infra | 23:44 | |
| jhesketh | mordred, jeblair: so, rethinking on the bower mirror stuff, I've rerolled https://review.openstack.org/#/c/154297/ if you get a chance :-) (no pressure) | 23:44 |
| *** Sukhdev has quit IRC | 23:45 | |
| *** nelsnelson has quit IRC | 23:45 | |
| *** nelsnelson has joined #openstack-infra | 23:45 | |
| *** nelsnelson has quit IRC | 23:46 | |
| *** dims_ has joined #openstack-infra | 23:46 | |
| *** emagana has quit IRC | 23:46 | |
| openstackgerrit | Merged openstack-infra/system-config: Only start 1 subunit worker per node https://review.openstack.org/151725 | 23:46 |
| *** emagana has joined #openstack-infra | 23:47 | |
| *** dims__ has quit IRC | 23:47 | |
| openstackgerrit | Merged openstack-infra/system-config: Update Jenkins doc to reference nodepool https://review.openstack.org/149051 | 23:49 |
| openstackgerrit | K Jonathan Harker proposed openstack-infra/puppet-elasticsearch: Make the number of shards and replicas tunable https://review.openstack.org/154711 | 23:49 |
| *** sabeen2 has quit IRC | 23:50 | |
| *** dims_ has quit IRC | 23:50 | |
| *** bswartz has joined #openstack-infra | 23:51 | |
| *** emagana has quit IRC | 23:51 | |
| openstackgerrit | Joe Gordon proposed openstack-infra/devstack-gate: DO NOT MERGE: testing aiopcpu with tempest full https://review.openstack.org/136504 | 23:53 |
| *** adalbas has quit IRC | 23:54 | |
| *** cdent has quit IRC | 23:54 | |
| *** carl_baldwin has quit IRC | 23:56 | |
| *** camunoz_gone is now known as camunoz | 23:56 | |
| *** mestery has quit IRC | 23:57 | |
| *** Alexandra_ has quit IRC | 23:58 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!