| fungi | confirmed heat-translator has indeed fixed their py34 unit test run. also the dev who picked up the python-heatclient bug has a stack of patches up to rework their mocks, though i haven't tested them quite yet | 00:00 |
|---|---|---|
| *** MaxV has joined #openstack-infra | 00:00 | |
| fungi | no new movement on the two open ubuntu sru requests though, nor any signs of activity for the python-glanceclient bug | 00:00 |
| clarkb | probably hung over from unicorn release | 00:00 |
| * clarkb hopes next week will be better | 00:00 | |
| fungi | we'll be too hung over to notice | 00:01 |
| *** msabramo has quit IRC | 00:02 | |
| anteaya | ianw: not so far, can we get it as an agenda item on tomorrow's meeting agenda? | 00:03 |
| anteaya | ianw: I won't be here for the meeting but will go by what the group decides | 00:03 |
| ianw | anteaya: yep. hopefully there is time :) | 00:04 |
| *** MaxV has quit IRC | 00:05 | |
| anteaya | hopefully | 00:05 |
| anteaya | I would like to hear more from nibalizer about puppet module names | 00:06 |
| anteaya | I'm still leaning toward getting rid of the underscore | 00:06 |
| *** mmaglana has quit IRC | 00:06 | |
| fungi | puppet-openstackci¯apache | 00:08 |
| nibalizer | anteaya: so i dont like puppet-openstackci_apache either really | 00:09 |
| nibalizer | but i dislike openstackciapache much more | 00:09 |
| fungi | puppet-opache | 00:09 |
| nibalizer | other modules have underscores https://forge.puppetlabs.com/openshift/openshift_origin/dependencies | 00:10 |
| nibalizer | openstackci_apache gives us a name that can be grepped on 'openstackci' 'apache' and can be split on '_' | 00:12 |
| nibalizer | and most of the interactions with it will be in the puppet code where the 'puppet-' prefix won't be visible | 00:12 |
| mordred | what about just calling it samuel | 00:12 |
| mordred | let's bring back meaningless names!!!! | 00:13 |
| waynr | this seems like a long shot but is there any way to interpolate the names of jobs in a job group in another part of the configuration? | 00:13 |
| *** stevemar has joined #openstack-infra | 00:13 | |
| *** stevemar is now known as notstevemar | 00:13 | |
| waynr | (jenkins job builder) | 00:14 |
| waynr | i have a bunch of jobs that use the same template whose names are of course parameterized--it'd be nice not to have to put them into a buildstep trigger field manually | 00:14 |
| lifeless | dhellmann: have you tried virtualenvwrappers with jython? | 00:15 |
| lifeless | dhellmann: I get, at the end of mkvirtualenv, | 00:15 |
| lifeless | ERROR: None | 00:15 |
| lifeless | ERROR: this script requires Python 2.6 or greater. | 00:15 |
| lifeless | ERROR: None | 00:15 |
| lifeless | ERROR: this script requires Python 2.6 or greater. | 00:15 |
| lifeless | erm, just one of those :) | 00:15 |
| *** oomichi has joined #openstack-infra | 00:17 | |
| *** unicell has joined #openstack-infra | 00:20 | |
| *** weshay has joined #openstack-infra | 00:24 | |
| *** msabramo has joined #openstack-infra | 00:24 | |
| ianw | 10 is the new cool version to be? puppet-apache10 | 00:29 |
| fungi | version 10 is so done even openstack has one. this needs to go to eleven | 00:29 |
| *** zaphod42_ has joined #openstack-infra | 00:32 | |
| *** achanda has quit IRC | 00:32 | |
| *** zaphod42_ has joined #openstack-infra | 00:32 | |
| reed | I think Samuel is a good name: In American the meaning of the name Samuel is: Heard of God; asked of God. (source:google) | 00:32 |
| *** achanda has joined #openstack-infra | 00:33 | |
| *** amcrn has quit IRC | 00:33 | |
| *** ayoung-dadmode is now known as ayoung | 00:33 | |
| clarkb | so rax-iad is predominantly preferred for 2-node builds | 00:35 |
| *** zaphod42 has quit IRC | 00:35 | |
| clarkb | I think the allocator might be built so that this happens | 00:35 |
| lifeless | booyah | 00:35 |
| lifeless | unittest2 0.6.0 just released, wheel and sdist formats. If I've broken the world, please tell me soon:) | 00:35 |
| clarkb | we should know as soon as bandersnatch can sync them :) | 00:36 |
| *** zaphod42_ has quit IRC | 00:37 | |
| *** achanda has quit IRC | 00:37 | |
| openstackgerrit | Angus Lees proposed a change to openstack/requirements: Allow newer versions of pylint https://review.openstack.org/131013 | 00:39 |
| openstackgerrit | Angus Lees proposed a change to openstack/requirements: Upgrade to pylint >=1.3.0 https://review.openstack.org/131310 | 00:39 |
| *** rkukura has quit IRC | 00:39 | |
| *** rmcall has quit IRC | 00:41 | |
| *** yamamoto has joined #openstack-infra | 00:41 | |
| anteaya | fungi: puppet-opache ++ | 00:42 |
| anteaya | mordred: samuel + | 00:43 |
| fungi | and failing that, i also like puppet-gazpacho | 00:43 |
| anteaya | nibalizer: ewwww, but you do make a point | 00:43 |
| anteaya | I also like puppet-gazpacho | 00:43 |
| anteaya | you can grep on pach | 00:44 |
| *** pc_m_ has quit IRC | 00:45 | |
| fungi | wow, i didn't realize that appledouble encoding dates all the way back to a/ux | 00:45 |
| fungi | the things you learn on the 'pedia | 00:45 |
| *** yamamoto has quit IRC | 00:46 | |
| *** david-lyle has joined #openstack-infra | 00:46 | |
| *** david-lyle has quit IRC | 00:51 | |
| *** sarob has joined #openstack-infra | 00:51 | |
| *** r1chardj0n3s_afk is now known as r1chardj0n3s | 00:51 | |
| *** sarob has quit IRC | 00:52 | |
| *** sarob has joined #openstack-infra | 00:53 | |
| *** yamahata has joined #openstack-infra | 00:53 | |
| *** koolhead17 has quit IRC | 00:54 | |
| *** sarob has quit IRC | 00:58 | |
| lifeless | fungi: aieeee | 01:01 |
| lifeless | fungi: flashbacks | 01:01 |
| *** rkukura has joined #openstack-infra | 01:04 | |
| *** Iron_Chef has joined #openstack-infra | 01:07 | |
| *** mriedem has quit IRC | 01:08 | |
| Iron_Chef | Hi openstack :) | 01:09 |
| *** freemanbrandon has joined #openstack-infra | 01:10 | |
| anteaya | hello Iron_Chef | 01:12 |
| *** Longgeek has joined #openstack-infra | 01:13 | |
| *** baoli has joined #openstack-infra | 01:16 | |
| *** julim has quit IRC | 01:17 | |
| *** Longgeek has quit IRC | 01:17 | |
| *** Iron_Chef has left #openstack-infra | 01:18 | |
| *** amitgandhinz has joined #openstack-infra | 01:24 | |
| *** signed8bit has joined #openstack-infra | 01:26 | |
| clarkb | well I cannot get 2-node builds in not rax | 01:27 |
| clarkb | so I am just going to wait on that for when nodepool is busy and allocator will possibly do what I want | 01:27 |
| *** david-lyle has joined #openstack-infra | 01:31 | |
| anteaya | clarkb: :( | 01:32 |
| ianw | clarkb: sorry, i had it in my head that puppet would restart nodepool on the git change. but it's nodepool that checks itself for updated config files | 01:33 |
| ianw | too many automagically moving pieces :) | 01:34 |
| fungi | clarkb: or tweak the config temporarily to only build them in hpcloud long enough to build one, then switch it back | 01:35 |
| jogo | clarkb: you are testing out the two node build stuff? | 01:35 |
| jogo | clarkb: is there an etherpad or something tracking the pieces of that | 01:35 |
| *** david-lyle has quit IRC | 01:36 | |
| clarkb | jogo: no etherpad, just the change and its children | 01:36 |
| clarkb | jogo: it works fine | 01:36 |
| clarkb | trying a specific variation of it (or attempting to) | 01:36 |
| clarkb | fungi: that is, hrm I should totally do that | 01:36 |
| clarkb | I am going to do that now | 01:36 |
| fungi | slightly faster than rolling loaded dice over and over | 01:37 |
| jogo | clarkb: cool, so we can run any tempest config with this? | 01:40 |
| clarkb | jogo: I mean I don't expect them to all work | 01:40 |
| clarkb | right now its just tempest + nova net and tempest + neutron and not full tempest best I can tell | 01:40 |
| jogo | clarkb: don't expect all configs to work or not TEMPEST_FULL | 01:40 |
| clarkb | both | 01:40 |
| jogo | clarkb: yeah your probably right, TEMPEST_FULL may have some kinks to sort out before we can try adding things like live migration | 01:41 |
| clarkb | woot fungi++ it is building an hpcloud pair now | 01:42 |
| clarkb | I will recheck as soon as I have at least one set of nodes ready | 01:42 |
| jogo | clarkb: have you gotten far enough to run tempest on a two node? | 01:43 |
| jogo | hmm I wonder if I can simulate with sdague's two node devstack vagrant thing | 01:43 |
| clarkb | yes afazekas_beer has had tempest running on two nodes for months | 01:44 |
| clarkb | jogo: https://review.openstack.org/#/c/106043/22 | 01:44 |
| jogo | clarkb: ohh aiopcpu | 01:44 |
| jogo | wow that is super cryptic | 01:45 |
| jogo | might as well call it ieaiaio | 01:45 |
| jogo | which is a system of a down song | 01:45 |
| clarkb | all in one plus cpu | 01:45 |
| clarkb | not that I am a fan | 01:45 |
| clarkb | but meh important part is making it go | 01:45 |
| clarkb | and it goes | 01:45 |
| jogo | clarkb: agreed | 01:46 |
| * jogo listens to system of a down | 01:46 | |
| clarkb | system of a down will get me too excited | 01:46 |
| *** signed8bit has quit IRC | 01:47 | |
| jogo | clarkb: haha | 01:48 |
| jogo | so this isn't tempest-full | 01:48 |
| clarkb | no I don't think so | 01:48 |
| jogo | would it make sense to get one of those jobs added to so we can work on sorting out the kinks in that in parallel? | 01:49 |
| clarkb | I don't understand the question we have two of them | 01:49 |
| jogo | yeah that didn't make sense, let me try again | 01:49 |
| *** Ryan_Lane has quit IRC | 01:49 | |
| openstackgerrit | A change was merged to openstack-dev/hacking: Add hacking check H105: don't use author tags https://review.openstack.org/126992 | 01:50 |
| jogo | thoughts on adding a tempest-dsvm-full-aiopcpu job, and maybe a tempest-dsvm-neutron-full-aiopcpu as well, to shake out any tempest related issues in parallel to the current work | 01:51 |
| clarkb | I would like to get the base code merged first so that we can run those jobs against arbitrary changes | 01:51 |
| *** weshay has quit IRC | 01:51 | |
| clarkb | I think we are really close at this point (I am testing what I believe is the last point of contention) | 01:52 |
| *** amitgandhinz has quit IRC | 01:52 | |
| jogo | clarkb: cool, I wasn't sure how far away we were since https://review.openstack.org/#/c/106043/22 has a few TODOs in the commit message | 01:53 |
| clarkb | I think we can work on those todos in parallel with expanding testing | 01:53 |
| clarkb | https://jenkins02.openstack.org/job/check-tempest-dsvm-aiopcpu/14/ finally that is running on hpcloud | 01:55 |
| clarkb | once the other job starts I will edit the nodepool.yaml and reenable puppet on nodepool.o.o | 01:55 |
| clarkb | jogo: also the neutron job takes much longer. Maybe it is running full? | 01:56 |
| clarkb | nope | 01:56 |
| clarkb | just slower I guess | 01:56 |
| *** nosnos has joined #openstack-infra | 01:59 | |
| clarkb | I have returned nodepool to its normal state. puppet is enabled and yaml file matches git | 02:04 |
| *** tsg has joined #openstack-infra | 02:06 | |
| lifeless | :q | 02:07 |
| clarkb | there are unwritten changes to he buffer | 02:07 |
| jogo | :q! | 02:09 |
| *** freemanbrandon has quit IRC | 02:11 | |
| anteaya | jogo: you can never leave | 02:12 |
| *** baoli has quit IRC | 02:13 | |
| *** rfolco has quit IRC | 02:19 | |
| *** VijayTripathi has quit IRC | 02:19 | |
| *** VijayTripathi has joined #openstack-infra | 02:20 | |
| *** bhuvan has quit IRC | 02:20 | |
| jogo | anteaya: noooo | 02:21 |
| jogo | clarkb: how many n-cpus should that aiopcpu have? | 02:21 |
| jogo | 2? | 02:21 |
| jogo | I don't see two logs | 02:21 |
| clarkb | 2 | 02:21 |
| clarkb | jogo: I think that is the todo fix logging | 02:21 |
| clarkb | jogo: but there should be two let me look | 02:22 |
| *** bdpayne has joined #openstack-infra | 02:22 | |
| *** bdpayne has quit IRC | 02:22 | |
| clarkb | http://logs.openstack.org/97/131297/1/experimental/check-tempest-dsvm-aiopcpu/692a986/logs/104.130.6.108/ and http://logs.openstack.org/97/131297/1/experimental/check-tempest-dsvm-aiopcpu/692a986/logs/ each have an n-cpu log | 02:22 |
| jogo | ohhh I see | 02:23 |
| jogo | I didn't realize that is how it was split up | 02:23 |
| jogo | that isn't very obvious | 02:23 |
| jogo | logs and logs/$IP/ | 02:23 |
| *** notstevemar has quit IRC | 02:24 | |
| *** VijayTripathi has quit IRC | 02:24 | |
| *** emagana has joined #openstack-infra | 02:26 | |
| *** ivar-laz_ has joined #openstack-infra | 02:30 | |
| clarkb | we do have volume related failures when using the floating ips in hpcloud | 02:31 |
| clarkb | not sure yet if due to floating IPs or hpcloud /me checks the base change | 02:31 |
| *** rfolco has joined #openstack-infra | 02:33 | |
| *** ivar-lazzaro has quit IRC | 02:34 | |
| *** david-lyle has joined #openstack-infra | 02:36 | |
| *** ivar-laz_ has quit IRC | 02:39 | |
| *** david-lyle has quit IRC | 02:41 | |
| *** dims_ has quit IRC | 02:44 | |
| *** dims__ has joined #openstack-infra | 02:47 | |
| jogo | if you could run E-R against the logs it would be able to tell you pretty quickly | 02:48 |
| jogo | as I don't think we have any unknown failures | 02:48 |
| jogo | I spoke too soon, we didn't at lunch time | 02:48 |
| jogo | http://logs.openstack.org/77/130077/2/gate/gate-tempest-dsvm-neutron-heat-slow/5a81254/logs/devstack-gate-setup-workspace-new.txt.gz#_2014-10-28_00_33_31_268 | 02:49 |
| jogo | well this is fun GnuTLS recv error (-9): A TLS packet with unexpected length was received. | 02:49 |
| *** dims__ has quit IRC | 02:49 | |
| clarkb | well this is completely brand new | 02:49 |
| clarkb | it would likely not be able to tell me if its node slowness or NAT or some other flavor variety between clouds | 02:50 |
| jesusaurus | heh, thats the same error i get from glowing bear when i try to connect my weechat relay </not_helpful> | 02:50 |
| jogo | clarkb: wow 13 hits in 24 hours | 02:51 |
| * jogo is afraid | 02:51 | |
| jogo | hmm a lot of them on successfull runs | 02:52 |
| jogo | oh right we do a retry | 02:52 |
| *** patrickeast has quit IRC | 02:54 | |
| *** jerryz has joined #openstack-infra | 02:54 | |
| *** liusheng has quit IRC | 02:55 | |
| mwhudson | i'm seeing tests fail (locally) with ResponseNotReady | 02:55 |
| mwhudson | frustratinly i know i've seen this before, but can't remember what it means :( | 02:55 |
| *** liusheng has joined #openstack-infra | 02:55 | |
| *** yamamoto_ has joined #openstack-infra | 02:56 | |
| *** sarob has joined #openstack-infra | 02:56 | |
| *** timrc is now known as timrc-afk | 02:59 | |
| *** sarob has quit IRC | 03:01 | |
| *** amitgandhinz has joined #openstack-infra | 03:03 | |
| *** emagana has quit IRC | 03:03 | |
| clarkb | ok the base aiopcpu change works on hpcloud | 03:04 |
| clarkb | jeblair: so the fixed IP thing may actually be required | 03:04 |
| *** emagana has joined #openstack-infra | 03:04 | |
| *** spzala has joined #openstack-infra | 03:04 | |
| *** alexpilotti has quit IRC | 03:04 | |
| *** alexpilotti has joined #openstack-infra | 03:06 | |
| *** alexpilotti has quit IRC | 03:07 | |
| *** amitgandhinz has quit IRC | 03:07 | |
| *** unicell has quit IRC | 03:08 | |
| *** emagana has quit IRC | 03:09 | |
| *** mmaglana has joined #openstack-infra | 03:10 | |
| *** alexpilotti has joined #openstack-infra | 03:10 | |
| *** sunil_ has quit IRC | 03:11 | |
| *** alexpilotti has quit IRC | 03:11 | |
| *** alexpilotti has joined #openstack-infra | 03:12 | |
| *** alexpilotti has quit IRC | 03:12 | |
| *** alexpilotti has joined #openstack-infra | 03:14 | |
| *** armax has quit IRC | 03:14 | |
| *** spzala has quit IRC | 03:15 | |
| *** armax has joined #openstack-infra | 03:15 | |
| *** liusheng has quit IRC | 03:15 | |
| *** unicell has joined #openstack-infra | 03:16 | |
| *** alexpilotti has quit IRC | 03:16 | |
| *** liusheng has joined #openstack-infra | 03:17 | |
| *** ChuckC___ has joined #openstack-infra | 03:18 | |
| *** Longgeek has joined #openstack-infra | 03:21 | |
| *** freemanbrandon has joined #openstack-infra | 03:22 | |
| *** camunoz is now known as camunoz_mtg | 03:25 | |
| *** emagana has joined #openstack-infra | 03:25 | |
| *** Longgeek has quit IRC | 03:26 | |
| *** nelsnelson has quit IRC | 03:26 | |
| *** freemanbrandon has quit IRC | 03:26 | |
| *** ChuckC___ has quit IRC | 03:27 | |
| openstackgerrit | Brian Saville proposed a change to openstack-infra/jenkins-job-builder: Add support for multiple port allocations https://review.openstack.org/131328 | 03:27 |
| *** harlowja is now known as harlowja_away | 03:27 | |
| *** doug-fish has joined #openstack-infra | 03:29 | |
| openstackgerrit | A change was merged to openstack/requirements: Update to newer requests-mock https://review.openstack.org/131064 | 03:30 |
| *** harlowja_away is now known as harlowja | 03:30 | |
| *** tomoe has joined #openstack-infra | 03:32 | |
| *** mbacchi has quit IRC | 03:32 | |
| *** MaxV has joined #openstack-infra | 03:33 | |
| *** mattoliverau has quit IRC | 03:35 | |
| *** zns has joined #openstack-infra | 03:35 | |
| *** mattoliverau has joined #openstack-infra | 03:35 | |
| *** rmcall has joined #openstack-infra | 03:35 | |
| openstackgerrit | Brian Saville proposed a change to openstack-infra/jenkins-job-builder: Add doxygen builder support https://review.openstack.org/131330 | 03:35 |
| *** nosnos has quit IRC | 03:36 | |
| *** nosnos has joined #openstack-infra | 03:37 | |
| *** MaxV has quit IRC | 03:37 | |
| *** mattoliverau has quit IRC | 03:38 | |
| *** miqui has quit IRC | 03:38 | |
| *** mattoliverau has joined #openstack-infra | 03:39 | |
| openstackgerrit | Brian Saville proposed a change to openstack-infra/jenkins-job-builder: Add SVN credential support https://review.openstack.org/131332 | 03:39 |
| *** pcrews has quit IRC | 03:41 | |
| *** nosnos has quit IRC | 03:41 | |
| *** terryw has quit IRC | 03:43 | |
| openstackgerrit | Brian Saville proposed a change to openstack-infra/jenkins-job-builder: Add support for multiple port allocations https://review.openstack.org/131335 | 03:46 |
| *** otherwiseguy has joined #openstack-infra | 03:50 | |
| *** dims__ has joined #openstack-infra | 03:50 | |
| *** VijayTripathi has joined #openstack-infra | 03:51 | |
| *** VijayTripathi1 has joined #openstack-infra | 03:54 | |
| *** dims__ has quit IRC | 03:55 | |
| *** VijayTripathi1 has quit IRC | 03:57 | |
| *** VijayTripathi has quit IRC | 03:57 | |
| *** achanda has joined #openstack-infra | 04:01 | |
| *** ayoung has quit IRC | 04:04 | |
| *** zns has quit IRC | 04:06 | |
| *** david-lyle has joined #openstack-infra | 04:06 | |
| *** zns has joined #openstack-infra | 04:06 | |
| *** gyee has joined #openstack-infra | 04:11 | |
| *** AzherKhan has joined #openstack-infra | 04:14 | |
| *** nosnos has joined #openstack-infra | 04:15 | |
| *** Khaazher has joined #openstack-infra | 04:16 | |
| *** Khaazher has quit IRC | 04:17 | |
| *** armax has quit IRC | 04:20 | |
| *** unicell has quit IRC | 04:22 | |
| *** MaxV has joined #openstack-infra | 04:34 | |
| *** unicell has joined #openstack-infra | 04:36 | |
| *** otherwiseguy has quit IRC | 04:37 | |
| *** MaxV has quit IRC | 04:38 | |
| *** links has joined #openstack-infra | 04:39 | |
| *** stevemar has joined #openstack-infra | 04:44 | |
| *** sarob has joined #openstack-infra | 04:54 | |
| *** mrmartin has joined #openstack-infra | 04:54 | |
| *** achanda has quit IRC | 04:54 | |
| *** achanda has joined #openstack-infra | 04:56 | |
| *** sarob has quit IRC | 04:58 | |
| *** rmcall has quit IRC | 05:02 | |
| *** sdake_ has joined #openstack-infra | 05:04 | |
| *** nelsnelson has joined #openstack-infra | 05:06 | |
| *** tomoe has quit IRC | 05:08 | |
| *** tomoe has joined #openstack-infra | 05:08 | |
| *** tomoe has quit IRC | 05:13 | |
| *** gyee has quit IRC | 05:19 | |
| *** fifieldt__ has quit IRC | 05:20 | |
| *** ildikov has quit IRC | 05:24 | |
| *** yfried has quit IRC | 05:30 | |
| *** tomoe has joined #openstack-infra | 05:32 | |
| *** nelsnelson has quit IRC | 05:32 | |
| jogo | did something happen to the git servers | 05:33 |
| jogo | there has been a spate of failures | 05:33 |
| *** mrmartin has quit IRC | 05:33 | |
| *** talluri has joined #openstack-infra | 05:33 | |
| *** mmaglana has quit IRC | 05:36 | |
| *** tomoe has quit IRC | 05:36 | |
| *** stevemar has quit IRC | 05:37 | |
| *** harlowja is now known as harlowja_away | 05:37 | |
| clarkb | links? | 05:38 |
| clarkb | should have a bot demand links | 05:38 |
| *** tomoe has joined #openstack-infra | 05:38 | |
| *** tsg has quit IRC | 05:38 | |
| *** Longgeek has joined #openstack-infra | 05:39 | |
| links | clarkb, ^ was that intended for me ? | 05:39 |
| clarkb | links no for jogo. sorry | 05:39 |
| links | :) okay | 05:39 |
| *** emagana has quit IRC | 05:42 | |
| jogo | clarkb: http://logs.openstack.org/66/130966/2/gate/gate-tempest-dsvm-neutron-full/baf4264/logs/devstack-gate-setup-workspace-new.txt.gz#_2014-10-28_00_14_10_252 | 05:47 |
| jogo | http://logs.openstack.org/77/130077/2/gate/gate-tempest-dsvm-neutron-heat-slow/5a81254/logs/devstack-gate-setup-workspace-new.txt.gz#_2014-10-28_00_33_31_268 | 05:47 |
| jogo | and I thought there was another but can't seem to find it right now | 05:48 |
| *** chandankumar has joined #openstack-infra | 05:48 | |
| clarkb | I think that is new | 05:48 |
| clarkb | we dont https on the mirrors | 05:48 |
| clarkb | oh wait | 05:49 |
| clarkb | its late. we do | 05:49 |
| clarkb | pypi mirror is no https. git does | 05:49 |
| *** reed has quit IRC | 05:49 | |
| lifeless | we should totally https up the pypi mirrors | 05:52 |
| *** sdake_ has quit IRC | 05:52 | |
| *** loki184 has joined #openstack-infra | 05:53 | |
| *** r1chardj0n3s is now known as r1chardj0n3s_afk | 05:58 | |
| *** koolhead17 has joined #openstack-infra | 05:59 | |
| *** yamamoto_ has quit IRC | 06:01 | |
| *** yamahata has quit IRC | 06:06 | |
| *** achanda has quit IRC | 06:08 | |
| *** afazekas_drunk is now known as afazekas | 06:08 | |
| *** ChuckC___ has joined #openstack-infra | 06:15 | |
| *** nikil89_ has joined #openstack-infra | 06:16 | |
| *** yfried has joined #openstack-infra | 06:16 | |
| *** jerryz has quit IRC | 06:17 | |
| *** ildikov has joined #openstack-infra | 06:18 | |
| *** david-lyle has quit IRC | 06:20 | |
| *** ChuckC___ has quit IRC | 06:21 | |
| *** talluri has quit IRC | 06:23 | |
| *** ivar-lazzaro has joined #openstack-infra | 06:24 | |
| *** talluri has joined #openstack-infra | 06:26 | |
| *** ivar-lazzaro has quit IRC | 06:32 | |
| *** ivar-lazzaro has joined #openstack-infra | 06:33 | |
| *** sdake_ has joined #openstack-infra | 06:34 | |
| *** akamyshnikova_ has quit IRC | 06:34 | |
| *** yamahata has joined #openstack-infra | 06:34 | |
| *** MaxV has joined #openstack-infra | 06:35 | |
| *** unicell has quit IRC | 06:37 | |
| *** ashaeron has joined #openstack-infra | 06:38 | |
| *** ivar-lazzaro has quit IRC | 06:39 | |
| *** MaxV has quit IRC | 06:40 | |
| *** sdake_ has quit IRC | 06:43 | |
| *** dims__ has joined #openstack-infra | 06:44 | |
| *** HeOS has joined #openstack-infra | 06:45 | |
| *** sdake_ has joined #openstack-infra | 06:46 | |
| *** talluri has quit IRC | 06:46 | |
| *** kingia has joined #openstack-infra | 06:47 | |
| *** kingia is now known as IaKing | 06:47 | |
| *** talluri has joined #openstack-infra | 06:47 | |
| *** dims__ has quit IRC | 06:49 | |
| *** david-lyle has joined #openstack-infra | 06:50 | |
| *** tomoe has quit IRC | 06:51 | |
| *** david-lyle has quit IRC | 06:54 | |
| *** HeOS has quit IRC | 06:54 | |
| *** k4n0 has joined #openstack-infra | 06:57 | |
| *** rakhmerov_ is now known as rakhmerov | 06:58 | |
| *** e0ne has joined #openstack-infra | 07:01 | |
| *** jerryz has joined #openstack-infra | 07:05 | |
| *** fifieldt has joined #openstack-infra | 07:06 | |
| *** achanda has joined #openstack-infra | 07:09 | |
| *** unicell has joined #openstack-infra | 07:14 | |
| *** cody-somerville has quit IRC | 07:16 | |
| *** cody-somerville has joined #openstack-infra | 07:16 | |
| *** _nadya_ has joined #openstack-infra | 07:16 | |
| *** achanda has quit IRC | 07:17 | |
| *** oomichi has quit IRC | 07:19 | |
| *** david-lyle has joined #openstack-infra | 07:21 | |
| *** david-lyle has quit IRC | 07:25 | |
| *** rushiagr_away is now known as rushiagr | 07:27 | |
| *** e0ne has quit IRC | 07:29 | |
| *** talluri has quit IRC | 07:30 | |
| *** tomoe has joined #openstack-infra | 07:35 | |
| *** leopoldj has joined #openstack-infra | 07:36 | |
| *** e0ne has joined #openstack-infra | 07:38 | |
| *** e0ne has quit IRC | 07:40 | |
| *** yamamoto_ has joined #openstack-infra | 07:40 | |
| *** andreykurilin_ has joined #openstack-infra | 07:41 | |
| *** sdake_ has quit IRC | 07:41 | |
| *** koolhead17 has quit IRC | 07:43 | |
| *** MaxV has joined #openstack-infra | 07:47 | |
| openstackgerrit | A change was merged to openstack-infra/project-config: Add the API working group guidelines to specs.o.org https://review.openstack.org/130701 | 07:47 |
| *** links has quit IRC | 07:48 | |
| *** e0ne has joined #openstack-infra | 07:50 | |
| *** e0ne has quit IRC | 07:52 | |
| openstackgerrit | Ian Wienand proposed a change to openstack-infra/project-config: Add openstack-infra/puppet-openstackci_apache https://review.openstack.org/129496 | 07:55 |
| *** _nadya_ has quit IRC | 07:55 | |
| *** mrmartin has joined #openstack-infra | 07:58 | |
| *** viglesias has quit IRC | 08:09 | |
| *** HeOS has joined #openstack-infra | 08:10 | |
| openstackgerrit | Magnus Lundin proposed a change to openstack-infra/jenkins-job-builder: Added support for Rich Text Publisher Plugin https://review.openstack.org/130455 | 08:11 |
| *** koolhead17 has joined #openstack-infra | 08:15 | |
| *** yfried is now known as yfried|afk | 08:17 | |
| *** yfried|afk is now known as yfried | 08:19 | |
| *** david-lyle has joined #openstack-infra | 08:21 | |
| *** yamahata has quit IRC | 08:22 | |
| *** david-lyle has quit IRC | 08:26 | |
| *** slashme has quit IRC | 08:26 | |
| *** talluri has joined #openstack-infra | 08:31 | |
| *** koolhead17 has quit IRC | 08:31 | |
| *** talluri has quit IRC | 08:35 | |
| *** doude has joined #openstack-infra | 08:36 | |
| *** rushiagr is now known as rushiagr_away | 08:40 | |
| *** jgallard has joined #openstack-infra | 08:40 | |
| *** Guest74700 has joined #openstack-infra | 08:45 | |
| *** talluri has joined #openstack-infra | 08:46 | |
| *** yamamoto_ has quit IRC | 08:51 | |
| *** jpich has joined #openstack-infra | 08:51 | |
| *** Longgeek has quit IRC | 08:54 | |
| *** Longgeek has joined #openstack-infra | 08:54 | |
| *** andreykurilin_ has quit IRC | 08:56 | |
| *** slashme has joined #openstack-infra | 08:57 | |
| *** parimala has joined #openstack-infra | 09:00 | |
| *** nikil89_ is now known as naggappan | 09:00 | |
| *** naggappan has left #openstack-infra | 09:00 | |
| *** naggappan has joined #openstack-infra | 09:01 | |
| *** links has joined #openstack-infra | 09:01 | |
| *** hdd has joined #openstack-infra | 09:02 | |
| *** liusheng has quit IRC | 09:02 | |
| parimala | Hello All, | 09:03 |
| naggappan | Hi is it possible to set devstack-gate to install devstack with my neutron repository insted of standard neutron repository? I have tried to set ZUUL_PROJECT=naggappan/test-neutron in jenkins job. Currently in my CI system if there any any changes in test-neutron repository then the zuul triggers a job and everything works fine. But it takes the standard neutron repository insted of my repositry . | 09:03 |
| *** vdo has joined #openstack-infra | 09:04 | |
| parimala | when I am doing "tox -e checklinks", it is giving error as the link is not available , so what should I do? | 09:05 |
| parimala | any clue? | 09:05 |
| parimala | it is the redhat stigs link which is broken | 09:08 |
| *** achanda has joined #openstack-infra | 09:15 | |
| *** derekh has joined #openstack-infra | 09:16 | |
| *** rcarrillocruz has quit IRC | 09:17 | |
| *** links has quit IRC | 09:17 | |
| *** achanda has quit IRC | 09:19 | |
| ttx | morganfainberg: no you can't. If it's really so important that people can't be trusted to cut and paste it, I can turn them into links from within sched itself | 09:20 |
| *** rcarrillocruz has joined #openstack-infra | 09:20 | |
| morganfainberg | ttx, <a ref> works | 09:20 |
| ttx | annegentle: the pods are at Le Meridien (like all the design summit) | 09:20 |
| morganfainberg | ttx, btw. | 09:20 |
| ttx | morganfainberg: heh. Sounds like a bug :) | 09:21 |
| morganfainberg | ttx, :) | 09:21 |
| ttx | maybe not actually. It allows a subset of "safe" html | 09:21 |
| morganfainberg | ttx, ah. yeah href is pretty *safe*-ish | 09:21 |
| morganfainberg | ttx, i wasn't going to ask you to go through and make them all links, if i couldn't do it via the tools copy-paste-it-was-to-be | 09:22 |
| *** david-lyle has joined #openstack-infra | 09:22 | |
| morganfainberg | oh gah, it's late. | 09:22 |
| morganfainberg | i need to sleep | 09:22 |
| ttx | or early | 09:22 |
| ttx | morganfainberg: if you're training for the new timezone, you're doing it wrong | 09:23 |
| *** viglesias has joined #openstack-infra | 09:23 | |
| morganfainberg | ttx, nope. was re-installing my desktop because of a complete fubar that nuked something in a way i couldn't fix. | 09:23 |
| morganfainberg | otherwise i'd be sleeping already | 09:23 |
| ttx | rule #124 of sumits: do not upgrade or reinstall laptop in the week before | 09:24 |
| *** eglynn-officeafk is now known as eglynn-office | 09:24 | |
| morganfainberg | ttx, i think i'd rather not upgrade / reinstall anytime if i could avoid it | 09:25 |
| *** david-lyle has quit IRC | 09:26 | |
| *** jamielennox_ has joined #openstack-infra | 09:28 | |
| *** parimala has quit IRC | 09:29 | |
| *** jp_at_hp has joined #openstack-infra | 09:29 | |
| *** ZZelle has quit IRC | 09:29 | |
| *** ZZelle has joined #openstack-infra | 09:30 | |
| *** IvanBerezovskiy has joined #openstack-infra | 09:34 | |
| *** pelix has joined #openstack-infra | 09:36 | |
| *** andreaf has joined #openstack-infra | 09:36 | |
| *** andreaf_ has joined #openstack-infra | 09:39 | |
| *** zns has quit IRC | 09:43 | |
| *** brad__ has joined #openstack-infra | 09:48 | |
| *** Longgeek has quit IRC | 09:51 | |
| *** jamielennox_ has quit IRC | 09:53 | |
| *** slashme has quit IRC | 09:54 | |
| *** teran has quit IRC | 09:59 | |
| *** jedimike has joined #openstack-infra | 10:02 | |
| *** pc_m_ has joined #openstack-infra | 10:02 | |
| *** koolhead17 has joined #openstack-infra | 10:02 | |
| *** pc_m_ has quit IRC | 10:03 | |
| *** pc_m_ has joined #openstack-infra | 10:03 | |
| *** Longgeek has joined #openstack-infra | 10:04 | |
| *** _nadya_ has joined #openstack-infra | 10:06 | |
| *** hdd has quit IRC | 10:08 | |
| *** _nadya_ has quit IRC | 10:08 | |
| *** hdd has joined #openstack-infra | 10:11 | |
| *** brad__ has quit IRC | 10:11 | |
| *** dizquierdo has joined #openstack-infra | 10:14 | |
| *** rlandy has joined #openstack-infra | 10:16 | |
| *** mkerrin has joined #openstack-infra | 10:16 | |
| *** slashme has joined #openstack-infra | 10:18 | |
| *** dims__ has joined #openstack-infra | 10:19 | |
| *** yfried is now known as yfried|afk | 10:20 | |
| *** david-lyle has joined #openstack-infra | 10:23 | |
| *** dims__ has quit IRC | 10:23 | |
| *** koolhead17 has quit IRC | 10:27 | |
| *** david-lyle has quit IRC | 10:28 | |
| openstackgerrit | A change was merged to openstack-infra/jenkins-job-builder: Support additional events for HipChat plugin https://review.openstack.org/129760 | 10:29 |
| sdague | jhesketh: you up? | 10:32 |
| sdague | ttx: can you kick the following into the req proj - https://review.openstack.org/#/c/131184/ ? | 10:33 |
| *** e0ne has joined #openstack-infra | 10:34 | |
| *** markmcclain has joined #openstack-infra | 10:36 | |
| ttx | sdague: looking | 10:36 |
| *** slashme has quit IRC | 10:37 | |
| ttx | sdague: approved | 10:37 |
| sdague | thanks | 10:37 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/system-config: only show most recent results per pipeline https://review.openstack.org/131249 | 10:37 |
| *** teran has joined #openstack-infra | 10:39 | |
| *** loki184 has quit IRC | 10:41 | |
| *** koolhead17 has joined #openstack-infra | 10:41 | |
| *** _nadya_ has joined #openstack-infra | 10:41 | |
| *** dims__ has joined #openstack-infra | 10:43 | |
| ianw | sdague: got 2 minutes to sync up on bashate? it's causing me some grief | 10:44 |
| *** teran_ has joined #openstack-infra | 10:44 | |
| sdague | sure, I'm on cup off coffee 1, so be forewarned :) | 10:44 |
| *** teran has quit IRC | 10:44 | |
| ianw | sdague: TOT doesn't work on devstack ATM, there are several reviews out | 10:44 |
| sdague | ok, I haven't found my way back to those yet | 10:45 |
| sdague | let me go look | 10:45 |
| ianw | sdague: i think we should beat it into shape any which way and do a release | 10:45 |
| sdague | sure, is there a particular sequence the patches should go in in your opinion? | 10:46 |
| ianw | i now have a devstack review that is probably incorrectly triggering bashate failures : https://review.openstack.org/#/c/131334/ | 10:46 |
| *** koolhead17 has quit IRC | 10:46 | |
| ianw | but see no way to skip that | 10:46 |
| ianw | sdague: i *think* the ones current causing devstack to fail are https://review.openstack.org/127052 & https://review.openstack.org/129771 | 10:47 |
| ianw | merging the `` was probably the wrong thing to do, we'll need to add an ignore to devstack for that | 10:48 |
| ianw | the `` check i mean | 10:48 |
| ianw | i feel like we should have the ability to turn it down to a warning, at least then we could be fixing it and see from the logs | 10:48 |
| *** teran_ has quit IRC | 10:48 | |
| ianw | https://review.openstack.org/129761 (add ability to ignore in-file) might be useful for situations such as the false-positive. haven't got to the comments on that yet | 10:49 |
| *** dmakogon_ is now known as denis_makogon | 10:49 | |
| ianw | https://review.openstack.org/128807 (Strip comments before checks) is also required for devstack TOT I think | 10:49 |
| sdague | https://review.openstack.org/#/c/127052/5 seems like a weird way to do that | 10:49 |
| sdague | ianw: https://review.openstack.org/#/c/128807/ is now in merge conflict, can you rebase? | 10:50 |
| sdague | https://review.openstack.org/#/c/128807/4 seems like a good idea, and I think it removes the need for 127052 | 10:51 |
| *** AzherKhan has quit IRC | 10:51 | |
| *** jklare has joined #openstack-infra | 10:52 | |
| ianw | sdague: 128807 should be rebased | 10:52 |
| sdague | hmmm... have we seen mrodden on reviews recently? | 10:53 |
| openstackgerrit | Sergey Skripnick proposed a change to openstack-infra/project-config: Add hook for building rally docker image https://review.openstack.org/131389 | 10:53 |
| *** markmcclain has quit IRC | 10:53 | |
| *** markmcclain has joined #openstack-infra | 10:53 | |
| sdague | so there is another way to do https://review.openstack.org/#/c/131334/ that's less tricksie, let me figure out where I did that | 10:53 |
| ianw | sdague: yeah, can probably abandon 127052 if 128807 goes in | 10:54 |
| sdague | I just fast approved 128807 | 10:54 |
| jklare | hi, can somebody help me with this issue http://logs.openstack.org/11/115711/4/check/gate-openstack-chef-repo/2237e9e/console.html | 10:55 |
| *** talluri has quit IRC | 10:55 | |
| jklare | this should have been fixed with the dep-selector-libgecode gem but apparently its not | 10:55 |
| jklare | and i cant repoduce this locally | 10:55 |
| *** andreaf has quit IRC | 10:55 | |
| *** andreaf_ is now known as andreaf | 10:55 | |
| ianw | sdague: happy for a better way on that expansion thing, that code does my head in | 10:56 |
| sdague | ianw: so... https://github.com/openstack-infra/devstack-gate/blob/master/functions.sh#L285-L286 works without having to subshell the heredoc | 10:57 |
| sdague | I think it's the same thing you are trying to do, right? | 10:57 |
| ianw | sdague: hmm, that doesn't seem to keep quotes correctly | 10:59 |
| *** teran has joined #openstack-infra | 10:59 | |
| ianw | merge_config_file test-quote: failed: | 10:59 |
| ianw | [foo] | 10:59 |
| ianw | foo = foo bar baz != | 10:59 |
| ianw | [foo] | 10:59 |
| ianw | foo = "foo bar" "baz" | 10:59 |
| *** IaKing has quit IRC | 10:59 | |
| ianw | using the "cat" I think does | 10:59 |
| *** tomoe has quit IRC | 10:59 | |
| *** baoli has joined #openstack-infra | 11:00 | |
| *** tomoe has joined #openstack-infra | 11:00 | |
| *** teran_ has joined #openstack-infra | 11:01 | |
| *** teran_ has quit IRC | 11:01 | |
| ianw | well, more the heredoc than the cat i guess | 11:01 |
| sdague | can you give me an example of where this is failing in the code? | 11:01 |
| sdague | like a real instance of the iniset that's breaking, so I can wrap my head around it better | 11:02 |
| *** teran_ has joined #openstack-infra | 11:02 | |
| *** BharatK has joined #openstack-infra | 11:03 | |
| ianw | sdague: pci_alias = {"name":"cisco","vendor_id":"8086","product_id":"10ed"} <- from https://bugs.launchpad.net/devstack/+bug/1374118 | 11:03 |
| uvirtbot | Launchpad bug 1374118 in devstack "use of local.conf doesn't preserve quotes in the string and can't configure multi-line items" [Undecided,In progress] | 11:03 |
| *** teran has quit IRC | 11:03 | |
| *** yfried|afk is now known as yfried | 11:03 | |
| ianw | sdague: i assume you mean breaking with your proposed fix? | 11:03 |
| ianw | sdague: we started single-quoting the argument to allow that type of line to work, however, in the process, broke expansion of variables in the line | 11:04 |
| *** baoli has quit IRC | 11:04 | |
| ianw | sdague: to keep everyone happy, we need to both keep quotes, and expand variables | 11:04 |
| *** tomoe has quit IRC | 11:04 | |
| *** baoli has joined #openstack-infra | 11:05 | |
| sdague | ianw: yeh, that's what I was looking for | 11:05 |
| *** dizquierdo has quit IRC | 11:05 | |
| *** dizquierdo has joined #openstack-infra | 11:06 | |
| *** weshay has joined #openstack-infra | 11:07 | |
| sdague | so... honestly, when did oslo start taking lines like that? | 11:07 |
| ianw | sdague: not sure. I assume it's just naturally heading toward Turing completeness | 11:08 |
| sdague | yeh, so my feeling is we should revert the thing that supported pci_alias | 11:09 |
| sdague | get old working behavior | 11:09 |
| sdague | and have a conversation about nested dicts inside of config options | 11:09 |
| sdague | because I think they are a terrible idea | 11:10 |
| sdague | it's been in olso config for a year, but I still think it's a terrible idea | 11:10 |
| ianw | sdague: ok, that's probably a conversation best started by someone with significant brand recognition :) | 11:12 |
| sdague | sure, but can you propose the reverts? I'll start the config conversation | 11:12 |
| sdague | zomg, it's not even using the olso DictOption | 11:15 |
| sdague | it's building a random custom in option parser | 11:15 |
| ianw | sdague: i won't get to it tonight, can tomorrow .au time | 11:17 |
| sdague | ianw: sounds good | 11:17 |
| sdague | https://github.com/openstack/nova/blob/master/nova/pci/devspec.py#L47-L60 | 11:17 |
| ianw | sdague: ok, TOT bashate runs on devstack with -i E040 (the `` v $() checking) | 11:18 |
| *** talluri has joined #openstack-infra | 11:18 | |
| ianw | sdague: it may have been a mistake to merge that ... maybe we should revert, figure out a way to have warnings, and re-instate it as a warning-level message -- not fatal? | 11:19 |
| sdague | ianw: which thing should we revert now? | 11:22 |
| ianw | sdague: the check in bashate for `foo`, erroring out that it should be $(foo) | 11:22 |
| *** david-lyle has joined #openstack-infra | 11:24 | |
| sdague | ah, gotcha | 11:26 |
| sdague | yeh, that's definitely not true everywhere | 11:26 |
| ekarlso | hey guys, is there a smart way to drop a column in sqla migrate and preserve the indexes after the migration when it's done in unit tests using sqlite ? | 11:26 |
| sdague | is there a revert up for that? | 11:26 |
| *** HeOS has quit IRC | 11:26 | |
| ianw | sdague: so the end game, is that if we have bashate in a release state, we can put out a new version and then merge https://review.openstack.org/#/c/128809/ (Fix up file-matching in bashate tox test) which actually starts running bashate on a whole bunch of stuff that's currently missed | 11:27 |
| *** david-lyle has quit IRC | 11:28 | |
| *** yfried is now known as yfried|afk | 11:31 | |
| *** yfried|afk is now known as yfried | 11:31 | |
| ianw | sdague: https://review.openstack.org/#/c/131396/ (Revert "Add E040: $(COMMAND) preferred over `COMMAND`") <- with that, bashate should pass on devstack with https://review.openstack.org/#/c/128809/ | 11:33 |
| *** aysyd has joined #openstack-infra | 11:34 | |
| *** _nadya_ has quit IRC | 11:35 | |
| *** koolhead17 has joined #openstack-infra | 11:40 | |
| *** cdent has joined #openstack-infra | 11:41 | |
| openstackgerrit | A change was merged to openstack/requirements: implement -s / --soft-update flag https://review.openstack.org/131184 | 11:44 |
| *** HeOS has joined #openstack-infra | 11:46 | |
| *** yamahata has joined #openstack-infra | 11:47 | |
| *** yamamoto has joined #openstack-infra | 11:50 | |
| *** pelix has quit IRC | 11:50 | |
| *** kgiusti has joined #openstack-infra | 11:50 | |
| *** pelix has joined #openstack-infra | 11:52 | |
| *** pelix has quit IRC | 11:53 | |
| *** yamamoto has quit IRC | 11:54 | |
| *** HeOS has quit IRC | 11:55 | |
| *** tomoe has joined #openstack-infra | 11:55 | |
| *** doug-fish1 has joined #openstack-infra | 11:57 | |
| *** koolhead17 has quit IRC | 11:59 | |
| *** yfried is now known as yfried|afk | 12:00 | |
| *** doug-fish has quit IRC | 12:00 | |
| *** markmcclain has quit IRC | 12:00 | |
| *** teran has joined #openstack-infra | 12:01 | |
| *** HeOS has joined #openstack-infra | 12:02 | |
| *** weshay has quit IRC | 12:03 | |
| *** dkranz has joined #openstack-infra | 12:03 | |
| *** teran_ has quit IRC | 12:05 | |
| *** dims__ has quit IRC | 12:07 | |
| *** dims__ has joined #openstack-infra | 12:07 | |
| openstackgerrit | meza proposed a change to openstack-infra/jenkins-job-builder: Added slack support https://review.openstack.org/131404 | 12:08 |
| mordred | ianw, sdague: wow. pci_alias = {"name":"cisco","vendor_id":"8086","product_id":"10ed"} | 12:08 |
| mordred | that's crazypants | 12:08 |
| sdague | mordred: right... agreed | 12:08 |
| sdague | I started an ML thread on it | 12:09 |
| mordred | _maybe_ we should drop a bomb on people and suggest that we drop ini for yaml? | 12:09 |
| *** doug-fish1 has left #openstack-infra | 12:09 | |
| sdague | well, honestly, the name = value here I think is overkill | 12:09 |
| sdague | we're talking about a few well defined fields | 12:09 |
| mordred | kk | 12:09 |
| sdague | we could just do a fixed record format | 12:09 |
| * mordred tries to keep his uninformed head out of most of these things | 12:09 | |
| sdague | see ML | 12:09 |
| mordred | sdague: oh - while you were out, I wrote a new library | 12:10 |
| *** mwagner_lap has quit IRC | 12:10 | |
| sdague | I just assume you write a new library every weekend :) | 12:10 |
| mordred | :) | 12:10 |
| mordred | sdague: this one is called "os-client-config" - it's a library to handle end-user/client cloud credential config stuffs | 12:10 |
| mordred | sdague: dtroyer has a patch up to add it to python-openstackclient | 12:11 |
| sdague | cool | 12:11 |
| *** yfried|afk is now known as yfried | 12:11 | |
| mordred | handles multi-clouds-in-config-file as well as env vars - and I think dtroyer has it doing something sensible with post-processing argparse namespaces | 12:11 |
| sdague | nice | 12:11 |
| mordred | I kinda think we should add it to all of the client libs - mainly because I'm sad that the nova command line client doesn't grok my clouds.yaml file now :) | 12:12 |
| *** bradjones has quit IRC | 12:12 | |
| *** bradjones has joined #openstack-infra | 12:13 | |
| sdague | that seems reasonable... oh, I should push up the python client from pypi testing patch out there. Maybe heads up on the ML first to get some acks on approach. | 12:14 |
| *** yamahata has quit IRC | 12:15 | |
| openstackgerrit | Sergey Skripnick proposed a change to openstack-infra/project-config: Add hook for building rally docker image https://review.openstack.org/131389 | 12:15 |
| sdague | also, I'm going to be that guy at least once today - https://www.flickr.com/gp/51681448@N00/o5D079 - my coding buddy coming for a visit yesterday | 12:15 |
| *** weshay has joined #openstack-infra | 12:16 | |
| *** BharatK has quit IRC | 12:16 | |
| *** unicell has quit IRC | 12:17 | |
| mordred | sdague: you're keeping to a strict diet of python, right? :) | 12:18 |
| sdague | yep :) | 12:18 |
| sdague | though, actually, I guess I was writing bash and javascript mostly yesterday | 12:18 |
| mordred | turns out that openstack is not all written in python | 12:18 |
| mordred | turns out it's turned in to "python and bash and javascript" | 12:19 |
| sdague | yep | 12:19 |
| sdague | https://review.openstack.org/#/c/131249/ should further simplify the test results table | 12:19 |
| dims__ | hmm, none of the tests in python27 seems to be running? | 12:19 |
| dims__ | Run: 0 in 0.0 sec. | 12:20 |
| sdague | dims__: on what? | 12:20 |
| mordred | dims__: that does not happy making me | 12:20 |
| dims__ | sample - http://logs.openstack.org/86/130086/6/check/gate-nova-python27/0f838dc/console.html | 12:20 |
| * mordred cries | 12:20 | |
| * dims__ hopes he is wrong | 12:20 | |
| mordred | lifeless: ^^ you still awake? | 12:20 |
| mrmartin | dims__: epic fast | 12:21 |
| dims__ | :) | 12:21 |
| mordred | I get 106 hits for that in the past 48 hours - but it seems to have picked up significantly in the last 8 hours | 12:22 |
| dims__ | ouch | 12:24 |
| *** david-lyle has joined #openstack-infra | 12:24 | |
| mordred | dims__: https://review.openstack.org/#/c/130870/ seems to be an early one that did it | 12:26 |
| *** slashme has joined #openstack-infra | 12:27 | |
| dims__ | thanks mordred looking | 12:28 |
| *** slashme has quit IRC | 12:28 | |
| *** slahsme has joined #openstack-infra | 12:29 | |
| *** david-lyle has quit IRC | 12:29 | |
| *** koolhead17 has joined #openstack-infra | 12:29 | |
| mordred | dims__: but - you know, may not be related - that's just me trying to pretend like I know how to use elasticsearch | 12:30 |
| dims__ | mordred: understood. staring at run_tests.sh | 12:30 |
| dims__ | but we are not running that | 12:30 |
| dims__ | tox.ini changes only the cover target | 12:31 |
| dims__ | so can't be that | 12:31 |
| mordred | I'm wondering if something changed to cause testr to fail open | 12:31 |
| sdague | yeh, that happens some times | 12:31 |
| mordred | lifeless: I know we've said this before, so I don't want to lay it on too much ... but I do want to reiterate that I think that failing open is never the right choice | 12:32 |
| sdague | so there is work around code in tempest to catch that case | 12:32 |
| * mordred cries | 12:32 | |
| sdague | I can bring that over to nova | 12:32 |
| sdague | dims__: you replicating this locally? | 12:33 |
| * sdague is kicking off a local run for it | 12:33 | |
| dims__ | sdague: ya, thats how i noticed it | 12:33 |
| mordred | sdague: is that workaround code in a wrapper script? | 12:33 |
| dims__ | "./run_tests.sh -N" also shows 0 tests run | 12:33 |
| dims__ | "cat: .testrepository/next-stream: No such file or directory" | 12:34 |
| *** ryanpetrello has quit IRC | 12:34 | |
| sdague | mordred: it is | 12:34 |
| *** ryanpetrello has joined #openstack-infra | 12:34 | |
| *** dprince has joined #openstack-infra | 12:35 | |
| sdague | mordred: https://github.com/openstack/tempest/blob/master/tools/subunit-trace.py#L237-L239 | 12:36 |
| sdague | mordred: also... what tz you in? | 12:36 |
| sdague | because this is freaking early for west coast :) | 12:36 |
| mordred | sdague: I'm in paris | 12:36 |
| mordred | sdague: wow. how is all of that code not just in testr? | 12:36 |
| sdague | oh, nice | 12:36 |
| *** miqui has joined #openstack-infra | 12:36 | |
| *** doug-fish has joined #openstack-infra | 12:36 | |
| sdague | because... testr is still in bzr | 12:37 |
| *** arosen has joined #openstack-infra | 12:37 | |
| arosen | anyone know where this file now lives? https://github.com/openstack-infra/system-config/blob/master/modules/openstack_project/files/nodepool/scripts/prepare_node.sh | 12:37 |
| mordred | sdague: I believe it has moved finally | 12:37 |
| *** koolhead_ has joined #openstack-infra | 12:38 | |
| mordred | https://github.com/testing-cabal/testrepository | 12:38 |
| *** koolhead17 has quit IRC | 12:39 | |
| *** mbacchi has joined #openstack-infra | 12:39 | |
| *** ihrachyshka has joined #openstack-infra | 12:39 | |
| *** freemanbrandon has joined #openstack-infra | 12:40 | |
| mordred | ok. I'm going to go find coffee | 12:41 |
| arosen | nvm found it. | 12:41 |
| *** hdd has quit IRC | 12:42 | |
| sdague | ok, cool | 12:42 |
| *** weshay has quit IRC | 12:43 | |
| *** hdd has joined #openstack-infra | 12:44 | |
| *** alexpilotti has joined #openstack-infra | 12:45 | |
| *** unicell has joined #openstack-infra | 12:45 | |
| *** alexpilotti has quit IRC | 12:45 | |
| *** alexpilotti has joined #openstack-infra | 12:46 | |
| *** k4n0 has quit IRC | 12:46 | |
| *** dkliban_afk is now known as dkliban | 12:46 | |
| sdague | dims__: it's related to the olso.concurency bit | 12:46 |
| dims__ | yea | 12:47 |
| sdague | is this one of those goofy things where oslo.concurrency exits because the tmpdir isn't specified? | 12:47 |
| *** jerryz has quit IRC | 12:47 | |
| openstackgerrit | meza proposed a change to openstack-infra/jenkins-job-builder: Fixed style violation https://review.openstack.org/131414 | 12:47 |
| *** talluri has quit IRC | 12:48 | |
| sdague | dims__: so, honestly, I think we need to fast revert the concurrency patch | 12:48 |
| sdague | because not running unit tests is not really an option | 12:48 |
| *** _nadya_ has joined #openstack-infra | 12:49 | |
| *** talluri has joined #openstack-infra | 12:51 | |
| *** vhoward has left #openstack-infra | 12:52 | |
| *** todd_dsm has joined #openstack-infra | 12:52 | |
| *** mwagner_lap has joined #openstack-infra | 12:52 | |
| openstackgerrit | meza proposed a change to openstack-infra/jenkins-job-builder: Fixed style violations in the properties https://review.openstack.org/131417 | 12:53 |
| *** mfink has joined #openstack-infra | 12:57 | |
| *** ryanpetrello has quit IRC | 13:02 | |
| *** weshay has joined #openstack-infra | 13:03 | |
| *** unicell has quit IRC | 13:04 | |
| *** aysyd has quit IRC | 13:05 | |
| *** bknudson has quit IRC | 13:05 | |
| *** ryanpetrello has joined #openstack-infra | 13:05 | |
| *** ryanpetrello has quit IRC | 13:06 | |
| *** dustins has joined #openstack-infra | 13:06 | |
| *** derekh has quit IRC | 13:06 | |
| *** rodrigods has joined #openstack-infra | 13:09 | |
| *** julim has joined #openstack-infra | 13:10 | |
| anteaya | so mikal and I have successfuly arm-twisted jogo into being the nova third-party liaison | 13:10 |
| anteaya | I've asked jogo to reply to http://lists.openstack.org/pipermail/openstack-dev/2014-October/049335.html and let him know we are happy to support his knowledge base so he can reply | 13:11 |
| *** ryanpetrello has joined #openstack-infra | 13:11 | |
| *** markmcclain has joined #openstack-infra | 13:11 | |
| *** markmcclain1 has joined #openstack-infra | 13:12 | |
| *** rushiagr_away is now known as rushiagr | 13:12 | |
| *** markmcclain1 has quit IRC | 13:12 | |
| dhellmann | lifeless: I haven't tried it with jython, no | 13:12 |
| *** unicell has joined #openstack-infra | 13:15 | |
| *** markmcclain has quit IRC | 13:16 | |
| *** hdd has quit IRC | 13:18 | |
| *** achanda has joined #openstack-infra | 13:18 | |
| fungi | mordred: sdague: dims__: still catching up on scrollback, but lifeless uploaded a new release around the time that picked up | 13:21 |
| *** aysyd has joined #openstack-infra | 13:21 | |
| dhellmann | sdague, dims__ : is there a bug for this thing with oslo.concurrency and nova not running tests? | 13:22 |
| *** teran_ has joined #openstack-infra | 13:22 | |
| dims__ | fungi: lockutils caused problem, sdague has a review for a revert of oslo.concurrency in nova and a review to make sure we bail out when we see "0 tests" | 13:23 |
| *** achanda has quit IRC | 13:23 | |
| fungi | mordred: sdague: dims__: unittest2 0.6.0 | 13:23 |
| fungi | he commented in here 2014-10-28 00:35:43 | 13:23 |
| arosen | Hi, I noticed that my 3rd party CI doesn't actually seem to be showing it's voting score on gerrit. Is this something I need to change or something that the infra team needs to set? The CI name is "VMWare Congress CI" | 13:25 |
| *** bknudson has joined #openstack-infra | 13:25 | |
| *** david-lyle has joined #openstack-infra | 13:25 | |
| *** teran has quit IRC | 13:26 | |
| *** ryanpetrello_ has joined #openstack-infra | 13:26 | |
| *** tsg has joined #openstack-infra | 13:27 | |
| fungi | arosen: the project on which it will vote needs to request voting rights for the account, just so we don't have accounts voting on projects' changes uninvited | 13:27 |
| *** mriedem has joined #openstack-infra | 13:29 | |
| *** david-lyle has quit IRC | 13:29 | |
| arosen | fungi: i see, i guess i need to send an email to third-party-requests@lists.openstack.org to get it enabled? | 13:29 |
| fungi | arosen: i'm not sure we've formalized that process, but it wouldn't hurt... on what project(s) will it be voting? | 13:31 |
| arosen | stackforge/congress, stackforge/python-congressclient for now. | 13:31 |
| fungi | arosen: does congress have a ptl? | 13:31 |
| arosen | fungi: yes it does | 13:32 |
| *** thedodd has joined #openstack-infra | 13:32 | |
| arosen | Tim Hinrichs | 13:32 |
| arosen | I'm set as the release manager though. | 13:32 |
| fungi | neat, haven't seen him around. and yes, i just confirmed that in the gerrit group membership | 13:32 |
| *** zns has joined #openstack-infra | 13:33 | |
| *** sarob has joined #openstack-infra | 13:34 | |
| fungi | i guess if you want to send couple sentence e-mail saying that as the release manager for teh stackforge/congress projects you're requesting voting rights for that account so that it can provide better feedback on your projects, and then i'll reply that i've done so | 13:34 |
| fungi | arosen: ^ | 13:34 |
| arosen | fungi: awesome i'll do that. Thanks. I'll CC tim too. | 13:34 |
| *** changbl has quit IRC | 13:35 | |
| fungi | arosen: and in the meantime i've already added voting rights for it... the e-mail is just so that we have a more easily searched public record of these sorts of status changes | 13:35 |
| arosen | fungi: thanks :), sending the email now. | 13:36 |
| dhellmann | sdague: I noticed yesterday that some of the oslo libs still send patches through the integrated gate. I don't think that's the intent any more, but can't remember how exactly that works to fix it myself. Can we work on that together after the summit? | 13:36 |
| sdague | fungi: can you gate queue this - https://review.openstack.org/#/c/131416/ | 13:36 |
| sdague | it's kind of important that no other nova changes land until that's in | 13:36 |
| sdague | dhellmann: sure, do you have an example? | 13:36 |
| *** dmsimard_away is now known as dmsimard | 13:37 | |
| fungi | sdague: 131416,1 is enqueued into the gate now | 13:37 |
| openstackgerrit | meza proposed a change to openstack-infra/jenkins-job-builder: Fixed style violations https://review.openstack.org/131431 | 13:38 |
| fungi | sdague: along with its child change | 13:38 |
| sdague | fungi: thanks | 13:38 |
| *** homeless has joined #openstack-infra | 13:39 | |
| *** timrc-afk is now known as timrc | 13:39 | |
| *** aysyd has quit IRC | 13:40 | |
| *** marun has joined #openstack-infra | 13:41 | |
| fungi | sdague: oddly, i thought we already had precautions in the tox wrapper script to fail out if no tests get run | 13:42 |
| sdague | fungi: so I think we do | 13:42 |
| sdague | but... | 13:42 |
| sdague | we have a different wrapper for nova now | 13:42 |
| sdague | that gives us more info | 13:42 |
| sdague | so we probably bypass that | 13:42 |
| fungi | i guess this didn't catch it then? http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/scripts/run-unittests.sh#n59 | 13:43 |
| anteaya | rain in france http://www.france-weather.com/ | 13:43 |
| clarkb | we do and nova managed to bypass | 13:44 |
| *** homeless1 has joined #openstack-infra | 13:44 | |
| clarkb | the check in our tests relies on .testrepository existing | 13:44 |
| fungi | nova's testr runs don't create a .testrepository file any longer? | 13:45 |
| clarkb | they do but not when broken as in this case | 13:45 |
| sdague | clarkb: passing --subunit makes .testrepository not exist? | 13:45 |
| clarkb | it was dying before any testr could run | 13:45 |
| clarkb | no oslo.concurrency had no main | 13:46 |
| fungi | oh | 13:46 |
| *** marun has quit IRC | 13:46 | |
| sdague | oh, gotcha | 13:46 |
| clarkb | so python -m lockstuff went "fail" | 13:46 |
| *** homeless has quit IRC | 13:46 | |
| clarkb | but it returned 0? | 13:46 |
| *** zns has quit IRC | 13:47 | |
| clarkb | thats the bit that has me confused | 13:48 |
| *** freemanbrandon has quit IRC | 13:48 | |
| clarkb | I am guessing python -m importedjust fine | 13:48 |
| sdague | yeh, I don't know | 13:48 |
| clarkb | and its up to the lib to execute code | 13:48 |
| *** amitgandhinz has joined #openstack-infra | 13:49 | |
| clarkb | since it didnt the import was successful and return 0 | 13:49 |
| *** armax has joined #openstack-infra | 13:49 | |
| *** amitgandhinz has quit IRC | 13:49 | |
| *** marun has joined #openstack-infra | 13:49 | |
| *** amitgandhinz has joined #openstack-infra | 13:49 | |
| fungi | right, the import was a no-op | 13:50 |
| *** pelix has joined #openstack-infra | 13:51 | |
| *** dizquierdo has quit IRC | 13:51 | |
| *** katyafervent is now known as katyafervent_awa | 13:51 | |
| *** aysyd has joined #openstack-infra | 13:52 | |
| *** zns has joined #openstack-infra | 13:52 | |
| *** freemanbrandon has joined #openstack-infra | 13:52 | |
| *** zns has quit IRC | 13:53 | |
| *** yfried has quit IRC | 13:54 | |
| *** koolhead_ has quit IRC | 13:55 | |
| *** hdd has joined #openstack-infra | 13:55 | |
| openstackgerrit | Mate Lakat proposed a change to openstack-infra/project-config: XenServer: Fix initramfs scripts https://review.openstack.org/131438 | 13:59 |
| *** ryanpetrello has quit IRC | 13:59 | |
| *** ryanpetrello_ is now known as ryanpetrello | 13:59 | |
| *** zns has joined #openstack-infra | 14:00 | |
| *** eharney has joined #openstack-infra | 14:02 | |
| *** zz_dimtruck is now known as dimtruck | 14:02 | |
| naggappan | hi what is git_base URL in devstack gate? its always pointing to local git_base=https://git.openstack.org. I wan to modify to my get server path. so that it takes neutron code from my server and others it takes from standard repository. currently its trying git clone https://git.openstack.org/naggappan/test-neutron and repository not found and it gets the the standard one. Now i want the devstack-gate to use my Repository how can i do that. | 14:04 |
| *** dims__ has quit IRC | 14:04 | |
| *** parimala has joined #openstack-infra | 14:04 | |
| *** unicell has quit IRC | 14:05 | |
| *** nosnos has quit IRC | 14:06 | |
| *** jerryz has joined #openstack-infra | 14:06 | |
| *** nosnos has joined #openstack-infra | 14:07 | |
| *** vhoward has joined #openstack-infra | 14:07 | |
| *** talluri has quit IRC | 14:07 | |
| naggappan | by default will devstack-gate clone all openstack projects only from https://git.openstack.org/ URL? is it possible to change it | 14:07 |
| *** dizquierdo has joined #openstack-infra | 14:07 | |
| *** chandankumar has quit IRC | 14:07 | |
| *** talluri has joined #openstack-infra | 14:07 | |
| *** unicell has joined #openstack-infra | 14:08 | |
| *** nelsnelson has joined #openstack-infra | 14:09 | |
| *** nelsnelson has quit IRC | 14:09 | |
| *** nelsnelson has joined #openstack-infra | 14:10 | |
| *** nosnos has quit IRC | 14:11 | |
| *** mattfarina has joined #openstack-infra | 14:14 | |
| *** katyafervent_awa is now known as katyafervent | 14:14 | |
| *** DuncanT- has joined #openstack-infra | 14:15 | |
| *** JayJ has joined #openstack-infra | 14:21 | |
| *** otherwiseguy has joined #openstack-infra | 14:23 | |
| fungi | naggappan: if you look in devstack-vm-gate-wrap.sh you'll see you can define a custom pre_test_hook function and export it in the calling environment. it runs after setup_workspace has setup all the listed projects but before gate_hook runs the devstack-vm-gate.sh script | 14:23 |
| *** msabramo has quit IRC | 14:24 | |
| *** spzala has joined #openstack-infra | 14:24 | |
| fungi | naggappan: your pre_test_hook could do something like call setup_project for your special project and override GIT_BASE to your git server url when doing so | 14:24 |
| *** miqui has quit IRC | 14:25 | |
| *** sarob has quit IRC | 14:25 | |
| *** sarob has joined #openstack-infra | 14:25 | |
| *** david-lyle has joined #openstack-infra | 14:26 | |
| sdague | clarkb: on - https://review.openstack.org/#/c/125095/ - do you have a snippet for only getting supported branches? my brain wasn't quite coming up with it | 14:26 |
| *** tsg has quit IRC | 14:26 | |
| sdague | naggappan: you probably don't want to override GIT_BASE, override the url for the specific project instead | 14:27 |
| *** jerryz has quit IRC | 14:27 | |
| *** amitgandhinz has quit IRC | 14:28 | |
| clarkb | sdague https://git.openstack.org/cgit/openstack-infra/project-config/tree/nodepool/elements/cache-devstack/extra-data.d/55-cache-devstack-repos#n57 something like that | 14:29 |
| *** radez_g0n3 is now known as radez | 14:29 | |
| *** sarob has quit IRC | 14:29 | |
| fungi | sdague: setup_project honors GIT_BASE from the calling environment but i don't see where you can pass in the specific url for the project you want to set up instead | 14:30 |
| *** david-lyle has quit IRC | 14:31 | |
| naggappan | @fungi so for specifig project i can't override the url. Because i want to use only neutron from my repository others i can use the same git.openstack.org. If possible please tell the the yamal file configuration where i can specify the environment variable | 14:32 |
| *** thedodd has quit IRC | 14:33 | |
| sdague | fungi: oh... right. I was thinking about the devstack case | 14:34 |
| *** yaguang has joined #openstack-infra | 14:34 | |
| fungi | naggappan: i'm not sure what you're asking. there are no yaml files i'm aware of in devstack-gate or consumed by devstack-gate | 14:35 |
| clarkb | features.yaml | 14:35 |
| fungi | oh, right | 14:35 |
| fungi | so anyway, no yaml files which let you modify clone urls or git remotes | 14:36 |
| clarkb | and no feature.yaml knows nothing about git | 14:36 |
| *** zns has quit IRC | 14:36 | |
| *** miqui has joined #openstack-infra | 14:37 | |
| naggappan | sorry i said wrong. Not feature.yamal. Acutally I am setting up CI setup . where all repo should take from standard git.openstack.org but only neutron it should take from my repository. | 14:37 |
| fungi | naggappan: one easy trick might just be to make pre_test_hook change the origin remote in the local copy of the neutron repo and then git reset --hard origin/master | 14:37 |
| openstackgerrit | Marton Kiss proposed a change to openstack-infra/system-config: Update drush dsd-tool of groups portal https://review.openstack.org/131452 | 14:37 |
| fungi | mrmartin: is that ^ urgent to approve? | 14:40 |
| fungi | or are you working around it okay for now? | 14:40 |
| mrmartin | fungi: I feel so | 14:40 |
| naggappan | @fungi so i have to first download the standared neutron repo. Then set this pre_test_hook with my repo and reset. Is that right? If so then where should i change this because devstackgate script automatically do git clone. where i should specify all this changes | 14:40 |
| fungi | clarkb: very simple patch if you feel like adding your +3 | 14:40 |
| mrmartin | writing software is hard :) and had a stupid permission bug that caused a White screen of death | 14:40 |
| fungi | mrmartin: i've had my share of those too :/ | 14:41 |
| mrmartin | we need some monitoring, because this stupid bug affected the prod portal also, I fixed that manually to bring it back today morning | 14:41 |
| fungi | naggappan: wherever you're executing the devstack-vm-gate-wrap.sh script from, you need to export your custom pre_test_hook function in the calling environment (are you using jenkins-job-builder?) | 14:42 |
| *** ayoung has joined #openstack-infra | 14:42 | |
| *** amitgandhinz has joined #openstack-infra | 14:42 | |
| *** yamahata has joined #openstack-infra | 14:42 | |
| naggappan | @fungi yes i am using JJB | 14:42 |
| mrmartin | groups-dev still suffering from that, but I don't want to fix manually, because I want to wait the fixed tool | 14:42 |
| *** chandankumar has joined #openstack-infra | 14:43 | |
| fungi | naggappan: then it would probably need to be exported in the shell builder for the job you're running, before calling devstack-vm-gate-wrap.sh | 14:43 |
| *** zns has joined #openstack-infra | 14:43 | |
| pleia2 | good morning | 14:43 |
| clarkb | approved | 14:44 |
| clarkb | morning | 14:44 |
| *** koolhead17 has joined #openstack-infra | 14:45 | |
| *** koolhead17 has joined #openstack-infra | 14:45 | |
| naggappan | @fungi i know to export the variables. But how can i export the complete pre_test_hook function. Could you point me some example link | 14:45 |
| fungi | naggappan: in http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/rally.yaml you can see examples of jobs and job-templates exporting a pre_test_hook function | 14:46 |
| *** ayoung has quit IRC | 14:46 | |
| fungi | mrmartin: as for monitoring for situations like that, who/how would you expect to get notified so as to reduce the impact duration? | 14:48 |
| naggappan | @fungi thanks let me gothrought and try that | 14:48 |
| *** garyh has joined #openstack-infra | 14:48 | |
| fungi | mrmartin: we've talked about monitoring before, but it's usually boiled down to that set of procedural questions (actually implementing a monitoring system is the "easy" part) | 14:48 |
| *** achanda has joined #openstack-infra | 14:48 | |
| mrmartin | fungi: I agree, I think we can discuss this on the summit | 14:49 |
| mrmartin | https://review.openstack.org/#/c/131452/ get -1 from jenkins | 14:49 |
| mrmartin | a dns / networking issue in the centos6 node | 14:50 |
| fungi | mrmartin: yep, i just rechecked it. should go through this time | 14:50 |
| *** markmcclain has joined #openstack-infra | 14:50 | |
| arosen | hrm, any ideas why i've been hitting this failure in my 3rd party ci related to the python-keystoneclient http://paste.openstack.org/show/125975/ ? | 14:51 |
| mrmartin | fungi, thnx | 14:51 |
| arosen | 2014-10-28 14:47:22.818 | No distributions matching the version for stevedore>=1.1.0 (from python-keystoneclient==0.11.2.10.g20f488d) | 14:51 |
| *** signed8bit has joined #openstack-infra | 14:51 | |
| *** tonytan4ever has joined #openstack-infra | 14:53 | |
| fungi | mrmartin: as for preparing for a productive discussion about monitoring, "we need monitoring" is too nebulous, but something like "it would be nice if i got an e-mail when this url returns no content for more than 5 minutes" would be an example of a more actionable request/proposal | 14:53 |
| *** wenlock has joined #openstack-infra | 14:53 | |
| fungi | arosen: i see "pypi.openstack.org" in that log | 14:54 |
| arosen | fungi: I guess the upstream CI doesn't use that? | 14:55 |
| fungi | arosen: nope, we don't. were you in the third-party meeting yesterday by any chance? or did you see my e-mail to third-party-announce about why the citrix xenserver ci account was disabled? | 14:55 |
| arosen | fungi: sadly no. I was on vacation for the last week. I just joined the 3rd party ci last | 14:55 |
| arosen | i can dig up the thread though so you don't have to rehash it for me. | 14:55 |
| mrmartin | fungi: of course :) I did some icinga / nagios projects before | 14:55 |
| fungi | mrmartin: point being, it's a proposal which addresses the people and notification mechanism desired | 14:56 |
| fungi | arosen: in particular the package mirrors maintained by the upstream project infrastructure are not stable public interfaces, and also present to actual benefit to third-party systems. you should either run a mirror of your own or just use pypi.python.org | 14:57 |
| arosen | fungi: yup found youre email http://lists.openstack.org/pipermail/third-party-announce/2014-October/000084.html | 14:58 |
| *** zz_jgrimm is now known as jgrimm | 14:58 | |
| *** mattfarina has quit IRC | 14:59 | |
| fungi | arosen: for pypi.openstack.org, that was abandoned some months ago when we switched to running individual mirrors within each provider/region, so it hasn't been updated in some time and is slated for removal | 14:59 |
| arosen | fungi: should i update jenkins_jobs/config/macros.yaml: s/http://pypi.openstack.org/http://pypi.python.org/g ? | 14:59 |
| *** dane_leblanc has joined #openstack-infra | 15:00 | |
| fungi | arosen: your nodepool prep scripts or dib elements, if you're reconsuming ours without modification, may be setting a pypi mirror | 15:00 |
| *** ayoung has joined #openstack-infra | 15:00 | |
| arosen | fungi: so you don't thinks it's being done via jenkins_jobs/config/macros.yaml? This is the only place in etc i see this string. | 15:01 |
| arosen | "pypi.openstack" | 15:01 |
| fungi | arosen: it's possible that's where you're getting it from | 15:01 |
| arosen | fungi: k let me try updating it here first | 15:01 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/project-config: enforce stable requirements https://review.openstack.org/125095 | 15:01 |
| sdague | clarkb: how about that? | 15:02 |
| *** carl_baldwin has joined #openstack-infra | 15:04 | |
| *** david-lyle has joined #openstack-infra | 15:05 | |
| nelsnelson | Greetings, channel. I've got a little problem trying to set up devstack-gate on an ubuntu vm. | 15:05 |
| nelsnelson | I'm following the instructions here https://github.com/openstack-infra/devstack-gate | 15:05 |
| nelsnelson | But I keep getting timeouts when I run the safe-devstack-vm-gate-wrap.sh script: https://gist.github.com/nelsnelson/f43b987a5d4e94a69277 | 15:05 |
| openstackgerrit | A change was merged to openstack-infra/system-config: Update drush dsd-tool of groups portal https://review.openstack.org/131452 | 15:06 |
| *** talluri has quit IRC | 15:06 | |
| fungi | nelsnelson: have you looked in the devstack log to see what it might have been doing when that timeout was reached? | 15:07 |
| *** achanda has quit IRC | 15:09 | |
| nelsnelson | fungi, thanks for responding. Yes, I checked the logs, but I can't really see anything conclusive. | 15:09 |
| *** SumitNaiksatam has quit IRC | 15:09 | |
| *** msabramo has joined #openstack-infra | 15:09 | |
| nelsnelson | The last thing I see is "Processing triggers for man-db ..." | 15:09 |
| nelsnelson | Prior to that, thtere's a bunch of "Reading database ..." messages, but it indicates progress, and completes. | 15:10 |
| fungi | dtroyer: if you override devstack.org and/or www.devstack.org to 192.237.219.31 you should be able to test the redirect to http://docs.openstack.org/developer/devstack/$1 | 15:10 |
| fungi | dtroyer: are you okay with me switching dns over to the redirect now? | 15:10 |
| *** tsg has joined #openstack-infra | 15:11 | |
| fungi | nelsnelson: sounds like it's still in the middle of package installation/updates when it times out | 15:11 |
| nelsnelson | Sounds about right. | 15:11 |
| fungi | nelsnelson: we're usually running it with pre-cached local packages, so perhaps it's taking a lot longer because you don't have any cached and need to up the timeout on that? | 15:11 |
| openstackgerrit | Mark Vanderwiel proposed a change to openstack-infra/project-config: Missed repo gate job for zuul layout filter. https://review.openstack.org/131467 | 15:12 |
| nelsnelson | Well, prior to that error message about the timeout, it just sits there for over 30 minutes at least. | 15:12 |
| nelsnelson | The package installs and everything appear to complete... | 15:12 |
| nelsnelson | Reading package lists... Done | 15:12 |
| nelsnelson | and so on | 15:12 |
| fungi | nelsnelson: oh, in that case it may be that package installation is not the issue, but rather whatever comes immediately after it in the script | 15:13 |
| nelsnelson | The last thing I see is this: | 15:13 |
| nelsnelson | Running devstack | 15:13 |
| nelsnelson | ... this takes 5 - 8 minutes (logs in logs/devstacklog.txt.gz) | 15:13 |
| nelsnelson | ./safe-devstack-vm-gate-wrap.sh: line 321: 1452 Killed | 15:13 |
| nelsnelson | ERROR: the main setup script run by this job failed - exit code: 137 | 15:13 |
| nelsnelson | but then there is not logs/devstacklog.txt.gz file | 15:14 |
| *** freemanbrandon has quit IRC | 15:14 | |
| nelsnelson | just a logs/devstacklog.txt file | 15:14 |
| fungi | yeah, not a very helpful line for it to point out unfortunately, as that's the definition for the gate_hook function | 15:14 |
| *** eglynn-office has quit IRC | 15:14 | |
| dtroyer | fungi: checking… | 15:14 |
| *** sdake_ has joined #openstack-infra | 15:17 | |
| marcusvrn_ | fungi: I saw that you disabled the CI that was failing because it was using pypi.openstack.org that will be removed. You mentioned to change the mirror to pypi.python.org or local mirror. I have the same problem....can you help me to say where I have to change it? | 15:18 |
| *** pcrews has joined #openstack-infra | 15:18 | |
| clarkb | sdague: one question on 125095 | 15:18 |
| fungi | marcusvrn_: it depends a lot on how you're building your job workers and running jobs on them | 15:18 |
| clarkb | sdague: but I think that mostly works | 15:18 |
| dtroyer | fungi: go for it…and thanks | 15:19 |
| clarkb | marcusvrn_: arosen: are you all using jaypipes' stuff? | 15:19 |
| arosen | clarkb: yes | 15:19 |
| clarkb | :/ | 15:19 |
| arosen | i tried updating the /etc/jenkins files but that didn't do the trick | 15:19 |
| marcusvrn_ | fungi:clarkb I'm using asselin repo (that is a fork of jaypipes repo) | 15:19 |
| clarkb | iirc devstack-gate copies a "base" set of files into all the places we need it | 15:19 |
| clarkb | so I think you just need to edit that location. give me a minute to look that up | 15:20 |
| *** sdake_ has quit IRC | 15:20 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/project-config: enforce stable requirements https://review.openstack.org/125095 | 15:20 |
| sdague | clarkb: ok, fixed | 15:20 |
| *** sdake_ has joined #openstack-infra | 15:21 | |
| clarkb | https://git.openstack.org/cgit/openstack-infra/devstack-gate/tree/functions.sh#n370 | 15:21 |
| clarkb | marcusvrn_: arosen ^ so that is copying whatever you install into jenkins homedir | 15:21 |
| *** mattfarina has joined #openstack-infra | 15:22 | |
| clarkb | https://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/files/pip.conf is likely the source of all the confusion | 15:22 |
| clarkb | along with https://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/files/pydistutils.cfg patches welcome | 15:22 |
| arosen | clarkb: ah these get built into my nodepool image from prepare_node.sh when it clones this repo? | 15:23 |
| openstackgerrit | Mate Lakat proposed a change to openstack-infra/project-config: Experimental patch to try overriding pip config https://review.openstack.org/131469 | 15:23 |
| clarkb | ya you end up with those files in jenkins' homedir via puppet | 15:23 |
| arosen | k | 15:23 |
| *** lttrl has quit IRC | 15:24 | |
| clarkb | we then override them with nodepool to have per region mirrors | 15:24 |
| kashyap | Hi, LaunchPad OpenID seems to be arbitrary strings - https://login.launchpad.net/+id/Nb3wnff, I wonder if it's possible to s/Nb3wnff/$USERNAME | 15:24 |
| fungi | arosen: so yes you should either modify those to use your own mirror, or if you remove them they'll default to pypi.python.org | 15:24 |
| clarkb | but if you just set those files to pypi.python.org instead of pypi.openstack.org that is a reasonable value | 15:24 |
| marcusvrn_ | clarkb: hmm..nace! I'll try it | 15:24 |
| clarkb | fungi: well I think we should fix upstream | 15:24 |
| clarkb | and set to pypi.python.org | 15:24 |
| marcusvrn_ | nice* | 15:24 |
| fungi | kashyap: you're wanting to look up the lp username associated with an lp openid? | 15:24 |
| *** doug-fish has left #openstack-infra | 15:25 | |
| kashyap | fungi, Yes | 15:25 |
| fungi | kashyap: if so, they have an api call to do that lookup | 15:25 |
| fungi | kashyap: you can see where we do it in openstack-infra/jeepyb i think... | 15:25 |
| fungi | getting you a quick example | 15:25 |
| fungi | kashyap: http://git.openstack.org/cgit/openstack-infra/jeepyb/tree/jeepyb/cmd/update_bug.py#n121 | 15:26 |
| kashyap | fungi, Hm, I wasn't thinking of doing it programmatically, but it's good to know it's possible to do it, thank you | 15:26 |
| fungi | kashyap: afaik that;s the _olly_ way to do it | 15:26 |
| fungi | er, _only_ | 15:26 |
| kashyap | Something a bit more human friendly like: USERNAME.id.openstack.org or some such | 15:26 |
| arosen | clarkb: is we fixed this upstream then you guys have to update youre puppet stuff to set this value right? | 15:27 |
| kashyap | But I'm just making random requests, you can disregard me :-) | 15:27 |
| arosen | or update it via another means | 15:27 |
| fungi | kashyap: well, https://openstackid.org/ is running now, though it's not in use yet | 15:27 |
| clarkb | arosen: we already update it via another method | 15:27 |
| clarkb | arosen: because we don't know what region a node will boot in until it boots (particularly in the dib world) we have nodepool write these files for us after boot | 15:27 |
| arosen | clarkb: ah okay, want me to update it upsteam then? | 15:27 |
| fungi | kashyap: and it uses the username in the url instead of a random hash | 15:28 |
| kashyap | fungi, Oh, so eventually, users _will_ have something like: USER.openstackid.org, cool. | 15:28 |
| arosen | i was just going to hack my prepare_node.sh script | 15:28 |
| *** hdd has quit IRC | 15:28 | |
| kashyap | fungi, Thanks for the quick response. | 15:28 |
| clarkb | arosen: ya I think we can just set to pypi.python.org upstream | 15:29 |
| fungi | kashyap: https://openstackid.org/jeremy.stanley | 15:29 |
| *** msabramo has quit IRC | 15:29 | |
| *** msabramo1 has joined #openstack-infra | 15:29 | |
| arosen | clarkb: cool, want me to push a patch for this change or are you going to? | 15:29 |
| clarkb | arosen: go ahead, that way my +2 counts :) | 15:30 |
| arosen | cool | 15:30 |
| fungi | mrmartin: out of curiosity, do you know how smarcet planned to keep openid urls stable when people change their name? | 15:30 |
| kashyap | fungi, Nice, thank you. | 15:30 |
| *** dimtruck is now known as zz_dimtruck | 15:30 | |
| mrmartin | fungi: I don't know this part | 15:31 |
| mrmartin | just trying to hack-in the prod one | 15:31 |
| mrmartin | into groups portal | 15:31 |
| *** zns has quit IRC | 15:31 | |
| fungi | mrmartin: thanks... i'll try to remember to ask him later | 15:31 |
| kashyap | fungi, I have to separately register or should my launchpad credentials will work? | 15:32 |
| fungi | kashyap: it's using the same database as www.openstack.org so the login is the same as you use for your foundation member profile | 15:32 |
| mrmartin | I think it will bring up some issues during the first few weeks | 15:32 |
| fungi | mrmartin: i agree, but that's why it's in beta ;) | 15:32 |
| mrmartin | so we can stabilize this openid service before using for something must-work service. | 15:33 |
| *** whoops has joined #openstack-infra | 15:33 | |
| fungi | mrmartin: right, i'm hesitant to tie it into, e.g., review.openstack.org or wiki.openstack.org until it's proven on other services for a little while | 15:34 |
| openstackgerrit | Aaron Rosen proposed a change to openstack-infra/system-config: Update pypi mirror so use pypi.python.org instead of pypi.openstack.org https://review.openstack.org/131474 | 15:34 |
| arosen | clarkb: ^ | 15:34 |
| fungi | arosen: there's a review out there for that i think... checking | 15:34 |
| openstackgerrit | Aaron Rosen proposed a change to openstack-infra/system-config: Update pypi mirror to use pypi.python.org instead of pypi.openstack.org https://review.openstack.org/131474 | 15:34 |
| kashyap | fungi, Cool, noted. | 15:34 |
| clarkb | oh you know what | 15:35 |
| clarkb | hrm | 15:35 |
| clarkb | arosen: fungi: our non nodepool built slaves use those files | 15:35 |
| clarkb | we have two at this point iirc | 15:35 |
| clarkb | proposal.slave.o.o and pypi.slave.o.o | 15:35 |
| fungi | arosen: ideally the comments in https://review.openstack.org/119466 need to be addressed | 15:35 |
| clarkb | do we want to default to one of our internal regions? | 15:35 |
| fungi | clarkb: yep, that's why i -1'd 119466 | 15:35 |
| clarkb | so would pypi.dfw.openstack.org be a better fallback there? | 15:36 |
| fungi | clarkb: probably yes | 15:36 |
| clarkb | arosen: ^ | 15:36 |
| clarkb | sorry that just occured to me | 15:36 |
| *** zz_dimtruck is now known as dimtruck | 15:36 | |
| *** zns has joined #openstack-infra | 15:37 | |
| *** msabramo1 has quit IRC | 15:40 | |
| clarkb | sdague: I am +2 on 125095 now. Will be interesting to see how many projects trip over that | 15:40 |
| *** rmcall has joined #openstack-infra | 15:40 | |
| clarkb | sdague: we likely don't want to rollback client changes but instead let the stable branches pull up? | 15:40 |
| arosen | clarkb: so update it to pypi.dfw.openstack.org ? | 15:41 |
| clarkb | arosen: ya Ithink that is a reasonable default for upstream and downstream (as much as we don't want downstream relying on our mirrors) | 15:41 |
| *** dizquierdo has quit IRC | 15:41 | |
| sdague | clarkb: yeh, I think so | 15:42 |
| fungi | clarkb: i think pypi.python.org is a reasonable default for downstream. pypi.*.openstack.org is not | 15:42 |
| fungi | but i also don't know how we stop them from using it | 15:43 |
| fungi | so meh | 15:43 |
| clarkb | fungi: right, that file has a reasonable default for us | 15:43 |
| clarkb | and then others reconsume it... | 15:43 |
| openstackgerrit | Aaron Rosen proposed a change to openstack-infra/system-config: Update pypi mirror to use pypi.dfw.openstack.org https://review.openstack.org/131474 | 15:43 |
| arosen | clarkb: ^ | 15:43 |
| *** SumitNaiksatam has joined #openstack-infra | 15:44 | |
| clarkb | afazekas_beer: fungi: did you guys see the results of multinode using floating ips? | 15:44 |
| clarkb | wasn't pretty | 15:44 |
| clarkb | so it is likely that we need to use the internal addresses | 15:44 |
| jeblair | clarkb: where can i find your conclusions? | 15:44 |
| clarkb | jeblair: I left a review on afazekas_beer's change | 15:44 |
| fungi | clarkb: so is it a nat triangulation issue with the vm trying to talk to its global address out in hpcloud's network? | 15:45 |
| clarkb | jeblair: basically ran floating ip change on rax and hpcloud. rax worked but hpcloud did not (you can see those results in the jenkins results on the child change of afazekas_beer's change). I made sure to run afazekas_beer's change against hpcloud as well to make sure this wasn't a common issue but the test that ran on hpcloud for that change passed | 15:45 |
| clarkb | fungi: I didn't debug the specific problem with NAT. merely got the tests to run there and checked the result | 15:45 |
| fungi | k | 15:46 |
| *** unicell1 has joined #openstack-infra | 15:46 | |
| clarkb | but the failing tests appeared to deal with volumes | 15:46 |
| clarkb | so I think there is a good chance the iscsi issues afazekas_beer mentioned were hit | 15:46 |
| jeblair | clarkb: did the configuration set the cinder ip address? | 15:46 |
| fungi | oh, so maybe this is a devstack/cinder bug? | 15:46 |
| clarkb | jeblair: no, I merely made the changes you suggested to cat the nodepool files | 15:46 |
| fungi | ahh, or missed configuration | 15:46 |
| jeblair | clarkb: okay, so i'm not convinced fixed ip is the answer. i believe that proper cinder configuration should be attempted. | 15:47 |
| *** unicell has quit IRC | 15:47 | |
| clarkb | so cinder needs special config? | 15:47 |
| *** andreaf has quit IRC | 15:48 | |
| clarkb | I think I missed that yesterday | 15:48 |
| * clarkb checks scrollback | 15:48 | |
| jeblair | clarkb: part of the problem with what hp is doing with not locating the actual public ip on their hosts is breaking the assumption that many programs have that they can determine the ip addresses of the machines they run on. | 15:49 |
| *** bitblt has joined #openstack-infra | 15:49 | |
| jeblair | this affects a surprisingly wide range of programs, apparently including cinder | 15:50 |
| clarkb | so we need to explicitly tell cinder what its IP is even though the IP isn't on a local interface | 15:50 |
| jeblair | but most of them, (including cinder, fortunately) have a method of saying "don't trust your ip address detection, just use this user-supplied one instead" | 15:50 |
| clarkb | I can look into supplying that in my change | 15:50 |
| jeblair | clarkb: https://github.com/openstack/cinder/blob/master/etc/cinder/cinder.conf.sample#L1039 | 15:51 |
| *** msabramo has joined #openstack-infra | 15:51 | |
| fungi | clearly the solution is for openstack projects to post a request to http://www.whatismyip.com/ | 15:52 |
| jeblair | fungi: that would be the RESTful service way, yeah? | 15:52 |
| fungi | absolutely... i'll save that cross-project spec proposal for april though | 15:53 |
| *** dizquierdo has joined #openstack-infra | 15:54 | |
| clarkb | jeblair: it looks like we set that value | 15:54 |
| clarkb | http://logs.openstack.org/97/131297/1/experimental/check-tempest-dsvm-aiopcpu/9ac7127/logs/localrc.txt.gz SERVICE_HOST is the public IP | 15:54 |
| clarkb | (it is set twice :( but second one is public IP) | 15:54 |
| *** otherwiseguy has quit IRC | 15:54 | |
| *** terryw has joined #openstack-infra | 15:54 | |
| clarkb | then in devstack we set cinder's my_ip to SERVICE_HOST https://git.openstack.org/cgit/openstack-dev/devstack/tree/lib/cinder#n224 | 15:55 |
| clarkb | note CINDER_SERVICE_HOST defaults to SERVICE_HOST and we don't override | 15:55 |
| *** groknix has quit IRC | 15:55 | |
| clarkb | and cinder config you link to implies that the iscsi IP defaults to my_ip. Though I am currently trying to cnfirm that | 15:55 |
| *** groknix has joined #openstack-infra | 15:56 | |
| *** zaphod42 has joined #openstack-infra | 15:56 | |
| *** slahsme has quit IRC | 15:56 | |
| jeblair | clarkb, sdague: left some -1 comments on 125095 | 15:57 |
| clarkb | https://github.com/openstack/cinder/blob/master/cinder/volume/driver.py#L51-L53 seems to default to my_ip | 15:57 |
| *** zns has quit IRC | 15:58 | |
| jeblair | wow '$my_ip' is actually a thing in oslo.config? i'm constantly amazed | 15:58 |
| clarkb | indeed | 15:58 |
| *** miqui has quit IRC | 15:59 | |
| *** miqui_ has joined #openstack-infra | 15:59 | |
| jeblair | clarkb: just so you know what i'm looking for -- i want to know what the actual problem is. i want to know if this is a cinder bug, or an hploud bug, or a tgtadm bug, etc... | 15:59 |
| clarkb | yes understood | 16:00 |
| jogo | anteaya: yup, I will be replying to it this morning | 16:00 |
| jgriffith | clarkb: clarkb clarkb jeblair anything I ca help with here? | 16:01 |
| *** fesp has joined #openstack-infra | 16:01 | |
| clarkb | jgriffith: possibly. We are trying to get multinode devstack + tempest testing working | 16:01 |
| clarkb | jgriffith: currently when cinder is configured to use a floating IP for my_ip in hpcloud tempest tests fail | 16:02 |
| jgriffith | jeblaihmmm... i tsees my connection is dieing | 16:02 |
| openstackgerrit | Aaron Rosen proposed a change to openstack-infra/system-config: Update pypi mirror to use pypi.dfw.openstack.org https://review.openstack.org/131474 | 16:02 |
| clarkb | jgriffith: they do not fail in rax which does not have NATed floating IPs | 16:02 |
| clarkb | jgriffith: http://logs.openstack.org/97/131297/1/experimental/check-tempest-dsvm-aiopcpu/9ac7127/ is link to the logs | 16:02 |
| *** fesp is now known as flaper87 | 16:02 | |
| *** flaper87 has joined #openstack-infra | 16:03 | |
| arosen | clarkb: I updated the rest of the places pypi.openstack.org was used there now here: https://review.openstack.org/131474 | 16:03 |
| *** ivar-lazzaro has joined #openstack-infra | 16:03 | |
| openstackgerrit | Sergey Skripnick proposed a change to openstack-infra/project-config: Add hook for building rally docker image https://review.openstack.org/131389 | 16:03 |
| clarkb | arosen: I don't think we want wholesale replacement. at least not in the docs like that | 16:04 |
| *** parimala has quit IRC | 16:04 | |
| *** ivar-laz_ has joined #openstack-infra | 16:04 | |
| clarkb | or in bandersnatch configs | 16:04 |
| clarkb | arosen: I think we should start with just those two files | 16:04 |
| *** gyee has joined #openstack-infra | 16:04 | |
| *** doug-fish has joined #openstack-infra | 16:04 | |
| arosen | clarkb: so my patchset 3? | 16:05 |
| *** amotoki has joined #openstack-infra | 16:05 | |
| clarkb | I think so | 16:05 |
| clarkb | fungi: ^ | 16:05 |
| fungi | clarkb: i guess the doc updates can get tacked onto https://review.openstack.org/119466 instead and then rebase it on top of 131474? | 16:05 |
| clarkb | fungi: ya. I see this change as fixing the immediate issue of pypi.o.o doesn't work and some slaves use it | 16:05 |
| fungi | clarkb: fair enough | 16:06 |
| jeblair | sdague: did you test either your "rewrite everything" patch to hideci or your "only show most recent" patch with the new change screen? | 16:06 |
| clarkb | proper cleanup of pypi.o.o can be a subsequent change | 16:06 |
| jeblair | sdague: because it does not appear that the new change screen works right now | 16:06 |
| openstackgerrit | Aaron Rosen proposed a change to openstack-infra/system-config: Update pypi mirror to use pypi.dfw.openstack.org https://review.openstack.org/131474 | 16:06 |
| arosen | okay i repushed patch set 3 with just those two files being update.d | 16:06 |
| jeblair | fungi, jhesketh, SergeyLukjanov: you approved https://review.openstack.org/#/c/124632/ . did you test it or verify that it was tested? | 16:07 |
| sdague | jeblair: I did not | 16:07 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/system-config: Revert "refactor of hideci for readability and modularity" https://review.openstack.org/131483 | 16:07 |
| *** ivar-lazzaro has quit IRC | 16:08 | |
| fungi | jeblair: oops, i tested it in-browser in the old change screen before approving, but forgot to test against the new change screen | 16:08 |
| fungi | jeblair: we can revert for now i guess | 16:08 |
| sdague | jeblair: let me take a look to see if it's an easy fix | 16:08 |
| *** mmaglana has joined #openstack-infra | 16:08 | |
| jgriffith | firewall? | 16:09 |
| *** jgriffit1 has joined #openstack-infra | 16:09 | |
| clarkb | jgriffith: the ndoe should be wide open between each other but I will double check | 16:09 |
| *** jgriffit1 is now known as jdg | 16:09 | |
| *** jgriffith has left #openstack-infra | 16:09 | |
| *** jdg is now known as jgriffith | 16:10 | |
| *** VijayTripathi has joined #openstack-infra | 16:10 | |
| *** jgriffith is now known as Guest58267 | 16:10 | |
| VijayTripathi | Hello everyone | 16:11 |
| *** carl_baldwin has quit IRC | 16:11 | |
| clarkb | Guest58267: -A openstack-INPUT -s 15.126.217.114/32 -j ACCEPT and -A openstack-INPUT -s 15.126.216.93/32 -j ACCEPT in http://logs.openstack.org/97/131297/1/experimental/check-tempest-dsvm-aiopcpu/9ac7127/logs/iptables.txt.gz should cover us | 16:11 |
| clarkb | Guest58267: at least on the cinder control plane side which is being connected to? I am checking the other side for completeness | 16:11 |
| clarkb | http://logs.openstack.org/97/131297/1/experimental/check-tempest-dsvm-aiopcpu/9ac7127/logs/15.126.217.114/iptables.txt.gz has the same rules so I don't think firewall is at play. I am still digging through service logs to see what I can see | 16:12 |
| arosen | fungi: clarkb thanks for the quick approval, hopefully I this should let me get the CI in ordering order again :) | 16:12 |
| fungi | dtroyer: dns updates for devstack.org are implemented now. the old ip addresses were 207.97.227.245 for the domain address record and 50.31.209.254 for the www subdomain address | 16:12 |
| sdague | jeblair: zomg, new screen provides no semantic markup? | 16:13 |
| Guest58267 | clarkb: not sure, but looking at logs I'm not sure those hosts are talking at all... ie able to ping or ssh regardless of the iscsi tgt | 16:13 |
| fungi | dtroyer: now the domain has a and aaaa address records matching static.openstack.org, and www is a cname to the same | 16:14 |
| *** Guest58267 is now known as jdg | 16:14 | |
| jdg | sorry... hopefully I have a connection that is stable now | 16:14 |
| jeblair | sdague: no. it just might be possible that the complexity you objected to might have been required in order to function with the new screen. | 16:14 |
| openstackgerrit | Sergey Skripnick proposed a change to openstack-infra/project-config: Add hook for building rally docker image https://review.openstack.org/131389 | 16:15 |
| sdague | yeh... I'm trying to figure out how we can have nice things with the new change screen at all | 16:15 |
| *** emagana has joined #openstack-infra | 16:15 | |
| sdague | it's amazing how they stripped out all the semantically meaningful content there | 16:15 |
| clarkb | Guest20522: http://logs.openstack.org/97/131297/1/experimental/check-tempest-dsvm-aiopcpu/9ac7127/logs/15.126.217.114/screen-n-cpu.txt.gz?level=INFO has errors currently looking at those now as they are the first bread crumbs that are useable | 16:15 |
| clarkb | er jdg ^ | 16:16 |
| *** msabramo has quit IRC | 16:16 | |
| jeblair | sdague: the previous version collected the comments from either change screen and put them in a rather simple, easy to work with data structure. i may be biased, but i think that perhaps instead of changing everything, you could probably start with that. | 16:16 |
| sdague | jeblair: so the complexity of that made a bunch of higher level manipulations really hard | 16:17 |
| zaro | morning | 16:17 |
| jeblair | sdague: it literally just pulled out the comments and put them in a data structure | 16:17 |
| jeblair | sdague: it was a function called "ci_find_comments" | 16:17 |
| sdague | jeblair: right but it didn't have the grouping | 16:18 |
| dtroyer | fungi: looks good from here, thanks again | 16:18 |
| jeblair | sdague: sure, build on top of that | 16:18 |
| clarkb | jdg: oh I see, for some reason nova is using the internal addresses and not the public addresses | 16:18 |
| clarkb | jdg: so the ssh key hasn't been added | 16:18 |
| sdague | jeblair: that's actually what I did | 16:18 |
| clarkb | jeblair: ^ so it may be that nova is being smart here | 16:18 |
| jeblair | sdague: it output a list of comments, each item in the list was a hash with attrs like 'name', 'psnum', 'results', 'comment' | 16:18 |
| * clarkb digs more | 16:18 | |
| jeblair | sdague: you could add 'pipeline' to that pretty easily | 16:18 |
| jeblair | sdague: and then collaps | 16:19 |
| sdague | so, I can probably rework ci_parse_comments to work in the new screen as well, we'll have less info, but maybe that's ok | 16:20 |
| jeblair | sdague: why would we have less info? | 16:21 |
| jeblair | sdague: the same information is provided in both change screens | 16:21 |
| sdague | it's actually not, email isn't, for instance | 16:21 |
| jeblair | sdague: we need email? | 16:21 |
| *** stevemar has joined #openstack-infra | 16:21 | |
| sdague | I don't know, I'm just saying they aren't actually exactly the same. | 16:22 |
| *** zns has joined #openstack-infra | 16:22 | |
| *** ildikov has quit IRC | 16:22 | |
| *** ivar-lazzaro has joined #openstack-infra | 16:22 | |
| jeblair | sdague: anyway, i don't care whether you extend what i wrote or completely rewrite it. up to you | 16:22 |
| jeblair | however, we do need something that works on both change screens now | 16:23 |
| jdg | clarkb: yeah... also this may help going forward: https://review.openstack.org/131491 | 16:23 |
| jeblair | and it does not sound like this is a quick fix | 16:23 |
| sdague | yeh, give me an hour to see how bad this is going to be | 16:23 |
| asselin | clarkb, arosen marcusvrn_ I see where the openstack pypi setup is in my repo. I will delete it: https://github.com/rasselin/os-ext-testing/blob/master/puppet/modules/os_ext_testing/templates/jenkins_job_builder/config/macros.yaml.erb#L64 | 16:23 |
| jeblair | sdague: okay | 16:23 |
| *** freemanbrandon has joined #openstack-infra | 16:24 | |
| asselin | clarkb, arosen marcusvrn_ this is where i set my own pip mirror for prepare_node.sh: https://github.com/rasselin/os-ext-testing/blob/master/puppet/modules/os_ext_testing/files/nodepool/prepare_node_devstack_http_proxy.sh#L23 | 16:24 |
| clarkb | http://logs.openstack.org/97/131297/1/experimental/check-tempest-dsvm-aiopcpu/9ac7127/logs/15.126.217.114/etc/nova/nova.conf.txt.gz nova's my_ip is the internal IP... | 16:25 |
| clarkb | cinders is confirmed to be correct | 16:25 |
| *** ivar-laz_ has quit IRC | 16:25 | |
| jdg | clarkb: cool... | 16:26 |
| *** ashaeron has quit IRC | 16:26 | |
| *** miqui_ has quit IRC | 16:26 | |
| *** zns has quit IRC | 16:26 | |
| *** reed has joined #openstack-infra | 16:27 | |
| naggappan | @fungi if i use pre_test_hook it tried to change remote url before downloading . But if i add in post_test_hook then i am able to set my repository url . Is this the right way? or we need to use pre_test_hook | 16:27 |
| *** freemanbrandon has quit IRC | 16:28 | |
| fungi | naggappan: well, you need to update the version in the git repository cache devstack-gate sets up. devstack shouldn't re-clone if it's already in the cache i think, but i've not actually tried to override a repository that way. you may have more luck talking to people who run third-party ci systems to find out how they're actually solving this problem | 16:29 |
| *** dangers_away is now known as dangers | 16:29 | |
| *** zz_gondoi is now known as gondoi | 16:29 | |
| *** doug-fish has quit IRC | 16:29 | |
| *** msabramo has joined #openstack-infra | 16:31 | |
| *** dimtruck is now known as zz_dimtruck | 16:32 | |
| clarkb | jeblair: https://review.openstack.org/131492 fixes at least some of the problem | 16:32 |
| clarkb | (I think, I don't actually grok all of the reasons devstack might use one var or another in these cases) | 16:32 |
| *** unicell1 has quit IRC | 16:32 | |
| *** yamahata has quit IRC | 16:32 | |
| *** mriedem has quit IRC | 16:32 | |
| *** unicell has joined #openstack-infra | 16:32 | |
| *** dims__ has joined #openstack-infra | 16:33 | |
| jeblair | clarkb: but does that affect cinder.conf? | 16:33 |
| *** thedodd has joined #openstack-infra | 16:33 | |
| *** jaypipes is now known as jaypipes-afk | 16:33 | |
| clarkb | jeblair: no, but I think cinder tests may have been failing due to nova failures (tests were along the lines of boot from volume) | 16:33 |
| jeblair | clarkb: gotcha | 16:34 |
| clarkb | jeblair: I am not sure this will fix all of the problems, but it is definiately a problem | 16:34 |
| naggappan | @fungi what you mean by "update the version in git repository cache devstack-gate" is there any cache for all repositories and i need to update over there also ? | 16:35 |
| openstackgerrit | A change was merged to openstack-infra/system-config: Update pypi mirror to use pypi.dfw.openstack.org https://review.openstack.org/131474 | 16:36 |
| *** zz_dimtruck is now known as dimtruck | 16:37 | |
| *** ivar-lazzaro has quit IRC | 16:37 | |
| *** zns has joined #openstack-infra | 16:38 | |
| *** ivar-lazzaro has joined #openstack-infra | 16:38 | |
| fungi | naggappan: the loop over setup_project in setup_workspace checks out all projects listed in $PROJECTS into $DEST | 16:39 |
| *** bhuvan has joined #openstack-infra | 16:39 | |
| fungi | er, clones them | 16:39 |
| fungi | naggappan: that happens before the pre_test_hook is called | 16:42 |
| *** Guest74700 has quit IRC | 16:43 | |
| gondoi | how can I kick off tests again on a review? | 16:44 |
| gondoi | it passed initial tests, but then failed verification of an already passing test | 16:45 |
| gondoi | http://logs.openstack.org/73/131473/2/gate/gate-satori-pypy/a1d1b68/console.html | 16:45 |
| *** markmcclain has quit IRC | 16:45 | |
| *** harlowja_away is now known as harlowja | 16:46 | |
| clarkb | gondoi: leave a comment of 'recheck' | 16:46 |
| gondoi | awesome, thank you | 16:46 |
| *** Longgeek has quit IRC | 16:47 | |
| naggappan | @ok fungi thanks. Let me try with the post_test_hook itself if its not working, let me look for other solution | 16:47 |
| *** bhuvan_ has joined #openstack-infra | 16:47 | |
| fungi | naggappan: post_test_hook runs after your tests, so probably won't do what you want | 16:48 |
| *** doug-fish has joined #openstack-infra | 16:49 | |
| *** bhuvan has quit IRC | 16:50 | |
| *** bhuvan_ is now known as bhuvan | 16:50 | |
| *** tnurlygayanov has joined #openstack-infra | 16:51 | |
| sdague | jeblair: ok, I have a version working with new change screen, got to fix one thing I broke on old in the process | 16:53 |
| *** jgriffit1 has joined #openstack-infra | 16:54 | |
| sdague | the date parse is annoying, because in old we get full date, and in new we only get display date, but such it is | 16:54 |
| *** VijayTripathi has quit IRC | 16:54 | |
| *** jgriffit1 is now known as jgriffith | 16:55 | |
| *** jgriffith is now known as Guest15018 | 16:56 | |
| *** sarob has joined #openstack-infra | 16:56 | |
| *** jgriffith has joined #openstack-infra | 16:57 | |
| asselin | clarkb, I reconfigured my tests to not use hooks as suggested. I abandoned the change. | 17:00 |
| asselin | clarkb, sdague, let me know if there's still interest in having the devstack-gate hook refactoring change: https://review.openstack.org/#/c/123578/ | 17:01 |
| marcusvrn_ | fungi: clarkb asselin arosen thx all! The CI is working now with python.org repo | 17:01 |
| clarkb | asselin: ya I think we can leave that one up | 17:01 |
| *** ivar-laz_ has joined #openstack-infra | 17:01 | |
| *** zns has quit IRC | 17:02 | |
| *** freemanbrandon has joined #openstack-infra | 17:02 | |
| *** ildikov has joined #openstack-infra | 17:02 | |
| *** ivar-lazzaro has quit IRC | 17:04 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/system-config: fix hideci for new change screen https://review.openstack.org/131500 | 17:05 |
| sdague | jeblair: ^^^ | 17:05 |
| *** msabramo has quit IRC | 17:07 | |
| *** msabramo1 has joined #openstack-infra | 17:07 | |
| *** zns has joined #openstack-infra | 17:07 | |
| clarkb | my devstack change doesn't seem to be broken during check testing so thats good | 17:09 |
| *** carl_baldwin has joined #openstack-infra | 17:10 | |
| clarkb | I wonder if I can get d-g to check out that change when running experimental tests | 17:10 |
| jeblair | clarkb: that's the idea behind adding cross-repo deps to zuul, but afaik that's not done yet | 17:10 |
| jeblair | clarkb: though i assume with devstack-gate, you could bash something in there | 17:11 |
| clarkb | ya I think I can just bash something into setup_workspace | 17:12 |
| *** jpich has quit IRC | 17:12 | |
| fungi | yeah, an explicit checkout of that change into the cached devstack repo after or instead of the reset to master ought to do it | 17:13 |
| *** IvanBerezovskiy has left #openstack-infra | 17:13 | |
| *** changbl has joined #openstack-infra | 17:14 | |
| *** yaguang has quit IRC | 17:17 | |
| *** zns has quit IRC | 17:20 | |
| *** zns has joined #openstack-infra | 17:20 | |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/devstack-gate: Test nodepool provided addresses in aiopcpu setup https://review.openstack.org/131297 | 17:21 |
| clarkb | there it is bashed in | 17:21 |
| clarkb | I am going to do the trick to force nodes onto hpcloud for 2-node before I check experimental that change | 17:21 |
| *** lttrl has joined #openstack-infra | 17:22 | |
| *** VijayTripathi has joined #openstack-infra | 17:23 | |
| *** bhuvan has quit IRC | 17:23 | |
| *** bhuvan has joined #openstack-infra | 17:24 | |
| *** MaxV has quit IRC | 17:25 | |
| *** Ryan_Lane has joined #openstack-infra | 17:25 | |
| *** MaxV has joined #openstack-infra | 17:25 | |
| *** dprince has quit IRC | 17:27 | |
| *** tomoe has quit IRC | 17:29 | |
| *** stevemar is now known as notstevemar | 17:29 | |
| *** patrickeast has joined #openstack-infra | 17:29 | |
| *** MaxV has quit IRC | 17:30 | |
| *** naggappan has quit IRC | 17:31 | |
| *** achanda has joined #openstack-infra | 17:32 | |
| *** amotoki has quit IRC | 17:33 | |
| clarkb | https://jenkins02.openstack.org/job/check-tempest-dsvm-aiopcpu/15/ woot that is running now | 17:34 |
| clarkb | should give us good info | 17:35 |
| *** terryw has quit IRC | 17:37 | |
| *** jp_at_hp has quit IRC | 17:38 | |
| *** mriedem has joined #openstack-infra | 17:38 | |
| *** vhoward has left #openstack-infra | 17:40 | |
| *** achanda has quit IRC | 17:42 | |
| *** freemanbrandon has quit IRC | 17:45 | |
| *** amcrn has joined #openstack-infra | 17:46 | |
| *** morganfainberg is now known as alsonotstevemar | 17:46 | |
| *** alsonotstevemar is now known as morganfainberg | 17:47 | |
| *** achanda has joined #openstack-infra | 17:47 | |
| *** garyh has quit IRC | 17:49 | |
| *** achanda has quit IRC | 17:49 | |
| *** jgallard has quit IRC | 17:49 | |
| *** achanda has joined #openstack-infra | 17:49 | |
| *** sarob_ has joined #openstack-infra | 17:50 | |
| *** freemanbrandon has joined #openstack-infra | 17:50 | |
| *** markmcclain has joined #openstack-infra | 17:51 | |
| *** emagana has quit IRC | 17:52 | |
| *** emagana has joined #openstack-infra | 17:53 | |
| *** ZZelle_ has joined #openstack-infra | 17:54 | |
| *** markmcclain has quit IRC | 17:55 | |
| *** jdg has quit IRC | 17:56 | |
| *** emagana_ has joined #openstack-infra | 17:57 | |
| *** emagana has quit IRC | 17:57 | |
| *** sdake_ has quit IRC | 17:59 | |
| *** chandankumar has quit IRC | 17:59 | |
| *** zns has quit IRC | 17:59 | |
| *** dimtruck is now known as zz_dimtruck | 17:59 | |
| *** rmcall has quit IRC | 18:00 | |
| *** otherwiseguy has joined #openstack-infra | 18:00 | |
| *** zz_dimtruck is now known as dimtruck | 18:01 | |
| *** rmcall has joined #openstack-infra | 18:02 | |
| *** ivar-laz_ has quit IRC | 18:03 | |
| *** ivar-lazzaro has joined #openstack-infra | 18:04 | |
| SergeyLukjanov | jeblair, re hideci - I've been thinking that I test it... heh | 18:08 |
| SergeyLukjanov | jeblair, sdague, how are you testing hidejs changes? | 18:08 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/project-config: enforce stable requirements https://review.openstack.org/125095 | 18:09 |
| *** ociuhandu has joined #openstack-infra | 18:09 | |
| fungi | SergeyLukjanov: paste the raw version of the js file contents into the browser console | 18:09 |
| sdague | yeh, what fungi said | 18:09 |
| *** sarob has quit IRC | 18:09 | |
| fungi | and then browse around review.o.o and see what it does | 18:09 |
| sdague | copy / paste the whole file into the chrome js console (or ff I think) | 18:09 |
| lifeless | mordred: failing open? context? I am at a loss | 18:09 |
| SergeyLukjanov | fungi, hm, that what's I've done for this patch... sounds like there was lack of coffeine in my blood and I've done it incorrectly | 18:09 |
| *** dprince has joined #openstack-infra | 18:10 | |
| SergeyLukjanov | fungi, sdague, thanks | 18:10 |
| *** _nadya_ has quit IRC | 18:10 | |
| sdague | SergeyLukjanov: did you use the new change screen? | 18:10 |
| SergeyLukjanov | sdague, yup | 18:10 |
| sdague | the issue was that it didn't work if you had new change screen preferences on | 18:10 |
| *** tgohad has joined #openstack-infra | 18:10 | |
| sdague | 131500 fixes it | 18:11 |
| SergeyLukjanov | sdague, oh, got it (added note to test that changes on both screen prefs) | 18:11 |
| mordred | lifeless: succeeding when no tests are found | 18:11 |
| clarkb | mordred: is this the nova thing? | 18:11 |
| lifeless | mordred: strangely I got pushback from [some] cpython devs http://bugs.python.org/issue18232 | 18:11 |
| clarkb | mordred: testr can't do anything about that | 18:12 |
| fungi | mordred: lifeless: in this case it turned out that the module being imported had no main() so the import was a successful noop, and this we never got a testrepository file to inspect to see how many tests were run | 18:12 |
| clarkb | the way nova (and a lot of projects run their tests) is something like python -m oslo.lockutils testr | 18:12 |
| lifeless | mordred: but as we've discussed I'm working on these things upstream | 18:12 |
| clarkb | if oslo.lockutil exits 0 then testr is never even invoked | 18:12 |
| lifeless | clarkb: -wat- | 18:12 |
| clarkb | lifeless: don't get me started | 18:13 |
| lifeless | clarkb: that makes no sense to me | 18:13 |
| fungi | yeah, seems like a bad choice to me | 18:13 |
| clarkb | lifeless: we should make a club | 18:13 |
| lifeless | clarkb: you don't wrap the orchestrator | 18:13 |
| sdague | so... we're fixing that | 18:13 |
| clarkb | lifeless: no no they do | 18:13 |
| lifeless | clarkb: _why_ | 18:13 |
| clarkb | lifeless: global locking | 18:13 |
| fungi | not for much longer apparently ;) | 18:13 |
| lifeless | still not following | 18:13 |
| lifeless | testr is concurrent | 18:13 |
| clarkb | lifeless: the wrapper sets up global locks | 18:13 |
| clarkb | yes | 18:13 |
| sdague | because oslo lockutils doesn't work without extra setup | 18:13 |
| *** tsg has quit IRC | 18:13 | |
| clarkb | and its silly | 18:13 |
| sdague | honestly, I think this is beer talk in paris :) | 18:14 |
| sdague | there are many layers of silly here | 18:14 |
| clarkb | well I tried to -1 it back in the day | 18:14 |
| sdague | I'm hoping to kill a few of them today | 18:14 |
| dhellmann | clarkb: the wrapper sets up the lock DIRECTORY not the locks themselves. It sets an environment variable. | 18:14 |
| lifeless | sdague: that should be per test backend though, with process long fixtures to isolate each backend, or you'll have terrible trrouble with N-context tests. | 18:14 |
| clarkb | dhellmann: but why | 18:14 |
| clarkb | dhellmann: just put it in the tests | 18:14 |
| lifeless | clarkb: I now understand why- see my comment^ | 18:14 |
| dhellmann | clarkb: sure. I'm just explaining what it actually does. | 18:14 |
| lifeless | sdague: sure, it can be a beer discussion, or I'll just get a patch together when I finish this EPIC yak shave | 18:15 |
| clarkb | dhellmann: back when I was more angry about it I was told it was for db tests to have a global lock | 18:15 |
| clarkb | dhellmann: except you don't need a global lock for db tests | 18:15 |
| dhellmann | clarkb: the script doesn't create any locks | 18:15 |
| clarkb | ok tempest is still failing with my bashed in devstack change. will need to look at logs more when it is done | 18:16 |
| clarkb | anyways we cargo culted a thing that should never have been written in the first place. It is very frustrating that we don't scrutinize the cargo when we cult | 18:16 |
| lifeless | sdague: but in terms of fitting in with the rest of the test architecture, there really isn't anything to discuss except 'how do we fix this' | 18:16 |
| clarkb | http://logs.openstack.org/97/131297/2/experimental/check-tempest-dsvm-aiopcpu/e49cb03/logs/ for the logs. looks like nova my_ip is correct now to the n-cpu logs | 18:20 |
| sdague | lifeless: sure, mostly I think the beer talk is to look at some other silly things we've let creep in and figure out which bits we can unwind | 18:22 |
| mtreinish | clarkb: http://logs.openstack.org/97/131297/2/experimental/check-tempest-dsvm-aiopcpu/e49cb03/logs/15.126.209.255/screen-n-cpu.txt.gz?level=TRACE looks suspicious | 18:22 |
| arosen | From time to time i see my nodepool nodes go offline in jenkins. Right now the error message is: Disconnected by anonymous : Offline due to Gearman request | 18:23 |
| mtreinish | which matches up with the resize test failures | 18:23 |
| arosen | is there anything i can do about this? | 18:23 |
| *** garyh has joined #openstack-infra | 18:23 | |
| *** carl_baldwin has quit IRC | 18:24 | |
| fungi | arosen: that is the node being offlined by the jenkins-gearman plugin, generally because either a job completed or was aborted | 18:24 |
| arosen | the jenkins log says: WARNING: Process leaked file descriptors. See http://wiki.jenkins-ci.org/display/JENKINS/Spawning+processes+from+build for more information | 18:24 |
| clarkb | mtreinish: ya so I fixed nova's my_ip which got us past the ssh errors | 18:24 |
| arosen | http://paste.openstack.org/show/126074/ | 18:24 |
| clarkb | mtreinish: but now qemu-img is failing on both nodes and those operations look local | 18:24 |
| lifeless | clarkb: oh | 18:25 |
| fungi | arosen: that warning is generally benign... just means your job created one or more files outside the workspace (unavoidable if you're running devstack-gate) | 18:25 |
| arosen | fungi: oh maybe you're right let me dig closer. | 18:25 |
| lifeless | clarkb: that failing open... what was the return code from the lockutils thing? | 18:25 |
| clarkb | lifeless: 0 | 18:25 |
| lifeless | clarkb: waaaaat | 18:25 |
| clarkb | lifeless: the reason is oslo.concurrency remove their if __main__ check | 18:25 |
| clarkb | so python -m happily imported the module and did nothing returned 0 | 18:26 |
| lifeless | clarkb: I'm just surprised and astounded at python right now | 18:26 |
| *** zns has joined #openstack-infra | 18:26 | |
| arosen | fungi: also it's weird because nodepool list ; doesn't show my nodes are in 'use' even though jenkins is using them. | 18:26 |
| lifeless | clarkb: and then I remembered, this is a consequence of python 1 | 18:26 |
| lifeless | clarkb: and there being no formal 'this is executable' in python land | 18:26 |
| fungi | arosen: if they all continue to show "ready" then your nodepool probably isn't successfully getting events from the zeromq event publisher plugin on your jenkins master | 18:27 |
| *** zns has quit IRC | 18:27 | |
| jeblair | clarkb: my_ip = 15.126.209.254 | 18:27 |
| jeblair | clarkb: http://logs.openstack.org/97/131297/2/experimental/check-tempest-dsvm-aiopcpu/e49cb03/logs/15.126.209.255/etc/nova/nova.conf.txt.gz | 18:27 |
| clarkb | jeblair: yup | 18:27 |
| jeblair | clarkb: i dunno how to run an openstack.... | 18:27 |
| clarkb | jeblair: so I think my devstack change fixes that bit | 18:27 |
| jeblair | clarkb: but i feel like saying "my ip is .254" on a host whose ip is ".255" may have problems | 18:28 |
| *** zns has joined #openstack-infra | 18:28 | |
| clarkb | oh is the ip wrong? | 18:28 |
| jeblair | clarkb: yeah, i think it's correct on the .254 host but incorrect on the .255 | 18:28 |
| clarkb | well its set to SERVICE_HOST everywhere | 18:28 |
| clarkb | I think my_ip is really "where is the controller ip" | 18:28 |
| clarkb | but I could be wrong | 18:29 |
| jeblair | clarkb: i mean, maybe "my_ip" should be .254 on .255 | 18:29 |
| jeblair | clarkb: but i can not reason about such a thing, so i'm going to go back to what i was doing before :) | 18:29 |
| clarkb | jeblair: :) probably a good idea | 18:29 |
| *** thedodd has quit IRC | 18:30 | |
| dhellmann | git experts: We may have a case where we want to take a module from oslo-incubator and put it in an existing library repository (oslo.utils). Is it possible to do that and preserve the history? I think I understand how to get the 2 commit trees into one repo and merged, but not with gerrit in the mix. | 18:32 |
| *** garyh has quit IRC | 18:33 | |
| dhellmann | it's not the end of the world if we can't, but I want to explore options before just copying the file and losing the history | 18:33 |
| jeblair | dhellmann: gerrit is just another git repo; we can push to a branch and then make a merge commit of that branch into master | 18:34 |
| *** HeOS has quit IRC | 18:34 | |
| jeblair | (and the merge commit can be reviewed) | 18:34 |
| *** otherwiseguy has quit IRC | 18:34 | |
| *** packet has joined #openstack-infra | 18:34 | |
| clarkb | I have held the neutron tests's pair of nodes to poke around on disk | 18:35 |
| dhellmann | jeblair: ok, cool. I'm happy with the "it's possible" answer for now but may come back for more detailed instructions after the summit. | 18:35 |
| arosen | fungi is rebooted the box and still seems like the node pool nodes aren't going to ready. Is there a specific log file i should look at for this? | 18:35 |
| clarkb | well thats interesting | 18:37 |
| clarkb | it looks like n-cpu on the primary node is running qemu-img for a host on the sub node | 18:37 |
| clarkb | and so the file does not exist because we are running on the wrong node | 18:38 |
| clarkb | maybe my_ip is actually wrong? | 18:38 |
| *** tonytan4ever has quit IRC | 18:38 | |
| clarkb | why is nova ssh'ing to localhost then | 18:38 |
| fungi | arosen: you should check that your zeromq event publisher plugin on the jenkins master is enabled, and probably look for traffic for that address/port | 18:39 |
| clarkb | so maybe what I need to do is abandon the devstack change and set HOST_IP to the value in the appropriate /etc/nodepool file | 18:39 |
| *** dane_leblanc has quit IRC | 18:40 | |
| clarkb | jeblair: except that if we grow beyond 1 subnode then those files are not sufficient to know what the external IPs are | 18:40 |
| arosen | fungi: ah okay it's started working again. I'm not sure if it was related to running this though: | 18:40 |
| arosen | sudo jenkins-jobs --flush-cache update /etc/jenkins_jobs/config | 18:40 |
| *** carl_baldwin has joined #openstack-infra | 18:40 | |
| arosen | seems like it shouldn't be... | 18:40 |
| clarkb | and in this case cinder works because it doesn't run on two nodes? | 18:41 |
| jeblair | dhellmann: cool. we also did once merge a git history into a repo without even a common shared point on the dag (two completely independent repos merged). if that's possible, anything is :) | 18:41 |
| dhellmann | jeblair: it's like magic! | 18:41 |
| clarkb | so why would nova ssh to itself though | 18:41 |
| jeblair | clarkb: not following: 18:39 < clarkb> so maybe what I need to do is abandon the devstack change and set HOST_IP to the value in the appropriate /etc/nodepool file | 18:41 |
| clarkb | jeblair: I believe my_ip is supposed to be the IP of the localhost | 18:42 |
| clarkb | jeblair: but in hpcloud we have no way of getting that info reliably | 18:43 |
| clarkb | for the public floating ip | 18:43 |
| *** sdake_ has joined #openstack-infra | 18:43 | |
| clarkb | http://logs.openstack.org/97/131297/1/experimental/check-tempest-dsvm-aiopcpu/9ac7127/logs/15.126.217.114/screen-n-cpu.txt.gz?level=TRACE is failing to ssh to the local ip | 18:43 |
| jeblair | clarkb: so you want to set my_ip to a value supplied by nodepool, but need that to scale to n-hosts that are not the primary | 18:43 |
| clarkb | jeblair: if we want to use the public floating ips on each node: yes | 18:43 |
| *** tonytan4ever has joined #openstack-infra | 18:44 | |
| jeblair | clarkb: i agree that is not possible now, that it is desirable, and that we should enhance nodepool to supply that information | 18:45 |
| clarkb | I am going to go ahead and try it with the sub_nodes file data now to see if I can get this to work if the IPs are correct | 18:45 |
| clarkb | also I have no idea why nova does ssh like that | 18:45 |
| jeblair | clarkb: ++ | 18:45 |
| clarkb | sdague: jogo ^ do you guys know why nova does that? | 18:47 |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/devstack-gate: Test nodepool provided addresses in aiopcpu setup https://review.openstack.org/131297 | 18:51 |
| clarkb | https://jenkins04.openstack.org/job/check-tempest-dsvm-aiopcpu/9/ is running with external IPs as HOST_IP now | 18:52 |
| *** cdent has quit IRC | 18:54 | |
| *** dane_leblanc has joined #openstack-infra | 18:55 | |
| *** baoli has quit IRC | 18:55 | |
| clarkb | it looks like nova does that ssh for "migration" operations where that operation can be entirely local? | 18:56 |
| *** andreykurilin_ has joined #openstack-infra | 18:56 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/nodepool: Add current node IP to /etc/nodepool https://review.openstack.org/131532 | 18:58 |
| clarkb | jeblair: that should probably come with a doc update (the other files are documented) | 18:59 |
| *** zns has quit IRC | 19:00 | |
| fungi | meeting time... | 19:00 |
| *** jaypipes-afk is now known as jaypipes | 19:01 | |
| *** cinderbld has joined #openstack-infra | 19:01 | |
| *** pelix has quit IRC | 19:02 | |
| *** emagana_ has quit IRC | 19:02 | |
| *** naggappan has joined #openstack-infra | 19:02 | |
| *** dgenin has joined #openstack-infra | 19:03 | |
| *** emagana has joined #openstack-infra | 19:03 | |
| dgenin | dtroyer: do you have a minute for a cinder lvm question? | 19:03 |
| *** dane_leblanc has quit IRC | 19:04 | |
| *** emagana_ has joined #openstack-infra | 19:05 | |
| *** emagana has quit IRC | 19:05 | |
| *** sdake_ has quit IRC | 19:06 | |
| *** tonytan4ever has quit IRC | 19:06 | |
| *** dprince has quit IRC | 19:07 | |
| *** AJaeger has joined #openstack-infra | 19:07 | |
| *** mestery has quit IRC | 19:07 | |
| *** dane_leblanc has joined #openstack-infra | 19:09 | |
| *** mestery has joined #openstack-infra | 19:09 | |
| *** darvon has quit IRC | 19:15 | |
| *** darvon has joined #openstack-infra | 19:16 | |
| *** sdake_ has joined #openstack-infra | 19:16 | |
| *** mjturek has quit IRC | 19:16 | |
| *** andreykurilin_ has quit IRC | 19:17 | |
| *** arosen has quit IRC | 19:17 | |
| *** andreykurilin_ has joined #openstack-infra | 19:18 | |
| *** garyh has joined #openstack-infra | 19:19 | |
| *** jedimike has quit IRC | 19:21 | |
| *** mjturek has joined #openstack-infra | 19:29 | |
| *** teran has joined #openstack-infra | 19:29 | |
| *** mjturek has left #openstack-infra | 19:30 | |
| *** gyee has quit IRC | 19:31 | |
| *** tonytan4ever has joined #openstack-infra | 19:31 | |
| *** dizquierdo has quit IRC | 19:32 | |
| *** dgenin has quit IRC | 19:33 | |
| *** teran_ has quit IRC | 19:33 | |
| jogo | clarkb: it uses ssh for libvirt based migration if no shared disk | 19:34 |
| jogo | clarkb: that doesn't have to be the public IP, that can be the private IP | 19:34 |
| clarkb | jogo: right but it does so even when talking to localhost for a disk resize apparently | 19:34 |
| clarkb | jogo: we want it to be the public ip | 19:34 |
| clarkb | what was confusing to me is nova basically does `ssh localhost mkdir` | 19:35 |
| *** arosen has joined #openstack-infra | 19:35 | |
| jogo | clarkb: a resize up needs to be rescheduled | 19:35 |
| jogo | clarkb: fewer code paths that way | 19:35 |
| clarkb | its a longer code path... | 19:35 |
| jogo | clarkb: why do you want it to be the public IP | 19:35 |
| clarkb | you are relying on network services that are not required | 19:35 |
| jogo | clarkb: how much overhead would that incur? | 19:36 |
| jogo | i am not convinced its worth changing, but I do agree it is odd | 19:36 |
| *** pack3t has joined #openstack-infra | 19:39 | |
| *** pack3t has quit IRC | 19:39 | |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/project-config: Add Juno stable-compat-jobs to trunk sahara client https://review.openstack.org/131542 | 19:40 |
| *** e0ne has quit IRC | 19:42 | |
| clarkb | jogo: Ithink the biggest overhead is for humans | 19:43 |
| clarkb | jogo: you immediately assume remote connection with ssh | 19:43 |
| jogo | clarkb: that is fair | 19:43 |
| jogo | I wonder what dansmith thinks ^ | 19:44 |
| clarkb | (like I did and completely went down the wrong hole for a while) | 19:44 |
| pleia2 | zaro: so this has been bothering me for quite some time: http://people.ubuntu.com/~lyz/openstack/OS-gerrit-screenshot-10.28.2014.png (see bottom) | 19:44 |
| pleia2 | turns out chrome on mobile does this awful FontBoosting thing that seems mostly broke, but there is a hacky fix | 19:44 |
| pleia2 | broken | 19:44 |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/project-config: Use common integrated jobs for sahara https://review.openstack.org/131543 | 19:44 |
| pleia2 | would this be upstream, or is there a way for us to modify our instance? I could only find the css I want to change in our gerrit repo :\ | 19:45 |
| dansmith | jogo: clarkb: reading | 19:45 |
| jogo | clarkb: btw OT, we broke py26 in nova already so hopefully we can deal with any concerns complaints early in the cycle | 19:45 |
| clarkb | that was fast | 19:46 |
| jogo | clarkb: it was intentional | 19:46 |
| *** hdd has joined #openstack-infra | 19:46 | |
| dansmith | jogo: what's the question? why we ssh to localhost? | 19:46 |
| jogo | dansmith: yes, for resize | 19:46 |
| jogo | dansmith: I thought it was to use the same code path to make things easier | 19:47 |
| dansmith | jogo: is it part of the libvirt connection or the disk stuff? | 19:47 |
| dansmith | jogo: right, I'm sure it's that | 19:47 |
| jogo | dansmith: http://logs.openstack.org/97/131297/1/experimental/check-tempest-dsvm-aiopcpu/9ac7127/logs/15.126.217.114/screen-n-cpu.txt.gz?level=TRACE | 19:47 |
| jogo | dansmith: clarkb found it counter intuitive making debugging much harder, which is a reasonable concern | 19:48 |
| *** _buzz_ has joined #openstack-infra | 19:48 | |
| dansmith | but was it fixed by setting my_ip properly? | 19:48 |
| *** julim has quit IRC | 19:48 | |
| clarkb | dansmith: no, it was proper before | 19:49 |
| clarkb | I thought it was wrong because I didn't realize nova would ssh to my_ip | 19:49 |
| clarkb | but I had to go around and do it wrong before I figured it out :) | 19:49 |
| dansmith | okay, I guess I'm missing something | 19:49 |
| dansmith | because this is what you were hitting before you changed it at all, right? | 19:49 |
| clarkb | yes | 19:49 |
| clarkb | our nodes have two IPs | 19:50 |
| clarkb | a public IP and a private IP | 19:50 |
| clarkb | we are trying to run stuff on the public IP | 19:50 |
| clarkb | but my_ip was the private IP | 19:50 |
| zaro | pleia2: not sure. since we are behind upstream, it might have already been fixed upstream. | 19:50 |
| clarkb | so when nova did ssh my_ip it hit an address for which it did not have an accepted ssh host key | 19:50 |
| *** mwagner_lap has quit IRC | 19:50 | |
| clarkb | I thought it was ssh'ing to the private IP of the other host because why would you ssh to local ip | 19:50 |
| dansmith | I see, but don't we ssh to both? | 19:51 |
| *** _buzz has quit IRC | 19:51 | |
| pleia2 | zaro: ok, I'll dig around and see, is there a vanilla upstream gerrit instance I can look at somewhere? (not openstack, just something some project is using) | 19:51 |
| dansmith | i.e.: ssh to localhost and ssh to the remote machine | 19:51 |
| clarkb | dansmith: not in this case, its just a local resize | 19:51 |
| dansmith | clarkb: there's no such thing | 19:51 |
| clarkb | wat | 19:51 |
| clarkb | you have to move a node to resize it? | 19:51 |
| dansmith | clarkb: resize is expected to move the instance to another machine | 19:51 |
| clarkb | thats ok | 19:52 |
| dansmith | clarkb: the hack in the config file is just there to support single-node testing | 19:52 |
| jogo | dansmith: don't we set a flag in devstack to turn on local resize | 19:52 |
| jogo | dansmith: we should drop that for multi-node right? | 19:52 |
| dansmith | jogo: we used to, otherwise resizes would fail because there were no hosts | 19:52 |
| openstackgerrit | Ramy Asselin proposed a change to openstack-infra/project-config: Refactor puppet-apply jobs into template https://review.openstack.org/131548 | 19:52 |
| dansmith | er, I read that wrong | 19:52 |
| zaro | pleia2: gerrit-review.googlesource.com is pretty close to upstream. | 19:52 |
| dansmith | jogo: right, we should drop that in multinode | 19:52 |
| pleia2 | zaro: thanks | 19:52 |
| clarkb | in any case ssh some local ip that isn't localhost but is effectively localhost is super duper confusing | 19:53 |
| jogo | iniset $NOVA_CONF DEFAULT allow_resize_to_same_host "True" | 19:53 |
| jogo | in lib/nova in devstack | 19:53 |
| dansmith | clarkb: well, I guess I can see that, but I'm not sure I think it's worth changing just for the debugging case | 19:53 |
| pleia2 | zaro: aha, not fixed! | 19:54 |
| dansmith | because I htink that code is designed to be able to be run from potentially a third party location | 19:54 |
| dansmith | where it's just ssh $src ; ssh $dst | 19:54 |
| clarkb | even `if ip == my_ip then ssh localhost` would help | 19:54 |
| clarkb | but that would potentially cause lots of deployments to need to go add the host key for localhost | 19:55 |
| clarkb | so probably not worth the pain of changing | 19:55 |
| dansmith | clarkb: right | 19:55 |
| dansmith | clarkb: was just going to say that :) | 19:55 |
| dansmith | clarkb: the way it is, makes it easy to distribute a known_hosts file | 19:55 |
| *** naggappan has quit IRC | 19:55 | |
| clarkb | ya | 19:55 |
| *** otherwiseguy has joined #openstack-infra | 19:56 | |
| jogo | dansmith: just running the same tests but with allow resize and allow migrate to same hosts off will be huge | 19:59 |
| sdague | resize_to_same_host isn't a live op | 20:00 |
| sdague | unless a thing changed | 20:00 |
| dansmith | jogo: I'm not sure what you mean | 20:00 |
| *** HeOS has joined #openstack-infra | 20:00 | |
| jogo | dansmith: w.r.t. testing | 20:00 |
| jogo | sdague: allow_resize_to_same_host is defined in nova.compute.api | 20:01 |
| *** emagana_ has quit IRC | 20:01 | |
| sdague | jogo: right, but I thought everything that used that path didn't need ssh | 20:01 |
| jogo | same for allow_migrate_to_same_host | 20:01 |
| AJaeger | jeblair: could you review https://review.openstack.org/#/c/129435/ , please? I'd like your comments on python 2.6 removal from infra projects | 20:01 |
| sdague | allow_migrate_to_same_host ... on the other hand | 20:01 |
| clarkb | jeblair: new thing I have discovered with this multinode stuff. we are using the pivate address in all hpcloud tests today | 20:01 |
| clarkb | also down to 2 failures | 20:01 |
| *** emagana has joined #openstack-infra | 20:02 | |
| clarkb | in http://logs.openstack.org/97/131297/3/experimental/check-tempest-dsvm-aiopcpu/065b025/logs/ VM netowrking appears to be broken | 20:02 |
| jeblair | AJaeger: clarkb pointed out in the infra meeting we actually do need 2.6 for zuul; do you want me to -1 that and request removal, or add it back with a followup patch? | 20:02 |
| clarkb | does nova network require everything to be on the same layer 2 network? | 20:02 |
| *** zns has joined #openstack-infra | 20:02 | |
| jogo | sdague: with allow_resize_to_same_host off it just adds the current host to the ignored_hosts list | 20:03 |
| dansmith | yeah, I don't think setting that has any effect on what we do and don't do re: ssh | 20:03 |
| *** baoli has joined #openstack-infra | 20:03 | |
| jeblair | fungi: i'm stopping by new york on my way to paris: https://www.softwarefreedom.org/events/2014/sflc-10th-anniversary-conference/ | 20:03 |
| pleia2 | zaro: so here's the issue: http://people.ubuntu.com/~lyz/openstack/OS-gerrit-screenshot-10.28.2014.png & http://people.ubuntu.com/~lyz/openstack/Gerrit-gerrit-screenshot-10.28.2014.png ; if anyone cares, it's a one line fix in our gerrit-gwtui/src/main/java/com/google/gerrit/client/gerrit.css | 20:03 |
| jogo | sdague: what would be the best way to make it optional to turn on *_to_same_host | 20:03 |
| pleia2 | zaro: but maybe it doesn't matter and I shouldn't be opening reviews in chrome on my phone :) | 20:04 |
| jeblair | pleia2: is your puppetconf talk available on video? | 20:04 |
| fungi | jeblair: oh, that looks exciting! | 20:04 |
| sdague | jogo: tc meeting now, can noodle on it after | 20:04 |
| pleia2 | jeblair: yep, let me grab the link | 20:04 |
| jogo | sdague: kk | 20:04 |
| clarkb | jeblair: I think that makes me somewhat partial to starting with those internal addresses on the first iteration of this | 20:05 |
| AJaeger | jeblair: Let me remove it and resubmit. | 20:05 |
| *** zns has quit IRC | 20:05 | |
| openstackgerrit | A change was merged to openstack-infra/jenkins-job-builder: Added support for Rich Text Publisher Plugin https://review.openstack.org/130455 | 20:05 |
| pleia2 | jeblair: https://puppetlabs.com/presentations/how-open-source-your-puppet-configuration-elizabeth-krumbach-joseph-hp | 20:05 |
| pleia2 | puppet-monty loses his glasses | 20:06 |
| *** emagana has quit IRC | 20:06 | |
| clarkb | dansmith: jogo: http://logs.openstack.org/97/131297/3/experimental/check-tempest-dsvm-aiopcpu/065b025/console.html has no route to host and ssh timeouts when trying to hit the cirros VM | 20:06 |
| *** ayoung has quit IRC | 20:07 | |
| dansmith | clarkb: neutron or n-net? | 20:07 |
| clarkb | dansmith: nova network | 20:07 |
| jeblair | pleia2: oh no! | 20:07 |
| *** freemanbrandon has quit IRC | 20:08 | |
| jogo | clarkb: so how are you setting up the networking underneath | 20:08 |
| *** thedodd has joined #openstack-infra | 20:08 | |
| jogo | clarkb: as in you have the aio and the ncpu nodes | 20:08 |
| clarkb | jogo: lol whatever devstack does | 20:08 |
| *** todd_dsm has quit IRC | 20:08 | |
| openstackgerrit | Andreas Jaeger proposed a change to openstack-infra/project-config: Remove python26 jobs from various projects https://review.openstack.org/129435 | 20:08 |
| jogo | how does aio know how to route which VM IPs to npcu | 20:08 |
| clarkb | note that neutron has the same failures too http://logs.openstack.org/97/131297/3/experimental/check-tempest-dsvm-neutron-aiopcpu/df58711/ | 20:08 |
| clarkb | jogo: I have no idea | 20:08 |
| jogo | clarkb: I dont't think devstack knows how to do that | 20:09 |
| dansmith | clarkb: multihost or normal? | 20:09 |
| clarkb | dansmith: both are mutlihost | 20:09 |
| *** markmcclain has joined #openstack-infra | 20:09 | |
| dansmith | okay | 20:09 |
| clarkb | jogo: I do not know how nova net expects that to be configured | 20:09 |
| clarkb | do they all have to be on the same layer 2 network? | 20:09 |
| AJaeger | clarkb, jeblair, fungi: 129435 should now be fine | 20:09 |
| jogo | clarkb: I don't remember off the top of my head | 20:10 |
| clarkb | http://logs.openstack.org/97/131297/3/experimental/check-tempest-dsvm-aiopcpu/065b025/logs/15.126.215.109/etc/nova/nova.conf.txt.gz is the nova config on the sub node | 20:10 |
| jogo | clarkb: I don't think its a nova issue actually | 20:10 |
| *** markmcclain1 has joined #openstack-infra | 20:10 | |
| clarkb | it probably isn't | 20:11 |
| *** freemanbrandon has joined #openstack-infra | 20:11 | |
| clarkb | hrm nova net runs on the primary node but not the secondary? | 20:11 |
| *** emagana has joined #openstack-infra | 20:11 | |
| jogo | clarkb: without adding special routing info I am not sure if this could even work | 20:11 |
| clarkb | jogo: sure, but lets not think about that yet | 20:11 |
| jogo | vishy: you around | 20:11 |
| clarkb | I am trying to understand what nova expects | 20:11 |
| *** cinderbld has quit IRC | 20:11 | |
| vishy | jogo: yeah but in tc meeting | 20:11 |
| clarkb | does nova-net need to run on every node? | 20:12 |
| *** carl_baldwin has quit IRC | 20:12 | |
| jogo | clarkb: in one mode yes, but I don't thin kthat is enabled | 20:12 |
| clarkb | ok if it doesn't then I assume every node must have to be on the same l2 network | 20:12 |
| *** markmcclain has quit IRC | 20:12 | |
| clarkb | otherwise how does anything work | 20:13 |
| *** _nadya_ has joined #openstack-infra | 20:13 | |
| jogo | dansmith: what config option is used to set nova network multihost mode | 20:13 |
| jogo | clarkb: I think the latter | 20:14 |
| *** emagana_ has joined #openstack-infra | 20:14 | |
| jogo | and there needs to be routing to the L2 network from the test node somehow | 20:14 |
| dansmith | jogo: think it's just multihost=yes somewhere in there | 20:14 |
| jogo | dansmith: looking at nova.conf.sample | 20:14 |
| pleia2 | going to go find a snack (and possibly a quick swim), bbiab | 20:14 |
| *** emagana has quit IRC | 20:14 | |
| jogo | ahh multi_host | 20:15 |
| *** aysyd has quit IRC | 20:15 | |
| jogo | vishy: we are trying to set up nova-network in a two node devstack gate job | 20:15 |
| jogo | for when you have a momment | 20:15 |
| *** markmcclain1 has quit IRC | 20:15 | |
| clarkb | I am reading nova network docs now | 20:16 |
| vishy | jogo: ok that should be prettty easy | 20:16 |
| vishy | * pretty | 20:16 |
| jogo | clarkb: yeah we aren't running multi_host | 20:16 |
| vishy | you want no multi-host? | 20:16 |
| *** mrmartin has quit IRC | 20:17 | |
| *** spzala has quit IRC | 20:17 | |
| jogo | vishy: I don't think it matters much either way at least not right now | 20:17 |
| jogo | whichever is easier for now | 20:17 |
| jogo | long term we want the one we would like to test more (multi-host?) | 20:17 |
| clarkb | yes we already test not mutlihost | 20:17 |
| clarkb | this test env should test the things we can only test on mutlihost | 20:17 |
| jogo | clarkb: what to get vishy up to speed | 20:17 |
| jogo | clarkb: like multi_node nova-network mode ;) | 20:18 |
| *** otherwiseguy has quit IRC | 20:19 | |
| vishy | ok so multihost is pretty simple | 20:19 |
| jogo | vishy: well right now this is the issue: http://logs.openstack.org/97/131297/3/experimental/check-tempest-dsvm-aiopcpu/065b025/console.html#_2014-10-28_19_23_31_235 | 20:19 |
| vishy | do we always use flaoting ips in tests right now | 20:19 |
| vishy | or are the instances reachable via fixed? | 20:20 |
| jogo | we cannot ssh into the VMs nova is creating | 20:20 |
| clarkb | lets backup | 20:20 |
| jogo | I don't think we are using floating IPs for the VMs at least not public ones | 20:20 |
| clarkb | everything currently works using the "fixed" ips | 20:20 |
| jogo | clarkb: yes please | 20:20 |
| clarkb | (I really dislike that we have about 3 terms for these things) | 20:20 |
| clarkb | internal, private, fixed | 20:20 |
| jogo | clarkb: lets clarify which level of 'VM's we are talking about | 20:21 |
| clarkb | the host VMs | 20:21 |
| clarkb | so the things hpcloud and rackspace give us | 20:21 |
| *** freemanbrandon has quit IRC | 20:21 | |
| jogo | right, just wanted to make sure we are all on the same page | 20:21 |
| clarkb | jeblair has asked that we use the public IPs instead of the "fixed" IPs | 20:22 |
| clarkb | it simplifies some things on the infra side to do so (we basically run everything assuming it is public) | 20:22 |
| clarkb | in rackspace we get normal interfaces with the public IPs | 20:22 |
| clarkb | in hpcloud we get NATed interfaces with floating IPs | 20:22 |
| fungi | (or if we can't use the public ips then at least identifying why so we know where to file the appropriate bug reports) | 20:22 |
| clarkb | now when running the current setup against "fixed" IPs it just works | 20:23 |
| clarkb | when running the current setup against public IPs in rackspace it just works | 20:23 |
| openstackgerrit | Ramy Asselin proposed a change to openstack-infra/project-config: Refactor puppet-apply jobs into template https://review.openstack.org/131548 | 20:23 |
| clarkb | when running the current setup against HPCloud public IPs it does not work | 20:23 |
| *** kgiusti has quit IRC | 20:23 | |
| clarkb | we are configured to use the flat dhcp manager right now | 20:24 |
| clarkb | it appears that that requires everything be on the same layer 2 network | 20:24 |
| clarkb | so that packets bound for addresses just end up on the right bridge | 20:25 |
| clarkb | if that is the case it would explain why it does not work when we go to hpcloud public IPs | 20:25 |
| clarkb | however reading nova network docs it appears that a cloudpipe vpn can be set up if we use the vlanmanager | 20:25 |
| clarkb | what is not clear at all is what does mutli_host do? | 20:25 |
| *** sdake_ has quit IRC | 20:25 | |
| jogo | vishy: ^ | 20:27 |
| *** JayJ has quit IRC | 20:27 | |
| clarkb | is multi_host a way of saying I am running nova net in multiple locations please don't trample each other? | 20:27 |
| clarkb | then we would have to run nova-net on both nodes? | 20:28 |
| *** JayJ has joined #openstack-infra | 20:28 | |
| vishy | clarkb: i was actually referring to tempest testing | 20:28 |
| *** nelsnelson has quit IRC | 20:28 | |
| vishy | against devstack | 20:28 |
| *** whoops has quit IRC | 20:28 | |
| vishy | clarkb: multi host means a nova-net on each node yes | 20:29 |
| *** nelsnelson has joined #openstack-infra | 20:29 | |
| *** _nadya_ has quit IRC | 20:29 | |
| clarkb | and the cloudpipe vpn thing can cross layer 3 networks? | 20:29 |
| *** r1chardj0n3s_afk is now known as r1chardj0n3s | 20:30 | |
| vishy | clarkb: ignore the cloudpipe vpn | 20:31 |
| vishy | it is useless :) | 20:31 |
| *** dkranz has quit IRC | 20:31 | |
| vishy | clarkb: you really need the nodes on the same layer 2 network for this to work properly | 20:31 |
| clarkb | vishy: thats, unfortunate | 20:31 |
| vishy | well, assuming you want the instances to talk to each other | 20:32 |
| clarkb | so I don't think we can actually make this work without a VPN then (even in the "fixed" ip case) | 20:32 |
| vishy | if you don’t care about that | 20:32 |
| vishy | it will be fine | 20:32 |
| fungi | clarkb: having them in the same provider neutron net is insufficient, or just means that bouncing through the provider's nat is out of the question? | 20:32 |
| *** todd_dsm has joined #openstack-infra | 20:32 | |
| clarkb | because we don't have an garuntee from rax that we will alays have nodes on the same l2 network | 20:32 |
| clarkb | fungi: both | 20:32 |
| vishy | clarkb: how much do we care about vms in the multinode devstack being able to talk to each other? | 20:33 |
| *** rmcall has quit IRC | 20:33 | |
| clarkb | vishy: well we need tempest to be able to talk to all of them | 20:33 |
| clarkb | and tempest runs on one node | 20:33 |
| vishy | but it uses floating ips? | 20:33 |
| vishy | or fixed? | 20:33 |
| clarkb | vishy: the floating ip vs fixed is actually not really important anymore | 20:33 |
| clarkb | because we cannot garuntee everything ends up on the same l2 network regardless | 20:33 |
| vishy | it has to be for tempest | 20:33 |
| clarkb | it isn't we can cross routers in both situations | 20:34 |
| vishy | ah i see | 20:34 |
| vishy | so yeah you will need to create a virtual layer 2 network | 20:34 |
| vishy | to do this | 20:34 |
| jogo | vishy: and we should be able to do that via neutron right? | 20:35 |
| clarkb | I think we can garuntee it in hpcloud | 20:35 |
| vishy | https://coreos.com/blog/introducing-rudder/ | 20:35 |
| vishy | :) | 20:35 |
| clarkb | but rax doesn't expose neutron so we can't do it there | 20:35 |
| vishy | chuckle | 20:35 |
| fungi | clarkb: they were working on making it available, right? | 20:35 |
| clarkb | vishy: we cannot run cinder in a container | 20:35 |
| clarkb | fungi: maybe? | 20:35 |
| vishy | clarkb: i wasn’t really referring to the container part, more the overlay network using udp | 20:36 |
| vishy | which is the only thing that works across most clouds | 20:36 |
| clarkb | ya so we can openvpn them together pretty easily | 20:36 |
| vishy | yeah openvp would work | 20:36 |
| fungi | or ipsec with ike+nat-t | 20:36 |
| fungi | or even just straight up gre | 20:37 |
| vishy | no gre no work | 20:37 |
| *** markmcclain1 has joined #openstack-infra | 20:37 | |
| fungi | it's not like we have security/privacy concerns | 20:37 |
| *** zns has joined #openstack-infra | 20:37 | |
| *** markmcclain1 has quit IRC | 20:37 | |
| vishy | its too bad that the udp encap never merged in openvswitch | 20:37 |
| fungi | gre with a point-to-point l2 won't work? | 20:37 |
| vishy | oh actually gre might work in rax + hp | 20:37 |
| vishy | its blocked by default | 20:37 |
| fungi | ahh | 20:38 |
| vishy | but if they are both using neutron | 20:38 |
| *** teran has quit IRC | 20:38 | |
| vishy | you can open it up | 20:38 |
| vishy | nova-network you can't | 20:38 |
| fungi | you mean the ip protocol for gre is likely blocked between networks. got it | 20:38 |
| vishy | it doesn’t have an option for allowing gre, just like amazon | 20:38 |
| fungi | because, you know, nobody really needs ip protocols other than icmp, udp and tcp right? | 20:38 |
| clarkb | ya so "fixed" ips would work in hpcloud today as we set one network per virtual provider thing in nodepool | 20:39 |
| vishy | yes amazon :) | 20:39 |
| fungi | i mean, the other 50+ are just wasting space | 20:39 |
| *** markmcclain1 has joined #openstack-infra | 20:39 | |
| clarkb | jroll: JayF: do you have any idea how rackspace is doing neutron and if all our nodes in one tenant end up in the same private network? | 20:39 |
| vishy | speaking of which I’ve been meaning to add other protos to security groups | 20:39 |
| *** dkranz has joined #openstack-infra | 20:39 | |
| vishy | but getting stuff into nova-network is sooo painful these days | 20:39 |
| jogo | vishy: you should just use neutron ;) | 20:40 |
| clarkb | vishy: so one more qusetion. If/when we sort out this layer 2 stuff we should probably go to testing multi_node by default beause we don't cover that today? | 20:40 |
| jroll | clarkb: on the virt side, I have no idea | 20:40 |
| clarkb | vishy: and if we do that should we keep using flatdhcpmanaer? | 20:40 |
| jroll | clarkb: I'm not entirely sure what you're asking, are you asking if by default a tenant has its own private network? | 20:40 |
| fungi | anyway, if it's just a concern over ip proto 49 getting blocked, we could forward it over a tcp socket | 20:40 |
| fungi | er, ip proto 47 | 20:40 |
| clarkb | jroll: if I nova boot 2 nodes using the same tenant/project credentails. Will those two nodes' private IP always be on the same private l2 network? | 20:41 |
| JayF | clarkb: define: private IP | 20:41 |
| clarkb | jroll: so that traffic between them never crosses a router (at least logically) | 20:41 |
| JayF | clarkb: a cloud network? servicenet? | 20:41 |
| jroll | clarkb: no | 20:41 |
| clarkb | JayF: the non globally routable address that every node gets | 20:41 |
| * fungi also hates the term "private ip address" | 20:41 | |
| JayF | clarkb: that's very likely to be routed, and is not a contiguous layer 2 domain | 20:41 |
| fungi | rfc-1918 in this case | 20:41 |
| JayF | clarkb: I believe cloud networks will behave as a contiguous layer 2 domain | 20:41 |
| JayF | clarkb: but not servicenet | 20:42 |
| jroll | ^ | 20:42 |
| clarkb | what is cloud networks? | 20:42 |
| clarkb | and what is servicenet? you are using words that have no meaning :P | 20:42 |
| openstackgerrit | Sahdev Zala proposed a change to openstack-infra/project-config: Rename the heat-translator project repo https://review.openstack.org/131558 | 20:42 |
| jroll | clarkb: cloud networks is private tenant networks | 20:42 |
| fungi | "cloud networks" is rax's productization of neutron? | 20:42 |
| jroll | clarkb: servicenet and publicnet are the two default neutron networks you get with a standard cloud server | 20:42 |
| jroll | yes | 20:42 |
| vishy | so lets clarify the layer 2 requirement | 20:42 |
| vishy | if the nodes can communicate with each other | 20:43 |
| vishy | i think that is good enough | 20:43 |
| clarkb | vishy: it isn't | 20:43 |
| jroll | fungi: that yes was for you | 20:43 |
| clarkb | vishy: I think the failing tests prove at least that much | 20:43 |
| vishy | :) | 20:43 |
| clarkb | vishy: because the inner openstack uses some 172.x.y.z address space | 20:43 |
| fungi | jroll: i always assume all yeses are for me | 20:43 |
| vishy | clarkb: well it wouldn’t work for mult-host mode | 20:44 |
| clarkb | vishy: and packets with those addresses cannot cross routers without service provider help | 20:44 |
| jroll | fungi: :P | 20:44 |
| *** baoli has quit IRC | 20:44 | |
| vishy | actually let me think for a sec | 20:44 |
| clarkb | vishy: if however everything is on the same l2 network those packets will get to their homes because arp | 20:44 |
| clarkb | (and broadcast if necessary) | 20:44 |
| vishy | yeah you would need two different private networks | 20:44 |
| vishy | with routing rules to the other host | 20:44 |
| vishy | so you’re right probably not feasible | 20:44 |
| openstackgerrit | Antoine Musso proposed a change to openstack-infra/zuul: Support stracktracing on Geard embedded server https://review.openstack.org/131559 | 20:44 |
| *** baoli has joined #openstack-infra | 20:44 | |
| clarkb | jroll: JayF: and cloud networks is not exposed yet? | 20:45 |
| fungi | clarkb: actually, i think without a tenant-specific layer 2 the providers have a definite incentive not to allow instances to add arbitrary unassigned ip addresses | 20:45 |
| fungi | since they could easily conflict between tenants | 20:45 |
| clarkb | fungi: I mean they probably should but it works | 20:45 |
| jroll | clarkb: afaik it is | 20:45 |
| * jroll looks for docs | 20:46 | |
| JayF | clarkb: it is; I don't know how to do it via the API as I've only done it via our control panels before | 20:46 |
| *** e0ne has joined #openstack-infra | 20:46 | |
| clarkb | I think we have been "lucky" that this has worked in rax so far | 20:47 |
| fungi | clarkb: also, if we fire up two multinode jobs in the same layer 2 broadcast domain, will their test instances conflict on arbitrary ip address assignment? i suspect we really need a virtual layer 2 network for each multinode if layer 2 is needed between subnodes at all | 20:47 |
| clarkb | fungi: good point, yes I think they will | 20:47 |
| clarkb | fungi: that would be fun to arp poison each other | 20:47 |
| clarkb | so ya we should just do openvpn | 20:47 |
| fungi | yep | 20:47 |
| clarkb | (or similar) | 20:47 |
| *** david-lyle has quit IRC | 20:48 | |
| * fungi takes his dusty old network engineer hat back off as quickly as possible | 20:48 | |
| ianw | clarkb / jeblair: can we do a nodepool restart soonish, so i can make sure the hpcloud centos7 images are building before it seems most "root" people disappear for a bit | 20:48 |
| clarkb | sdague: jeblair ^ you will probably want to review scrollback when the TC meeting is over | 20:48 |
| clarkb | ianw: I can do it now | 20:49 |
| *** IaKing has joined #openstack-infra | 20:49 | |
| ianw | clarkb: thanks, i'll check in on the logs in a bit and make sure things are going | 20:49 |
| ianw | clarkb: glusterfs people are keen to use centos -- it can work on ubuntu but package support is better and more up to date on centos i believe | 20:50 |
| *** radez is now known as radez_g0n3 | 20:51 | |
| clarkb | restarted | 20:52 |
| *** teran has joined #openstack-infra | 20:52 | |
| *** hashar has joined #openstack-infra | 20:54 | |
| *** carl_baldwin has joined #openstack-infra | 20:54 | |
| *** zns has quit IRC | 20:55 | |
| *** baoli has quit IRC | 20:55 | |
| *** baoli has joined #openstack-infra | 20:56 | |
| jogo | can we use this as pressure to make rax give us neutron | 20:57 |
| *** koolhead17 has quit IRC | 20:57 | |
| clarkb | jogo: it won't help us | 20:57 |
| clarkb | so no not likely | 20:57 |
| jogo | clarkb: why wouldn't neutron help us? | 20:57 |
| * jogo is not a networking person | 20:58 | |
| clarkb | jogo: I could be wrong but my understanding is that the 172.x.y.z addresses for the inner cloud work on a layer 2 network because of arp | 20:59 |
| *** teran has quit IRC | 20:59 | |
| clarkb | jogo: before you send packet to 172.something you say who has 172.something. When there are 50 inner clouds all on the same network there can be many 172.somethings | 20:59 |
| clarkb | they will effectively arp poison each other | 20:59 |
| jogo | clarkb: if each 2 node devstack setup each had its own private overlay network though | 21:00 |
| clarkb | then tests will likely fail when they shouldnt or pass when they should fail because the wrong thing answered | 21:00 |
| *** spzala has joined #openstack-infra | 21:00 | |
| clarkb | jogo: ya if they had that no problem. But neutron doesn't help us with that | 21:00 |
| jogo | clarkb: it doesn't? I thought that was the point of it | 21:00 |
| clarkb | jogo: except it doesn't work all that well... | 21:00 |
| fungi | jogo: right, private overlay network would be something specifically negotiated between the subnodes in a given multi-node group | 21:01 |
| clarkb | jogo: there are so many pieces you have to manage in order to "give me a network" | 21:01 |
| clarkb | we could say create a new router, a new network, a new set of ports and floating IPs for every pair of servers we spin up | 21:01 |
| clarkb | or we just start openvpn on the pair | 21:01 |
| clarkb | my feeling is openvpn is going to be simpler and easier to debug | 21:01 |
| fungi | oh, i see. right it could in theory be possible for nodepool to provision multi-node-specific ephemeral networks which then get deleted again when teh node group is deleted | 21:01 |
| jogo | clarkb: ahh, that is a good point. but this sounds like a failing of neutron | 21:01 |
| clarkb | jogo: yes it is | 21:02 |
| fungi | by asking neutron at the provider to create one, then asking it to attach to the nodes in the group | 21:02 |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard: Updated & Expanded Subscription API. https://review.openstack.org/131564 | 21:02 |
| jogo | marun salv-orlando markmcclain1 mestery: ^ we are making fun of neutron | 21:02 |
| jogo | marun salv-orlando markmcclain1 mestery: and using openvpn instead | 21:02 |
| marun | jogo: that's an invitation to join in the fun? | 21:02 |
| clarkb | jogo: not to mention quota | 21:02 |
| *** hashar has quit IRC | 21:02 | |
| marun | :) | 21:02 |
| clarkb | I think we are allowed 10 netowrks and routes | 21:02 |
| jogo | marun: yes it is, this sounds like a massive shortcomming of neutron to me | 21:02 |
| clarkb | we can ask for more but I do not know what is involved on that backend and if asking for 600 networks is likely to come back with a yes response | 21:03 |
| * mestery is quadruple tasking | 21:03 | |
| jogo | clarkb: I think long term we should use neutron for this. Short term using openvpn sounds like a really good idea | 21:03 |
| * markmcclain1 is slooooooooooow pipe | 21:03 | |
| *** david-lyle has joined #openstack-infra | 21:03 | |
| jogo | clarkb: for the long term its a good dog fooding thing | 21:04 |
| fungi | especially if they're not using some sort of vlan aggregation protocol... they might be physically limited to direct 802.1q mapping or something insane | 21:04 |
| *** sdake_ has joined #openstack-infra | 21:04 | |
| clarkb | jogo: ya. mostly I want to see neutron abstract enough of these common operations that I don't hve to track quota and uuids for every single network, port, floating ip and router and subnet | 21:04 |
| clarkb | just give me a netowrk | 21:04 |
| mestery | clarkb: neutron net-create | 21:04 |
| mestery | clarkb: There you go ;) | 21:04 |
| clarkb | mestery: does that do the right thing with routers and ports and external nets and so on? | 21:05 |
| apmelton | Has anyone here ever seen devstack-gate hung trying to install some dependencies through apt? | 21:05 |
| apmelton | logs: https://gist.github.com/ramielrowe/96e8ff07e896783a5a37 | 21:05 |
| fungi | or they might be bridging systems through hardware with other limitations on vlan count, such that it's cost-prohibitive for them to significantly up our network quota or deal with the create/delete churn | 21:05 |
| mestery | clarkb: Your definition of right thing may be different than other peoples. Define yours. :) | 21:05 |
| clarkb | seems like last time I did any of this stuff it all had to be in a very specific order with all of the object created in the proper sequence | 21:05 |
| apmelton | seems to be hung after "Processing triggers for man-db ..." | 21:05 |
| salv-orlando | I think the bottom line of what clarkb and jogo say is that the API is mostly pointless crap | 21:05 |
| clarkb | mestery: neutron should have a single command along the lines of `neutron give me a network called foobar` | 21:05 |
| mestery | salv-orlando: Summarized brilliantly | 21:05 |
| *** marun has quit IRC | 21:06 | |
| clarkb | mestery: that would return a uuid for a network that is attached to a router with a subnet | 21:06 |
| jogo | salv-orlando: yup | 21:06 |
| *** marun has joined #openstack-infra | 21:06 | |
| clarkb | mestery: and that router would be attached to the enxternal network | 21:06 |
| jogo | salv-orlando: and that this would be a great use case for neutron | 21:06 |
| clarkb | and I could just use this bigger network abstraction | 21:06 |
| markmcclain1 | clarkb: the correct api call is mordred-wants-a-network-now | 21:06 |
| clarkb | because in reality I don't care about routers and ports and external networks | 21:06 |
| mestery | lol | 21:06 |
| clarkb | (and if I really did I would use the lower level api to muck with that stuff) | 21:06 |
| mestery | clarkb: Porcelain APIs. | 21:07 |
| mestery | :) | 21:07 |
| *** zns has joined #openstack-infra | 21:07 | |
| clarkb | now where this is really problematic isn't the creation portion | 21:07 |
| clarkb | its the accounting issues that arrise | 21:07 |
| clarkb | we already leak floating IPs everywhere | 21:07 |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard: Added Vagrant VM for development purposes. https://review.openstack.org/131306 | 21:07 |
| *** dimtruck is now known as zz_dimtruck | 21:07 | |
| salv-orlando | I am already aware that the infra team as a user of neutron has realized that at the end of the day all the features that neutron provides are a hindrance for them | 21:07 |
| clarkb | because there is no atomic just kill the damn thing for me | 21:07 |
| fungi | (and ports apparently?) | 21:07 |
| salv-orlando | and many other teams like them as well | 21:07 |
| clarkb | instead you have to kill the node then the floating IP | 21:08 |
| tsufiev | hi there! I've noticed a new Gerrit code review message 'The change requires a local merge to resolve...' recently in my commits (I've never met it before). Is it a new way to say 'There is merging conflict'? | 21:08 |
| clarkb | but if you kill the node you lose your pointer to the floating IP | 21:08 |
| salv-orlando | possibly all users except the ones which are forced into buying stuff from hw vendors? | 21:08 |
| clarkb | so if the floating IP delete fails the first time around we have a leak | 21:08 |
| salv-orlando | clarkb: arosen did patch a while ago for that | 21:08 |
| salv-orlando | both in nodepool and nova | 21:08 |
| clarkb | and I would be very worried about similar issues when it comes to routers, ports, subnets, and networks as in the api today | 21:08 |
| clarkb | salv-orlando: ya we just have to wait for it downstream | 21:09 |
| tsufiev | the thing that confused me is that after I did 'Rebase change' in gerrit, Jenkins said +1 | 21:09 |
| fungi | tsufiev: there are two different subsystems which can complain about merge conflicts. that one is gerrit itself realizing that its internal jgit will be unable to merge a commit, even though zuul's cgit backend had no problem with it | 21:09 |
| openstackgerrit | Andreas Jaeger proposed a change to openstack-infra/project-config: Remove obsolete rtfd-id entry https://review.openstack.org/131569 | 21:09 |
| fungi | tsufiev: there are unfortunately minor differences in how merge conflict resolution is implemented between jgit and cgit from what we've seen | 21:10 |
| arosen | clarkb: yup we backported this issue here to nova: https://review.openstack.org/#/q/I53b0c9d949404288e8687b304361a74b53d69ef3,n,z | 21:10 |
| arosen | salv-orlando: | 21:10 |
| tsufiev | fungi, does that mean that it's pointless to press 'Rebase change' in web-interface in that case? | 21:10 |
| clarkb | arosen: we are still seeing the leaks so I don't think hpcloud has deployed those changes | 21:10 |
| fungi | tsufiev: jgit seems to have particular trouble with n-way merges beyond ~4, so if you had a patch series of a lot of changes gerrit likely threw that on a change partway into the stack | 21:11 |
| clarkb | arosen: or do we need to modify how we do deletions client side? | 21:11 |
| *** yjiang5 has quit IRC | 21:11 | |
| fungi | tsufiev: you can try the "rebase" button, but in my prior experience i've found i needed to locally rebase the commit and push it up as a new patchset | 21:11 |
| jeblair | clarkb, fungi: istr that we thought openvpn would eventually be necessary to do some neutron testing anyway; openvpn sounds good to me. i also like being protected from the vagaries of our cloud providers' environments. | 21:12 |
| arosen | clarkb: i assume you have this patch applied to youre node pool: https://review.openstack.org/#/c/113659/ | 21:12 |
| clarkb | yup | 21:12 |
| clarkb | arosen: so likely we just need to wait for downstream cloud providers to deploy those changes | 21:12 |
| fungi | jeblair: agreed. i do worry that trying to leverage the provider's neutron to create and delete tons of tiny networks on the fly may not work out so well | 21:12 |
| arosen | clarkb: that patch should fix the leaking issue if not we should need to investigate more what else could be causing the leak | 21:12 |
| clarkb | arosen: well we knew it wouldn't fix all of the leaks | 21:13 |
| arosen | maybe jogo or armax could help us find out if hp has though changes deployed. | 21:13 |
| clarkb | it would just shorten the locations of leaks | 21:13 |
| *** sdake_ has quit IRC | 21:13 | |
| clarkb | and were dependent on the clouds themselves deploying the server side code to make it more atomic and fix it properly | 21:13 |
| tsufiev | fungi, thanks, will use it as a rule of thumb. Didn't understand the difference between jgit and cgit though (which acts when) :(. Is there some ML thread I'm missing? | 21:13 |
| arosen | yup | 21:13 |
| armax | arosen: I am afraid I wouldn’t know how to help | 21:13 |
| arosen | meh :P | 21:13 |
| armax | perhaps with jogo you’re luckier | 21:14 |
| jogo | armax: I am not | 21:14 |
| armax | arosen: sorry :O | 21:14 |
| armax | arosen: oh well | 21:14 |
| jogo | armax: I don't even know who is in charge of neutron in HP public cloud | 21:14 |
| jogo | mestery: works for us now right? | 21:14 |
| armax | maybe carl_baldwin could be a good source | 21:14 |
| arosen | carl_baldwin: maybe | 21:14 |
| mestery | jogo: Indeed | 21:14 |
| armax | arosen: I am a small fish in a big pond | 21:15 |
| * mestery is a freshly caught fish | 21:15 | |
| armax | it takes me ages to swim from one side to the other and back | 21:15 |
| * carl_baldwin just joining now ... | 21:15 | |
| clarkb | jeblair: ya agreed, if we openvpn we shouldn't ever need to worry about it again | 21:15 |
| fungi | tsufiev: zuul is our testing orchestration layer, and also has some pseudo-jobs which will opportunistically report on mergeability of proposed changes. the version of gerrit we're running right now only figures out whether a change can merge when asked to permanently merge it to the target branch | 21:15 |
| jogo | arosen: wait this is a nova patcH? | 21:15 |
| arosen | jogo: yes. | 21:15 |
| jogo | arosen: I may be able to help | 21:16 |
| fungi | tsufiev: zuul uses the usual cgit implementation like most people use from the command line, while gerrit is a java-based service running in a jvm and uses a java-based git implementation called jgit | 21:16 |
| * jogo sends an email | 21:16 | |
| jogo | I emailed phil day and paul murray but both are in the UK | 21:18 |
| tsufiev | fungi, and java implementation is more fussy that the cgit? Very Java-like :) | 21:18 |
| fungi | tsufiev: basically yes | 21:19 |
| nelsnelson | Greetings, channel. So, when trying to get a devstack-gate set up and following these instructions https://github.com/openstack-infra/devstack-gate, I can get as far as executing safe-devstack-vm-gate-wrap.sh, but then that process seems to hang. The last thing I see in the new/devstacklog.txt is 'Processing triggers for man-db ...' | 21:19 |
| *** yjiang5 has joined #openstack-infra | 21:19 | |
| fungi | nelsnelson: hi again... apmelton also just commented he's running into the same issue | 21:20 |
| nelsnelson | Oh okay, thank you. | 21:20 |
| tsufiev | fungi, thanks again for the explanation! | 21:20 |
| apmelton | fungi: we're working together on that issue | 21:20 |
| *** harlowja is now known as harlowja_away | 21:21 | |
| apmelton | fungi: the apt process seems to be stuck in Stopped state | 21:21 |
| nelsnelson | fungi, I think you might have mentioned trying to pre-install all the dependencies prior to running safe-devstack-vm-gate-wrap.sh. Are there instructions that include a list of those dependencies, or is there a sub script somewhere that handles those installations? | 21:21 |
| fungi | apmelton: nelsnelson: fwiw, the package installations you're seeing are likely coming from devstack itself, since devstack-gate doesn't apt-get install any packages aside from git and python-yaml | 21:21 |
| *** unicell has quit IRC | 21:22 | |
| nelsnelson | Okay, that makes sense. | 21:22 |
| apmelton | fungi: I've noticed in the past, I can get past thing, by removing the timeout wrapper | 21:22 |
| *** hdd has quit IRC | 21:22 | |
| fungi | apmelton: nelsnelson: if you do think it's just timing out on something time-consuming, you can adjust the timeout without removing it entirely | 21:22 |
| apmelton | fungi: it's not actually timing out | 21:23 |
| nelsnelson | I personally don't think it's actually doing anything. | 21:23 |
| apmelton | it hangs way before the timeout would actually hit | 21:23 |
| fungi | right, so whatever devstack should do next after installing packages seems to not be proceeding, and not resulting in any output in the devstack log | 21:23 |
| nelsnelson | correct | 21:24 |
| *** zz_dimtruck is now known as dimtruck | 21:24 | |
| clarkb | strace it? | 21:24 |
| apmelton | fungi: when the gate is run, are most of the dependencies pre-installed on the image? | 21:24 |
| apmelton | clarkb: root@devstack-tempest-1414396776:/opt/stack/new# strace -p 2788 | 21:25 |
| apmelton | Process 2788 attached - interrupt to quit | 21:25 |
| apmelton | --- SIGTTOU (Stopped (tty output)) @ 0 (0) --- | 21:25 |
| apmelton | --- SIGINT (Interrupt) @ 0 (0) --- | 21:25 |
| apmelton | Process 2788 detached | 21:25 |
| apmelton | I'm not very familiar with strace, but it basically killed the process | 21:25 |
| apmelton | put it in zombie state | 21:25 |
| *** AJaeger has quit IRC | 21:25 | |
| fungi | apmelton: yes, we use something called nodepool to precreate images which we later use to boot our test systems. it does a download-only install of the debs or rpms (depending on platform) mentioned in devstack | 21:26 |
| *** armax has quit IRC | 21:27 | |
| fungi | that way devstack's install step only has to retrieve updated package lists and maybe a handful of new packages but should find most of the ones it wants already in the cache | 21:27 |
| *** rushiagr is now known as rushiagr_away | 21:27 | |
| *** markmcclain1 has quit IRC | 21:27 | |
| apmelton | fungi: but ultimately, it's devstack is still installing those packages, right? | 21:27 |
| fungi | apmelton: but given that package installation seems to be completing well in advance of hitting the timeout, i doubt that's the cause of your current issue | 21:28 |
| fungi | apmelton: and yes, devstack still installs them because we need to be able to reliably self-test proposed changes to devstack's package lists | 21:28 |
| apmelton | nelsnelson: how did you kick off this script? just running the wrapper script in an ssh session? | 21:29 |
| nelsnelson | Yeah. Regular shell session. Just replicated all the commands indicated by https://github.com/openstack-infra/devstack-gate. | 21:30 |
| *** bcourt has joined #openstack-infra | 21:30 | |
| apmelton | fungi: how does the devstack gate script get kicks off normally? I'm wondering if maybe executing it from ssh could be causing issues | 21:32 |
| apmelton | get kicked* | 21:32 |
| fungi | apmelton: via a jenkins slave agent. we define it as a shell builder in a jenkins job | 21:32 |
| fungi | apmelton: it's possible if you're doing noninteractive ssh, not allocating a tty, that it could have some bearing on the result | 21:33 |
| *** armax has joined #openstack-infra | 21:33 | |
| fungi | nelsnelson: apmelton: maybe compare your devstack log to a working one from our ci like http://logs.openstack.org/93/131193/4/check/check-tempest-dsvm-full/1da1ae9/logs/devstacklog.txt.gz and see what should be coming next after where your output ends | 21:33 |
| *** armax has quit IRC | 21:34 | |
| apmelton | fungi: dunno if this is interesting or not, but it would seem our run is installing libsigsegv2 while that run didn't | 21:35 |
| *** dustins has quit IRC | 21:36 | |
| apmelton | though, there seem to be a couple other packages like that as well | 21:36 |
| fungi | apmelton: may have to do with what envvars you're passing to devstack-gate. it has some switches to turn different things on and off | 21:36 |
| *** e0ne has quit IRC | 21:36 | |
| *** emagana_ has quit IRC | 21:37 | |
| *** emagana has joined #openstack-infra | 21:38 | |
| nelsnelson | fungi, I'm only setting the environment variables exactly as indicated in the https://github.com/openstack-infra/devstack-gate readme. | 21:38 |
| *** zns has quit IRC | 21:39 | |
| fungi | nelsnelson: so you're using DEVSTACK_GATE_TEMPEST=1 and DEVSTACK_GATE_TEMPEST_FULL=1 in this case? should be mostly similar to the job run i linked (tempest-dsvm-full) | 21:40 |
| apmelton | fungi: what image are y'all using when running against rackspace? | 21:40 |
| nelsnelson | Correct. | 21:40 |
| *** andreykurilin_ has quit IRC | 21:41 | |
| *** emagana_ has joined #openstack-infra | 21:41 | |
| apmelton | fungi: was it just Ubuntu 14.04 PVHVM? | 21:41 |
| clarkb | yes | 21:41 |
| fungi | "Ubuntu 14.04 LTS (Trusty Tahr) (PVHVM)" | 21:41 |
| fungi | for that particular test run | 21:41 |
| apmelton | nelsnelson: which image did you use? | 21:41 |
| *** emagana has quit IRC | 21:41 | |
| *** packet has quit IRC | 21:42 | |
| fungi | and an 8gb performance flavor | 21:42 |
| annegentle | what is the name of the open source project that ask.openstack.org is built on? | 21:42 |
| annegentle | upon? | 21:42 |
| clarkb | annegentle: askbot | 21:42 |
| nelsnelson | Well, I thought that I was using the Ubuntu 14.04 LTS (Trusty Tahr) (PVHVM), but I now realize that I am mistaken. | 21:42 |
| nelsnelson | :-/ | 21:43 |
| *** dkranz has quit IRC | 21:43 | |
| apmelton | nelsnelson: lets try with 14.04 PVHVM and see what happens | 21:43 |
| annegentle | clarkb: thank you! | 21:43 |
| apmelton | thanks for the help clarkb and fungi, hopefully using that image will fix this for us | 21:45 |
| *** harlowja_away is now known as harlowja | 21:46 | |
| *** dims__ has quit IRC | 21:46 | |
| nelsnelson | Thank you, apmelton, clarkb, and fungi... I'll let you know if this fixes my problem. | 21:46 |
| *** marun has quit IRC | 21:49 | |
| *** mattfarina has quit IRC | 21:50 | |
| *** teran has joined #openstack-infra | 21:51 | |
| fungi | okay, gonna go watch nasa retry to launch this rocket, then grab some dinner and pack. mostly afk for the evening | 21:52 |
| *** david-lyle has quit IRC | 21:52 | |
| *** andreykurilin_ has joined #openstack-infra | 21:53 | |
| *** david-lyle has joined #openstack-infra | 21:53 | |
| clarkb | wait nasa launches rockets? | 21:57 |
| *** pc_m_ has quit IRC | 21:57 | |
| *** arosen has left #openstack-infra | 21:58 | |
| fungi | when there's not a sailboat in the way, they do | 21:58 |
| clarkb | I thought they paid other people to do all that now | 21:58 |
| *** dimtruck is now known as zz_dimtruck | 22:01 | |
| *** doug-fish has left #openstack-infra | 22:01 | |
| *** dangers is now known as dangers_away | 22:01 | |
| *** JayJ has quit IRC | 22:02 | |
| *** JayJ has joined #openstack-infra | 22:03 | |
| *** thedodd has quit IRC | 22:09 | |
| *** bknudson has quit IRC | 22:11 | |
| *** bcourt has quit IRC | 22:11 | |
| *** ZZelle_ has quit IRC | 22:13 | |
| greghaynes | they just dont do manned low earth orbit | 22:14 |
| greghaynes | or sailboats | 22:14 |
| greghaynes | or any combination thereof | 22:14 |
| *** todd_dsm has quit IRC | 22:14 | |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard: Test Version Controller https://review.openstack.org/131584 | 22:14 |
| *** e0ne has joined #openstack-infra | 22:15 | |
| *** armax has joined #openstack-infra | 22:16 | |
| clarkb | rereading this stuff it looks like afazekas_beer was already setting up gre bridges between the nodes | 22:18 |
| *** dizquierdo has joined #openstack-infra | 22:18 | |
| clarkb | or trying to? I don't see the interfaces | 22:18 |
| *** amitgandhinz has quit IRC | 22:18 | |
| *** ryanpetrello has quit IRC | 22:19 | |
| *** thedodd has joined #openstack-infra | 22:19 | |
| *** julim has joined #openstack-infra | 22:20 | |
| clarkb | I am going to guess that maybe my IP manipulation didn't do the correct thing there | 22:20 |
| *** Ryan_Lane1 has joined #openstack-infra | 22:21 | |
| *** Ryan_Lane has quit IRC | 22:21 | |
| *** lttrl has quit IRC | 22:22 | |
| nibalizer | is this just weird dns failures or is there something wrong with my test? http://logs.openstack.org/86/126086/3/check/gate-puppet-storyboard-puppet-beaker-rspec/207fc61/console.html | 22:22 |
| openstackgerrit | A change was merged to openstack/requirements: Add keystoneclient-kerberos repository to projects https://review.openstack.org/131065 | 22:25 |
| *** changbl has quit IRC | 22:26 | |
| *** rlandy has quit IRC | 22:26 | |
| clarkb | ok I think I may know why this broke at least for nova network | 22:26 |
| clarkb | not sure about neutron yet | 22:26 |
| *** emagana_ has quit IRC | 22:26 | |
| *** sarob_ has quit IRC | 22:27 | |
| *** tonytan4ever has quit IRC | 22:27 | |
| clarkb | `remote_command $node sudo -i ip link add ${if_name} type gretap local $node remote $host_ip key $key` the remote host_ip is 15.something | 22:28 |
| clarkb | but none of the traffic will original from that address when it talks because hpcloud knows not about it | 22:28 |
| clarkb | jeblair: I think we should give consideration to using the "fixed" ips | 22:29 |
| clarkb | jeblair: since we are already testing everything with them today and it keeps the crazy routing and bridging sane | 22:29 |
| ianw | clarkb: so i'm not seeing hpcloud centos7 builds in http://nodepool.openstack.org/image.log ... | 22:30 |
| jeblair | clarkb: what do you think we should do on rax? | 22:30 |
| clarkb | jeblair: same thing, rax gives us a "fixed" ip ish thing too | 22:31 |
| jeblair | clarkb: well, they give us public and private ips | 22:31 |
| jeblair | clarkb: so which one do you think we should use there? | 22:31 |
| clarkb | the private ip | 22:31 |
| jeblair | clarkb: why? | 22:31 |
| clarkb | because that is more consistent with what hp gives us | 22:31 |
| jeblair | in what way? | 22:32 |
| *** dmsimard is now known as dmsimard_away | 22:32 | |
| mordred | it's an ip that the node knows about | 22:32 |
| clarkb | they are not globally routable addresses | 22:32 |
| clarkb | and they are btoh ips the node knows about | 22:32 |
| jeblair | mordred: rax knows about both ips | 22:32 |
| clarkb | and we won't have to special case anything to get both behaviors | 22:32 |
| *** david-lyle_ has joined #openstack-infra | 22:33 | |
| jeblair | clarkb: what 'both behaviors'? | 22:33 |
| clarkb | node knows about the IP address and IP address is not globally routable | 22:34 |
| mordred | amusingly enough, I've got a function already which will return the fixed IP on hp and the private IP on rax | 22:34 |
| mordred | https://github.com/emonty/shade/blob/master/shade/meta.py#L21-L32 - called like: https://github.com/emonty/shade/blob/master/shade/meta.py#L81-L82 | 22:35 |
| mordred | because "public" and "private" and "fixed" and "floating" are not consistent terms across clouds | 22:35 |
| *** thedodd has quit IRC | 22:35 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/nodepool: Add current node IP to /etc/nodepool https://review.openstack.org/131532 | 22:35 |
| *** gondoi is now known as zz_gondoi | 22:36 | |
| jeblair | clarkb: well, not globally routable sounds like an arbitrary decision at this point | 22:36 |
| *** david-lyle has quit IRC | 22:36 | |
| jeblair | mordred: i think afazekas proposed a change to nodepool for that. perhaps you could review it? | 22:36 |
| mordred | jeblair: yes | 22:36 |
| jeblair | mordred: i only -1d it on its necessity, which clarkb is now making a case for | 22:36 |
| clarkb | jeblair: but it isn't necessary | 22:37 |
| jeblair | clarkb: oh, i thought you wanted to use fixed ips | 22:37 |
| *** andreaf has joined #openstack-infra | 22:37 | |
| clarkb | yes but we don't need nodepool for that | 22:37 |
| jeblair | or, let's say "node-aware rfc1918 addresses" | 22:37 |
| jeblair | clarkb: why not? | 22:37 |
| clarkb | afazekas_beer's current change figures it out just fine best I can tell | 22:37 |
| clarkb | each node already knows those addresses so it doesn't need nodepool to tell it | 22:38 |
| mordred | on the host? why? we're already putting info into the file in nodepool - why perform ip address figuring out logic in two places | 22:38 |
| jeblair | clarkb: i can not tell that it does that, nor can i tell that it will do that in future cloud configurations. i'd much rather nodepool be the source of this information. i think he would too. | 22:38 |
| mordred | ++ | 22:38 |
| jeblair | clarkb: i imagine that's why he wrote the nodepool change and said "this gets rid of a lot of complexity in devstack-gate" | 22:38 |
| mordred | I would also prefer nodepool to be teh source of the information - its view of the world is richer | 22:38 |
| clarkb | jeblair: sure I am not saying don't put it in nodepool | 22:38 |
| clarkb | just saying it isn't necessary | 22:38 |
| clarkb | this is the nice thing about nodes knowing their own IPs | 22:38 |
| mordred | jeblair: I do not see a nodepool change from afazekas_beer | 22:38 |
| mordred | oh - it's abandoned | 22:39 |
| mordred | one sec | 22:39 |
| *** zns has joined #openstack-infra | 22:39 | |
| *** gyee has joined #openstack-infra | 22:39 | |
| jeblair | clarkb: also, i want to make devstack-gate smaller | 22:39 |
| mordred | jeblair: his change is incomplete | 22:40 |
| jeblair | mordred: can you link it? | 22:40 |
| mordred | https://review.openstack.org/#/c/114840 | 22:40 |
| mordred | it only handles fixed ips - doesn't handle a neutron-based cloud which has a 'private' network | 22:41 |
| jeblair | mordred: missing rax? yah | 22:41 |
| *** david-lyle has joined #openstack-infra | 22:41 | |
| mordred | jeblair: lemme fix - one sec | 22:41 |
| jeblair | i love that we have servers that don't know their own ip, and i love that everything we do is still wrapped in an "if hpcloud/elif rax" clause :( | 22:42 |
| mordred | well ... it's on my list to port nodepool to shade so that none of our code has any cloud-differences-aware code ... but I haven't even gotten around to writting up the spec for that :) | 22:42 |
| *** zns has quit IRC | 22:44 | |
| *** nelsnelson has quit IRC | 22:44 | |
| *** jgrimm is now known as zz_jgrimm | 22:44 | |
| *** zns has joined #openstack-infra | 22:44 | |
| *** ryanpetrello has joined #openstack-infra | 22:44 | |
| *** david-lyle_ has quit IRC | 22:45 | |
| *** dane_leblanc has quit IRC | 22:45 | |
| *** dims__ has joined #openstack-infra | 22:46 | |
| jeblair | i restored that change and left a comment with my current understanding | 22:47 |
| jeblair | mordred: it may want to be rebased onto https://review.openstack.org/131532 and an equivalent file added | 22:48 |
| jeblair | also, we should avoid the use of the word fixed | 22:48 |
| mordred | yah. since it's "fixed or private" | 22:48 |
| *** mfink has quit IRC | 22:48 | |
| *** zz_dimtruck is now known as dimtruck | 22:49 | |
| mordred | jeblair: how about just private? | 22:49 |
| jeblair | mordred: that works for me since our fixed addresses are also private | 22:49 |
| *** dims__ has quit IRC | 22:52 | |
| *** amcrn has quit IRC | 22:52 | |
| *** tgohad has quit IRC | 22:52 | |
| *** ryanpetrello has quit IRC | 22:53 | |
| *** dims__ has joined #openstack-infra | 22:55 | |
| *** dims__ has quit IRC | 22:55 | |
| *** emagana has joined #openstack-infra | 22:55 | |
| *** dims__ has joined #openstack-infra | 22:56 | |
| jeblair | clarkb, fungi: the antares rocket exploded just after liftoff; http://www.nasa.gov/mission_pages/station/structure/launch/orbital.html#.VFAe_4WVtph | 22:56 |
| jeblair | no one hurt according to nasa | 22:57 |
| *** ihrachyshka has quit IRC | 22:57 | |
| *** dims__ has quit IRC | 22:57 | |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/nodepool: Add private ip to the /etc/nodepool https://review.openstack.org/114840 | 22:57 |
| *** dims__ has joined #openstack-infra | 22:57 | |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/nodepool: Add private ip to the /etc/nodepool https://review.openstack.org/114840 | 22:59 |
| *** msabramo1 has quit IRC | 22:59 | |
| mordred | missed updating the docs in the first patch | 22:59 |
| jeblair | mordred: how about a "node_private" file? | 23:00 |
| jeblair | mordred: private ip address of current node | 23:00 |
| mordred | does it not add one of those? | 23:00 |
| *** timrc is now known as timrc-afk | 23:01 | |
| jeblair | mordred: primary_node_private, and _sub_nodes_private. | 23:01 |
| mordred | AH - gotcha | 23:01 |
| *** baoli has quit IRC | 23:01 | |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/nodepool: Add private ip to /etc/nodepool https://review.openstack.org/114840 | 23:03 |
| *** tomoe has joined #openstack-infra | 23:03 | |
| *** msabramo has joined #openstack-infra | 23:04 | |
| mordred | jeblair: these files just gave me an idea - bear with me, it'll take a few seconds to explain | 23:06 |
| jeblair | mordred: i will beer with you | 23:07 |
| *** timrc-afk is now known as timrc | 23:07 | |
| mordred | much of the info that we're writing out into /etc/nodepool is very similar to the hostvars stuff that we get in ansible inventory, which I have a function that returns as a dict of lists and dicts as you'd expect - which ansible then turns in to json when needed for reconsumtion | 23:08 |
| *** andreaf has quit IRC | 23:08 | |
| mordred | now, we're not using ansible for this- so json/yaml files are not useful | 23:08 |
| mordred | thus all the individual files in /etc/nodepool - shell scripts | 23:08 |
| *** JayJ has quit IRC | 23:08 | |
| mordred | SO - the idea is - what if we wrote a small generalized tool that would take a json/yaml/simple datastructure | 23:08 |
| mordred | and splat it out into files and dirs under a top level output dir | 23:09 |
| *** msabramo has quit IRC | 23:09 | |
| mordred | so that we could plop down a yaml/json file for any things like python that can consume all the data in a structured form | 23:09 |
| mordred | but then have an exactly analogous dir of simple files that shell scripts could use | 23:09 |
| jeblair | mordred: doesn't sound bad, though i don't think we have a current problem that necessitates it | 23:10 |
| mordred | no. we don't | 23:10 |
| mordred | just made me think that the problem of "I have json/yaml and I want to use from shell scripts" does come up from time to time | 23:10 |
| mordred | and it usually gets solved in some one-off crazy manner | 23:10 |
| mordred | so maybe we can invent new goodness | 23:11 |
| jeblair | mordred: perhaps, though i think the rules about how to translate a complex data structure into a flat structure may vary | 23:11 |
| jeblair | so generalizing that may be hard | 23:11 |
| *** andreaf has joined #openstack-infra | 23:11 | |
| mordred | they may - and it may be hard to get it "right" every time | 23:11 |
| mordred | but I think I'll take a stab at an 80% good enough approach and see what that looks like | 23:11 |
| mordred | may suck, in which case I may delete it :) | 23:12 |
| *** teran_ has joined #openstack-infra | 23:12 | |
| jeblair | though i guess things like '../subnodes/0/ip ..subnodes/1/ip' would probably work | 23:12 |
| mordred | yah. just as understandable as the equiv in the yaml :) | 23:13 |
| clarkb | after much investigating an attempts to further debug I am relatively certain that the gre tunnel stuff that afazekas_beer sets up is sufficient for our needs | 23:13 |
| jeblair | mordred: so actually, yeah, you could probably translate lists and dicts directly to directories | 23:13 |
| openstackgerrit | Khai Do proposed a change to openstack-infra/storyboard: Add a rest endpoint to retreive system information https://review.openstack.org/131599 | 23:13 |
| clarkb | it creates a bridge on the primary node that all subnodes are attached to via gretap devices | 23:13 |
| clarkb | then devstack/nova is told to use the resulting pub_if interface for the Flat netwrok | 23:14 |
| clarkb | so the VMs we spin up nested all end up on that bridge | 23:14 |
| clarkb | and I am pretty sure that this doesn't work for tempest in the case of public floating IPs because the node is unaware of the 15.something addresses so linux cannot route our packets appropriately | 23:14 |
| *** teran has quit IRC | 23:15 | |
| clarkb | but when traffic originates on the "fixed" private address it does know how to get those packets from source to destination | 23:15 |
| jeblair | clarkb: cool, so way forward: 1) land nodepool change, 2) combine both afazekas devstack-gate changes (the one that adds stuff along with the one that removes complexity), 3) restart nodepool, 4) test/merge change ? | 23:15 |
| clarkb | that sounds about right | 23:15 |
| clarkb | note I haven't quite investigated this for the neutron case yet | 23:16 |
| jroll | mordred: seems like a useful thing | 23:16 |
| clarkb | but if my memory is not completely terrible neutron basically sets up the same gretap bridge situtation for you | 23:16 |
| *** emagana has quit IRC | 23:16 | |
| *** andreaf has quit IRC | 23:16 | |
| clarkb | and neutron doing that would also explain the brokeness when we set the gretap device hosts as using the floating IPs | 23:17 |
| clarkb | so to recap | 23:17 |
| *** emagana has joined #openstack-infra | 23:17 | |
| *** andreaf has joined #openstack-infra | 23:17 | |
| clarkb | I don't think cinder specifically cares | 23:17 |
| clarkb | I do think that communication to arbitrary VMs does care | 23:17 |
| clarkb | because of the gretap bridge | 23:17 |
| clarkb | mordred: you need to update the nodepool test fakes | 23:19 |
| *** emagana_ has joined #openstack-infra | 23:21 | |
| *** emagana has quit IRC | 23:21 | |
| clarkb | ianw: I am going to start a manual hpcloud centos7 image build | 23:22 |
| *** emagana_ has quit IRC | 23:22 | |
| *** emagana has joined #openstack-infra | 23:23 | |
| ianw | clarkb: ok, great. has it just not got around to it? | 23:23 |
| clarkb | it had a bunch of older ones | 23:23 |
| clarkb | I am deleting those first | 23:23 |
| clarkb | ok one just started | 23:24 |
| clarkb | you should probably start seeing it in the log now? | 23:24 |
| ianw | clarkb: downloading ... thank heavens for gzip :) | 23:25 |
| ianw | clarkb: yay, building! i expect that will go ok | 23:27 |
| *** emagana has quit IRC | 23:27 | |
| *** emagana has joined #openstack-infra | 23:28 | |
| openstackgerrit | Khai Do proposed a change to openstack-infra/storyboard: Add a rest endpoint to retreive system information https://review.openstack.org/131599 | 23:29 |
| *** amitgandhinz has joined #openstack-infra | 23:29 | |
| *** teran has joined #openstack-infra | 23:30 | |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/nodepool: Test label with snapshot and dib images https://review.openstack.org/130878 | 23:31 |
| *** teran_ has quit IRC | 23:31 | |
| clarkb | jeblair: ianw ^ I think that should better cover the case where a label has both snapshot and dib provided images | 23:31 |
| *** emagana has quit IRC | 23:32 | |
| *** amitgandhinz has quit IRC | 23:34 | |
| mordred | jroll, jeblair: done: http://paste.openstack.org/show/126205 | 23:35 |
| clarkb | mordred: did you see comment about nodepool test fakes? | 23:36 |
| mordred | clarkb: yup. on it now | 23:36 |
| jroll | neat | 23:36 |
| * jroll saves for later | 23:36 | |
| *** rushiagr_away is now known as rushiagr | 23:36 | |
| mordred | jroll: after I'm done with test fakes, I'm gong to cookiecutter that and publish to pypi just because why the heck not | 23:36 |
| jroll | cool | 23:36 |
| *** carl_baldwin has quit IRC | 23:37 | |
| *** andreykurilin_ has quit IRC | 23:40 | |
| *** marun has joined #openstack-infra | 23:40 | |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/nodepool: Add private ip to /etc/nodepool https://review.openstack.org/114840 | 23:41 |
| *** amcrn has joined #openstack-infra | 23:42 | |
| *** andreaf has quit IRC | 23:43 | |
| *** sarob has joined #openstack-infra | 23:44 | |
| *** SumitNaiksatam has quit IRC | 23:48 | |
| lifeless | dhellmann: so any ideas of the jython issue? Or is it terra incognita? | 23:50 |
| lifeless | mordred: 'that'? | 23:50 |
| *** david-lyle has quit IRC | 23:50 | |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/nodepool: Test label with snapshot and dib images https://review.openstack.org/130878 | 23:50 |
| clarkb | lifeless: http://paste.openstack.org/show/126205 | 23:51 |
| clarkb | is the that aiui | 23:51 |
| fungi | yes, antares rocket go boom six seconds off the pad. not our space program's finest hour | 23:51 |
| *** packet has joined #openstack-infra | 23:51 | |
| *** mwagner_lap has joined #openstack-infra | 23:51 | |
| jeblair | clarkb, mordred, afazekas: i left some -1 comments on the nodepool change | 23:51 |
| *** dizquierdo has quit IRC | 23:52 | |
| fungi | we were far enough away we couldn't hear the blast either | 23:52 |
| * fungi gets back to packing | 23:52 | |
| jeblair | fungi, clarkb, mordred, jhesketh, SergeyLukjanov: i think it would be good to either merge my revert change or review and merge sdague's changes to fix the test displays on gerrit asap. that's a serious regression that new change folks are living with. | 23:53 |
| mordred | jroll: https://github.com/emonty/explode https://pypi.python.org/pypi/explode | 23:54 |
| jhesketh | jeblair: links? | 23:55 |
| jeblair | jhesketh: 131500 and 131483 | 23:55 |
| jhesketh | ta | 23:55 |
| fungi | they looked okay to me, but i did not find time to test them | 23:56 |
| *** Ryan_Lane1 is now known as Ryan_Lane | 23:57 | |
| *** Ryan_Lane is now known as Guest91880 | 23:57 | |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/devstack-gate: Enable nova multi host on aiopcpu tests https://review.openstack.org/131604 | 23:57 |
| mordred | jroll, jeblair: I believe that is my absolute fasted time from idea in IRC to 1.0 release of a thing :) | 23:58 |
| *** rmcall has joined #openstack-infra | 23:58 | |
| mordred | fastest | 23:58 |
| *** Guest91880 has quit IRC | 23:59 | |
| jhesketh | jeblair: 131500 lgtm if you want to merge it | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
