| *** dboik has joined #openstack-infra | 00:00 | |
| *** mriedem has quit IRC | 00:01 | |
| clarkb | sdague: so far the job hsa been more reliable than nova >_> | 00:01 |
|---|---|---|
| sdague | oh, with the live migration regressions? | 00:02 |
| *** dboik has quit IRC | 00:02 | |
| clarkb | ya | 00:03 |
| *** mahito has joined #openstack-infra | 00:03 | |
| sdague | yeh, well in the real world most people doing that are using ceph | 00:04 |
| sdague | so it's a slightly convoluted setup | 00:04 |
| *** claudiub has quit IRC | 00:04 | |
| jogo | sdague: http://jogo.github.io/gate/aiopcpu | 00:05 |
| jogo | we aren't getting a lot of runs at the moment | 00:05 |
| sdague | I guess the patch flow is still in lull with the release | 00:05 |
| jogo | sdague: yeah | 00:06 |
| sdague | jogo: oh, that's gate only, yeh, what about check queue | 00:06 |
| clarkb | harlowja: you are likely going to need a gerrit admin to force push those commits in | 00:06 |
| clarkb | harlowja: how did we miss these when we sycned the repo in? | 00:06 |
| sdague | clarkb: so, would you object if I changed the name to 'multinode' ? | 00:06 |
| sdague | because, then people would know what it was | 00:06 |
| clarkb | sdague: no, I think that would be fine | 00:07 |
| sdague | based on the d-g code, I think we can do that in project config, and it will all be fine | 00:07 |
| jogo | sdague: that is check queue the title is wrong | 00:08 |
| clarkb | sdague: ya as long as that job sets the mode to aiopcpu for d-g | 00:08 |
| clarkb | sdague: and don't forget to update zuul's node selection function to pick the 2-node env for "multinode" | 00:08 |
| sdague | clarkb: the logic in d-g is actually inverted | 00:08 |
| clarkb | sdague: oh right its if !=aio | 00:08 |
| sdague | clarkb: yeh, git grep is my fried | 00:08 |
| clarkb | so ya should just work | 00:08 |
| Krinkle | jeblair: Thx, I'll re-triage a few things on our end and see to it that we have some actionable items for the next few days. | 00:09 |
| *** julim has joined #openstack-infra | 00:10 | |
| openstackgerrit | Sean Dague proposed openstack-infra/project-config: change name from aiopcpu to multinode https://review.openstack.org/173614 | 00:12 |
| sdague | clarkb: there you go | 00:12 |
| *** otter768 has joined #openstack-infra | 00:12 | |
| *** koolhead17 has quit IRC | 00:13 | |
| jogo | sdague: excellent | 00:14 |
| sdague | ok, have a good night folks | 00:15 |
| jogo | sdague: only 3 multinode runs since the job was fixed | 00:15 |
| *** otter768 has quit IRC | 00:17 | |
| *** baoli has joined #openstack-infra | 00:18 | |
| *** mbacchi1 has quit IRC | 00:22 | |
| *** r-daneel has quit IRC | 00:24 | |
| openstackgerrit | Armando Migliaccio proposed openstack-infra/devstack-gate: Follow up with some comments to clarify OVS commands https://review.openstack.org/173621 | 00:24 |
| armax | sdague, clarkb: ^^^ | 00:24 |
| *** salv-orlando has joined #openstack-infra | 00:25 | |
| jamielennox | is there any facility in infra to rename a repo or do i need to essentially propose it again? | 00:25 |
| openstackgerrit | Spencer Krum proposed openstack-infra/system-config: Use envassert for integration tests https://review.openstack.org/173592 | 00:28 |
| clarkb | armax: I am wondering if we should be more specific. For the add port bit say something like "Create a GRE tunnel between remote_ip and local_ip uniquely identified by key | 00:29 |
| clarkb | jamielennox: its even more fun than that, we need to take a gerrit downtime to update databases | 00:29 |
| armax | clarkb: can do, I was not sure about the level of verbosity needed | 00:29 |
| jamielennox | clarkb: ... i think i'll just repropose :p | 00:29 |
| clarkb | jamielennox: if the project isn't created yet that would be easiest | 00:29 |
| armax | clarkb: the command seems pretty self explainatory to me, or at least not any more convoluted than the equivalent ip/brctl syntax | 00:29 |
| armax | clarkb: let me make the addition, since I need to fix a stupid syntactic error | 00:30 |
| jamielennox | clarkb: it's created but there hasn't been a release or anything so it shouldn't be a history problem or anything, i can set upstream to the existing, and then later we remove the original project? | 00:31 |
| *** wenlock has joined #openstack-infra | 00:31 | |
| jamielennox | clarkb: ? is does that make sense | 00:31 |
| fungi | we batch up project renames, so we'd rather rename your project than have a dead copy of it on the servers in addition to your working one | 00:31 |
| jamielennox | ok, fun - how do i get a project on the rename queue? | 00:32 |
| fungi | we have about a dozen scheduled to rename this friday, and could make it a dozen-plus | 00:32 |
| jamielennox | oh - right, i thought i was doing something really unusual :) | 00:32 |
| fungi | make a change like the ones you see linked at https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting#Upcoming_Project_Renames | 00:32 |
| *** tiswanso has joined #openstack-infra | 00:32 | |
| *** salv-orlando has quit IRC | 00:32 | |
| fungi | i should have said "baker's dozen" | 00:33 |
| fungi | oh well, the moment has passed | 00:33 |
| clarkb | jamielennox: right you cna't just repropose | 00:33 |
| *** nelsnelson has quit IRC | 00:33 | |
| clarkb | jamielennox: we have to gerrit downtime, update db, move files around etc | 00:34 |
| openstackgerrit | Armando Migliaccio proposed openstack-infra/devstack-gate: Follow up with some comments to clarify OVS commands https://review.openstack.org/173621 | 00:34 |
| fungi | for the curious, our process is documented at http://ci.openstack.org/gerrit.html#renaming-a-project | 00:34 |
| *** Krinkle is now known as Krinkle|detached | 00:34 | |
| lifeless | clarkb: / fungi: I'd like to get a node held at some point | 00:35 |
| fungi | lifeless: one running a particular job? one which hasn't run any jobs yet? | 00:35 |
| *** dims_ has joined #openstack-infra | 00:35 | |
| lifeless | https://review.openstack.org/#/c/123597/ | 00:35 |
| lifeless | it passes unit tests locally | 00:35 |
| lifeless | failed in CI | 00:35 |
| lifeless | I need to rebase it but thats hygiene, not functional | 00:36 |
| *** dims has quit IRC | 00:36 | |
| fungi | is the pep8 failure at all related? | 00:36 |
| *** nelsnelson has joined #openstack-infra | 00:36 | |
| jamielennox | clarkb, fungi: will put on the list - thanks | 00:36 |
| lifeless | its irrelevant | 00:36 |
| fungi | k | 00:36 |
| openstackgerrit | lifeless proposed openstack-dev/pbr: Honour MANIFEST.in excludes https://review.openstack.org/123597 | 00:37 |
| fungi | the problem is pbr.tests.test_packaging.TestPackagingInGitRepoWithCommit.test_manifest_exclude_honoured(preversioned) in this case? | 00:37 |
| *** arosen2 has joined #openstack-infra | 00:37 | |
| fungi | so you're looking for a bare-trusty node. i guess even a used one will do as long as it's not still running a job at that point | 00:38 |
| lifeless | fungi: yes | 00:38 |
| lifeless | fungi: which works here :) | 00:38 |
| *** arosen2 has left #openstack-infra | 00:38 | |
| fungi | since the jobs which run on bare-.* nodes at least mostly don't ever do things with sudo | 00:38 |
| lifeless | fungi: I've just pushed up a rebase, which means we should get a node for this itself soonish | 00:38 |
| fungi | okay, i'll hold the one running its python27 job once it pops up | 00:38 |
| lifeless | thanks | 00:39 |
| *** arosen1 has quit IRC | 00:39 | |
| *** nelsnelson has quit IRC | 00:39 | |
| fungi | held | 00:40 |
| fungi | which ssh key do you want authorized on it? | 00:40 |
| *** emagana has quit IRC | 00:40 | |
| *** emagana has joined #openstack-infra | 00:41 | |
| lifeless | that was fast | 00:41 |
| lifeless | sec | 00:41 |
| lifeless | see query | 00:41 |
| fungi | lifeless: ssh jenkins@15.126.248.54 | 00:42 |
| *** jogo has quit IRC | 00:42 | |
| fungi | let's make sure i pieced that pubkey back together right ;) | 00:42 |
| *** tiswanso has quit IRC | 00:44 | |
| lifeless | am in | 00:44 |
| fungi | cool. let me know when you're done and i'll recycle it back into the pool | 00:44 |
| *** tiswanso has joined #openstack-infra | 00:44 | |
| *** yamamoto has joined #openstack-infra | 00:46 | |
| lifeless | ok, so reproduced | 00:46 |
| lifeless | now the hard bit - figuring out wtf | 00:46 |
| fungi | isn't that most of what we all spend our days doing? | 00:46 |
| lifeless | /wrists | 00:47 |
| *** tiswanso_ has joined #openstack-infra | 00:47 | |
| lifeless | lunch then diving in | 00:47 |
| *** emagana has quit IRC | 00:48 | |
| *** tiswanso has quit IRC | 00:50 | |
| *** nelsnelson has joined #openstack-infra | 00:51 | |
| *** kmartin has quit IRC | 00:51 | |
| harlowja | clarkb i think it was mainly just the one that was on git that got synced from was out of date | 00:56 |
| harlowja | *wasn't resynced with bzr before the sync event.... | 00:56 |
| *** nelsnelson has quit IRC | 00:56 | |
| openstackgerrit | Merged openstack-infra/devstack-gate: Add Neutron DVR aiopcpu support https://review.openstack.org/158525 | 00:59 |
| clarkb | harlowja ok, so ya you will likely need to construct the repo state you want then have a gerrit admin force push in | 00:59 |
| harlowja | k | 00:59 |
| harlowja | thx; will see about that | 01:00 |
| *** dimtruck is now known as zz_dimtruck | 01:00 | |
| *** tqtran has joined #openstack-infra | 01:01 | |
| *** mahito has quit IRC | 01:01 | |
| openstackgerrit | Clark Laughlin proposed openstack-infra/puppet-jenkins: Change installation of the JDK to use the architecture of the host https://review.openstack.org/168222 | 01:02 |
| openstackgerrit | David Stanek proposed stackforge/gertty: Removes the need to pass around the app object https://review.openstack.org/172678 | 01:02 |
| openstackgerrit | David Stanek proposed stackforge/gertty: Add the initial CLI implementation https://review.openstack.org/172681 | 01:02 |
| openstackgerrit | David Stanek proposed stackforge/gertty: Use a named argument for server https://review.openstack.org/172680 | 01:02 |
| openstackgerrit | David Stanek proposed stackforge/gertty: wip: add sync CLI command https://review.openstack.org/172682 | 01:02 |
| openstackgerrit | David Stanek proposed stackforge/gertty: Refactor: move getRepo out of the App object https://review.openstack.org/173633 | 01:02 |
| openstackgerrit | Jamie Lennox proposed openstack-infra/project-config: Rename keystoneclient-federation to saml2 https://review.openstack.org/173634 | 01:03 |
| *** yamamoto has quit IRC | 01:05 | |
| *** mahito has joined #openstack-infra | 01:06 | |
| *** ayoung has joined #openstack-infra | 01:09 | |
| *** achanda has quit IRC | 01:10 | |
| *** jogo has joined #openstack-infra | 01:10 | |
| morganfainberg | want to see something neat? not sure if we care to fix / disable this: | 01:10 |
| morganfainberg | https://bugzilla.mozilla.org/show_bug.cgi?id=667429 | 01:10 |
| openstack | Mozilla bug 667429 in Networking: HTTP "Shibboleth Replay Detection Failure" [Normal,Resolved: invalid] - Assigned to nobody | 01:10 |
| morganfainberg | *points* OMG see it. | 01:10 |
| morganfainberg | :) | 01:11 |
| harlowja | clarkb https://github.com/stackforge/cloud-init/compare/0.7.x...harlowja:0.7.x-fixed is the changes that need to sync up, will talk to the other cloud-init folks to see what/when to do this... | 01:13 |
| harlowja | pita :-P | 01:13 |
| *** marun has quit IRC | 01:13 | |
| fungi | morganfainberg: fix/disable it where? | 01:14 |
| morganfainberg | fungi, not sure. | 01:14 |
| morganfainberg | fungi, in the puppet config for the bot that makes it talk to bugzilla | 01:15 |
| morganfainberg | fungi, we can probably make it only do LP | 01:15 |
| * morganfainberg isn't sure it is worth caring about | 01:15 | |
| fungi | morganfainberg: oh! i thought you meant fix/disable the apache issue mentioned there | 01:15 |
| fungi | we as a project do actually care about red hat bugs from time to time | 01:15 |
| morganfainberg | well that is a mozilla bug. | 01:16 |
| fungi | er, right | 01:16 |
| morganfainberg | but :P | 01:16 |
| fungi | less so about those ;) | 01:16 |
| morganfainberg | i wont worry, just surprised me to see it | 01:16 |
| morganfainberg | was like OH HAI | 01:16 |
| fungi | though it does a lot of bug trackers, for example https://bugs.debian.org/781027 | 01:16 |
| openstack | Debian bug 781027 in wnpp "ITP: nodepool -- nodepool a system to launch disposable OpenStack VM for testing purposes" [Wishlist,Open] | 01:16 |
| *** sdake_ has joined #openstack-infra | 01:17 | |
| clarkb | harlowja we sync what you told us | 01:18 |
| clarkb | harlowja we have very little way of knowing if it is correct | 01:18 |
| *** sdake has quit IRC | 01:21 | |
| *** dboik_ has joined #openstack-infra | 01:21 | |
| *** tqtran has quit IRC | 01:23 | |
| *** esker has joined #openstack-infra | 01:24 | |
| *** Daisy has joined #openstack-infra | 01:25 | |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/os-loganalyze: Tidy up generators into contained objects https://review.openstack.org/172711 | 01:26 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/os-loganalyze: Add support for setting filters and views per file https://review.openstack.org/173329 | 01:26 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/os-loganalyze: Allow the config to set the filter and view https://review.openstack.org/173328 | 01:26 |
| jhesketh | clarkb: ^ | 01:26 |
| *** Swami_ has joined #openstack-infra | 01:28 | |
| *** sarob has joined #openstack-infra | 01:30 | |
| *** Daisy_ has joined #openstack-infra | 01:30 | |
| *** yamamoto has joined #openstack-infra | 01:30 | |
| *** Swami has quit IRC | 01:32 | |
| *** Daisy has quit IRC | 01:33 | |
| *** Daisy_ is now known as Daisy | 01:33 | |
| *** Sukhdev has quit IRC | 01:34 | |
| *** nelsnelson has joined #openstack-infra | 01:39 | |
| *** woodster_ has quit IRC | 01:40 | |
| *** nelsnelson has quit IRC | 01:41 | |
| *** miqui has joined #openstack-infra | 01:44 | |
| *** shardy_z has quit IRC | 01:46 | |
| *** sdake has joined #openstack-infra | 01:52 | |
| *** yamahata has quit IRC | 01:53 | |
| *** baoli has quit IRC | 01:53 | |
| *** baoli has joined #openstack-infra | 01:54 | |
| *** baoli has quit IRC | 01:54 | |
| *** sdake_ has quit IRC | 01:56 | |
| *** kaisers has joined #openstack-infra | 01:58 | |
| *** kaisers1 has quit IRC | 01:59 | |
| *** gyee has quit IRC | 02:02 | |
| *** ivar-lazzaro has quit IRC | 02:02 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/devstack-gate: [WIP] Report memory usage low-point https://review.openstack.org/173647 | 02:04 |
| *** jokke_ has joined #openstack-infra | 02:09 | |
| jokke_ | anyone here still awake? | 02:09 |
| *** achanda has joined #openstack-infra | 02:10 | |
| *** amotoki has joined #openstack-infra | 02:11 | |
| ianw | jokke_: it's midday somewhere | 02:11 |
| jokke_ | ianw: fair point | 02:12 |
| *** otter768 has joined #openstack-infra | 02:13 | |
| jokke_ | So We'd like to create stable brances for glanceclient and glance_store ... where/how I should get that done? I'd expect I should propose change on some of the infra configs, but not sure where. | 02:13 |
| *** sarob has quit IRC | 02:13 | |
| jokke_ | I'd expect at least on the project-config acls | 02:14 |
| *** changbl has joined #openstack-infra | 02:14 | |
| *** unicell has quit IRC | 02:16 | |
| *** achanda has quit IRC | 02:17 | |
| *** harlowja is now known as harlowja_away | 02:17 | |
| *** otter768 has quit IRC | 02:18 | |
| *** woodster_ has joined #openstack-infra | 02:19 | |
| *** dims_ has quit IRC | 02:21 | |
| *** fedexo has joined #openstack-infra | 02:23 | |
| *** spzala has quit IRC | 02:26 | |
| *** sabeen1 has joined #openstack-infra | 02:27 | |
| *** yamahata has joined #openstack-infra | 02:28 | |
| *** salv-orlando has joined #openstack-infra | 02:28 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/devstack-gate: [WIP] Report memory usage low-point https://review.openstack.org/173647 | 02:32 |
| *** tkelsey has joined #openstack-infra | 02:36 | |
| lifeless | man | 02:37 |
| lifeless | someday we'll be allowed to have nice things | 02:37 |
| *** patrickeast has quit IRC | 02:40 | |
| *** salv-orlando has quit IRC | 02:41 | |
| *** davideagnello has quit IRC | 02:42 | |
| *** tkelsey has quit IRC | 02:42 | |
| *** julim has quit IRC | 02:43 | |
| *** sabeen1 has quit IRC | 02:43 | |
| *** SumitNaiksatam has joined #openstack-infra | 02:45 | |
| lifeless | fungi: how can do I root things on this node? | 02:46 |
| *** sabeen1 has joined #openstack-infra | 02:47 | |
| *** julim has joined #openstack-infra | 02:55 | |
| *** esker has quit IRC | 02:57 | |
| *** esker has joined #openstack-infra | 02:57 | |
| *** nijaba has quit IRC | 03:01 | |
| *** mriedem has joined #openstack-infra | 03:02 | |
| *** mahito has quit IRC | 03:03 | |
| *** mriedem1 has quit IRC | 03:04 | |
| *** otter768 has joined #openstack-infra | 03:05 | |
| *** nijaba has joined #openstack-infra | 03:06 | |
| lifeless | fungi: thanks, node can be freed | 03:06 |
| lifeless | clarkb: ^ | 03:06 |
| openstackgerrit | lifeless proposed openstack-dev/pbr: Test that MANIFEST.in excludes work https://review.openstack.org/123597 | 03:07 |
| *** mriedem has quit IRC | 03:08 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/devstack-gate: [WIP] Report memory usage low-point https://review.openstack.org/173647 | 03:13 |
| *** achanda has joined #openstack-infra | 03:13 | |
| *** sarob has joined #openstack-infra | 03:14 | |
| *** Sukhdev has joined #openstack-infra | 03:17 | |
| *** achanda has quit IRC | 03:17 | |
| *** Longgeek has joined #openstack-infra | 03:18 | |
| *** aswadr has joined #openstack-infra | 03:25 | |
| *** sputnik13 has quit IRC | 03:25 | |
| *** sputnik13 has joined #openstack-infra | 03:25 | |
| *** ashleighfarnham has joined #openstack-infra | 03:37 | |
| *** Longgeek has quit IRC | 03:41 | |
| *** achanda has joined #openstack-infra | 03:49 | |
| *** tiswanso_ has quit IRC | 03:50 | |
| *** ayoung is now known as ayoung_ZZZzzz | 03:52 | |
| *** aswadr has quit IRC | 03:54 | |
| *** amotoki_ has joined #openstack-infra | 03:54 | |
| *** ashleighfarnham has quit IRC | 03:56 | |
| *** SumitNaiksatam has quit IRC | 04:00 | |
| *** otter768 has quit IRC | 04:01 | |
| *** stevemar has quit IRC | 04:01 | |
| *** SumitNaiksatam has joined #openstack-infra | 04:01 | |
| *** ashleighfarnham has joined #openstack-infra | 04:04 | |
| *** sgordon has quit IRC | 04:04 | |
| *** SumitNaiksatam has quit IRC | 04:06 | |
| *** krtaylor has quit IRC | 04:07 | |
| *** mahito has joined #openstack-infra | 04:09 | |
| *** SumitNaiksatam has joined #openstack-infra | 04:09 | |
| *** sgordon has joined #openstack-infra | 04:10 | |
| *** krtaylor has joined #openstack-infra | 04:10 | |
| *** sdake has quit IRC | 04:11 | |
| *** ashleighfarnham has quit IRC | 04:13 | |
| *** patrickeast has joined #openstack-infra | 04:16 | |
| *** sarob has quit IRC | 04:16 | |
| *** sarob_ has joined #openstack-infra | 04:19 | |
| tchaypo | jokke_: I’d love to point you at the relevant docs in http://docs.openstack.org/infra/manual/drivers.html | 04:20 |
| tchaypo | but they don’t seem to exist | 04:20 |
| *** btran has quit IRC | 04:21 | |
| tchaypo | but there *are* instructions for creating release branches and I’m assuming the process for creating a stable/ branch would be similar | 04:21 |
| *** koolhead17 has joined #openstack-infra | 04:21 | |
| *** sks has joined #openstack-infra | 04:23 | |
| *** amotoki_ has quit IRC | 04:24 | |
| tchaypo | lifeless: I misread the “excludes” in that subject line as a verb and “work” as a noun. | 04:26 |
| tchaypo | english is delightful | 04:26 |
| *** crc32 has joined #openstack-infra | 04:28 | |
| *** mgedmin has quit IRC | 04:28 | |
| *** salv-orlando has joined #openstack-infra | 04:29 | |
| *** mgedmin has joined #openstack-infra | 04:29 | |
| *** sputnik13 has quit IRC | 04:29 | |
| *** davideagnello has joined #openstack-infra | 04:30 | |
| *** Daisy has quit IRC | 04:31 | |
| *** davideagnello has quit IRC | 04:35 | |
| *** salv-orlando has quit IRC | 04:36 | |
| *** Longgeek has joined #openstack-infra | 04:37 | |
| *** luqas has joined #openstack-infra | 04:39 | |
| *** luqas has quit IRC | 04:43 | |
| *** Longgeek has quit IRC | 04:44 | |
| *** Swami_ has quit IRC | 04:47 | |
| *** achanda has quit IRC | 04:50 | |
| *** achanda has joined #openstack-infra | 04:50 | |
| *** xyang1 has quit IRC | 04:52 | |
| *** BharatK has joined #openstack-infra | 04:53 | |
| *** wenlock has quit IRC | 04:54 | |
| *** stevemar has joined #openstack-infra | 04:55 | |
| *** sabeen1 has quit IRC | 04:56 | |
| *** yamada-h has quit IRC | 05:00 | |
| *** unicell has joined #openstack-infra | 05:01 | |
| *** yamada-h has joined #openstack-infra | 05:02 | |
| yamamoto | can anyone push this https://review.openstack.org/#/c/158415/ ? | 05:07 |
| yamamoto | the dependent patch has been merged | 05:07 |
| *** deepakcs has joined #openstack-infra | 05:08 | |
| *** mrodden has quit IRC | 05:12 | |
| *** Daisy has joined #openstack-infra | 05:16 | |
| lifeless | tchaypo: heh, no. | 05:16 |
| *** tnovacik has joined #openstack-infra | 05:17 | |
| *** Sukhdev has quit IRC | 05:17 | |
| *** thingee has quit IRC | 05:29 | |
| *** ociuhandu has quit IRC | 05:29 | |
| *** resker has joined #openstack-infra | 05:32 | |
| *** esker has quit IRC | 05:35 | |
| *** ociuhandu has joined #openstack-infra | 05:38 | |
| *** patrickeast has quit IRC | 05:40 | |
| *** sputnik13 has joined #openstack-infra | 05:40 | |
| *** yamada-h has quit IRC | 05:44 | |
| *** samwan has joined #openstack-infra | 05:45 | |
| samwan | is this the right place for devstack-gate query? | 05:45 |
| samwan | does anyone know why devstack-gate does not support redhat? | 05:46 |
| clarkb | samwan it should. we run tests on centos and fedora with it | 05:48 |
| samwan | no it does not support REDHAT | 05:48 |
| *** Longgeek has joined #openstack-infra | 05:48 | |
| samwan | there's a function named start_timer in functions.sh | 05:49 |
| samwan | it only allows fedora,centos | 05:49 |
| samwan | but not redhat | 05:49 |
| clarkb | can you explain further? both centos and fedora are redhat related | 05:49 |
| samwan | function is_fedora { | 05:49 |
| samwan | # note we consider CentOS 7 as fedora for now | 05:49 |
| samwan | lsb_release -i 2>/dev/null | grep -iq "fedora" || \ | 05:49 |
| samwan | lsb_release -i 2>/dev/null | grep -iq "CentOS" | 05:49 |
| samwan | } | 05:49 |
| samwan | because of this | 05:49 |
| samwan | if is_fedora; then | 05:49 |
| samwan | local ntp_service='ntpd' | 05:49 |
| samwan | elif uses_debs; then | 05:49 |
| samwan | local ntp_service='ntp' | 05:49 |
| samwan | else | 05:49 |
| samwan | echo "Unsupported platform, can't determine ntp service" | 05:49 |
| samwan | exit 1 | 05:49 |
| samwan | fi | 05:49 |
| clarkb | please use a paste service for large pastes | 05:50 |
| samwan | ok | 05:50 |
| samwan | sorry for that | 05:50 |
| clarkb | but looks like it just needs an updated check | 05:50 |
| clarkb | we do not test on rhel because it costs money | 05:50 |
| *** salv-orlando has joined #openstack-infra | 05:50 | |
| samwan | :) | 05:50 |
| clarkb | centos is the alternative we do support | 05:50 |
| samwan | i tested | 05:50 |
| clarkb | anyone can use that instead | 05:51 |
| samwan | i think we should add redhat in the function | 05:51 |
| samwan | in is_fedora | 05:51 |
| samwan | wo should include grep -iq "redhat" | 05:51 |
| clarkb | feel free to propose the addition | 05:51 |
| samwan | good | 05:51 |
| samwan | but i don't know how to propose for devstack-gate | 05:51 |
| clarkb | its the same as any openstack project. make changes then use git review to push to gerrit | 05:52 |
| samwan | oh yeah | 05:52 |
| samwan | thanks clarkb | 05:52 |
| *** mrodden has joined #openstack-infra | 05:55 | |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/shade: Invalidate user cache on user create https://review.openstack.org/173589 | 05:56 |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/shade: Add test for user_cache https://review.openstack.org/173584 | 05:56 |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/shade: MonkeyPatch time.sleep in unit tests to avoid wait https://review.openstack.org/173110 | 05:56 |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/shade: Test that deleting user invalidates user cache https://review.openstack.org/173596 | 05:56 |
| *** ociuhandu has quit IRC | 05:56 | |
| *** mrunge has joined #openstack-infra | 05:57 | |
| *** yfried is now known as yfried|afk | 05:59 | |
| *** yfried|afk is now known as yfried | 05:59 | |
| *** salv-orlando has quit IRC | 06:00 | |
| *** otter768 has joined #openstack-infra | 06:02 | |
| *** unicell has quit IRC | 06:02 | |
| *** luqas has joined #openstack-infra | 06:04 | |
| *** mrunge has quit IRC | 06:05 | |
| *** garyk has joined #openstack-infra | 06:06 | |
| *** otter768 has quit IRC | 06:07 | |
| *** mrunge has joined #openstack-infra | 06:08 | |
| *** hdd has joined #openstack-infra | 06:09 | |
| *** yfried is now known as yfried|afk | 06:10 | |
| *** fedexo has quit IRC | 06:13 | |
| *** crc32 has quit IRC | 06:15 | |
| *** stevemar has quit IRC | 06:15 | |
| openstackgerrit | Merged openstack-infra/project-config: Enable DVR for Neutron aiopcpu job https://review.openstack.org/158415 | 06:18 |
| *** alex_xu has quit IRC | 06:18 | |
| *** isviridov_away is now known as isviridov | 06:20 | |
| *** yfried|afk is now known as yfried | 06:20 | |
| *** miqui has quit IRC | 06:21 | |
| openstackgerrit | greghaynes proposed openstack-infra/project-config: Add diskimage-builder functests https://review.openstack.org/173585 | 06:22 |
| *** scheuran has joined #openstack-infra | 06:25 | |
| *** armax has quit IRC | 06:28 | |
| *** jlibosva has joined #openstack-infra | 06:28 | |
| *** jcoufal has joined #openstack-infra | 06:29 | |
| lifeless | jd__: tchaypo: thanks for the reviews.. but https://review.openstack.org/#/c/124651/ deeper in the stack is what actually needs reviews :) | 06:29 |
| jd__ | lifeless: yeah I'm reading :) | 06:30 |
| *** alex_xu has joined #openstack-infra | 06:30 | |
| lifeless | jd__: thanks :) | 06:30 |
| *** yamada-h has joined #openstack-infra | 06:30 | |
| tchaypo | lifeless: starred for review later | 06:31 |
| *** yfried is now known as yfried|afk | 06:32 | |
| tchaypo | lifeless: on which topic, 172691 is the latest gertty magic from jeblair - batches a bunch of small pushes up to the server. Upstream from that is a patch that avoids some duplicate tasks that were hitting the queue. | 06:33 |
| jlibosva | /quit | 06:33 |
| *** jlibosva has quit IRC | 06:33 | |
| tchaypo | I’m not sure which one you’re at, but I’m having a much better experience running with the latest 172691 | 06:33 |
| lifeless | tchaypo: I am not currently running gertty | 06:34 |
| lifeless | tchaypo: I didn't have time to fix it, and I didn't have time to use it | 06:34 |
| *** jlibosva has joined #openstack-infra | 06:34 | |
| *** yamada-h has quit IRC | 06:35 | |
| tchaypo | Would you be interested in me pinging you when those two things land? | 06:35 |
| *** ildikov has quit IRC | 06:38 | |
| lifeless | tchaypo: sure | 06:38 |
| *** tkelsey has joined #openstack-infra | 06:39 | |
| StevenK | lifeless: testtools will automatically pull in unittest2, or it needs some help? | 06:39 |
| lifeless | StevenK: define 'pull in' | 06:40 |
| StevenK | lifeless: import and use it. Trying to debug AttributeError: 'TestObject' object has no attribute 'assertRaisesRegex' | 06:41 |
| lifeless | StevenK: testtools has a versioned dep on unittest2 | 06:41 |
| lifeless | StevenK: whats the __mro__ for TestObject there? | 06:41 |
| *** soren has joined #openstack-infra | 06:42 | |
| *** tkelsey has quit IRC | 06:44 | |
| StevenK | lifeless: That's going to require patching it in, because docker. :-/ | 06:44 |
| *** unicell has joined #openstack-infra | 06:45 | |
| *** shardy has joined #openstack-infra | 06:45 | |
| lifeless | StevenK: isn't there some way to get a debug session? | 06:45 |
| lifeless | StevenK: of course, if you're using docker, are you sure you're installing testtools from PyPI and not an ancient-ass-binary? | 06:46 |
| StevenK | lifeless: I am not installing testtools from pypi. It's coming from a local bootstrap archive and is 0.9.39. | 06:47 |
| lifeless | thats your problem | 06:47 |
| *** achanda has quit IRC | 06:48 | |
| * StevenK grabs 1.7.1 from pypi and tries to package it up | 06:48 | |
| *** mwhudson has quit IRC | 06:48 | |
| *** woodster_ has quit IRC | 06:50 | |
| *** sushilkm has joined #openstack-infra | 06:52 | |
| *** sushilkm has left #openstack-infra | 06:52 | |
| *** yfried|afk is now known as yfried | 06:52 | |
| lifeless | StevenK: you'll probably need to package linecache2, traceback2, unittest2 latest versions to go with | 06:53 |
| *** rfolco has quit IRC | 06:55 | |
| *** sputnik1_ has joined #openstack-infra | 06:57 | |
| *** Ala has joined #openstack-infra | 06:59 | |
| *** sputnik13 has quit IRC | 07:00 | |
| *** sputnik13 has joined #openstack-infra | 07:01 | |
| *** yamada-h has joined #openstack-infra | 07:01 | |
| *** sputnik__ has joined #openstack-infra | 07:02 | |
| *** mrmartin has joined #openstack-infra | 07:03 | |
| *** samwan has quit IRC | 07:03 | |
| *** sputnik1_ has quit IRC | 07:03 | |
| *** yamada-h has quit IRC | 07:05 | |
| *** jeblair has quit IRC | 07:05 | |
| *** sputnik13 has quit IRC | 07:06 | |
| openstackgerrit | lifeless proposed openstack-dev/pbr: Fixup semver https://review.openstack.org/124651 | 07:06 |
| openstackgerrit | lifeless proposed openstack-dev/pbr: Test that MANIFEST.in excludes work https://review.openstack.org/123597 | 07:06 |
| *** dtantsur|afk is now known as dtantsur | 07:06 | |
| *** jeblair has joined #openstack-infra | 07:07 | |
| *** salv-orlando has joined #openstack-infra | 07:08 | |
| *** rwsu has quit IRC | 07:09 | |
| *** Guest87529 has joined #openstack-infra | 07:09 | |
| *** Guest87529 has quit IRC | 07:11 | |
| *** Guest87529 has joined #openstack-infra | 07:11 | |
| *** yfried is now known as yfried|afk | 07:12 | |
| *** markus_z has joined #openstack-infra | 07:12 | |
| *** yfried|afk is now known as yfried | 07:15 | |
| *** jistr has joined #openstack-infra | 07:24 | |
| *** sergsh has joined #openstack-infra | 07:28 | |
| *** sputnik13 has joined #openstack-infra | 07:31 | |
| *** yfried is now known as yfried|afk | 07:33 | |
| *** vlaza has joined #openstack-infra | 07:34 | |
| *** sputnik__ has quit IRC | 07:34 | |
| *** chlong has quit IRC | 07:35 | |
| *** yamada-h has joined #openstack-infra | 07:37 | |
| *** [HeOS] has quit IRC | 07:38 | |
| *** rfolco has joined #openstack-infra | 07:39 | |
| *** Guest87529 is now known as mwhudson | 07:40 | |
| *** jlanoux has joined #openstack-infra | 07:42 | |
| *** SumitNaiksatam has quit IRC | 07:44 | |
| *** SumitNaiksatam has joined #openstack-infra | 07:45 | |
| *** fhubik has joined #openstack-infra | 07:47 | |
| *** belmoreira has joined #openstack-infra | 07:47 | |
| *** mwhudson has left #openstack-infra | 07:47 | |
| *** yamada-h has joined #openstack-infra | 07:48 | |
| *** ildikov has joined #openstack-infra | 07:49 | |
| *** arxcruz has joined #openstack-infra | 07:50 | |
| *** hashar has joined #openstack-infra | 07:55 | |
| *** Somay has joined #openstack-infra | 07:58 | |
| *** e0ne has joined #openstack-infra | 08:02 | |
| *** otter768 has joined #openstack-infra | 08:02 | |
| *** e0ne has quit IRC | 08:05 | |
| *** otter768 has quit IRC | 08:07 | |
| *** Hal has joined #openstack-infra | 08:08 | |
| *** e0ne has joined #openstack-infra | 08:08 | |
| *** Hal is now known as Guest93251 | 08:08 | |
| *** kiseok7_ has quit IRC | 08:08 | |
| *** davideagnello has joined #openstack-infra | 08:08 | |
| *** MaxV has joined #openstack-infra | 08:10 | |
| *** ildikov has quit IRC | 08:12 | |
| *** MaxV has quit IRC | 08:12 | |
| *** tkelsey has joined #openstack-infra | 08:13 | |
| *** davideagnello has quit IRC | 08:13 | |
| *** mpavone has joined #openstack-infra | 08:14 | |
| *** yfried|afk is now known as yfried | 08:17 | |
| *** e0ne has quit IRC | 08:19 | |
| *** Somay has quit IRC | 08:21 | |
| *** e0ne has joined #openstack-infra | 08:23 | |
| *** yamahata has quit IRC | 08:25 | |
| *** e0ne has quit IRC | 08:29 | |
| *** ociuhandu has joined #openstack-infra | 08:31 | |
| *** MaxV has joined #openstack-infra | 08:32 | |
| *** e0ne has joined #openstack-infra | 08:33 | |
| *** mahito has quit IRC | 08:34 | |
| *** cdent has joined #openstack-infra | 08:36 | |
| *** dtantsur is now known as dtantsur|brb | 08:37 | |
| *** vdo has joined #openstack-infra | 08:38 | |
| *** teran has quit IRC | 08:39 | |
| *** ildikov has joined #openstack-infra | 08:40 | |
| *** e0ne has quit IRC | 08:41 | |
| *** sarob_ has quit IRC | 08:46 | |
| *** otherwiseguy has quit IRC | 08:47 | |
| *** ihrachyshka has joined #openstack-infra | 08:49 | |
| *** otherwiseguy has joined #openstack-infra | 08:49 | |
| *** pelix has joined #openstack-infra | 08:50 | |
| *** ssam2 has joined #openstack-infra | 08:55 | |
| *** hichihara has quit IRC | 08:55 | |
| *** krtaylor has quit IRC | 08:57 | |
| *** pblaho has joined #openstack-infra | 08:59 | |
| *** amotoki has quit IRC | 09:01 | |
| *** pblaho has quit IRC | 09:02 | |
| *** pblaho has joined #openstack-infra | 09:02 | |
| *** krtaylor has joined #openstack-infra | 09:04 | |
| *** yfried is now known as yfried|afk | 09:06 | |
| *** yfried|afk is now known as yfried | 09:08 | |
| *** sdake has joined #openstack-infra | 09:10 | |
| openstackgerrit | Antoine Musso proposed openstack-infra/nodepool: Soften sqlalchemy and pyzmq requirements https://review.openstack.org/173731 | 09:13 |
| *** fhubik is now known as fhubik_afk | 09:21 | |
| *** freerunner has quit IRC | 09:23 | |
| *** [HeOS] has joined #openstack-infra | 09:23 | |
| *** Daisy has quit IRC | 09:23 | |
| *** freerunner has joined #openstack-infra | 09:25 | |
| *** freerunner has quit IRC | 09:25 | |
| *** derekh has joined #openstack-infra | 09:25 | |
| *** freerunner has joined #openstack-infra | 09:26 | |
| *** freerunner has quit IRC | 09:26 | |
| *** freerunner has joined #openstack-infra | 09:27 | |
| *** ildikov has quit IRC | 09:28 | |
| *** freerunner has quit IRC | 09:29 | |
| *** freerunner has joined #openstack-infra | 09:29 | |
| *** fhubik_afk is now known as fhubik | 09:30 | |
| *** freerunner has quit IRC | 09:31 | |
| *** freerunner has joined #openstack-infra | 09:32 | |
| *** freerunner has quit IRC | 09:33 | |
| *** e0ne has joined #openstack-infra | 09:34 | |
| *** freerunner has joined #openstack-infra | 09:34 | |
| *** freerunner has quit IRC | 09:35 | |
| *** freerunner has joined #openstack-infra | 09:36 | |
| *** freerunner has quit IRC | 09:37 | |
| *** freerunner has joined #openstack-infra | 09:38 | |
| *** freerunner has quit IRC | 09:38 | |
| *** freerunner has joined #openstack-infra | 09:39 | |
| *** garyk has quit IRC | 09:39 | |
| *** garyk has joined #openstack-infra | 09:40 | |
| *** freerunner has quit IRC | 09:40 | |
| *** freerunner has joined #openstack-infra | 09:41 | |
| *** freerunner has quit IRC | 09:42 | |
| *** ZZelle has quit IRC | 09:44 | |
| *** freerunner has joined #openstack-infra | 09:44 | |
| *** fhubik is now known as fhubik_afk | 09:45 | |
| *** freerunner has quit IRC | 09:45 | |
| *** Guest93251 has quit IRC | 09:47 | |
| *** e0ne is now known as e0ne_ | 09:49 | |
| *** liusheng has quit IRC | 09:51 | |
| *** freerunner has joined #openstack-infra | 09:51 | |
| *** jamielennox is now known as jamielennox|away | 09:52 | |
| *** freerunner has quit IRC | 09:52 | |
| *** freerunner has joined #openstack-infra | 09:54 | |
| *** freerunner has quit IRC | 09:54 | |
| *** jhesketh has quit IRC | 09:57 | |
| *** Somay has joined #openstack-infra | 09:58 | |
| *** jhesketh has joined #openstack-infra | 10:00 | |
| *** amotoki has joined #openstack-infra | 10:00 | |
| *** ildikov has joined #openstack-infra | 10:01 | |
| *** e0ne_ has quit IRC | 10:01 | |
| *** Longgeek has quit IRC | 10:02 | |
| *** Longgeek_ has joined #openstack-infra | 10:02 | |
| *** fhubik_afk is now known as fhubik | 10:03 | |
| *** e0ne has joined #openstack-infra | 10:03 | |
| *** Kennan2 has joined #openstack-infra | 10:03 | |
| *** otter768 has joined #openstack-infra | 10:03 | |
| *** Kennan has quit IRC | 10:04 | |
| *** ssam2 has quit IRC | 10:05 | |
| *** isviridov is now known as isviridov_away | 10:06 | |
| openstackgerrit | Erno Kuvaja proposed openstack-infra/project-config: Add stable acls for python-glanceclient https://review.openstack.org/173748 | 10:06 |
| openstackgerrit | Erno Kuvaja proposed openstack-infra/project-config: Add stable acls for python-glanceclient https://review.openstack.org/173748 | 10:08 |
| *** yamamoto has quit IRC | 10:08 | |
| *** otter768 has quit IRC | 10:08 | |
| *** teran has joined #openstack-infra | 10:09 | |
| *** dtantsur|brb is now known as dtantsur | 10:10 | |
| *** teran has quit IRC | 10:16 | |
| *** pc_m has joined #openstack-infra | 10:17 | |
| *** sarob has joined #openstack-infra | 10:19 | |
| *** ssam2 has joined #openstack-infra | 10:20 | |
| *** achanda has joined #openstack-infra | 10:22 | |
| *** dguitarbite has joined #openstack-infra | 10:24 | |
| *** sarob has quit IRC | 10:24 | |
| *** jcoufal has quit IRC | 10:27 | |
| *** achanda has quit IRC | 10:27 | |
| *** jcoufal has joined #openstack-infra | 10:27 | |
| *** tnovacik has quit IRC | 10:29 | |
| *** koolhead17 has quit IRC | 10:31 | |
| *** hashar is now known as hasharLunch | 10:33 | |
| *** ociuhandu has quit IRC | 10:34 | |
| *** claudiub has joined #openstack-infra | 10:34 | |
| *** MIDENN_ has quit IRC | 10:35 | |
| *** Somay has quit IRC | 10:36 | |
| *** Somay_ has joined #openstack-infra | 10:36 | |
| *** ssam2 has quit IRC | 10:38 | |
| *** e0ne is now known as e0ne_ | 10:39 | |
| *** yfried is now known as yfried|afk | 10:39 | |
| *** amotoki_ has joined #openstack-infra | 10:40 | |
| *** teran has joined #openstack-infra | 10:41 | |
| *** e0ne_ is now known as e0ne | 10:42 | |
| *** amotoki has quit IRC | 10:43 | |
| *** mfmcdonagh has joined #openstack-infra | 10:44 | |
| *** yfried|afk is now known as yfried | 10:45 | |
| *** juggler has quit IRC | 10:45 | |
| *** sdake has quit IRC | 10:46 | |
| *** deepakcs has quit IRC | 10:46 | |
| *** Somay_ has quit IRC | 10:48 | |
| *** asselin_ has quit IRC | 10:49 | |
| *** ssam2 has joined #openstack-infra | 10:50 | |
| *** fhubik has quit IRC | 10:53 | |
| *** e0ne has quit IRC | 10:53 | |
| *** e0ne has joined #openstack-infra | 10:54 | |
| *** fhubik has joined #openstack-infra | 10:54 | |
| yolanda | here, do you know if updating an upstream from an already existing project works? i amended one, but after running manage_projects, same content in the buggy repo | 10:55 |
| *** dhritishikhar has joined #openstack-infra | 10:56 | |
| *** mattfarina has joined #openstack-infra | 10:58 | |
| *** arxcruz has quit IRC | 11:01 | |
| *** arxcruz has joined #openstack-infra | 11:02 | |
| *** amotoki has joined #openstack-infra | 11:02 | |
| openstackgerrit | Igor Belikov proposed openstack-infra/jenkins-job-builder: Add node-label parameter to trigger-builds builder https://review.openstack.org/173775 | 11:04 |
| *** mattfarina has quit IRC | 11:04 | |
| *** amotoki_ has quit IRC | 11:05 | |
| *** zz_johnthetubagu is now known as johnthetubaguy | 11:07 | |
| *** dims has joined #openstack-infra | 11:08 | |
| *** ildikov has quit IRC | 11:09 | |
| *** BharatK has quit IRC | 11:09 | |
| *** mattfarina has joined #openstack-infra | 11:11 | |
| *** btully has quit IRC | 11:14 | |
| *** mrmartin has quit IRC | 11:16 | |
| *** btully has joined #openstack-infra | 11:16 | |
| *** e0ne is now known as e0ne_ | 11:19 | |
| *** mwagner_lap has quit IRC | 11:21 | |
| openstackgerrit | Bharat Kumar Kobagana proposed openstack-infra/project-config: Skip test_volume_boot_pattern for GlusterFS until bug 1441050 is fixed https://review.openstack.org/173408 | 11:22 |
| openstack | bug 1441050 in Cinder "test_volume_boot_pattern tempest test failure for glusterfs backend" [Undecided,In progress] https://launchpad.net/bugs/1441050 - Assigned to Bharat Kumar Kobagana (bharat-kobagana) | 11:22 |
| *** dhritishikhar has quit IRC | 11:22 | |
| *** BharatK has joined #openstack-infra | 11:23 | |
| *** [HeOS] has quit IRC | 11:25 | |
| *** sergmelikyan has quit IRC | 11:25 | |
| *** [HeOS] has joined #openstack-infra | 11:26 | |
| *** e0ne_ has quit IRC | 11:29 | |
| *** Longgeek has joined #openstack-infra | 11:29 | |
| *** otter768 has joined #openstack-infra | 11:31 | |
| *** Longgeek_ has quit IRC | 11:31 | |
| *** nadya has joined #openstack-infra | 11:33 | |
| *** nadya is now known as Guest27177 | 11:34 | |
| *** spredzy is now known as spredzy|afk | 11:34 | |
| *** Somay has joined #openstack-infra | 11:34 | |
| vladiskuz | Hi, folks! I've been playing with OAuth2.0 on openstackid.org and with User REST API on it. User API get following fields: name, family_name, nickname, picture, birthdate, gender, and email. Question: How can I distinguish between users? because in https://www.openstack.org/profile/ I can change my email and other fields | 11:34 |
| *** Guest27177 is now known as _nadya_ | 11:35 | |
| *** ociuhandu has joined #openstack-infra | 11:37 | |
| *** Mmike has quit IRC | 11:38 | |
| *** e0ne has joined #openstack-infra | 11:38 | |
| *** Mmike has joined #openstack-infra | 11:41 | |
| *** Mmike has joined #openstack-infra | 11:41 | |
| *** macjack has left #openstack-infra | 11:44 | |
| BobBall | Can someone please remove the email address from the "XenProject CI" gerrit account so we don't get automatic emails from gerrit? | 11:44 |
| *** davideagnello has joined #openstack-infra | 11:46 | |
| *** fhubik is now known as fhubik_afk | 11:47 | |
| *** hasharLunch is now known as hashar | 11:48 | |
| *** davideagnello has quit IRC | 11:50 | |
| *** doug-fish has joined #openstack-infra | 11:53 | |
| *** tnovacik has joined #openstack-infra | 11:54 | |
| *** otter768 has quit IRC | 11:56 | |
| *** dkranz has joined #openstack-infra | 11:56 | |
| *** freerunner has joined #openstack-infra | 11:57 | |
| *** freerunner has quit IRC | 11:57 | |
| *** freerunner has joined #openstack-infra | 11:58 | |
| *** freerunner has quit IRC | 11:59 | |
| *** isaic has joined #openstack-infra | 12:00 | |
| *** freerunner has joined #openstack-infra | 12:01 | |
| *** julim has quit IRC | 12:01 | |
| *** freerunner has quit IRC | 12:03 | |
| *** freerunner has joined #openstack-infra | 12:03 | |
| *** Hal has joined #openstack-infra | 12:04 | |
| *** notnownikki has quit IRC | 12:04 | |
| *** isaic has left #openstack-infra | 12:04 | |
| *** Hal is now known as Guest32383 | 12:04 | |
| *** freerunner has quit IRC | 12:05 | |
| *** isaic has joined #openstack-infra | 12:05 | |
| *** isaic has left #openstack-infra | 12:05 | |
| *** ldnunes has joined #openstack-infra | 12:05 | |
| *** notnownikki has joined #openstack-infra | 12:05 | |
| *** freerunner has joined #openstack-infra | 12:07 | |
| *** mrmartin has joined #openstack-infra | 12:07 | |
| *** dprince has joined #openstack-infra | 12:08 | |
| *** freerunner has quit IRC | 12:08 | |
| *** sarob has joined #openstack-infra | 12:08 | |
| *** freerunner has joined #openstack-infra | 12:09 | |
| *** freerunner has quit IRC | 12:09 | |
| *** mwagner_lap has joined #openstack-infra | 12:09 | |
| *** freerunner has joined #openstack-infra | 12:12 | |
| *** freerunner has quit IRC | 12:12 | |
| *** sarob has quit IRC | 12:12 | |
| *** yamamoto has joined #openstack-infra | 12:13 | |
| *** laurelm has quit IRC | 12:13 | |
| *** freerunner has joined #openstack-infra | 12:14 | |
| *** e0ne is now known as e0ne_ | 12:14 | |
| *** laurelm has joined #openstack-infra | 12:15 | |
| fungi | jokke_: check with ttx. the release managers have been coordinating creation of client stable branches so far from what i've seen | 12:15 |
| fungi | tchaypo: ^ (for reference) | 12:15 |
| mrmartin | fungi, morning | 12:16 |
| mrmartin | have you seen my email about openstackid-dev ? | 12:16 |
| tchaypo | Thanks fungi | 12:16 |
| *** freerunner has quit IRC | 12:16 | |
| *** fhubik_afk is now known as fhubik | 12:17 | |
| tchaypo | I'd like to add something about this to the docs | 12:17 |
| jokke_ | fungi: thanks ... yeah I noticed that the acl change needs to happen through the project-config repo, but I do not have rights to push the actual branch | 12:17 |
| *** freerunner has joined #openstack-infra | 12:17 | |
| tchaypo | It sounds like, as a start, a pointer to the release manager email address (I'm assuming there is one) should be good? | 12:17 |
| *** aysyd has joined #openstack-infra | 12:18 | |
| fungi | yolanda: manage-projects intentionally ignores the upstream parameter except on initial import (when the project does not yet exist in gerrit), or for its somewhat obscure upstream tracking feature (where you keep upstream and a fork in separate branches) | 12:18 |
| *** freerunner has quit IRC | 12:19 | |
| fungi | tchaypo: jokke_: probably best to ask ttx what he wants the documentation to recommend | 12:20 |
| ttx | we'll be craeting stable branches for clients this afternoon | 12:21 |
| fungi | mrmartin: thanks, fixing! | 12:21 |
| vladiskuz | Folks, have you seen my question above? | 12:22 |
| jokke_ | ttx: glance_store would need one as well, thanks | 12:22 |
| ttx | yes, it's on the list | 12:22 |
| jokke_ | ttx: gr8 | 12:22 |
| *** bswartz has quit IRC | 12:22 | |
| ttx | also been discussing the client bump with stuart, we have that under control | 12:22 |
| *** freerunner has joined #openstack-infra | 12:23 | |
| jokke_ | ttx: I know, that's why I tried to get the stable branches done | 12:23 |
| *** yfried is now known as yfried|afk | 12:23 | |
| vladiskuz | I've been playing with OAuth2.0 on openstackid.org and with User REST API on it. User API get following fields: name, family_name, nickname, picture, birthdate, gender, and email. Question: How can I distinguish between users? because on https://www.openstack.org/profile/ I can change my email and other fields | 12:24 |
| mrmartin | vladiskuz: yeap, it is a problem | 12:25 |
| *** fhubik has quit IRC | 12:25 | |
| *** amotoki has quit IRC | 12:25 | |
| *** yfried|afk is now known as yfried | 12:26 | |
| vladiskuz | If I change my email than I get new openid with suffix .1 | 12:26 |
| *** fhubik has joined #openstack-infra | 12:26 | |
| *** e0ne_ has quit IRC | 12:26 | |
| *** spredzy|afk is now known as spredzy | 12:26 | |
| mrmartin | sure, we can ask smarcet, what he recommends here. | 12:26 |
| fungi | mrmartin: fixed now | 12:26 |
| *** amotoki has joined #openstack-infra | 12:27 | |
| *** vlaza1 has joined #openstack-infra | 12:28 | |
| *** vlaza has quit IRC | 12:28 | |
| mrmartin | fungi: ack, works well. | 12:28 |
| mrmartin | thank you! | 12:28 |
| *** cdent has quit IRC | 12:28 | |
| fungi | thanks for spotting my mistake! | 12:28 |
| mrmartin | anyway Sebastian plan to move the openstackid_id db too on dev. | 12:28 |
| *** e0ne has joined #openstack-infra | 12:30 | |
| *** bknudson has joined #openstack-infra | 12:30 | |
| *** amotoki has quit IRC | 12:31 | |
| yolanda | fungi, thx, and if i did a mistake and create the wrong upstream, what's better? i remove the project and i recreate? | 12:32 |
| *** koolhead17 has joined #openstack-infra | 12:33 | |
| fungi | yolanda: elevate yourself to a member of whatever your equivalent of our project bootstrappers group is, and then push --force a new local repo in on top of it | 12:33 |
| *** koolhead17 has quit IRC | 12:33 | |
| *** koolhead17 has joined #openstack-infra | 12:33 | |
| yolanda | ok, thx | 12:33 |
| fungi | yolanda: i assume you're talking about a gerrit you manage, and not review.openstack.org | 12:33 |
| yolanda | yes | 12:33 |
| yolanda | our gerrit | 12:33 |
| fungi | mrmartin: oh, that's good to know. hopefully he'll give us a heads up when he's doing it this time | 12:34 |
| *** eharney has joined #openstack-infra | 12:34 | |
| *** gordc has joined #openstack-infra | 12:34 | |
| mrmartin | yeah, I asked him to announce *before* the change | 12:34 |
| *** dkliban_afk is now known as dkliban | 12:35 | |
| *** ZZelle has joined #openstack-infra | 12:39 | |
| *** dkranz has quit IRC | 12:40 | |
| *** baoli has joined #openstack-infra | 12:42 | |
| *** dhritishikhar has joined #openstack-infra | 12:42 | |
| *** jaypipes has quit IRC | 12:42 | |
| *** otter768 has joined #openstack-infra | 12:43 | |
| *** julim has joined #openstack-infra | 12:43 | |
| *** Somay has quit IRC | 12:45 | |
| *** otter768 has quit IRC | 12:46 | |
| *** smccully has joined #openstack-infra | 12:46 | |
| *** rlandy has joined #openstack-infra | 12:46 | |
| *** arxcruz has quit IRC | 12:47 | |
| *** annegentle has joined #openstack-infra | 12:50 | |
| *** woodster_ has joined #openstack-infra | 12:50 | |
| *** mrmartin has quit IRC | 12:52 | |
| openstackgerrit | Merged openstack-infra/project-config: change name from aiopcpu to multinode https://review.openstack.org/173614 | 12:53 |
| *** baoli_ has joined #openstack-infra | 12:54 | |
| openstackgerrit | Merged openstack-infra/system-config: Turn F20 min-ready down to zero before removal https://review.openstack.org/171485 | 12:54 |
| *** garyk has quit IRC | 12:54 | |
| openstackgerrit | Igor Belikov proposed openstack-infra/jenkins-job-builder: Add node-label parameter to trigger-builds builder https://review.openstack.org/173775 | 12:56 |
| openstackgerrit | Merged openstack-infra/system-config: Remove get-pip.py after using it in install_puppet https://review.openstack.org/168140 | 12:56 |
| *** baoli has quit IRC | 12:56 | |
| *** jaypipes has joined #openstack-infra | 12:56 | |
| openstackgerrit | Merged openstack-infra/system-config: Rename summit server to design-summit-prep https://review.openstack.org/170132 | 12:57 |
| openstackgerrit | Julia Kreger proposed openstack-infra/shade: Wrap ironicclient methods that leak objects https://review.openstack.org/173567 | 12:57 |
| openstackgerrit | Julia Kreger proposed openstack-infra/shade: Change Ironic node lookups to support names https://review.openstack.org/171164 | 12:57 |
| *** claudiub has quit IRC | 12:57 | |
| *** bswartz has joined #openstack-infra | 12:58 | |
| *** resker has quit IRC | 12:58 | |
| *** mrunge has quit IRC | 12:59 | |
| *** dhritishikhar has quit IRC | 13:00 | |
| *** ayoung_ZZZzzz is now known as ayoung | 13:06 | |
| *** mattfarina has quit IRC | 13:06 | |
| *** kgiusti has joined #openstack-infra | 13:06 | |
| *** miqui has joined #openstack-infra | 13:06 | |
| *** dboik_ has quit IRC | 13:08 | |
| *** mattfarina has joined #openstack-infra | 13:08 | |
| *** dhritishikhar has joined #openstack-infra | 13:10 | |
| *** adalbas has quit IRC | 13:12 | |
| *** dims has quit IRC | 13:13 | |
| eantyshev | Hello! Are there any dsvm jobs with centos7 images? | 13:14 |
| *** fhubik is now known as fhubik_afk | 13:14 | |
| *** mjturek1 has left #openstack-infra | 13:14 | |
| *** dhritishikhar has quit IRC | 13:14 | |
| *** dims has joined #openstack-infra | 13:15 | |
| *** fhubik_afk is now known as fhubik | 13:16 | |
| *** dhritishikhar has joined #openstack-infra | 13:17 | |
| mordred | jeblair: I have a strange request for gertty ... in unified diff mode - maybe skip printing the line numbers on the left-hand side in favor of a old,new line at the top of the section like diff -u does | 13:18 |
| *** duran has joined #openstack-infra | 13:19 | |
| mordred | jeblair: because in a normal 80 char xterm, unified diffs of openstack code should actually never wrap, since we're pedantic about that sort of thing | 13:19 |
| mordred | jeblair: but the line numbers make it wrap (this is also not really a problem and is easily fixable on my side - just musing outloud) | 13:20 |
| *** david-lyle has quit IRC | 13:24 | |
| *** dustins has joined #openstack-infra | 13:25 | |
| *** dboik has joined #openstack-infra | 13:25 | |
| *** dhritishikhar has quit IRC | 13:26 | |
| *** dhritishikhar has joined #openstack-infra | 13:26 | |
| *** mriedem has joined #openstack-infra | 13:27 | |
| *** tiswanso has joined #openstack-infra | 13:29 | |
| *** peristeri has joined #openstack-infra | 13:30 | |
| *** wenlock has joined #openstack-infra | 13:30 | |
| *** r-daneel has joined #openstack-infra | 13:31 | |
| *** Daisy has joined #openstack-infra | 13:32 | |
| openstackgerrit | Julia Kreger proposed openstack-infra/shade: Basic test for meta method obj_list_to_dict https://review.openstack.org/173871 | 13:33 |
| *** garyk has joined #openstack-infra | 13:33 | |
| openstackgerrit | Russell Bryant proposed openstack-infra/project-config: Add #openstack-neutron-ovn to gerritbot. https://review.openstack.org/173873 | 13:33 |
| *** BharatK has quit IRC | 13:35 | |
| openstackgerrit | Russell Bryant proposed openstack-infra/system-config: Log #openstack-neutron-ovn. https://review.openstack.org/173876 | 13:36 |
| openstackgerrit | Merged openstack-infra/shade: Add meta method obj_list_to_dict https://review.openstack.org/173600 | 13:36 |
| spredzy | if there is any core around, could I get a review https://review.openstack.org/#/c/173255/ . Thanks. | 13:38 |
| *** yfried has quit IRC | 13:39 | |
| *** marcusvrn has quit IRC | 13:41 | |
| *** garyk has quit IRC | 13:41 | |
| *** yfried has joined #openstack-infra | 13:41 | |
| openstackgerrit | Julia Kreger proposed openstack-infra/shade: Add update_machine method and operator unit test substrate https://review.openstack.org/171165 | 13:42 |
| *** e0ne is now known as e0ne_ | 13:43 | |
| *** e0ne_ is now known as e0ne | 13:43 | |
| *** hdd has quit IRC | 13:46 | |
| *** mtanino has joined #openstack-infra | 13:48 | |
| *** markvoelker has joined #openstack-infra | 13:48 | |
| openstackgerrit | Merged openstack-infra/shade: Add API method create_subnet() https://review.openstack.org/170217 | 13:49 |
| *** hdd has joined #openstack-infra | 13:50 | |
| openstackgerrit | Merged openstack-infra/shade: Add API method delete_subnet() https://review.openstack.org/170477 | 13:52 |
| openstackgerrit | Merged openstack-infra/shade: Add API method delete_subnet() https://review.openstack.org/170477 | 13:52 |
| openstackgerrit | Merged openstack-infra/shade: Add API method update_subnet() https://review.openstack.org/170516 | 13:52 |
| *** dkliban has left #openstack-infra | 13:53 | |
| openstackgerrit | Merged openstack-infra/shade: Add get_subnet() method https://review.openstack.org/171760 | 13:53 |
| sdague | fungi: our deleting servers get wedged? | 13:54 |
| *** vlaza1 is now known as vlaza | 13:54 | |
| sdague | I see kind of steady growth over time there | 13:54 |
| openstackgerrit | Merged openstack-dev/cookiecutter: Don't omit tests from coverage https://review.openstack.org/170594 | 13:56 |
| *** marcusvrn has joined #openstack-infra | 13:56 | |
| *** sarob has joined #openstack-infra | 13:57 | |
| openstackgerrit | Steven Weston proposed openstack-infra/infra-specs: CI Dashboard Spec https://review.openstack.org/135170 | 13:57 |
| *** fhubik has quit IRC | 13:57 | |
| *** cference_ has joined #openstack-infra | 13:57 | |
| *** cference_ has quit IRC | 13:57 | |
| sdague | fungi: yeh, we're down to about 100 working nodes | 13:58 |
| fungi | sdague: looking to see if it's still rax having problems | 13:58 |
| mordred | fungi: you have a jeepyb change from ages ago with 2 +2s on it - about checking to see if a project uses github before using the pull request closer | 13:59 |
| mordred | fungi: any reasno I should avoid a +A? | 13:59 |
| fungi | mordred: not to my knowledge. we can hold it up as proof that changes proposed by our core reviewers don't get fast-tracked though! | 14:00 |
| *** dustins has quit IRC | 14:00 | |
| mordred | fungi: hah. first +2 was in december | 14:00 |
| *** amitgandhinz has joined #openstack-infra | 14:00 | |
| mordred | scuse me - september | 14:01 |
| *** sarob has quit IRC | 14:01 | |
| fungi | so only 7 months. that's not so bad ;) | 14:02 |
| *** dustins has joined #openstack-infra | 14:02 | |
| fungi | sdague: wow, that drop in building nodes is really something. i wonder if that was rax fixing itself | 14:03 |
| fungi | oh, nevermind. that was a graph glitch | 14:03 |
| openstackgerrit | Merged openstack-infra/jeepyb: Be more careful about when to close pull requests https://review.openstack.org/116414 | 14:04 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config: Configure stable branch ACLs for non-Oslo libs https://review.openstack.org/173892 | 14:05 |
| *** dizquierdo has joined #openstack-infra | 14:05 | |
| *** esker has joined #openstack-infra | 14:06 | |
| openstackgerrit | Merged openstack-infra/jeepyb: Replace 'javascript' method/parameternames https://review.openstack.org/127748 | 14:06 |
| *** sabeen1 has joined #openstack-infra | 14:06 | |
| openstackgerrit | Merged openstack-infra/jeepyb: Rename upload_rss and move non-swift-relevant parts into the main method https://review.openstack.org/127749 | 14:06 |
| *** bswartz has quit IRC | 14:06 | |
| dhellmann | fungi, clarkb, jeblair, mordred: ttx and I are working on client stable branches this morning, and we're going to need some help with gerrit acls. See https://review.openstack.org/173892 to start | 14:07 |
| openstackgerrit | Merged openstack-infra/jeepyb: Add global flag to control replication https://review.openstack.org/161786 | 14:07 |
| openstackgerrit | Merged openstack-infra/jeepyb: Add global flag to control replication https://review.openstack.org/161786 | 14:07 |
| *** Kennan has joined #openstack-infra | 14:07 | |
| *** Kennan2 has quit IRC | 14:08 | |
| fungi | dhellmann: ttx: cool--should we also be cleaning up the refs/heads/proposed/* sections in some of those for better consistency? also what is our proposed/* branch control plan these days? i'd love to discuss dropping that section from the all-projects global acl and leaving it to the individual repo acls | 14:09 |
| *** stevemar has joined #openstack-infra | 14:09 | |
| dhellmann | fungi: I don't think we're using the proposed branch namespace for clients, but I'd want to verify with ttx | 14:10 |
| dhellmann | and I'd be happy to clean up the files I'm already touching. same patch, or a second one? | 14:11 |
| *** zz_jgrimm is now known as jgrimm | 14:11 | |
| fungi | dhellmann: that was my thought too. let's do the same patch--this is a release management administrivia change for official projects, so making them consistent is very much on topic | 14:11 |
| dhellmann | fungi: wfm, I'll start updating | 14:11 |
| * mordred stops reviewing patch | 14:11 | |
| dhellmann | fungi: remove the "proposed" blocks entirely? | 14:12 |
| fungi | dhellmann: yep, unless ttx objects | 14:12 |
| fungi | i suspect they were just cargo-culted from the api server acls | 14:12 |
| *** markvoelker_ has joined #openstack-infra | 14:13 | |
| dhellmann | probably | 14:13 |
| fungi | sdague: yep, as i suspected rax-dfw is still in a bad way... 180 nodes (our entire quota there i think) is stuck in delete | 14:15 |
| fungi | sdague: for reference https://status.rackspace.com/index/viewincidents?group=21&start=1429070400 | 14:16 |
| fungi | seems to still be ongoing | 14:16 |
| *** markvoelker has quit IRC | 14:16 | |
| *** scheuran has quit IRC | 14:17 | |
| *** tonytan4ever has joined #openstack-infra | 14:17 | |
| *** bswartz has joined #openstack-infra | 14:17 | |
| ttx | fungi: you can drop proposed/* from libs ACLs | 14:18 |
| mtreinish | mordred: oh too bad, I was hoping you'd get to: https://review.openstack.org/#/c/164374/ | 14:18 |
| ttx | I'd rather keep it in all-projects since that's a way to say "if you create a proposed branch, releasemanagement will own it" | 14:18 |
| ttx | but we could certainly trickle that down if necessary | 14:19 |
| *** achanda has joined #openstack-infra | 14:20 | |
| fungi | ttx: well, part of the issue is that release management doesn't own it. there are projects overriding it in their acls to give control to their .*-milestone teams | 14:20 |
| *** sigmavirus24_awa is now known as sigmavirus24 | 14:20 | |
| mordred | fungi: awesome. so I'm guessing that their folks would probably be less interested in runnign manual delete cleanups for us if they're in an incident | 14:20 |
| *** ujuc_ has joined #openstack-infra | 14:21 | |
| *** ujuc_ has quit IRC | 14:21 | |
| fungi | ttx: for example http://git.openstack.org/cgit/openstack-infra/project-config/tree/gerrit/acls/openstack/nova.config | 14:21 |
| ttx | fungi: hmm, good point | 14:21 |
| ttx | I thought it was marked non-overfridable | 14:21 |
| mordred | johnthetubaguy: just on the off chance you're around and feel like ignoring your employer and helping us | 14:21 |
| fungi | ttx: nah, whatever's in the repo-specific acl for a particular ref pattern overrides the equivalent pattern from all-projects | 14:22 |
| *** dizquierdo has quit IRC | 14:22 | |
| *** marcusvrn1 has joined #openstack-infra | 14:22 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config: Configure stable branch ACLs for non-Oslo libs https://review.openstack.org/173892 | 14:22 |
| dhellmann | fungi, mordred, ttx: updated ^^ | 14:23 |
| fungi | ttx: looking at the nova acl, i think what they currently have is nova-core and nova-milestone granted control over refs/heads/proposed/* though that's worth confirming it's overriding in the direction i believe it's designed to | 14:23 |
| dhellmann | fungi: while you're reviewing my acl changes, I think I addressed your comments on the ones for oslo, too: https://review.openstack.org/#/c/173075/ | 14:24 |
| *** dkranz has joined #openstack-infra | 14:24 | |
| *** achanda has quit IRC | 14:24 | |
| mordred | johnthetubaguy: ^^ we have 180 nodes in deleting at rax | 14:24 |
| mordred | johnthetubaguy: (remember a while back when I told you you were going to become our go-to guy for rax issues? :) ) | 14:24 |
| *** marcusvrn has quit IRC | 14:24 | |
| fungi | dhellmann: thanks. checking both now | 14:24 |
| dhellmann | fungi: ty | 14:25 |
| fungi | ttx: anyway, i don't think we need to change up the refs/heads/proposed/* section in all-projects for now, but post kilo release we should revisit streamlining this | 14:25 |
| ttx | dhellmann: +1 | 14:26 |
| ttx | ack | 14:26 |
| ttx | fungi: ^ | 14:26 |
| dhellmann | fungi: ++, I'm happy to help with that | 14:27 |
| *** sarob has joined #openstack-infra | 14:27 | |
| fungi | ttx: dhellmann: main reason being any special ref patterns we make exclusive in the all-projects global acl means that much more complication and confusion for stackforge and loosely-affiliated official projects who happen to have branch names colliding with that | 14:28 |
| fungi | and also gerrit's acl inheritance rules are a little opaque, so the less we expose people to that the better in my opinion | 14:28 |
| johnthetubaguy | mordred: hey, I should try get some folks to look into things for you | 14:28 |
| dhellmann | fungi: that makes sense | 14:29 |
| openstackgerrit | Merged openstack-infra/shade: Change Ironic node lookups to support names https://review.openstack.org/171164 | 14:29 |
| johnthetubaguy | mordred: stuck in a meeting at this second, should be out soon | 14:29 |
| ttx | fungi: agreed | 14:29 |
| fungi | johnthetubaguy: is that meeting in dfw? are you sure it isn't just pending deletion since early yesterday? ;) | 14:29 |
| johnthetubaguy | fungi: I hurd some folks restarted things in DFW, I suspect thats related… | 14:30 |
| fungi | cool--here's hoping | 14:31 |
| *** armax has joined #openstack-infra | 14:31 | |
| *** dboik_ has joined #openstack-infra | 14:32 | |
| johnthetubaguy | fungi: i am snug at home in the UK sadly, I guess its DFW where the issue is right? | 14:32 |
| fungi | johnthetubaguy: yeah. it's the issue posted at status.rackspace.com | 14:33 |
| *** zz_dimtruck is now known as dimtruck | 14:33 | |
| dhellmann | fungi, ttx: I wonder if we should consider encouraging core teams to think of stable-maint as their responsibility, too, and just stop having separate stable teams. in my (limited) experience, we usually end up with much the same teams anyway (that's why Oslo decided to go the route we chose) | 14:33 |
| *** achanda has joined #openstack-infra | 14:34 | |
| ttx | dhellmann: so the trick there is... we tried that and it failed | 14:34 |
| ttx | dhellmann: we need to check that core people actually understand the stable branch policy | 14:35 |
| dhellmann | "if it's everyone's job, it's noone's job"? | 14:35 |
| ttx | a surprisingly little number of them do | 14:35 |
| dhellmann | ah, yeah | 14:35 |
| sdague | dhellmann: yeh, the thing is, very different people care about stable 'products' | 14:35 |
| fungi | dhellmann: i think part of the concern was that not enough of the core reviewers on some of the larger core teams paid attention to the branches changes were being proposed to, and didn't know (or want to know) the subtleties of the stable backport workflow and separate review criteria | 14:35 |
| *** dkranz has quit IRC | 14:35 | |
| ttx | I'm also all for splitting "core" as a single status | 14:35 |
| ttx | i.e. core reviewing != stable fitness judgment call | 14:36 |
| dhellmann | ok, those arguments make sense | 14:36 |
| ttx | so the groups can certainly overlap | 14:36 |
| ttx | but being the same is counterproductive | 14:36 |
| fungi | we could rename them after the branches over which they have approval. "master reviewers" totally doesn't sound like an elite group to me ;) | 14:36 |
| *** Ala has quit IRC | 14:36 | |
| *** emagana has joined #openstack-infra | 14:36 | |
| *** dboik has quit IRC | 14:36 | |
| ttx | fungi: "HEAD reviewers" maybe :) | 14:36 |
| *** markvoelker has joined #openstack-infra | 14:36 | |
| fungi | ttx wins! | 14:36 |
| ttx | "trunk reviewers" sounds dirty | 14:37 |
| sdague | heh | 14:37 |
| *** dizquierdo has joined #openstack-infra | 14:37 | |
| fungi | i get that terminology matters, but i think we're going to just end up playing renaming games forever because the truth is that any time you have a specific set of permissions delegated to a specific group of people and the membership of those groups is published, they will inevitably be viewed as "elite" groups by external parties. no amount of calling everyone janitors and drudges will fix that | 14:38 |
| *** markvoel_ has joined #openstack-infra | 14:38 | |
| *** russellb has quit IRC | 14:38 | |
| fungi | though power to the janitors and all that | 14:38 |
| *** hdd has quit IRC | 14:39 | |
| sdague | honestly, the naming part isn't important here. | 14:40 |
| *** markvoelker_ has quit IRC | 14:40 | |
| openstackgerrit | Merged openstack-infra/shade: Make cache key generator ignore cache argument https://review.openstack.org/173109 | 14:40 |
| sdague | dhellmann: also, I think that the ability/desire of a team owning their stable branch largely has to do with how big the influx is on master | 14:40 |
| *** markvoel_ has quit IRC | 14:41 | |
| fungi | splitting various roles and responsibilities is always an option. we do it all the time in a lot of places in openstack already | 14:41 |
| BobBall | If you're suggesting naming reviewers after the branch then surely 'stable reviewers; implies that the rest are unstable | 14:41 |
| dhellmann | yeah, I'm fine with leaving them split, I was just looking at that as a way to reduce the special casing | 14:41 |
| dhellmann | lol | 14:41 |
| *** russellb has joined #openstack-infra | 14:42 | |
| *** markvoelker has quit IRC | 14:42 | |
| *** marcusvrn has joined #openstack-infra | 14:45 | |
| *** sarob has quit IRC | 14:45 | |
| fungi | dhellmann: i think the big-tent scaling solution though is to not create more centralized groups. decentralize and let a thousand stars blossom, or something like that | 14:45 |
| *** achanda has quit IRC | 14:45 | |
| *** sks has quit IRC | 14:45 | |
| fungi | doesn't mean they can't be split roles to multiple groups, just should avoid making those centralized groups over multiple projects when avoidable | 14:45 |
| *** otter768 has joined #openstack-infra | 14:46 | |
| *** marcusvrn1 has quit IRC | 14:46 | |
| dhellmann | fungi: yeah, I'm trying to balance the self-service model with making sure stable-maint-core has permission to do things in repos in emergencies (or even just in cases like today) | 14:47 |
| *** marcusvrn1 has joined #openstack-infra | 14:47 | |
| mordred | so - since we can't encode that exactly in gerrit | 14:48 |
| mordred | what if we encode it as a unittest on things in the openstack dir of the acls dir? | 14:48 |
| *** dboik_ has quit IRC | 14:48 | |
| mordred | like, "everything in openstack/ must include X team as a last-resort override team regardless of that team's policy" | 14:48 |
| *** BharatK has joined #openstack-infra | 14:49 | |
| *** dizquierdo has quit IRC | 14:49 | |
| *** dboik has joined #openstack-infra | 14:49 | |
| *** marcusvrn has quit IRC | 14:49 | |
| mordred | that way people can define their own stable maint team, but we can have an uber team for dealing with things like today | 14:49 |
| fungi | mordred: we could probably add things to the current project-config-gerrit which does acl normalization checks | 14:49 |
| fungi | er, what i mean is, we already do acl normalization checks in the project-config-gerrit job, so that seems like it would fit there and probably not be much extra code to implement | 14:50 |
| mordred | yah | 14:50 |
| mordred | because there are times when a thing needs done and it's not a thing that needs 100 individual conversations | 14:51 |
| *** otter768 has quit IRC | 14:51 | |
| fungi | dhellmann: ttx: both 173075 and 173892 lgtm now | 14:51 |
| mordred | and those times are few and far between - like, maybe one per cycle | 14:51 |
| *** hdd has joined #openstack-infra | 14:52 | |
| openstackgerrit | Merged openstack-infra/shade: Invalidate volume list cache when creating https://review.openstack.org/172595 | 14:53 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/release-tools: update make_library_stable_branch.sh to set defaultbranch https://review.openstack.org/173919 | 14:53 |
| *** asselin has joined #openstack-infra | 14:53 | |
| *** annegentle has quit IRC | 14:54 | |
| *** patrickeast has joined #openstack-infra | 14:55 | |
| dhellmann | fungi: thanks | 14:55 |
| dhellmann | mordred: I like that. if there's a way to just add stable-maint-core in the global acls file as fungi suggests, that would do it, too | 14:56 |
| mordred | yup | 14:56 |
| jeblair | dhellmann: i finished about half of the change to do "file:" searching in gertty on the trip back from montreal | 14:56 |
| dhellmann | jeblair: nice! | 14:56 |
| mordred | look! it's a jeblair! | 14:56 |
| dhellmann | jeblair: I tried to sleep on my flight, so no coding for me :-/ | 14:57 |
| jeblair | dhellmann: (actually, i finished almost all of it, but i also think it needs to arrive with a fairly major schema change, otherwise the database has some major duplication, so i'm working on that now) | 14:58 |
| *** kmartin has joined #openstack-infra | 14:58 | |
| *** Daisy has quit IRC | 14:58 | |
| dhellmann | jeblair: typical developer. "It's done, I just need to make one more tweak..." | 14:58 |
| jeblair | dhellmann: i'm worried my database professor will find it and tell me it's not in the appropriate normal form | 14:59 |
| dhellmann | haha | 14:59 |
| dhellmann | jeblair: all the cool kids are using denormalized schemaless databases these days, didn't you know? | 14:59 |
| fungi | dhellmann: you mean filesystems? | 15:00 |
| fungi | (what's old is new again!) | 15:00 |
| *** rwsu has joined #openstack-infra | 15:00 | |
| dhellmann | fungi: you're giving away all the secrets that keep us ol' timers ahead of the curve | 15:01 |
| *** dizquierdo has joined #openstack-infra | 15:01 | |
| * fungi puts his data in files. it's revolutionary! | 15:02 | |
| jeblair | fungi: especially if the files are on spinning disk | 15:02 |
| jeblair | mordred: you use unified diff with gertty? | 15:02 |
| *** jkraj has joined #openstack-infra | 15:02 | |
| mordred | I was using it this morning, yes | 15:02 |
| rcarrillocruz | fungi: i notice the review manifest doesn't define mysql_user, i assume it uses some default | 15:03 |
| mordred | (still experimenting with how it works best for me) | 15:03 |
| rcarrillocruz | (hi btw) | 15:03 |
| rcarrillocruz | but can't figure what that is | 15:03 |
| rcarrillocruz | gerrit2 as well? | 15:03 |
| fungi | jeblair: i use our galaxy's rotation within the andromeda cluster to spin my storage these days. _really_ old-school | 15:04 |
| *** duran has quit IRC | 15:04 | |
| jeblair | mordred: i bet we could improve on what diff -u does. we could probably have it always display the top line number in the 2nd line on the screen (where i put the filenames). or even have it display the cursor line numbers as you move. | 15:04 |
| fungi | rcarrillocruz: you mean the openstack_project::review class? | 15:05 |
| jeblair | fungi: what do you think? is it worth trying to get rid of the line numbers on the side in unified mode, so things really fit in 80 cols? | 15:05 |
| rcarrillocruz | yeah that one | 15:05 |
| rcarrillocruz | fungi: ^ | 15:05 |
| mordred | rcarrillocruz: you want to look in templates/gerrit.config.erb | 15:05 |
| mordred | rcarrillocruz: in puppet-gerrit | 15:05 |
| rcarrillocruz | ahhh | 15:05 |
| rcarrillocruz | ok | 15:05 |
| rcarrillocruz | set in template | 15:05 |
| rcarrillocruz | thx | 15:05 |
| mordred | and yes - it's hardcoded because history | 15:05 |
| *** xyang1 has joined #openstack-infra | 15:05 | |
| fungi | jeblair: i would love that, but it already manages to wrap gracefully on 70-something column lines and not wrap when it doesn't need to, so it's not terrible | 15:06 |
| rcarrillocruz | :-) | 15:06 |
| rcarrillocruz | seeing so many hardcoded things lately in manifests, yeah | 15:06 |
| openstackgerrit | Doug Hellmann proposed openstack/requirements: Remove OpenStack library caps for liberty https://review.openstack.org/173924 | 15:06 |
| *** thedodd has joined #openstack-infra | 15:07 | |
| *** belmoreira has quit IRC | 15:08 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/project-config: Remove stable-compat-jobs for projects that have stable branchs https://review.openstack.org/173478 | 15:08 |
| DuncanT | Does anybody know Daniel Berrange's IRC nick, please? | 15:09 |
| fungi | rcarrillocruz: well, we do try to hard-code things that shouldn't need changing and avoid unnecessary proliferation of class parameters | 15:09 |
| fungi | DuncanT: that's the thing... he doesn't hang out in irc | 15:09 |
| DuncanT | fungi :-( Thanks | 15:09 |
| DuncanT | Anybody know gerrymander reasonably well? | 15:09 |
| fungi | DuncanT: he's said in the past that if it didn't happen in e-mail then it didn't happen as far as he's concerned | 15:09 |
| *** flaviosr has joined #openstack-infra | 15:10 | |
| *** soren has quit IRC | 15:10 | |
| DuncanT | fungi: I'll email him if needed, thanks | 15:10 |
| fungi | DuncanT: i think harlowja_away may have done some things with gerrymander... now i don't recall for sure | 15:10 |
| DuncanT | He's got a commit in it, good point, cheers | 15:11 |
| *** markvoelker has joined #openstack-infra | 15:12 | |
| rcarrillocruz | speaking of class parameters hiera and things | 15:13 |
| openstackgerrit | Ryan Carey proposed openstack-infra/jenkins-job-builder: Artifactory plugin https://review.openstack.org/173932 | 15:13 |
| mordred | fungi: funny, I consider that if it wasn't mentioned in IRC it didn't happen | 15:13 |
| mordred | rcarrillocruz: yah? | 15:13 |
| rcarrillocruz | what you think about using hiera as an ENC? | 15:13 |
| rcarrillocruz | site.pp is becoming hard to manage | 15:13 |
| rcarrillocruz | why not putting on hiera what service classes belong to each node | 15:13 |
| rcarrillocruz | that will make code cleaner | 15:13 |
| rcarrillocruz | and allow for a better experience on deploying new infras | 15:13 |
| openstackgerrit | Ryan Carey proposed openstack-infra/jenkins-job-builder: Add support for Artifactory plugin https://review.openstack.org/173932 | 15:14 |
| rcarrillocruz | and testing infra itself | 15:14 |
| mordred | rcarrillocruz: I'm not sure I can fully visualize what that looks like | 15:14 |
| fungi | rcarrillocruz: i think the plan is to get rid of the monolithic site.pp in system-config anyway | 15:14 |
| rcarrillocruz | fungi: and what's the alternative | 15:14 |
| mordred | but yeah - once we've moved to puppet apply, I think there are several thigns that can get more flexible | 15:14 |
| fungi | rcarrillocruz: that site.pp is basically an artifact of using a puppetmaster | 15:14 |
| rcarrillocruz | mordred: https://docs.puppetlabs.com/hiera/1/puppet.html#assigning-classes-to-nodes-with-hiera-hierainclude | 15:14 |
| mordred | rcarrillocruz: one we're doing puppet apply, we could have a top-level manifest per type if we wanted | 15:14 |
| fungi | so, yeah, moving to masterless puppet makes it unnecessary | 15:15 |
| mordred | rcarrillocruz: that is drastically more confusing to me | 15:15 |
| jeblair | rcarrillocruz: also, i believe nibalizer was looking into moving more information into hiera after http://specs.openstack.org/openstack-infra/infra-specs/specs/public_hiera.html is implemented | 15:15 |
| mordred | but public hiera is a great idea | 15:15 |
| rcarrillocruz | yes, we've been talking about an in-tree hiera in gozer for a while | 15:16 |
| rcarrillocruz | one hiera for public conf | 15:16 |
| rcarrillocruz | another one for secrets | 15:16 |
| mordred | yes. that's right | 15:16 |
| mordred | I'd rather get that finished before we start thinking about larger structural changes like the ENC thing you mentioend | 15:16 |
| *** markvoelker has quit IRC | 15:16 | |
| rcarrillocruz | agreed | 15:16 |
| mordred | however, I do want to say -that if I have to understand a magical hiera directory structure to be able to infer what classes are on a node, I'm going to be unhappy | 15:16 |
| *** _nadya_ has quit IRC | 15:16 | |
| rcarrillocruz | hiera is a good thing, and it improves dramatically downstream consumption of infra | 15:17 |
| mordred | ++ | 15:17 |
| rcarrillocruz | right now automating an infra deploy is nightmarish, because of the hardocded things around and highly coupled manifests | 15:17 |
| nibalizer | yea i havent worked on an in tree hiera dir in a while | 15:17 |
| nibalizer | we shoukd return to tgat | 15:17 |
| rcarrillocruz | nibalizer: well, we'll meet next week | 15:18 |
| nibalizer | mordred: did i see you grump a bit about fabric yesterday? | 15:18 |
| rcarrillocruz | providing gozer shares a lot of things with infra, we can discuss and get back to it on that spec | 15:18 |
| *** bswartz has quit IRC | 15:19 | |
| *** david-lyle has joined #openstack-infra | 15:19 | |
| *** david-lyle_ has joined #openstack-infra | 15:19 | |
| *** david-lyle_ has quit IRC | 15:19 | |
| yolanda | from my side as well, i'm finding very difficult to be on sync with upstream due to that hardcoded bits, generating a huge drift | 15:20 |
| *** hdd has quit IRC | 15:20 | |
| *** Krinkle|detached is now known as Krinkle | 15:20 | |
| yolanda | so improving isolation, moving all possible logic to modules, and add parameters (even if upstream has its sane defaults to avoid mess), is the path to follow, from my opinion | 15:21 |
| jeblair | yolanda: that's what we're doing, right? | 15:21 |
| yolanda | jeblair, yes, but honestly is not easy sometimes | 15:21 |
| jeblair | nope it's not. it's difficult and slow | 15:21 |
| fungi | we have a large and complex set of systems and lots to do | 15:22 |
| yolanda | i'm filing changes and changes for that, as well as Ricardo and other ones | 15:22 |
| jeblair | yes, and we're reviewing them and merging them rather quickly since this is one of our priorities | 15:22 |
| openstackgerrit | Fabien Boucher proposed openstack-infra/system-config: Move server class call outside of jenkins*.pp class https://review.openstack.org/170487 | 15:23 |
| yolanda | jeblair, but sometimes is not easy as we have that policy that system-config is not reconsumable. Even if it's not, that concept of avoiding hardcoded things, moving all logic to puppet modules, should be on top | 15:23 |
| dhellmann | sdague: can you take a look at this change to uncap our libs in master? https://review.openstack.org/#/c/173924/ | 15:23 |
| rcarrillocruz | jeblair: i get sometimes feedback that system-config is not downstream consumable when i want to refactor stuff up. Un-hardcoding things doesn't mean i want to use system-config to deploy my own infra (we will always need our very own system-config branch and repo), it helps to test things in infra when we need to fix bugs that imply several components or automation workflows | 15:23 |
| openstackgerrit | Julia Kreger proposed openstack-infra/shade: Wrap ironicclient methods that leak objects https://review.openstack.org/173567 | 15:23 |
| openstackgerrit | Julia Kreger proposed openstack-infra/shade: Add patch_machine method and operator unit test substrate https://review.openstack.org/171165 | 15:24 |
| *** dizquierdo has quit IRC | 15:24 | |
| fungi | rcarrillocruz: yolanda: well, the answer to that is not to make system-config more complex so that downstream can consume it. it's to split system-config up and/or move more of what's there down into the individual service modules so that we can maintain a better defined api around those things | 15:24 |
| yolanda | fungi, that's what were are trying in the background | 15:25 |
| jeblair | rcarrillocruz: right. we're changing this system one step at a time. first split out puppet modules. make their api's better. then start working on building those up into system units. | 15:25 |
| rcarrillocruz | un-hardcoding things is not making them more complex, is making them more flexible | 15:25 |
| jeblair | rcarrillocruz: the openstackci module is an example of that: http://specs.openstack.org/openstack-infra/infra-specs/specs/openstackci.html | 15:25 |
| openstackgerrit | Fabien Boucher proposed openstack-infra/system-config: Move server class call outside of jenkins*.pp class https://review.openstack.org/170487 | 15:25 |
| yolanda | at the end i don't care if parameters come from site.pp, hiera or whatever, but having that parameters exposed is a necessary thing | 15:25 |
| johnthetubaguy | fungi: mordred: have you got some instance uuids you are having a problem with right now? | 15:25 |
| rcarrillocruz | interesting jeblair | 15:26 |
| rcarrillocruz | so that's some sort of a minimal reference design for Openstack CI | 15:26 |
| rcarrillocruz | ? | 15:26 |
| fungi | rcarrillocruz: more flexible is basically _always_ more complex. choices add complexity. taking choice away reduces complexity | 15:26 |
| jeblair | rcarrillocruz: so let's focus on implementing the parts of the plan we've already discussed and approved, and then take the next step from there. | 15:26 |
| *** jcoufal_ has joined #openstack-infra | 15:26 | |
| yolanda | how is status on that spec btw? i wouldn't mind to help | 15:26 |
| yolanda | we are on the same boat | 15:27 |
| rcarrillocruz | indeed | 15:27 |
| fungi | we acknowledge that to make these things more reusable downstream they need to be more flexible, but the complexity that brings needs to go into places where we can corral it and wrap api stability expectations around it, rather than just making system-config more and more of a rat's nest | 15:27 |
| openstackgerrit | Sergey Lukjanov proposed openstack-infra/project-config: Add all Sahara images build to check pipeline as non-voting https://review.openstack.org/173939 | 15:27 |
| rcarrillocruz | i don't intend to use system-config for a prod CI | 15:28 |
| fungi | johnthetubaguy: sure, just a sec. getting an example now | 15:28 |
| *** jerryz has quit IRC | 15:28 | |
| rcarrillocruz | i'd be happy if i could just put my args in site.pp/hiera | 15:28 |
| rcarrillocruz | fire puppet | 15:28 |
| rcarrillocruz | and more or less have something functional | 15:28 |
| johnthetubaguy | fungi: thanks | 15:28 |
| *** stevemar has quit IRC | 15:28 | |
| jeblair | yes, the efforts already in progress are all about moving things into appropriate API layers and giving us the flexibility to reorganize things | 15:28 |
| jroll | fungi: chance that these are stuck from the thing I mentioned yesterday? | 15:28 |
| jeblair | rcarrillocruz, yolanda: so please do pitch in on the public hiera and openstackci efforts | 15:29 |
| *** tonytan4ever has quit IRC | 15:29 | |
| fungi | jroll: my assumption is that they definitely are. the rackspace status page claims this incident is still ongoing | 15:29 |
| *** jcoufal has quit IRC | 15:29 | |
| rcarrillocruz | we will | 15:29 |
| jroll | fungi: oh, interesting, did not realize it was still a thing | 15:29 |
| yolanda | asselin, count with us to help, can we have a talk and see how can we collaborate? | 15:29 |
| openstackgerrit | Ryan Carey proposed openstack-infra/jenkins-job-builder: fix https://review.openstack.org/173941 | 15:30 |
| rcarrillocruz | so asselin | 15:30 |
| rcarrillocruz | if i'm not mistaken , we work for the same company | 15:30 |
| rcarrillocruz | and you are EMEA based | 15:30 |
| rcarrillocruz | ? | 15:30 |
| fungi | jroll: https://status.rackspace.com/index/viewincidents?group=21&start=1429070400 | 15:30 |
| asselin | yolanda, in a meeting-4....with catch up after | 15:30 |
| yolanda | k, thx | 15:30 |
| rcarrillocruz | k, will ping you later then | 15:30 |
| openstackgerrit | Merged openstack-infra/reviewstats: Make local cache more permanent. https://review.openstack.org/172550 | 15:31 |
| fungi | jroll: so i guess it got resolved according to that? | 15:31 |
| jroll | fungi: yeah, looks resolved | 15:31 |
| *** harlowja_at_home has joined #openstack-infra | 15:31 | |
| jroll | fungi: but maybe some stuff got stuck during that time that isn't getting unstuck | 15:31 |
| *** tonytan4ever has joined #openstack-infra | 15:32 | |
| *** russellb has quit IRC | 15:32 | |
| *** dangers_away is now known as dangers | 15:33 | |
| notmyname | ttx: now that we've got all the ec stuff onto master, there's an opportunity for some branch cleanup. feature/ec_review has been merged to master and can be deleted. feature/ec IMO should be tagged for history and then deleted | 15:34 |
| notmyname | ttx: (for swift) | 15:34 |
| openstackgerrit | Ryan Carey proposed openstack-infra/jenkins-job-builder: Artifactory plugin https://review.openstack.org/173932 | 15:34 |
| *** sdake has joined #openstack-infra | 15:35 | |
| openstackgerrit | Andrey Pavlov proposed openstack-infra/project-config: upgrade ec2api jobs https://review.openstack.org/173945 | 15:35 |
| yolanda | jeblair, will you have a sec to talk about nodepool queues today? | 15:35 |
| fungi | johnthetubaguy: jroll: here's 111 uuids for instances we've been trying to delete for ~20 hours http://paste.openstack.org/show/203997/ | 15:35 |
| jeblair | yolanda: sure, but i haven't reviewed your patch yet. do i need to do that first? | 15:35 |
| yolanda | yes, you should | 15:35 |
| johnthetubaguy | fungi: cool, thanks, I will pass that on | 15:35 |
| jeblair | yolanda: i'll ping you when i do then | 15:36 |
| yolanda | it's following the 2 queues approach , one for create/delete, but i want to know your feedback | 15:36 |
| ttx | notmyname: ok. Note that while I can create feature branches, I can't delete them (don't ask). So you should ask for infra help on that | 15:36 |
| notmyname | ttx: ok. that's why I asked in here. I wasn't sure who does that | 15:36 |
| *** russellb has joined #openstack-infra | 15:36 | |
| notmyname | jeblair: ^^ | 15:36 |
| notmyname | jeblair: now that we've got all the ec stuff onto master, there's an opportunity for some branch cleanup. feature/ec_review has been merged to master and can be deleted. feature/ec IMO should be tagged for history and then deleted | 15:37 |
| *** dboik_ has joined #openstack-infra | 15:37 | |
| clarkb | notmyname might be best for you to tag feature/ec first then we can remove the branches (easier to tage the correct sha1) that way | 15:38 |
| *** Sukhdev has joined #openstack-infra | 15:38 | |
| fungi | johnthetubaguy: jroll: spot-checking that list against what nova list reports, some of those uuids don't seem to correspond to an existing instance at all, while others are showing in an active state. i'll try to force nodepool to clean up what it can and see what lingers | 15:38 |
| notmyname | clarkb: I wasn't aware that I could push tags to gerrit for swift (just swift client) | 15:39 |
| clarkb | oh right | 15:39 |
| *** sdake_ has joined #openstack-infra | 15:39 | |
| clarkb | notmyname what name should the tag be theb? | 15:39 |
| notmyname | clarkb: help me, clarkb, you're my only hope ;-) | 15:39 |
| clarkb | feature-erasure-coding? | 15:39 |
| notmyname | clarkb: erasure_code_dev_history | 15:39 |
| notmyname | ? | 15:39 |
| *** sarob has joined #openstack-infra | 15:39 | |
| *** dboik has quit IRC | 15:40 | |
| *** samueldmq has quit IRC | 15:40 | |
| *** plaurin has joined #openstack-infra | 15:42 | |
| openstackgerrit | Ryan Carey proposed openstack-infra/jenkins-job-builder: Artifactory plugin https://review.openstack.org/173932 | 15:42 |
| plaurin | Hello, in a jenkin job, how to I configure a nodepool to make devstack-gate build a multi node environnement? | 15:42 |
| *** sdake has quit IRC | 15:42 | |
| clarkb | I initially thought feature-ec-eol to be like stable tags when they eol but I think that may confuse people into thinking the feature was end of lifed | 15:42 |
| clarkb | notmyname: erasure_code_dev_history wfm | 15:43 |
| *** spredzy is now known as spredzy|afk | 15:43 | |
| notmyname | clarkb: ok | 15:43 |
| clarkb | plaurin if you grep for multinode in openstack-infra/project-config you will see examplea | 15:43 |
| clarkb | plaurin what are you trying to set up? | 15:44 |
| *** marcusvrn has joined #openstack-infra | 15:44 | |
| plaurin | a netvirt "ping" test using robot framework, I am currently working in the opendaylight jenkins sandbox. I am able to run devstack gate, but I want to add two jcloud instances and make devstack-gate build compute nodes in them | 15:45 |
| *** amotoki has joined #openstack-infra | 15:45 | |
| fungi | hrm, nodepool is basically our replacement for the jenkins jclouds plug-in. which are you using? nodepool or jclouds? | 15:46 |
| *** marcusvrn1 has quit IRC | 15:46 | |
| clarkb | fungi I think this may be third party ci | 15:46 |
| fungi | sounds like it to me, yes | 15:46 |
| clarkb | but yes we dont use jclouds for multinode envs, we assume nodepool | 15:47 |
| plaurin | jclouds, but I don't mind if it uses nodepool | 15:47 |
| *** mbacchi1 has joined #openstack-infra | 15:47 | |
| *** flaviosr has quit IRC | 15:47 | |
| *** dboik_ has quit IRC | 15:48 | |
| ttx | fungi, jeblair: we have an issue with our stable/kilo testing. It doesn't seem to fall back to proposed/kilo when no stable/kilo is available | 15:48 |
| *** dboik has joined #openstack-infra | 15:49 | |
| ttx | http://logs.openstack.org/15/173815/1/check/check-tempest-dsvm-full/0636154/logs/devstack-gate-setup-workspace-new.txt.gz | 15:49 |
| ttx | That seems to point to stable/kilo requirements changes testing against trove master, according to sdague | 15:49 |
| *** thinrichs has joined #openstack-infra | 15:50 | |
| plaurin | how is the multinode triggered? | 15:51 |
| *** thinrichs has left #openstack-infra | 15:51 | |
| openstackgerrit | Merged openstack-infra/system-config: Create a list to coordinate internships https://review.openstack.org/167458 | 15:51 |
| *** garyk has joined #openstack-infra | 15:51 | |
| clarkb | plaurin zuul triggers the job for us | 15:51 |
| fungi | ttx: that may need a patch to devstack-gate. checking now | 15:52 |
| jeblair | fungi, ttx: er.... | 15:52 |
| ttx | fungi, jeblair: so it is likely that the proposed/kilo testing falls back to stable/kilo when no proposed/kilo is around, but not the other way around | 15:52 |
| jeblair | fungi, ttx: can we just create a proposed/kilo branch? | 15:52 |
| ttx | for requirements ? | 15:52 |
| jeblair | er, sorry, stable/kilo for requirements | 15:52 |
| jeblair | yeah | 15:52 |
| ttx | stable/kilo for requirements is what we have | 15:53 |
| ttx | I think someone said that it wouldn't work if we cut proposed/kilo. Wouldn't sync to stable/kilo already-cut branches | 15:53 |
| jeblair | oh, sorry, so we'd need stable/kilo trove | 15:53 |
| *** cdent has joined #openstack-infra | 15:53 | |
| *** annegentle has joined #openstack-infra | 15:54 | |
| ttx | So currently we have stable/kilo branches for libraries. proposed/kilo branches for all RCs. stable/kilo branch for requirements. | 15:54 |
| ttx | proposed/kilo branch testing seems to work fine and uses stable/kilo stuff where available | 15:55 |
| *** thingee has joined #openstack-infra | 15:55 | |
| ttx | stable/kilo branch testing on the other hand does not seem to use propsoed/kilo stuff if no stable/kilo is available ? | 15:55 |
| fungi | jeblair: ttx: confirmed we don't have kilo branches handled in devstack-gate yet. i can add them unless anyone knows of a change in flight to do that already | 15:56 |
| *** sarob has quit IRC | 15:56 | |
| *** deepakcs has joined #openstack-infra | 15:56 | |
| ttx | fungi: so this has nothing to do with proposed/stable, but more to do with kilo ? | 15:57 |
| jeblair | fungi: what do you mean? i don't think we should have to add anything to d-g. | 15:57 |
| *** mrmartin has joined #openstack-infra | 15:57 | |
| ttx | fungi: no such change that I know of | 15:57 |
| fungi | jeblair: all the branch logic for e.g. grenade ordering? | 15:57 |
| *** MarkAtwood has joined #openstack-infra | 15:57 | |
| *** FallenPegasus has joined #openstack-infra | 15:57 | |
| fungi | is that stuff determined automagically outside of devstack-gate now? | 15:58 |
| fungi | it looks like features.yaml has been updated for kilo at least | 15:58 |
| jeblair | fungi: yes, grenade needs some help | 15:58 |
| jeblair | ttx: i'm uncertain that there's any kind of stable<->proposed fallback logic in either direction. | 15:58 |
| ttx | jeblair: there certainly was in previous releases :) | 15:59 |
| *** bswartz has joined #openstack-infra | 15:59 | |
| *** sdake has joined #openstack-infra | 15:59 | |
| *** tqtran has joined #openstack-infra | 15:59 | |
| fungi | depends on where the bug is. if this is just a problem with requirements rewriting (i haven't looked at the logs yet) then it may be something we added between juno and now | 16:00 |
| *** sarob has joined #openstack-infra | 16:00 | |
| * plaurin looking for multinode examples in openstack jenkins configs | 16:00 | |
| *** unicell1 has joined #openstack-infra | 16:00 | |
| fungi | oh, hrm, no devstack branch yet? | 16:00 |
| krtaylor | asselin, sorry we ran out of time | 16:01 |
| *** unicell has quit IRC | 16:01 | |
| ttx | fungi: apparently not | 16:01 |
| fungi | ttx: jeblair: dtroyer: sdague: mtreinish: that error looks like we need a stable/kilo branch in the openstack-dev/devstack repo | 16:01 |
| sdague | fungi: normally we don't cut that until release | 16:02 |
| jeblair | fungi: can you link to the error? | 16:02 |
| sdague | it's not typically needed | 16:02 |
| asselin | krtaylor, no problem, hope that was helpful | 16:02 |
| ttx | sdague: makes sense to have it now that there are */kilo branches everywhere though | 16:02 |
| fungi | ttx: jeblair: dtroyer: sdague: mtreinish: my bad, thrown off by benign error messages in the log. still looking | 16:03 |
| *** sdake_ has quit IRC | 16:03 | |
| *** zz_ja is now known as zz_zz_ja | 16:03 | |
| *** derekh has quit IRC | 16:03 | |
| *** tqtran_ has joined #openstack-infra | 16:03 | |
| plaurin | fungi: I think DEVSTACK_GATE_TOPOLOGY is what I am looking for | 16:03 |
| *** emagana has quit IRC | 16:03 | |
| clarkb | plaurin yes | 16:03 |
| sdague | ttx: well, except it means a lot more backporting. We delay the branch for a reason to fix up last bits before the release | 16:03 |
| *** jistr has quit IRC | 16:03 | |
| jeblair | ttx, sdague: i'm having trouble understanding exactly what the error is here. | 16:03 |
| ttx | sdague: ack | 16:03 |
| jeblair | ttx, sdague: can somene point me to the thing they think is wrong? | 16:03 |
| sdague | jeblair: propose a change to stable/kilo requirements | 16:03 |
| sdague | it is testing against master for the projects | 16:04 |
| sdague | not against proposed/kilo | 16:04 |
| *** jistr has joined #openstack-infra | 16:04 | |
| sdague | http://logs.openstack.org/15/173815/1/check/check-tempest-dsvm-full/0636154/logs/devstack-gate-setup-workspace-new.txt.gz | 16:04 |
| jeblair | sdague: i agree that is expected behavior for the current code. why is there a stable/kilo for requirements instead of a proposed/kilo? | 16:05 |
| sdague | http://logs.openstack.org/15/173815/1/check/check-tempest-dsvm-full/0636154/logs/devstack-gate-setup-workspace-new.txt.gz#_2015-04-15_14_01_27_493 | 16:05 |
| fungi | ttx: sdague: jeblair: are you sure? http://logs.openstack.org/15/173815/1/check/check-tempest-dsvm-full/0636154/logs/devstack-gate-setup-workspace-new.txt.gz#_2015-04-15_14_00_09_662 | 16:05 |
| openstackgerrit | Merged openstack-infra/devstack-gate: Set standard "swappiness" https://review.openstack.org/173221 | 16:05 |
| *** jlanoux has quit IRC | 16:05 | |
| sdague | fungi: the clients have stable/kilo branches | 16:06 |
| fungi | "Switched to a new branch 'stable/kilo'" (for openstack/python-novaclient) | 16:06 |
| sdague | yeh, see my link | 16:06 |
| sdague | nova is running off liberty | 16:06 |
| sdague | jeblair: right, that might be the fix | 16:06 |
| jeblair | sdague, ttx, fungi: so the new thing here is that we have cut stable branches for clients, and we have done so without stable branches of other things. | 16:06 |
| fungi | oh, got it, so it's testing stable/kilo requirements changes against stable/kilo of anything that has that branch or master of anything which doesn't (even if it has a proposed/kilo branch) | 16:07 |
| ttx | jebayes | 16:07 |
| sdague | I'm just saying the current branch structure doesn't do the right thing | 16:07 |
| ttx | jeblair: yes | 16:07 |
| *** luqas has quit IRC | 16:07 | |
| ttx | jeblair: I'm fine calling it proposed/kilo requirements instead if that would still get the library tested correctly | 16:07 |
| *** marcusvrn1 has joined #openstack-infra | 16:07 | |
| sdague | ttx: you're also going to have to do that for client libraries, because otherwise it will fail the other way, right? | 16:07 |
| ttx | depends on where and in what direction the stable <-> proposed fallback is done | 16:07 |
| ttx | because there is one | 16:08 |
| ttx | there was always one | 16:08 |
| fungi | wondering if we've got a logic error in test-matrix.normalize_branch() | 16:08 |
| sdague | fungi: could be | 16:08 |
| jeblair | ttx, sdague: we have one level of fallback support in branch checkouts, so we can do either stable/foo->master or proposed/foo->master, but we have no support for stable/foo->proposed/foo->master. | 16:08 |
| jeblair | test-matrix does not control this. | 16:08 |
| sdague | jeblair: yeh, that's what I thought | 16:08 |
| ttx | ah | 16:09 |
| ttx | weird that it worked before then | 16:09 |
| *** jistr has quit IRC | 16:09 | |
| jeblair | we had not imagined that we would have all three in play at a single time; we previously only either had proposed or stable in use, but not both at once | 16:09 |
| *** marcusvrn has quit IRC | 16:09 | |
| fungi | okay, so changes proposed to stable/X branches of projects won't be tested with proposed/X branches of other projects, but the reverse is true | 16:09 |
| *** emagana has joined #openstack-infra | 16:10 | |
| ttx | jeblair: ack | 16:10 |
| *** carl_baldwin has joined #openstack-infra | 16:10 | |
| jeblair | fungi: i don't think the reverse is true... | 16:10 |
| jeblair | fungi: i don't think we'll end up with stable/foo and proposed/foo branches of different projects in the same test in any situation | 16:10 |
| ttx | fungi: if you're right, creating proposed/kilo requirements should solve it | 16:10 |
| sdague | ttx: during this window of release time we actually rely on being lucky a lot and that patch rate is slow | 16:10 |
| *** harlowja_at_home has quit IRC | 16:10 | |
| ttx | (which is what we might have done in juno) | 16:10 |
| *** carl_baldwin has quit IRC | 16:11 | |
| jeblair | sdague: and that "*/foo" and master are still mostly compatible | 16:11 |
| fungi | which does also complicate the suggestions we'd been making to the driver/plugin maintainers to just use stable branches when the servers they're integrating with are in rc on proposed branches | 16:11 |
| *** carl_baldwin has joined #openstack-infra | 16:11 | |
| anteaya | it would be nice if what ever suggestions we make to driver/plugin maintainers was something they could rely on for a bit | 16:12 |
| anteaya | whatever those suggestions end up being | 16:12 |
| fungi | sure. first we have to sort out what we're suggesting to ourselves i think | 16:12 |
| anteaya | yes, that is what it sounds like to me | 16:12 |
| *** markvoelker has joined #openstack-infra | 16:13 | |
| *** koolhead17 has quit IRC | 16:13 | |
| anteaya | is an inchannel conversation sufficient for that? | 16:13 |
| *** dosaboy has quit IRC | 16:13 | |
| *** unicell1 has quit IRC | 16:14 | |
| *** markus_z has quit IRC | 16:14 | |
| ttx | So it looks like we have 3 options | 16:14 |
| *** dosaboy has joined #openstack-infra | 16:14 | |
| fungi | so... if we switch the requirements branch from stable/kilo to proposed/kilo we're going to get proposed/kilo branches of projects that have them tested with proposed/kilo requirements and master of projects that already have stable/kilo branches. conversely, changes to stable/kilo branches of the projects that have those will get tested with master requirements and master branches of projects which | 16:14 |
| fungi | are part of the integrated release rc model | 16:14 |
| *** jistr has joined #openstack-infra | 16:15 | |
| * dhellmann waits to hear ttx's 3 options | 16:15 | |
| ttx | still processing what fungi just said | 16:15 |
| *** sergsh has quit IRC | 16:16 | |
| ttx | might be only two options | 16:16 |
| ttx | option 1 - switch to using stable/kilo everywhere | 16:16 |
| jeblair | fungi: i believe that is correct | 16:16 |
| dhellmann | 1. Take up goat herding. | 16:16 |
| dhellmann | 2. ? | 16:16 |
| fungi | proposed/kilo changes get tested with kilo requirements and liberty clients/libs. stable/kilo changes to clients/libs get tested with liberty requirements and api servers | 16:16 |
| clarkb | dhellmann make cheese | 16:16 |
| dhellmann | clarkb: ++ | 16:16 |
| *** annegentle has quit IRC | 16:16 | |
| ttx | option 2 - fix tooling so that it just ignores the * in */kilo | 16:16 |
| dhellmann | ttx: yeah, I'm strongly in favor of your option 1 | 16:17 |
| ttx | My original option 3 was to cut a proposed/kilo req branch, but apparently that won't solve it if I read fungi correctly | 16:17 |
| zaro | morning | 16:17 |
| anteaya | morning zaro | 16:17 |
| jeblair | ttx, fungi, sdague: if we are going to have stable and proposed branches live at the same time, i think perhaps we do need both stable and proposed branches on the requirements repo. | 16:17 |
| dhellmann | ttx: that is also what I understood jeblair and fungi to be saying | 16:17 |
| jeblair | ttx: ^ i think that's your option 3 | 16:17 |
| *** ihrachyshka has quit IRC | 16:17 | |
| *** jistr has quit IRC | 16:17 | |
| ttx | jeblair: oh, havbe /both/ | 16:18 |
| *** Swami has joined #openstack-infra | 16:18 | |
| clarkb | jeblair how do you keep them synced? | 16:18 |
| jeblair | (i had typed it in but was reading fungi's reply; the prose doesn't make sense now, forgive me) | 16:18 |
| fungi | i've been in favor of option 1 since we went through this about a year ago, and then six months ago. the problem with it is that we'll need to make control group changes if we want different people controlling that branch during the rc period than after release | 16:18 |
| ttx | OK, let's rule out option 2 as way too risky and painful | 16:18 |
| jeblair | clarkb: hard work. | 16:18 |
| dhellmann | yeah, syncing sounds like it would be a real pain | 16:18 |
| clarkb | why not just do stable -> proposed -> master fallback | 16:18 |
| *** dboik_ has joined #openstack-infra | 16:18 | |
| ttx | The historical two issues with option 1 are: | 16:18 |
| anteaya | clarkb: I believe gerrit doesn't allow that option | 16:18 |
| clarkb | essentially the */kilo ignores * | 16:18 |
| dhellmann | fungi: I think having to change acls is ok, if we think we actually have to change acls | 16:19 |
| jeblair | so i just about have my head wrapped around the tech here, but i don't actually know why we have stable client branches now? | 16:19 |
| ttx | clarkb: that's option 2 | 16:19 |
| clarkb | ttx right | 16:19 |
| anteaya | clarkb: if I read jeblair's notes from above correctly | 16:19 |
| ttx | clarkb: I fear that would be a bit complex to track everywhere | 16:19 |
| *** markvoelker has quit IRC | 16:19 | |
| dhellmann | jeblair: because we're capping requirements in stable branch test environments now | 16:19 |
| clarkb | jeblair because we use clients in servers and backward compat for that is hard | 16:19 |
| ttx | so 1a- we need different acls for stable/kilo and stable/* (* not being kilo) | 16:19 |
| *** tqtran_ has quit IRC | 16:19 | |
| fungi | i guess the question is "why now rather than on release day?" | 16:20 |
| jeblair | fungi: yeah | 16:20 |
| dhellmann | ah | 16:20 |
| ttx | 1b- we convey the wrong meaning (the branch is not stable yet) -- we can survive that one | 16:20 |
| clarkb | jeblair so capping reqs with mappings to releases for the clients | 16:20 |
| dhellmann | well, we need to get the caps in place before we release the other projects | 16:20 |
| ttx | 1c- release tooling needs to change | 16:20 |
| fungi | basically doing it on release day means we're telling clients to freeze development between now and then, i guess | 16:20 |
| dhellmann | yeah, and we're already getting some push back because the oslo requirements in master are capped still, so I'd like to undo that | 16:21 |
| fungi | because client/library master branches will need to remain compatible with anything that's using proposed/kilo until kilo release day | 16:21 |
| jeblair | i sort of imagined capping requirements would happen _after_ release and the branch was cut. | 16:21 |
| *** dkranz has joined #openstack-infra | 16:21 | |
| ttx | jeblair: unfortiunately some people want to release liberty libs before kilo final release | 16:21 |
| *** dboik has quit IRC | 16:22 | |
| ttx | since liberty is "open" you know | 16:22 |
| ttx | so we can't freeze them | 16:22 |
| ttx | anyway | 16:22 |
| asselin | yolanda, rcarrillocruz hi | 16:22 |
| fungi | this may also just be a case of "sorry, that's too complicated for us to support, don't do that, wait your turn" | 16:22 |
| dhellmann | fungi: that's fair | 16:22 |
| ttx | Looks like the simplest is to just use stable/kilo everywhere | 16:22 |
| *** emagana has quit IRC | 16:22 | |
| asselin | yolanda, rcarrillocruz I'm in california. There are tasks in storyboard to track the effort: https://storyboard.openstack.org/#!/story/2000101 | 16:23 |
| ttx | Does everyone agree that would solve the current issue and future ones ? | 16:23 |
| dhellmann | ttx: what do we lose by doing that? we have proposed separately for some reason, but I don't think I ever got it? | 16:23 |
| yolanda | hi asselin | 16:23 |
| fungi | the other minor objection to that option, as i recall, is some people might think we released kilo already if we have a stable/kilo branch in existence. but i'm not terribly concerned about that | 16:23 |
| dhellmann | ttx: I think it does solve this problem | 16:23 |
| ttx | dhellmann: see 1a 1b 1c above | 16:23 |
| *** dtantsur is now known as dtantsur|afk | 16:23 | |
| clarkb | ttx yup I think that solves it | 16:23 |
| *** mpavone has quit IRC | 16:23 | |
| anteaya | all I need is something to tell the plugin/driver maintainers, and acl docs to reference myself when reviewing patches | 16:23 |
| *** jlibosva has quit IRC | 16:24 | |
| yolanda | ah, nice, i see them | 16:24 |
| anteaya | I don't know enough about release management to offer an opinion on the direction itself | 16:24 |
| yolanda | we are in the same direction as you, so let's work together | 16:24 |
| fungi | dhellmann: proposed.* was and is currently used to make it so we don't have to switch who controls approval of changes to the branch at release time | 16:24 |
| asselin | yolanda, rcarrillocruz I have the log server done. Needs reviews really. This is the first one, so hopefully once everyone is on board and it's gets merged, the other parts should follow | 16:24 |
| dhellmann | ttx: why different acls? to give $project-core greater access before the release? | 16:24 |
| asselin | yolanda, +1 | 16:24 |
| ttx | dhellmann: lesser access | 16:24 |
| dhellmann | ah | 16:24 |
| yolanda | going to check that reviews | 16:24 |
| ttx | you wouldn't believe what people do with +As the day before release | 16:24 |
| asselin | yolanda, that's why we share the same topic: downstream-puppet :) | 16:24 |
| ttx | proposed/* is restricted to pTL + release managers | 16:25 |
| ttx | while stable/* is stable-maint team | 16:25 |
| rcarrillocruz | asselin: will check thx | 16:25 |
| rcarrillocruz | asselin: what timezone are you based btw | 16:25 |
| yolanda | asselin, yes, but i've also been focusing on improving system-config direclty | 16:25 |
| rcarrillocruz | ? | 16:25 |
| ttx | it was a risk management tool. We can emulate it using ACLs jumping | 16:25 |
| rcarrillocruz | ah nm | 16:25 |
| rcarrillocruz | california | 16:25 |
| rcarrillocruz | didn't read | 16:25 |
| ttx | i.e. specifying stable/kilo specific ACLs and remove them on release day | 16:25 |
| asselin | rcarrillocruz, yolanda, so the spec focuses on the modules needed by 3rd party ci. That's really to limit scope & a starting point. | 16:25 |
| dhellmann | ttx, fungi : we can control that with a single change in the global acls file, right? by adding a stable/foo that sets it to just stable-maint-core? or do we actually want it to be $project-stable-maint? | 16:25 |
| fungi | dhellmann: yeah, we had something like the first thing, and now it's the second thing | 16:26 |
| anteaya | can acl files reference an acl file? can you nest them? | 16:26 |
| *** dustins has quit IRC | 16:26 | |
| fungi | dhellmann: we _could_ control it by changes to group membership, or by changes to per-repo acls | 16:26 |
| ttx | dhellmann: no we want release managers + ptl to control the +A | 16:26 |
| ttx | fungi: feels like it's the simplest option | 16:27 |
| fungi | anteaya: they can "inherit" from another specific acl | 16:27 |
| dhellmann | fungi, ttx: this seems like something we could script, to make toggling the settings easier next time | 16:27 |
| anteaya | fungi: usually from the global acl file, yeah? | 16:27 |
| ttx | although we have a boatload of proposed/kilo changes in gerrit | 16:27 |
| jeblair | ttx, clarkb: we _can_ add multi-level fallback to the git checkouts if we need to. let's make the decision based on what we think is best for humans -- wether we think it's better to have the multi-tier system or not. | 16:27 |
| *** marcusvrn has joined #openstack-infra | 16:27 | |
| asselin | yolanda, yes, and some of those improvements are needed to even be able to split out | 16:27 |
| rcarrillocruz | asselin: that's ok, providing we have something working and tested we can then build things on top of it | 16:27 |
| fungi | ttx: on the up-side, if we want to do it via group membership, there's an api which can be used to update the groups en-masse but we'd probably need to switch to a more consistent group name for all of them to support that easily | 16:28 |
| yolanda | asselin, looking at logserver, so from here we also have another needs, that is , pointing to private git instead of git.openstack | 16:28 |
| fungi | anteaya: yep | 16:28 |
| yolanda | will you accept to parameterize that? | 16:28 |
| fungi | anteaya: but you can specify a different acl, and chain them that way | 16:28 |
| asselin | yolanda, yes, as a follow-on patch | 16:28 |
| clarkb | I was always pro single stable/kilo branch but appreciate the acl issues with that | 16:28 |
| anteaya | fungi: oh | 16:28 |
| *** ociuhandu has quit IRC | 16:28 | |
| *** dannywilson has joined #openstack-infra | 16:28 | |
| yolanda | ok, don't want to block these ones with -1, so let's push them and we can start working based on that | 16:29 |
| asselin | yolanda, I'm trying to keep the original as close to the original. | 16:29 |
| yolanda | it's really cool | 16:29 |
| ttx | jeblair: multilevel fallback would be proposed->stable->master and stable->proposed->master | 16:29 |
| ttx | to cover all the cases | 16:29 |
| rcarrillocruz | asselin: i've been working on a tool to bring up a system-config from scratch, it does the provisioning instances with ansible, and then orchestrates the config management bits running puppet | 16:29 |
| rcarrillocruz | what you think about the scope of this? | 16:29 |
| openstackgerrit | Khai Do proposed openstack-infra/project-config: Make gerritbot report changes from Gerrit 2.10.2 branch https://review.openstack.org/155450 | 16:29 |
| *** marcusvrn1 has quit IRC | 16:29 | |
| asselin | yolanda, yes, it's easier from a review point of view to separate them out | 16:29 |
| rcarrillocruz | would you say the provisioning is a separate project, or you think it would be useful to put in on yours | 16:29 |
| fungi | ttx: jeblair: clarkb: sdague: i wonder if we could simplify it by special-casing stable/ and proposed/ and assuming that any given repo can only have one at a time but not both? | 16:30 |
| jeblair | clarkb, fungi, ttx: yeah, so maybe in an ideal world, we have a single stable/branch and hide the acl complexity with a tool? does that make things easiest for humans to understand? it violates ttx's 1b -- sends the wrong message about the release, but i think most folks think that's minor? | 16:30 |
| asselin | rcarrillocruz, I don't know enough about provisioning to comment. | 16:30 |
| ttx | fungi: that is for sure. Never both | 16:30 |
| jeblair | ttx: i would expect only "stable->proposed->master". do we need "proposed->stable->master" ? | 16:30 |
| *** _nadya_ has joined #openstack-infra | 16:30 | |
| anteaya | jeblair: it sounds to me like if we have the multi-tier system, we can deal with many things in different states, which I believe would be easier for us than trying to get and keep everyone in the same state | 16:31 |
| rcarrillocruz | provisioning in openstack itself btw, leveraging the new openstack ansible modules from mordred, shrews, TheJulia and others | 16:31 |
| jeblair | ttx: oh. i see. | 16:31 |
| dhellmann | jeblair: yeah, I think the message issue isn't as big of a deal | 16:31 |
| ttx | we can survive the message thing | 16:31 |
| yolanda | asselin, so i've also been working to move features out from system-config to modules, this could even simplify the openstackci module | 16:31 |
| ttx | especially with us using stable/* now everywhere | 16:31 |
| fungi | so (proposed|stable)->master | 16:31 |
| ttx | well | 16:31 |
| fungi | if we can somehow work it so that the logic treats proposed/X and stable/X as equals | 16:32 |
| rcarrillocruz | asselin: do you use a particular topic on reviews for this work, so I can look at reviews and push changes with that same topic? | 16:32 |
| ttx | fungi: the question is when a change is proposed on requirements stable/kilo and pulls projects proposed/kilo | 16:32 |
| *** emagana has joined #openstack-infra | 16:32 | |
| asselin | yolanda, +1 | 16:32 |
| ttx | fungi: +1 | 16:32 |
| asselin | rcarrillocruz, downstream-puppet | 16:32 |
| yolanda | same as ours | 16:32 |
| rcarrillocruz | ok | 16:32 |
| rcarrillocruz | let me see then | 16:32 |
| ttx | proposed/X and stable/X should be the same thing that can have two names | 16:33 |
| ttx | So... | 16:33 |
| jeblair | ttx: that seems harder to express | 16:33 |
| openstackgerrit | Clark Boylan proposed openstack-infra/devstack-gate: Make multinode first class with ansible https://review.openstack.org/172614 | 16:33 |
| ttx | Plan A is to renbame all branches to stable/kilo, somewhow move existing proposed/kilo reveiws to stable/kilo, fix all the ACLs to have stable/kilo ACL equivalent to proposed/kilo | 16:33 |
| asselin | rcarrillocruz, here are the one's I proposed: https://review.openstack.org/#/q/topic:downstream-puppet+owner:%22Ramy+Asselin+%253Cramy.asselin%2540hp.com%253E%22,n,z | 16:34 |
| ttx | Plan B is to somehow express in the tooling that proposed/kilo and stable/kilo are the same thing | 16:34 |
| ttx | Choice of plan depends how hard and risky it is to do plan B | 16:34 |
| dhellmann | what tool actually handles that checkout fallback behavior? | 16:34 |
| jeblair | i think we need further design on plan b if we want to pursue it | 16:34 |
| dhellmann | is that in a script, zuul, gerrit, something else? | 16:35 |
| jeblair | i don't think i fully understand it or agree that it would work | 16:35 |
| ttx | also in plan A: have to rewrite release tooling, rewrite release doc, redirect everyone to stable/kilo testing | 16:35 |
| ttx | Plan A: No rocket science, just pain | 16:35 |
| jeblair | dhellmann: it's two things: shell script in devstack-gate and python code in zuul-cloner. they do the same thing. zuul-cloner will replace the shell script in devstack-gate shortly after kilo releases. | 16:35 |
| fungi | dhellmann: devstack-gate in most cases (not sure if anything's using zuul-cloner for this purpose yet) | 16:35 |
| ttx | Plan B: potential rocket science | 16:36 |
| *** vdo has quit IRC | 16:36 | |
| ttx | jeblair, fungi: how about you evaluate the feasibility of plan B and we come back here to make a decision based on that ? | 16:36 |
| dhellmann | ttx: ok, after I get some food I can help with the tooling changes for plan a | 16:36 |
| ttx | Plan A is defintely doable | 16:36 |
| yolanda | asselin, what about project-config, to bring up all the projects and jobs, do you have any thoughts on how to incorporate that? | 16:36 |
| ttx | so if plan B is to complex / risky / impossible, let's do plan A | 16:37 |
| *** thingee has quit IRC | 16:37 | |
| asselin | yolanda, nothing great | 16:37 |
| jeblair | ttx: the thing i need to work out with plan b is whether we can implement it as a hierarchy, or if we need to do the (A|B)->master thing. | 16:37 |
| fungi | yeah, i need to go get lunch but can refresh my memory on how setup_project() in devstack-gate's functions.sh is doing this | 16:37 |
| asselin | yolanda, right now, for 3rd party ci, I copy & override | 16:37 |
| jeblair | fungi, ttx: the implementation is not hard. the logic is. | 16:38 |
| fungi | right, we have a lot of subtlety around it there already | 16:38 |
| asselin | yolanda, for example: https://github.com/rasselin/os-ext-testing/blob/master/puppet/modules/os_ext_testing/manifests/master.pp#L313 | 16:38 |
| yolanda | we also need some more services like a puppetmaster at this moment, some static, git servers...are we ok to propose there? | 16:38 |
| fungi | and we've seriously broken ourselves there in the past | 16:38 |
| *** EmilienM is now known as EmilienM|afk | 16:38 | |
| clarkb | asselin: yolanda IMO that module should be strictly what is required to run tests | 16:39 |
| jeblair | 16:32 < ttx> fungi: the question is when a change is proposed on requirements stable/kilo and pulls projects proposed/kilo | 16:39 |
| clarkb | so zuul, nodepool, jenkins, logs | 16:39 |
| clarkb | roughly | 16:39 |
| ttx | jeblair, fungi: Let's give it a bit more thought... I'll be back in a couple of hours to discuss that | 16:39 |
| jeblair | ttx: ^ what do you think should happen in that example? | 16:39 |
| ttx | jeblair: I expect a change in */foo to be tested with */foo whatever that is | 16:39 |
| yolanda | ok | 16:40 |
| ttx | if none available, master | 16:40 |
| fungi | we already have a change to requirements updating to look for proposed/X if stable/X doesn't exist for a given project, not sure if it's been merged yet... checking | 16:40 |
| jeblair | ttx: but stable/foo is always preferred to proposed/foo, right? | 16:40 |
| johnthetubaguy | mordred: fungi: I am seeming most of those instances as deleted now, has it resolved its-self? | 16:40 |
| ttx | jeblair: doesn't matter, they don't exist at the same time. | 16:40 |
| ttx | (for a given project) | 16:40 |
| clarkb | right so .*/kilo will only match a single branch on any project at one time | 16:41 |
| dhellmann | *that* says to me that using the name to indicate the status is the real issue | 16:41 |
| fungi | johnthetubaguy: yeah, it looks like explicitly deleting those has worked now | 16:41 |
| ttx | so proposed/foo -> proposed/foo, */foo, master | 16:41 |
| ttx | clarkb: yes | 16:41 |
| *** patrickeast has quit IRC | 16:41 | |
| asselin | yolanda, rcarrillocruz dropping off for a bit | 16:41 |
| fungi | johnthetubaguy: i suspect it was api calls with unexpected responses or something while the systems in that region were having trouble | 16:41 |
| johnthetubaguy | fungi: do you keep retrying delete if delete fails? | 16:42 |
| *** asselin has quit IRC | 16:42 | |
| fungi | johnthetubaguy: we do | 16:42 |
| johnthetubaguy | fungi: I thought so, hmm its odd it took so long to heal | 16:42 |
| fungi | johnthetubaguy: so in this case i'm not sure why nodepool didn't successfully retry deletes for these, but don't have time to look into it right now | 16:42 |
| jeblair | ttx, clarkb, fungi: we need to express this in a generalized way. we can't actually encode things like "stable/" and "proposed/" in the tooling itself. so if we go this route, we need to define the rules that we want to have happen, then find a way to express those rules in a generalized way. | 16:42 |
| johnthetubaguy | fungi: ah, no worries | 16:43 |
| *** dannywilson has quit IRC | 16:43 | |
| johnthetubaguy | fungi: I suspect it was 500 errors, we can check the logs on our side if thats helpful when you need to dig into that | 16:43 |
| *** unicell has joined #openstack-infra | 16:43 | |
| clarkb | jeblair: yes I think the rule is that we have (potentially arbitrary) branch prefixes that prefix a unique release suffix | 16:43 |
| ttx | jeblair: I think we can. */foo -> */foo if available, master if not | 16:43 |
| fungi | jeblair: well, we encode stable and proposed into test-matrix.py so could in theory call out to that? | 16:44 |
| *** markvoelker has joined #openstack-infra | 16:44 | |
| jeblair | fungi: no, this has to be able to happen entirely within zuul-cloner | 16:44 |
| *** dhritishikhar has quit IRC | 16:44 | |
| fungi | oh, right. devstack-gate is oldthink | 16:44 |
| clarkb | any prefix/suffix pairs will be treated as matching other prefix/suffix pairs if the suffix matches | 16:44 |
| ttx | clarkb: ++ | 16:44 |
| clarkb | (this will likely also help cloud and gozer with their >1 kilo branches) | 16:45 |
| jeblair | if we can encode a hierarchy into that (even if we don't use it), that's probably easier... | 16:45 |
| fungi | alpha order? | 16:45 |
| clarkb | *hpcloud | 16:45 |
| ttx | if you really want a hierarchy, make stable the "best" choice | 16:45 |
| *** hughhalf has quit IRC | 16:45 | |
| ttx | reverse alpha order | 16:45 |
| *** unicell has quit IRC | 16:46 | |
| *** Guest32383 has quit IRC | 16:46 | |
| *** unicell1 has joined #openstack-infra | 16:46 | |
| fungi | user-defined ordering? | 16:46 |
| ttx | so that you could use prefix for versions and use the "latest" | 16:46 |
| jeblair | so 'match (.*)/(.*)'; try: 'stable/$2', 'proposed/$2', 'master' | 16:46 |
| ttx | jeblair: that works too. | 16:46 |
| *** sarob has quit IRC | 16:47 | |
| *** sarob has joined #openstack-infra | 16:47 | |
| ttx | got to run, be back in 2 hours | 16:47 |
| *** kgiusti has quit IRC | 16:47 | |
| *** hughhalf has joined #openstack-infra | 16:47 | |
| jeblair | kk | 16:47 |
| clarkb | jeblair: ya I think that generally works. The one potential issue is feature branches | 16:47 |
| *** otter768 has joined #openstack-infra | 16:47 | |
| *** afazekas has joined #openstack-infra | 16:47 | |
| clarkb | but if a feature branch overlaps with a release name then likely they should be treated the same | 16:47 |
| *** dhritishikhar has joined #openstack-infra | 16:48 | |
| anteaya | do feature branch have consistent naming? | 16:48 |
| clarkb | otherwise it iwll fall back onto master which is how we currently test feature branches | 16:48 |
| anteaya | branches | 16:48 |
| clarkb | anteaya: they are all prefixed with "feature/" | 16:48 |
| ttx | yeah, retrospectively we should have done kilo/proposed | 16:48 |
| anteaya | ah | 16:48 |
| ttx | or foobar/feature | 16:48 |
| jeblair | if we're changing it anyway... :) | 16:48 |
| *** MaxV has quit IRC | 16:49 | |
| * ttx runs away sceraming | 16:49 | |
| fungi | match (.*)/(.*) and from all listed branches $b matching (.*)/$2 try $b/$2 in a defined sort order (reverse alpha?) using the first match found, else use master | 16:49 |
| ttx | and screaming | 16:49 |
| *** MaxV has joined #openstack-infra | 16:49 | |
| ttx | fungi wins | 16:49 |
| *** adreznec has quit IRC | 16:49 | |
| *** mrmartin has quit IRC | 16:50 | |
| *** mfink_ has quit IRC | 16:50 | |
| *** markvoelker has quit IRC | 16:50 | |
| *** tqtran_ has joined #openstack-infra | 16:50 | |
| fungi | this assumes that, e.g., our local clone is up to date enough to know what remote branches are valid for that repo | 16:51 |
| jeblair | https://etherpad.openstack.org/p/zuul-cloner-branches | 16:51 |
| anteaya | I am looking for a way to work the wilhelm scream into something: https://www.youtube.com/watch?v=cdbYsoEasio | 16:51 |
| fungi | (or that we have some other source of information providing a list of valid branches for that project) | 16:51 |
| * dhellmann goes for food, too | 16:51 | |
| jeblair | the etherpad has the current fallback logic | 16:52 |
| *** otter768 has quit IRC | 16:52 | |
| jeblair | (which is, of course, slightly more complex than we've been talking about. we have the fallback branche, then we have overrides, then we have per-project overrides) | 16:52 |
| *** koolhead17 has joined #openstack-infra | 16:52 | |
| *** dhritishikhar has quit IRC | 16:54 | |
| *** tqtran_ has quit IRC | 16:54 | |
| jeblair | fungi: are you proposing that we examine the git repos before calling zuul cloner, or encode the sorted(refs/origin) in zuul-cloner? | 16:55 |
| fungi | jeblair: examine the git repo to build the list of expected branch names | 16:55 |
| *** e0ne has quit IRC | 16:56 | |
| jeblair | fungi: but then we have to do git operations before calling zuul cloner, and it is supposed to do all of those for us | 16:56 |
| fungi | jeblair: which seems like the only way around this if we explicitly insist on not hard-coding any prefixes into the tooling anywhere | 16:57 |
| fungi | jeblair: i meant as a feature _of_ zuul-cloner | 16:57 |
| jeblair | fungi: i'm not proposing we encode the fallback logic in zuul-cloner, i'm saying we should create a way so we can specify it external to the tool | 16:57 |
| *** david-lyle has quit IRC | 16:57 | |
| *** dustins has joined #openstack-infra | 16:57 | |
| fungi | oh! sure then your proposal sounds fine. it at first sounded like you were opposed to encoding these anywhere at all | 16:57 |
| jeblair | fungi: i put a sample command line in the etherpad | 16:57 |
| clarkb | jeblairs proposal makes sense to me and will accomodate feature branches too | 16:58 |
| fungi | yep, i see it there. i was confused as to why you were encoding branch prefixes into your sample command line since you seemed not to want any | 16:58 |
| jeblair | fungi: so i'm suggesting we have zuul-cloner accept arguments to match the branch, and then you can add additional arguments to specify the fallback branches using that match | 16:58 |
| jeblair | fungi: sorry, i meant i want the tool to be general, but to accept our project-specific logic as input | 16:58 |
| fungi | i'm fine with that example syntax | 16:58 |
| fungi | i thought you wanted devstack-gate itself to also not have to encode any branch prefixes | 16:59 |
| *** bswartz has quit IRC | 16:59 | |
| fungi | but putting them in devstack-gate's invocation of zuul-cloner seems reasonable | 16:59 |
| anteaya | I think I even understand what is going on | 16:59 |
| jeblair | yeah, and possibly later with a zuul cloner jjb macro (or zuulv3 equivalent) so you can use it without having to think too hard | 17:00 |
| fungi | wfm | 17:00 |
| fungi | i need to go get lunch, but will catch up on the rest of the conversation when i get back | 17:00 |
| jeblair | ok | 17:01 |
| anteaya | how does this translate to what driver/plugin maintainers should do, or are we too early in the conversation for this | 17:01 |
| anteaya | should we wait until this is built? | 17:01 |
| pc_m | lifeless: hi | 17:02 |
| anteaya | as a for instance a split out neutron driver testing itself with latest neutron | 17:02 |
| clarkb | anteaya: if they use zuul-cloner then they can use either proposed/foo -> stable/foo or just go straight to stable/foo and it should just work for them | 17:02 |
| *** unicell1 has quit IRC | 17:02 | |
| anteaya | okay | 17:02 |
| *** unicell has joined #openstack-infra | 17:02 | |
| anteaya | clarkb: can I help with documenting that anywhere as a series of steps? | 17:03 |
| clarkb | anteaya: I think we should have something more concrete first (like a zuul-cloner change) | 17:03 |
| anteaya | fair enough | 17:03 |
| clarkb | otherwise our docs may not be very correct | 17:03 |
| anteaya | okay | 17:03 |
| * anteaya goes back to ansible things | 17:03 | |
| jeblair | sdague: as the designer of half of the current fallback logic, can you look at https://etherpad.openstack.org/p/zuul-cloner-branches and make sure it looks sane? | 17:06 |
| *** bswartz has joined #openstack-infra | 17:06 | |
| *** dhritishikhar has joined #openstack-infra | 17:06 | |
| jeblair | sdague: it's in zuul-cloner terms, but "--project-branch tempest=master" == "OVERRIDE_TEMPEST_PROJECT_BRANCH=master" in devstack-gate terms, and likewise "--branch stable/icehouse" == "OVERRIDE_ZUUL_BRANCH=stable/icehouse" | 17:07 |
| *** andrey-mp has joined #openstack-infra | 17:07 | |
| *** tonytan4ever has quit IRC | 17:07 | |
| *** dboik_ has quit IRC | 17:08 | |
| *** dboik has joined #openstack-infra | 17:08 | |
| *** david-lyle has joined #openstack-infra | 17:09 | |
| clarkb | notmyname: I am going to grab breakfast then will look at the swift tagging | 17:09 |
| clarkb | ttx: are you happy with erasure_code_dev_history as the tag for the feature/ec branch that will be deleted? | 17:09 |
| *** ianw has quit IRC | 17:10 | |
| *** amotoki has quit IRC | 17:10 | |
| *** david-lyle has quit IRC | 17:11 | |
| *** david-lyle has joined #openstack-infra | 17:11 | |
| *** andrey-mp has quit IRC | 17:11 | |
| *** samueldmq has joined #openstack-infra | 17:11 | |
| notmyname | clarkb: thanks | 17:12 |
| *** MaxV has quit IRC | 17:12 | |
| *** adalbas has joined #openstack-infra | 17:13 | |
| cinerama | ugh, still having problems with openstackid-dev.o.o. i can't use openstackid.o at the moment because my patch hasn't been applied there yet (i think) | 17:14 |
| *** AJaeger_ has joined #openstack-infra | 17:15 | |
| anteaya | cinerama: I can't help you but posting the url of the patch you are referencing might help the next person who can | 17:16 |
| *** dhritishikhar has quit IRC | 17:16 | |
| *** ssam2 has quit IRC | 17:17 | |
| cinerama | anteaya: hey that's a good idea...let me see if i can find it | 17:17 |
| cinerama | this is the change: https://review.openstack.org/#/c/165696/ | 17:19 |
| *** sigmavirus24 is now known as sigmavirus24_awa | 17:19 | |
| anteaya | okay so when fungi or clarkb comes back from eating or jeblair returns from his task that helps them evaluate if that code is on the server | 17:21 |
| *** asselin has joined #openstack-infra | 17:23 | |
| *** dhritishikhar has joined #openstack-infra | 17:24 | |
| marcusvrn | anteaya: hi! Is there a problem with gerrit? My Zuul stop watching Gerrit events...then I tried it manually (ssh -p 29418 USERNAME@review.openstack.org gerrit stream-events) and it doesn't return anything... I can ping review.openstack.org | 17:25 |
| anteaya | marcusvrn: hi | 17:25 |
| *** Krinkle is now known as Krinkle|detached | 17:25 | |
| anteaya | let's walk through this one at a time, is there a problem with gerrit? not to my knowledge | 17:26 |
| anteaya | your zuul stopped watching gerrit, okay so you are asking about your third party ci account | 17:26 |
| marcusvrn | anteaya: yeah, sry! hehe | 17:27 |
| anteaya | you tried to run stream-events manually using your third party ci credentials and were unsuccessful | 17:27 |
| *** sputnik13 has quit IRC | 17:27 | |
| anteaya | you have tried pinging gerrit and were successful | 17:27 |
| anteaya | let's first look at the possiblity that your third party ci account credentials are incorrect or your account is disabled | 17:28 |
| anteaya | what is the name of your account? | 17:28 |
| *** davideagnello has joined #openstack-infra | 17:28 | |
| *** Krinkle|detached is now known as Krinkle | 17:28 | |
| *** tnovacik has quit IRC | 17:28 | |
| marcusvrn | hds-hnas-ci | 17:29 |
| marcusvrn | anteaya: I think it's not disabled, because I can "connect" via ssh -> http://paste.openstack.org/show/204020/ | 17:30 |
| anteaya | paste the output of running the stream-events command | 17:31 |
| anteaya | your wikipage was last updated today by you: https://wiki.openstack.org/wiki/ThirdPartySystems/HDS_HNAS_CI | 17:32 |
| *** hdd has joined #openstack-infra | 17:32 | |
| marcusvrn | I have updated the wikipage yesterday | 17:32 |
| anteaya | ah sorry yesterday | 17:32 |
| *** sputnik13 has joined #openstack-infra | 17:33 | |
| marcusvrn | and about stream-events command, nothing is returned... | 17:33 |
| anteaya | no mention of hds on infra, dev or third-party-announce mailing lists for this month | 17:33 |
| anteaya | it just hangs? | 17:33 |
| marcusvrn | http://paste.openstack.org/show/204021/ | 17:33 |
| anteaya | marcusvrn: don't control-C it | 17:34 |
| anteaya | it if is open it means it is working and no events are being broadcast | 17:34 |
| *** dhritishikhar has quit IRC | 17:34 | |
| anteaya | we are in release candidate stage | 17:34 |
| anteaya | the only patches you should be seeing are bug fixes and you shouldn't see many of them | 17:34 |
| *** achanda has joined #openstack-infra | 17:35 | |
| marcusvrn | hmm.... but I have tried to recheck a patch, and nothing was shown in zuul/debug.log | 17:36 |
| *** tonytan4ever has joined #openstack-infra | 17:36 | |
| *** kgiusti has joined #openstack-infra | 17:36 | |
| anteaya | in your zuul/debug.log? | 17:36 |
| *** harlowja_away is now known as harlowja | 17:36 | |
| marcusvrn | yep | 17:36 |
| armax | fungi, sdague, dtroyer, jeblair is there a plan to switch to using/storing local.conf instead of localrc in the OpenStack infra job? | 17:36 |
| anteaya | marcusvrn: what patch? | 17:37 |
| armax | or any dsvm jobs for that matter | 17:37 |
| *** deepakcs has quit IRC | 17:37 | |
| *** e0ne has joined #openstack-infra | 17:37 | |
| *** sputnik13 has quit IRC | 17:37 | |
| marcusvrn | anteaya: https://review.openstack.org/#/c/157085/ | 17:37 |
| *** Sukhdev has quit IRC | 17:38 | |
| adam_g | hi, a failing job accidentally slipped into stable/juno gate pipelines earlier this week, anyone have a second for a trivial project-config review that removes it and unblocks them? https://review.openstack.org/#/c/173319/ | 17:38 |
| anteaya | marcusvrn: I don't see that jenkins picked it up either | 17:39 |
| anteaya | marcusvrn: has your stream-events posted any output back to you yet? | 17:40 |
| AJaeger_ | anteaya: I already +2ed 173319... | 17:40 |
| anteaya | AJaeger_: does this have anything to do with what I am talking about with marcusvrn? | 17:41 |
| AJaeger_ | anteaya: no - that was about adam_g. Sorry for interrupting. | 17:41 |
| *** SumitNaiksatam has quit IRC | 17:41 | |
| anteaya | AJaeger_: oh okay | 17:41 |
| marcusvrn | anteaya: nothing yet | 17:41 |
| ttx | clarkb: happy | 17:42 |
| anteaya | marcusvrn: okay, so stream-events I can help with or try to, I don't run zuul myself so I have no idea what should or should not be in the logs | 17:42 |
| adam_g | AJaeger, thanks for that. | 17:42 |
| anteaya | marcusvrn: I jsut started stream-events myself and also have nothing so far | 17:43 |
| *** SumitNaiksatam has joined #openstack-infra | 17:43 | |
| marcusvrn | anteaya: ok, if I solve the stream-events problem, I guess zuul will work too | 17:44 |
| harlowja | clarkb so talked to smoser , what needs to happen to get https://github.com/stackforge/cloud-init/compare/0.7.x...harlowja:0.7.x-fixed to get pushed into stackforge (only on the 0.7.x branch) | 17:44 |
| harlowja | ' Able to merge. These branches can be automatically merged. ' (so shouldn't be a problem) | 17:44 |
| marcusvrn | anteaya: I can recheck another patch to check if stream-events will posted some output | 17:45 |
| marcusvrn | post* | 17:45 |
| *** duran has joined #openstack-infra | 17:45 | |
| mordred | harlowja: one of us will need to merge it in to a repo locally and force-push it up to gerrit | 17:45 |
| harlowja | mordred cool; hopefully not to much of a pita :-/ | 17:46 |
| mordred | harlowja: I can help with that ... but before I do, the other branches are good, right? | 17:46 |
| *** [HeOS] has quit IRC | 17:46 | |
| *** _nadya_ has quit IRC | 17:46 | |
| harlowja | master/head is just gonna be the home of 2.0 so its fine | 17:46 |
| mordred | cool | 17:46 |
| harlowja | and there are no other branches, so ya :-P | 17:46 |
| harlowja | all good afaik | 17:46 |
| anteaya | marcusvrn: okay, when you leave a recheck is recheck the very first word in the first line starting in the first space of the comment? | 17:47 |
| marcusvrn | anteaya: just rechecked and nothing happens (https://review.openstack.org/#/c/173157/) | 17:47 |
| *** signed8bit has joined #openstack-infra | 17:47 | |
| jeblair | anteaya, marcusvrn, fungi, clarkb, mordred, pleia2: i think all our ssh stream threads are stuck again | 17:47 |
| *** markvoelker has joined #openstack-infra | 17:47 | |
| anteaya | jeblair: ah | 17:47 |
| AJaeger_ | yeah, something is strange with gerrit - there are no new events going to the gate queue currently | 17:47 |
| anteaya | in which case thank you marcusvrn for alerting us to that | 17:47 |
| *** johnthetubaguy is now known as zz_johnthetubagu | 17:48 | |
| jeblair | https://review.openstack.org/#/c/170948/ has merged | 17:48 |
| jeblair | i would like to deploy it so that we can debug this problem | 17:48 |
| marcusvrn | anteaya: np | 17:48 |
| anteaya | jeblair: I am in favour of that action | 17:49 |
| *** marun has joined #openstack-infra | 17:49 | |
| mordred | harlowja: you want the 0.7.x branch to be what is in the contents of your 0.7.x-fixed branch, right? | 17:49 |
| mordred | jeblair: awesome | 17:49 |
| mordred | jeblair: I am also in favor of that action | 17:49 |
| harlowja | mordred ya | 17:49 |
| *** e0ne is now known as e0ne_ | 17:50 | |
| *** SumitNaiksatam has quit IRC | 17:50 | |
| *** erlon has joined #openstack-infra | 17:50 | |
| *** Longgeek has quit IRC | 17:51 | |
| *** adalbas has quit IRC | 17:51 | |
| *** markvoelker has quit IRC | 17:52 | |
| *** mpaolino has joined #openstack-infra | 17:52 | |
| jeblair | clarkb, fungi, mordred, pleia2, zaro: https://review.openstack.org/174008 | 17:53 |
| clarkb | mordred: harlowja no, cloud init needs to merge it then give us the repo to push imo | 17:53 |
| clarkb | mordred: I don't want to be responsible for figuring out what they want their git tree to look like | 17:53 |
| harlowja | https://github.com/harlowja/cloud-init/tree/0.7.x-fixed ? | 17:54 |
| harlowja | or do u want more than just a branch? | 17:54 |
| harlowja | which i can do also | 17:54 |
| jeblair | clarkb, mordred, zaro: that build includes https://review.openstack.org/#/c/94675/ | 17:54 |
| mordred | harlowja: nope. that's fine | 17:54 |
| jeblair | which, if i understand zaro correctly from the meeting on tuesday, means we're about to break launchpad integration | 17:54 |
| mordred | clarkb: they did. it's harlowja's branch | 17:55 |
| clarkb | mordred: then what needs merging? | 17:55 |
| jeblair | mordred, harlowja, clarkb: any chance we could shelve this conversation for a minute | 17:55 |
| jeblair | ? | 17:55 |
| harlowja | seems like nothing anymore | 17:55 |
| harlowja | lol | 17:55 |
| harlowja | mordred i think did it? | 17:55 |
| clarkb | jeblair: sure, I +2'd the war update | 17:55 |
| mordred | jeblair: yah | 17:55 |
| clarkb | jeblair: that brings the change you link and better connection logging which sounds good to me | 17:55 |
| *** sarob has quit IRC | 17:55 | |
| jeblair | clarkb: yeah, i'm saying that if i follow zaro correctly, https://review.openstack.org/#/c/94675/ will break gerrit-lp integration in jeepyb | 17:56 |
| mordred | jeblair: there is a correspondong jeepyb change which adds consumption of those parameters ... | 17:56 |
| jeblair | mordred: has it merged? | 17:56 |
| mordred | no: https://review.openstack.org/#/c/173571/ | 17:56 |
| clarkb | wouldn't --change-owner just be ignored by argparse? | 17:56 |
| mordred | I just reviewed it this morning | 17:56 |
| clarkb | or is argparse going to complain? | 17:57 |
| mordred | I did not raise it as important because it mentioned 2.10 and not 2.8 | 17:57 |
| clarkb | I guess I don't know what argparse does in that situation | 17:57 |
| mordred | so it seems landing the jeepyb change before any gerrit changes would be safe | 17:57 |
| *** ivar-lazzaro has joined #openstack-infra | 17:57 | |
| *** rfolco has quit IRC | 17:57 | |
| clarkb | mordred: I agree, I can approve 173571 now | 17:58 |
| *** andreykurilin has joined #openstack-infra | 17:58 | |
| clarkb | exept it depends on https://review.openstack.org/#/c/155463/ which is wip | 17:58 |
| *** openstackstatus has quit IRC | 17:58 | |
| clarkb | zaro: ^ can you remove the dep there? | 17:58 |
| mordred | clarkb: argparse does error on unknown arguments | 17:58 |
| mordred | clarkb: it is? | 17:59 |
| mordred | clarkb: it does not show as dependent to me | 17:59 |
| sdague | jeblair: I think https://etherpad.openstack.org/p/zuul-cloner-branches makes sense to me | 17:59 |
| clarkb | mordred: its a depends on | 17:59 |
| mordred | ah - well, we can pull that from the commit message ourselves | 17:59 |
| *** openstackstatus has joined #openstack-infra | 17:59 | |
| *** ChanServ sets mode: +v openstackstatus | 17:59 | |
| mordred | done | 18:00 |
| zaro | clarkb: was getting this in the log: http://paste.openstack.org/show/204022/ | 18:00 |
| mordred | yes - argparse will error on unknown params | 18:00 |
| jeblair | zaro: i don't think we shoud have merged that change to 2.8.4 | 18:00 |
| *** carl_baldwin has quit IRC | 18:00 | |
| jeblair | i don't really know why it was necessary | 18:00 |
| sdague | jeblair: are we sure it will stack up branches in order and keep them correctly? I think that's my only concern with fallback branch as multi option instead of comma | 18:00 |
| clarkb | mordred: +2 | 18:00 |
| jeblair | sdague: i need to focus on gerrit things for a few mins, will get back to you | 18:01 |
| jeblair | #status alert Gerrit has stopped emitting events so Zuul is not alerted to changes. We will restart Gerrit shortly to correct the problem. | 18:01 |
| openstackstatus | jeblair: sending alert | 18:01 |
| sdague | jeblair: no worries | 18:01 |
| mordred | in any case, the depends-on was backwards | 18:01 |
| mordred | the jeepyb change was to allow jeepyb to not break when the new params exist, then the gerrit change adds the params ... zaro for next time you have a similar thing | 18:02 |
| zaro | jeblair: the purpose was so we could pass owner info back to LP | 18:02 |
| zaro | jeblair: i believe there was a request in LP for that. | 18:02 |
| -openstackstatus- NOTICE: Gerrit has stopped emitting events so Zuul is not alerted to changes. We will restart Gerrit shortly to correct the problem. | 18:02 | |
| *** ChanServ changes topic to "Gerrit has stopped emitting events so Zuul is not alerted to changes. We will restart Gerrit shortly to correct the problem." | 18:02 | |
| mordred | harlowja: your 0.7.x branch should be correct now | 18:03 |
| *** vlaza has left #openstack-infra | 18:03 | |
| jeblair | mordred: we need jeepyb deployed before we restart gerrit 2.8.4.17 | 18:03 |
| *** patrickeast has joined #openstack-infra | 18:03 | |
| mordred | jeblair: hrm. so - want me to manually deploy a copy of it with that patch applied? | 18:03 |
| *** signed8bit is now known as signed8bit_ZZZzz | 18:04 | |
| *** annegentle has joined #openstack-infra | 18:04 | |
| jeblair | mordred: i think you need to force-merge that change, and i need to force-merge the gerrit change, and then perhaps if we run ansipuppet after checking out system-config on puppetmaster it will do both? | 18:04 |
| mordred | jeblair: ok | 18:04 |
| mordred | jeblair: I have force-merged the jeepyb change | 18:04 |
| clarkb | jeblair: yes, if you don't use the run_all.sh script then you need to update system-config directly | 18:05 |
| *** e0ne_ is now known as e0ne | 18:05 | |
| openstackstatus | jeblair: finished sending alert | 18:05 |
| mordred | jeblair: I've got ansipuppet when you're ready | 18:05 |
| *** signed8bit_ZZZzz is now known as signed8bit | 18:05 | |
| jeblair | mordred: okay, will you make sure the system-config repo is up to date? | 18:05 |
| mordred | yup | 18:05 |
| jeblair | mordred: i have force-merged the puppet change | 18:05 |
| jeblair | mordred: so all yours | 18:05 |
| *** ashleighfarnham has joined #openstack-infra | 18:06 | |
| mordred | Upgrade production gerrit to 2.8.4.17 | 18:06 |
| mordred | is the latest there now | 18:06 |
| mordred | ansipuppeting | 18:06 |
| zaro | mordred: the 'kind' param is not in 2.8 will jeepby ignore that? | 18:07 |
| mordred | zaro: yes | 18:07 |
| *** bswartz has quit IRC | 18:07 | |
| mordred | zaro: when you add it to argparse, it will be fine if it doesn't recieve it | 18:07 |
| mordred | zaro: because you did | 18:08 |
| mordred | parser.add_argument('--kind', default=None) | 18:08 |
| fungi | okay, back full of fish and catching up | 18:08 |
| mordred | that means args.kind will be None | 18:08 |
| mordred | which is fine, because we don't consume it anywhere | 18:08 |
| *** _nadya_ has joined #openstack-infra | 18:08 | |
| *** mpaolino has quit IRC | 18:09 | |
| fungi | cinerama: i got openstackid-dev.o.o back on track this morning (tipit moved one of its databases) and was able to authenticate to it successfully thereafter. what specifically is it doing for you? | 18:09 |
| cinerama | fungi: let me try again | 18:09 |
| clarkb | mordred: how is puppet doing? | 18:10 |
| *** sigmavirus24_awa is now known as sigmavirus24 | 18:10 | |
| *** david-lyle has quit IRC | 18:10 | |
| cinerama | fungi: heh. now it works. | 18:10 |
| mordred | clarkb: running | 18:11 |
| *** mayurig has joined #openstack-infra | 18:11 | |
| jeblair | mordred, clarkb: from inspecting the filesystem, the new gerrit war is in place. | 18:11 |
| _qb | hello, I'm trying to estimate how much hardware I need to throw at an environment to do third party testing. it's daunting. anyone want to help? | 18:13 |
| mordred | jeblair, clarkb: should be done | 18:13 |
| *** dosaboy has quit IRC | 18:13 | |
| *** amitgandhinz has quit IRC | 18:13 | |
| anteaya | _qb: we are dealing with a maintainance issue at the moment | 18:13 |
| jeblair | mordred: i see '--change-owner' in "update-bug --help" | 18:14 |
| *** rfolco has joined #openstack-infra | 18:14 | |
| *** dosaboy has joined #openstack-infra | 18:14 | |
| jeblair | mordred: ready for me to restart gerrit? | 18:14 |
| *** tkelsey has quit IRC | 18:15 | |
| *** dannywilson has joined #openstack-infra | 18:15 | |
| mordred | jeblair: yes | 18:15 |
| fungi | okay, caught up now. i seem to have missed all the excitement (hopefully!) | 18:15 |
| jeblair | oh, er... | 18:15 |
| jeblair | does puppet restart gerrit? | 18:15 |
| anteaya | fungi: all but the cheering | 18:16 |
| fungi | i have not seen puppet restart gerrit normally, though it may start it if it was stopped? | 18:16 |
| clarkb | I don't think it does | 18:16 |
| * clarkb reads some puppet | 18:16 | |
| jeblair | gerrit2 7803 0.0 0.0 4440 652 ? Ss 18:06 0:00 sh -c /etc/init.d/gerrit stop; /usr/bin/java -jar /home/gerrit2/review_site/bin/gerrit.war init -d /home/gerrit2/review_site --batch --no-auto-start; /usr/bin/java -jar /home/gerrit2/review_site/bin/gerrit.war reindex -d /home/gerrit2/review_site | 18:16 |
| fungi | perhaps that's what you meant by restart though | 18:16 |
| jeblair | i think it has stopped it and is running a reindex | 18:16 |
| jeblair | so, er, i guess gerrit is down for the next 10 minutes :) | 18:17 |
| fungi | ahh, yep | 18:17 |
| anteaya | oh my, that takes how long? 30 minutes? | 18:17 |
| anteaya | oh just 10 | 18:17 |
| fungi | 10-15 | 18:17 |
| *** mayurig has quit IRC | 18:17 | |
| jeblair | well, it's 10 minutes into it, so if it takes 15 we have 5 left | 18:17 |
| clarkb | it will run a /etc/init.d/gerrit start after it installs a new war | 18:18 |
| clarkb | but I don't see it doing a stop | 18:18 |
| anteaya | _qb: can you join #openstack-dev and I can talk to you there | 18:18 |
| clarkb | oh the stop in in the "init" exec | 18:18 |
| clarkb | thats not very intuitive | 18:18 |
| clarkb | jeblair: so yes it iwll restart gerrit for us | 18:18 |
| jeblair | it seems to have finished the index and not restarted | 18:19 |
| jeblair | it is possible i broke it with my service gerrit stop | 18:19 |
| *** teran has quit IRC | 18:19 | |
| jeblair | but i also don't understand how ansipuppet could have returned while that was still runinng | 18:19 |
| *** teran has joined #openstack-infra | 18:19 | |
| jeblair | someone should look into this | 18:19 |
| clarkb | if the init script returned early that could happen | 18:19 |
| jeblair | on a test node | 18:19 |
| jeblair | starting gerrit | 18:20 |
| zaro | jeblair: it will take a while to start after the reinfex | 18:20 |
| *** dosaboy has quit IRC | 18:20 | |
| jeblair | 2.8.4-17-ge391a2d is running | 18:20 |
| clarkb | its up an appears to be happy for me | 18:20 |
| mordred | woot | 18:20 |
| anteaya | what are we confirming, launchpad login? | 18:21 |
| clarkb | notmyname: you were saying we should delete feature/ec_review as well? should that branch get a tag too? | 18:21 |
| *** dosaboy has joined #openstack-infra | 18:21 | |
| clarkb | anteaya: I opened a change and looked at my changes dashboard | 18:21 |
| *** _qb has left #openstack-infra | 18:21 | |
| anteaya | I can login with launchpad in chrome | 18:22 |
| *** EmilienM|afk is now known as EmilienM | 18:22 | |
| zaro | anteaya: that a change in gerrit puts an LP bug to 'in progress' | 18:22 |
| *** SumitNaiksatam has joined #openstack-infra | 18:22 | |
| *** dosaboy has quit IRC | 18:22 | |
| zaro | anteaya: but i think the bigger one is gerrit events are working again | 18:23 |
| notmyname | clarkb: ec_review has been merged to master, so it is accessible from there | 18:23 |
| clarkb | notmyname: I see | 18:23 |
| *** teran has quit IRC | 18:23 | |
| jeblair | #status ok Gerrit has been restarted. New patches, approvals, and rechecks between 17:30 and 18:20 UTC may have been missed by Zuul and will need rechecks or new approvals added. | 18:23 |
| openstackstatus | jeblair: sending ok | 18:23 |
| *** dosaboy has joined #openstack-infra | 18:24 | |
| *** _nadya_ has quit IRC | 18:24 | |
| *** dosaboy has quit IRC | 18:24 | |
| *** dosaboy has joined #openstack-infra | 18:24 | |
| jeblair | soooo many git push tasks | 18:24 |
| anteaya | marcusvrn: open stream-events again and tell me what your experience is | 18:24 |
| clarkb | notmyname: I am tagging ff7f7f474b634a921561aae1d0093a2881efcde4 with name erasure_code_dev_history and will push that momentarily | 18:25 |
| *** ChanServ changes topic to "Discussion of OpenStack Developer and Community Infrastructure | docs http://docs.openstack.org/infra/manual/ http://ci.openstack.org/ | bugs https://storyboard.openstack.org/ | source https://git.openstack.org/cgit/openstack-infra/" | 18:25 | |
| -openstackstatus- NOTICE: Gerrit has been restarted. New patches, approvals, and rechecks between 17:30 and 18:20 UTC may have been missed by Zuul and will need rechecks or new approvals added. | 18:25 | |
| clarkb | notmyname: can you double check that commit for me (it is head of feature/ec) | 18:25 |
| marcusvrn | anteaya: now it's working!! :) | 18:25 |
| notmyname | clarkb: ack ff7f7f474b634a921561aae1d0093a2881efcde4 | 18:26 |
| jeblair | sdague: yes, i think if you use 'append' with argparse you get the arguments in order | 18:26 |
| *** tonytan4ever has quit IRC | 18:26 | |
| jeblair | sdague: i think i like that because then we don't have to parse an argument that might already be a complex expression itself | 18:27 |
| anteaya | marcusvrn: wonderful, thank you for sharing your experience and alerting us to the problem, my gratitude | 18:27 |
| openstackstatus | jeblair: finished sending ok | 18:27 |
| *** bswartz has joined #openstack-infra | 18:28 | |
| *** signed8bit is now known as signed8bit_ZZZzz | 18:28 | |
| *** sambetts has quit IRC | 18:29 | |
| sdague | jeblair: ok, sure, might be good to ensure that it's documented, because wrong order is going to cause very subtle bugs. | 18:29 |
| jeblair | sdague: ++ | 18:30 |
| jeblair | incidentally, i think this solves a wikimedia problem where they don't have master branches | 18:30 |
| *** e0ne has quit IRC | 18:30 | |
| jeblair | (so the hardcoded 'master' fallback didn't always make sense to them) | 18:30 |
| marcusvrn | anteaya: you're welcome! thank you all for solving the problem so fast! :) | 18:31 |
| clarkb | notmyname: tag pushed | 18:31 |
| anteaya | marcusvrn: :) | 18:31 |
| notmyname | clarkb: thanks | 18:31 |
| clarkb | mordred: I need to remove myself from project bootstrappers should I remove you? | 18:31 |
| clarkb | notmyname: deleting feature/ec and feature/ec_review now | 18:31 |
| notmyname | clarkb: ack. looks good (tag) | 18:32 |
| *** sambetts has joined #openstack-infra | 18:32 | |
| mordred | clarkb: yes | 18:33 |
| jeblair | clarkb, fungi, ttx: okay, so i think plan B from earlier is doable as described in https://etherpad.openstack.org/p/zuul-cloner-branches, and we can probably land the changes for it this week if we are in a rush. | 18:33 |
| clarkb | notmyname: branches are deleted, you should be set | 18:33 |
| notmyname | clarkb: thanks | 18:33 |
| clarkb | mordred: done | 18:33 |
| jeblair | clarkb, fungi, ttx: for actual implementation, we can land the devstack-gate version of it first, just so long as we have a plan for zuul-cloner | 18:33 |
| *** thingee has joined #openstack-infra | 18:34 | |
| clarkb | jeblair: that seems reasonable since d-g is primarily what will affect kilo | 18:34 |
| fungi | that seems like a pragmatic path | 18:34 |
| fungi | no objection | 18:34 |
| AJaeger_ | jeblair: totally different topic, please put it in your queue and answer later. I'm signing of here but will read backlog tomorrow: | 18:34 |
| AJaeger_ | jeblair: anteaya, fungi and myself discussed yesterday gerrit permissions a bit on this etherpad: https://etherpad.openstack.org/p/E7dQRd2gyy . Should I clean this up and propose a patch somewhere? Question is where exactly... | 18:34 |
| jeblair | clarkb, fungi, ttx: so i think we can make the decision based on what we _want_ to do. technically we are able to execute plan B with what we wrote in the etherpad, and we can execute plan A with tooling around changing acls/groups, etc. | 18:34 |
| *** annegentle has quit IRC | 18:35 | |
| fungi | AJaeger_: it should probably wait until after kilo since we're discussing shaking some of that up anyway | 18:35 |
| *** annegentle has joined #openstack-infra | 18:35 | |
| fungi | AJaeger_: though i'm okay with documenting it and then changing it in a few weeks | 18:35 |
| *** eharney has quit IRC | 18:36 | |
| *** andreykurilin has quit IRC | 18:36 | |
| *** duran has quit IRC | 18:36 | |
| clarkb | I do have a preference for a single branch for a release | 18:36 |
| *** mrmartin has joined #openstack-infra | 18:36 | |
| clarkb | but I am not the person juggling acls and branches typically so defer to ttx's desire to have reviewer control around that | 18:36 |
| fungi | jeblair: clarkb: ttx: i do still find the simplicity of not having a proposed/* branch attractive, but will defer to ttx on that | 18:37 |
| jeblair | clarkb: me too | 18:37 |
| AJaeger_ | fungi: I'm happy either way, just want some guidance ;) Thanks! | 18:37 |
| jeblair | ttx: so we might have a room full of people that would prefer planA but are happy to implement planB if it makes release people happy :) | 18:37 |
| fungi | AJaeger_: for example, _right now_ we're discussing possibly getting rid of one of the items you described there entirely | 18:38 |
| *** ivar-laz_ has joined #openstack-infra | 18:38 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/devstack-gate: Make multinode first class with ansible https://review.openstack.org/172614 | 18:38 |
| *** notnownikki has quit IRC | 18:38 | |
| AJaeger_ | fungi: ah, ok ;) Missed the start of it | 18:38 |
| *** andreykurilin has joined #openstack-infra | 18:40 | |
| anteaya | AJaeger_: and thanks for writing up the etherpad, very valuable for me, thanks | 18:40 |
| *** ivar-lazzaro has quit IRC | 18:42 | |
| anteaya | I have no opinion on what the thing is as long as ttx and dhellmann are happy and I know what to tell neutron driver/plugin maintainers as well as how to review acl patches | 18:42 |
| *** tonytan4ever has joined #openstack-infra | 18:43 | |
| *** bswartz has quit IRC | 18:44 | |
| *** bswartz has joined #openstack-infra | 18:44 | |
| harlowja | mordred thx much for the cloud-init repo /merge/push/update | 18:44 |
| AJaeger_ | anteaya: I've learned a lot as well - and like to have a reference that I can point others to. | 18:44 |
| anteaya | AJaeger_: ++ | 18:44 |
| *** isviridov_away is now known as isviridov | 18:45 | |
| *** jcoufal_ has quit IRC | 18:45 | |
| *** isviridov is now known as isviridov_away | 18:45 | |
| *** sarob has joined #openstack-infra | 18:46 | |
| mordred | harlowja: sure nuff! | 18:47 |
| harlowja | the other question that smoser had that idk if he asked was is there anyway for https://github.com/stackforge/cloud-init to automatically push to https://github.com/cloud-init/cloud-init (or gerrit could do this?) | 18:47 |
| mordred | no | 18:47 |
| harlowja | so that cloud-init/cloud-init is mirroring stackforge | 18:47 |
| harlowja | kk | 18:47 |
| harlowja | smoser so there's your answer :-P | 18:48 |
| smoser | harlowja, thanks | 18:48 |
| *** otter768 has joined #openstack-infra | 18:48 | |
| harlowja | mordred do people usually setup there own little cron-jobs that do this then? | 18:49 |
| fungi | jeblair: clarkb: ttx: also, as a thought experiment on simplifying the shift of power between rc and release, what if the people approving backports from master to fix rc bugs are the same people who have a vested interest in the state of the stable branch which emerges from that, and follow relatively similar guidelines as to what's worth backporting to it in both states? | 18:49 |
| *** tqtran is now known as tqtran_afk | 18:49 | |
| mordred | harlowja: I don't know of anyone who uses a second github repo that is different than the github repo that gerrit replicates to | 18:50 |
| *** amitgandhinz has joined #openstack-infra | 18:50 | |
| harlowja | mordred kk | 18:50 |
| *** kgiusti has quit IRC | 18:50 | |
| harlowja | smoser ^ | 18:50 |
| dhellmann | fungi: how does that differ from what we do now? | 18:52 |
| *** andreykurilin has quit IRC | 18:52 | |
| *** duran has joined #openstack-infra | 18:52 | |
| *** otter768 has quit IRC | 18:53 | |
| *** hdd has quit IRC | 18:53 | |
| fungi | dhellmann: right now there are different groups of people approving changes to proposed/kilo branches vs those who will be approving backports to stable/kilo after release day | 18:53 |
| fungi | dhellmann: i'm just suggesting that perhaps there might actually be a lot of overlap between those groups, and that maybe we can come up with one acl which covers both the rc period and post-release | 18:54 |
| dhellmann | fungi: ok, I guess I didn't realize that because in oslo it's largely the same set of people. | 18:54 |
| openstackgerrit | Russell Bryant proposed openstack-infra/project-config: Add #openstack-neutron-ovn to gerritbot. https://review.openstack.org/173873 | 18:54 |
| dhellmann | fungi: like $project-stable-maint ? :-) | 18:54 |
| fungi | dhellmann: sounds like i already have a data point in favor of my conjecture! | 18:54 |
| ttx | fungi: one aspect is that stable people didn't sign up for that task at all though | 18:54 |
| *** ivar-laz_ has quit IRC | 18:55 | |
| dhellmann | fungi: yeah, though we are trying to go the other way in oslo and say that oslo-core replaces oslo-stable-maint and we don't have 2 teams -- if the tools are made simpler by us implementing that with group membership in gerrit, we can do that, too | 18:55 |
| *** tnovacik has joined #openstack-infra | 18:55 | |
| fungi | ttx: sure, though how many of them are the same people? | 18:55 |
| fungi | that's more or less what i'm asking | 18:55 |
| ttx | So whiel I could introduce the idea to them, i don't want to change the process to match the tooling | 18:55 |
| dhellmann | could we figure that out by looking at the gerrit db? | 18:55 |
| *** ivar-lazzaro has joined #openstack-infra | 18:55 | |
| fungi | not talking about .*-core vs .*-stable-maint, but rather .*-milestone vs .*-stable-maint | 18:56 |
| *** baoli_ has quit IRC | 18:56 | |
| fungi | yeah, we could run some numbers to identify overlap. i worry that it's all a bit ad-hoc so far though and might be hard to draw actual conclusions from across projects | 18:57 |
| dhellmann | yeah, we don't have a milestone team in oslo so I always forget about that | 18:57 |
| ttx | RC handling is completely different set of processes and procedures though. We can't just assume people inherit knowledge magically | 18:57 |
| ttx | It's also more time-sensitive and therefore risky | 18:57 |
| ttx | which is why the current set of people is so restricted | 18:57 |
| ttx | catching up on backlog | 18:57 |
| *** ivar-lazzaro has quit IRC | 18:58 | |
| fungi | fair enough. sounds like it really does need to be a separate set between rc approvers and and stable approvers | 18:58 |
| *** tsg_ has joined #openstack-infra | 18:58 | |
| *** ivar-lazzaro has joined #openstack-infra | 18:59 | |
| ttx | fungi, jeblair, clarkb: if the plan takes until end of week to land, I think plan A is faster, and time is a factor here | 18:59 |
| ttx | fungi: at least for this release yes | 18:59 |
| *** [HeOS] has joined #openstack-infra | 19:00 | |
| *** bswartz has quit IRC | 19:00 | |
| fungi | so for plan a (there is no proposed, only zuul^H^H^H^Hstable), we need to decide if we want to go with new groups and/or twiddle acls or group membership at release time | 19:02 |
| *** tqtran has joined #openstack-infra | 19:02 | |
| ttx | I'm drafting plan A at bottom of https://etherpad.openstack.org/p/zuul-cloner-branches | 19:03 |
| *** bswartz has joined #openstack-infra | 19:03 | |
| openstackgerrit | Julia Kreger proposed openstack-infra/shade: Add Ironic maintenance state pass-through https://review.openstack.org/172283 | 19:03 |
| *** sabeen1 has quit IRC | 19:03 | |
| openstackgerrit | Sean Dague proposed openstack-infra/elastic-recheck: fix query for bug 1285323 https://review.openstack.org/174030 | 19:04 |
| openstack | bug 1285323 in grenade "Services fail to shut down on the old side of Grenade" [Undecided,In progress] https://launchpad.net/bugs/1285323 - Assigned to Sean Dague (sdague) | 19:04 |
| ttx | fungi: is step 2 of the plan doable ? | 19:04 |
| *** eharney has joined #openstack-infra | 19:04 | |
| ttx | (see bottom of https://etherpad.openstack.org/p/zuul-cloner-branches) | 19:05 |
| fungi | i think they'd need to be reproposed | 19:05 |
| fungi | it _might_ be doable with some database modifications during a gerrit outage, but i wouldn't guarantee it (or feel safe about doing it without a lot of testing) | 19:06 |
| ttx | We have about 24 patches up at https://review.openstack.org/#/q/status:open+branch:proposed/kilo,n,z | 19:06 |
| *** tqtran has quit IRC | 19:06 | |
| dhellmann | I can spend some time resubmitting those when we have the right branches | 19:07 |
| ttx | I guess they can be manually reproposed | 19:07 |
| *** soren has joined #openstack-infra | 19:07 | |
| fungi | yeah, unique changes take target branch into account, so if you just try to push a new patchset but change the branch, you'll get an entirely separate change | 19:07 |
| openstackgerrit | Russell Bryant proposed openstack-infra/reviewstats: Catch exception on exec_command() https://review.openstack.org/174031 | 19:07 |
| ttx | fungi: it's just that we can't remove proposed/kilo until all review there are abandoned, right ? | 19:07 |
| *** luqas has joined #openstack-infra | 19:08 | |
| fungi | right | 19:08 |
| fungi | afaik there's no gerrit feature for moving a change to another branch, nor for renaming an existing branch | 19:08 |
| dhellmann | are we talking about renaming the existing branch, or making a new one? | 19:08 |
| ttx | so if it takes us 4 hours to repropose stuff, both branches coexist | 19:08 |
| ttx | dhellmann: same thing | 19:08 |
| ttx | dhellmann: you can't rename a branch, you create a different ref | 19:08 |
| ttx | and delete the old one | 19:09 |
| dhellmann | ok, that's what I thought | 19:09 |
| dhellmann | so we can make a new branch, resubmit against it and abandon them one at a time | 19:09 |
| ttx | I don't feel very comfortable with the idea of those branches coexisting for a long time, and I can't sign up to repropose them all in record time | 19:09 |
| openstackgerrit | Julia Kreger proposed openstack-infra/shade: Update register_machine to use tasks https://review.openstack.org/171324 | 19:09 |
| openstackgerrit | Julia Kreger proposed openstack-infra/shade: Update unregister_machine to use tasks https://review.openstack.org/171325 | 19:09 |
| ttx | If we have volunteers to do the reproposing job in the coming hours, I guess plan A is doable | 19:10 |
| lifeless | pc_m: hi? | 19:10 |
| ttx | I'd like to close most of this today if possible. Traveling tomorrow | 19:10 |
| ttx | and meeting all day | 19:10 |
| ttx | fungi: if we abandon them and remove branch, can we still check out the proposed change as a step to repropose it ? Or is it gone ? | 19:11 |
| dhellmann | ttx: my idea was to abandon them after they had been resubmitted | 19:11 |
| fungi | you can still check them out and repropose them to another branch even if already abandoned | 19:12 |
| fungi | but whichever | 19:12 |
| dhellmann | ok, well, that works too | 19:12 |
| *** luqas has quit IRC | 19:12 | |
| fungi | okay, i can propose the acl changes to add a refs/heads/stable/kilo section to each repo which duplicates the permissions from refs/heads/proposed/* | 19:12 |
| fungi | if this is the path forward | 19:12 |
| ttx | Sounds like the simplest way | 19:12 |
| ttx | I'm +1 on it | 19:12 |
| dhellmann | I'll start pulling these patches down | 19:12 |
| ttx | dhellmann: ? | 19:12 |
| openstackgerrit | Julia Kreger proposed openstack-infra/shade: Add Ironic machine power state pass-through https://review.openstack.org/172284 | 19:12 |
| dhellmann | ttx: ++ let's go with plan A | 19:12 |
| fungi | jeblair: clarkb: any last-minute objections? | 19:12 |
| *** stevemar has joined #openstack-infra | 19:13 | |
| *** vlaza has joined #openstack-infra | 19:13 | |
| ttx | fungi: assuming we are 99.9% sure that will work :) | 19:13 |
| dhellmann | ttx: should we try one project with a small number of patches first? | 19:13 |
| ttx | dhellmann: don't know how to test that "things are operating correctly" | 19:13 |
| ttx | since the test that doesn't operate correctly still reports success | 19:14 |
| dhellmann | well, sdague saw that the test job was checking out the wrong branch, right? | 19:14 |
| fungi | i am 100% sure the change reproposal part will work. i _expect_ the stable/kilo integration testing to work because, well, it will need to after release and there's no reason it shouldn't also work now | 19:14 |
| ttx | he has eagle eyes indeed | 19:14 |
| ttx | ok, let's move to #openstack-relmgr-office for execution of plan | 19:15 |
| dhellmann | ok | 19:15 |
| *** mrmartin has quit IRC | 19:17 | |
| openstackgerrit | Kyle Rockman proposed openstack-infra/jenkins-job-builder: Adding hipchat publisher https://review.openstack.org/174036 | 19:18 |
| *** jaypipes has quit IRC | 19:18 | |
| sdague | ttx: you just need to read logs :) | 19:19 |
| *** otherwiseguy has quit IRC | 19:20 | |
| ttx | sdague: I already don't read them when they FAIL | 19:20 |
| sdague | :) | 19:20 |
| ttx | now I have to read them when they SUCCEED! | 19:20 |
| *** spzala has joined #openstack-infra | 19:21 | |
| *** jaypipes has joined #openstack-infra | 19:21 | |
| anteaya | ha ha ha | 19:23 |
| *** peristeri has quit IRC | 19:24 | |
| *** ashleighfarnham has quit IRC | 19:25 | |
| *** dboik_ has joined #openstack-infra | 19:25 | |
| openstackgerrit | Merged openstack-infra/reviewstats: Sync infra projects to governance repo list https://review.openstack.org/173021 | 19:27 |
| *** dboik has quit IRC | 19:29 | |
| *** otherwiseguy has joined #openstack-infra | 19:29 | |
| *** david-lyle has joined #openstack-infra | 19:30 | |
| *** e0ne has joined #openstack-infra | 19:30 | |
| lifeless | mordred: around? can has +2+A on https://review.openstack.org/#/c/124651/ ? | 19:31 |
| *** armax has quit IRC | 19:31 | |
| *** vlaza has quit IRC | 19:35 | |
| *** vlaza has joined #openstack-infra | 19:35 | |
| jeblair | fungi: no, but i don't see the plan in the etherpad anymore | 19:38 |
| *** signed8bit_ZZZzz is now known as signed8bit | 19:38 | |
| fungi | i think ttx may have deleted it? | 19:38 |
| *** crc32 has joined #openstack-infra | 19:38 | |
| fungi | it's been integrated into https://etherpad.openstack.org/p/the-big-thaw | 19:38 |
| openstackgerrit | Sean Dague proposed openstack-infra/elastic-recheck: remove query for bug 1285323 https://review.openstack.org/174030 | 19:39 |
| openstack | bug 1285323 in grenade "Services fail to shut down on the old side of Grenade" [Undecided,In progress] https://launchpad.net/bugs/1285323 - Assigned to Sean Dague (sdague) | 19:39 |
| *** signed8bit is now known as signed8bit_ZZZzz | 19:39 | |
| *** e0ne has quit IRC | 19:41 | |
| *** koolhead17 has quit IRC | 19:42 | |
| *** alex_xu has quit IRC | 19:48 | |
| *** alex_xu has joined #openstack-infra | 19:49 | |
| *** markvoelker has joined #openstack-infra | 19:50 | |
| *** spzala has quit IRC | 19:50 | |
| *** kmartin has quit IRC | 19:51 | |
| *** Rockyg has joined #openstack-infra | 19:51 | |
| elarson | not sure if this is the right place to ask, but using jenkins job builder, I get an error when I configure a scm to do a clean up with scm: clean: after: true | 19:52 |
| elarson | I noticed the docs mention that flag as an extension, but I didn't see anything outside verifying the jenkins git plugin is installed. | 19:53 |
| elarson | apologies if this isn't the best channel to ask about jenkins job builder. it's a shot in the dark | 19:54 |
| anteaya | elarson: this is the best channel to ask about jenkins job builder | 19:54 |
| anteaya | elarson: have you a stacktrace you can paste to paste.openstack.org or another paste service? | 19:54 |
| *** markvoelker has quit IRC | 19:55 | |
| elarson | anteaya: sure | 19:57 |
| elarson | anteaya: http://paste.openstack.org/show/204034/ | 19:58 |
| openstackgerrit | Ryan Carey proposed openstack-infra/jenkins-job-builder: Add Artifactory plugin support. https://review.openstack.org/173932 | 19:58 |
| anteaya | elarson: what command produced this stacktrace? | 19:59 |
| elarson | anteaya: jenkins-jobs --conf /jjb/jenkins_jobs.ini --ignore-cache update /jjb/designate --delete-old | 19:59 |
| openstackgerrit | Ryan Carey proposed openstack-infra/jenkins-job-builder: Artifactory plugin https://review.openstack.org/173932 | 19:59 |
| anteaya | the after: true seems to be the issue, is that part of what you created, where did you come up with that key: value pair? | 20:00 |
| *** dustins has quit IRC | 20:00 | |
| *** smccully has quit IRC | 20:01 | |
| anteaya | can you paste the part of the file where that code occurs? with a line or two before and after | 20:01 |
| elarson | anteaya: I didn't write it originally, but I believe http://ci.openstack.org/jenkins-job-builder/scm.html#scm.git is where it came from (down a bit in the extensions section) | 20:01 |
| elarson | anteaya: here is the section that breaks http://paste.openstack.org/show/204036/ | 20:02 |
| *** mrmartin has joined #openstack-infra | 20:03 | |
| *** crc32 has quit IRC | 20:05 | |
| anteaya | elarson: that syntax appears to match the expected syntax from the test: http://git.openstack.org/cgit/openstack-infra/jenkins-job-builder/tree/tests/scm/fixtures/git-clean01.yaml | 20:06 |
| anteaya | I don't know where to go from here to offer assistance | 20:07 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config: Temporarily copy proposed/* ACLs to stable/kilo https://review.openstack.org/174074 | 20:07 |
| anteaya | zaro: are you about? | 20:07 |
| mtreinish | anteaya: if you get a sec mriedem was bugging me about: https://review.openstack.org/#/c/173319/1 | 20:08 |
| mriedem | heh | 20:08 |
| mtreinish | although besides a +1 there wasn't much I could do for him | 20:08 |
| *** crc32 has joined #openstack-infra | 20:09 | |
| *** dustins has joined #openstack-infra | 20:09 | |
| elarson | anteaya: I think we found it. it is deprecated. I can use wipe-workspace: true instead | 20:09 |
| elarson | anteaya: thanks so much for looking into it! | 20:09 |
| anteaya | ah | 20:11 |
| anteaya | elarson: good work | 20:11 |
| anteaya | we have a chorus line of +1's there | 20:11 |
| *** teran has joined #openstack-infra | 20:12 | |
| anteaya | you wrote all 3 patches I see | 20:12 |
| anteaya | why is ceph gating at all? | 20:13 |
| anteaya | they are a third party job | 20:13 |
| anteaya | which are open source so happen to be running by us, and I have no problem with them voting in check, but why are they in the gate? | 20:14 |
| anteaya | I am very against third party anything in the gate | 20:14 |
| anteaya | mriedem: ^ | 20:14 |
| clarkb | anteaya its not third party | 20:15 |
| anteaya | I thought it was | 20:15 |
| anteaya | that is what I have been told | 20:15 |
| clarkb | its us running it, cinder and nova wanted it to vote | 20:15 |
| anteaya | ceph and sheepdog and glusterfs | 20:15 |
| anteaya | right | 20:15 |
| fungi | ceph is not an openstack project, but that doesn't mean we can't gate on it | 20:15 |
| fungi | mysql and postgresql are not openstack projects either, but we gate on whether or not our code works with them | 20:16 |
| anteaya | I have been told that they are being treated as being in the same category | 20:17 |
| mriedem | anteaya: so...if you want the dirty details it's in the cinder meeting logs from last week when we talked about making the ceph job voting on nova/cinder/glance changes in the gate | 20:18 |
| mriedem | it's not 3rd party, and it's a job that's already hitting all 3 projects and is the only project in infra that is testing nova with shared storage | 20:18 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config: Configure stable branch ACLs for non-Oslo libs https://review.openstack.org/173892 | 20:18 |
| * anteaya goes to read the dirty details in the cinder meeting logs from last week | 20:18 | |
| mriedem | and it's also the top used cinder backend according to the user survey | 20:18 |
| anteaya | I'm just gathering resources when I get hammered by third party operators who yet again want to make the case that they should be in the gate | 20:19 |
| anteaya | we go through this often | 20:19 |
| clarkb | anteaya they have to make their case to the projects | 20:19 |
| clarkb | ceph did so | 20:19 |
| clarkb | also its open source and doesnt require special hardware | 20:19 |
| mriedem | for closed source vendor backends the response is easy | 20:19 |
| anteaya | great, then we just tell them to hammer the ptls | 20:19 |
| anteaya | mriedem: please state it for the record | 20:20 |
| anteaya | clarkb: and the ptls and me are getting tired of being hammered | 20:20 |
| *** tkelsey has joined #openstack-infra | 20:20 | |
| mriedem | state that infra isn't going to run a storwize cinder backend? | 20:20 |
| openstackgerrit | Ryan Carey proposed openstack-infra/jenkins-job-builder: Artifactory plugin https://review.openstack.org/173932 | 20:20 |
| mriedem | or db2? | 20:20 |
| clarkb | then stop drmanding everyone test their drivers... the ptls made that decision... | 20:20 |
| mriedem | if it's closed source and the distros don't have packages to run the stuff, infra shouldn't be required to run the jobs | 20:20 |
| anteaya | mriedem: thank you | 20:21 |
| *** adreznec has joined #openstack-infra | 20:21 | |
| *** mfink has joined #openstack-infra | 20:21 | |
| clarkb | (also it doesnt have to be hammering the ptl, it could be a ml discussion or a meeting agenda item as was in this case) | 20:22 |
| mriedem | is it too late for me to run for ptl? :) | 20:22 |
| fungi | mriedem: almost a week too late | 20:22 |
| fungi | mriedem: though there's still time to make a bid for tc! | 20:23 |
| fungi | mriedem: or you can get a head start on your campaign for october | 20:23 |
| mtreinish | mriedem: well you've used a storewize right? It's something I don't have fond memories of... | 20:23 |
| AJaeger_ | anteaya: change was https://review.openstack.org/#/c/170913/ | 20:23 |
| mriedem | mtreinish: i haven't | 20:23 |
| mriedem | fungi: too much work probably | 20:24 |
| mtreinish | mriedem: aren't you already ptl for qa? | 20:24 |
| mriedem | sometimes | 20:24 |
| fungi | doppleganger! | 20:24 |
| *** tkelsey has quit IRC | 20:24 | |
| clarkb | I think we should not discourage testing when we are able to address a need | 20:25 |
| *** kmartin has joined #openstack-infra | 20:25 | |
| fungi | clarkb: jeblair: mordred: the new world release order cordially requests your review of these stylish and sweeping acl changes... https://review.openstack.org/173892 https://review.openstack.org/174074 | 20:26 |
| *** dboik_ has quit IRC | 20:26 | |
| *** dboik has joined #openstack-infra | 20:26 | |
| openstackgerrit | Ryan Carey proposed openstack-infra/jenkins-job-builder: Add support for Artifactory plugin https://review.openstack.org/173932 | 20:26 |
| fungi | clarkb: jeblair: mordred: oh, and perhaps also https://review.openstack.org/173075 | 20:27 |
| clarkb | though we should consider rolling ceph testing into an existing job (I didnt want to make that a req before release since we are a bit time sensitive) | 20:27 |
| openstackgerrit | Ryan Carey proposed openstack-infra/jenkins-job-builder: Add support for Artifactory plugin https://review.openstack.org/173932 | 20:27 |
| *** dizquierdo has joined #openstack-infra | 20:28 | |
| *** hdd has joined #openstack-infra | 20:28 | |
| openstackgerrit | Ryan Carey proposed openstack-infra/jenkins-job-builder: Add support for Artifactory plugin https://review.openstack.org/173932 | 20:28 |
| *** vlaza has quit IRC | 20:28 | |
| *** vlaza1 has joined #openstack-infra | 20:28 | |
| *** bswartz has quit IRC | 20:28 | |
| *** dboik has quit IRC | 20:28 | |
| *** dboik has joined #openstack-infra | 20:29 | |
| *** AJaeger_ has quit IRC | 20:29 | |
| *** vlaza1 is now known as vlaza | 20:29 | |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/project-config: Enable python3 jobs for oslo.messaging https://review.openstack.org/174088 | 20:30 |
| *** baoli has joined #openstack-infra | 20:30 | |
| *** whoops has joined #openstack-infra | 20:30 | |
| openstackgerrit | Timo Tijhof proposed openstack-infra/zuul: Status: Don't raise "abort" as error to the user https://review.openstack.org/172241 | 20:32 |
| anteaya | just to clarify from the log 16:11:02 <thingee> I think Ceph is unique situation. It's integrated in a variety of projects and maintained by infra. | 20:32 |
| anteaya | the ceph job is run by the openstack infrastructure, ceph isn't maintained by infra | 20:33 |
| clarkb | I think he means the job is maintained upstream | 20:33 |
| clarkb | it was initially written by russellb iirc | 20:34 |
| russellb | yeah | 20:35 |
| anteaya | 16:13:42 <thingee> xyang2: I think the idea is eventually CI's would vote. they're just not usually stable | 20:35 |
| russellb | actively maintained by someone else now | 20:35 |
| anteaya | I have consistently said that ci's won't be in the gate | 20:35 |
| anteaya | since it comes up all the time | 20:35 |
| anteaya | they all want to be in the gate | 20:35 |
| asselin | which ci's are in the gate? they should at most be in 'check' | 20:36 |
| *** spzala has joined #openstack-infra | 20:36 | |
| clarkb | I think any test we can run can gate if prohects are happy with that | 20:36 |
| anteaya | 16:15:41 <e0ne> imo, any _stable_ ci must bt voting | 20:37 |
| *** sarob has quit IRC | 20:37 | |
| fungi | jobs being run on someone else's (not upstream openstack's) infrastructure shouldn't be voting in the gate pipeline. jobs being run on openstack's infrastructure can do so if the projects they're voting on want that | 20:37 |
| *** sarob has joined #openstack-infra | 20:37 | |
| *** mattt has joined #openstack-infra | 20:37 | |
| anteaya | no idea who e0ne is | 20:37 |
| clarkb | I dont tell cinder and nova what to gate one we merely make sure that the jobs run | 20:37 |
| anteaya | just saying it will open a flood gate | 20:37 |
| fungi | the problem is that the term "third party ci" has become ubiquitous and people are referring to jobs for optional drivers/plugins as "third-party" now even when they're not running in a third-party's ci infrastructure | 20:38 |
| anteaya | yes | 20:38 |
| *** tonytan4ever has quit IRC | 20:38 | |
| fungi | the ceph job is not "third party ci" it's "an openstack ci job" | 20:38 |
| anteaya | that is not how it is percieved in cinder | 20:39 |
| * anteaya continues to read the cinder meeting logs | 20:39 | |
| fungi | i don't care if they paint the sky green personally, doesn't change reality | 20:39 |
| *** palendae has joined #openstack-infra | 20:40 | |
| anteaya | I agree | 20:41 |
| anteaya | however that is not how they see things | 20:41 |
| *** stevemar has quit IRC | 20:41 | |
| anteaya | 16:22:36 <DuncanT> thingee: I'm not totally opposed, as long as (a) the goal is to do this with other drivers too (b) we can turn off the voting easily if the job gets unstable | 20:42 |
| anteaya | DuncanT fully expects this to be the start of a direction, not the exception | 20:42 |
| fungi | yep, so long as "other drivers" are, like ceph, something that we can test upstream, then that's entirely possible | 20:43 |
| anteaya | 16:23:47 <DuncanT> thingee: If the plan is not to allow other stable CIs to also vote with similar criteria for stability (whatever we pick), then I'm very much opposed | 20:43 |
| anteaya | fungi: I'm not feeling that caveat in DuncanT's assertion | 20:43 |
| clarkb | it came up later in that discussion | 20:44 |
| clarkb | that has never been negotiable | 20:44 |
| *** gyee has joined #openstack-infra | 20:44 | |
| anteaya | and thingee is in agreement with him: 16:24:07 <thingee> DuncanT: I think that would be the goal eventually to allow others. That's the point of this discussion | 20:44 |
| *** gyee has quit IRC | 20:44 | |
| anteaya | clarkb: I know that but it appears they don't | 20:44 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config: oslo.messaging: enable Python 3.4 check https://review.openstack.org/172221 | 20:45 |
| anteaya | 16:26:12 <thingee> So one thing being opposed to drivers for me is we have around 45 drivers (counting all supported protocols) | 20:45 |
| fungi | well, anyway, there's nothing about that ceph job that's particularly different from anything else we currently have voting in the gate. our testing incorporates all manner of software we don't write. if there are jobs we can run upstream and projects want them voting, then that's pretty much the end of it | 20:46 |
| anteaya | okay so having to head off the herd beliveing this is the start of 100+ cis in the gate isn't really filling me with warm thoughts | 20:46 |
| anteaya | fungi: right | 20:47 |
| anteaya | except that cinder has expectations that are unrealistic | 20:47 |
| *** belmoreira has joined #openstack-infra | 20:47 | |
| fungi | yeah, them and lots of other people | 20:47 |
| *** dustins has quit IRC | 20:48 | |
| *** baoli has quit IRC | 20:48 | |
| *** baoli has joined #openstack-infra | 20:49 | |
| *** otter768 has joined #openstack-infra | 20:49 | |
| anteaya | right, but I'm just evaluating my actions in terms of their unrealistic expectations at the moment | 20:49 |
| fungi | this has come up many times before (docker, glusterfs, et cetera). openstack infra is willing to assist free software authors in devising jobs which integrate their components if those are things projects want jobs for. and when those jobs cease to be desirable they can be made non-voting again or removed as needed | 20:49 |
| *** thedodd has quit IRC | 20:49 | |
| *** spzala has quit IRC | 20:49 | |
| *** spzala has joined #openstack-infra | 20:50 | |
| anteaya | if you don't want me to take this time to review this patch the way I'm comfortable go on ahead, you don't need me | 20:50 |
| anteaya | approve and move on | 20:50 |
| *** dboik_ has joined #openstack-infra | 20:51 | |
| *** mrmartin has quit IRC | 20:51 | |
| anteaya | I haven't -1'd a thing | 20:51 |
| fungi | that wasn't my intent at all. simply suggesting that the infra team should attempt to remain neutral when it comes to projects deciding what sorts of jobs make sense to test their software | 20:51 |
| anteaya | right | 20:51 |
| anteaya | and I am trying to be neutral | 20:51 |
| russellb | i seem to be pretty consistently getting "Too many concurrent connections" from gerrit ... is that a global thing, or a per-source thing? | 20:52 |
| anteaya | but having worked with third party folks and having heard them talk about being in the gate for some time, I am concerned about what action they will take as a result | 20:52 |
| fungi | russellb: what's emitting that error? ssh api calls? | 20:52 |
| *** viglesias has quit IRC | 20:52 | |
| russellb | fungi: well, was trying to get the commit hook this instant | 20:53 |
| russellb | so ssh, yeah | 20:53 |
| fungi | as in git review -s? | 20:53 |
| fungi | that's no good. checking logs now | 20:53 |
| *** SumitNaiksatam has quit IRC | 20:53 | |
| russellb | yep | 20:54 |
| russellb | i wonder if my last change to reviewstats has made it go crazy | 20:54 |
| *** otter768 has quit IRC | 20:54 | |
| russellb | and i'm getting blocked :) | 20:54 |
| fungi | perhaps. i'm checking show-connections output. it's taking a _while_ to return details | 20:54 |
| *** sarob has quit IRC | 20:54 | |
| fungi | i hope this isn't courtesy of our new patch to provide more details in the show-connections list output | 20:55 |
| *** dboik has quit IRC | 20:55 | |
| fungi | pewpew | 20:56 |
| fungi | it works, just takes a few minutes | 20:56 |
| fungi | oh, i wonder if it's just now coming down off something | 20:57 |
| openstackgerrit | Russell Bryant proposed openstack-infra/reviewstats: Catch exception on exec_command() https://review.openstack.org/174031 | 20:57 |
| *** spzala has quit IRC | 20:57 | |
| russellb | last iteration of that didn't sleep before retrying the connection, oops | 20:58 |
| russellb | could have got into a bad loop | 20:58 |
| fungi | if it just stopped a couple minutes ago, that might explain the behavior i was seeing too | 20:58 |
| russellb | yes, i just killed it | 20:58 |
| russellb | couple minutes ago | 20:58 |
| russellb | sorry :( | 20:58 |
| fungi | i think it may have been starving gerrit's ssh interface | 20:59 |
| russellb | yeah | 20:59 |
| *** spzala has joined #openstack-infra | 20:59 | |
| fungi | because show-connections is snappy again nopw | 20:59 |
| russellb | in better news, I did recently change it so it doesn't download the entire review history of every project every day :-X | 20:59 |
| fungi | heh | 20:59 |
| anteaya | 16:30:09 <mriedem> DuncanT: i think that's the goal for all jobs that infra hosts which are non-voting today | 20:59 |
| russellb | fungi: https://review.openstack.org/#/c/172550/ | 20:59 |
| anteaya | do we have an infra goal of having open source jobs we run in the gate? | 21:00 |
| russellb | fungi: sorry for the disturbance :) | 21:00 |
| *** gyee has joined #openstack-infra | 21:00 | |
| *** SumitNaiksatam has joined #openstack-infra | 21:01 | |
| mriedem | anteaya: i think cinder wants to get the glusterfs and sheepdog jobs voting at some point for cinder, which are already hosted on infra | 21:01 |
| fungi | anteaya: that sounds realistic to me. i don't know whether it's an explicit goal, but we do want to see openstack projects tested and we are able to provide the resources needed for them to test themselves with free software | 21:01 |
| anteaya | mriedem: that is a cinder goal, I didn't know infra had that as a goal | 21:01 |
| * anteaya didn't have that as a goal | 21:02 | |
| fungi | so if that's cinder's goal, that seems fine to me and is something we're able to provide the systems to make possible as long as developers step up to maintain those jobs | 21:02 |
| *** mattfarina has quit IRC | 21:02 | |
| nibalizer | jeblair: I saw some chatter that sounded like in your view "secure secret storage(hiera)" and zuulv3 are related somehow | 21:02 |
| nibalizer | is that your view? can you expand on that? | 21:03 |
| anteaya | I'm fine supporting cinder's goals whatever they are, but I'm not going to say they are my goals | 21:03 |
| anteaya | well thingee put this in the meeting: 16:31:59 <thingee> #agreed infra-hosted open-source cinder jobs will be eligable to vote in future once they've proved stable | 21:04 |
| anteaya | which is exactly what I was looking for | 21:04 |
| fungi | i should clarify when using the word "we" that these are things as a group we've come forward in the past saying we were willing to provide help to accomplish if they're things the projects want | 21:04 |
| *** sputnik13 has joined #openstack-infra | 21:04 | |
| * anteaya paints that on a sign to show the third party ci hoards | 21:04 | |
| *** duran has quit IRC | 21:04 | |
| sdague | mriedem: I think it's also important that any job has a sponsor on it | 21:05 |
| *** aysyd has quit IRC | 21:05 | |
| fungi | yes, that ^ | 21:05 |
| anteaya | and sdague had moved ahead and approved so you didn't need me anyway | 21:05 |
| fungi | hence my comment "as long as developers step up to maintain those jobs" | 21:05 |
| dhellmann | jeblair, mordred, clarkb: can one of you take a look at https://review.openstack.org/#/c/174074/ for us? We need the ACL changes there to finish setting up the stable branches (we can't approve anything into the branches now) | 21:05 |
| * anteaya moves off to do something else | 21:06 | |
| sdague | anteaya: which patch? | 21:06 |
| sdague | the one I just approved removed ceph from stable | 21:06 |
| fungi | sdague: the ceph job change | 21:06 |
| sdague | ok, but that was a delete | 21:06 |
| mriedem | anteaya: fungi: sdague: yes, fine print - devs must own the jobs, if they aren't we nack them | 21:06 |
| fungi | yep | 21:06 |
| sdague | mriedem: not just own jobs, I want a name of a person | 21:07 |
| sdague | and that person needs to be in -infra and -qa | 21:07 |
| fungi | also street address at which to target icbms | 21:07 |
| mtreinish | sdague: sounds like mriedem is volunteering :) | 21:07 |
| sdague | and if their job starts failing and they don't step up to debug fast, we turn it off | 21:07 |
| *** ZZelle has quit IRC | 21:07 | |
| mriedem | jbernard is the ceph guy | 21:07 |
| mriedem | yes | 21:07 |
| mriedem | that came up in the cinder meeting last week | 21:07 |
| sdague | mriedem: ok, and he'll have vacation coverage? | 21:08 |
| *** mwagner_lap has quit IRC | 21:08 | |
| mriedem | cells is basically alaski since !comstud :) | 21:08 |
| mriedem | sdague: then it's dansmith :) | 21:08 |
| fungi | he does seem to be lurking in the infra and qa channels | 21:08 |
| sdague | fungi: yes, he is | 21:08 |
| sdague | because I made that a requirement of my +2 :) | 21:08 |
| fungi | heh | 21:08 |
| sdague | in the ceph case, I'm kind of ok, I do think we have enough people that have now said they are fine being on the hook | 21:09 |
| sdague | for some of these others, I think we need to make it really clear you need to have folks around and engaged that are debugging these | 21:09 |
| mriedem | for other things, if they want to gate on a project, they need to get that project's permission | 21:09 |
| sdague | otherwise we nix them | 21:09 |
| mriedem | like if sheepdog wanted to vote on nova patches, we'd probably nack that right now | 21:09 |
| sdague | yeh, like how the docker job keeps getting demoted | 21:10 |
| fungi | getting that project's permissions is step the first. if they don't have that there's little point | 21:10 |
| sdague | because there are no owners | 21:10 |
| mriedem | step the first, i like it | 21:10 |
| sdague | mriedem: but I do think it's more than a project level concern, because we can be in blocking situations where we need a cinder fix landed | 21:10 |
| alaski | mriedem: me and bauzas and melwitt now :) | 21:11 |
| *** vlaza has quit IRC | 21:11 | |
| sdague | right, again, I feel like we've got a couple people legit stepping up on cells that are around regularlly, that's goodness :) | 21:11 |
| mriedem | sdague: yeah true | 21:11 |
| *** esker has quit IRC | 21:12 | |
| *** dprince has quit IRC | 21:12 | |
| *** Sukhdev has joined #openstack-infra | 21:13 | |
| openstackgerrit | Merged openstack-infra/project-config: Don't run gate-tempest-dsvm-full-ceph job on stable icehouse/juno https://review.openstack.org/173319 | 21:13 |
| openstackgerrit | gordon chung proposed openstack-infra/project-config: enable ceilometerclient functional tests on master only https://review.openstack.org/174103 | 21:14 |
| jeblair | fungi, dhellmann: has ttx signed off on https://review.openstack.org/173075 in principle? | 21:14 |
| dhellmann | jeblair: hmm, we didn't discuss it with him | 21:14 |
| *** ldnunes has quit IRC | 21:14 | |
| dhellmann | we can wait, if it makes you more comfortable | 21:15 |
| openstackgerrit | David Lenwell proposed openstack-infra/project-config: new-project https://review.openstack.org/174104 | 21:15 |
| jeblair | dhellmann: if it's not urgent / blocking the other stuff, let's do that. | 21:15 |
| fungi | jeblair: yeah, i didn't ask. in general he's seemed fine with whatever group a project's leadership wanted taking care of stable maintenance in the new order (since we removed the iron fist of stable-maint-core a while back) | 21:15 |
| *** esker has joined #openstack-infra | 21:15 | |
| *** yolanda has quit IRC | 21:15 | |
| dhellmann | jeblair: yeah, that one isn't blocking us | 21:15 |
| *** rfolco has quit IRC | 21:15 | |
| *** jkraj has quit IRC | 21:16 | |
| dhellmann | jeblair: in practice the stable team is going to be the core team, which is really only 3-4 people anyway, but I'm OK with waiting for his sign-off | 21:16 |
| *** yolanda has joined #openstack-infra | 21:16 | |
| jeblair | yeah, i feel like he wouldn't object, but would just like to keep him in the loop if it's no rush. | 21:16 |
| fungi | makes total sense | 21:16 |
| jeblair | dhellmann, fungi: aprvd https://review.openstack.org/#/c/174074/ | 21:17 |
| dhellmann | jeblair: thanks! | 21:17 |
| fungi | thanks. that at least gets us out of the danger zone of stable core groups mucking with pre-release branches | 21:17 |
| *** rlandy has quit IRC | 21:17 | |
| jeblair | nibalizer: i believe that for the purposes of providing credentials to _jobs_, we need a solution in zuul so that we can eliminate the remaining static build slaves and can use nodepool nodes for those jobs | 21:18 |
| fungi | also once that merges we can consider removing the refs/heads/proposed/* section from the all-projects acl, simplifying it and our documentation some more | 21:18 |
| *** rlandy has joined #openstack-infra | 21:19 | |
| *** hashar has quit IRC | 21:19 | |
| nibalizer | jeblair: makes sense, doesn't seem tightly coupled to encrypted hiera tho | 21:19 |
| openstackgerrit | David Lenwell proposed openstack-infra/project-config: new-project https://review.openstack.org/174104 | 21:19 |
| nibalizer | if I am understanding you correctly | 21:19 |
| gordc | hi folks, if free, can somoene validate patch to unblock ceilometerclient stable/kilo branch: https://review.openstack.org/#/c/174103/ | 21:20 |
| jeblair | nibalizer: i agree, i don't think it is. if we plumb the credentials through zuul, i think there's very little, possibly no, interaction with puppet and ehiera. | 21:21 |
| jeblair | nibalizer: basically, if the problem statement is "we need to be able to provide credentials to jobs" then i think this is the way we should look. | 21:21 |
| jeblair | nibalizer: if there is another (non-job-running) reason to encrypt hiera, then i don't think this idea addresses that. | 21:22 |
| jbernard | sdague: o/ | 21:23 |
| jeblair | nibalizer: but the chatter i have heard is "we need people to be able to provide passwords for the jobs that we run for them and we don't want them to send them to us in the clear", which seems to point in the direction of zuul | 21:23 |
| jbernard | sdague: ill be here and in -qa for any ceph related issues | 21:23 |
| mordred | jeblair: I agree with you | 21:24 |
| nibalizer | jeblair: ah, i think we crossed streams on you | 21:24 |
| nibalizer | we have a couple problems downstream | 21:24 |
| jeblair | don't cross the streams :) | 21:24 |
| nibalizer | well, we are the ghostbusters | 21:24 |
| mordred | who you gonna call? | 21:24 |
| nibalizer | so one problem is we need to centralize and code-review our changes to hiera, which is used exclusively for secret-storage | 21:25 |
| *** andreykurilin has joined #openstack-infra | 21:25 | |
| nibalizer | our code-review system doesn't have good acl support, so the plan is to just encrypt all the data in it and clunk along with manual reviews on the command line | 21:25 |
| *** dboik_ has quit IRC | 21:25 | |
| openstackgerrit | David Lenwell proposed openstack-infra/project-config: New repos for Akanda https://review.openstack.org/174104 | 21:26 |
| *** dizquierdo has quit IRC | 21:26 | |
| nibalizer | another problem is a cambrian explosion of jobs that require some form of secret, for which a zuul component that could inject the right secrets at the right time would be a good fit | 21:26 |
| *** dboik has joined #openstack-infra | 21:26 | |
| *** viglesias has joined #openstack-infra | 21:27 | |
| jeblair | i think i'm not opposed to adopting ehiera for general puppet secrets for openstack-infra if the other roots are okay with it. i don't think it's a big deal for us, but i don't think it would hurt. makes the DR story a little better. :) | 21:27 |
| nibalizer | cool | 21:27 |
| jeblair | i'd like mordred to review that to make sure it works with ansible-puppet | 21:28 |
| jeblair | and where all that is going | 21:28 |
| clarkb | how does master unlock it? | 21:28 |
| nibalizer | key on the filesystem | 21:28 |
| jeblair | nibalizer: is in asymmetric or symmetric? | 21:28 |
| clarkb | so its not more secure | 21:28 |
| jeblair | is it | 21:28 |
| mordred | I think it's no more secure than the current file | 21:28 |
| jeblair | clarkb: yeah, frankly i don't believe we can improve on the security here, just make it "safe" to actually publish the yaml file | 21:29 |
| nibalizer | this page has the details https://github.com/sihil/hiera-eyaml-gpg | 21:29 |
| mordred | the qusetion is does it provide the currently level of security with the ability for us to not have to shell in to server to add new keys | 21:29 |
| nibalizer | downstream we have multiple masters, so one of the problems we are trying to solve is keeping the hiera dir the same on all masters | 21:29 |
| nibalizer | pushing it into gerrit and then all masters pull and decrypt is a good way to solve that, we think | 21:30 |
| mordred | nibalizer: that will also get better with ansible+puppet-apply for you | 21:30 |
| mordred | btw | 21:30 |
| nibalizer | mordred: no it wont | 21:30 |
| mordred | yeah it will | 21:30 |
| mordred | you won't need two masters | 21:30 |
| clarkb | dat ssh | 21:30 |
| mordred | because you can ssh OUT to anything | 21:30 |
| sdague | jbernard: yep, like I said, I think we're covered for now in the ceph case. But just wanted to make sure people knew we really wanted sponsors on things | 21:30 |
| nibalizer | and we use only one master then... maybe yea | 21:30 |
| mordred | nibalizer: the puppetmaster model involves the agent connecting back to the master | 21:30 |
| lifeless | I'm trying to diagnose the failure on https://review.openstack.org/#/c/124651/ | 21:31 |
| mordred | so you need a master in each network zone | 21:31 |
| lifeless | but http://logs.openstack.org/51/124651/10/gate/gate-pbr-docs/635e6a5/ is a 404 | 21:31 |
| jeblair | mordred, clarkb: if it asymmetric (which i _think_ it is from a quick scan of the page) it would _theoretically_ allow non-roots to add credentials to the file, however, i'm not sure we would want to do that as a matter of policy since it would mean a non-root would _know_ a credential. | 21:31 |
| jeblair | nibalizer: ^ | 21:31 |
| mordred | jeblair: ++ | 21:31 |
| clarkb | +1 | 21:31 |
| jeblair | (that may be a _desired_ behavior downstream, but possibly not because i think most needs for that would be obviated by the zuul-credential path) | 21:31 |
| fungi | lifeless: this? https://jenkins06.openstack.org/job/gate-pbr-docs/143/console | 21:32 |
| mordred | yah. in fact, I think downstream should have less problems with needing to empower non-roots | 21:32 |
| mordred | but more problems with zuul-credentials | 21:32 |
| lifeless | fungi: thanks, how did you get to that? | 21:32 |
| fungi | lifeless: i was also looking at it while it was showing failed in the gate | 21:32 |
| mordred | nibalizer: in any case, I don't think ehiera breaks what we did towards puppet-apply | 21:32 |
| fungi | lifeless: "curl: (22) The requested URL returned error: 502 Proxy Error" means jenkins06 may be struggling | 21:33 |
| mordred | nibalizer: assuming that the ehiera stuff still works with apply | 21:33 |
| clarkb | mordred not quite | 21:33 |
| mordred | nibalizer: it means our "copy the appropriate hiera files" code will need to also copy the key | 21:33 |
| nibalizer | mordred: yep | 21:33 |
| mordred | but, as before, that's no less secure than what we're doing today | 21:33 |
| clarkb | mordred and it has to be key per node | 21:33 |
| mordred | clarkb: nope | 21:33 |
| mordred | clarkb: one key - lives everywhere | 21:33 |
| lifeless | fungi: so, whats the current incantation to retry a +A ? | 21:34 |
| clarkb | that would allow every node access to all secrets | 21:34 |
| mordred | yup. | 21:34 |
| mordred | but that is true now | 21:34 |
| clarkb | thats a bug | 21:34 |
| mordred | it's not | 21:34 |
| nibalizer | mordred: as for 'do we still need multiple masters' thats a more internal question than -infra really needs to hear but my initial response is that each major network zone should have its own control node | 21:34 |
| clarkb | no its not | 21:34 |
| clarkb | puppet master doesnt allow arbitrary lookups | 21:34 |
| mordred | clarkb: neither does this | 21:34 |
| *** achanda has quit IRC | 21:34 | |
| mordred | clarkb: have you seen what the hiera solution is for this? | 21:34 |
| fungi | lifeless: yeah, remove and readd your workflow +1 (or add a second workflow +1 if yours was not the original) | 21:34 |
| mordred | clarkb: it's already covered | 21:34 |
| clarkb | mordred it does because the data is in gerrit | 21:34 |
| clarkb | mordred its not ... | 21:35 |
| mordred | clarkb: AHHH | 21:35 |
| mordred | thank you | 21:35 |
| mordred | that was the missing piece | 21:35 |
| jeblair | nibalizer: also, if the zuul-credential path is needed before zuulv3, i think that's an option. it may involve some work on gearman-jenkins-plugin too, but other than that, could be a separate workitem from the rest of zuulv3. iow, it is required for zuulv3, but zuulv3 is not required for it. just a matter of prioritization. | 21:35 |
| mordred | jeblair, clarkb: in that case, I may want to revert my thinking - I think it's too complex for us | 21:35 |
| nibalizer | yea hrm | 21:36 |
| mordred | I do not want key-per-host | 21:36 |
| clarkb | mordred I agree | 21:36 |
| jeblair | mordred: your solution involves copying host-specific hiera data to the host, right? | 21:36 |
| mordred | jeblair: yes - but the overall file would be publically fetchable over the internet from git.o.o in the ehiera model | 21:36 |
| jeblair | mordred: so what about decrypting on master and then copying only that host's subset to the host? | 21:36 |
| fungi | mordred: worth noting, we do already have unique keys for those hosts to which we could encrypt, though we may still not want to because complicated | 21:37 |
| thingee | anteaya, fungi, sdague: so mriedem and dansmith assured me that this is a community supported CI. I think it's important to define what community means there. I'm assuming jbernard is part of this. | 21:37 |
| mordred | jeblair: ooh | 21:37 |
| jeblair | mordred: so hosts never see the keys | 21:37 |
| mordred | nibalizer: ^^ | 21:37 |
| mordred | jeblair: yes.. I think that's sane and acceptable | 21:37 |
| anteaya | thingee: definitions of words are good | 21:37 |
| fungi | thingee: yes, we're all part of this. it's the openstack ci | 21:37 |
| *** achanda has joined #openstack-infra | 21:37 | |
| *** achanda has quit IRC | 21:37 | |
| nibalizer | mordred: jeblair interesting | 21:38 |
| nibalizer | im not sure you'd want to use heira-gpg there then | 21:38 |
| *** achanda has joined #openstack-infra | 21:38 | |
| nibalizer | you'd probably want to do it more traditionally | 21:38 |
| anteaya | thingee: if the ceph job breaks I expect a ceph dev to offer a patch to fix it | 21:38 |
| openstackgerrit | gordon chung proposed openstack-infra/project-config: enable ceilometerclient functional tests on master only https://review.openstack.org/174103 | 21:38 |
| mordred | nibalizer: what do you mean? | 21:38 |
| *** doug-fish has left #openstack-infra | 21:38 | |
| fungi | the openstack project infrastructure is maintained by the openstack community at large, and facilitated by the project infrastructure team. hopefully the devs working on ceph integration in cinder are part of our community | 21:38 |
| mordred | nibalizer: so - let's start from a set of steps | 21:38 |
| clarkb | mordred: basically we aren't using ehiera at that point we are just decrypting a tar of files | 21:38 |
| mordred | nibalizer: step 1 - add encrypted values into a hiera file so that puppetmaster decrypts individual values before providing them to clients | 21:39 |
| mordred | nibalizer: step 2 - split hiera files into mutliple files, but puppetmaster still ehiera's them | 21:39 |
| gordc | fungi: thanks for review, i hope this is what you meant: https://review.openstack.org/#/c/174103/ | 21:40 |
| mordred | nibalizer: step 3 - puppetmaster goes away, _something_ decrypts individual values (not whole files) before copying file content to hosts | 21:40 |
| mordred | nibalizer, clarkb: ^^ right? | 21:40 |
| *** armax has joined #openstack-infra | 21:40 | |
| nibalizer | mordred: yea that could work | 21:40 |
| * mordred doesn't care if ehiera does the decrypting in step 3 - but the format should be such that if it was run through a puppetmaster, ehiera would do the right thing | 21:40 | |
| nibalizer | unfortunately the ehiera utility is ruby | 21:40 |
| nibalizer | mordred: so thats the question i think | 21:40 |
| jeblair | but it's just using gpg, right? | 21:41 |
| mordred | nod | 21:41 |
| nibalizer | do you use the ehiera encrypted format | 21:41 |
| clarkb | mordred: I wouldn't encrypt individual values | 21:41 |
| fungi | gordc: i think your editor may have messed up your indentation there | 21:41 |
| clarkb | I would encrypt a tarball | 21:41 |
| mordred | clarkb: that's how ehiera works | 21:41 |
| clarkb | and splat that onto the master | 21:41 |
| mordred | clarkb: and that's the point of this | 21:41 |
| nibalizer | or do you gpg /armor the entire file | 21:41 |
| clarkb | mordred: right but this model is different | 21:41 |
| mordred | clarkb: the point is so that only the secret value goes into code review | 21:41 |
| clarkb | mordred: because we are using ansible to copy files | 21:41 |
| gordc | fungi: ah i see it now... apologies | 21:41 |
| mordred | so that you can see what the key names are | 21:41 |
| mordred | clarkb: no - that's not a problem we need solved | 21:41 |
| clarkb | mordred: you can ou just have to pull them locally | 21:41 |
| mordred | the idea is that keynames are still readable | 21:41 |
| nibalizer | jeblair: yes there is nothing magical, just that it has interspersed encrypetd and non encrypted data | 21:41 |
| mordred | by people without the key | 21:41 |
| nibalizer | so you have to make a software to do that | 21:42 |
| nibalizer | which probably isn't super hard | 21:42 |
| openstackgerrit | gordon chung proposed openstack-infra/project-config: enable ceilometerclient functional tests on master only https://review.openstack.org/174103 | 21:42 |
| nibalizer | but ehiera is the utility kindof | 21:42 |
| mordred | clarkb: if we're goign to have a completely opaque blob, there is zero point to putting it in gerrit | 21:42 |
| nibalizer | i have to run off and do a thing | 21:42 |
| jeblair | nibalizer: yeah, i like that it's just some syntax around stardard gpg | 21:42 |
| clarkb | mordred: I think there is zero point regardless | 21:42 |
| nibalizer | ill come back and we can bang things out | 21:42 |
| nibalizer | timrc: i think there is value to you proposing a spec for this upstream | 21:42 |
| clarkb | mordred: this will only serve to make our config management more complicated without any benefit to security | 21:42 |
| mordred | clarkb: I believe I'm mostly inclined to agree with you. otoh - I don't think it will hurt security | 21:43 |
| jbernard | thingee: yep, ill do my best to keep it healthy | 21:43 |
| jeblair | nibalizer, timrc: ++ it sounds like splitting into two parts -- stuff for puppet and stuff for jobs would help | 21:43 |
| mordred | so I think it's tenable | 21:43 |
| gordc | s | 21:43 |
| *** dizquierdo has joined #openstack-infra | 21:43 | |
| * timrc blinks | 21:43 | |
| mordred | whether it's desirable for us is a different question | 21:43 |
| gordc | whoops wrong window. | 21:43 |
| jeblair | clarkb: yeah, it's not a security benefit. but it does mean we can put our secret yaml file in public git | 21:43 |
| fungi | jeblair: nibalizer: mordred: clarkb: uunfortunately the values when encrypted will need a fair amount of padding to avoid revealing their relative lengths, which means the lines will be seriously long i expect | 21:44 |
| jeblair | clarkb: which is not something we've been clamoring for, but, honestly, it is good for disaster recovery, system visibility, and lets us do things like "Depends-On: some change that adds a credential" | 21:44 |
| timrc | hiera-eyaml-gpg looks pretty nice... | 21:44 |
| clarkb | jeblair: I am not sure I find that particularly beneficial when balanced against the extra complications it introduces | 21:44 |
| jeblair | fungi: yeah, that could be a big drawback | 21:44 |
| mordred | I thinkk Depends-On: change that adds a credential is the biggest benefit for us | 21:45 |
| timrc | the default pkcs7 was rather limited... | 21:45 |
| mordred | the other things are largely meh-ish | 21:45 |
| timrc | it made decrypting a review a lot more awkard | 21:45 |
| timrc | Unless copying around private keys is how you roll. | 21:45 |
| *** tiswanso has quit IRC | 21:45 | |
| timrc | You'd need to invent some weird thing to log into some central place where the private key exists to decrypt stuff. | 21:45 |
| jeblair | yeah, honestly, i'd probably never decrypt a change in review. i'd just look at it and say "yep, it adds a key". | 21:46 |
| *** dims_ has joined #openstack-infra | 21:46 | |
| mordred | ++ | 21:46 |
| *** jamielennox|away is now known as jamielennox | 21:46 | |
| *** Rockyg has quit IRC | 21:46 | |
| timrc | jeblair, Yeah to me the most critical thing is making sure the key is correct... value? Maybe not as important, though jesusaurus brings up a good point about wanting to make sure the certificate you encrypted is actually valid. | 21:46 |
| notmyname | what's the policy for updating a dependency version during an RC? (asking for a friend) | 21:46 |
| *** andreykurilin has quit IRC | 21:46 | |
| clarkb | notmyname ffe to requirements | 21:47 |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck: Add fingerprint for bug 1442272 https://review.openstack.org/173591 | 21:47 |
| openstack | bug 1442272 in neutron "Ovs monitor tests fail with eventlet timeout" [High,In progress] https://launchpad.net/bugs/1442272 - Assigned to Maru Newby (maru) | 21:47 |
| fungi | we could in theory use delegate keys or encrypt to multiple keys (and boy would that be bloated in the file) but i agree that i would be unlikely to bother | 21:47 |
| jeblair | notmyname: are you going to propose the change for your friend too? ;) | 21:47 |
| *** annegentle has quit IRC | 21:47 | |
| notmyname | jeblair: ;-) | 21:47 |
| notmyname | clarkb: thanks | 21:47 |
| timrc | So anyway... I should probably read everything before someone though to mention 'timrc' | 21:48 |
| *** dims has quit IRC | 21:48 | |
| timrc | thought* | 21:48 |
| *** mwagner_lap has joined #openstack-infra | 21:48 | |
| *** whoops has quit IRC | 21:49 | |
| jeblair | fungi, clarkb: so i'm seeing some very slight benefits for this (visibility). there may be some drawbacks (like file size), and a lot of stuff that doesn't really change at all (security). | 21:49 |
| jeblair | fungi, clarkb: is it reasonable to write up a spec and evaluate those further? | 21:49 |
| jeblair | i'd like to see what it actually looks like and what it would entail before making a final decision | 21:49 |
| lifeless | jogo: ping | 21:50 |
| lifeless | jogo: you say we can't use pip install -r .... successfully. Why not? | 21:50 |
| mordred | yeah. at the very least, if it's a thing that downstreams want to use, and we need to add $something to allow them to do so even if we don't want to use the feature, and that thing is not particularly onerous, I think that's worth consideration too | 21:50 |
| fungi | jeblair: i'm not opposed to the idea as stated. on the fence about possible benefits but a more detailed description might help me decide | 21:50 |
| *** dkranz has quit IRC | 21:51 | |
| clarkb | mordred: I don't think we need to add anything, they can use ehiera transparent to us | 21:51 |
| clarkb | so my concern is in the 5 minutes of discussion here it has already been confusing to figure out where we can leak credentials | 21:51 |
| jogo | lifeless: because of transitive dependency resolution | 21:51 |
| clarkb | the current system does not leak them (at least by design and our understanding of said design) and has worked well | 21:52 |
| lifeless | jogo: I don't understand | 21:52 |
| clarkb | the current system also does not prevent anyone else from using ehiera | 21:52 |
| jogo | lifeless: if reqs.txt is pulled directly from pip freeze | 21:52 |
| lifeless | jogo: In my head it will work. Is there a bug or example case where it fails. | 21:52 |
| jeblair | clarkb: yeah, that's why i want to divert this to a conversation about having a spec to think this all out rather than a conversation about whether we use it. | 21:52 |
| clarkb | if someone wants to design how it may be used I don't have a problem with that | 21:52 |
| *** annegentle has joined #openstack-infra | 21:52 | |
| mordred | clarkb: I agree - except that I think there _will_ be a difference when we move to puppet-apply | 21:52 |
| jeblair | clarkb: i mean, it's a security topic, we ought to really get a lot of people thinking about it. | 21:52 |
| *** tqtran has joined #openstack-infra | 21:52 | |
| *** dboik_ has joined #openstack-infra | 21:52 | |
| clarkb | it just seems odd to spin cycles on a non issue | 21:53 |
| clarkb | but maybe I am not understanding what the issue is | 21:53 |
| mordred | clarkb: that's what I was thinking about in terms of feature additions - basically to ansible-puppet | 21:53 |
| *** ianw has joined #openstack-infra | 21:53 | |
| jeblair | (also, i haven't called this out explicitly as a benefit, but we have a bad track record of git committing our changes to common.yaml) | 21:53 |
| *** spzala has quit IRC | 21:53 | |
| *** markvoelker has joined #openstack-infra | 21:53 | |
| mordred | I could also see it as more of a benefit once we have more than one file - in taht it will allow us to doublecheck that a key goes into the _correct_ file | 21:54 |
| *** bknudson has quit IRC | 21:54 | |
| mordred | so - worth a conversation- not convinced | 21:54 |
| *** spzala has joined #openstack-infra | 21:54 | |
| *** pc_m has quit IRC | 21:54 | |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck: Add fingerprint for bug 1439371 https://review.openstack.org/173595 | 21:54 |
| openstack | bug 1439371 in Cinder "Volume creation from image fails for UEC and Glance API version 2" [High,Fix committed] https://launchpad.net/bugs/1439371 - Assigned to Jon Bernard (jbernard) | 21:54 |
| jroll | is there a tl;dr on backporting patches to RC versions? | 21:55 |
| jogo | lifeless: trying to remember the case | 21:55 |
| jogo | AFAIK sdague may remember | 21:55 |
| clarkb | jroll: its basically the same as backporting to stable | 21:55 |
| clarkb | jroll: should fix bugs etc etc and be land(ed|ing) on master | 21:55 |
| jroll | clarkb: ok, I guess I'm just not seeing a stable/kilo branch | 21:55 |
| *** dboik has quit IRC | 21:56 | |
| *** tqtran has quit IRC | 21:57 | |
| clarkb | jroll: there may be a proposed/kilo branch | 21:57 |
| *** dboik_ has quit IRC | 21:57 | |
| jogo | lifeless: Ahttps://github.com/pypa/pip/issues/2141 ? | 21:57 |
| jogo | lifeless: https://github.com/pypa/pip/issues/2141 ? | 21:57 |
| clarkb | jroll: but if neither of those two branches exist then I don't think the project in question has done an RC yet | 21:57 |
| jroll | clarkb: yeah, just learned that, ty much :) | 21:57 |
| *** markvoelker has quit IRC | 21:58 | |
| jogo | lifeless: so I may be wrong about the need for --no-deps if reqs.txt includes all transitive | 22:00 |
| lifeless | jogo: so that case doesn't describe a freeze-equivalent | 22:00 |
| jogo | lifeless: correct | 22:01 |
| lifeless | jogo: the requirements file there was broad and allowed a too-low version to be selected | 22:01 |
| *** esker has quit IRC | 22:02 | |
| *** annegentle has quit IRC | 22:03 | |
| *** sarob has joined #openstack-infra | 22:03 | |
| *** patrickeast has quit IRC | 22:04 | |
| *** Swami has quit IRC | 22:10 | |
| *** hdd has quit IRC | 22:10 | |
| timrc | I guess I should probably listen to my laptop when it tells me it's run out of power and is going to shut off. | 22:13 |
| mordred | timrc: :) | 22:13 |
| timrc | So not sure anything I said got sent, but... | 22:13 |
| clarkb | timrc: doesn't apepar to have made it to the freenode ether | 22:14 |
| timrc | The fundamental problem we're solving downstream with eyaml is being able to gate changes to Hiera yaml since a change to Hiera yaml now is essentially an unreviewed change to production. Also, if we were to lose the system today, the only place those secrets are backed up to is tape, which is a 24 hour turn around time, since the repo is local to the puppet master. Having the yaml in a Gerrit project would give us all the benefits of having a | 22:15 |
| timrc | dvcs in that regard. | 22:15 |
| *** yamahata has joined #openstack-infra | 22:15 | |
| nibalizer | timrc: another problem is we have 4 masters and need the hiera data on them to be consistent | 22:15 |
| nibalizer | spoiler alert: its not right now | 22:15 |
| timrc | We had two incidents recently. The first was a bad cert was deployed to a production system and then a good cert was deployed, but to the wrong system. | 22:15 |
| timrc | And, yeah, what nibalizer said. | 22:15 |
| nibalizer | i think those we both this guy <--- | 22:15 |
| clarkb | jogo: http://logs.openstack.org/14/172614/19/experimental/check-tempest-dsvm-multinode-full/9843e88/console.html#_2015-04-15_19_41_29_714 iirc you had run into that before right? (the dhcp discover from cirros VM failing to get a lease), was that a problem on the single node job(s) too? | 22:15 |
| timrc | So I'd like to eliminate localized repos and make sure that changes are reviewed by someone other than the person making them. | 22:16 |
| jogo | lifeless: yeah, I may be mistaken about needing --no-deps | 22:16 |
| notmyname | fungi: clarkb: what's the process for doing a requirements FFE? | 22:17 |
| clarkb | notmyname: I think send mail to the list with details /me finds an example | 22:17 |
| clarkb | dhellmann: ^ iirc you and ttx were dealing with those? | 22:17 |
| jogo | clarkb: yes i have seen that | 22:17 |
| jogo | still don't know what the issue is though | 22:17 |
| clarkb | jogo: ok, let me try setting up my iptables capture for dhcp and see how ugly that makes our logs | 22:18 |
| jogo | that is the main blocker to getting the job stable | 22:18 |
| jogo | clarkb: awesome | 22:18 |
| clarkb | jogo: iirc that was my idea for debugging this as it is infrequent | 22:18 |
| jeblair | timrc, nibalizer: so when you write the spec, having a good problem statement would help a lot, keeping both downstream and upstream in mind. | 22:18 |
| *** belmoreira has quit IRC | 22:18 | |
| *** rlandy has quit IRC | 22:18 | |
| clarkb | notmyname: ya I see at least one requirements FFE to the dev list | 22:18 |
| *** amitgandhinz has quit IRC | 22:19 | |
| clarkb | notmyname: mestery sent the message thread subject is Kilo stable branches for "other" libraries | 22:19 |
| jogo | clarkb: I think it happens 10 or 20% of the time or so | 22:19 |
| timrc | There are days I want to just give up on managing secrets and he help rms turn emacs into a word processor. | 22:19 |
| fungi | nibalizer: also quaint stories about you blowing up servers because you did not have this feature is right on point ;) | 22:19 |
| fungi | timrc: the emacs operating system already contains word processor technology baked right in between the newsreader and the dogflosser | 22:20 |
| *** annegentle has joined #openstack-infra | 22:20 | |
| notmyname | clarkb: ok, just looks like a very brief note with a link to the patch | 22:21 |
| timrc | fungi, :) | 22:21 |
| *** patrickeast has joined #openstack-infra | 22:21 | |
| jeblair | timrc: i edit gpg encrypted files directly with emacs :) | 22:21 |
| clarkb | notmyname: yup, so I think push the change to requirements then point to it with a blurb about why the hcange is desired (I would put the same blurb in the commit message so that the reason follows the edit in git) | 22:21 |
| notmyname | right | 22:21 |
| notmyname | good news is this is a swift-only dependency (ie not something that affects other projects) | 22:22 |
| *** julim has quit IRC | 22:22 | |
| *** annegentle has quit IRC | 22:24 | |
| jeblair | yolanda: i have reviewed your change! :) i think it's 90% there and is generally a good approach, but i have suggested one api change. | 22:24 |
| fungi | i think our clouds are mostly broken today | 22:25 |
| jeblair | we are not going to space today | 22:25 |
| fungi | wrong end pointing toward ground, yes | 22:25 |
| *** ivar-laz_ has joined #openstack-infra | 22:25 | |
| jeblair | fungi, clarkb: should we just open an ec2 account and expense it to the foundation? :) | 22:26 |
| clarkb | ha | 22:26 |
| *** SumitNaiksatam has quit IRC | 22:26 | |
| jeblair | anyone have a cloud we can use? | 22:26 |
| fungi | i believe rax-dfw may still be in trouble. we have 106 instances there in a building state right now | 22:26 |
| *** dboik has joined #openstack-infra | 22:26 | |
| fungi | jeblair: itym "borrow" | 22:26 |
| jeblair | we'll give it back | 22:26 |
| fungi | we'll give it back when we're done, honest ;) | 22:26 |
| timrc | Let's get on Azure. | 22:27 |
| timrc | And use snappy ubuntu core. | 22:27 |
| fungi | we've also got rather a lot of nodes building in hpcloud and almost none running jobs there | 22:27 |
| *** dboik_ has joined #openstack-infra | 22:27 | |
| fungi | looks to me like hpcloud is mostly broken, and a third of rax is also broken | 22:28 |
| jeblair | status notice If anyone has an OpenStack cloud they aren't using, would you please give credentials to the infra team in #openstack-infra? We will return it in the condition we found it aside from normal wear and tear. | 22:28 |
| clarkb | jeblair: ha | 22:28 |
| clarkb | calling all clouds | 22:28 |
| fungi | send in the clouds | 22:28 |
| jeblair | 10 points for fungi | 22:29 |
| fungi | at $oldjob we used to get our yuks by replacing the word "cloud" with "clown" in marketing materials | 22:29 |
| clarkb | fungi: you should install the cloud2butt firefox/chromium plugin | 22:29 |
| *** ivar-lazzaro has quit IRC | 22:29 | |
| fungi | got to the point where whenever i heard the sales guys talking about our cloud services i would have even more trouble than usual holding back my laughter | 22:30 |
| *** SumitNaiksatam has joined #openstack-infra | 22:30 | |
| *** SumitNaiksatam has quit IRC | 22:30 | |
| *** dboik has quit IRC | 22:31 | |
| jeblair | um. so if, at the summit, you find that i am spending all of my time rolling around on the floor laughing, you will know why. | 22:31 |
| fungi | the words sounded similar enough i'd start hearing it as "clown" all the time, and even accidentally mispronouncing it clown at times | 22:31 |
| *** SumitNaiksatam has joined #openstack-infra | 22:32 | |
| greghaynes | I had to uninstall that when I started openstacking | 22:32 |
| greghaynes | it was too much | 22:32 |
| greghaynes | also, the notice wikipedia had for editors to uninstall that plugin because they were unintentionally changing entire wikipedia pages was amazing | 22:33 |
| jeblair | fungi: i expect some of us will now try to pronounce it just exactly in the middle so you're never quite sure which work was said. | 22:33 |
| fungi | i certainly hope so | 22:33 |
| jeblair | greghaynes: haha! | 22:33 |
| fungi | yeah, rax-dfw has 106 nodes building, 74 deleting | 22:34 |
| fungi | it looks like all the ones in a building state are from ~7 hours ago when i deleted all the nodes which had been stuck in a delete state there since yesterday | 22:35 |
| jeblair | TypeError: None object cannot be quoted | 22:35 |
| fungi | i'm going to see about purging any alien nodes. seems like nodepool might be fighting back an invasion | 22:35 |
| jeblair | that's showing up a lot in the logs. | 22:35 |
| *** annegentle has joined #openstack-infra | 22:36 | |
| fungi | russellb: you have 63 connections open to gerrit | 22:37 |
| fungi | in good news, show-connections is correctly showing account names now! | 22:37 |
| *** mriedem has quit IRC | 22:38 | |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool: Don't try to delete nonexistent jenkins nodes https://review.openstack.org/174131 | 22:38 |
| jeblair | fungi: i thought it always did that? | 22:38 |
| *** dizquierdo has quit IRC | 22:39 | |
| openstackgerrit | Khai Do proposed openstack-infra/system-config: Allow puppet to manage replication keys on review-dev.o.o https://review.openstack.org/174133 | 22:39 |
| jeblair | fungi, clarkb: ^ we may want to push that nodepool change in. i believe we may be underutilizing quota because of it. | 22:39 |
| clarkb | jeblair: will take a look in just a moment | 22:39 |
| zaro | fungi: ^ may need your help setting up that key in heira? | 22:40 |
| fungi | zaro: yep, i can do that part no problem | 22:40 |
| jeblair | fungi, clarkb: (of course, that's only happening because of extreme breakange from earlier, but it would still theoretically make it harder to recover from. and of course if clouds can't satisfy demand anyway, it probably won't help. but hey, it might a little.) | 22:40 |
| *** annegentle has quit IRC | 22:41 | |
| jeblair | fungi: i think my change to gerrit should show you the username or connection or something in the "show-queue" command if one of the tasks it is running is a stream-event task (which almost never shows up there unless there's a problem) | 22:41 |
| fungi | jeblair: oh, good catch. so that's why we're getting that logged | 22:41 |
| fungi | jeblair: ahh, yep, now i remember | 22:42 |
| jeblair | OverLimit: Maximum number of ports exceeded | 22:42 |
| jeblair | we're also getting that | 22:42 |
| *** bswartz has joined #openstack-infra | 22:42 | |
| fungi | i'm currently looking for disassociated floating ips to delete too | 22:42 |
| clarkb | may need to rerun the port cleaner script | 22:42 |
| fungi | proof this is janitorial duty | 22:43 |
| openstackgerrit | Clark Boylan proposed openstack-infra/devstack-gate: Capture dhcp during multinode n-net runs https://review.openstack.org/174139 | 22:43 |
| clarkb | jogo: ^ | 22:43 |
| jeblair | fungi, clarkb: once we're done mopping up, we could consider giving yolanda's change https://review.openstack.org/170788 a test run in production | 22:44 |
| jeblair | fungi, clarkb: it's not ready to land for API reasons, but I think should _work_ and it might bump hpcloud throughput a little | 22:44 |
| jeblair | anyway, something to keep in the back pocket | 22:45 |
| clarkb | jeblair: by not ready to land you mean not ready to merge but you are ok deploying it as is temporarily? | 22:45 |
| jeblair | clarkb: yeah | 22:45 |
| *** gordc has quit IRC | 22:45 | |
| clarkb | ok, I will review that one now | 22:45 |
| fungi | yeah, a few hundred disassociated fips. deleting them now | 22:46 |
| *** ivar-laz_ has quit IRC | 22:46 | |
| *** ivar-lazzaro has joined #openstack-infra | 22:46 | |
| jeblair | (also, i hear the process to actually fix this in hpcloud is proceeding, but not implemented yet) | 22:46 |
| fungi | oh, awesome! | 22:47 |
| jeblair | biab | 22:48 |
| openstackgerrit | Merged openstack-infra/elastic-recheck: remove query for bug 1285323 https://review.openstack.org/174030 | 22:48 |
| openstack | bug 1285323 in grenade "Services fail to shut down on the old side of Grenade" [Undecided,In progress] https://launchpad.net/bugs/1285323 - Assigned to Sean Dague (sdague) | 22:48 |
| *** achanda has quit IRC | 22:48 | |
| *** patrickeast has quit IRC | 22:48 | |
| fungi | removing >400 alien nodes now too | 22:50 |
| *** otter768 has joined #openstack-infra | 22:50 | |
| *** markvoelker_ has joined #openstack-infra | 22:53 | |
| lifeless | fungi: I hope you removed the one I had yesterday :) | 22:53 |
| fungi | lifeless: no idea, but i'll check in a bit. thanks for the reminder | 22:54 |
| fungi | i take it you worked out the pbr oddity there? | 22:54 |
| *** otter768 has quit IRC | 22:54 | |
| lifeless | fungi: kindof. I have NFI why it worked for me locally. It was broken ass code. | 22:55 |
| fungi | heh | 22:55 |
| fungi | and yes, i've _now_ deleted that helpd node. thanks again | 22:55 |
| lifeless | so I figured out why it was breaking there, and fixed it. | 22:55 |
| *** soren has quit IRC | 22:57 | |
| *** baoli has quit IRC | 22:57 | |
| *** achanda has joined #openstack-infra | 22:58 | |
| *** baoli has joined #openstack-infra | 22:58 | |
| *** dimtruck is now known as zz_dimtruck | 22:59 | |
| *** erlon has quit IRC | 23:01 | |
| *** Rockyg has joined #openstack-infra | 23:01 | |
| *** baoli has quit IRC | 23:03 | |
| clarkb | mordred: jeblair re the nodepool change with queues. Does shade implement the task manager? No right? instead you tell shade here is a task manager you can use if the code using shade needs it | 23:04 |
| clarkb | mordred: jeblair: then shade submits tasks into that manager | 23:04 |
| clarkb | (trying to make sure I grok jeblairs comments) | 23:04 |
| fungi | checked all the jenkins masters and i think only 06 is struggling. i'm putting it into shutdown to get ready for a restart | 23:04 |
| clarkb | fungi: thanks | 23:05 |
| fungi | that's also the one that ate lifeless's pbr docs job in the gate earlier, so seems to be related | 23:05 |
| clarkb | fungi: yes I think that the wget/curl for the console.html has a high chance of failing when jenkins has thread leaked | 23:06 |
| fungi | i wouldn't be surprised if the number of 5xx proxy errors from jenkins increases when it gets into this state | 23:06 |
| clarkb | yup | 23:06 |
| lifeless | wow, lots of building nodes | 23:06 |
| lifeless | flaky cloud? | 23:06 |
| clarkb | I think that happens because jenkins wants to make a thread for your http session but it fails to do so in a timely manner | 23:07 |
| jroll | lifeless: rhetorical question? :) | 23:07 |
| lifeless | jroll: no, it could be other reasons. | 23:07 |
| lifeless | jroll: so genuine q | 23:07 |
| jroll | lifeless: was a joke | 23:07 |
| lifeless | jroll: ah :) | 23:07 |
| *** jaypipes has quit IRC | 23:08 | |
| clarkb | lifeless: flaky clouds :) | 23:08 |
| clarkb | hpcloud had leaked a bunch of resources (fungi just cleaned them up) and rax dfw is having trouble booting/deleting nodes | 23:08 |
| fungi | yeah, for whatever reason nodepool can't seem to build nodes in rax-dfw at all. i haven't debugged it yet but seems that all its attempts to build new instances there since i cleared all the hung deleting ones from their incident in dfw yesterday are now spinning indefinitely with no response | 23:09 |
| clarkb | fungi: have you tried spinning a node up there by hand? | 23:09 |
| fungi | not yet | 23:09 |
| clarkb | fungi: I can do that if it will help | 23:09 |
| fungi | please do | 23:09 |
| clarkb | kk starting that now | 23:09 |
| fungi | i sort of wonder if something happened to the api endpoint there and nodepool is just unhappy with it, which would also explain why i was able to manually delete nodes it should have retried deletion for on its own | 23:10 |
| *** chlong has joined #openstack-infra | 23:11 | |
| *** Sukhdev has quit IRC | 23:12 | |
| clarkb | I am booting a clarkbtest VM based on our latest devstack-trusty image in dfw | 23:13 |
| jroll | fungi: I'm happy to look into things if you have a tenant id or node uuids | 23:15 |
| clarkb | jroll: I can hand you clarkbtest's uuid if it doesn't boot up properly | 23:15 |
| jroll | sure | 23:15 |
| clarkb | nope it booted fine. fungi 104.130.131.52 | 23:16 |
| *** otter768 has joined #openstack-infra | 23:16 | |
| *** mhu has quit IRC | 23:16 | |
| *** patrickeast has joined #openstack-infra | 23:16 | |
| *** mhu has joined #openstack-infra | 23:16 | |
| *** asettle has quit IRC | 23:19 | |
| clarkb | I am going to delete it and try again | 23:19 |
| jroll | clarkb: what's the tenant ID, I can look for api weirdness | 23:19 |
| clarkb | random note: it would be nice if our clients handled insecure platform warnings sanely | 23:19 |
| reed | clarkb, like a 'no panic'-'ok panic' sign? | 23:21 |
| clarkb | jroll: you want the username or the tenant number? | 23:21 |
| jroll | clarkb: number? | 23:21 |
| clarkb | reed: one print per invocation would be nice | 23:21 |
| *** jaypipes has joined #openstack-infra | 23:22 | |
| fungi | clarkb: other than a handful in an error state and our long-running (proposal, pypi) slaves, we have nothing in that region for that tenant. i think nodepool is trying to build/delete and never getting a response from the api endpoint | 23:23 |
| clarkb | second boot came up fine too | 23:24 |
| fungi | clarkb: forcing that socket to close from the nodepoold end (possibly by restarting nodepoold itself) will probably clear this up | 23:24 |
| clarkb | fungi: we can put jeblairs patch for the jenkins thing in when we do it | 23:24 |
| fungi | that's what i'm thinking, yes | 23:24 |
| clarkb | jeblair: ^ want ot try yolanda's queues change too since a restart looks handy anyways? | 23:24 |
| jeblair | clarkb: just got back, catching up | 23:25 |
| fungi | also, nodepool could benefit from a periodic no-op api "ping" just to test that the provider's endpoint is still responding, and then close and reopen the socket if not | 23:25 |
| fungi | we've seen this before, though it was usually with the tripleo cloud's api endpoint going away in an outage/hard reboot | 23:26 |
| jeblair | clarkb: yeah, i'll stage both of those changes locally on nodepool | 23:26 |
| clarkb | fungi: iirc we applied a timeout to hpcloud but not nodepool connections | 23:26 |
| clarkb | fungi: let me find that and I can update the rax config to use it too | 23:26 |
| fungi | clarkb: oh! that's right, forgot about that | 23:27 |
| fungi | that's even better | 23:27 |
| *** hichihara has joined #openstack-infra | 23:27 | |
| jroll | fungi: clarkb: so to start, I don't see any POST requests to dfw from your tenant between 19:16:35 and 23:11:38 | 23:27 |
| fungi | especially since it already exists ;) | 23:27 |
| fungi | jroll: yep, i think your dfw api endpoint went away ungracefully at some point yesterday and nodepoold never noticed because the socket was still in an established state at our end | 23:28 |
| jroll | and only two since 23:11 which I presume is clark's stuff | 23:28 |
| jroll | hrm | 23:28 |
| jroll | I guess it could have, LB failover or something | 23:29 |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config: Set nodepool api-timeout on rax providers https://review.openstack.org/174153 | 23:29 |
| fungi | jroll: 19:16 likely corresponds to when i was doing out-of-band deletes | 23:29 |
| jeblair | clarkb, fungi: those two changes staged. i'll wait until clarkb is ready with the rax timeout config change so we don't immediately yank the provider out after the restart | 23:29 |
| clarkb | fungi: ^ that should do it | 23:29 |
| jroll | fungi: that wouldn't be a POST though | 23:29 |
| *** amitgandhinz has joined #openstack-infra | 23:29 | |
| fungi | oh, the api uses post for creates? i get you | 23:30 |
| * fungi is woefully unfamiliar with openstack's underlying rest apis | 23:30 | |
| jeblair | clarkb, fungi: want to force-merge that patch, then stop nodepool, then manually apply that patch, then start nodepool? | 23:30 |
| jroll | heh | 23:30 |
| *** teran_ has joined #openstack-infra | 23:30 | |
| jroll | yeah, POST is create, DELETE is delete | 23:30 |
| fungi | jeblair: sounds fine | 23:30 |
| jeblair | (i think that should keep the config from flapping with puppet) | 23:31 |
| clarkb | jeblair: I think we have to disable puppet too until the next run_all.sh pass | 23:31 |
| jroll | fungi: so this would have been like 4 hours ago that it stopped, I didn't hear anything about things happening to APIs but I'm only halfway in the loop on stuff like that :) | 23:31 |
| clarkb | jeblair: but I am on board with the rest of your plan | 23:31 |
| jeblair | clarkb: well, i was thinking if we force-merged it, it should work out | 23:31 |
| *** armax_ has joined #openstack-infra | 23:32 | |
| *** miqui_ has joined #openstack-infra | 23:32 | |
| jeblair | clarkb: i guess there's a race in there | 23:32 |
| clarkb | jeblair: ya run_all.sh only updates system-config at the start of its run | 23:32 |
| *** a13x2121 has joined #openstack-infra | 23:32 | |
| jeblair | okay, we'll disable puppet then | 23:32 |
| *** crc32 has quit IRC | 23:32 | |
| jeblair | puppet is disabled on nodepool and not running, so we should have the mutex. | 23:32 |
| jeblair | fungi: you want to force-merge https://review.openstack.org/#/c/174153/ when you're ready? | 23:33 |
| *** mestery_ has joined #openstack-infra | 23:33 | |
| fungi | jroll: we must have been doing something else around that timeframe. this looks like nodepoold hasn't actually been communicating successfully with your api since sometime yesterday (since it never successfully retried deletes on the stuff which it should have from yesterday and had more than 100 nodes it thought were building from 8 hours ago when i deleted those) | 23:33 |
| fungi | jeblair: can do | 23:33 |
| *** pblaho_ has joined #openstack-infra | 23:33 | |
| *** tteggel_ has joined #openstack-infra | 23:34 | |
| *** amitgandhinz has quit IRC | 23:34 | |
| *** kmartin_ has joined #openstack-infra | 23:34 | |
| jroll | fungi: hrm, ok, if there's anything else you'd like me to dig into let me know (yesterday there was network shenanigans in DFW so that may make more sense) | 23:34 |
| *** Ryan_Lan1 has joined #openstack-infra | 23:34 | |
| *** Shrews_ has joined #openstack-infra | 23:35 | |
| *** frickler_ has joined #openstack-infra | 23:35 | |
| *** dhp_ has joined #openstack-infra | 23:35 | |
| *** HenryG_ has joined #openstack-infra | 23:35 | |
| *** sarob_ has joined #openstack-infra | 23:35 | |
| *** salv-orl_ has joined #openstack-infra | 23:36 | |
| *** Shrews has quit IRC | 23:36 | |
| *** Shrews_ is now known as Shrews | 23:36 | |
| openstackgerrit | Merged openstack-infra/system-config: Set nodepool api-timeout on rax providers https://review.openstack.org/174153 | 23:36 |
| fungi | that's ^ merged. stopping nodepool and applying manually now | 23:37 |
| jeblair | fungi: oh i already staged the config change | 23:37 |
| fungi | just saw that--thanks | 23:37 |
| fungi | i'll start nodepoold back up again in that case | 23:37 |
| jeblair | BadHTTPException: Error communicating with server[https://jenkins06.openstack.org/] | 23:38 |
| *** aarefiev_ has joined #openstack-infra | 23:38 | |
| *** skolekonov_ has joined #openstack-infra | 23:38 | |
| *** a13x212 has quit IRC | 23:38 | |
| *** dmsimard has quit IRC | 23:38 | |
| *** teran has quit IRC | 23:38 | |
| *** Ryan_Lane has quit IRC | 23:38 | |
| *** gnuoy has quit IRC | 23:38 | |
| *** Mithrandir has quit IRC | 23:38 | |
| *** grantbow has quit IRC | 23:38 | |
| *** kmartin has quit IRC | 23:38 | |
| *** miqui has quit IRC | 23:38 | |
| *** HenryG has quit IRC | 23:38 | |
| *** skolekonov has quit IRC | 23:38 | |
| *** jamespag` has joined #openstack-infra | 23:38 | |
| *** gnuoy` has joined #openstack-infra | 23:38 | |
| *** jamespag` has quit IRC | 23:38 | |
| *** jamespag` has joined #openstack-infra | 23:38 | |
| *** reed_ has joined #openstack-infra | 23:38 | |
| clarkb | jeblair: ya fungi put it in shutdown mode because its thread leaking | 23:38 |
| *** cburgess_ has joined #openstack-infra | 23:38 | |
| *** dougshelley66_ has joined #openstack-infra | 23:38 | |
| fungi | yeah, it's in prepare for shutdown. i'm getting ready to restart 06 as soon as a couple lingering jobs wrap up | 23:38 |
| jeblair | ok cool | 23:38 |
| *** Mithrandir has joined #openstack-infra | 23:38 | |
| *** markvoelker has joined #openstack-infra | 23:39 | |
| *** vhoward- has joined #openstack-infra | 23:39 | |
| *** cdent has quit IRC | 23:39 | |
| *** ctlaugh_ has joined #openstack-infra | 23:39 | |
| *** Krinkle has quit IRC | 23:39 | |
| *** jesusaurus has quit IRC | 23:39 | |
| *** vhoward has quit IRC | 23:39 | |
| *** sarob has quit IRC | 23:39 | |
| *** armax has quit IRC | 23:39 | |
| *** pblaho has quit IRC | 23:39 | |
| *** salv-orlando has quit IRC | 23:39 | |
| *** heyongli has quit IRC | 23:39 | |
| *** frickler has quit IRC | 23:39 | |
| *** mestery has quit IRC | 23:39 | |
| *** aarefiev has quit IRC | 23:39 | |
| *** reed has quit IRC | 23:39 | |
| *** jamespage has quit IRC | 23:39 | |
| *** dougshelley66 has quit IRC | 23:39 | |
| *** tteggel has quit IRC | 23:39 | |
| *** dhp has quit IRC | 23:39 | |
| *** ctlaugh has quit IRC | 23:39 | |
| *** cburgess has quit IRC | 23:39 | |
| *** heyongli has joined #openstack-infra | 23:39 | |
| *** dmsimard_ has joined #openstack-infra | 23:39 | |
| *** dmsimard_ is now known as dmsimard | 23:39 | |
| *** dougshelley66_ is now known as dougshelley66 | 23:39 | |
| *** heyongli has quit IRC | 23:39 | |
| *** heyongli has joined #openstack-infra | 23:39 | |
| *** armax_ is now known as armax | 23:39 | |
| *** sarob_ is now known as sarob | 23:39 | |
| fungi | it 5xx'd a console log curl on a gate job earlier, which prompted me reviewing all the masters for signs of the thread leak. that seemed to be the only one | 23:39 |
| jeblair | clarkb: re earlier question -- shade will implement a simple synchronous task manager that it will use by default. nodepool will supply it's threading taskmanager when it uses shade | 23:39 |
| *** ajmiller has quit IRC | 23:39 | |
| clarkb | jeblair: roger | 23:39 |
| *** jesusaurus has joined #openstack-infra | 23:39 | |
| mordred | yes | 23:40 |
| *** markvoelker_ has quit IRC | 23:40 | |
| *** reed_ is now known as reed | 23:40 | |
| *** markvoelker has quit IRC | 23:40 | |
| *** Krinkle has joined #openstack-infra | 23:40 | |
| *** markvoelker has joined #openstack-infra | 23:40 | |
| jeblair | clarkb: (and so all the task classes will end up being defined in shade and removed from nodepool) | 23:40 |
| clarkb | jeblair: right, and just the taskmanager would be passed from nodepool to zuul | 23:41 |
| clarkb | er | 23:41 |
| clarkb | s/zuul/shade/ | 23:41 |
| jeblair | yep | 23:41 |
| mordred | yup | 23:41 |
| *** markvoelker_ has joined #openstack-infra | 23:41 | |
| mordred | jeblair: we said the same thing but spelled it with different regional accents | 23:41 |
| jeblair | mordred: uh-huh | 23:41 |
| *** grantbow has joined #openstack-infra | 23:41 | |
| *** grantbow has quit IRC | 23:41 | |
| *** grantbow has joined #openstack-infra | 23:41 | |
| jeblair | so far, nodepool is not spewing exceptions to the log. | 23:42 |
| fungi | you bet yer sweet bippy! | 23:42 |
| mordred | woot! | 23:42 |
| *** emagana has quit IRC | 23:43 | |
| jeblair | oh, it's doing that thing where the flavor cache isn't populated so 50 listflavortasks get enqueued. i'm assuming that'll be fixed by shade, yeah? | 23:43 |
| jeblair | (i mean, should be easy to fix in nodepool, but i think we're about to just drop that code, and it only happens at startup) | 23:43 |
| jeblair | also extensions | 23:43 |
| mordred | jeblair: yes - I it should be | 23:44 |
| *** esker has joined #openstack-infra | 23:44 | |
| *** Ryan_Lan1 is now known as Ryan_Lane | 23:45 | |
| mordred | SpamapS: ^^ as we're working on the caching stuff, we should think about the thundering herd case above ^^ and make sure we're doing a good job with it | 23:45 |
| *** Ryan_Lane has quit IRC | 23:45 | |
| *** Ryan_Lane has joined #openstack-infra | 23:45 | |
| *** markvoelker has quit IRC | 23:45 | |
| fungi | looks like jenkins07 just threw a shoe on a change in the gate too... https://jenkins07.openstack.org/job/gate-oslo.utils-pep8/98/console | 23:45 |
| mordred | Shrews: ^^ you too | 23:45 |
| SpamapS | mordred: we're not. ;) | 23:45 |
| SpamapS | mordred: we need to write a dogpile.cache backend that uses gearman ;-) | 23:46 |
| mordred | SpamapS: I know we're not _now_ | 23:46 |
| mordred | heh | 23:46 |
| clarkb | fungi: hrm we are supposed to retry | 23:46 |
| SpamapS | or teach dogpile.cache to do soft expires | 23:46 |
| jeblair | SpamapS: clearly yes, but i don't know why | 23:46 |
| clarkb | fungi: there may be a bug in the retry logic, I will take a look | 23:46 |
| * mordred throws a nicely seasoned salmon at SpamapS | 23:46 | |
| jeblair | SpamapS: and probably AFS too? :) | 23:46 |
| fungi | clarkb: thanks | 23:46 |
| SpamapS | jeblair: this is where coalesce comes in | 23:46 |
| mordred | SpamapS: can we reimplement gearman using AFS | 23:46 |
| SpamapS | IIRC geard doesn't actually implement coalesce? | 23:46 |
| clarkb | SpamapS: it does not | 23:47 |
| SpamapS | mordred: geArFS? | 23:47 |
| mordred | SpamapS: ++ | 23:47 |
| SpamapS | Ok, so without coalesce, gearman is not good for thundering herd protection | 23:47 |
| jeblair | SpamapS: there was nothing called coalesce in the docs when i wrote gear | 23:47 |
| SpamapS | With it, it does all the magic for you. | 23:47 |
| clarkb | jeblair: its still not there | 23:47 |
| clarkb | jeblair: gearman appears to have grown quite a few features without the protocol being updated | 23:47 |
| SpamapS | jeblair: unique jobs should only be queued once, and if a unique arrives with the same unique as an in-progress job and it is not backgrounded, its client should be sent the response for the in-progress job. | 23:48 |
| SpamapS | This was basically the first feature of gearman. | 23:48 |
| SpamapS | Like, the perl gearman. | 23:48 |
| fungi | since it looks like jenkins07 may also be starting to show signs of unhappy, i'll start quiescing it as soon as i get 06 restarted | 23:48 |
| SpamapS | So it's possible nobody wrote it down because it was the reason the wrote gearman. ;) | 23:49 |
| SpamapS | they wrote | 23:49 |
| jeblair | i think there's a lesson here | 23:49 |
| jeblair | SpamapS: i thought unique ids were added later | 23:49 |
| clarkb | fungi: I wonder, could nodepool restarts somehow trigger the jenkins unhappyness? thought 06 was unhappy prior | 23:49 |
| SpamapS | Also the word coalescence does appear _once_ in libgearman's documentation. :) | 23:50 |
| fungi | maybe we have a thundering herd condition with nodepool hitting the jenkins api at startup too? seems unlikely but possible | 23:50 |
| jeblair | SpamapS: not here: http://gearman.org/protocol/ | 23:50 |
| SpamapS | jeblair: uniques being fed to the worker was added later | 23:51 |
| SpamapS | jeblair: but it was always in the client request | 23:51 |
| jeblair | SpamapS: yeah | 23:51 |
| jeblair | SpamapS: it never says what the server will do with it. other than (possibly) provide it to the worker | 23:51 |
| SpamapS | so the protocol isn't what implements it, and I totally agree geard implements the protocol. :) | 23:51 |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config: Fix retries in grab_console_log.sh https://review.openstack.org/174160 | 23:52 |
| clarkb | fungi: ^ | 23:53 |
| jeblair | SpamapS: i would just expect the documentation for SUBMIT_JOB.* to say "if the unique thingy matches an existing job, the client will get the results from that job"... more or less in the way that it says "If on of the BG versions is used, the client is not updated with status or notified when the job has completed (it is detached)." | 23:53 |
| SpamapS | jeblair: also my merge proposals against gearmand have sat ignored by upstream for a while, so I am a little worried that protocol document is unmaintained. I'd love to add a note "This unique ID may be used by the server to coalesce requests to a single worker." | 23:53 |
| fungi | clarkb: thanks! | 23:53 |
| jeblair | SpamapS: yeah i think that's what's needed | 23:53 |
| SpamapS | jeblair: yeah, the thing is, the protocol doesn't need to say that.. because coalescing is invisible to worker and client. :-P | 23:54 |
| jeblair | SpamapS: anyway, we can implement coalesce; i don't think it will break gear | 23:54 |
| jeblair | SpamapS: i think it should because it's an important behavior for a client to know about | 23:54 |
| SpamapS | but if the protocol _did_ say that it would help implementors and might explain "why would I ever want a unique ID?" to the protocol users. | 23:54 |
| jeblair | SpamapS: i think that if a client submits a job, it should know whether it's going to run once or twice | 23:54 |
| openstackgerrit | Clark Boylan proposed openstack-infra/devstack-gate: Capture dhcp during multinode n-net tempest runs https://review.openstack.org/174139 | 23:54 |
| SpamapS | jeblair: the problem is that you can't know that if you have multiple servers. :-P | 23:54 |
| jeblair | SpamapS: so i think it's important for it to be defined, since jobs might have side effects | 23:54 |
| jeblair | SpamapS: the protocol only applies to one server. a client that cared would know how many servers it submitted the job to | 23:55 |
| SpamapS | and it could even run twice-ish .. if it just finished, you submit again, that looks like twice. :-P | 23:55 |
| SpamapS | anyway, I 100% agree, protocol doc should point out the server can use this for coalesce. Submitting merge proposal now. | 23:56 |
| SpamapS | Hopefully it merges before 2016. | 23:56 |
| * mordred wishes SpamapS good luck | 23:56 | |
| SpamapS | https://code.launchpad.net/gearmand/+activereviews | 23:56 |
| SpamapS | Not looking too active. :( | 23:56 |
| *** tqtran_afk is now known as tqtran | 23:57 | |
| jeblair | nodepool used is about 2x what we had before | 23:57 |
| fungi | much of which is probably just having dfw back in the pool again | 23:58 |
| jeblair | 281 currently | 23:58 |
| jeblair | /usr/local/lib/python2.7/dist-packages/novaclient/v1_1/__init__.py:30: UserWarning: Module novaclient.v1_1 is deprecated (taken as a basis for novaclient.v2). The preferable way to get client class or object you can find in novaclient.client module. warnings.warn("Module novaclient.v1_1 is deprecated (taken as a basis for " | 23:59 |
| jeblair | um. | 23:59 |
| clarkb | http://15.125.65.125/dashboard/db/nodepool-performance-hp-cloud hopefully that graph will look better | 23:59 |
| *** dboik_ has quit IRC | 23:59 | |
| *** wolsen_ is now known as wolsen | 23:59 | |
| jeblair | like i get the general sense that something about client deprecation is being communicated to me | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!