| *** pcrews_ has quit IRC | 00:00 | |
| fungi | #status ok the gate is still fairly backed up, though nodepool is back on track and chipping away at remaining changes. some py3k/pypy node starvation is slowing recovery | 00:02 |
|---|---|---|
| openstackstatus | fungi: sending ok | 00:02 |
| *** melwitt has quit IRC | 00:02 | |
| *** ChanServ changes topic to "Gerrit upgraded to 2.8. See: https://wiki.openstack.org/wiki/GerritUpgrade" | 00:04 | |
| -openstackstatus- NOTICE: the gate is still fairly backed up, though nodepool is back on track and chipping away at remaining changes. some py3k/pypy node starvation is slowing recovery | 00:04 | |
| *** matsuhashi has joined #openstack-infra | 00:04 | |
| *** aconrad has quit IRC | 00:04 | |
| openstackstatus | fungi: finished sending ok | 00:06 |
| *** david-lyle has joined #openstack-infra | 00:09 | |
| *** david-lyle has quit IRC | 00:13 | |
| *** yjiang5 is now known as yjiang5_away | 00:19 | |
| jogo | fungi: I like how fairly backed up is just 7 hours now | 00:23 |
| jogo | and 15 patches :_ | 00:23 |
| jogo | althoguh check is pretty big | 00:23 |
| *** bhuvan has quit IRC | 00:23 | |
| jeblair | the zuul job queue looks like half-dome | 00:23 |
| * jeblair goes to play space quest | 00:24 | |
| fungi | woah i haven't thought about space quest in years. once of the funniest sierra game series, for sure. great writing | 00:26 |
| *** aconrad has joined #openstack-infra | 00:26 | |
| *** packet has quit IRC | 00:30 | |
| tchaypo | StevenK: plus it's morning, pre-coffee. this is when i stare at gerrit and go "huh?" rather than trying to write anything useful | 00:30 |
| StevenK | Lies. The coffee is right next to you. | 00:31 |
| tchaypo | yes, it's next to me. | 00:32 |
| tchaypo | see, if it was *in* me I might have noticed that this isn't #tripleo | 00:33 |
| fungi | tchaypo: well, it had a # and a few of the same letters, so you were close anyway | 00:34 |
| *** lcostantino has joined #openstack-infra | 00:34 | |
| tchaypo | more than just the same letters | 00:34 |
| tchaypo | it had some of the same characters! | 00:34 |
| SlickNik | question: Do all infra jobs run on nodepool instances that are jenkins slaves? Or only the devstack-vm-gate ones? | 00:36 |
| jeblair | SlickNik: all | 00:36 |
| *** arnaud__ has joined #openstack-infra | 00:38 | |
| SlickNik | jeblair: thanks! The reason I ask is this: I'm trying to write a new infra job that builds the trove guest-image from dib-elements; it needs to be able to check out the elements repo, and a couple of tripleo repos and run diskimage-builder. | 00:39 |
| SlickNik | I'm thinking that Gerrit-git-prep should be sufficient as a builder, and that I shouldn't have to use the hooks in devstack-gate to do this. | 00:41 |
| SlickNik | Or is there a better way to do this, that you'd suggest? Thoughts? | 00:42 |
| jeblair | SlickNik: if the projects involved depend on each other and will end up gating, you may want to use devstack-gate since it's the only thing that knows how to set up multiple projects in zuul's queue | 00:42 |
| jeblair | SlickNik: (hashar is working on a script that will generalize that, so we can do this sort of thing outside of d-g in the future) | 00:42 |
| jeblair | SlickNik: other than that, the bare nodes and devstack nodes are converging, hopefully we'll just have one node type eventually. main difference at this point is whether they have all the extra devstack stuff cached (images, packages, etc) | 00:43 |
| jeblair | SlickNik: they both have sudo ability and local caches of all the git repos | 00:44 |
| *** yamahata has quit IRC | 00:44 | |
| *** rhsu has quit IRC | 00:45 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/nodepool: Fix race in tests https://review.openstack.org/91491 | 00:46 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/nodepool: Improve logging/stats around launch errors https://review.openstack.org/91492 | 00:46 |
| *** aconrad has quit IRC | 00:47 | |
| jeblair | fungi, mordred, phschwartz: ^ that should tell us _a lot_ more about what's going on in nodepool | 00:47 |
| SlickNik | jeblair: Thanks! That's really helpful. | 00:48 |
| SlickNik | jeblair: Right now we're planning to gate on just the one repository that contains the trove-guest elements. | 00:48 |
| SlickNik | jeblair: But let me give the gating requirements pertaining to this a bit more thought. | 00:49 |
| *** jhesketh has quit IRC | 00:49 | |
| *** crc32 has quit IRC | 00:51 | |
| jeblair | SlickNik: okay, if it's only one repo, you might be able to get away with ggp for now, and maybe we will have the multi-repo script ready by the time you need it | 00:52 |
| jeblair | (definitely the much simpler way to go) | 00:52 |
| jogo | hrmm 'gate-python-novaclient-pypy: queue' at top of gate | 00:52 |
| jogo | fungi: I assume you are aware already ^ | 00:56 |
| SlickNik | jeblair: Okay, I think that's the route I'll go with for now (and plan on switching to the multi-repo script when we have that). Thanks for your help! | 00:57 |
| *** weshay has joined #openstack-infra | 00:58 | |
| *** mriedem has joined #openstack-infra | 00:59 | |
| *** manishg has joined #openstack-infra | 00:59 | |
| *** lcostantino has quit IRC | 01:02 | |
| krotscheck | nibalizer: https://github.com/krotscheck/puppet-storyboard | 01:04 |
| krotscheck | That _should_ work to set up your own standalone storyboard instance. | 01:05 |
| krotscheck | Where “should” means “Some of the dependencies might be a bit messed up” | 01:05 |
| nibalizer | oh wild | 01:07 |
| nibalizer | is that the same as the one in the openstack-infra/config project | 01:07 |
| *** jhesketh has joined #openstack-infra | 01:08 | |
| mgagne | greatest meta puppet module I ever saw so far... https://github.com/krotscheck/puppet-storyboard/tree/master/modules/puppet_module | 01:08 |
| krotscheck | mgagne: No kidding | 01:09 |
| phschwartz | jeblair: That should give us more info when there are failures which will let us debug it for you better. | 01:09 |
| krotscheck | nibalizer: No, it’s based on that though. | 01:09 |
| *** amcrn has quit IRC | 01:09 | |
| krotscheck | nibalizer: Give me a couple of minutes, I’ll get the README working. | 01:09 |
| mgagne | should have been puppet module install yodawg/puppet_module | 01:09 |
| krotscheck | Also, that puppet_module isnt’ required anymore | 01:09 |
| *** amcrn has joined #openstack-infra | 01:12 | |
| krotscheck | nibalizer: Done. Readme now reflects the install process I used. | 01:13 |
| krotscheck | nibalizer: One thing- the module doesn’t automatically set up any admin users. You’ll have to manually go into the database and set is_superuser in the users table. | 01:14 |
| nibalizer | okay | 01:14 |
| nibalizer | thanks | 01:14 |
| nibalizer | im still at $WORKJOB | 01:14 |
| nibalizer | so cant play with it | 01:14 |
| nibalizer | ubt ill try to get going soon | 01:15 |
| nibalizer | also there is a changset pending against infra/config to make storyboard its own puppet module on openstack | 01:15 |
| nibalizer | so hopefully we can separate those | 01:15 |
| *** nati_ueno has joined #openstack-infra | 01:16 | |
| krotscheck | nibalizer: Oh, right. I should go review that. | 01:16 |
| *** rhsu has joined #openstack-infra | 01:16 | |
| krotscheck | nibalizer: Thanks :) | 01:16 |
| *** mwagner_lap has joined #openstack-infra | 01:17 | |
| *** zns_ has joined #openstack-infra | 01:17 | |
| *** weshay has quit IRC | 01:18 | |
| jesusaurus | nibalizer: do you have a link handy for that change? i wanna see what all is needed to do that | 01:18 |
| nibalizer | uh no | 01:20 |
| nibalizer | but if you search on me should come up | 01:20 |
| nibalizer | i think its -1 right now :( | 01:21 |
| *** amcrn has quit IRC | 01:21 | |
| *** zns has quit IRC | 01:22 | |
| *** zns_ has quit IRC | 01:22 | |
| *** saschpe has quit IRC | 01:28 | |
| *** saschpe has joined #openstack-infra | 01:29 | |
| *** marcoemorais has quit IRC | 01:36 | |
| openstackgerrit | A change was merged to openstack-infra/config: add subunit log test sanity https://review.openstack.org/85797 | 01:43 |
| *** yamahata has joined #openstack-infra | 01:44 | |
| *** nati_ueno has quit IRC | 01:47 | |
| *** nosnos has joined #openstack-infra | 01:49 | |
| *** amcrn has joined #openstack-infra | 01:52 | |
| *** homeless has quit IRC | 01:55 | |
| *** rhsu1 has joined #openstack-infra | 01:56 | |
| *** rhsu has quit IRC | 01:56 | |
| openstackgerrit | A change was merged to openstack-infra/config: Add django_openstack_auth to the integrated gate https://review.openstack.org/86528 | 01:58 |
| *** zhiyan_ is now known as zhiyan | 01:59 | |
| openstackgerrit | A change was merged to openstack-infra/config: Add a check for nova-docker https://review.openstack.org/86910 | 02:01 |
| *** Alexandra_ has joined #openstack-infra | 02:01 | |
| *** marun has quit IRC | 02:01 | |
| *** mengxd has joined #openstack-infra | 02:02 | |
| openstackgerrit | A change was merged to openstack-dev/pbr: Fixed a typo in the documentation. https://review.openstack.org/90516 | 02:02 |
| *** yidclare has quit IRC | 02:03 | |
| openstackgerrit | A change was merged to openstack-infra/config: Fixes to nodepool installation on jenkins-dev.o.o https://review.openstack.org/88026 | 02:04 |
| openstackgerrit | A change was merged to openstack-infra/config: Integrate Launchpad for compass-* projects https://review.openstack.org/87418 | 02:05 |
| openstackgerrit | A change was merged to openstack-infra/config: Fix redis.conf pid and data dir settings https://review.openstack.org/88323 | 02:06 |
| openstackgerrit | A change was merged to openstack-infra/config: Extract Murano python jobs in a template https://review.openstack.org/87974 | 02:07 |
| *** Ryan_Lane1 has quit IRC | 02:07 | |
| portante | anybody around that can help with a openstack-infra/config change failure? | 02:13 |
| portante | http://logs.openstack.org/24/91424/1/check/gate-config-layout/0192c4d/console.html | 02:13 |
| portante | it is saying that gate-swift-unittests-func is not defined, but not sure why | 02:14 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add fingerprint for bug 1314786 https://review.openstack.org/91453 | 02:19 |
| uvirtbot | Launchpad bug 1314786 in openstack-ci "/opt/stack/new/devstack/files/Fedora-x86_64-20-20131211.1-sda.qcow2 is not cached" [Undecided,New] https://launchpad.net/bugs/1314786 | 02:19 |
| fifieldt | sdague, hi | 02:23 |
| openstackgerrit | A change was merged to openstack-infra/config: Fix gate-heat-templates-dsvm shell script https://review.openstack.org/89875 | 02:24 |
| *** wenlock has joined #openstack-infra | 02:25 | |
| openstackgerrit | A change was merged to openstack-infra/config: use buck from master branch https://review.openstack.org/90219 | 02:26 |
| openstackgerrit | A change was merged to openstack-infra/config: Fixed several typos in the docs. https://review.openstack.org/90544 | 02:27 |
| openstackgerrit | A change was merged to openstack-infra/config: Configuring bundler to use multiple cpus/cores https://review.openstack.org/90563 | 02:27 |
| openstackgerrit | A change was merged to openstack-infra/config: index n-sched in grenade runs https://review.openstack.org/90054 | 02:27 |
| *** zehicle_at_dell has joined #openstack-infra | 02:32 | |
| *** mriedem has quit IRC | 02:32 | |
| *** julim has quit IRC | 02:33 | |
| *** sweston has quit IRC | 02:36 | |
| *** arnaud__ has quit IRC | 02:47 | |
| mattoliverau | portante: in your patch, under zuul's layout you place 'gate-swift-unittests-func' under the project 'openstack/swift' which makes sense, but in jenkins job builder its not under swift but under 'sqlalchemy-migrate' is this what you wanted? | 02:51 |
| openstackgerrit | A change was merged to openstack-infra/nodepool: Create launch-timeout setting for providers https://review.openstack.org/85150 | 02:59 |
| portante | ugh, mattoliverau thanks | 03:02 |
| *** Alexandra_ has quit IRC | 03:06 | |
| *** mengxd has quit IRC | 03:08 | |
| *** talluri has quit IRC | 03:09 | |
| openstackgerrit | Peter Portante proposed a change to openstack-infra/config: Add swift "in-process" functional test runs https://review.openstack.org/91424 | 03:09 |
| portante | embarrassing ... | 03:09 |
| *** manishg has quit IRC | 03:13 | |
| *** manishg has joined #openstack-infra | 03:15 | |
| *** manishg has quit IRC | 03:16 | |
| mattoliverau | portante: well you had alot of +1's so your not the only one who didn't notice. I only noticed because I was reading the yaml files trying to peice together how they worked, as I haven't set one up before. So yay for fresh eyes on a problem :) | 03:17 |
| portante | yes, thank you! | 03:18 |
| *** zhiyan is now known as zhiyan_ | 03:18 | |
| openstackgerrit | A change was merged to openstack-infra/jenkins-job-builder: Tiny spelling fix: YMAL -> YAML https://review.openstack.org/91291 | 03:19 |
| *** harlowja is now known as harlowja_away | 03:20 | |
| *** harlowja_away is now known as harlowja | 03:21 | |
| *** Ryan_Lane has joined #openstack-infra | 03:21 | |
| *** crc32 has joined #openstack-infra | 03:22 | |
| *** alexpilotti has quit IRC | 03:25 | |
| *** matsuhashi has quit IRC | 03:28 | |
| openstackgerrit | A change was merged to openstack-infra/config: Added gate-magnetodb-devstack-dsvm as gate https://review.openstack.org/91014 | 03:35 |
| *** aconrad has joined #openstack-infra | 03:36 | |
| *** Ryan_Lane1 has joined #openstack-infra | 03:36 | |
| openstackgerrit | Joshua Harlow proposed a change to stackforge/gertty: Handle exiting more gracefully https://review.openstack.org/91504 | 03:37 |
| harlowja | jeblair ^ | 03:37 |
| *** nosnos has quit IRC | 03:46 | |
| harlowja | jeblair some other examples u might find useful, https://excess.org/svn/urwid/contrib/trunk/rbreu_menus.py could be useful for a menu | 03:52 |
| *** crc32 has quit IRC | 04:05 | |
| *** rlandy|afk has quit IRC | 04:07 | |
| *** AaronGreen is now known as AaronGr | 04:07 | |
| *** miqui has quit IRC | 04:09 | |
| lifeless | it would be great if we can get https://review.openstack.org/#/c/89691/ in tomorrow please; would really like time to get stuff into it before the summit | 04:16 |
| *** yfried has quit IRC | 04:16 | |
| *** Ryan_Lane1 has quit IRC | 04:19 | |
| *** UtahDave has joined #openstack-infra | 04:21 | |
| *** rhsu has joined #openstack-infra | 04:22 | |
| *** rhsu1 has quit IRC | 04:22 | |
| *** matsuhashi has joined #openstack-infra | 04:35 | |
| *** nosnos has joined #openstack-infra | 04:36 | |
| *** marcoemorais has joined #openstack-infra | 04:42 | |
| *** marcoemorais1 has joined #openstack-infra | 04:45 | |
| *** marcoemorais has quit IRC | 04:46 | |
| *** matsuhashi has quit IRC | 04:48 | |
| *** matsuhashi has joined #openstack-infra | 04:53 | |
| *** morganfainberg is now known as morganfainberg_Z | 05:05 | |
| *** Ryan_Lane has quit IRC | 05:23 | |
| *** Ryan_Lane has joined #openstack-infra | 05:24 | |
| *** UtahDave has quit IRC | 05:30 | |
| *** ihrachyshka has joined #openstack-infra | 05:31 | |
| *** ihrachyshka has quit IRC | 05:33 | |
| *** ihrachyshka has joined #openstack-infra | 05:33 | |
| *** yfried has joined #openstack-infra | 05:35 | |
| openstackgerrit | Derek Higgins proposed a change to openstack-infra/config: Remove setting of RE_EXEC from nova-docker job https://review.openstack.org/91515 | 05:35 |
| *** harlowja is now known as harlowja_away | 05:37 | |
| *** ihrachyshka has quit IRC | 05:37 | |
| *** derekh has joined #openstack-infra | 06:02 | |
| *** luisg_ has joined #openstack-infra | 06:02 | |
| *** luisg__ has quit IRC | 06:02 | |
| derekh | is zuul having problems, we have check-tripleo jobs running in jenkins but nothing listed in the queue | 06:06 |
| *** dkehn has quit IRC | 06:10 | |
| *** dkehn_ has quit IRC | 06:10 | |
| *** AaronGr has quit IRC | 06:11 | |
| *** rhsu1 has joined #openstack-infra | 06:22 | |
| *** rhsu has quit IRC | 06:22 | |
| *** afazekas has joined #openstack-infra | 06:26 | |
| *** aconrad has quit IRC | 06:29 | |
| *** AaronGr has joined #openstack-infra | 06:41 | |
| *** chandan_kumar has joined #openstack-infra | 06:44 | |
| openstackgerrit | A change was merged to openstack-infra/devstack-gate: Allow setting services up under apache mod_wsgi https://review.openstack.org/90812 | 06:45 |
| *** jooools has joined #openstack-infra | 06:56 | |
| *** shardy_afk is now known as shardy | 07:06 | |
| *** chandan_kumar has quit IRC | 07:07 | |
| *** matsuhashi has quit IRC | 07:09 | |
| *** wenlock has quit IRC | 07:09 | |
| *** matsuhas_ has joined #openstack-infra | 07:12 | |
| *** derekh has quit IRC | 07:13 | |
| *** praneshp has quit IRC | 07:18 | |
| *** oomichi has joined #openstack-infra | 07:19 | |
| *** jamielennox is now known as jamielennox|away | 07:23 | |
| *** rhsu1 has quit IRC | 07:26 | |
| *** Ryan_Lane has quit IRC | 07:31 | |
| *** _nadya_ has joined #openstack-infra | 07:33 | |
| *** Ryan_Lane has joined #openstack-infra | 07:40 | |
| *** _nadya_ has quit IRC | 07:46 | |
| *** Ryan_Lane has quit IRC | 07:48 | |
| *** Longgeek has joined #openstack-infra | 07:48 | |
| *** jpich has joined #openstack-infra | 07:49 | |
| *** _nadya_ has joined #openstack-infra | 07:55 | |
| *** mkerrin has joined #openstack-infra | 08:03 | |
| *** andreykurilin_ has joined #openstack-infra | 08:05 | |
| mattoliverau | Time to call it a day for me, have a great day/night all. | 08:10 |
| *** amcrn has quit IRC | 08:12 | |
| *** _nadya_ has quit IRC | 08:17 | |
| *** darraghb has joined #openstack-infra | 08:18 | |
| *** e0ne has joined #openstack-infra | 08:21 | |
| *** andreykurilin_ has quit IRC | 08:23 | |
| *** Longgeek has quit IRC | 08:24 | |
| *** e0ne has quit IRC | 08:25 | |
| *** _nadya_ has joined #openstack-infra | 08:25 | |
| *** andreykurilin_ has joined #openstack-infra | 08:26 | |
| *** andreaf has joined #openstack-infra | 08:26 | |
| *** jp_at_hp has joined #openstack-infra | 08:30 | |
| *** matsuhas_ has quit IRC | 08:33 | |
| *** dkehn has joined #openstack-infra | 08:33 | |
| *** markwash has quit IRC | 08:34 | |
| *** dkehn_ has joined #openstack-infra | 08:37 | |
| *** flaper87|afk is now known as flaper87 | 08:37 | |
| *** jcoufal has joined #openstack-infra | 08:40 | |
| *** matsuhashi has joined #openstack-infra | 08:41 | |
| *** andreykurilin_ has quit IRC | 08:42 | |
| *** _nadya_ has quit IRC | 08:43 | |
| *** andreykurilin_ has joined #openstack-infra | 08:43 | |
| *** e0ne has joined #openstack-infra | 08:45 | |
| *** flaper87 is now known as flaper87|afk | 08:57 | |
| *** markmc has joined #openstack-infra | 08:58 | |
| *** derekh has joined #openstack-infra | 09:07 | |
| *** marcoemorais1 has quit IRC | 09:11 | |
| *** e0ne has quit IRC | 09:12 | |
| *** jcoufal has quit IRC | 09:23 | |
| *** jcoufal has joined #openstack-infra | 09:23 | |
| *** oomichi has quit IRC | 09:28 | |
| *** sdake_ has quit IRC | 09:31 | |
| *** matsuhashi has quit IRC | 09:32 | |
| *** matsuhashi has joined #openstack-infra | 09:33 | |
| *** sdake_ has joined #openstack-infra | 09:53 | |
| *** nosnos has quit IRC | 09:54 | |
| openstackgerrit | Graham Hayes proposed a change to openstack-infra/config: Add stackforge/designate-specs https://review.openstack.org/87540 | 10:06 |
| sdague | fifieldt: is this normal awake hours for you now? | 10:06 |
| *** Longgeek has joined #openstack-infra | 10:14 | |
| *** chandan_kumar has joined #openstack-infra | 10:15 | |
| *** ociuhandu has quit IRC | 10:16 | |
| openstackgerrit | A change was merged to stackforge/python-jenkins: Wrap tests and linting with tox https://review.openstack.org/89521 | 10:20 |
| *** rcarrill` has quit IRC | 10:23 | |
| *** rcarrillocruz has joined #openstack-infra | 10:23 | |
| *** andreykurilin_ has quit IRC | 10:31 | |
| *** jp_at_hp has quit IRC | 10:32 | |
| *** andreykurilin_ has joined #openstack-infra | 10:32 | |
| openstackgerrit | Derek Higgins proposed a change to openstack-infra/config: Create tripleo-specs git repo https://review.openstack.org/89691 | 10:52 |
| *** chandan_kumar has quit IRC | 10:52 | |
| *** e0ne has joined #openstack-infra | 10:54 | |
| *** andreykurilin_ has quit IRC | 10:54 | |
| *** ociuhandu has joined #openstack-infra | 10:56 | |
| *** sdake_ has quit IRC | 10:58 | |
| *** rcarrill` has joined #openstack-infra | 11:00 | |
| *** rcarrillocruz has quit IRC | 11:02 | |
| *** rcarrillocruz has joined #openstack-infra | 11:04 | |
| *** alexpilotti has joined #openstack-infra | 11:04 | |
| *** ociuhandu has quit IRC | 11:04 | |
| *** rcarrill` has quit IRC | 11:06 | |
| derekh | sdague: about https://review.openstack.org/#/c/89691/ am I able to have a docs job for a blank repo , or should I populate it with stuff first and import a prepopulated repo ? | 11:13 |
| *** ociuhandu has joined #openstack-infra | 11:13 | |
| sdague | derekh: that's a good question | 11:15 |
| sdague | I thought the cookiecutter had enough to get rolling here | 11:16 |
| sdague | you definitely shouldn't start with a blank repo, you should start with the cookiecutter | 11:16 |
| derekh | sdague: ok, I think I get it now, will take a look at cookie cutter | 11:19 |
| *** ociuhandu has quit IRC | 11:24 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/config: fix overflow on firefox https://review.openstack.org/91548 | 11:28 |
| openstackgerrit | A change was merged to openstack-infra/jenkins-job-builder: Add more tests https://review.openstack.org/90524 | 11:31 |
| openstackgerrit | A change was merged to openstack-infra/jenkins-job-builder: Add additional yaml/xml tests, literalinclude them https://review.openstack.org/91131 | 11:31 |
| *** jcoufal has quit IRC | 11:34 | |
| *** miarmak_ has joined #openstack-infra | 11:35 | |
| *** miarmak has quit IRC | 11:35 | |
| *** bogdando has quit IRC | 11:36 | |
| *** bogdando has joined #openstack-infra | 11:39 | |
| *** _nadya_ has joined #openstack-infra | 11:43 | |
| *** maxbit has joined #openstack-infra | 11:44 | |
| *** doug-fish has left #openstack-infra | 11:44 | |
| *** yamahata has quit IRC | 11:45 | |
| *** _nadya_ has quit IRC | 11:47 | |
| *** dkehn__ has joined #openstack-infra | 11:48 | |
| *** jp_at_hp has joined #openstack-infra | 11:48 | |
| *** rlandy has joined #openstack-infra | 11:48 | |
| *** dkehn__ has quit IRC | 11:49 | |
| *** dkehn_ has quit IRC | 11:49 | |
| *** dkehn__ has joined #openstack-infra | 11:49 | |
| *** dkehn has quit IRC | 11:50 | |
| *** dkehn has joined #openstack-infra | 11:52 | |
| *** hashar has joined #openstack-infra | 11:52 | |
| *** bogdando has quit IRC | 11:59 | |
| *** miarmak_ has quit IRC | 11:59 | |
| *** miarmak has joined #openstack-infra | 11:59 | |
| *** bogdando has joined #openstack-infra | 11:59 | |
| *** bookwar is now known as bookwar_away | 12:00 | |
| *** talluri has joined #openstack-infra | 12:04 | |
| anteaya | sdague: morning | 12:08 |
| sdague | morning | 12:08 |
| anteaya | sdague: what is the harm in starting with a blank repo? | 12:08 |
| anteaya | I thought is was okay to do that | 12:08 |
| *** mbacchi has joined #openstack-infra | 12:08 | |
| sdague | I think these projects should be starting with the cookiecutter as an import point | 12:09 |
| anteaya | which projects? | 12:09 |
| anteaya | every new project? | 12:09 |
| *** aysyd has joined #openstack-infra | 12:11 | |
| *** pdmars has joined #openstack-infra | 12:11 | |
| openstackgerrit | Thomas Leaman proposed a change to openstack/requirements: Move to newer version of python-swiftclient https://review.openstack.org/76170 | 12:12 |
| sdague | the new specs projects | 12:14 |
| *** rfolco has joined #openstack-infra | 12:14 | |
| anteaya | ah okay thanks | 12:17 |
| *** weshay has joined #openstack-infra | 12:17 | |
| *** thomasem has joined #openstack-infra | 12:18 | |
| *** maxbit has quit IRC | 12:22 | |
| *** weshay has quit IRC | 12:27 | |
| *** alexpilotti has quit IRC | 12:29 | |
| openstackgerrit | A change was merged to openstack/requirements: Update sphinxcontrib-pecanwsme to 0.8 https://review.openstack.org/91370 | 12:34 |
| *** thomasem has quit IRC | 12:36 | |
| *** thomasem has joined #openstack-infra | 12:36 | |
| *** thomasem has quit IRC | 12:37 | |
| *** adalbas has joined #openstack-infra | 12:41 | |
| *** alexpilotti has joined #openstack-infra | 12:41 | |
| *** bknudson has quit IRC | 12:42 | |
| *** yamahata has joined #openstack-infra | 12:43 | |
| *** talluri has quit IRC | 12:46 | |
| *** alexpilotti has quit IRC | 12:47 | |
| *** thomasem has joined #openstack-infra | 12:48 | |
| *** hashar has quit IRC | 12:56 | |
| *** miqui has joined #openstack-infra | 12:57 | |
| *** changbl has quit IRC | 12:59 | |
| *** thomasem has quit IRC | 13:03 | |
| *** weshay has joined #openstack-infra | 13:04 | |
| *** thomasem has joined #openstack-infra | 13:05 | |
| openstackgerrit | Derek Higgins proposed a change to openstack-infra/config: Create tripleo-specs git repo https://review.openstack.org/89691 | 13:09 |
| *** mriedem has joined #openstack-infra | 13:12 | |
| *** aconrad has joined #openstack-infra | 13:14 | |
| *** signed8bit has joined #openstack-infra | 13:14 | |
| *** julim has joined #openstack-infra | 13:16 | |
| *** doug-fish has joined #openstack-infra | 13:17 | |
| *** eharney has quit IRC | 13:17 | |
| *** smarcet has joined #openstack-infra | 13:18 | |
| *** rcarrill` has joined #openstack-infra | 13:20 | |
| anteaya | mrmartin: when you are around, what do you see as "content source for static pages of community portal" which are the motivation for https://review.openstack.org/#/c/89734/ | 13:21 |
| *** rcarrillocruz has quit IRC | 13:22 | |
| derekh | Could zuul be loosing track of jobs ? according to the zuul status page there are no f20 jobs running in the check-tripleo queue | 13:25 |
| derekh | but | 13:25 |
| anteaya | I think if jobs are lost track of that is nodepool | 13:25 |
| derekh | there currently appears to be 12 running on jenkins goodsquishy.com/downloads/tripleo-jobs.html | 13:25 |
| anteaya | if I have the relationships correct in my mind | 13:26 |
| *** matty_dubs|gone is now known as matty_dubs | 13:26 | |
| anteaya | what am I seeing at the goodsquishy.com url? | 13:26 |
| anteaya | lots of data, how is it gathered? | 13:27 |
| derekh | andreaf: pooling the jenkins API, green/red are success/failed | 13:27 |
| derekh | anteaya: and grey are currently running | 13:27 |
| anteaya | okay | 13:28 |
| derekh | Jobs have been kicked off multiple times all day, instead of reporting back to gerrit and leaving the queue | 13:28 |
| anteaya | so you are saying that goodsquishy and status.openstack.org/zuul are showing you different states for the same jobs? | 13:28 |
| anteaya | derekh: do you have a url for a patch that has had testing on it without leaving the queue? | 13:29 |
| derekh | anteaya: yup, take https://review.openstack.org/#/c/91161/3 for example | 13:30 |
| anteaya | thanks | 13:30 |
| * anteaya clicks | 13:30 | |
| *** zns has joined #openstack-infra | 13:30 | |
| derekh | zuul thinks it is waiting on the f20 job to start | 13:30 |
| derekh | but infact its run 4 times | 13:31 |
| pleia2 | morganfainberg_Z: thanks for looking into reviewday wip, I had completely forgotten that the upgrade would impact it | 13:31 |
| derekh | and one is currently running https://jenkins07.openstack.org/job/check-tripleo-overcloud-f20/261/ | 13:31 |
| derekh | but zull status page doesn't know about it | 13:31 |
| derekh | anteaya: ^ | 13:31 |
| *** manishg has joined #openstack-infra | 13:31 | |
| *** dprince has joined #openstack-infra | 13:33 | |
| anteaya | derekh: interesting | 13:33 |
| derekh | anteaya: hmm, the jenkins status page doesn't contain "Triggered by change: 91161,3" | 13:33 |
| anteaya | the jenkins status page? https://jenkins07.openstack.org/job/check-tripleo-overcloud-f20/261/ | 13:34 |
| anteaya | hmmmm | 13:35 |
| anteaya | Building remotely on tripleo-f20-tripleo-test-cloud-rh1-3928696 in workspace /home/jenkins/workspace/check-tripleo-overcloud-f20 | 13:35 |
| derekh | anteaya: yup that one | 13:35 |
| anteaya | 2014-05-01 12:59:00.587 | Triggered by: https://review.openstack.org/91161 patchset 3 | 13:35 |
| *** zns has quit IRC | 13:36 | |
| *** manishg has quit IRC | 13:36 | |
| *** zns has joined #openstack-infra | 13:36 | |
| derekh | anteaya: yup, thats in the console log but not in the status page | 13:36 |
| *** manishg has joined #openstack-infra | 13:36 | |
| anteaya | so that is in the console log but not on this page: https://jenkins07.openstack.org/job/check-tripleo-overcloud-f20/261/ | 13:36 |
| derekh | anteaya: infact looking at another job that ran 6 times , its the same situation (triggered by line missing on status page) , for all the jobs that ran except the most recent one | 13:37 |
| anteaya | derekh: I'm suspecting nodepool is playing a role in the confusion | 13:37 |
| anteaya | derekh: have you a link to that patch? | 13:37 |
| *** dkehn_ has joined #openstack-infra | 13:38 | |
| *** maxbit has joined #openstack-infra | 13:39 | |
| derekh | anteaya: yup, this patch https://review.openstack.org/#/c/91161/ resulted in the same job getting run 6 times https://jenkins07.openstack.org/job/check-tripleo-undercloud-precise/664/ https://jenkins03.openstack.org/job/check-tripleo-undercloud-precise/532/ https://jenkins05.openstack.org/job/check-tripleo-undercloud-precise/658/ https://jenkins04.openstack.org/job/check-tripleo-undercloud-precise/578/ https://jenkins06.openstack.org/job/check-tri | 13:39 |
| derekh | pleo-undercloud-precise/574/ | 13:39 |
| *** dkehn has quit IRC | 13:39 | |
| derekh | and only the most recent https://jenkins07.openstack.org/job/check-tripleo-undercloud-precise/672/ contains the gerrit patchnumber in the status page | 13:40 |
| *** markmc_ has joined #openstack-infra | 13:40 | |
| *** dkehn__ has quit IRC | 13:40 | |
| *** dkehn has joined #openstack-infra | 13:41 | |
| openstackgerrit | A change was merged to openstack/requirements: Bump python-keystoneclient to >= 0.8.0 https://review.openstack.org/90745 | 13:42 |
| *** markmc_ has quit IRC | 13:42 | |
| anteaya | derekh: so that is the check-tripelo-undercloud-precise job | 13:42 |
| anteaya | so whatever is going on is not isolated to f20 | 13:43 |
| fungi | anteaya: i only skimmed, but i'm missing the gist of the hand-wavey argument about nodepool having something to do with running jobs (nodepool neither runs jobs nor knows anything about them, it just builds instances, assigns them to jenkins masters, and then deletes them again when jenkins says it's done with them) | 13:43 |
| *** bknudson has joined #openstack-infra | 13:44 | |
| derekh | anteaya: well I'm thinking it might be related , zuul didn't know about the first 5 times that job ran and only registered the 6th one | 13:44 |
| *** maxbit has quit IRC | 13:44 | |
| *** dkranz has joined #openstack-infra | 13:44 | |
| *** maxbit has joined #openstack-infra | 13:44 | |
| anteaya | fungi: okay then I am wrong | 13:44 |
| anteaya | when jobs become disconnected I had thought that nodepool had a role in that | 13:45 |
| anteaya | anyway, it appears that tripleo jobs are running multiple times | 13:45 |
| fungi | what do you mean by jobs becoming disconnected? | 13:46 |
| anteaya | derekh feels zuul doesn't know about running jobs | 13:46 |
| anteaya | perhaps that is an incorrect characterization | 13:46 |
| fungi | zuul will re-run a job if jenkins fails to report a job status on completion (or in the case of a few other jenkins failure modes as well) | 13:47 |
| derekh | fungi: It looks like there are jobs running in the check-tripleo queue but zuul doesn't show them on the status page and for a lot of them is rerunning them | 13:47 |
| anteaya | well perhaps the issue is with jenkins reporting | 13:47 |
| anteaya | derekh: I'm curious how you can see running jobs that aren't showing on the zuul status page | 13:48 |
| anteaya | how are you collecting that data | 13:48 |
| derekh | anteaya: pooling jenkins | 13:48 |
| anteaya | I don't know what that means | 13:48 |
| derekh | anteaya: that is where the goodsquishy page gets its data, by talking to the 7 jenkins API's | 13:49 |
| anteaya | oh okay talking to the jenkins apis | 13:49 |
| fungi | right now nodepool believes there are 7 tripleo-f20 nodes in use | 13:49 |
| anteaya | now I got it | 13:49 |
| derekh | fungi: for the jobs zuul appears to have "lost", the jenkins status page for the jobs are missing the "Triggered by" line (at least the ones I have looked at) | 13:50 |
| derekh | https://jenkins05.openstack.org/job/check-tripleo-undercloud-precise/658/ | 13:50 |
| derekh | fungi: ok, but zuul thinks there is non according to status.openstack.org/zuul/ | 13:51 |
| fungi | Triggered by: https://review.openstack.org/91161 patchset 3 | 13:51 |
| *** yamahata has quit IRC | 13:52 | |
| *** yamahata has joined #openstack-infra | 13:52 | |
| fungi | and zuul's status page shows that job ran on that change and has recorded a result of success | 13:53 |
| derekh | fungi: your seeing that here ? https://jenkins05.openstack.org/job/check-tripleo-undercloud-precise/658/ | 13:53 |
| fungi | yes | 13:53 |
| fungi | in the console log | 13:53 |
| *** eharney has joined #openstack-infra | 13:53 | |
| derekh | fungi: no I'm not talking about the console log, I'm talking about the page you get when you hit that URL | 13:53 |
| derekh | fungi: compare to https://jenkins07.openstack.org/job/check-tripleo-undercloud-precise/672/ | 13:54 |
| fungi | oh, yeah no idea how jenkins parses that metadata to display the build status page | 13:54 |
| fungi | looks like the build information is empty | 13:55 |
| *** dkranz has quit IRC | 13:55 | |
| derekh | fungi: so basically , as far as I can see zuul keeps kicking off a job until it gets one that displays the build information | 13:55 |
| openstackgerrit | Alex Conrad proposed a change to openstack-infra/jenkins-job-builder: support execute-on for postbuildscript https://review.openstack.org/90507 | 13:56 |
| fungi | that seems possible. have you noticed a pattern of it only affecting certain jobs or certain node labels or certain jenkins masters? | 13:56 |
| derekh | fungi: nope looks like all jobs, any label or jenkins http://goodsquishy.com/downloads/tripleo-jobs.html | 13:57 |
| derekh | fungi: green lines are jobs that successfully ran, I wouldn't expect to see them again | 13:57 |
| fungi | oh, but only for changes in the tripleo queues or have you seen it other places too? | 13:57 |
| derekh | fungi: I only monitor the tripleo jobs | 13:58 |
| derekh | fungi: If you want I can that the script and check another job in another queue | 13:58 |
| *** prad has joined #openstack-infra | 13:59 | |
| fungi | i'm just spot-checking the other pipelines and don't see any obvious signs of that behavior, so it could be something specific to how those jobs are completing or to the configuration of those particular nodes | 13:59 |
| sdague | derekh: ++ on extending the monitoring across all the queues, would be good to see if this is more wide spread | 14:00 |
| fungi | definitely would be good to knowe | 14:00 |
| fungi | know | 14:00 |
| *** mfer has joined #openstack-infra | 14:00 | |
| *** chandan_kumar has joined #openstack-infra | 14:01 | |
| derekh | sdague: fungi: will pick a job/queue combo and take a look | 14:01 |
| *** zz_gondoi is now known as gondoi | 14:02 | |
| *** yamahata has quit IRC | 14:03 | |
| fungi | i've got a couple currently running in that state (both appear to be f20 jobs). i'll have a look in the jenkins log on the corresponding master for any insights | 14:03 |
| *** freyes has quit IRC | 14:03 | |
| *** maxbit_ has joined #openstack-infra | 14:03 | |
| *** yamahata has joined #openstack-infra | 14:04 | |
| *** maxbit_ has quit IRC | 14:04 | |
| *** maxbit_ has joined #openstack-infra | 14:05 | |
| *** maxbit has quit IRC | 14:05 | |
| *** maxbit_ has quit IRC | 14:05 | |
| *** maxbit has joined #openstack-infra | 14:06 | |
| jeblair | sdague, anteaya: as a matter of policy, i don't think we should require an existing repo to import -- but i also agree that specs repos should not start empty. in addition to importing, i would also be okay with blank repo and docs gating jobs, with the subsequent initial commit being cookiecutter output. | 14:07 |
| jeblair | fungi: let me know if you want me to look into anything | 14:07 |
| *** aconrad has quit IRC | 14:08 | |
| derekh | fungi: thanks , while your looking I'll see if I can get some results for other jobs | 14:08 |
| anteaya | jeblair: *nod | 14:08 |
| fungi | so far the only currently running jobs i find exhibiting this behavior are on tripleo-f20 nodes assigned to jenkins04, 06 and 07 | 14:09 |
| anteaya | fungi: what about the check-tripleo-undercloud-precise job that ran 6 times on https://review.openstack.org/#/c/91161/3 | 14:10 |
| *** dkranz has joined #openstack-infra | 14:10 | |
| fungi | yes, that one's not currently running | 14:10 |
| fungi | i'm not suggesting it's completely isolated to f20 nodes, just that they seem to have a much higher incidence of it | 14:10 |
| derekh | fungi: In general our F20 jobs all run together at the end (which is another problem I have) https://bugs.launchpad.net/nodepool/+bug/1308407 | 14:11 |
| uvirtbot | Launchpad bug 1308407 in tripleo "Nodepool favouring precise nodes over f20" [Critical,Triaged] | 14:11 |
| anteaya | k | 14:11 |
| *** jgrimm has joined #openstack-infra | 14:11 | |
| *** thedodd has joined #openstack-infra | 14:12 | |
| *** homeless has joined #openstack-infra | 14:12 | |
| *** akerr has joined #openstack-infra | 14:13 | |
| *** zns has quit IRC | 14:13 | |
| akerr | So I'm probably doing something wrong, but since the gerrit update, when I follow the instructions to grab the new keys I get the following: "ssh_exchange_identification: read: Operation timed out" any ideas? This is when I do the ssh -p 29418… command | 14:14 |
| akerr | i can ping review.openstack.org and see the web interface no problem | 14:15 |
| anteaya | akerr: are you behind a firewall of any kind? | 14:15 |
| fungi | derekh: as for that, yeah i suspect that you have more demand for tripleo-precise nodes and the build time for bringing up new nodes to meet the perceived demand is resulting in some hysteresis | 14:15 |
| akerr | corporate firewall, but it worked up until the update | 14:15 |
| anteaya | akerr: some people are finding the new ip blocked when the other was allowed | 14:16 |
| anteaya | not sure if that is playing a role for you or not | 14:16 |
| akerr | anteaya: ok, I hoped that wasn't the answer, dealing with IT to open ports isn't fun :/ | 14:16 |
| fungi | akerr: the ip addresses of review.openstack.org also changed as part of the upgrade. if your corporate firewall had explicitly allowed egress to the old ip addresses they may need to do the same now for the new addresses | 14:16 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add a fingerprint for bug 1298355 https://review.openstack.org/90795 | 14:16 |
| uvirtbot | Launchpad bug 1298355 in neutron "Lock wait timeout in update VIP status" [Undecided,New] https://launchpad.net/bugs/1298355 | 14:16 |
| anteaya | akerr: *nod | 14:16 |
| fungi | akerr: or talk your corporate firewall admins into letting developers reach the internet without deciding what protocols make up the internet ;) | 14:16 |
| akerr | fungi: so I should find a wall to hit my head on? | 14:17 |
| fungi | akerr: i managed to do it... *once* | 14:17 |
| anteaya | might be easier on you | 14:17 |
| *** talluri has joined #openstack-infra | 14:17 | |
| akerr | well thanks, I guess I'll go start the boulder up the hill | 14:17 |
| fungi | akerr: by becoming one of them, then fixing all the broken systems, watching them all get fired for incompetence, and rewriting policy sanely | 14:17 |
| fungi | akerr: the drawback to that is that then you're stuck as the corporate firewall admin | 14:18 |
| openstackgerrit | A change was merged to openstack-infra/nodepool: Fix race in tests https://review.openstack.org/91491 | 14:18 |
| *** e0ne has quit IRC | 14:18 | |
| *** talluri has quit IRC | 14:19 | |
| anteaya | can we submit patches over the gerrit http api? | 14:19 |
| fungi | anteaya: now that we have gerrit 2.8, i think we can start experimenting with that, yes | 14:19 |
| *** talluri has joined #openstack-infra | 14:19 | |
| jeblair | 2014-04-30 19:48:20,294 INFO zuul.Gearman: Gearman job <gear.Job 0x7fc8ec26b850 handle: H:127.0.0.1:768035 name: build:check-tripleo-undercloud-precise unique: 71ea9bab53b247ca97da4d8dddb018f1> lost due to disconnect | 14:19 |
| fungi | anteaya: i believe much of the code needed for it has already been contributed to git-review by others running newer gerrit servers | 14:20 |
| openstackgerrit | A change was merged to openstack-infra/nodepool: Improve logging/stats around launch errors https://review.openstack.org/91492 | 14:20 |
| phschwartz | morning | 14:20 |
| anteaya | akerr: did you want to investigate the gerrit http api? | 14:20 |
| anteaya | akerr: whilst you set the uphill boulder in motion | 14:20 |
| anteaya | morning phschwartz | 14:20 |
| phschwartz | jeblair: I tested the logging change in my vm test env for my infra setup internal here and it is a very nice improvement +1 | 14:21 |
| jeblair | fungi, derekh: ^ | 14:21 |
| jeblair | phschwartz: ah cool, i just approved the changes and was going to try for a nodepool restart in a bit | 14:21 |
| fungi | jeblair: derekh: yeah, i'm finding entries in the jenkins log for one of these where it's having trouble communicating with the slave agent for monitoring | 14:22 |
| *** gondoi is now known as zz_gondoi | 14:22 | |
| derekh | fungi: as I understand it, problem is that we have demand for multiple node type and grants are becomming available one at a time, nodepool is always servicing the precise node as its first in the list, hence my patch to randomly shuffle the list | 14:22 |
| *** crc32 has joined #openstack-infra | 14:23 | |
| jeblair | fungi: i think that particular error is zuul disconnecting from its geard | 14:23 |
| *** talluri has quit IRC | 14:24 | |
| jeblair | fungi: (there may be more than one thing going on) | 14:24 |
| *** zz_gondoi is now known as gondoi | 14:24 | |
| fungi | jeblair: yeah, i don't yet have a clear picture of whether this is something also encountered by other slaves which don't exhibit the problem behavior | 14:24 |
| *** alexpilotti has joined #openstack-infra | 14:24 | |
| jeblair | but we did drop 5021 jobs yesterday because of the geard disconnection | 14:25 |
| *** dstanek has joined #openstack-infra | 14:25 | |
| dstanek | bnemec: you around? | 14:25 |
| bnemec | dstanek: On a call, but should be done shortly. | 14:26 |
| dstanek | bnemec: i have a question about your patch: https://review.openstack.org/#/c/91173 | 14:26 |
| dstanek | bnemec: ok | 14:26 |
| *** gondoi is now known as zz_gondoi | 14:26 | |
| fungi | derekh: oh, good point, though perhaps something deterministic like a round-robin interleave would be preferable (always a little wary of randomization) | 14:26 |
| *** ArxCruz has joined #openstack-infra | 14:27 | |
| jeblair | derekh, fungi: it's not strictly that it allocates precise because it's first -- it allocates precise because its demand is very high. | 14:28 |
| derekh | sdague: fungi jeblair : doesn't seem to be happening at the moment on other jobs http://goodsquishy.com/downloads/other-jobs.html but scroll down to 90476,5 and you'll see a bunch of jobs that got kicked off multiple times | 14:28 |
| jeblair | (order does play a part, but not nearly as much as demand and the min-ready ratios) | 14:28 |
| derekh | jeblair: it doesn't have to be very high the ratio just has to be above .5 for f20 nodes to be ignored (as we only have one node available on each cycle) | 14:29 |
| fungi | jeblair: do you happen to know what normally populates the "build information" in jenkins (where it displays the link to the gerrit change, list of assicoated builds and list of other changes tested with it, et cetera)? | 14:29 |
| jeblair | fungi: zuul | 14:30 |
| fungi | okay, so that's a field zuul injects html into? | 14:30 |
| jeblair | yup | 14:30 |
| *** david-lyle has joined #openstack-infra | 14:31 | |
| derekh | fungi: I could try a round-robin algoritm, will see if I can get it right given different weights | 14:32 |
| fungi | jeblair: and zuul does that via jenkins api after the job task is grabbed from the gearman queue, or is it part of the gearman packet? | 14:32 |
| jeblair | fungi: it's another gearman job | 14:34 |
| *** wenlock has joined #openstack-infra | 14:36 | |
| anteaya | jeblair: is it worth trying to extend git review to allow for commands over the REST api? | 14:37 |
| anteaya | jeblair: or would it be better to refine gertty and suggest folks that want to use the api instead of ssh use gertty? | 14:37 |
| jeblair | 2014-04-30 19:21:35,739 INFO zuul.Gearman: Gearman job <gear.Job 0x7fc8b454a410 handle: H:127.0.0.1:761517 name: build:check-tempest-dsvm-full unique: d3f737f004d543e3b9883d3555ab5ea2> lost due to disconnect | 14:37 |
| jeblair | 2014-04-30 19:48:20,307 INFO zuul.Gearman: Gearman job <gear.Job 0x7fc8e128e850 handle: H:127.0.0.1:767750 name: build:check-tempest-dsvm-full unique: a03fb3c899154365b9555e645ab158b1> lost due to disconnect | 14:37 |
| jeblair | 2014-04-30 19:55:34,878 INFO zuul.Gearman: Gearman job <gear.Job 0x7fc9166d3490 handle: H:127.0.0.1:786256 name: build:check-tempest-dsvm-full unique: 645813b16f594886b3ec6d9dd6e63f91> lost due to disconnect | 14:38 |
| jeblair | 2014-04-30 20:10:53,810 INFO zuul.Gearman: Gearman job <gear.Job 0x7fc8e2143f10 handle: H:127.0.0.1:791682 name: build:check-tempest-dsvm-full unique: 9c3c6bb2463442beb4459bdee9c6b94e> lost due to disconnect | 14:38 |
| fungi | i wonder whether there were network issues in one of our providers last night | 14:38 |
| jeblair | 2014-05-01 01:25:34,589 INFO zuul.Gearman: Build <gear.Job 0x7fc8e01869d0 handle: H:127.0.0.1:800047 name: build:check-tempest-dsvm-full unique: 7819a9c3325e445fa29b541266c0724a> complete, result SUCCESS | 14:39 |
| fungi | nothing obvious in rackspace's status page | 14:39 |
| anteaya | also is the date and time in the gerrit 2.8 comments utc time? | 14:39 |
| jeblair | derekh: that's the outcome of each build of 90476,5 | 14:39 |
| jeblair | fungi: zuul and geard are on the same server | 14:39 |
| *** zz_gondoi is now known as gondoi | 14:39 | |
| *** akerr has left #openstack-infra | 14:39 | |
| jeblair | 2014-04-30 18:58:26,506 ERROR gear.Client.unknown: Connection <gear.Connection 0x29ab7d0 host: 127.0.0.1 port: 4730> timed out waiting for a response to a submit job request: <gear.Job 0x7fc915a28710 handle: None name: stop:jenkins05.openstack.org unique: 9c69ff5edd7440d68fb64df07f4e1033> | 14:39 |
| jeblair | fungi: ^ | 14:39 |
| fungi | oh, right, so this is local socket disconnects between zuul and its gear daemon process | 14:40 |
| jeblair | yep, because geard was too slow | 14:40 |
| *** dkehn_ is now known as dkehnx | 14:40 | |
| jeblair | (for an unknown reason) | 14:41 |
| fungi | it's too bad cacti is abandoned on ubuntu 12.04 lts or we'd be able to look at performance graphs for the server... http://people.canonical.com/~ubuntu-security/cve/2014/CVE-2014-2708.html | 14:41 |
| uvirtbot | fungi: Multiple SQL injection vulnerabilities in graph_xport.php in Cacti 0.8.7g, 0.8.8b, and earlier allow remote attackers to execute arbitrary SQL commands via the (1) graph_start, (2) graph_end, (3) graph_height, (4) graph_width, (5) graph_nolegend, (6) print_source, (7) local_graph_id, or (8) rra_id parameter. (http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2014-2708) | 14:41 |
| jeblair | fungi: hrm, cacti is running. | 14:41 |
| fungi | oh, so it is | 14:42 |
| phschwartz | jeblair: I have noticed that geard is not the most efficient when it comes to actually responding to requests. Seems like a latency issue in the software | 14:42 |
| jeblair | fungi: i assumed someone started it because the patch was applied | 14:42 |
| *** atiwari has joined #openstack-infra | 14:42 | |
| bnemec | dstanek: Okay, I'm free | 14:45 |
| jeblair | fungi: ironically, i think it was rebooted because of heartbleed and apache was not stopped then. | 14:45 |
| fungi | jeblair: fixed in debian with the 0.8.8b+dfsg-4 upload a few weeks ago https://bugs.debian.org/743565 but not backported to anything besides trusty | 14:45 |
| jeblair | fungi: that's pretty lame. :( | 14:45 |
| dstanek | bnemec: i tried the patch and it does seem to parse the patchset data better, but it looks like it's not dealing with age properly | 14:46 |
| dstanek | bnemec: were you not having that issue? i'm trying to see if it's just my environment | 14:46 |
| bnemec | dstanek: Yeah, but as I mentioned in the commit message I don't think that's reviewstats. The data is actually bad. | 14:46 |
| bnemec | It appears that every vote's timestamp got updated to the same time during the upgrade. | 14:46 |
| jeblair | fungi: how about i "rm /usr/share/cacti/site/graph_xport.php" ? | 14:46 |
| bnemec | So every vote that happened pre-2.8 is now showing up as a couple of days old. | 14:46 |
| bnemec | I've been meaning to follow-up on that and see if it can be fixed somehow. | 14:47 |
| fungi | jeblair: i would accept that as an option | 14:47 |
| *** pcrews_ has joined #openstack-infra | 14:48 | |
| dstanek | bnemec: oh, that's interesting | 14:48 |
| *** rfolco has quit IRC | 14:49 | |
| jeblair | fungi: done | 14:49 |
| fungi | bnemec: dstanek: i blame gerrit. it makes extensive use of mysql auto-timestamp fields rather than bothering to manage its own timestamping on things, which is hella-lame | 14:49 |
| bnemec | dstanek: fungi: I think it was only two timestamps (one for code reviews and one for approvals), so we could theoretically filter out the bad data in reviewstats. | 14:50 |
| bnemec | It means we'll only have stats back to the upgrade, but that's probably more useful than what we've got now. | 14:50 |
| *** manishg has quit IRC | 14:50 | |
| fungi | bnemec: dstanek: which means for certain tables, if you go in and update row contents for a data migration of some sort, mysql "conveniently" updates the timestamp for the row | 14:50 |
| *** annegentle has joined #openstack-infra | 14:51 | |
| dstanek | bnemec: i think that would be great - i use reviewstats as sort of a heartbeat to make sure i'm spending enough time on reviews | 14:51 |
| jeblair | we do have the old data though, so it may be possible to correct | 14:51 |
| bnemec | dstanek: Likewise. That's why I wanted to get it fixed. :-) | 14:52 |
| dstanek | fungi: can that be turned off during a migration in MySQL? | 14:52 |
| fungi | agreed, but it's probably a fair amount of work... maybe zaro wants to investigate and put together a recommended set of queries for that | 14:52 |
| sdague | I found a new interesting gerrit upgrade bug | 14:53 |
| jeblair | bnemec: can you work up an etherpad documenting the issues? | 14:53 |
| sdague | when you click on the first file to review | 14:53 |
| bnemec | jeblair: Sure | 14:53 |
| fungi | dstanek: well, there are workarounds for it (like adding colname=colname for the timestamp column in your update query) but now that dhip has sailed | 14:53 |
| sdague | the vertical scroll bar possition stays at what it was on the overview page | 14:53 |
| sdague | which means you might be deep into the patch | 14:53 |
| fungi | s/dhip/ship/ | 14:54 |
| dstanek | fungi: i'm just thinking ahead to the next one :-) | 14:54 |
| *** matsuhashi has quit IRC | 14:54 | |
| fungi | dstanek: yeah, we discovered it a while back and implemented workarounds in our documented processes for things like project renames/moves | 14:54 |
| fungi | but probably missed it in the gerrit upgrade (or it's missed in gerrit's own upgrade scripts) | 14:54 |
| *** yamahata has quit IRC | 14:54 | |
| *** markmcclain has joined #openstack-infra | 14:54 | |
| dstanek | fungi: sounds like fun | 14:55 |
| fungi | add to the list of "things which seem like a convenient option, but really aren't and you should never make use of them in your own software" | 14:55 |
| dstanek | the other thing i noticed with the new gerrit is that i periodicallly get blank pages | 14:55 |
| dstanek | they contain a little javascript snippet, but that's it - a refresh fixes the issue though | 14:56 |
| dstanek | actually it may be anytime i open the link in a new tab | 14:57 |
| *** markmcclain1 has joined #openstack-infra | 14:58 | |
| fungi | dstanek: maybe you're conflicting with the change gerrit made to start forcing links to open in a new tab/window | 14:58 |
| dstanek | fungi: could be, but if i go here https://review.openstack.org/#/q/status:open+project:openstack/keystone,n,z | 14:59 |
| fungi | oh, though it's mostly not configured to do that for internal links back to its own pages, only for links to outside resources | 14:59 |
| dstanek | those links don't open in new tabs so i do that myself | 14:59 |
| *** markmcclain has quit IRC | 14:59 | |
| hughsaunders | hey all, who looks after paste.openstack.org? seems to be struggling a little today | 15:00 |
| fungi | dstanek: in firefox 24 a ctrl-click on those links doesn't get me a blank tab. what are you running? | 15:00 |
| anteaya | my gertty log today is full of this error: http://paste.openstack.org/show/78234/ | 15:01 |
| anteaya | hughsaunders: hi, we do, paste.o.o is slow | 15:02 |
| dstanek | fungi: i'm running chrome | 15:02 |
| dstanek | fungi: on mac | 15:02 |
| hughsaunders | anteaya: cool, thanks for running it, most useful :) | 15:02 |
| fungi | hughsaunders: watching top on paste.o.o, drizzled seems to eat 100% of a cpu for extended periods when i request a paste with my browser | 15:03 |
| fungi | i've heard it suggested that the version of drizzle providing the db backend on it is prone to sluggishness | 15:03 |
| anteaya | hughsaunders: thanks but looking at cacti, the cached memory took a jump this morning: http://cacti.openstack.org/cacti/graph_view.php?action=tree&tree_id=1&leaf_id=14&page=2 | 15:03 |
| *** marun has joined #openstack-infra | 15:04 | |
| fungi | also drizzled seems to be eating over a gigabyte of virtual memory, though that's only about half of the allocated system ram and there's little else contending for it on the vm | 15:04 |
| jeblair | fungi: mordred said something about not having any buffers configured the other day. i don't know if he did anything about that though. | 15:06 |
| fungi | ohhh | 15:06 |
| *** yamahata has joined #openstack-infra | 15:07 | |
| jeblair | 2014-04-22T22:16:51 <mordred> there's no real reason that our current paste should be slow ... oh, except there is no buffer pool configured | 15:07 |
| jeblair | 2014-04-22T22:16:54 <mordred> *facepalm* | 15:07 |
| hughsaunders | is it using innodb? | 15:07 |
| jeblair | 2014-04-22T22:17:27 <clarkb> mordred: does that mean you are going to fix it? | 15:07 |
| jeblair | 2014-04-22T22:17:42 <mordred> clarkb: well, by "fix it" first I mean" move it to trove" | 15:07 |
| jeblair | fungi: http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2014-04-22.log | 15:08 |
| fungi | jeblair: great context, thanks | 15:08 |
| fungi | also, since we have cacti after all, i'm seeing that zuul is about to run out of space on / again | 15:08 |
| fungi | probably debug logs | 15:08 |
| *** gondoi is now known as zz_gondoi | 15:09 | |
| fungi | yeah, about 70% of the data on / is in /var/log/zuul | 15:09 |
| fungi | i guess it's not eating through it all that quickly though | 15:10 |
| jeblair | yup, we need to compress those | 15:10 |
| jeblair | fungi: i just manually deleted some. it'll run out again if we don't fix it. | 15:10 |
| fungi | k | 15:11 |
| jeblair | so... if you have 1000 jenkins slaves (which we do, more or less), and they can run 2500 jobs (which ours can, more or less). that's 2.5 million gearman function registrations. | 15:11 |
| mordred | jeblair: dude | 15:12 |
| mordred | that's a lot | 15:12 |
| jeblair | (i'm setting up a geard stress test, and that's the first thing i noticed -- it takes a while for my 1000 slaves to register their functions) | 15:12 |
| anteaya | mordred: can you look at these? https://review.openstack.org/#/q/status:open+project:openstack-infra/gitdm,n,z poor StevenK has been waiting over a month | 15:20 |
| mordred | anteaya: on it | 15:21 |
| anteaya | mordred: thanks | 15:21 |
| mordred | fungi: think we should migrate paste to trove? | 15:21 |
| sdague | anteaya: I've got +2 on that as well if needed | 15:21 |
| anteaya | sdague: awesome thanks | 15:22 |
| anteaya | let's see what mordred does | 15:22 |
| mordred | OMG | 15:22 |
| anteaya | just as a note, I am waiting for input from jeblair on https://review.openstack.org/#/c/89421/ since I need to here what direction he wants to go in, the infra-manuals publish job | 15:22 |
| anteaya | hear | 15:22 |
| *** jaypipes has quit IRC | 15:23 | |
| fungi | mordred: seems like a sane idea. drizzle is mysql-like enough that we can just mysqldump and then source the result in trove, right? | 15:23 |
| mordred | why would I ever want the link to a project on a change to take me to the admin screen for that project and not for other changes for that project | 15:23 |
| mordred | fungi: that's right | 15:23 |
| *** mrodden has joined #openstack-infra | 15:23 | |
| sdague | mordred: oh, yeh, I notice that this morning as well | 15:23 |
| mordred | fungi: I think I put up a change already to add hostname support | 15:23 |
| *** zz_gondoi is now known as gondoi | 15:23 | |
| *** zns has joined #openstack-infra | 15:23 | |
| sdague | the magnifying glass takes you where you want | 15:23 |
| fungi | mordred: i'll find it | 15:23 |
| mordred | sdague: yah | 15:23 |
| *** ociuhandu has joined #openstack-infra | 15:26 | |
| openstackgerrit | A change was merged to openstack-infra/gitdm: Add StevenK to the HP group list https://review.openstack.org/82706 | 15:26 |
| *** changbl has joined #openstack-infra | 15:27 | |
| *** manishg has joined #openstack-infra | 15:27 | |
| *** mestery has quit IRC | 15:27 | |
| *** yfried has quit IRC | 15:28 | |
| *** manishg has quit IRC | 15:29 | |
| openstackgerrit | A change was merged to openstack-infra/gitdm: Update mtreinish's employment history https://review.openstack.org/90439 | 15:33 |
| *** zns has quit IRC | 15:34 | |
| anteaya | thanks | 15:35 |
| *** ociuhandu has quit IRC | 15:36 | |
| jeblair | sdague, mordred: yep, there are 8x more pixels devoted to navigating you to the admin screen than there are for getting you to the list of changes to the project. i'm pretty sure that's because the admin screen in 8x more important than the list of changes in a project. | 15:36 |
| *** zns has joined #openstack-infra | 15:38 | |
| sdague | jeblair: heh | 15:38 |
| *** chandan_kumar has quit IRC | 15:39 | |
| *** maxbit_ has joined #openstack-infra | 15:39 | |
| *** maxbit_ has quit IRC | 15:40 | |
| annegentle | hey mordred I had a great call with Colette Alexander and Doug Moran this week | 15:40 |
| *** manishg has joined #openstack-infra | 15:41 | |
| mordred | annegentle: awesome! I hear that went well | 15:42 |
| *** rfolco has joined #openstack-infra | 15:43 | |
| *** maxbit has quit IRC | 15:43 | |
| jeblair | anteaya: why are you waiting for input from me on that? | 15:47 |
| anteaya | jeblair: yes | 15:47 |
| anteaya | oh why | 15:47 |
| anteaya | because there are two option outlined | 15:47 |
| anteaya | and I have my inclinations but would like to hear your preference | 15:48 |
| *** gondoi is now known as zz_gondoi | 15:48 | |
| anteaya | I can just go with my inclination if you would like | 15:48 |
| jeblair | anteaya: where are the two options? | 15:48 |
| *** zz_gondoi is now known as gondoi | 15:48 | |
| anteaya | mgagne: gave two suggestions: https://review.openstack.org/#/c/89421/9/modules/openstack_project/files/jenkins_job_builder/config/infra-publish-jobs.yaml | 15:50 |
| anteaya | AFAIK, jeblair would like the folder to be named "manual". In this case, the JJB variable {name} will be replaced by the JJB project name: infra-manual. The manual will get uploaded to infra/infra-manual which isn't good. | 15:50 |
| anteaya | In the end, if the project name seldom corresponds to the folder name into which the docs is published, I suggest introducing a new JJB variable: doc-publisher-folder. | 15:50 |
| anteaya | This JJB variable should be added to projects.yaml and defined with the value "manual": | 15:50 |
| anteaya | it is to do with the format for naming the job and the resultant url | 15:51 |
| bnemec | dstanek: fungi: jeblair: https://etherpad.openstack.org/p/gerrit-2.8-reviewstats | 15:53 |
| jeblair | anteaya: option (A) | 15:55 |
| anteaya | jeblair: thank you | 15:55 |
| anteaya | that was the one I like too | 15:55 |
| *** UtahDave has joined #openstack-infra | 15:58 | |
| *** eharney has quit IRC | 15:59 | |
| jeblair | bnemec: so we need to find the fields in the db with the wrong data | 16:00 |
| jeblair | bnemec: in order to do that, i need to find the fields in the query with the wrong data | 16:00 |
| jeblair | bnemec: so let's come up with a query that illustrates it | 16:00 |
| *** thomasem has quit IRC | 16:00 | |
| *** packet has joined #openstack-infra | 16:01 | |
| *** packet has quit IRC | 16:01 | |
| *** thomasem has joined #openstack-infra | 16:01 | |
| *** thomasem has quit IRC | 16:01 | |
| jeblair | bnemec: let's start with finding one change that illustrates (at least one, ideally all of) the problem(s) | 16:01 |
| *** thomasem has joined #openstack-infra | 16:02 | |
| bnemec | jeblair: A db query or the Gerrit query reviewstats uses? I don't think reviewstats directly queries the db so I'm not sure I have access to that. | 16:02 |
| openstackgerrit | Derek Higgins proposed a change to openstack-infra/config: Create tripleo-specs git repo https://review.openstack.org/89691 | 16:02 |
| *** packet has joined #openstack-infra | 16:02 | |
| jeblair | bnemec: no, an ssh query (which is what reviewstats uses) | 16:02 |
| bnemec | jeblair: Cool, I should be able to do that. | 16:03 |
| jeblair | bnemec: cool | 16:03 |
| derekh | jeblair: fungi: is there anything I can do for the vanishing jobs issue ? or is there more investigations to be done? | 16:03 |
| *** sweston has joined #openstack-infra | 16:04 | |
| mordred | jeblair: just read the etherpad above - do you think it's worth trying to extract the old timestamps from the db dump? | 16:04 |
| jeblair | derekh: no i think we identified the problem a while ago -- zuul dropped the gearman connection and relaunched running jobs | 16:04 |
| jeblair | mordred: that's what i'm trying to facilitate. | 16:04 |
| *** markmcclain1 has quit IRC | 16:05 | |
| jeblair | mordred: i'm trying to identify what timestamps need to be extracted. | 16:05 |
| mordred | kk | 16:06 |
| derekh | jeblair: ok, so there are still running jobs that zuul doesn't know about, do I just need to wait it out | 16:07 |
| jeblair | derekh: link to one? the example i looked at was from yesterday | 16:08 |
| derekh | jeblair: https://jenkins04.openstack.org/job/check-tripleo-overcloud-f20/229/ | 16:08 |
| *** weshay has quit IRC | 16:08 | |
| fungi | this one is currently running and has no build status... https://jenkins07.openstack.org/job/check-tripleo-overcloud-f20/264/ | 16:09 |
| jeblair | 2014-04-30 20:10:53,501 INFO zuul.Gearman: Gearman job <gear.Job 0x7fc8c4919f10 handle: H:127.0.0.1:792065 name: build:check-tripleo-overcloud-f20 unique: 275628aeaa5f46a291c6aab17b9cb3be> lost due to disconnect | 16:09 |
| fungi | ahh, right, "from yesterday" meaning the change was enqueued yesterday, not that the job started yesterday | 16:10 |
| *** mrodden has quit IRC | 16:11 | |
| *** eharney has joined #openstack-infra | 16:11 | |
| jeblair | i'm a little surprised that ended up running; i need to think about whether that's the right thing for geard to do in this case. | 16:12 |
| *** matty_dubs is now known as matty_dubs|lunch | 16:12 | |
| *** talluri has joined #openstack-infra | 16:13 | |
| fungi | the one currently running which i linked was similarly lost between zuul/gear at 20:10:53 yesterday | 16:13 |
| zaro | morning | 16:14 |
| fungi | i find it odd that it's just tripleo jobs which are getting perpetually restarted due to this. we don't seem to have changes in other pipelines sticking around exhibiting this behavior | 16:15 |
| fungi | even though there were changes in other pipelines affected by the disconnects, according to the logs | 16:16 |
| jeblair | fungi: are you saying no changes other than tripleo-related ones were restarted? | 16:19 |
| anteaya | morning zaro | 16:20 |
| anteaya | zaro: is gerrit 2.8 commenting using utc time? | 16:20 |
| fungi | jeblair: i don't know that no other changes were restarted, just that we have tripleo changes with jobs getting rerun which were originally enqueued yesterday | 16:21 |
| fungi | and it doesn't look like we have changes in any other queues for anywhere near that long | 16:21 |
| jeblair | fungi: tripleo doesn't have enough nodes | 16:21 |
| jeblair | fungi: those changes were likely actually waiting that long | 16:22 |
| fungi | yeah, pinning down whether they're getting restarted more than once since yesterday is proving challenging | 16:22 |
| jogo | how did this happen? http://logs.openstack.org/75/91375/1/gate/gate-glance-python26/9c03553/console.html#_2014-05-01_02_40_45_316 | 16:22 |
| jogo | a dependency escaped the pypi mirror | 16:23 |
| bnemec | jeblair: I think you already saw it, but I added a query for a single change demonstrating the problem in https://etherpad.openstack.org/p/gerrit-2.8-reviewstats | 16:23 |
| fungi | though at this point i'm really only finding restarts on f20-based jobs which would support the supposition that they really did just get disconnected and not rerun until 20+ hours later | 16:23 |
| fungi | or rather, that they got lost in zuul but took 20+ hours before a jenkins master had an available node of that type to go ahead and run the job (even though at this point zuul doesn't know it's being run) | 16:24 |
| jeblair | bnemec: so it looks like it's all the votes, but not the submit time. any other times affected? | 16:24 |
| *** weshay has joined #openstack-infra | 16:24 | |
| *** mrodden has joined #openstack-infra | 16:24 | |
| jogo | fungi: ^ | 16:25 |
| bnemec | jeblair: Not that I've seen. The createdOn and lastUpdated times look reasonable on that change, and those are the only other timestamps I see. | 16:26 |
| jeblair | fungi: i think i found a bug in geard | 16:26 |
| derekh | fungi: but the jobs I was looking at origionaly started yesterday evening and have been continuously restarted overnight some upto 6 times | 16:26 |
| derekh | fungi: the problems on check-tempest-dsvm-fullcheck-tempest-dsvm-neutron | 16:26 |
| derekh | also ran upto 6 times | 16:26 |
| derekh | fungi: but got cleared a lot quicker | 16:26 |
| jeblair | derekh: because it runs on precise nodes and they come up more often. | 16:27 |
| fungi | jogo: that's interesting... that job seems to have tried to download the req from somewhere other than our mirror | 16:27 |
| jeblair | it's worth noting that if geard has jobs queued that zuul does not know about, nodepool can not take them into consideration when it's calculating load. | 16:28 |
| jogo | fungi: yeah I can't figure out how that happened | 16:28 |
| derekh | jeblair: I'm talking about the check queue, not tripleo, what I mean is the same problem happened at the same time, only the multiple jobs got cleared a lot faster because the quota to handle them is higher | 16:28 |
| *** jpich has quit IRC | 16:28 | |
| jogo | fungi: oh shit I found it | 16:29 |
| *** talluri has quit IRC | 16:29 | |
| fungi | jogo: http://git.openstack.org/cgit/openstack/glance/tree/test-requirements.txt#n22 | 16:29 |
| jogo | http://git.openstack.org/cgit/openstack/glance/tree/test-requirements.txt#n22 | 16:29 |
| jogo | yuyp | 16:29 |
| jogo | derp | 16:29 |
| fungi | heh | 16:29 |
| fungi | git blame here i come | 16:29 |
| jeblair | derekh: yeah. i'm trying to figure out if we're talking past each other. i think the problem is well explained at this point; is there something you think we're missing? | 16:30 |
| dstanek | bnemec: nice | 16:30 |
| mordred | jogo: how did that make it past the requirements gate? | 16:31 |
| derekh | jeblair: no if ye are happy, I'm happy, I just saw a comment saying this was possibly only related to the check-tripleo queue, all I was saying is that wasn't true | 16:31 |
| fungi | jogo: https://review.openstack.org/#/c/59699/12/test-requirements.txt | 16:31 |
| *** jooools has quit IRC | 16:31 | |
| dstanek | bnemec: instead of filtering out specific timestamps can you just pick one and filter anything before it? | 16:31 |
| fungi | wtf? | 16:31 |
| *** nati_ueno has joined #openstack-infra | 16:31 | |
| jogo | mordred: because the global reqs check probablly doesn't work when a specific file is called for | 16:31 |
| mordred | jogo: oh! I think we explicitly don't try to check those because of legacy | 16:32 |
| mordred | we should remove that exclusion to the check | 16:32 |
| jogo | mordred: ++ | 16:32 |
| fungi | clearly it allows people to act carelessly | 16:32 |
| bnemec | dstanek: Yeah, potentially. I figured out why Approvals aren't showing up too. I probably need to update my patch Approved -> Workflow. | 16:32 |
| jeblair | derekh: cool, i agree with you that it is not tripleo-specific (we may be more likely to see evidence of it there due to the differences in node availability) | 16:33 |
| *** marcoemorais has joined #openstack-infra | 16:33 | |
| derekh | jeblair: yup | 16:33 |
| jeblair | bnemec: approved will show up in older changes, workflow in newer ones; probably best to handle both | 16:34 |
| *** alexpilotti has quit IRC | 16:34 | |
| jogo | mordred: also pysendfile is in our mirror already | 16:34 |
| bnemec | jeblair: Yeah, makes sense. | 16:34 |
| mordred | jogo: yah. you fixing or you want me to? | 16:34 |
| *** zns has quit IRC | 16:34 | |
| jogo | mordred: we should just ban '-f ' to external deps | 16:35 |
| jogo | mordred: I am writting up a glance patch | 16:35 |
| fungi | jogo: mordred: i'm thinking it may have been in response to this comment in the bug linked from that commit message... | 16:35 |
| fungi | https://bugs.launchpad.net/glance/+bug/1271806/comments/3 | 16:35 |
| uvirtbot | Launchpad bug 1271806 in glance "unable to run tests due to missing deps in the virtual env" [Wishlist,Fix released] | 16:35 |
| mordred | jogo: we should just hard-ban anything thats not in global requirements without exclusions | 16:35 |
| jogo | fungi: hehe wow | 16:37 |
| fungi | strangely, pysendfile has been in our mirror for more than a year... http://pypi.openstack.org/openstack/pysendfile/ | 16:37 |
| jogo | mordred: ++ | 16:37 |
| jogo | fungi: it looks like that was a local failure the guy saw or something | 16:37 |
| anteaya | I have a concern about the commiter offering patch 12, not a trivial change with no other reviews and self-approving | 16:37 |
| mordred | where is our actual "must match global requirements" check? | 16:37 |
| jogo | mordred: in PBR I think | 16:38 |
| jogo | fungi: https://review.openstack.org/91603 | 16:38 |
| fungi | jogo: ahh, yep, that was around the time pip updated to start refusing externally-linked packages, so that was probably a contributor to the confusion there | 16:38 |
| jogo | yeah | 16:39 |
| mordred | that's the integration job - I don't see the check for does your change match | 16:39 |
| * jogo files a glance bug to track | 16:39 | |
| zaro | anteaya: i'm not sure. can you tell? | 16:39 |
| fungi | jogo: good, goof | 16:39 |
| fungi | s/goof/good/ | 16:39 |
| mordred | foudn it | 16:40 |
| *** sacharya has joined #openstack-infra | 16:40 | |
| anteaya | zaro: I was hoping someone would know, I will keep an eye on it and see if I can track it | 16:41 |
| anteaya | zaro: thanks | 16:41 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Stop giving -f and -e lines a pass in requirements https://review.openstack.org/91604 | 16:42 |
| mordred | fungi, jogo: ^^ | 16:43 |
| *** Longgeek has quit IRC | 16:43 | |
| jogo | mordred: I saw that out of the corner of my eye and thought it said "stop giving an f*" | 16:43 |
| jogo | a f* | 16:43 |
| fungi | quite the opposite, in fact ;) | 16:43 |
| jogo | mordred fungi: https://bugs.launchpad.net/glance/+bug/1315062 tracking bug | 16:43 |
| uvirtbot | Launchpad bug 1315062 in glance "don't pull pysendfile from googlecode" [Undecided,New] | 16:43 |
| jogo | fungi: yup hehe | 16:44 |
| fungi | mordred: does the reqs sync proposal script also ignore -e and -f lines when building patches? if so, we should probably fix that too | 16:45 |
| *** bhuvan has joined #openstack-infra | 16:46 | |
| fungi | mordred: also, this might end up leaving us unable to update requirements on projects in havana... weren't we linking to a tarball of an oslo.messaging beta in the havana timeframe? | 16:46 |
| *** lakshmi has joined #openstack-infra | 16:46 | |
| fungi | oh, actually no i think we successfully avoided releasing havana with links to unreleased beta reqs | 16:47 |
| fungi | so if we're not fine there, it needs to be broken so it becomes more visible | 16:48 |
| mordred | jogo: looks like we're slow on the draw and someone else already filed the bug and the patch | 16:49 |
| jogo | mordred: oh | 16:50 |
| jogo | link? | 16:50 |
| jogo | mordred: damn | 16:50 |
| jogo | ahh https://review.openstack.org/#/c/91547/ | 16:50 |
| fungi | yeah, the reqs proposal script seems to ignore things which don't look like normal requirements (to handle comments and whatnot)... https://review.openstack.org/#/c/88719/6/test-requirements.txt | 16:51 |
| *** etoews has joined #openstack-infra | 16:51 | |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Add fingerprint for bug 1315062 https://review.openstack.org/91605 | 16:54 |
| uvirtbot | Launchpad bug 1315062 in glance "don't pull pysendfile from googlecode (dup-of: 1314968)" [Undecided,New] https://launchpad.net/bugs/1315062 | 16:54 |
| uvirtbot | Launchpad bug 1314968 in glance "Installing test-requirements fails because pysendfile.2.0.0.tar.gz cannot be found" [Undecided,In progress] https://launchpad.net/bugs/1314968 | 16:54 |
| openstackgerrit | Ben Nemec proposed a change to openstack-infra/reviewstats: Update for Gerrit 2.8 https://review.openstack.org/91173 | 16:54 |
| openstackgerrit | Ben Nemec proposed a change to openstack-infra/reviewstats: Workaround the Gerrit upgrade timestamp issue https://review.openstack.org/91606 | 16:54 |
| *** dprince has quit IRC | 16:54 | |
| bnemec | dstanek: ^ should filter out the bad timestamps | 16:54 |
| *** markmcclain has joined #openstack-infra | 16:55 | |
| *** rcarrillocruz has joined #openstack-infra | 16:56 | |
| jogo | mordred: http://paste.openstack.org/show/78256 | 16:57 |
| jogo | looks like ironic is the only one | 16:58 |
| jogo | with pecan | 16:58 |
| jogo | devananda: ^ | 16:58 |
| *** ociuhandu has joined #openstack-infra | 16:58 | |
| *** rcarrill` has quit IRC | 16:59 | |
| jogo | oh that has been fixed | 16:59 |
| *** maxbit has joined #openstack-infra | 17:02 | |
| jogo | mordred: a better list of the '-e ' offenders http://paste.openstack.org/show/78257 | 17:03 |
| jogo | looks like we have some folks installing other openstack components from !tarballs | 17:04 |
| openstackgerrit | Vasudev Kamath proposed a change to openstack-dev/pbr: Documentation updates https://review.openstack.org/91609 | 17:04 |
| *** maxbit_ has joined #openstack-infra | 17:05 | |
| mordred | jogo: all of those are fixable pretty easily | 17:06 |
| *** markmc has quit IRC | 17:06 | |
| fungi | yeah, those probably indicate there are reqs which could stand to be (pre)released to pypi, worst case | 17:06 |
| *** derekh has quit IRC | 17:06 | |
| jogo | yup | 17:06 |
| fungi | which is likely to be beneficial regardless | 17:06 |
| jogo | I am a little surprised at how not too bad they are | 17:06 |
| *** Ryan_Lane has joined #openstack-infra | 17:06 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/gear: geard: be more careful removing jobs https://review.openstack.org/91610 | 17:06 |
| jeblair | derekh is gone... :( | 17:07 |
| jeblair | fungi: ^ that should address the "ghost jobs" part of the problem, though the underlying cause of "geard did not respond for 30 seconds" is still unknown. | 17:07 |
| mordred | jogo: making patches for those | 17:08 |
| *** maxbit has quit IRC | 17:08 | |
| fungi | jeblair: it's at least a start | 17:09 |
| *** SumitNaiksatam has quit IRC | 17:10 | |
| *** ociuhandu has quit IRC | 17:10 | |
| *** hcc has joined #openstack-infra | 17:11 | |
| fungi | lgtm | 17:11 |
| *** _nadya_ has joined #openstack-infra | 17:12 | |
| mordred | jogo: submitted changes for both of them | 17:12 |
| *** salv-orlando has quit IRC | 17:13 | |
| *** alexpilotti has joined #openstack-infra | 17:14 | |
| *** matty_dubs|lunch is now known as matty_dubs | 17:14 | |
| *** talluri has joined #openstack-infra | 17:16 | |
| * anteaya goes for a walk | 17:16 | |
| *** zns has joined #openstack-infra | 17:17 | |
| *** clayg has joined #openstack-infra | 17:18 | |
| clayg | in this grenade job n-net failed to start and caused vms' to go into error - but I can't find the root cause of the n-net failure? http://logs.openstack.org/02/91502/1/check/check-grenade-dsvm/912e89e/logs/new/ | 17:18 |
| *** zns has quit IRC | 17:18 | |
| *** arosen has left #openstack-infra | 17:18 | |
| *** zns has joined #openstack-infra | 17:19 | |
| *** annegentle has quit IRC | 17:19 | |
| clayg | there seems to be some mention of a tee "/opt/stack/status/stack/n-net.failure" - but I can't see where/if this status/stack directory is captured? | 17:21 |
| *** bhuvan has quit IRC | 17:24 | |
| *** SumitNaiksatam has joined #openstack-infra | 17:24 | |
| *** Ajaeger has joined #openstack-infra | 17:25 | |
| *** e0ne has joined #openstack-infra | 17:26 | |
| *** arnaud__ has joined #openstack-infra | 17:26 | |
| *** maxbit has joined #openstack-infra | 17:27 | |
| *** markmcclain has quit IRC | 17:27 | |
| *** maxbit__ has joined #openstack-infra | 17:29 | |
| *** reed has joined #openstack-infra | 17:30 | |
| *** morganfainberg_Z is now known as morganfainberg | 17:30 | |
| *** maxbit_ has quit IRC | 17:30 | |
| fungi | clayg: perplexing. it doesn't look from the console log xtrace like screen thinks it failed to start n-net, so the n-net.failure file wouldn't have been created in that case | 17:30 |
| fungi | however i agree there seems to be no screen log archived for it either | 17:30 |
| fungi | and plenty of n-cpu errors | 17:30 |
| fungi | which seem to stem from being unable to interact with the (presumably not running) n-net | 17:31 |
| clayg | yeah and all the n-cpu errors basically say, "no one seems to be handling n-net requests) | 17:31 |
| fungi | agreed | 17:31 |
| fungi | Shrews: finally found a moment to look into the update_bug.py failure you spotted--traceback makes me wonder whether we changed the account or authentication method we're using... http://paste.openstack.org/show/78260/ | 17:31 |
| dstanek | bnemec: seems to be working. thanks! | 17:31 |
| *** maxbit has quit IRC | 17:32 | |
| bnemec | dstanek: np, glad to hear it :-) | 17:32 |
| reed | hello working folks | 17:32 |
| *** reed has quit IRC | 17:32 | |
| *** reed has joined #openstack-infra | 17:33 | |
| fungi | Shrews: seems the user i would *expect* it to be using is in the bug team for that bug/chanfge | 17:33 |
| fungi | welcome back reed! | 17:33 |
| reed | hi fungi | 17:34 |
| Shrews | fungi: zoinks! this is obviously mordred's fault, though i have no direct evidence | 17:34 |
| reed | looks like I got disconnected briefly | 17:34 |
| fungi | reed: "<-- reed (~reed@fsf/member/reed) has quit (Excess Flood)" | 17:34 |
| reed | gerrit seems to be complaining that I'm committing with the wrong email address but I don't know where to change it | 17:34 |
| fungi | reed: best place to do it is in your ~/.git/config | 17:35 |
| fungi | er, ~/.gitconfig | 17:35 |
| reed | ah! | 17:36 |
| mordred | reed: "git config --global user.email blah@blah.com" | 17:36 |
| fungi | that's where i usually set my name, email and signingkey | 17:36 |
| reed | I wonder why that's there now... and I wonder why gerrit didn't pick it up earlier | 17:36 |
| mordred | reed: or, if you disagree, you can add an email address to gerrit | 17:36 |
| *** gyee has joined #openstack-infra | 17:36 | |
| fungi | yeah, the git config --global command basically edits that file for you | 17:36 |
| *** praneshp has joined #openstack-infra | 17:37 | |
| fungi | reed: also, if you've already committed something, the previous name and email are already in the commit as the author and won't be reset without some additional magic and an --amend | 17:37 |
| reed | damn, now I need to undo the commit | 17:37 |
| fungi | er, as the committer | 17:37 |
| *** ramashri has joined #openstack-infra | 17:37 | |
| reed | yeah, right | 17:37 |
| reed | git uncommit | 17:37 |
| fungi | i think it's --reset-committer or something like that | 17:37 |
| fungi | don't need to undo the commit itselg | 17:37 |
| fungi | git commit --amend --reset-author | 17:38 |
| *** thedodd has quit IRC | 17:38 | |
| fungi | i guess --amend normally updates the committer but not the author | 17:38 |
| openstackgerrit | Stefano Maffulli proposed a change to openstack-infra/config: Adding list for Superuser magazine's team https://review.openstack.org/91623 | 17:39 |
| reed | :) | 17:39 |
| fungi | superuser magazine, huh? | 17:40 |
| *** e0ne has quit IRC | 17:40 | |
| reed | fungi, it's a new initiative of the Foundation, there will be a publication featuring users | 17:40 |
| fungi | ahh | 17:41 |
| *** darraghb has quit IRC | 17:45 | |
| openstackgerrit | Arnaud Legendre proposed a change to openstack-infra/config: Create glance-specs git repo https://review.openstack.org/90461 | 17:46 |
| *** maxbit has joined #openstack-infra | 17:46 | |
| *** praneshp has quit IRC | 17:46 | |
| *** maxbit__ has quit IRC | 17:47 | |
| *** yidclare has joined #openstack-infra | 17:48 | |
| *** e0ne has joined #openstack-infra | 17:49 | |
| SpamapS | hm | 17:51 |
| openstackgerrit | Arnaud Legendre proposed a change to openstack-infra/config: Create glance-specs git repo https://review.openstack.org/90461 | 17:51 |
| SpamapS | check-tripleo seems to have no f20 slaves | 17:52 |
| SpamapS | but I see 12 on the cloud | 17:52 |
| *** praneshp has joined #openstack-infra | 17:52 | |
| fungi | SpamapS: yeah, see earlier scrollback with jeblair and derekh | 17:52 |
| SpamapS | ah ok I thought I'd see jobs running | 17:53 |
| *** bhuvan has joined #openstack-infra | 17:53 | |
| SpamapS | fungi: so "just wait it out" was what derekh said, but is there a place I can see how long we have to wait? | 17:53 |
| SpamapS | I want to start rolling out trusty to that cloud so we can get rid of our network problems | 17:53 |
| fungi | SpamapS: apparently communication between zuul and its gear daemon got choked yesterday, which due to a bug (patch pending) caused the jobs to still end up in the queue and run by jenkins even though zuul was then blind to the fact that they were running | 17:54 |
| *** rcarrillocruz has quit IRC | 17:54 | |
| *** _nadya_ has quit IRC | 17:54 | |
| SpamapS | Ok, so there's stuff running that we can't see.. | 17:54 |
| SpamapS | and when it finishes, we'll see? | 17:54 |
| *** markmcclain has joined #openstack-infra | 17:55 | |
| fungi | SpamapS: and due to the general starvation on the tripleo cloud, in particular for less-commonly-requested node types, those queued jobs are still running on f20 nodes as they become available | 17:55 |
| SpamapS | | free_ram_mb | 77100 | | 17:55 |
| SpamapS | | vcpus | 456 | | 17:55 |
| SpamapS | | vcpus_used | 114 | | 17:55 |
| fungi | eventually they get rerun by zuul using a gearman queue item it's aware exists and then the result for that rerun will report | 17:56 |
| SpamapS | fungi: do we need to bump up the quota? | 17:56 |
| SpamapS | I only see 41 instances, error, build, and active included | 17:56 |
| fungi | SpamapS: i think the current quota was constrained by the number of tripleo "work environments" configured on the te-broker? | 17:56 |
| SpamapS | oh that's entirely possible | 17:57 |
| * fungi is a little fuzzy on the terminology for that | 17:57 | |
| SpamapS | testenvs | 17:57 |
| fungi | i remember someone saying there were only 40 environments built out | 17:57 |
| SpamapS | THere are either 7 or 10 testenvs.. can't tell if the errors in undercloud mean anything | 17:58 |
| SpamapS | each one can do 4 jobs at once. | 17:58 |
| SpamapS | I think | 17:58 |
| SpamapS | 40 would make sense | 17:58 |
| fungi | okay, that's probably what it was | 17:58 |
| SpamapS | I have more hardware now | 17:58 |
| fungi | enough environments for 40 jobs then | 17:58 |
| SpamapS | so once lifeless wakes up.. I'll ask him if we should add a couple. | 17:58 |
| fungi | sounds like you could make use of it, yeah | 17:58 |
| *** clayg has left #openstack-infra | 17:58 | |
| *** mestery has joined #openstack-infra | 17:58 | |
| SpamapS | I have 10 boxes available, but 5 are _mine_ ;) | 17:58 |
| *** rcarrillocruz has joined #openstack-infra | 17:59 | |
| fungi | anyway, the pending zuul patch is to make sure those gearman items get dequeued if zuul is going to ignore them, so that they don't end up running and wasting resources | 17:59 |
| *** dstanek is now known as dstanek_zzz | 18:00 | |
| fungi | though the jury's still out on why geard gets bogged down when we flood it with millions of requests. jeblair is trying to profile it under manufactured stress for a better picture of what's happening | 18:00 |
| *** hcc is now known as hdd | 18:00 | |
| *** melwitt has joined #openstack-infra | 18:01 | |
| *** etoews has quit IRC | 18:04 | |
| fungi | Shrews: i'm definitely confused on the update_bug.py failures... the ~gerrit2/.launchpadlib/creds file being used to authenticate to lp is definitely identical... | 18:04 |
| fungi | still digging | 18:05 |
| *** dangers_away is now known as dangers | 18:06 | |
| *** etoews has joined #openstack-infra | 18:06 | |
| openstackgerrit | Andreas Jaeger proposed a change to openstack/requirements: Update openstack-doc-tools for translation of api-site https://review.openstack.org/90746 | 18:07 |
| *** moted has joined #openstack-infra | 18:07 | |
| *** aconrad has joined #openstack-infra | 18:08 | |
| JayF | fungi: ty for modifying the SPF record, I can tell when you did it because the emails started going to my inbox :) | 18:09 |
| openstackgerrit | Ramy Asselin proposed a change to openstack-infra/config: nodepool image update works behind a http proxy https://review.openstack.org/87384 | 18:10 |
| *** manishg has quit IRC | 18:10 | |
| SpamapS | fungi: "Python" ? ;) | 18:10 |
| jogo | who is part of testing-cabal besides lifeless? | 18:10 |
| *** manishg has joined #openstack-infra | 18:10 | |
| SpamapS | fungi: have we tried geard under pypy ? | 18:10 |
| jogo | I want to use https://github.com/testing-cabal/testtools/pull/78 | 18:10 |
| SpamapS | jogo: I think jml? | 18:11 |
| SpamapS | https://launchpad.net/~testing-cabal | 18:11 |
| SpamapS | jelmer too | 18:11 |
| fungi | SpamapS: actually an intresting suggestion, and no i don't suspect it's been tested under pypy but might be interesting to see the performance numbers compared | 18:11 |
| SpamapS | ok.. caffeine and lunch needed.. | 18:11 |
| *** harlowja_away is now known as harlowja | 18:12 | |
| SpamapS | fungi: it's also possible that even though the threads are just for I/O, the GIL is still eating geard's lunch | 18:13 |
| SpamapS | lunch.. right.. need.. | 18:13 |
| * SpamapS will return | 18:13 | |
| *** ArxCruz has quit IRC | 18:16 | |
| *** ociuhandu has joined #openstack-infra | 18:18 | |
| *** amcrn has joined #openstack-infra | 18:21 | |
| *** sdake has quit IRC | 18:21 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add fingerprint for bug 1315062 https://review.openstack.org/91605 | 18:21 |
| uvirtbot | Launchpad bug 1315062 in glance "don't pull pysendfile from googlecode (dup-of: 1314968)" [Undecided,New] https://launchpad.net/bugs/1315062 | 18:22 |
| uvirtbot | Launchpad bug 1314968 in glance "Installing test-requirements fails because pysendfile.2.0.0.tar.gz cannot be found" [Undecided,In progress] https://launchpad.net/bugs/1314968 | 18:22 |
| *** sdake has joined #openstack-infra | 18:22 | |
| *** jaypipes has joined #openstack-infra | 18:23 | |
| *** ramashri has quit IRC | 18:23 | |
| *** jp_at_hp has quit IRC | 18:24 | |
| fungi | has anybody in here submitted abstracts on openstack-infra topics for linuxcon-na? | 18:26 |
| fungi | proposal submission period ends tomorrow | 18:27 |
| *** dstanek_zzz is now known as dstanek | 18:27 | |
| anteaya | yeah I just got that email too | 18:32 |
| anteaya | they have a goverance topic | 18:32 |
| anteaya | might submit something about our ptl and tc election process | 18:32 |
| fungi | oh, i didn't see an e-mail, it was just on my list of things i was procrastinating on | 18:33 |
| fungi | but you should totally do that | 18:34 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/gear: Restructure logging https://review.openstack.org/91643 | 18:34 |
| jeblair | fungi: ^ that is not a fix to the underlying issue, however, i think that if we merge that and move zuul's logging to /opt, we may be in a better place to track it down the next time it happens | 18:35 |
| sdague | fungi: jogo submitted an ER talk | 18:35 |
| sdague | I just submitted something that I called OpenStack in 10 minutes with Devstack | 18:35 |
| *** changbl has quit IRC | 18:36 | |
| anteaya | cool | 18:40 |
| *** arnaud___ has joined #openstack-infra | 18:40 | |
| anteaya | how long is the time slot for a "presentation"? | 18:40 |
| anteaya | I have to make_a_new_account and don't think I can fill 40 minutes on our election process | 18:40 |
| fungi | you could probably go into greater detail on other parts of our project governance besides just the elections | 18:41 |
| openstackgerrit | Derek Higgins proposed a change to openstack-infra/devstack-gate: Set default for DEVSTACK_GATE_USE_APACHE https://review.openstack.org/91646 | 18:42 |
| anteaya | my brain is foggy, I haven't had a decent night sleep in weeks (you don't want to know) can you elaborate? | 18:42 |
| anteaya | perhaps I am so close to it I can't see what others might want to know about | 18:43 |
| *** thedodd has joined #openstack-infra | 18:43 | |
| *** arnaud__ has quit IRC | 18:44 | |
| fungi | well, you could talk about how the subprojects self-organize to govern things like code review, documentation, translations and so on | 18:44 |
| fungi | at levels below the more visible ptl/tc strata | 18:44 |
| *** e0ne has quit IRC | 18:45 | |
| fungi | or you could talk about the foundation board elections as well and how the election model differs, what's been going on with the efforts to change that and the current challenges faced | 18:45 |
| anteaya | ah yes | 18:46 |
| anteaya | hmmmm, might steer away from the board level | 18:46 |
| fungi | it is a sticky wicket | 18:46 |
| anteaya | I am sure there are better informed folks than myself to address that | 18:46 |
| anteaya | I need less sticky | 18:46 |
| anteaya | so the self-organizing bit sounds good | 18:46 |
| *** e0ne has joined #openstack-infra | 18:47 | |
| anteaya | I'm so immersed in it, i don't know other projects don't do it | 18:47 |
| anteaya | but you are right, that would be good to talk about as well | 18:47 |
| anteaya | thanks fungi | 18:47 |
| *** zns has quit IRC | 18:47 | |
| fungi | in that case an interesting bit might be the new specs-as-code-review experiment/movement | 18:47 |
| fungi | as well, as the fact that our governance documents are now starting to get managed as code review | 18:48 |
| anteaya | hmmmmm, I can hand-wavey it in and then assess how much time it gets, based on my understanding, when I go to talk | 18:48 |
| anteaya | yes, I like that we have a governance repo | 18:48 |
| anteaya | you are great at this | 18:49 |
| fungi | not really. if i were great at it i wouldn't have been procrastinating about submitting an abstract ;) | 18:49 |
| *** zns has joined #openstack-infra | 18:49 | |
| anteaya | I'm out of excuses for not putting in a talk submission | 18:49 |
| anteaya | fungi: tell me about what you would like to talk about | 18:49 |
| *** yfried has joined #openstack-infra | 18:49 | |
| *** changbl has joined #openstack-infra | 18:50 | |
| fungi | eh, something which won't compete with anyone who already submitted talks they want to give. i'm happy to be filler ;) | 18:58 |
| fungi | which is why i'm curious who/what's already submitted so i don't step on their toes at the last minute | 18:59 |
| anteaya | I think you have lots of options | 19:01 |
| *** e0ne has quit IRC | 19:01 | |
| fungi | yep | 19:01 |
| anteaya | I for one would like to watch you walk through how you track down problems in our system | 19:01 |
| *** manishg has quit IRC | 19:01 | |
| anteaya | you have a very systematic approach for going through the logs | 19:02 |
| *** e0ne has joined #openstack-infra | 19:02 | |
| anteaya | unfortunatly anything on servers is hidden | 19:02 |
| anteaya | so how zuul jenkins and nodepool all talk to each other | 19:02 |
| anteaya | I would like to see that from a persepective of sifting through the logs | 19:02 |
| anteaya | what ever you talk about I hope you include code | 19:03 |
| *** rcarrillocruz has quit IRC | 19:03 | |
| fungi | well, the infra overview already got presented at linuxcon-na last year, so probably best not to repeat that one ;) | 19:05 |
| *** e0ne has quit IRC | 19:05 | |
| anteaya | they system has changed a bit since last year | 19:05 |
| *** e0ne has joined #openstack-infra | 19:05 | |
| fungi | oh, actually that's the zuul deep-dive, not the infra overview | 19:06 |
| fungi | but there's a fair amount of overlap | 19:06 |
| *** andreykurilin_ has joined #openstack-infra | 19:06 | |
| *** rcarrillocruz has joined #openstack-infra | 19:07 | |
| *** dkliban_afk is now known as dkliban | 19:07 | |
| *** maxbit has quit IRC | 19:10 | |
| anteaya | I haven't memorized all the presentations maybe | 19:11 |
| anteaya | has their been a nodepool deep dive slidedeck yet? | 19:11 |
| openstackgerrit | Ben Nemec proposed a change to openstack-infra/config: Add dib-utils project https://review.openstack.org/90281 | 19:11 |
| *** manishg has joined #openstack-infra | 19:11 | |
| fungi | not yet, though i question my availability to write one between now and then | 19:12 |
| *** marcoemorais has quit IRC | 19:15 | |
| anteaya | fair enough | 19:15 |
| anteaya | they want slides submitted a week in advance | 19:15 |
| anteaya | ha ha ha | 19:15 |
| *** marcoemorais has joined #openstack-infra | 19:21 | |
| notmyname | anteaya: is there a list of current PTLs anywhere? | 19:26 |
| *** andreykurilin_ has quit IRC | 19:26 | |
| notmyname | anteaya: ah, I see it on an election result page (not findable on a search, though) https://wiki.openstack.org/wiki/PTL_Elections_March/April_2014 | 19:27 |
| *** andreykurilin_ has joined #openstack-infra | 19:27 | |
| openstackgerrit | Derek Higgins proposed a change to openstack-infra/devstack-gate: Add the sbins to PATH https://review.openstack.org/91655 | 19:28 |
| anteaya | notmyname: http://git.openstack.org/cgit/openstack/governance/tree/reference/programs.yaml | 19:29 |
| anteaya | notmyname: yeah I find the wiki search unusable | 19:30 |
| fungi | anteaya: last time i talked to Ryan_Lane he suggested that a recent patch/upgrade may have caused the search index to stop getting updated | 19:30 |
| notmyname | anteaya: I was looking under https://wiki.openstack.org/wiki/Governance (where the TC has a list, along with details about BoD and UC) | 19:30 |
| anteaya | fungi: cool, that would help things | 19:31 |
| fungi | anteaya: for example https://wiki.openstack.org/w/index.php?title=Special%3ASearch&profile=default&search=openstack-ci-admins&fulltext=Search | 19:31 |
| *** Ajaeger has quit IRC | 19:31 | |
| fungi | that returns a couple of results i removed from articles a month or two ago | 19:31 |
| anteaya | fungi: awesome | 19:31 |
| fungi | that search should have been returning zero results for a long time now | 19:32 |
| anteaya | notmyname: I wouldn't want to make changes to the Governance wiki page without having a chat with ttx first | 19:32 |
| anteaya | since it would be him maintaining it | 19:32 |
| anteaya | he has mused about scraping governance/programs.yaml for wiki updates | 19:32 |
| anteaya | I can mention it to him when he is back next week | 19:33 |
| *** eharney has quit IRC | 19:33 | |
| anteaya | I use the governance repo as the canonical source | 19:33 |
| fungi | http://git.openstack.org/cgit/openstack/governance/tree/reference/programs.yaml does seem to be current as of the most recent elections | 19:33 |
| anteaya | but the info should be deseminated better, I do agree | 19:33 |
| anteaya | April 16th it was updated, yes | 19:34 |
| *** zehicle_at_dell has quit IRC | 19:37 | |
| *** ihrachyshka has joined #openstack-infra | 19:38 | |
| portante | jeblair, is there a way to try out https://review.openstack.org/91424 prior to accepting the commit? | 19:43 |
| *** marcoemorais has quit IRC | 19:44 | |
| *** marcoemorais has joined #openstack-infra | 19:45 | |
| *** nati_ueno has quit IRC | 19:45 | |
| fungi | portante: you could launch a plain ubuntu 12.04 lts vm, grab openstack-infra/config, use install_jenkins_slave.sh in the root of that repo to set up the system with our usual bits, then check out swift and try to 'tox -e func' | 19:46 |
| *** jaypipes has quit IRC | 19:48 | |
| *** e0ne has quit IRC | 19:48 | |
| notmyname | fungi: http://b.z19r.com/post/did-you-just-tell-me-to-go-fuck-myself ;-) | 19:48 |
| *** e0ne has joined #openstack-infra | 19:48 | |
| fungi | notmyname: bwahahaha | 19:48 |
| *** zhiyan_ is now known as zhiyan | 19:49 | |
| *** maxbit has joined #openstack-infra | 19:53 | |
| *** schofield has joined #openstack-infra | 19:55 | |
| *** schofield has quit IRC | 19:56 | |
| *** schofield has joined #openstack-infra | 19:57 | |
| *** khyati_ has joined #openstack-infra | 19:57 | |
| *** packet has quit IRC | 19:58 | |
| *** nati_ueno has joined #openstack-infra | 19:58 | |
| *** bookwar_away is now known as bookwar | 19:58 | |
| *** eharney has joined #openstack-infra | 20:03 | |
| *** schofield has left #openstack-infra | 20:04 | |
| *** markwash has joined #openstack-infra | 20:04 | |
| jeblair | portante: heh, thus the 'start off non-voting suggestion' | 20:05 |
| *** e0ne has quit IRC | 20:05 | |
| portante | jeblair: okay, so it is likely that won't gate this getting accepted, then? | 20:06 |
| jeblair | portante: correct, as long as it's non-voting, it should be fine. i just aprvd it. | 20:06 |
| *** derekh_ has joined #openstack-infra | 20:06 | |
| portante | great, thanks | 20:06 |
| *** annegentle has joined #openstack-infra | 20:06 | |
| mriedem | i just hit this in a stable/icehouse patch, but the tests passed, never seen it before: "sub_unit.log was > 50 MB of uncompressed data!!!" | 20:07 |
| mriedem | is that an infra issue? | 20:07 |
| fungi | mriedem: there were recent qa checks added to help prevent changes from generating waaaaay too much log noise | 20:08 |
| *** timrc is now known as timrc-afk | 20:08 | |
| mriedem | fungi: hmm, it's a backport from master | 20:08 |
| mriedem | so if it were a problem i'd expect to also see it on master | 20:08 |
| fungi | mriedem: not sure if it checks sub_unit.log though... what's the change? | 20:09 |
| mriedem | fungi: https://review.openstack.org/#/c/91635/ | 20:09 |
| fungi | this could certainly be something else | 20:09 |
| mriedem | fungi: not just my change http://goo.gl/rTgBwV | 20:11 |
| fungi | mriedem: sdague's https://review.openstack.org/85797 merged today | 20:11 |
| mriedem | 6 hits today, all stable branches | 20:11 |
| openstackgerrit | A change was merged to openstack-infra/config: Add swift "in-process" functional test runs https://review.openstack.org/91424 | 20:11 |
| mriedem | sdague: does that change depend on something in the stable branches for it to not puke all over? | 20:12 |
| fungi | mriedem: it's possible nova was already very noisy in icehouse and we should consider only enforcing that on master? | 20:12 |
| mriedem | fungi: sdague: https://bugs.launchpad.net/openstack-ci/+bug/1315138 | 20:14 |
| uvirtbot | Launchpad bug 1315138 in openstack-ci "stable backports failing with "sub_unit.log was > 50 MB of uncompressed data!!!"" [Undecided,New] | 20:14 |
| jeblair | the subunit log _is_ 181M uncompressed; i'm not sure we just want to _ignore_ that | 20:14 |
| *** timrc-afk is now known as timrc | 20:15 | |
| mriedem | jeblair: so i'm not sure what to do here, this is a cherry pick | 20:16 |
| *** andreykurilin_ has quit IRC | 20:16 | |
| mriedem | i have no idea why this particular cherry pick would cause issues either... | 20:16 |
| mriedem | it shouldn't, it's not really a code change | 20:16 |
| mriedem | it's a hacking rule change | 20:16 |
| *** derekh_ has quit IRC | 20:17 | |
| jeblair | mriedem: well as fungi said, it's not this change that is the problem, it's enforcing the rule | 20:17 |
| jeblair | i believe there were some patches merged to fix the log spamminess | 20:17 |
| jeblair | so perhaps we can find them and merge them to the stable branches | 20:17 |
| fungi | mriedem: for comparison, the final py27 run for the master version of that change is much, much smaller... http://logs.openstack.org/49/88649/1/gate/gate-nova-python27/6b7f0fb/ | 20:17 |
| mriedem | yeah, so have to hunt those changes on master down and cherry pick them first | 20:18 |
| mriedem | i'd argue that https://review.openstack.org/#/c/85797/ shouldn't merge until the log spamming is cleaned up in stable branches also | 20:18 |
| mriedem | hard to enforce the rule otherwise | 20:18 |
| jeblair | 27% of the log lines contain the string "stevedore.extension" | 20:19 |
| fungi | it's possible that nova on stable/icehouse has been generating much larger subunit logs for a while and the enforcement numbers were based on master instead | 20:19 |
| jeblair | mriedem: hopefully sdague or mtreinish will know more context | 20:21 |
| fungi | something was tweaked to make debug loglevel be local to the project and not extend to imported dependencies yeah? | 20:21 |
| mriedem | jogo: ^ you know anything about that on master? | 20:21 |
| mriedem | i remember it vaguely | 20:21 |
| jeblair | just found I7317092c0a2bad8efd11ece3ba521d58c2acae42 from jogo | 20:22 |
| fungi | that's the one i was thinking of. also looks very safe to backport | 20:23 |
| mriedem | i'll propose it | 20:23 |
| jeblair | also sdague did I04ff2f0650f1e6035d8de5c5802b96ce4294891b | 20:23 |
| SergeyLukjanov | jeblair, fungi, clarkb, mordred, sdague, I'll be traveling for the next few days, will be available starting from saturday afternoon and completely alive (hopefully) since monday pst (I'll be in CA before the summit) | 20:24 |
| *** flaper87|afk is now known as flaper87 | 20:24 | |
| fungi | SergeyLukjanov: i hope someone told you ca is nowhere near the summit ;) | 20:24 |
| fungi | safe travels! | 20:24 |
| SergeyLukjanov | fungi, yup, I know :) google maps told it | 20:24 |
| jeblair | mriedem: the commit msg in the 2nd one indicates both changes are related | 20:24 |
| SergeyLukjanov | fungi, just some work stuff | 20:24 |
| SergeyLukjanov | fungi, thx ;) | 20:25 |
| jeblair | SergeyLukjanov: thanks, sadly, i won't be in ca next week so i won't see you until the summit | 20:25 |
| mriedem | jeblair: thanks, i'll backport them | 20:25 |
| mriedem | jeblair: and then bug the shit out of sdague and jogo if i missed anything :) | 20:25 |
| *** _david_ has joined #openstack-infra | 20:25 | |
| jeblair | mriedem: send them email copies of the log files ;) | 20:26 |
| _david_ | jeblair, have a look, please: https://gerrit-review.googlesource.com/56712 | 20:26 |
| anteaya | fungi: here is the content of my proposal thus far: https://etherpad.openstack.org/p/Ig9fEfzRQC | 20:26 |
| jeblair | _david_: cool idea! | 20:27 |
| SergeyLukjanov | jeblair, oh, ok :) | 20:27 |
| _david_ | jeblair, the idea is to (ab-)use magic branch to implement wip stickeness: git push wip | 20:27 |
| mgagne | anteaya: She is a Gerrit upstream contributor :D | 20:27 |
| _david_ | jeblair, that way we could seamlessly extend git-review to support it ;-) | 20:28 |
| anteaya | mgagne: I am | 20:30 |
| anteaya | what the heck was it, oh yeah "the" | 20:30 |
| *** khyati_ has quit IRC | 20:30 | |
| anteaya | a doc was missing a "the" | 20:30 |
| anteaya | wooo | 20:30 |
| *** annegentle has quit IRC | 20:30 | |
| bknudson | can we disable a check job for stable/icehouse? any examples? | 20:32 |
| fungi | anteaya: lgtm. i'd attend that! | 20:32 |
| mgagne | bknudson: disable or mark as non-voting ? | 20:32 |
| anteaya | fungi: thanks | 20:33 |
| *** ihrachyshka has quit IRC | 20:33 | |
| anteaya | copy/pasting then pressing send | 20:33 |
| *** _david_ has quit IRC | 20:33 | |
| bknudson | mgagne: there's no point to running the rally job on stable/icehouse for keystone since it was only set up for juno | 20:33 |
| bknudson | it's already nonvoting | 20:33 |
| mgagne | bknudson: =) | 20:33 |
| fungi | bknudson: you specify a branch name or regular expression | 20:34 |
| *** SumitNaiksatam has quit IRC | 20:34 | |
| fungi | bknudson: in the same place where you set voting | 20:34 |
| fungi | bknudson: you should see tons of examples there in the jobs list within layout.yaml | 20:34 |
| bknudson | ok, will take a look | 20:35 |
| *** rfolco has quit IRC | 20:35 | |
| jeblair | huh | 20:35 |
| anteaya | submitted | 20:35 |
| fungi | bknudson: a lot of them set negative lookahead matching on the branch(es) which are undesirable, so they're be skipped | 20:35 |
| anteaya | now i just have to remember to give them slides the week prior | 20:35 |
| jeblair | no wonder they were confused about landin the rally jobs | 20:35 |
| anteaya | jeblair: what do you use to create the slides that go in the presentation repo? | 20:36 |
| jeblair | apparently we have already landed two of them | 20:36 |
| mriedem | jeblair: jogo: sdague: fungi: mega backport https://review.openstack.org/#/q/status:open+project:openstack/nova+branch:stable/icehouse+topic:bug/1315138,n,z | 20:36 |
| *** adalbas has quit IRC | 20:36 | |
| fungi | jeblair: yeah, i approved one the other day where the ptl of the project in question had already +1'd it (and after discussing the details of the tests, and that they're mainly contained within the projects being tested) | 20:37 |
| fungi | though it didn't come to light that they were only written to run on specific branches | 20:38 |
| jeblair | fungi: yeah, i had a few objections to the one i reviewed: a) no ptl weigh in, b) i thought it should start in experimental, c) not keen on gating openstack projects on stackforge projects | 20:38 |
| jeblair | c isn't _technically_ an issue yet, but it's a matter of setting expectations. | 20:39 |
| fungi | right, c) worried me less since it was non-voting check only and the job duration is primarily under the control of the project being tested (depends on the scope of tests they implement in their project) | 20:39 |
| *** andreaf has quit IRC | 20:39 | |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard: Added priorities to tasks https://review.openstack.org/91675 | 20:41 |
| jeblair | the bad news is that the primary key for the approvals table is composed of 4 fields | 20:41 |
| jeblair | the good news is that both it and the timestamp field appear to have the same definition | 20:41 |
| jeblair | https://etherpad.openstack.org/p/gerrit-2.8-reviewstats | 20:42 |
| jeblair | (in the old and new gerrit dbs) | 20:42 |
| jeblair | (i bolded the diffs) | 20:42 |
| fungi | oh, and they dropped a key in the new schema | 20:45 |
| fungi | a secondary key anyway | 20:47 |
| *** mbacchi has quit IRC | 20:47 | |
| *** lakshmi has quit IRC | 20:48 | |
| fungi | i can kind of see the reasoning behind updating that timestamp on change. i think the one i took most exception to is that the change creation timestamp in the changes table had on update set as well, and i couldn't fathom why you'd ever update the creation date | 20:49 |
| *** e0ne has joined #openstack-infra | 20:49 | |
| *** adalbas has joined #openstack-infra | 20:49 | |
| jogo | mriedem: that isn't so big | 20:49 |
| jogo | mriedem: I like how you have the explication marks in the bug title | 20:50 |
| mriedem | jogo: just copy/paste from the log message :) | 20:50 |
| *** yfried has quit IRC | 20:50 | |
| jogo | mriedem: oh hehe | 20:50 |
| mriedem | THE SKY IS FALLING OMFG!!! | 20:50 |
| mriedem | jogo: that's failing on stable/havana also...not sure how though since your debug logging stuff didn't get into havana | 20:51 |
| mriedem | unless it's neutron jobs failing on havana, i didn't check | 20:51 |
| SpamapS | Where would be the best place to report bugs regarding the new skin for gerrit? | 20:52 |
| jogo | mriedem: hmm it could be log related but not sure | 20:52 |
| *** e0ne has quit IRC | 20:52 | |
| *** lakshmi has joined #openstack-infra | 20:53 | |
| jeblair | SpamapS: if it's a css thing, probably openstack-ci on lp. | 20:53 |
| jeblair | SpamapS: (it might be our fault, or we might be able to fix it; if neither of those apply, then upstream gerrit.) | 20:54 |
| SpamapS | jeblair: in this case, the "Report Bug" link is _really_ close to the => arrow on listings | 20:55 |
| SpamapS | jeblair: I click it about 10% of the time :-/ | 20:55 |
| *** e0ne has joined #openstack-infra | 20:56 | |
| SpamapS | ironically, I was pretty sure that is our CSS, and so that particular "Report Bug" link is not the one I want. ;) | 20:56 |
| jeblair | SpamapS: https://gerrit-review.googlesource.com/#/q/status:open | 20:56 |
| SpamapS | also the project name link in reviews used to take you back to open reviews for the project, but now takes you to the completely worthless project description page... _that_ seems like a gerrit bug | 20:56 |
| jeblair | SpamapS: upstream has the too-close-link bug too | 20:57 |
| *** e0ne has quit IRC | 20:57 | |
| SpamapS | jeblair: ahh good | 20:57 |
| jeblair | SpamapS: you might want to file both of those as upstream bugs; we might be able to fix the report-bug closeness by doing more local css customization, so maybe also file that locally (or just fix it :) | 20:57 |
| *** e0ne has joined #openstack-infra | 20:57 | |
| SpamapS | will report upstream then | 20:57 |
| SpamapS | jeblair: before I run off and report another one.. are we also looking to widen the comment boxes so that jenkins results don't wrap? | 20:58 |
| jeblair | SpamapS: where? | 20:59 |
| jeblair | SpamapS: (link to change illustrating problem) | 20:59 |
| jeblair | fungi: though the interpretation of category_id has changed :( | 21:00 |
| jeblair | but i have to write a script anyway, so i can map | 21:00 |
| SpamapS | jeblair: never mind, they're already wider since I looked yesterday. ;) | 21:00 |
| *** e0ne has quit IRC | 21:01 | |
| SpamapS | https://code.google.com/p/gerrit/issues/detail?id=2631 upstream bug about Report Bug proximity to => | 21:02 |
| SpamapS | https://bugs.launchpad.net/openstack-ci/+bug/1315151 openstack-ci version of same report | 21:03 |
| uvirtbot | Launchpad bug 1315151 in openstack-ci "Report Bug is very close to "next page" link in change listings" [Undecided,New] | 21:03 |
| *** david-lyle has quit IRC | 21:03 | |
| *** andreaf has joined #openstack-infra | 21:04 | |
| jeblair | SpamapS: so i have 1216846 rows i need to update... | 21:05 |
| jeblair | SpamapS: like: | 21:05 |
| jeblair | update patch_set_approvals set granted='2011-11-23 17:04:14' where change_id=1861 and patch_set_id=2 and account_id=112 and category_id='Code-Review'; | 21:05 |
| jeblair | SpamapS: do you have any suggestions about the best way to go about that? | 21:06 |
| *** ramashri has joined #openstack-infra | 21:08 | |
| jeblair | fungi: it's possible these entries may have legitimately changed since then (if a user clears and then re-sets a vote to the same value) | 21:09 |
| SpamapS | jeblair: is the only way you can identify them by PK? | 21:09 |
| jeblair | fungi: as we have found -- if a user just re-votes with the same value, it will not update the timestamp. that seems the most likely case, the other one seems less likely. | 21:09 |
| jeblair | SpamapS: yeah, see the schema(s) at the bottom of https://etherpad.openstack.org/p/gerrit-2.8-reviewstats | 21:10 |
| SpamapS | jeblair: http://dev.mysql.com/doc/refman/5.5/en/load-data.html might be interesting to you | 21:10 |
| jeblair | also, some of these rows may have been deleted; and i think we just want to ignore that case. | 21:10 |
| SpamapS | jeblair: note that LOAD DATA INFILE corresponds exactly with SELECT INTO OUTFILE | 21:11 |
| jeblair | SpamapS: however, if rows don't exist, we don't want to re-create them | 21:11 |
| jeblair | SpamapS: so load data probably isn't what we want in this case | 21:12 |
| SpamapS | http://dev.mysql.com/doc/refman/5.5/en/load-data.html | 21:13 |
| SpamapS | doh sorry | 21:13 |
| sdague | jeblair: where is the granted ts coming from? | 21:14 |
| SpamapS | jeblair: so I'd create a new table, and then delete using an outer join. | 21:14 |
| jeblair | sdague: old database | 21:14 |
| *** matty_dubs is now known as matty_dubs|gone | 21:14 | |
| *** smarcet has quit IRC | 21:15 | |
| sdague | can you load it in a 2nd table in the new db, then subselect that field? | 21:15 |
| sdague | it won't be fast, but it will be what you want | 21:16 |
| SpamapS | jeblair: so, on the old database, select into outfile.. new database.. create table new_thing like old_thing; .. load data infile into it.. delete new_table from new_table outer join old_table on .. where old_table.pk is null ... is my short hand making sense? | 21:16 |
| SpamapS | jeblair: finally select into outfile from new_table .. load data infile into old table | 21:17 |
| jeblair | SpamapS: how complicated is that if pk is 4 fields? | 21:17 |
| *** jamielennox|away is now known as jamielennox | 21:18 | |
| *** thomasem has quit IRC | 21:19 | |
| SpamapS | jeblair: just 4 part joins instad of 1 part. as complicated as writing update queries with 4 part where's. | 21:19 |
| *** miqui has quit IRC | 21:19 | |
| *** jgrimm has quit IRC | 21:20 | |
| jeblair | k | 21:20 |
| SpamapS | actually the last part might not need load data / select outfile ... update .. select is a thing. | 21:21 |
| jeblair | i think the thing to decide now is what to do in the case of "If a user has deleted their vote, then left a new one of the same value, we won't be able to tell." | 21:21 |
| jeblair | (a) ignore the issue and let the script overwrite it to the value of their pre-migration vote | 21:21 |
| jeblair | (b) attempt to identify the time range of inproperly updated records and only adjust records with timestamps within that range | 21:21 |
| jeblair | fungi, sdague: ^ opinions? | 21:22 |
| *** sweston has quit IRC | 21:22 | |
| *** bookwar has left #openstack-infra | 21:22 | |
| *** flaper87 is now known as flaper87|afk | 21:23 | |
| sdague | honestly, I'd be fine with a) | 21:23 |
| sdague | it seems like the level of history corruption is very low there | 21:24 |
| *** zhiyan is now known as zhiyan_ | 21:24 | |
| SpamapS | how does one even delete their own vote? | 21:25 |
| jeblair | okay, then i think we can use SpamapS's idea (after we translate the category ids; i can just do that on the intermediate data file directly) | 21:25 |
| jeblair | SpamapS: little X next to your name, or change it to something else then change it back will produce the same result for this issue. | 21:26 |
| *** dkranz has quit IRC | 21:26 | |
| *** e0ne has joined #openstack-infra | 21:26 | |
| *** changbl has quit IRC | 21:27 | |
| jeblair | SpamapS: how do i do that without local filesystem access on the database server? | 21:27 |
| jeblair | (yay trove!) | 21:27 |
| SpamapS | jeblair: load data local infile | 21:29 |
| SpamapS | jeblair: select into outfile.. hm | 21:29 |
| *** adalbas has quit IRC | 21:30 | |
| SpamapS | jeblair: ahh! the mysql client will spit it out in the right flavor of csv I think | 21:31 |
| SpamapS | jeblair: so mysql -e 'select ...' --silent > textfile | 21:31 |
| mordred | ola | 21:32 |
| jeblair | SpamapS: cool, thx | 21:33 |
| *** manishg has quit IRC | 21:33 | |
| *** pdmars has quit IRC | 21:34 | |
| openstackgerrit | A change was merged to openstack-infra/config: Adds insecure option to launch_node.py https://review.openstack.org/86690 | 21:35 |
| *** homeless has quit IRC | 21:35 | |
| *** aysyd has quit IRC | 21:35 | |
| *** zns has quit IRC | 21:37 | |
| portante | jeblair: fyi, in-process func tests can use run-unittests.sh, it takes a "version" for the first argument, not a pure environment | 21:37 |
| portante | so if we had yaml setup that used run-tox.sh that would work | 21:38 |
| *** dangers is now known as dangers_away | 21:38 | |
| portante | but I'll have to look at this later | 21:38 |
| jeblair | portante: i think we do | 21:38 |
| portante | thanks for the help anyway, see http://logs.openstack.org/78/87078/4/check/gate-swift-unittests-func/8ce760f/console.html | 21:38 |
| jeblair | SpamapS: actually, your shorthand isn't making sense | 21:38 |
| jeblair | SpamapS: can you spell it out a bit more at the bottom of https://etherpad.openstack.org/p/gerrit-2.8-reviewstats ? | 21:38 |
| jeblair | SpamapS: particularly, i think you changed some metavars in your shorthand (not sure what happens to the table "new_thing" | 21:39 |
| jeblair | SpamapS: i'm not as concerned about the specific query syntax as i am in understand which things you think should be done to which tables and in what sequence | 21:40 |
| SpamapS | jeblair: yeah I did not think it through :) will add to etherpad | 21:40 |
| openstackgerrit | A change was merged to openstack-infra/config: Move dsvm-virtual-ironic out of experimental https://review.openstack.org/90452 | 21:41 |
| *** adalbas has joined #openstack-infra | 21:43 | |
| mordred | SpamapS: why not just mysqldump -hold_gerrit reviewdb patch_set_approvals > psa.sql ; mysql -hnew_gerrit_trove -e 'create schema fixthings' ; cat psa.sql | mysql -hnew_gerrit_trove fixthings ? | 21:46 |
| SpamapS | jeblair: ok | 21:47 |
| *** changbl has joined #openstack-infra | 21:48 | |
| mordred | SpamapS: and then do a multi-table update query with a join between the two so yo're just updating the ts columns and setting them to the values in the old table? | 21:48 |
| SpamapS | mordred: we'd need a --where in there.. but yeah that could work | 21:48 |
| SpamapS | mordred: probably a lot simpler. We started from a very different place than we finished. :) | 21:48 |
| SpamapS | jeblair: anyway, my now exposed as overly complex instructions are at the bottom of gerrit-2.8-reviewstats ... | 21:49 |
| mordred | update new_psa n, old_psa set n.ts_field = o.ts_field where o.key1 = n.key1 and o.key2 = n.key2 ... | 21:49 |
| jeblair | SpamapS: woo! thanks | 21:49 |
| SpamapS | Yeah there's only one column that needs updating isn't there? | 21:49 |
| jeblair | SpamapS: yeah | 21:49 |
| mordred | SpamapS: I get the benefit of swooping in and reading all of your work | 21:49 |
| SpamapS | ok so let's go to the better idea.. | 21:49 |
| *** bhuvan has quit IRC | 21:49 | |
| jeblair | mordred: you want to spell that out in the etherpad? | 21:49 |
| mordred | jeblair: I think SpamapS is about to | 21:50 |
| jeblair | k | 21:50 |
| *** ArxCruz has joined #openstack-infra | 21:50 | |
| *** zehicle_at_dell has joined #openstack-infra | 21:51 | |
| *** aconrad has quit IRC | 21:51 | |
| fungi | i step away to cook dinner and suddenly so many pings in scrollback | 21:52 |
| *** adalbas has quit IRC | 21:52 | |
| mordred | fungi: NEVER EAT AGAIN!!! | 21:52 |
| lifeless | sphi | 21:52 |
| mordred | jeblair: the better idea SpamapS is writing out can also be tested in a local db too, to make sure the update query is right | 21:52 |
| mordred | jeblair: you know, load a dump of the two tables into local schemas | 21:53 |
| *** mburned is now known as mburned_out | 21:53 | |
| mordred | lifeless: that's one of the odder things you've ever said | 21:53 |
| jeblair | mordred: yeah, i was planning on doing that regardless. so far i've learned my version of mysql doesn't support load data local infile. | 21:53 |
| fungi | 1.2 million rows before breakfast | 21:54 |
| SpamapS | mordred: ok, look now.. | 21:54 |
| SpamapS | https://etherpad.openstack.org/p/gerrit-2.8-reviewstats | 21:54 |
| SpamapS | jeblair: ^ better idea done | 21:54 |
| *** bhuvan has joined #openstack-infra | 21:55 | |
| mordred | jeblair: that's because load data local infile is a security fisk | 21:55 |
| fungi | jeblair: still catching up, but perhaps the "problem" updates all have the exact same timestamp rather than that time or newer | 21:55 |
| *** marun has quit IRC | 21:55 | |
| lifeless | jogo: jelmer, jml, uhm need to check the acls to remember who else | 21:55 |
| SpamapS | jeblair: did load data local infile get removed finally? | 21:55 |
| mordred | SpamapS: do you really need the edit things step? | 21:55 |
| SpamapS | why hasn't mysql just copied the postgres COPY command yet? | 21:55 |
| lifeless | SpamapS: 'oh hi' | 21:55 |
| jeblair | fungi: it's a range. | 21:55 |
| SpamapS | mordred: I don't know | 21:55 |
| SpamapS | mordred: I got the feeling jeblair didn't want to revert _ALL_ of the votes | 21:55 |
| mordred | I don't think you do | 21:55 |
| *** marun has joined #openstack-infra | 21:55 | |
| SpamapS | lifeless: yo | 21:56 |
| jogo | lifeless: you can review it right ;) | 21:56 |
| mordred | how do you select the ones you don't want to revert? | 21:56 |
| SpamapS | don't know :-/ | 21:56 |
| * mordred fine with whatever - just asking annoying questions | 21:56 | |
| SpamapS | that's for jeblair | 21:56 |
| jeblair | er | 21:56 |
| jeblair | we only want to update rows that exist | 21:56 |
| jeblair | that's the only selection criteria | 21:56 |
| mordred | right. that should be handled by the join condition | 21:57 |
| jeblair | there is an edit step to change the label values | 21:57 |
| mordred | gotcha | 21:57 |
| jeblair | which i have written twice now for TSV files, and am about to write a third time for CSV ones :/ | 21:57 |
| openstackgerrit | Brant Knudson proposed a change to openstack-infra/config: Add stable filter for keystone python33 and rally https://review.openstack.org/91690 | 21:57 |
| mordred | you could do it as a query after you load the dump? | 21:57 |
| jeblair | mordred: i think so | 21:57 |
| *** marun has quit IRC | 21:58 | |
| fungi | i didn't know you could do csv with mysqlclient stdout redirection... only ever knew about tsv | 21:58 |
| * mordred feels bad he was offline travelling on the day with the fun db queries | 21:58 | |
| SpamapS | thats what you get | 21:59 |
| SpamapS | well that, and jet lag | 21:59 |
| SpamapS | and a mediocre meal | 22:00 |
| SpamapS | and all the orange juice you can drink | 22:00 |
| *** marcoemorais has quit IRC | 22:00 | |
| fungi | tang, the drink that went to the moon | 22:01 |
| jeblair | SpamapS, mordred: create database is denied in trove | 22:01 |
| fungi | i think you can probably do it from troveclient (though i've only ever done it from the provider's webui) | 22:02 |
| jeblair | i mean, i guess we could use the trove api to make a new db and give the user access | 22:02 |
| bknudson | I can't WIP my review on stable/icehouse? | 22:02 |
| *** marcoemorais has joined #openstack-infra | 22:03 | |
| *** dstanek is now known as dstanek_zzz | 22:03 | |
| fungi | bknudson: work-in-progress is now a workflow -1 vote since the gerrit upgrade | 22:03 |
| fungi | bknudson: so review the change and set your workflow vote to -i | 22:04 |
| bknudson | fungi: I don't have that button -- https://review.openstack.org/#/c/91677/ | 22:04 |
| bknudson | it shows up on this one -- https://review.openstack.org/#/c/91690/ | 22:04 |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard-webclient: Added task priority to edit and display of tasks https://review.openstack.org/91693 | 22:04 |
| *** yfried has joined #openstack-infra | 22:05 | |
| fungi | oh, interesting... i wonder whether we have an exclusive something set on stable/.* branches | 22:05 |
| jeblair | fungi, mordred: i have granted gerrit2 access to fixthings and reviewdb on the production trove instance | 22:06 |
| fungi | jeblair: k | 22:06 |
| mattoliverau | Morning | 22:08 |
| *** ramashri has quit IRC | 22:09 | |
| mordred | jeblair: ++ | 22:11 |
| *** e0ne has quit IRC | 22:11 | |
| bknudson | opened a bug https://bugs.launchpad.net/openstack-ci/+bug/1315178 | 22:12 |
| uvirtbot | Launchpad bug 1315178 in openstack-ci "User can't mark stable review as work in progress" [Undecided,New] | 22:12 |
| *** e0ne has joined #openstack-infra | 22:12 | |
| fungi | zaro: what's the chance that our exclusive label workflow 0..+1 for openstack-stable-maint group on | 22:12 |
| fungi | refs/heads/stable/* is preventing change owners from setting workflow -1? | 22:13 |
| fungi | and any suggestions on a workaround? | 22:13 |
| *** markmcclain has quit IRC | 22:14 | |
| mgagne | is there an place I can find stats about openstack infra? ie like the number of nodes created per day | 22:15 |
| *** marcoemorais has quit IRC | 22:15 | |
| fungi | mgagne: http://graphite.openstack.org/ plus elbow grease | 22:15 |
| mgagne | fungi: could I borrow your elbow? any wiki with pre-defined queries? | 22:16 |
| fungi | mgagne: though the graphs at the bottom of http://status.openstack.org/zuul/ provide some great starting points | 22:16 |
| fungi | (protip... see the get params in the urls for those images) | 22:17 |
| SlickNik | jeblair / fungi / mordred: At some point I want to catch up with you guys and get a quick brain-dump on how you're using trove — what's working well, and what's painful / lacking. Perhaps f2f in ATL? | 22:18 |
| mordred | SlickNik: we'd love to give you that feedback :) | 22:18 |
| fungi | SlickNik: i'm pretty sure we have feedback | 22:19 |
| *** changbl has quit IRC | 22:19 | |
| jeblair | mordred: double check what i've been putting in the ep | 22:19 |
| mordred | jeblair: looking | 22:19 |
| jeblair | mordred, fungi: i did those 3 update statements, and i was a little surprised that granted didn't get updated; am i missing something? | 22:19 |
| fungi | SlickNik: any insight into why rackspace trove instances are reachable on 3306/tcp from every tenant in the same region? | 22:19 |
| jeblair | (i thought i was going to have to do granted=granted) | 22:20 |
| *** lakshmi has quit IRC | 22:21 | |
| *** doug-fish has left #openstack-infra | 22:22 | |
| *** dstanek_zzz is now known as dstanek | 22:22 | |
| *** lakshmiS has joined #openstack-infra | 22:22 | |
| *** lakshmiS has quit IRC | 22:23 | |
| mordred | jeblair: that looks like good output | 22:23 |
| jeblair | mordred: any idea why granted didn't update when i ran those update statements? | 22:23 |
| mordred | it says it changed a million rows | 22:24 |
| mordred | wait- when you say "those update statements" | 22:24 |
| SlickNik | fungi: AFAIK they have been using something different from the upstream secgroups extension (which only landed in Havana). | 22:24 |
| jeblair | mordred: no, the 3 update statements above | 22:24 |
| *** rlandy has quit IRC | 22:24 | |
| mordred | I wouldn't expect those to change granted at all | 22:24 |
| SlickNik | fungi: Not sure about the details of that though. | 22:24 |
| jeblair | mordred: then why do we have this problem in the first place? | 22:25 |
| mordred | I would guess something having to do with the restore of the dump | 22:25 |
| mordred | and/or something to do with the data migrations in gerrit | 22:25 |
| SlickNik | jeblair / mordred / fungi: I'll ensure I come find you guys in ATL. :) | 22:25 |
| mordred | like, part of the upgrade ran data migration scripts | 22:25 |
| mordred | maybe I should go read those | 22:26 |
| jeblair | mordred: not the dump; the problem hasn't appeared loading data so far, and it did not affect SUMB changes -- those labels didn't update. | 22:26 |
| fungi | SlickNik: good answer to a somewhat rhetorical question (part of our complaints probably stem from a provider's implementation choices) | 22:26 |
| jeblair | mordred: it only affected the rows where lables changed | 22:26 |
| jeblair | mordred: which, of course, is what i'm recreating in those update statements | 22:26 |
| jeblair | `granted` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP, | 22:26 |
| jeblair | mordred: ^ i would have thought the problem stemmed from "ON UPDATE CURRENT_TIMESTAMP" | 22:26 |
| mordred | yah. I follow you now | 22:27 |
| jeblair | which i thought meant "if this row is updated, always set granted to the current time" | 22:27 |
| *** bknudson has quit IRC | 22:27 | |
| *** yidclare has quit IRC | 22:27 | |
| jeblair | if we don't have an explanation for why we haven't seen that, then i think i should do another dry run on the trove instance | 22:28 |
| mordred | jeblair: ++ | 22:28 |
| mordred | jeblair: also, I'm going to go read the schema migrations | 22:28 |
| *** lakshmiS has joined #openstack-infra | 22:28 | |
| jeblair | i will make a 3rd database on the prod trove instance to stand in for reviewdb and dry run again | 22:29 |
| *** aconrad has joined #openstack-infra | 22:29 | |
| mgagne | fungi: close enough: http://docs.openstack.org/infra/publications/2014-lca-zuul/#(8) | 22:29 |
| fungi | SlickNik: at the moment _my_ biggest concern is simply that we have to mask out the dns names of our trove instances in our otherwise public configuration because anyone who knows the ip address can potentially leverage pre-authentication bugs in mysqld or start trying to brute-force our account credentials (and i'm very much not a fan of "security by obscurity" there) | 22:30 |
| fungi | mgagne: those are fairly recent numbers if you're not looking for *current* numbers, yes | 22:31 |
| mgagne | fungi: just wanted to get an order of magnitude, thanks =) | 22:31 |
| *** marcoemorais has joined #openstack-infra | 22:32 | |
| *** prad has quit IRC | 22:33 | |
| jeblair | fungi, mordred: we need to fix mysql backups on review.o.o (they aren't happening because of the incorrect defaults file) | 22:36 |
| jeblair | after we do that, we should check/fix bup. | 22:37 |
| fungi | oh, great point. the old backups probably assumed a defaults file which pointed at the local mysqld | 22:38 |
| *** thedodd has quit IRC | 22:38 | |
| sdague | so... do we have a policy around people uploading other people's patches without credit? | 22:38 |
| sdague | an instance of that just happened in the devstack queue | 22:38 |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Remove Closed bugs that haven't been seen in two weeks https://review.openstack.org/91695 | 22:38 |
| anteaya | sdague: not so much policy that I know but frowning certainly | 22:39 |
| fungi | sdague: uploading someone else's work withouit credit shouldn't require a policy. it's just flat-out wrong | 22:39 |
| sdague | yes, once it was pointed out I changed my -1 to a -2 | 22:39 |
| jeblair | sdague: link? | 22:40 |
| sdague | https://review.openstack.org/#/c/90965/ is the review in question | 22:40 |
| *** wenlock has quit IRC | 22:40 | |
| fungi | copyright infringement is illegal in most jurisdictions | 22:40 |
| fungi | "don't break the law" probably doesn't need encoding in policy | 22:40 |
| anteaya | ah sorry, i was meaning someone committing on behalf of someone else, not theft | 22:41 |
| *** jhesketh has quit IRC | 22:41 | |
| sdague | that's fine, I wasn't sure if we had some more gentle redirection | 22:41 |
| anteaya | yeah, I think there is already policy on outright stealing someone else's work | 22:41 |
| anteaya | I think someone who does this has moved beyond the gentle redirection stage | 22:42 |
| cody-somerville | It may be a mistake. | 22:42 |
| jeblair | yeah, let's not go overboard | 22:42 |
| sdague | yeah, agreed | 22:43 |
| jeblair | it's worth noting that the original patch had no copyright information | 22:43 |
| sdague | well, it doesn't need to | 22:43 |
| jeblair | which means there was no authorship information to remove as far as that goes | 22:43 |
| fungi | well, keep in mind that different cultures place different levels of value on "intellectual property" | 22:43 |
| cody-somerville | Plus it's probably not copyright infringement either. The other patch was submitted under CLA. Attribution is not required by Apache license. | 22:43 |
| *** gondoi is now known as zz_gondoi | 22:43 | |
| sdague | cody-somerville: sure, though it is a CLA violation | 22:43 |
| sdague | https://review.openstack.org/static/cla.html - #5 | 22:43 |
| jeblair | sdague: i'm not saying it needed it, i'm saying that the second person may not have violated the license | 22:43 |
| sdague | sure, I don't actually want to flame the person. I was just trying to figure out if we had a more general education in "don't do that" | 22:44 |
| *** marcoemorais has quit IRC | 22:44 | |
| sdague | already sorted out | 22:44 |
| fungi | i tend to assume ignorance rather than malice, however it does need crediting if it merges in my (personal) opinion | 22:44 |
| sdague | the original contributor emailed me and was pretty mad about it | 22:45 |
| anteaya | about what part of it? | 22:45 |
| sdague | that it was his work | 22:45 |
| fungi | a "co-authored-by" header in the commit message might have averted all of that | 22:45 |
| sdague | with no reference ot it | 22:45 |
| sdague | yeh | 22:45 |
| anteaya | right, that makes sense | 22:45 |
| *** marcoemorais has joined #openstack-infra | 22:45 | |
| sdague | which is all fair | 22:45 |
| anteaya | has the author of the second patch said anything? | 22:45 |
| sdague | not yet | 22:45 |
| sdague | I just got the email an hour ago | 22:46 |
| *** marcoemorais has quit IRC | 22:46 | |
| anteaya | makes sense | 22:46 |
| sdague | so it's all late breaking | 22:46 |
| jeblair | the way the project operates things like oslo, however, goes to show that as a project we don't actually value git-level attribution very much | 22:46 |
| anteaya | I would be interested to hear his perspective, if he shares it | 22:46 |
| sdague | honestly, if you haven't been steeped in open source for a long time, copyright nuances escape most people. Especially under a certain age | 22:47 |
| fungi | gently pointing them toward the co-authored-by header discussed at https://wiki.openstack.org/wiki/GitCommitMessages#Including_external_references is probably warranted | 22:47 |
| *** marcoemorais has joined #openstack-infra | 22:47 | |
| cody-somerville | Although I'd probably be inclined to credit the other author in some way, the original change was abandoned and it would be a shame if the wrong message was given that you can't pick up abandoned changes and continue with them. | 22:47 |
| sdague | so trying to figure out the best education path | 22:47 |
| anteaya | I think in these circumstances direct communication with the original author is called for | 22:48 |
| anteaya | at least an attempt to email the person | 22:48 |
| anteaya | that is why there is an email in every git commit | 22:48 |
| *** rahmu has left #openstack-infra | 22:49 | |
| openstackgerrit | A change was merged to openstack-dev/specs-cookiecutter: Add optional spell checker target https://review.openstack.org/88072 | 22:49 |
| cody-somerville | fungi: ++ on sending link to that discussion. | 22:49 |
| fungi | cody-somerville: i'm tempted to support amending the (new in gerrit 2.8!) abandon control acl to all registered users. the problem is that i believe it would allow people to abandon the patches of others as well. maybe an upstream bug to allow others to adopt abandoned changes is in order | 22:49 |
| cody-somerville | maybe we should move back to bzr and then the original history will be preserved ;) | 22:50 |
| fungi | i envision an "adopt this change" button on abandoned changes, which also solves the problem of conveying the change owner field to another account too | 22:51 |
| jeblair | cody-somerville: i realize you are not serious, but nonetheless, i don't see how that would help....? | 22:51 |
| *** sacharya has quit IRC | 22:52 | |
| sdague | fungi: that would be a great feature | 22:52 |
| *** andreykurilin_ has joined #openstack-infra | 22:52 | |
| cody-somerville | jeblair: They'd bzr clone the other person's change, make their commits, push it up for review. Those changes get merged into trunk as one commit but the history of the branch submitted remains "inside" that single commit. | 22:52 |
| jeblair | (the only thing you lose by not being able to adopt an abandoned change is the review history, which is not something that is in either scm) | 22:53 |
| jeblair | cody-somerville: you can do that with git too; just change the change-id and push up a new patchset. which is not entirely dissimilar to what happened here. | 22:53 |
| fungi | jeblair: well, with normal workflows, the change author would also remain as it had been originally unless you reset it | 22:53 |
| fungi | but yes, changing the change-id would also get you the same | 22:53 |
| cody-somerville | jeblair: that's just a single commit. with bzr the original separate commits would be retained in the history. You wouldn't use commit --amend with bzr. | 22:54 |
| fungi | bazaar is bizarre | 22:54 |
| cody-somerville | on the target branch, you'd just see a normal single commit | 22:55 |
| jeblair | cody-somerville: we're kind of comparing apples and oranges here; you're suggesting that things about gerrit's workflow would be solved by using bzr | 22:55 |
| jeblair | cody-somerville: git (and even gerrit) can certainly support merging branches | 22:56 |
| jeblair | we have chosen quite deliberately not to do that | 22:56 |
| jeblair | (except in unusual circumstances) | 22:56 |
| cody-somerville | Ah. I've only really ever used git with gerrit. | 22:56 |
| jeblair | because of the way we want the history to look | 22:56 |
| cody-somerville | You can get same thing with bzr. It basically smooshes all the commits being proposed into a single one. | 22:57 |
| *** arnaud___ has quit IRC | 22:57 | |
| cody-somerville | but the actual ancestry, and the individual commits, are still preserved and can be seen if you pass in the right arguments to bzr | 22:58 |
| fungi | git calls that "squash" but yes | 22:58 |
| fungi | oh, well it also just concatenates all the commit messages in git when you do that | 22:58 |
| fungi | so not really multi-layered when you squash | 22:58 |
| sdague | yeh, you could do all the same things with feature branches though | 22:58 |
| fungi | merge commits may indeed be a better analogy | 22:58 |
| jeblair | mordred: aha, i was wrong -- the timestamps were updated, | 22:59 |
| sdague | but people don't really want intermediate crap patches in the history :) | 22:59 |
| jeblair | mordred: that reassures me | 22:59 |
| *** etoews has quit IRC | 22:59 | |
| fungi | sdague: well, some people seem to think they do (there have been discussions about people who dislike gerrit's requirements around squashing and rebasing and who prefer github's pull requests featuring branches with iterative commits on them) | 23:00 |
| jeblair | fungi: those people don't tend to be reviewers :) | 23:00 |
| fungi | i find the github workflow very annoying because, yes, nearly impossible to review sanely | 23:01 |
| mgagne | fungi: "oops, fixed typo" | 23:01 |
| *** andreaf has quit IRC | 23:02 | |
| mordred | jeblair: oh good. except we don't want them updated when we load the dump | 23:02 |
| jeblair | mordred: i think it's the update that does it, and that should be easy t ofix. | 23:02 |
| fungi | there is, after all, a couple decades of experience which has led to the lkml workflow. and as much as new devs tend to complain about how hard it is to get a patch into the mainline linux kernel, it's still around | 23:02 |
| sdague | right. That's for projects with 1 - 3 developers, where everyone has all the project context in their head all the time :) | 23:02 |
| mordred | jeblair: yah. just alter the column in the old table to not have the on update, yeah? | 23:02 |
| mgagne | fungi: some changes have 40 patch sets, imagine having 40 commits to review in a PR :-/ | 23:02 |
| sdague | mgagne: only 40 | 23:03 |
| mordred | oh - which is what you did in the alter | 23:03 |
| fungi | mgagne: "can you squash that so i can review it in context?" ;) | 23:03 |
| cody-somerville | In launchpad, you review the diff of the branch, not each 40 commits. | 23:03 |
| sdague | initial baremetal series was about 30 patches, and got into the 60s on revisiosn | 23:03 |
| jeblair | mordred: no, the alter doesn't seem to do it; just update | 23:03 |
| fungi | cody-somerville: so lp does the squash for you. awesome | 23:03 |
| *** ArxCruz has quit IRC | 23:03 | |
| mordred | jeblair: I mean, if you alter the table as you have in the script there to remove the on update clause, then do the alter | 23:03 |
| mordred | it _shoudln't_ update the timestamp at that point, right? | 23:04 |
| jeblair | mordred: i didn't remove the onupdate clause | 23:04 |
| mordred | bah. I just saw the alter on the other column, my bad | 23:04 |
| *** andreykurilin_ has quit IRC | 23:04 | |
| jeblair | mordred, fungi: i updated the etherpad to reflect using 'granted=granted' in the updates | 23:05 |
| fungi | cody-somerville: i'll be the first to admit that post-rcs/cvs (and to some extent post-svn) there were several contenders with better models/features compared to git. in the end git won, and is still better than svn, so i'm happy enough to use it ;) | 23:05 |
| jeblair | mordred: er i don't think what you are writing is necessary | 23:05 |
| fungi | jeblair: that's basically what we do with the update queries on project renames too | 23:05 |
| *** andreykurilin_ has joined #openstack-infra | 23:05 | |
| cody-somerville | fungi: nice thing is you can see commit message for every change they make in response to reviews plus diff (and just the diff of the commit, so you don't get other cruft in there). | 23:06 |
| jeblair | mordred, fungi: i just dry-ran the current instructions locally and it works | 23:06 |
| mordred | ok. cool | 23:06 |
| fungi | jeblair: excellent | 23:06 |
| cody-somerville | fungi: but then when it lands, you get the single merge commit | 23:06 |
| mordred | cody-somerville: that's just how the output of the log commands work | 23:06 |
| jeblair | doing it again on the prod server | 23:07 |
| cody-somerville | fungi: but the intermediate commits can still be found hidden within that merge commit (hence why I even brought this up) | 23:07 |
| mordred | cody-somerville: it's not a difference in the tools themselves. you can configure git to not show the intermediary commits in its log output too | 23:07 |
| mordred | and you can configure bzr to show them all the time | 23:07 |
| fungi | cody-somerville: yeah, i get that a commit object is multi-layered there. it's definitely an interesting model | 23:07 |
| SpamapS | I often wonder if keeping all those intermediary commits is one of the reasons git appears so much faster than bzr | 23:08 |
| mordred | people in the gh pr model keep the intermediary commits, and the review is done on the diff of the branches | 23:08 |
| cody-somerville | I think part of the reason git appears so much faster is that there are better/faster places to host other than launchpad ;) | 23:08 |
| *** signed8bit is now known as signed8bit_ZZZzz | 23:08 | |
| fungi | cody-somerville: touché | 23:08 |
| mattoliverau | good old sourceforge did bzr... git still won :P | 23:09 |
| mordred | I think we could have a gerrit-like branch-merge review model that is a mix of both - what we'd be losing is a cleanly bisectable hisotyr | 23:09 |
| cody-somerville | mordred: How? | 23:09 |
| mordred | the tools could be made to present the diffs for review in the manner consistent with our current review workflow | 23:09 |
| SpamapS | bisect works fine w/ merges last I checked. | 23:10 |
| fungi | mordred: well, if you bisect along the branch history and not within the sub-branches, i'm not sure you really lose bisectability | 23:10 |
| mordred | possibly so | 23:10 |
| jeblair | mordred: this takes me back 2.5 years. :) | 23:10 |
| mordred | but git by default flattens it, as cody mentioned | 23:10 |
| mordred | so, by default, you get a thing that looks like each of the intermediary commits is a real thing | 23:10 |
| sdague | also, on the anti bzr pov, I'd have to stab you all if we were using it :) | 23:10 |
| mordred | even though they aren't interesting | 23:10 |
| *** Alexandra_ has joined #openstack-infra | 23:11 | |
| mordred | sdague: be glad you arrived post-switch then :) | 23:11 |
| cody-somerville | I wish git had a better "ui". :) | 23:11 |
| cody-somerville | bzr was simple and intuitive to use for the most part. | 23:11 |
| mordred | cody-somerville: I actually at this point think git's ui is 1000000000 better | 23:11 |
| sdague | mordred: yes, very much so | 23:11 |
| mordred | but it does have a steeper learning curve | 23:11 |
| mordred | I find going back to bzr very painful now | 23:11 |
| sdague | cody-somerville: I've only heard that ever said by people that spent forever with bzr :) | 23:11 |
| mordred | even though I was a fanboi for quite a while | 23:11 |
| fungi | i still know darcs and hg worshippers too | 23:12 |
| sdague | hg at least has a reasonable size community around it | 23:12 |
| * SlickNik wonders if he should throw hg into the mix | 23:12 | |
| *** SumitNaiksatam has joined #openstack-infra | 23:12 | |
| mordred | I think the python community obsession with it will keep it around for a while | 23:13 |
| jeblair | mordred, fungi: testreview db on prod trove has final test version of patch_set_approvals table. want to double check it? | 23:13 |
| mordred | jeblair: looking | 23:13 |
| fungi | jeblair: looking | 23:13 |
| *** jhesketh_ has joined #openstack-infra | 23:13 | |
| sdague | mordred: probably, though this paints a pretty clear picture - http://redmonk.com/sogrady/2013/12/19/dvcs-and-git-2013/ | 23:13 |
| *** jhesketh_ is now known as jhesketh | 23:14 | |
| jhesketh | Morning | 23:14 |
| fungi | jeblair: did we have any examples of the incorrectly updated patchset approvals in prod? | 23:14 |
| jeblair | fungi: the etherpad has one | 23:15 |
| mordred | jeblair: I see a mix of Code-Review style and SUBM in the table | 23:15 |
| *** zns has joined #openstack-infra | 23:16 | |
| mordred | jeblair: does SUBM need to get changed too? | 23:16 |
| jeblair | mordred: check out line 118 in the etherpad | 23:16 |
| fungi | mordred: i think that's expected | 23:16 |
| mordred | ossum | 23:16 |
| jeblair | mordred: those are the values in the current db (short answer, no, subm stays the same) | 23:16 |
| mordred | thanks | 23:16 |
| mordred | that's great. | 23:16 |
| clarkb | hg is terrible. bzr > hg. that is all back to afk | 23:16 |
| fungi | clarkb: go back to your vacation. i DID NOT see you in here | 23:17 |
| jeblair | there's no arguing with that :) | 23:17 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Remove Closed bugs that haven't been seen in two weeks https://review.openstack.org/91695 | 23:17 |
| *** mikal is now known as stillbert | 23:18 | |
| mordred | clarkb: ++ | 23:19 |
| fungi | anyway, we should all just be thankful. i have a friend who runs infotech at epic games, and he has to support their perforce revision control server cluster | 23:19 |
| *** moted has quit IRC | 23:20 | |
| fungi | pretty much all he does is whine about how he wishes they could just switch to *any* modern open/free-developed dvcs | 23:20 |
| mordred | jeblair: it looks good to me | 23:20 |
| mordred | jeblair: there are old timestamp values, and there are also tiemstamp values still from today | 23:20 |
| jeblair | mordred: since i've done all the steps to test, there's no reason i can't just run the last step again changing the schema name to reviewdb, right? | 23:21 |
| mordred | jeblair: that's right | 23:21 |
| jeblair | (to do it for realz) | 23:21 |
| *** melwitt has quit IRC | 23:21 | |
| jeblair | cool | 23:21 |
| jeblair | just waiting on the go from fungi (no rush) | 23:21 |
| fungi | jeblair: i'm still staring. i switched computers so took me a minute to re-find the etherpad | 23:22 |
| fungi | evening and all | 23:22 |
| morganfainberg | jeblair, clarkb, ping wanted to ask if there was a serious issue if keystone under mod_wsgi produced a slightly different format for the log. notably, apache 2.2 prepends some... data that can't be customized to the log line | 23:22 |
| mordred | morganfainberg: got examples? | 23:23 |
| morganfainberg | mordred, yeah [Thu May 01 16:22:52 2014] [error] 2014-05-01 16:22:52.092 32280 DEBUG keystone.common.wsgi [-] arg_dict: {} __call__ /opt/stack/keystone/keystone/common/wsgi.py:181 | 23:23 |
| Alex_Gaynor | So there's a revision at the top of the gate pipeline, and one if it's jobs hasn't started, and there's no obvious reason why not | 23:23 |
| SpamapS | fungi: Perforce was really awesome. 10 years ago. :) | 23:23 |
| morganfainberg | mordred, that is w/ apache, w/o take the [<datestamp>][error] off | 23:23 |
| mordred | morganfainberg: wow. it's so nice of it to prepend that | 23:23 |
| SpamapS | fungi: I will say, their meld-ish tool did a hell of a lot better showing you "what was yours, what was theirs, what is the common base" | 23:24 |
| mordred | you can't customize that with logline? | 23:24 |
| *** imcsk8 has quit IRC | 23:24 | |
| *** asettle is now known as Captain_Kirk | 23:24 | |
| morganfainberg | mordred, nope, this is errorlog. 2.4 has the ErrorLogFormat directive that lets you customize | 23:24 |
| *** imcsk8 has joined #openstack-infra | 23:24 | |
| sdague | why is debug in error log? | 23:24 |
| mordred | oh! wait - why wouldn't we just use 2.4 then? | 23:24 |
| sdague | mordred: need Ubuntu >= 13.10 | 23:24 |
| *** e0ne has quit IRC | 23:24 | |
| mordred | gotcha | 23:24 |
| morganfainberg | sdague, all output / debug from the CGIs mod_wsgi etc go to apache errorlog | 23:25 |
| morganfainberg | sdague, customlog (accesslog?) is meant for request/response etc info | 23:25 |
| sdague | so I feel like in this situation you really shouldn't be just printing to stdout | 23:25 |
| *** Captain_Kirk is now known as SettleDown | 23:25 | |
| sdague | instead should have a dedicated log file | 23:25 |
| sdague | is that an option with wsgi? | 23:26 |
| morganfainberg | sdague, that was the next step to try, i am pretty sure i can, but, it means the screen log watcher will need to be fixed in devstack in this case | 23:26 |
| *** imcsk8 has quit IRC | 23:26 | |
| sdague | sure | 23:26 |
| morganfainberg | sdague, not a big deal, but was asking the path of least resistance :) | 23:26 |
| *** imcsk8 has joined #openstack-infra | 23:27 | |
| fungi | jeblair: apologies for the delay. the test table lgtm | 23:27 |
| sdague | but I think the volume of log data we do, real people would not appreciate that in their apache error_log | 23:27 |
| morganfainberg | sdague, i expected to need to fix some logging stuff in these cases | 23:27 |
| fungi | jeblair: select * from patch_set_approvals where change_id=51114; shows granted as circa 2013 for all votes | 23:27 |
| morganfainberg | sdague, well it is an isolated error log specific to keystone (we run it in a vhost) so only keystone logs are in there | 23:27 |
| fungi | which is as it should be | 23:27 |
| morganfainberg | sdague, not the global errorlog | 23:28 |
| jeblair | Alex_Gaynor: unfortunately, logging that would answer that question is in a pending change, so i don't have it right now. | 23:28 |
| sdague | morganfainberg: ok, fair, still seems weird | 23:28 |
| morganfainberg | sdague, it's how many things run under apache works. i agree, this is something i think i can fix with logging.conf | 23:28 |
| Alex_Gaynor | jeblair: hmm, is there any way to jiggle the switch, or similar so it doesn't block the whole pipeline? | 23:28 |
| morganfainberg | sdague, and consistent logging amkes it easier to point people in the right direction | 23:28 |
| sdague | yeh, agreed. | 23:29 |
| morganfainberg | sdague, with apache 2.4 (when we move there) i might just make it use errorlogformat...but i'll cross that hurdle when we get to that point | 23:29 |
| jeblair | Alex_Gaynor: yeah, uploading a new patchset to it will kick it out, or we could restart zuul. | 23:29 |
| sdague | did the apache issue where logging too much data can stall the server ever get fixed? | 23:29 |
| sdague | I thought if you write > 4k in one go it goes missing | 23:30 |
| jeblair | mordred, fungi: running the update in prod now. | 23:30 |
| Alex_Gaynor | jeblair: it's not my patch unfortunately, it's just blocking mine :-) | 23:30 |
| morganfainberg | sdague, oh, huh, i remember that bug now. | 23:30 |
| *** amcrn has quit IRC | 23:30 | |
| jeblair | Rows matched: 1216479 Changed: 1143230 Warnings: 0 | 23:30 |
| morganfainberg | sdague, i hope it was fixed... but tbh, haven't tried logging >4k in a long time | 23:30 |
| mordred | jeblair: woot | 23:30 |
| sdague | yeh, some of those stack traces.... might be interesting :) | 23:30 |
| morganfainberg | sdague, oh.. hmm. might need to funnel this through rsyslog. | 23:30 |
| morganfainberg | sdague, multiple processes trying to write to the same file = bad. | 23:31 |
| jeblair | bnemec: old gerrit approval timestamps have been restored | 23:31 |
| *** mestery has quit IRC | 23:31 | |
| morganfainberg | sdague, i'll work on some stuff to get this going and consistent one way or another, we can discuss further once i get a proposed fix in gerrit. | 23:31 |
| sdague | morganfainberg: so honestly, if we are going to use apache this way, then probably you just want to change the log format line for 2.2 and drop the date | 23:32 |
| jlk | clarkb: with j-j-b, have you dealt with creating some sort of singular job grouping, that you can use in a publisher trigger, so that you can trigger one thing, rather than 20? | 23:32 |
| sdague | I think making rsyslog a requirement for apache is differently od | 23:32 |
| morganfainberg | sdague, oh the keystone log format? | 23:32 |
| jeblair | Alex_Gaynor: i'm trying one more thing | 23:32 |
| sdague | yeh, then at least you wouldnt' be putting in double dates | 23:32 |
| sdague | in 2.4 you could fix the apache log | 23:33 |
| sdague | but not having subsec resolution is all you give up | 23:33 |
| sdague | anyway, got to run off and make some dinner | 23:33 |
| morganfainberg | sdague, well we have some time to tune this if we don't like how it comes out. i'll propose that type of fix then. initially this will be only a nonvote, and if it needs more work we can fix it :) | 23:33 |
| morganfainberg | sdague, ++ sounds good will post that change to devstack tonight (keystone log change under apache) | 23:34 |
| Alex_Gaynor | jeblair: godspeed, and best of luck :-) | 23:34 |
| *** crc32 has quit IRC | 23:34 | |
| jeblair | Alex_Gaynor: nope that didn't work. i'll restart zuul. | 23:34 |
| Alex_Gaynor | jeblair: funtimes, sorry :-( | 23:34 |
| jeblair | i'm mostly just sorry i don't get to put the logging changes in yet :( | 23:35 |
| *** hogepodge has quit IRC | 23:35 | |
| jlk | clarkb: and is there any way to manage jenkins views from j-j-b? | 23:35 |
| *** andreykurilin_ has quit IRC | 23:35 | |
| *** luisg_ has quit IRC | 23:36 | |
| Alex_Gaynor | jeblair: do we need to "recheck no bug" everything now? | 23:36 |
| jeblair | Alex_Gaynor: yep | 23:37 |
| *** amcrn has joined #openstack-infra | 23:37 | |
| Alex_Gaynor | jeblair: cheers. thanks so much | 23:37 |
| fungi | jlk: jjb doesn't configure jenkins views that i'm aware | 23:37 |
| jlk | that's what I'm seeing. | 23:38 |
| mordred | jeblair: I'm going to submit a change to upstream gerrit to fix the migration | 23:38 |
| jeblair | mordred: wow | 23:38 |
| *** luisg_ has joined #openstack-infra | 23:38 | |
| fungi | jlk: we never really made heavy use of views because, for the most part, it was easier to abstract that out to other interfaces | 23:38 |
| jlk | yeah, that's fair. We're using views now, but mostly because we just don't have anything better in front of us | 23:39 |
| mordred | jeblair: beacuse, you know, it's a data loss thing which is kinda bad | 23:39 |
| jlk | I think we're about 2 steps away from directly editing jenkins job builder yaml though, and going to something which can /generate/ jenkins job builder yaml. Not enough programmatic control to be truly DRY. | 23:39 |
| fungi | jlk: also, we have 1.5k devs interested in data and don't really want them all logging into the jenkins interface... and then there's the fact that we have 8 jenkins masters now, for scalability reasons | 23:40 |
| jlk | hah yes | 23:40 |
| fungi | so views on one master don't really mean much any more | 23:40 |
| fungi | jlk: i'll agree that jjb yaml isn't as dry as it could be. more templating and macro magic in jjb is probably worth investing in | 23:42 |
| jlk | that and using job names generated in one yaml as data in another yaml | 23:42 |
| jlk | e.g. I generate 8 jobs from one yaml, but I want a 9th job to trigger those 8 jobs | 23:42 |
| jlk | so right now I have to enumerate out all 8 jobs in the 9th' yaml | 23:42 |
| *** bknudson has joined #openstack-infra | 23:42 | |
| fungi | jlk: ahh, i think zuul was basically an answer to that (among other things) | 23:43 |
| anteaya | I am off to a motel, since I can get no sleep in my house, I will be online once i return tomorrow I have no estimated time for that | 23:43 |
| anteaya | night | 23:43 |
| fungi | anteaya: i hope you get some rest | 23:43 |
| anteaya | thanks | 23:43 |
| anteaya | I hope so too | 23:43 |
| fungi | jlk: zuul now has templates of its own, so adding a template to a project can basically cause multiple jobs to run in various pipelines for any change to a project now | 23:44 |
| jlk | hrm. | 23:45 |
| jlk | we're not yet into zuul | 23:45 |
| *** bknudson1 has joined #openstack-infra | 23:45 | |
| fungi | jlk: just something to keep in mind for if/when you get there | 23:45 |
| jlk | yeah, thanks | 23:45 |
| mordred | https://gerrit-review.googlesource.com/56750 in case anyone is interested | 23:46 |
| *** bknudson has quit IRC | 23:47 | |
| fungi | mordred: though really, internalizing timestamp management rather than relying on the underlying database to update timestamp fields when you change a row is where sanity can eventually be found | 23:48 |
| mordred | fungi: well, sure | 23:49 |
| mordred | but I'm not going to write that patch | 23:49 |
| fungi | i can understand why ;) | 23:49 |
| *** oomichi has joined #openstack-infra | 23:49 | |
| *** oomichi has quit IRC | 23:53 | |
| *** ramashri has joined #openstack-infra | 23:55 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!