| *** achanda has quit IRC | 00:00 | |
| *** dingyichen has joined #openstack-infra | 00:00 | |
| mmmpork | whoops i lied | 00:00 |
|---|---|---|
| *** gildub has quit IRC | 00:01 | |
| timrc | lol | 00:01 |
| *** achanda_ has quit IRC | 00:01 | |
| nibalizer | haha | 00:01 |
| nibalizer | classic | 00:01 |
| mmmpork | i left a TODO in that i meant to remove | 00:01 |
| *** dims has joined #openstack-infra | 00:03 | |
| mmmpork | should be good now, openstackgerrit just didn't want to tell anyone | 00:03 |
| clarkb | wait for it | 00:03 |
| *** oomichi has joined #openstack-infra | 00:03 | |
| clarkb | sometimes the throttle mechanism has it slow down | 00:03 |
| *** bnemec has quit IRC | 00:03 | |
| *** ybathia has quit IRC | 00:04 | |
| *** sflanigan has joined #openstack-infra | 00:04 | |
| *** sflanigan has joined #openstack-infra | 00:04 | |
| *** ashleighfarnham has quit IRC | 00:04 | |
| *** salv-orlando has quit IRC | 00:04 | |
| *** dims has quit IRC | 00:05 | |
| *** dims has joined #openstack-infra | 00:05 | |
| *** enikanorov has quit IRC | 00:05 | |
| timrc | Can we get this one merged soon? It's not super important to me but it's been stewing for a good while... https://review.openstack.org/186889 | 00:06 |
| *** enikanorov has joined #openstack-infra | 00:07 | |
| clarkb | timrc: see line 131 | 00:07 |
| clarkb | it most definitely can be run directly... | 00:07 |
| pabelanger | mriedem: yes, ianw and I are working on fedora22 now. Having some dnf cache issues, if you haven't seen. diskimage-builder should work however | 00:07 |
| clarkb | anyways maybe jhesketh wants to approve as he already +2'd | 00:08 |
| *** tsekiyam_ has joined #openstack-infra | 00:09 | |
| *** dims has quit IRC | 00:10 | |
| *** thorst1 has quit IRC | 00:10 | |
| timrc | clarkb, Doh. | 00:10 |
| openstackgerrit | Merged openstack-infra/puppet-iptables: Disable firewalld on centos7 and greater https://review.openstack.org/217928 | 00:10 |
| mriedem | pabelanger: ok, i'm interested in fc23 since it has a new enough libvirt to test some new features added to nova in liberty | 00:11 |
| mriedem | pabelanger: i'll just wait until november or something to check on an fc23 job | 00:11 |
| *** mtanino has quit IRC | 00:12 | |
| pabelanger | mriedem: right | 00:12 |
| *** tsekiyama has quit IRC | 00:13 | |
| *** tsekiyam_ has quit IRC | 00:13 | |
| pabelanger | It would be cool if we could some how create a rawhide dib to use for an experimental perodic job or something | 00:13 |
| mriedem | pabelanger: are the problems you're running into just wrt to the dib tests? | 00:13 |
| clarkb | pabelanger: tumbleweed would be my preference | 00:14 |
| mriedem | i figured things would be fine if we just had a fc23 image | 00:14 |
| pabelanger | no, dibs works last I checked | 00:14 |
| pabelanger | for fedora22 | 00:14 |
| *** amrith is now known as _amrith_ | 00:14 | |
| clarkb | pabelanger: if thats the sort of testing you want to do since that upstream does CI before updating their repos | 00:14 |
| pabelanger | we are just having some caching issues for devstack packages | 00:14 |
| clarkb | (so in theory we would spend less time debugging the distro for them) | 00:14 |
| pabelanger | clarkb: never used tumbleweed, should give it a try | 00:15 |
| clarkb | but I imagine that any jobs we run on that sort of distro will end up with lots of debug distro time | 00:15 |
| mriedem | heh, today is the first time i've heard of dnf | 00:16 |
| clarkb | mriedem: back away slowly | 00:16 |
| pabelanger | mriedem: well, we have some -infra issues with our puppet manifests and custom DIB elements right now. with fedora 22. That is the blocker for getting them online right now | 00:16 |
| clarkb | also dnf uses zypper's dep solver lib but zypper will cache rpms for you | 00:16 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config: (WIP) Add compute class for Infra Cloud https://review.openstack.org/213980 | 00:16 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config: (WIP) Add controller class for Infra Cloud https://review.openstack.org/209698 | 00:16 |
| pabelanger | puppet is working, dnf is working. Just some bindep stuff left right now | 00:16 |
| *** bhunter71 has joined #openstack-infra | 00:16 | |
| *** dims has joined #openstack-infra | 00:17 | |
| mriedem | dnf is a replacement for yum or what? | 00:17 |
| clarkb | pabelanger: no dnf doesn't work, it explicitly does not support the feature we want | 00:17 |
| clarkb | mriedem: yes but it doesn't support package caches | 00:17 |
| mriedem | do we need to use dnf with devstack and fc23? | 00:17 |
| clarkb | (and every other package manager ever does this even pip) | 00:17 |
| mriedem | or fc22 | 00:17 |
| clarkb | mriedem: I think yum still works | 00:17 |
| mriedem | so why not use yum? | 00:17 |
| clarkb | I think the idea is to use the new shiny? I dunno | 00:18 |
| mriedem | i see http://www.linux-magazine.com/Online/Features/Will-DNF-Replace-Yum | 00:18 |
| mriedem | "and is tentatively scheduled to replace Fedora’s package manager Yum entirely by the end of 2014." | 00:18 |
| mriedem | orly | 00:18 |
| mriedem | so yum is deprecated with fedora? | 00:18 |
| pabelanger | clarkb: right, dnf download is not feature compat. So, either we write a plugin to fix, fight dnf team to add support, maybe revert to yum. *shurgs* | 00:18 |
| pabelanger | either way, it sucks | 00:19 |
| pabelanger | mriedem: Yum, dnf by default now | 00:19 |
| pabelanger | haven't tested yum on fedora 22 as of yet | 00:20 |
| clarkb | greghaynes: ubuntu-trusty with growroot fix is uploading now | 00:21 |
| clarkb | so ya 6 7 ish is probably still correct | 00:22 |
| greghaynes | awesome | 00:22 |
| *** btully has joined #openstack-infra | 00:22 | |
| *** gildub has joined #openstack-infra | 00:25 | |
| *** pvaneck_ has quit IRC | 00:25 | |
| *** jeadre has quit IRC | 00:25 | |
| *** jeadre has joined #openstack-infra | 00:26 | |
| openstackgerrit | Merged openstack-infra/grafyaml: Use safe_load when parsing yaml https://review.openstack.org/213361 | 00:26 |
| *** asettle has joined #openstack-infra | 00:32 | |
| *** asettle has quit IRC | 00:33 | |
| *** asettle has joined #openstack-infra | 00:33 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Experimental multinode job for puppet-openstack-integration https://review.openstack.org/215696 | 00:36 |
| *** sridhar_ram has joined #openstack-infra | 00:38 | |
| nibalizer | fungi: ah darn I went off list on the puppet-httpd thread, how much of that should we copy in to the main ML? | 00:39 |
| *** markvoelker has joined #openstack-infra | 00:39 | |
| *** tqtran_ has quit IRC | 00:41 | |
| *** rlandy has quit IRC | 00:41 | |
| *** hdd has quit IRC | 00:42 | |
| *** rlandy has joined #openstack-infra | 00:42 | |
| mordred | mriedem, clarkb, pabelanger: I would say this is yet another instance of people just changing _BASIC_ and fundamentla parts of linux without being feature complete | 00:42 |
| mordred | it seems there are a bunch of people working in the linux space who actually just hate linux | 00:43 |
| fungi | nibalizer: feel free to quote it all back onto the ml, or we can just start over ;) | 00:43 |
| *** esker has joined #openstack-infra | 00:43 | |
| nibalizer | hehe | 00:43 |
| *** FallenPegasus has quit IRC | 00:44 | |
| pabelanger | mordred: agreed | 00:44 |
| mriedem | mordred: yeah there is some grumbling in the comments like that here http://www.linux-magazine.com/Online/Features/Will-DNF-Replace-Yum | 00:45 |
| fungi | mordred: i saw beos has a new forkish thing gaining steam. we could just exile them all there instead | 00:45 |
| clarkb | fungi: no! tumbleweed | 00:45 |
| clarkb | when AJaeger comes back from vacation we can surprise him by runing suse for all the things | 00:45 |
| pleia2 | hehe | 00:45 |
| *** yuanying-alt has quit IRC | 00:45 | |
| fungi | haiku is the one i was remembering | 00:45 |
| jesusaurus | i have weird termcap issues with tumbleweed | 00:45 |
| *** bknudson has joined #openstack-infra | 00:46 | |
| fungi | https://www.haiku-os.org/ | 00:46 |
| *** akraminsky has quit IRC | 00:46 | |
| *** tjones has left #openstack-infra | 00:46 | |
| jesusaurus | oh cool, haiku is still a thing | 00:47 |
| fungi | clarkb: no, i was saying send all the problem people to haiku so we can take gnu/linux back | 00:47 |
| clarkb | fungi: I see | 00:47 |
| jesusaurus | and still alpha, too! | 00:47 |
| *** rlandy has quit IRC | 00:47 | |
| clarkb | mordred: I got a huge dose of systemd today | 00:47 |
| jesusaurus | fungi: oh thats a much better idea | 00:47 |
| clarkb | mordred: added our first unit files to puppet | 00:47 |
| openstackgerrit | Sahdev Zala proposed openstack/requirements: Add tosca-parser to global requirements https://review.openstack.org/217991 | 00:47 |
| fungi | jeblair: maybe some day it will graduate backward to pre-alpha, and then pre-pre-alpha | 00:47 |
| fungi | er, jesusaurus ^ | 00:48 |
| fungi | (sorry jeblair!) | 00:48 |
| mordred | clarkb: oh. urgh | 00:48 |
| *** yuanying-alt has joined #openstack-infra | 00:48 | |
| clarkb | mordred: but it really is/was the best way to run git-daemon to not fight the system | 00:48 |
| clarkb | mordred: othrewise you have selinux problems | 00:48 |
| clarkb | mordred: and more systemd issues because systemd running sysv scripts didn't really work | 00:48 |
| mordred | clarkb: I mean, honestely, if the system is a systemd system, we should use systemd | 00:48 |
| fungi | i wrote a systemd unit file for ezstream not long ago. tried to shower it off, but they don't make water hot enough | 00:49 |
| clarkb | mordred: right | 00:49 |
| mordred | as much as it is a) clunky b) gross c) difficult d) unenjoyable e) stupid | 00:49 |
| clarkb | fungi: did you see the IPs I listed for the new git08 host? | 00:49 |
| clarkb | fungi: its there for testing though only the nova repo has content | 00:49 |
| fungi | clarkb: no, i skimmed scrollback but missed. i'll look closer, thanks | 00:49 |
| clarkb | and my testing of shallow clones worked | 00:49 |
| fungi | clarkb: great! any clue how/when that was fixed? | 00:50 |
| mordred | it's the New World Order imposed on us by our new Lennart overlords and far be it for us, mere users, to have an opinion in the shadow of his magnetic greatness | 00:50 |
| fungi | mordred: i like to refer to it as "lennux" now | 00:50 |
| clarkb | fungi: nope, though wouldn't be surprised if centos7 git is too oldto have the issue | 00:50 |
| fungi | ahh, that seems likely in fact | 00:50 |
| fungi | so... lucky us? | 00:50 |
| clarkb | though its 1.9.3 on centos7 and 1.9.1 on trusty so maybe I am wrong | 00:50 |
| clarkb | in any case it seems to work after battling apache, iptables/firewalld, selinux, and git-daemon | 00:51 |
| fungi | clarkb: your name will be forever recorded upon the wall of heroes | 00:51 |
| *** yuanying-alt has quit IRC | 00:52 | |
| dstufft | clarkb: I like how "even pip" is used as a "even pip does this thing wtf are you doing" ;) | 00:54 |
| clarkb | dstufft: :) | 00:54 |
| *** ramashri has quit IRC | 00:56 | |
| clarkb | greghaynes: 1/8 image uploads done | 00:56 |
| clarkb | greghaynes: this is slow | 00:57 |
| *** zhenguo has joined #openstack-infra | 00:57 | |
| *** jeadre has quit IRC | 00:57 | |
| *** jeadre has joined #openstack-infra | 00:58 | |
| *** dannywilson has quit IRC | 01:03 | |
| *** sergmelikyan has quit IRC | 01:03 | |
| *** hdd has joined #openstack-infra | 01:04 | |
| *** alexsyip has quit IRC | 01:05 | |
| *** sputnik13 has quit IRC | 01:06 | |
| *** juggler_ has joined #openstack-infra | 01:07 | |
| openstackgerrit | Colleen Murphy proposed openstack/diskimage-builder: Force install grub packages https://review.openstack.org/218007 | 01:07 |
| openstackgerrit | Merged openstack/requirements: Cap alembic version to < 0.8.1 https://review.openstack.org/216328 | 01:07 |
| *** juggler_ is now known as juggler | 01:08 | |
| ianw | greghaynes: can i just revive https://review.openstack.org/#/c/193051/ (doc: migrate from README.rst to Sphinx) and can i just add documentation to doc/* ? it seems to make sense to just have README as something very basic and fill out the documentation | 01:12 |
| ianw | merge failures on text is just the worst | 01:13 |
| *** sputnik13 has joined #openstack-infra | 01:14 | |
| *** thorst1 has joined #openstack-infra | 01:15 | |
| *** tlian2 has quit IRC | 01:18 | |
| *** jeadre has quit IRC | 01:18 | |
| *** sputnik13 has quit IRC | 01:18 | |
| *** jeadre has joined #openstack-infra | 01:19 | |
| *** bharathm has quit IRC | 01:19 | |
| *** spzala has quit IRC | 01:21 | |
| *** thorst1 has quit IRC | 01:23 | |
| *** tlian has joined #openstack-infra | 01:25 | |
| *** sridhar_ram has quit IRC | 01:26 | |
| *** darrenc is now known as darrenc_afk | 01:31 | |
| *** FallenPegasus has joined #openstack-infra | 01:35 | |
| *** signed8bit has quit IRC | 01:35 | |
| *** yfried_ has quit IRC | 01:36 | |
| *** woodster_ has quit IRC | 01:39 | |
| *** darrenc_afk is now known as darrenc | 01:47 | |
| *** yuanying-alt has joined #openstack-infra | 01:49 | |
| *** tiswanso has joined #openstack-infra | 01:50 | |
| *** spzala has joined #openstack-infra | 01:52 | |
| *** stevemar has joined #openstack-infra | 01:53 | |
| *** yuanying-alt has quit IRC | 01:53 | |
| *** madhu_ak has quit IRC | 01:53 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: doc: migrate from README.rst to Sphinx https://review.openstack.org/193051 | 01:53 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Make README.rst a bit more generic https://review.openstack.org/218043 | 01:53 |
| *** tiswanso_ has joined #openstack-infra | 01:55 | |
| *** tjones has joined #openstack-infra | 01:56 | |
| *** stevemar has quit IRC | 01:57 | |
| *** tiswanso has quit IRC | 01:57 | |
| clarkb | greghaynes: the roots look grown and the jobs are running | 02:03 |
| *** yamamoto has joined #openstack-infra | 02:03 | |
| *** bdemers has quit IRC | 02:04 | |
| *** sdake_ is now known as sdake | 02:04 | |
| nibalizer | they grow up so fast *sniff* | 02:04 |
| *** bdemers has joined #openstack-infra | 02:05 | |
| *** stevemar has joined #openstack-infra | 02:05 | |
| *** dims has quit IRC | 02:06 | |
| *** JoshNang has quit IRC | 02:06 | |
| *** stevemar has quit IRC | 02:07 | |
| *** bdemers has quit IRC | 02:07 | |
| *** bdemers has joined #openstack-infra | 02:08 | |
| *** shardy_ has joined #openstack-infra | 02:08 | |
| *** shardy has quit IRC | 02:09 | |
| *** JoshNang has joined #openstack-infra | 02:09 | |
| *** boris-42 has quit IRC | 02:10 | |
| *** unicell1 has quit IRC | 02:13 | |
| *** asettle is now known as asettle-afk | 02:13 | |
| *** shardy_ has quit IRC | 02:13 | |
| *** asettle-afk has quit IRC | 02:13 | |
| *** shardy has joined #openstack-infra | 02:14 | |
| *** yamamoto has quit IRC | 02:17 | |
| *** yamahata has quit IRC | 02:19 | |
| *** hdd has quit IRC | 02:19 | |
| *** raginbajin has quit IRC | 02:23 | |
| *** tjones has left #openstack-infra | 02:23 | |
| *** angerson has quit IRC | 02:26 | |
| *** raginbajin has joined #openstack-infra | 02:28 | |
| *** achanda has joined #openstack-infra | 02:28 | |
| *** angerson has joined #openstack-infra | 02:29 | |
| *** purp has quit IRC | 02:33 | |
| *** dims has joined #openstack-infra | 02:33 | |
| *** coreycb has quit IRC | 02:33 | |
| *** chen12 has left #openstack-infra | 02:33 | |
| *** zxiiro has quit IRC | 02:35 | |
| *** mrodden has quit IRC | 02:35 | |
| *** legoktm has quit IRC | 02:35 | |
| *** ctlaugh has quit IRC | 02:35 | |
| *** coreycb has joined #openstack-infra | 02:36 | |
| *** purp has joined #openstack-infra | 02:36 | |
| *** sthillma has quit IRC | 02:38 | |
| *** legoktm has joined #openstack-infra | 02:38 | |
| *** mrodden has joined #openstack-infra | 02:38 | |
| *** legoktm is now known as Guest99591 | 02:38 | |
| *** ctlaugh has joined #openstack-infra | 02:39 | |
| *** FallenPegasus has quit IRC | 02:39 | |
| *** larainema has joined #openstack-infra | 02:40 | |
| openstackgerrit | Sahdev Zala proposed openstack/requirements: Add tosca-parser to global requirements https://review.openstack.org/217991 | 02:44 |
| openstackgerrit | Merged openstack/diskimage-builder: Fix a typo in README.rst https://review.openstack.org/216147 | 02:44 |
| *** bdemers has quit IRC | 02:46 | |
| *** bdemers has joined #openstack-infra | 02:47 | |
| *** zxiiro has joined #openstack-infra | 02:47 | |
| cody-somerville | Is there a wikipage on Multi-node Openstack Testing? | 02:49 |
| openstackgerrit | Yushiro FURUKAWA proposed openstack-infra/project-config: Add python-jobs into networking-fujitsu https://review.openstack.org/218067 | 02:50 |
| pabelanger | cody-somerville: not that I am aware of. Mind you, I haven't really looked. I've been just reading nodepool and JJB right now | 02:51 |
| *** spzala has quit IRC | 02:51 | |
| *** gyee has quit IRC | 02:51 | |
| *** yamamoto has joined #openstack-infra | 02:56 | |
| openstackgerrit | Richard Jones proposed openstack-infra/project-config: Promote horizon selenium test job https://review.openstack.org/218068 | 02:56 |
| clarkb | the nodepool docs cover the generic setup, d-g implements setup for devstack in that env | 02:56 |
| *** jeadre has quit IRC | 02:57 | |
| *** jeadre has joined #openstack-infra | 02:58 | |
| *** baoli has joined #openstack-infra | 03:01 | |
| TheJulia | greg, that sounds like a bug... a nasty bug at that. is there a bug report filed that i can look at tomorrow? | 03:02 |
| *** tlian has quit IRC | 03:04 | |
| *** mriedem has quit IRC | 03:06 | |
| *** asettle has joined #openstack-infra | 03:07 | |
| *** jerryz has quit IRC | 03:07 | |
| *** nelsnels_ has quit IRC | 03:17 | |
| *** nelsnelson has joined #openstack-infra | 03:18 | |
| *** jeadre has quit IRC | 03:20 | |
| *** jeadre has joined #openstack-infra | 03:20 | |
| *** madong_ has joined #openstack-infra | 03:21 | |
| *** larainema has quit IRC | 03:21 | |
| *** dims has quit IRC | 03:23 | |
| *** ams__ has quit IRC | 03:26 | |
| *** vaidy has quit IRC | 03:27 | |
| *** isviridov_away has quit IRC | 03:27 | |
| *** rm_work has quit IRC | 03:27 | |
| *** vaidy has joined #openstack-infra | 03:27 | |
| *** isviridov_away has joined #openstack-infra | 03:27 | |
| *** dtroyer has quit IRC | 03:27 | |
| *** sigmavirus24_awa has quit IRC | 03:27 | |
| *** rainya has quit IRC | 03:27 | |
| *** hockeynut has quit IRC | 03:27 | |
| *** tbarron has quit IRC | 03:28 | |
| *** comstud has quit IRC | 03:28 | |
| *** matt____ has joined #openstack-infra | 03:28 | |
| *** othermordred has quit IRC | 03:28 | |
| *** fungi has quit IRC | 03:28 | |
| *** jroll has quit IRC | 03:28 | |
| *** zz_dimtruck has quit IRC | 03:28 | |
| *** gondoi has quit IRC | 03:28 | |
| *** lsell has quit IRC | 03:28 | |
| *** mattoliverau has quit IRC | 03:28 | |
| *** mgagne has quit IRC | 03:28 | |
| *** jbryce has quit IRC | 03:28 | |
| *** darrenc_ has joined #openstack-infra | 03:28 | |
| *** d34dh0r53 has quit IRC | 03:28 | |
| *** asahlin has quit IRC | 03:28 | |
| *** Trozz_ has quit IRC | 03:28 | |
| *** lbragstad has quit IRC | 03:28 | |
| *** wendar has quit IRC | 03:28 | |
| *** darrenc has quit IRC | 03:28 | |
| *** freyes has quit IRC | 03:28 | |
| *** gus has quit IRC | 03:28 | |
| *** Nakato has quit IRC | 03:28 | |
| *** JayF has quit IRC | 03:28 | |
| *** wendar_ has joined #openstack-infra | 03:28 | |
| *** eglute has quit IRC | 03:28 | |
| *** reed has quit IRC | 03:29 | |
| *** apmelton has quit IRC | 03:29 | |
| *** othermordred has joined #openstack-infra | 03:29 | |
| *** salv-orlando has joined #openstack-infra | 03:30 | |
| *** jeadre has quit IRC | 03:30 | |
| *** reed has joined #openstack-infra | 03:31 | |
| *** matt____ is now known as mattoliverau | 03:31 | |
| *** Nakato has joined #openstack-infra | 03:31 | |
| *** eglute has joined #openstack-infra | 03:31 | |
| *** gus has joined #openstack-infra | 03:31 | |
| *** jeadre has joined #openstack-infra | 03:31 | |
| *** rm_work has joined #openstack-infra | 03:31 | |
| *** asahlin has joined #openstack-infra | 03:31 | |
| *** apmelton has joined #openstack-infra | 03:32 | |
| *** ams__ has joined #openstack-infra | 03:32 | |
| *** d34dh0r53 has joined #openstack-infra | 03:32 | |
| *** jroll has joined #openstack-infra | 03:32 | |
| *** tbarron has joined #openstack-infra | 03:32 | |
| *** zz_dimtruck has joined #openstack-infra | 03:32 | |
| *** fungi has joined #openstack-infra | 03:32 | |
| *** zz_dimtruck is now known as dimtruck | 03:32 | |
| *** Trozz_ has joined #openstack-infra | 03:32 | |
| *** freyes has joined #openstack-infra | 03:33 | |
| *** sigmavirus24_awa has joined #openstack-infra | 03:33 | |
| *** fawadkhaliq has joined #openstack-infra | 03:34 | |
| *** rainya has joined #openstack-infra | 03:34 | |
| *** mgagne has joined #openstack-infra | 03:34 | |
| *** salv-orlando has quit IRC | 03:36 | |
| *** hockeynut has joined #openstack-infra | 03:36 | |
| *** dtroyer has joined #openstack-infra | 03:37 | |
| *** comstud has joined #openstack-infra | 03:37 | |
| *** JayF has joined #openstack-infra | 03:37 | |
| *** lbragstad has joined #openstack-infra | 03:37 | |
| *** jbryce has joined #openstack-infra | 03:37 | |
| *** zz_gondoi has joined #openstack-infra | 03:37 | |
| *** lsell has joined #openstack-infra | 03:37 | |
| *** zz_gondoi is now known as gondoi | 03:37 | |
| *** yuanying-alt has joined #openstack-infra | 03:37 | |
| *** sergmelikyan has joined #openstack-infra | 03:39 | |
| *** harlowja_at_home has joined #openstack-infra | 03:40 | |
| *** achanda has quit IRC | 03:41 | |
| *** yuanying-alt has quit IRC | 03:42 | |
| *** sdake_ has joined #openstack-infra | 03:44 | |
| *** tiswanso_ has quit IRC | 03:45 | |
| *** sdake has quit IRC | 03:48 | |
| cody-somerville | clarkb: How does the primary node end up knowing its multi-node and what the other nodes are? | 03:50 |
| cody-somerville | ah, maybe different ready script? | 03:51 |
| *** sdake has joined #openstack-infra | 03:57 | |
| *** baoli has quit IRC | 03:57 | |
| *** doug-fish has joined #openstack-infra | 03:58 | |
| clarkb | http://docs.openstack.org/infra/nodepool/scripts.html#ready-script | 04:01 |
| *** harlowja_at_home has quit IRC | 04:01 | |
| *** sdake_ has quit IRC | 04:01 | |
| *** doug-fish has quit IRC | 04:03 | |
| *** jeadre has quit IRC | 04:03 | |
| *** jeadre has joined #openstack-infra | 04:04 | |
| *** unicell has joined #openstack-infra | 04:04 | |
| *** jeadre has quit IRC | 04:06 | |
| *** jeadre has joined #openstack-infra | 04:07 | |
| *** ajmiller_ has joined #openstack-infra | 04:09 | |
| *** armax has quit IRC | 04:09 | |
| *** sergmelikyan has quit IRC | 04:09 | |
| *** links has joined #openstack-infra | 04:09 | |
| *** tjones has joined #openstack-infra | 04:10 | |
| *** andersonvom has joined #openstack-infra | 04:13 | |
| *** sdake_ has joined #openstack-infra | 04:14 | |
| *** armax has joined #openstack-infra | 04:16 | |
| openstackgerrit | Richard Jones proposed openstack-infra/project-config: Promote horizon selenium test job https://review.openstack.org/218068 | 04:16 |
| *** sdake has quit IRC | 04:18 | |
| *** bharathm has joined #openstack-infra | 04:20 | |
| *** sdake_ is now known as sdake | 04:22 | |
| *** darrenc_ is now known as darrenc | 04:22 | |
| *** gildub has quit IRC | 04:24 | |
| *** bharathm has quit IRC | 04:24 | |
| *** sdake_ has joined #openstack-infra | 04:25 | |
| *** _amrith_ is now known as amrith | 04:25 | |
| *** tjones has left #openstack-infra | 04:26 | |
| *** bmjen has joined #openstack-infra | 04:28 | |
| *** sdake has quit IRC | 04:29 | |
| *** hurgleburgler has joined #openstack-infra | 04:30 | |
| bmjen | Hello all. I'm having an issue getting my environment set up to push a bugfix to gerrit. I've accepted the ICLA agreement, however `git review` says I need to update my contact information. The review.openstack.org website is throwing an error when I attempt to update contact info. Can anyone help? | 04:30 |
| *** frickler has quit IRC | 04:30 | |
| *** Mmike has quit IRC | 04:31 | |
| *** amotoki has joined #openstack-infra | 04:33 | |
| *** fitoduarte has joined #openstack-infra | 04:33 | |
| *** sthillma has joined #openstack-infra | 04:35 | |
| *** Guest99591 has quit IRC | 04:35 | |
| *** Guest99591 has joined #openstack-infra | 04:35 | |
| *** Guest99591 is now known as legoktm | 04:35 | |
| *** sthillma_ has joined #openstack-infra | 04:36 | |
| *** puranamr has joined #openstack-infra | 04:37 | |
| crinkle | bmjen: does this help? https://ask.openstack.org/en/question/56720/cannot-store-contact-information-when-updating-info-in-openstack-gerrit/ | 04:38 |
| bmjen | o/ crinkle! | 04:38 |
| bmjen | familiar face | 04:38 |
| crinkle | hi bmjen \o | 04:38 |
| *** Mmike has joined #openstack-infra | 04:39 | |
| *** sthillma has quit IRC | 04:40 | |
| *** sthillma_ is now known as sthillma | 04:40 | |
| *** BharatK has joined #openstack-infra | 04:42 | |
| *** puranamr has quit IRC | 04:44 | |
| *** madong_ has quit IRC | 04:44 | |
| openstackgerrit | Steve Kowalik proposed openstack-infra/project-config: Do not ignore the exit status creating Zanata XML https://review.openstack.org/218087 | 04:46 |
| bmjen | that was it. Thanks crinkle++ | 04:46 |
| crinkle | \o/ | 04:50 |
| *** ajmiller_ has quit IRC | 04:50 | |
| *** FallenPegasus has joined #openstack-infra | 04:51 | |
| *** fedexo has joined #openstack-infra | 04:51 | |
| *** oomichi has quit IRC | 04:53 | |
| *** deepakcs has joined #openstack-infra | 04:56 | |
| *** larainema has joined #openstack-infra | 05:02 | |
| *** hdd has joined #openstack-infra | 05:02 | |
| nibalizer | bmjen: o/ | 05:02 |
| openstackgerrit | Motohiro/Yuanying Otsuka proposed openstack-infra/project-config: Enable Barbican in Magnum job definition https://review.openstack.org/216933 | 05:04 |
| *** hurgleburgler has quit IRC | 05:05 | |
| *** bhunter71 has quit IRC | 05:05 | |
| *** rossella_s has quit IRC | 05:05 | |
| *** rossella_s has joined #openstack-infra | 05:06 | |
| greghaynes | clarkb: so, success on growroot? | 05:07 |
| greghaynes | TheJulia: no bug yet, still digging but good to know its not expected behavior | 05:10 |
| *** FallenPegasus has quit IRC | 05:14 | |
| *** ildikov has joined #openstack-infra | 05:19 | |
| *** jeadre has quit IRC | 05:20 | |
| *** jeadre has joined #openstack-infra | 05:21 | |
| *** nelsnelson has quit IRC | 05:25 | |
| *** flepied1 has quit IRC | 05:26 | |
| *** yuanying-alt has joined #openstack-infra | 05:26 | |
| openstackgerrit | Naohiro Tamura proposed openstack-infra/project-config: Add openstack/python-scciclient project https://review.openstack.org/218097 | 05:26 |
| *** doug-fish has joined #openstack-infra | 05:27 | |
| *** yuanying-alt has quit IRC | 05:31 | |
| *** salv-orlando has joined #openstack-infra | 05:31 | |
| *** doug-fish has quit IRC | 05:32 | |
| *** salv-orlando has quit IRC | 05:36 | |
| *** yamamoto has quit IRC | 05:40 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Make README.rst a bit more generic https://review.openstack.org/218043 | 05:40 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: doc: migrate from README.rst to Sphinx https://review.openstack.org/193051 | 05:40 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Add DIB_KEEP_PACKAGE_CACHE variable https://review.openstack.org/208321 | 05:40 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Add DIB_USE_PACKAGE_CACHE variable https://review.openstack.org/211434 | 05:40 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Remove grub2 in redhat-common/pre-install.d/15-remove-grub https://review.openstack.org/211124 | 05:40 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Add #!/bin/bash to library functions https://review.openstack.org/218100 | 05:40 |
| *** bdemers has quit IRC | 05:44 | |
| *** bdemers has joined #openstack-infra | 05:45 | |
| *** skylerberg has quit IRC | 05:47 | |
| *** yamamoto has joined #openstack-infra | 05:48 | |
| *** BharatK has quit IRC | 05:51 | |
| *** yamahata has joined #openstack-infra | 05:52 | |
| *** armax has quit IRC | 05:56 | |
| *** bdemers has quit IRC | 05:56 | |
| *** bdemers has joined #openstack-infra | 05:57 | |
| *** mrmartin has joined #openstack-infra | 05:58 | |
| *** hdd has quit IRC | 05:59 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack-infra/project-config: Normalize projects.yaml https://review.openstack.org/218107 | 06:01 |
| *** sdake_ is now known as sdake | 06:03 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/requirements: Updated from generate-constraints https://review.openstack.org/216992 | 06:06 |
| *** BharatK has joined #openstack-infra | 06:06 | |
| *** asettle has quit IRC | 06:07 | |
| *** liam__ has joined #openstack-infra | 06:12 | |
| openstackgerrit | Huang Rui proposed openstack-infra/project-config: Add python-jobs for ceilometer-zvm https://review.openstack.org/218113 | 06:13 |
| *** puranamr has joined #openstack-infra | 06:15 | |
| *** salv-orlando has joined #openstack-infra | 06:18 | |
| *** flepied has joined #openstack-infra | 06:18 | |
| *** flepied1 has joined #openstack-infra | 06:19 | |
| *** larainema has quit IRC | 06:19 | |
| *** Sree has joined #openstack-infra | 06:21 | |
| *** flepied has quit IRC | 06:23 | |
| *** larainema has joined #openstack-infra | 06:23 | |
| *** dizquierdo has joined #openstack-infra | 06:24 | |
| *** bdemers has quit IRC | 06:25 | |
| *** puranamr has quit IRC | 06:28 | |
| *** fawadkhaliq has quit IRC | 06:29 | |
| *** gildub has joined #openstack-infra | 06:31 | |
| *** ryanpetrello has quit IRC | 06:32 | |
| *** clayton has quit IRC | 06:33 | |
| *** clayton has joined #openstack-infra | 06:33 | |
| *** sthillma has quit IRC | 06:35 | |
| *** madong_ has joined #openstack-infra | 06:37 | |
| *** camunoz has quit IRC | 06:39 | |
| *** larainema has quit IRC | 06:40 | |
| *** ryanpetrello has joined #openstack-infra | 06:40 | |
| *** liam__ has quit IRC | 06:44 | |
| openstackgerrit | Merged openstack-infra/project-config: Normalize projects.yaml https://review.openstack.org/218107 | 06:45 |
| *** vivekd has joined #openstack-infra | 06:48 | |
| *** mrunge has joined #openstack-infra | 06:53 | |
| Kennan | hi EmilienM: there ? | 06:53 |
| *** vikram has joined #openstack-infra | 06:53 | |
| vikram | Can somenone please help me to understand the release process for big-tent project | 06:53 |
| openstackgerrit | Steven Dake proposed openstack-infra/project-config: Kolla: Add build gate for ubuntu source https://review.openstack.org/213531 | 06:54 |
| *** dingyichen has quit IRC | 06:54 | |
| vikram | I have created a big-tent project in neutron "networking-onos".. Want to understand how the release of this repo will be available to the end-user | 06:54 |
| *** devvesa has joined #openstack-infra | 06:55 | |
| *** peristeri has joined #openstack-infra | 06:55 | |
| *** dtantsur|afk is now known as dtantsur | 06:58 | |
| *** dingyichen has joined #openstack-infra | 06:58 | |
| *** ifarkas has joined #openstack-infra | 06:59 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/project-config: Make gate-infra-ansible-ansible-lint non-voting https://review.openstack.org/218121 | 07:01 |
| *** achanda has joined #openstack-infra | 07:05 | |
| *** luqas has joined #openstack-infra | 07:07 | |
| *** dizquierdo has quit IRC | 07:13 | |
| *** HeOS has joined #openstack-infra | 07:13 | |
| *** [HeOS] has joined #openstack-infra | 07:14 | |
| *** yuanying-alt has joined #openstack-infra | 07:15 | |
| *** HeOS has quit IRC | 07:18 | |
| *** fawadkhaliq has joined #openstack-infra | 07:18 | |
| *** mpaolino has joined #openstack-infra | 07:19 | |
| *** bharathm has joined #openstack-infra | 07:19 | |
| *** yuanying-alt has quit IRC | 07:19 | |
| *** jeadre has quit IRC | 07:21 | |
| *** jeadre has joined #openstack-infra | 07:22 | |
| *** jerryz has joined #openstack-infra | 07:23 | |
| *** b10n1k_ has quit IRC | 07:29 | |
| *** cdent has joined #openstack-infra | 07:30 | |
| *** sflanigan has quit IRC | 07:31 | |
| *** frickler has joined #openstack-infra | 07:32 | |
| *** k4n0 has joined #openstack-infra | 07:34 | |
| *** Ala has joined #openstack-infra | 07:40 | |
| openstackgerrit | Kai Qiang Wu(Kennan) proposed openstack-infra/project-config: Add docs and publish jobs for kolla https://review.openstack.org/218137 | 07:43 |
| *** mpaolino has quit IRC | 07:48 | |
| *** arxcruz has joined #openstack-infra | 07:51 | |
| *** doude has joined #openstack-infra | 07:52 | |
| *** fawadk has joined #openstack-infra | 07:53 | |
| *** fawadkhaliq has quit IRC | 07:53 | |
| *** achanda has quit IRC | 07:56 | |
| *** jeadre has quit IRC | 08:04 | |
| *** leopoldj has joined #openstack-infra | 08:04 | |
| *** jeadre has joined #openstack-infra | 08:05 | |
| *** nfedotov has joined #openstack-infra | 08:05 | |
| *** hashar has joined #openstack-infra | 08:05 | |
| *** fhubik has joined #openstack-infra | 08:08 | |
| *** vikram has quit IRC | 08:09 | |
| *** derekh has joined #openstack-infra | 08:10 | |
| *** jistr has joined #openstack-infra | 08:12 | |
| openstackgerrit | Kai Qiang Wu(Kennan) proposed openstack/requirements: Add kolla to requirements family https://review.openstack.org/218144 | 08:14 |
| *** sdake_ has joined #openstack-infra | 08:14 | |
| *** Sree has quit IRC | 08:15 | |
| *** yamahata has quit IRC | 08:17 | |
| *** sdake has quit IRC | 08:18 | |
| *** vikram has joined #openstack-infra | 08:22 | |
| *** fedexo has quit IRC | 08:26 | |
| *** Sree has joined #openstack-infra | 08:27 | |
| *** fawadk has quit IRC | 08:30 | |
| *** ihrachys has joined #openstack-infra | 08:33 | |
| *** vikram has quit IRC | 08:37 | |
| *** dbite has quit IRC | 08:41 | |
| *** dguitarbite has joined #openstack-infra | 08:41 | |
| *** HeOS has joined #openstack-infra | 08:43 | |
| *** [HeOS] has quit IRC | 08:45 | |
| *** bexelbie has joined #openstack-infra | 08:46 | |
| openstackgerrit | Mateusz Matuszkowiak proposed openstack-infra/project-config: Added new SF repository for Fuel wstunnel plugin integration https://review.openstack.org/218154 | 08:47 |
| *** ihrachyshka has joined #openstack-infra | 08:49 | |
| *** amit213 has quit IRC | 08:50 | |
| *** amit213 has joined #openstack-infra | 08:51 | |
| *** HeOS has quit IRC | 08:51 | |
| *** vikram has joined #openstack-infra | 08:51 | |
| *** sshnaidm has joined #openstack-infra | 08:53 | |
| *** e0ne has joined #openstack-infra | 08:54 | |
| *** hashar has quit IRC | 08:57 | |
| *** fawadkhaliq has joined #openstack-infra | 08:58 | |
| *** marzif has joined #openstack-infra | 08:59 | |
| *** ilyashakhat has quit IRC | 08:59 | |
| *** skraynev has quit IRC | 09:00 | |
| *** fawadk has joined #openstack-infra | 09:00 | |
| *** hashar has joined #openstack-infra | 09:00 | |
| *** skolekonov has quit IRC | 09:01 | |
| *** dizquierdo has joined #openstack-infra | 09:01 | |
| *** fawadkhaliq has quit IRC | 09:02 | |
| *** e0ne has quit IRC | 09:03 | |
| *** yuanying-alt has joined #openstack-infra | 09:04 | |
| Kennan | hi infra cores | 09:05 |
| Kennan | https://review.openstack.org/#/c/218137/ from logs I did not figure out why gate-project-config-layout fail | 09:05 |
| Kennan | it not give much information, or I not familiar with it to get the reason | 09:05 |
| Kennan | could someone give a hint? | 09:05 |
| *** rossella_s has quit IRC | 09:05 | |
| *** pelix has joined #openstack-infra | 09:06 | |
| *** rossella_s has joined #openstack-infra | 09:06 | |
| rm_work | heading to bed, but want to shill for https://review.openstack.org/#/c/217458/ one last time :) | 09:08 |
| *** yuanying-alt has quit IRC | 09:08 | |
| *** marzif has quit IRC | 09:12 | |
| *** marzif has joined #openstack-infra | 09:12 | |
| *** Sree has quit IRC | 09:14 | |
| *** akraminsky has joined #openstack-infra | 09:19 | |
| Kennan | lifeless there? | 09:19 |
| *** zeih has joined #openstack-infra | 09:19 | |
| openstackgerrit | pangwa proposed openstack-infra/git-review: fix encoding issue on Windows https://review.openstack.org/218167 | 09:21 |
| *** GheRivero_Out is now known as GheRivero | 09:22 | |
| *** bauzas is now known as bauwser | 09:23 | |
| *** zeih has quit IRC | 09:24 | |
| *** amotoki has quit IRC | 09:25 | |
| *** e0ne has joined #openstack-infra | 09:25 | |
| *** claudiub has joined #openstack-infra | 09:31 | |
| *** skraynev has joined #openstack-infra | 09:33 | |
| *** ilyashakhat has joined #openstack-infra | 09:33 | |
| *** skolekonov has joined #openstack-infra | 09:33 | |
| *** skraynev has quit IRC | 09:38 | |
| *** jerryz has quit IRC | 09:39 | |
| *** ilyashakhat has quit IRC | 09:39 | |
| *** k4n0 has quit IRC | 09:39 | |
| *** marzif has quit IRC | 09:41 | |
| *** salv-orlando has quit IRC | 09:42 | |
| *** HeOS has joined #openstack-infra | 09:43 | |
| *** skraynev has joined #openstack-infra | 09:45 | |
| *** hashar has quit IRC | 09:57 | |
| *** Mmike has quit IRC | 09:57 | |
| *** Mmike has joined #openstack-infra | 09:57 | |
| *** hashar has joined #openstack-infra | 09:57 | |
| *** kiran-r has joined #openstack-infra | 09:59 | |
| *** zeih has joined #openstack-infra | 10:03 | |
| openstackgerrit | Kai Qiang Wu(Kennan) proposed openstack/requirements: Add kolla to requirements family https://review.openstack.org/218144 | 10:10 |
| *** pc_m has joined #openstack-infra | 10:13 | |
| openstackgerrit | Jesse Pretorius proposed openstack-infra/project-config: Rename stackforge/os-ansible-deployment{,specs} https://review.openstack.org/200730 | 10:14 |
| *** zeih has quit IRC | 10:15 | |
| openstackgerrit | Jesse Pretorius proposed openstack-infra/project-config: Rename stackforge/os-ansible-deployment{,specs} https://review.openstack.org/200730 | 10:19 |
| openstackgerrit | Romil Gupta proposed openstack/requirements: Add networking-vsphere to the list https://review.openstack.org/218189 | 10:20 |
| openstackgerrit | Alexander Makarov proposed openstack-dev/pbr: Protect WSGI application with a critical section https://review.openstack.org/217733 | 10:20 |
| *** ildikov has quit IRC | 10:21 | |
| *** shardy_ has joined #openstack-infra | 10:22 | |
| *** shardy has quit IRC | 10:23 | |
| *** akraminsky has quit IRC | 10:24 | |
| *** pc_m has quit IRC | 10:27 | |
| *** dizquierdo has quit IRC | 10:27 | |
| *** dguitarbite has quit IRC | 10:27 | |
| *** jeadre has quit IRC | 10:27 | |
| *** ryanpetrello has quit IRC | 10:27 | |
| *** lsell has quit IRC | 10:27 | |
| *** gondoi has quit IRC | 10:27 | |
| *** mgagne has quit IRC | 10:27 | |
| *** rainya has quit IRC | 10:27 | |
| *** apmelton has quit IRC | 10:27 | |
| *** asahlin has quit IRC | 10:27 | |
| *** rm_work has quit IRC | 10:27 | |
| *** wendar_ has quit IRC | 10:27 | |
| *** darrenc has quit IRC | 10:27 | |
| *** mattoliverau has quit IRC | 10:27 | |
| *** isviridov_away has quit IRC | 10:27 | |
| *** vaidy has quit IRC | 10:27 | |
| *** zxiiro has quit IRC | 10:27 | |
| *** legoktm has quit IRC | 10:27 | |
| *** purp has quit IRC | 10:27 | |
| *** juggler has quit IRC | 10:27 | |
| *** bknudson has quit IRC | 10:27 | |
| *** sarob has quit IRC | 10:27 | |
| *** adam_g has quit IRC | 10:27 | |
| *** Kennan has quit IRC | 10:27 | |
| *** jgriffith has quit IRC | 10:27 | |
| *** mika has quit IRC | 10:27 | |
| *** jamielennox has quit IRC | 10:27 | |
| *** phschwartz has quit IRC | 10:27 | |
| *** Ng has quit IRC | 10:27 | |
| *** kzaitsev_ws has quit IRC | 10:27 | |
| *** ruagair has quit IRC | 10:27 | |
| *** persia has quit IRC | 10:27 | |
| *** mhu has quit IRC | 10:27 | |
| *** michchap has quit IRC | 10:27 | |
| *** flaper87 has quit IRC | 10:27 | |
| *** d0ugal has quit IRC | 10:27 | |
| *** jogo has quit IRC | 10:27 | |
| *** mugsie has quit IRC | 10:27 | |
| *** Hunner has quit IRC | 10:27 | |
| *** svasheka has quit IRC | 10:27 | |
| *** dmsimard has quit IRC | 10:27 | |
| *** lxsli has quit IRC | 10:27 | |
| *** tcammann has quit IRC | 10:27 | |
| *** gilliard has quit IRC | 10:27 | |
| *** dguerri` has quit IRC | 10:27 | |
| *** pleia2 has quit IRC | 10:27 | |
| *** tlbr has quit IRC | 10:27 | |
| *** links has quit IRC | 10:27 | |
| *** jbryce has quit IRC | 10:27 | |
| *** comstud has quit IRC | 10:27 | |
| *** dtroyer has quit IRC | 10:27 | |
| *** lbragstad has quit IRC | 10:27 | |
| *** hockeynut has quit IRC | 10:27 | |
| *** Trozz_ has quit IRC | 10:27 | |
| *** tbarron has quit IRC | 10:27 | |
| *** freyes has quit IRC | 10:27 | |
| *** jroll has quit IRC | 10:27 | |
| *** dimtruck has quit IRC | 10:27 | |
| *** d34dh0r53 has quit IRC | 10:27 | |
| *** sigmavirus24_awa has quit IRC | 10:27 | |
| *** gus has quit IRC | 10:27 | |
| *** Nakato has quit IRC | 10:27 | |
| *** eglute has quit IRC | 10:27 | |
| *** markvoelker has quit IRC | 10:27 | |
| *** btully has quit IRC | 10:27 | |
| *** davideagnello has quit IRC | 10:27 | |
| *** afazekas has quit IRC | 10:27 | |
| *** barthalion has quit IRC | 10:27 | |
| *** ttx has quit IRC | 10:27 | |
| *** tteggel_ has quit IRC | 10:27 | |
| *** spredzy has quit IRC | 10:27 | |
| *** adreznec has quit IRC | 10:27 | |
| *** dteselkin has quit IRC | 10:27 | |
| *** NikitaKonovalov has quit IRC | 10:27 | |
| *** DinaBelova has quit IRC | 10:27 | |
| *** ruhe has quit IRC | 10:27 | |
| *** krotscheck has quit IRC | 10:27 | |
| *** anteaya has quit IRC | 10:27 | |
| *** rakhmerov has quit IRC | 10:27 | |
| *** tcammann1 has quit IRC | 10:27 | |
| *** asilenkov has quit IRC | 10:27 | |
| *** katyafervent has quit IRC | 10:27 | |
| *** shardy_ has quit IRC | 10:28 | |
| *** shardy has joined #openstack-infra | 10:28 | |
| openstackgerrit | Greg Chapple proposed openstack-infra/git-review: Allow reviewers to be defined in .gitreview file https://review.openstack.org/207795 | 10:28 |
| *** john-davidge_ has joined #openstack-infra | 10:28 | |
| *** links has joined #openstack-infra | 10:29 | |
| *** jbryce has joined #openstack-infra | 10:29 | |
| *** lbragstad has joined #openstack-infra | 10:29 | |
| *** comstud has joined #openstack-infra | 10:29 | |
| *** dtroyer has joined #openstack-infra | 10:29 | |
| *** hockeynut has joined #openstack-infra | 10:29 | |
| *** sigmavirus24_awa has joined #openstack-infra | 10:29 | |
| *** freyes has joined #openstack-infra | 10:29 | |
| *** Trozz_ has joined #openstack-infra | 10:29 | |
| *** dimtruck has joined #openstack-infra | 10:29 | |
| *** tbarron has joined #openstack-infra | 10:29 | |
| *** jroll has joined #openstack-infra | 10:29 | |
| *** d34dh0r53 has joined #openstack-infra | 10:29 | |
| *** gus has joined #openstack-infra | 10:29 | |
| *** eglute has joined #openstack-infra | 10:29 | |
| *** Nakato has joined #openstack-infra | 10:29 | |
| *** markvoelker has joined #openstack-infra | 10:29 | |
| *** btully has joined #openstack-infra | 10:29 | |
| *** davideagnello has joined #openstack-infra | 10:29 | |
| *** barthalion has joined #openstack-infra | 10:29 | |
| *** afazekas has joined #openstack-infra | 10:29 | |
| *** ttx has joined #openstack-infra | 10:29 | |
| *** tteggel_ has joined #openstack-infra | 10:29 | |
| *** spredzy has joined #openstack-infra | 10:29 | |
| *** adreznec has joined #openstack-infra | 10:29 | |
| *** dteselkin has joined #openstack-infra | 10:29 | |
| *** NikitaKonovalov has joined #openstack-infra | 10:29 | |
| *** DinaBelova has joined #openstack-infra | 10:29 | |
| *** ruhe has joined #openstack-infra | 10:29 | |
| *** krotscheck has joined #openstack-infra | 10:29 | |
| *** anteaya has joined #openstack-infra | 10:29 | |
| *** rakhmerov has joined #openstack-infra | 10:29 | |
| *** tcammann1 has joined #openstack-infra | 10:29 | |
| *** asilenkov has joined #openstack-infra | 10:29 | |
| *** katyafervent has joined #openstack-infra | 10:29 | |
| *** pc_m has joined #openstack-infra | 10:30 | |
| *** dizquierdo has joined #openstack-infra | 10:30 | |
| *** dguitarbite has joined #openstack-infra | 10:30 | |
| *** jeadre has joined #openstack-infra | 10:30 | |
| *** ryanpetrello has joined #openstack-infra | 10:30 | |
| *** lsell has joined #openstack-infra | 10:30 | |
| *** gondoi has joined #openstack-infra | 10:30 | |
| *** mgagne has joined #openstack-infra | 10:30 | |
| *** rainya has joined #openstack-infra | 10:30 | |
| *** apmelton has joined #openstack-infra | 10:30 | |
| *** asahlin has joined #openstack-infra | 10:30 | |
| *** rm_work has joined #openstack-infra | 10:30 | |
| *** wendar_ has joined #openstack-infra | 10:30 | |
| *** darrenc has joined #openstack-infra | 10:30 | |
| *** mattoliverau has joined #openstack-infra | 10:30 | |
| *** isviridov_away has joined #openstack-infra | 10:30 | |
| *** vaidy has joined #openstack-infra | 10:30 | |
| *** zxiiro has joined #openstack-infra | 10:30 | |
| *** legoktm has joined #openstack-infra | 10:30 | |
| *** purp has joined #openstack-infra | 10:30 | |
| *** juggler has joined #openstack-infra | 10:30 | |
| *** bknudson has joined #openstack-infra | 10:30 | |
| *** sarob has joined #openstack-infra | 10:30 | |
| *** adam_g has joined #openstack-infra | 10:30 | |
| *** Kennan has joined #openstack-infra | 10:30 | |
| *** jgriffith has joined #openstack-infra | 10:30 | |
| *** mika has joined #openstack-infra | 10:30 | |
| *** jamielennox has joined #openstack-infra | 10:30 | |
| *** Ng has joined #openstack-infra | 10:30 | |
| *** kzaitsev_ws has joined #openstack-infra | 10:30 | |
| *** ruagair has joined #openstack-infra | 10:30 | |
| *** persia has joined #openstack-infra | 10:30 | |
| *** mhu has joined #openstack-infra | 10:30 | |
| *** michchap has joined #openstack-infra | 10:30 | |
| *** flaper87 has joined #openstack-infra | 10:30 | |
| *** svasheka has joined #openstack-infra | 10:30 | |
| *** d0ugal has joined #openstack-infra | 10:30 | |
| *** jogo has joined #openstack-infra | 10:30 | |
| *** mugsie has joined #openstack-infra | 10:30 | |
| *** Hunner has joined #openstack-infra | 10:30 | |
| *** dmsimard has joined #openstack-infra | 10:30 | |
| *** lxsli has joined #openstack-infra | 10:30 | |
| *** tcammann has joined #openstack-infra | 10:30 | |
| *** gilliard has joined #openstack-infra | 10:30 | |
| *** pleia2 has joined #openstack-infra | 10:30 | |
| *** dguerri` has joined #openstack-infra | 10:30 | |
| *** tlbr has joined #openstack-infra | 10:30 | |
| *** flaper87 has quit IRC | 10:30 | |
| *** flaper87 has joined #openstack-infra | 10:30 | |
| *** phschwartz has joined #openstack-infra | 10:30 | |
| *** pc_m has quit IRC | 10:30 | |
| *** dizquierdo has quit IRC | 10:30 | |
| *** dguitarbite has quit IRC | 10:30 | |
| *** jeadre has quit IRC | 10:30 | |
| *** ryanpetrello has quit IRC | 10:30 | |
| *** lsell has quit IRC | 10:30 | |
| *** gondoi has quit IRC | 10:30 | |
| *** mgagne has quit IRC | 10:30 | |
| *** rainya has quit IRC | 10:30 | |
| *** apmelton has quit IRC | 10:30 | |
| *** asahlin has quit IRC | 10:30 | |
| *** rm_work has quit IRC | 10:30 | |
| *** wendar_ has quit IRC | 10:30 | |
| *** darrenc has quit IRC | 10:30 | |
| *** mattoliverau has quit IRC | 10:30 | |
| *** isviridov_away has quit IRC | 10:30 | |
| *** vaidy has quit IRC | 10:30 | |
| *** zxiiro has quit IRC | 10:30 | |
| *** legoktm has quit IRC | 10:30 | |
| *** purp has quit IRC | 10:30 | |
| *** juggler has quit IRC | 10:30 | |
| *** bknudson has quit IRC | 10:30 | |
| *** sarob has quit IRC | 10:30 | |
| *** adam_g has quit IRC | 10:30 | |
| *** Kennan has quit IRC | 10:30 | |
| *** jgriffith has quit IRC | 10:30 | |
| *** mika has quit IRC | 10:30 | |
| *** jamielennox has quit IRC | 10:30 | |
| *** Ng has quit IRC | 10:30 | |
| *** kzaitsev_ws has quit IRC | 10:30 | |
| *** ruagair has quit IRC | 10:30 | |
| *** persia has quit IRC | 10:30 | |
| *** mhu has quit IRC | 10:30 | |
| *** michchap has quit IRC | 10:30 | |
| *** flaper87 has quit IRC | 10:30 | |
| *** d0ugal has quit IRC | 10:30 | |
| *** jogo has quit IRC | 10:30 | |
| *** mugsie has quit IRC | 10:30 | |
| *** Hunner has quit IRC | 10:30 | |
| *** svasheka has quit IRC | 10:30 | |
| *** dmsimard has quit IRC | 10:30 | |
| *** lxsli has quit IRC | 10:30 | |
| *** tcammann has quit IRC | 10:30 | |
| *** gilliard has quit IRC | 10:30 | |
| *** dguerri` has quit IRC | 10:30 | |
| *** pleia2 has quit IRC | 10:30 | |
| *** tlbr has quit IRC | 10:30 | |
| *** pc_m has joined #openstack-infra | 10:33 | |
| *** dizquierdo has joined #openstack-infra | 10:33 | |
| *** dguitarbite has joined #openstack-infra | 10:33 | |
| *** jeadre has joined #openstack-infra | 10:33 | |
| *** ryanpetrello has joined #openstack-infra | 10:33 | |
| *** lsell has joined #openstack-infra | 10:33 | |
| *** mgagne has joined #openstack-infra | 10:33 | |
| *** rainya has joined #openstack-infra | 10:33 | |
| *** apmelton has joined #openstack-infra | 10:33 | |
| *** asahlin has joined #openstack-infra | 10:33 | |
| *** rm_work has joined #openstack-infra | 10:33 | |
| *** wendar_ has joined #openstack-infra | 10:33 | |
| *** darrenc has joined #openstack-infra | 10:33 | |
| *** mattoliverau has joined #openstack-infra | 10:33 | |
| *** isviridov_away has joined #openstack-infra | 10:33 | |
| *** vaidy has joined #openstack-infra | 10:33 | |
| *** zxiiro has joined #openstack-infra | 10:33 | |
| *** legoktm has joined #openstack-infra | 10:33 | |
| *** purp has joined #openstack-infra | 10:33 | |
| *** juggler has joined #openstack-infra | 10:33 | |
| *** bknudson has joined #openstack-infra | 10:33 | |
| *** sarob has joined #openstack-infra | 10:33 | |
| *** adam_g has joined #openstack-infra | 10:33 | |
| *** Kennan has joined #openstack-infra | 10:33 | |
| *** jgriffith has joined #openstack-infra | 10:33 | |
| *** mika has joined #openstack-infra | 10:33 | |
| *** jamielennox has joined #openstack-infra | 10:33 | |
| *** Ng has joined #openstack-infra | 10:33 | |
| *** kzaitsev_ws has joined #openstack-infra | 10:33 | |
| *** ruagair has joined #openstack-infra | 10:33 | |
| *** persia has joined #openstack-infra | 10:33 | |
| *** mhu has joined #openstack-infra | 10:33 | |
| *** michchap has joined #openstack-infra | 10:33 | |
| *** flaper87 has joined #openstack-infra | 10:33 | |
| *** svasheka has joined #openstack-infra | 10:33 | |
| *** d0ugal has joined #openstack-infra | 10:33 | |
| *** jogo has joined #openstack-infra | 10:33 | |
| *** mugsie has joined #openstack-infra | 10:33 | |
| *** Hunner has joined #openstack-infra | 10:33 | |
| *** dmsimard has joined #openstack-infra | 10:33 | |
| *** lxsli has joined #openstack-infra | 10:33 | |
| *** tcammann has joined #openstack-infra | 10:33 | |
| *** gilliard has joined #openstack-infra | 10:33 | |
| *** pleia2 has joined #openstack-infra | 10:33 | |
| *** dguerri` has joined #openstack-infra | 10:33 | |
| *** tlbr has joined #openstack-infra | 10:33 | |
| *** gondoi has joined #openstack-infra | 10:34 | |
| john-davidge_ | Are the current gate-tempest-dsvm-neutron-dvr failures a known issue? Or do we still need to find the cause? Example: https://review.openstack.org/#/c/215791/ | 10:36 |
| pc_m | john-davidge_: You can look at the L3 meeting minutes from yesterday. I thought I recalled them talking about test issues with DVR. | 10:37 |
| john-davidge_ | pc_m: I don’t remember a gate-blocker being discussed. Will take a look | 10:38 |
| pc_m | john-davidge_: Thought I did, but then again I was multi-tasking during the meeting :) | 10:39 |
| pc_m | gates been backed up for several days now... :( | 10:39 |
| *** zhenguo has quit IRC | 10:40 | |
| *** nikil89_ has joined #openstack-infra | 10:41 | |
| *** dtantsur is now known as dtantsur|brb | 10:43 | |
| openstackgerrit | YAMAMOTO Takashi proposed openstack-infra/project-config: Set DEVSTACK_GATE_SETTINGS for midonet gate job https://review.openstack.org/207731 | 10:43 |
| *** zeih has joined #openstack-infra | 10:44 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/puppet-jeepyb: Use new logging feature of jeepyb https://review.openstack.org/218202 | 10:45 |
| *** fhubik is now known as fhubik_brb | 10:45 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/system-config: Move manage_projects logic to puppet-jeepyb https://review.openstack.org/171565 | 10:46 |
| *** zeih_ has joined #openstack-infra | 10:48 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/system-config: Start using jfryman/puppet-selinux https://review.openstack.org/188357 | 10:49 |
| *** zeih has quit IRC | 10:49 | |
| openstackgerrit | Merged openstack-infra/release-tools: Fix formatting in process_bugs.py output https://review.openstack.org/216434 | 10:49 |
| openstackgerrit | yolanda.robla proposed openstack-infra/system-config: Remove graphite wrapper class https://review.openstack.org/216693 | 10:51 |
| *** yuanying-alt has joined #openstack-infra | 10:52 | |
| *** dimsum__ has joined #openstack-infra | 10:53 | |
| *** yuanying-alt has quit IRC | 10:57 | |
| *** ildikov has joined #openstack-infra | 11:06 | |
| *** Hal has joined #openstack-infra | 11:11 | |
| *** Hal is now known as Guest74995 | 11:11 | |
| *** shardy has quit IRC | 11:14 | |
| *** thorst1 has joined #openstack-infra | 11:19 | |
| *** yamamoto has quit IRC | 11:21 | |
| *** fhubik_brb is now known as fhubik | 11:28 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/puppet-jeepyb: Separate the fetch remotes log dir and file https://review.openstack.org/192187 | 11:28 |
| openstackgerrit | yolanda.robla proposed openstack-infra/system-config: Move manage_projects logic to puppet-jeepyb https://review.openstack.org/171565 | 11:29 |
| *** dprince has joined #openstack-infra | 11:30 | |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Update DELOREAN_URL to one from 2015-08-26 https://review.openstack.org/214591 | 11:31 |
| *** marzif has joined #openstack-infra | 11:32 | |
| *** dtantsur|brb is now known as dtantsur | 11:34 | |
| *** devvesa has quit IRC | 11:37 | |
| *** luqas has quit IRC | 11:38 | |
| *** luqas has joined #openstack-infra | 11:39 | |
| *** esker has quit IRC | 11:40 | |
| *** fhubik is now known as fhubik_brb | 11:45 | |
| *** zeih_ has quit IRC | 11:45 | |
| openstackgerrit | Igor Shishkin proposed openstack-infra/lodgeit: Turning NullPool to avoid MySQL server gone away https://review.openstack.org/218224 | 11:46 |
| *** fhubik_brb is now known as fhubik | 11:50 | |
| *** kiran-r has quit IRC | 11:51 | |
| *** julim has joined #openstack-infra | 11:51 | |
| *** gordc has joined #openstack-infra | 11:51 | |
| *** devvesa has joined #openstack-infra | 11:52 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/puppet-jenkins: Don't configure memory swap bytes on cgroups for Trusty https://review.openstack.org/218227 | 11:54 |
| TheJulia | greghaynes: so thinking about it, I'm reminded of a bug towards the end of kilo with regards to IPA and the write-out of configuration drives, but I'm 99.95% sure we fixed that. | 11:56 |
| TheJulia | greghaynes: Then, basically, it was every other deployment the config drive would be preserved from the previous reboot because the old drive was locked due to coreos. | 11:56 |
| *** trown|outttypeww is now known as trown | 11:57 | |
| *** hashar has quit IRC | 11:58 | |
| openstackgerrit | Igor Shishkin proposed openstack-infra/lodgeit: Security fix for possbile private paste bruteforcing https://review.openstack.org/218228 | 11:58 |
| *** zeih has joined #openstack-infra | 12:02 | |
| openstackgerrit | Igor Shishkin proposed openstack-infra/lodgeit: Security fix for possible private paste bruteforcing https://review.openstack.org/218228 | 12:02 |
| *** dingyichen has quit IRC | 12:03 | |
| openstackgerrit | Igor Shishkin proposed openstack-infra/lodgeit: Security fix for possible private paste bruteforcing https://review.openstack.org/218228 | 12:03 |
| openstackgerrit | Alexander Makarov proposed openstack-dev/pbr: Protect WSGI application with a critical section https://review.openstack.org/217733 | 12:04 |
| *** yamamoto has joined #openstack-infra | 12:04 | |
| *** yamamoto has quit IRC | 12:05 | |
| *** yamamoto has joined #openstack-infra | 12:05 | |
| *** vikram has quit IRC | 12:05 | |
| *** yamamoto_ has joined #openstack-infra | 12:05 | |
| *** pradk has quit IRC | 12:07 | |
| *** madong_ has quit IRC | 12:10 | |
| *** yamamoto has quit IRC | 12:10 | |
| *** yfried_ has joined #openstack-infra | 12:11 | |
| *** zeih has quit IRC | 12:12 | |
| *** deepakcs has quit IRC | 12:14 | |
| *** zeih has joined #openstack-infra | 12:15 | |
| *** Kennan has quit IRC | 12:18 | |
| *** vikram has joined #openstack-infra | 12:19 | |
| *** glauco_ has joined #openstack-infra | 12:19 | |
| *** glauco_ has quit IRC | 12:21 | |
| *** fawadk has quit IRC | 12:21 | |
| *** cburroughs has joined #openstack-infra | 12:22 | |
| *** glauco has joined #openstack-infra | 12:22 | |
| *** thiagop has joined #openstack-infra | 12:22 | |
| *** vivekd has quit IRC | 12:23 | |
| openstackgerrit | Glauco Oliveira proposed openstack-infra/puppet-zuul: Fix puppet-lint errors https://review.openstack.org/217788 | 12:25 |
| *** fawadkhaliq has joined #openstack-infra | 12:26 | |
| glauco | yolanda: ^^^ | 12:26 |
| glauco | yolanda: just included the .gitignore you mentioned | 12:26 |
| *** ihrachys has quit IRC | 12:27 | |
| *** aysyd has joined #openstack-infra | 12:27 | |
| *** ihrachys has joined #openstack-infra | 12:27 | |
| yolanda | hi glauco, i'll check | 12:27 |
| *** rfolco has joined #openstack-infra | 12:29 | |
| *** fawadk has joined #openstack-infra | 12:29 | |
| *** glauco has quit IRC | 12:29 | |
| *** fawadkhaliq has quit IRC | 12:30 | |
| *** salv-orlando has joined #openstack-infra | 12:30 | |
| *** salv-orlando has quit IRC | 12:32 | |
| *** Kennan has joined #openstack-infra | 12:33 | |
| *** chlong has joined #openstack-infra | 12:34 | |
| *** trown is now known as trown|brb | 12:36 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: copy_puppet_logs: copy tempest.conf in txt file https://review.openstack.org/218245 | 12:36 |
| EmilienM | Kennan: looking https://review.openstack.org/#/c/218137/ | 12:36 |
| EmilienM | Kennan: how can I help? | 12:37 |
| *** yamamoto_ has quit IRC | 12:39 | |
| *** yamamoto has joined #openstack-infra | 12:39 | |
| *** yamamoto has quit IRC | 12:40 | |
| *** BharatK has quit IRC | 12:42 | |
| *** samueldmq has quit IRC | 12:43 | |
| *** spzala has joined #openstack-infra | 12:45 | |
| *** Kennan has quit IRC | 12:46 | |
| *** dizquierdo has quit IRC | 12:46 | |
| *** zul has quit IRC | 12:49 | |
| *** glauco has joined #openstack-infra | 12:50 | |
| *** nelsnelson has joined #openstack-infra | 12:50 | |
| *** hashar has joined #openstack-infra | 12:51 | |
| *** fawadk has quit IRC | 12:51 | |
| *** zul has joined #openstack-infra | 12:52 | |
| *** jcoufal has joined #openstack-infra | 12:53 | |
| openstackgerrit | Julien Danjou proposed openstack-infra/project-config: Fix Aodh permissions to let the release team do its job https://review.openstack.org/218254 | 12:53 |
| *** baoli has joined #openstack-infra | 12:54 | |
| *** mpaolino has joined #openstack-infra | 12:55 | |
| *** mpaolino has quit IRC | 12:55 | |
| *** fawadkhaliq has joined #openstack-infra | 12:55 | |
| *** tlian has joined #openstack-infra | 12:55 | |
| *** woodster_ has joined #openstack-infra | 12:55 | |
| *** Kennan has joined #openstack-infra | 13:00 | |
| *** marzif has quit IRC | 13:00 | |
| *** baoli has quit IRC | 13:01 | |
| *** baoli has joined #openstack-infra | 13:02 | |
| *** bknudson has left #openstack-infra | 13:02 | |
| *** dzamboni has joined #openstack-infra | 13:02 | |
| *** thiagop_ has joined #openstack-infra | 13:03 | |
| *** hashar_ has joined #openstack-infra | 13:04 | |
| *** amuller has joined #openstack-infra | 13:04 | |
| *** hdd has joined #openstack-infra | 13:05 | |
| *** hashar has quit IRC | 13:06 | |
| derekh | Is there anybody else around who can take a look at a reshuffle of tripleo jobs/repos please https://review.openstack.org/#/c/205479/ https://review.openstack.org/#/c/215186/ | 13:09 |
| openstackgerrit | Merged openstack/requirements: Use requests-kerberos for python 2.6 as well https://review.openstack.org/216530 | 13:09 |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack-infra/devstack-gate: [WIP] Try to run against all oslo libraries from master https://review.openstack.org/217875 | 13:10 |
| *** glauco has quit IRC | 13:11 | |
| *** pradk has joined #openstack-infra | 13:13 | |
| *** annegentle has joined #openstack-infra | 13:17 | |
| *** nicodemos has joined #openstack-infra | 13:17 | |
| *** asselin has joined #openstack-infra | 13:22 | |
| *** rlandy has joined #openstack-infra | 13:23 | |
| *** ldnunes has joined #openstack-infra | 13:24 | |
| *** leopoldj has quit IRC | 13:24 | |
| *** trown|brb is now known as trown | 13:26 | |
| openstackgerrit | Sahdev Zala proposed openstack/requirements: Add tosca-parser to global requirements https://review.openstack.org/217991 | 13:26 |
| *** bdemers has joined #openstack-infra | 13:26 | |
| *** nelsnelson has quit IRC | 13:27 | |
| *** amuller is now known as amuller_afk | 13:27 | |
| *** hurgleburgler has joined #openstack-infra | 13:29 | |
| *** links has quit IRC | 13:29 | |
| *** sigmavirus24_awa is now known as sigmavirus24 | 13:30 | |
| *** amotoki has joined #openstack-infra | 13:32 | |
| *** puranamr has joined #openstack-infra | 13:33 | |
| *** wenlock has joined #openstack-infra | 13:33 | |
| *** samuelBartel has joined #openstack-infra | 13:36 | |
| *** alivigni has joined #openstack-infra | 13:36 | |
| *** mriedem has joined #openstack-infra | 13:37 | |
| *** salv-orlando has joined #openstack-infra | 13:37 | |
| *** bknudson has joined #openstack-infra | 13:38 | |
| *** puranamr has quit IRC | 13:39 | |
| *** yfried_ has quit IRC | 13:40 | |
| *** dansmith is now known as superdan | 13:42 | |
| *** zeih has quit IRC | 13:43 | |
| *** Kennan2 has joined #openstack-infra | 13:44 | |
| *** Kennan has quit IRC | 13:44 | |
| *** glauco_ has joined #openstack-infra | 13:46 | |
| *** yamamoto has joined #openstack-infra | 13:46 | |
| *** bnemec has joined #openstack-infra | 13:46 | |
| *** mmedvede has quit IRC | 13:47 | |
| *** nikil89_ is now known as naggappan | 13:47 | |
| *** HeOS has quit IRC | 13:48 | |
| *** gildub has quit IRC | 13:48 | |
| sdague | fungi: can we get a promote on - https://review.openstack.org/#/c/217804/3 | 13:48 |
| sdague | it should reduce gate failure rate | 13:48 |
| sdague | there was a missing wait in the tempest test | 13:48 |
| *** peristeri has quit IRC | 13:49 | |
| *** alaski is now known as lascii | 13:49 | |
| *** dimsum__ has quit IRC | 13:49 | |
| *** mmedvede has joined #openstack-infra | 13:50 | |
| *** dimsum__ has joined #openstack-infra | 13:50 | |
| openstackgerrit | Merged openstack-infra/elastic-recheck: Add query for test boot from volume snapshot bug 1489581 https://review.openstack.org/217868 | 13:51 |
| openstack | bug 1489581 in tempest "test_create_ebs_image_and_check_boot is race failing" [High,In progress] https://launchpad.net/bugs/1489581 - Assigned to Andrey Pavlov (apavlov-e) | 13:51 |
| *** yamamoto has quit IRC | 13:51 | |
| *** jordanP has joined #openstack-infra | 13:52 | |
| *** skolekonov has quit IRC | 13:52 | |
| *** eharney has joined #openstack-infra | 13:53 | |
| *** dkranz has joined #openstack-infra | 13:53 | |
| *** regXboi has joined #openstack-infra | 13:53 | |
| *** zeih has joined #openstack-infra | 13:55 | |
| *** skolekonov has joined #openstack-infra | 13:56 | |
| *** fawadk has joined #openstack-infra | 13:57 | |
| *** amuller_afk is now known as amuller | 13:57 | |
| *** jaosorior has joined #openstack-infra | 13:57 | |
| *** kgiusti has joined #openstack-infra | 13:58 | |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck: Update query for ebs test bug 1489581 https://review.openstack.org/218276 | 13:59 |
| openstack | bug 1489581 in tempest "test_create_ebs_image_and_check_boot is race failing" [High,In progress] https://launchpad.net/bugs/1489581 - Assigned to Andrey Pavlov (apavlov-e) | 13:59 |
| *** Kennan2 has quit IRC | 14:00 | |
| *** fawadkhaliq has quit IRC | 14:00 | |
| *** hashar_ is now known as hashar | 14:00 | |
| *** dprince has quit IRC | 14:01 | |
| *** shayneburgess has left #openstack-infra | 14:01 | |
| *** kaisers has left #openstack-infra | 14:02 | |
| fungi | sdague: bumping it up now | 14:02 |
| *** fhubik has quit IRC | 14:03 | |
| sdague | fungi: thank you | 14:04 |
| sdague | there are also some neutron fixes, but there seem to be at least three big failures and they are all failing each other's patch | 14:05 |
| fungi | that's marvellous | 14:05 |
| sdague | yeh, that's what happens when folks don't stay on top of them, and we get backed up | 14:05 |
| fungi | saw the py34 failures for neutron early yesterday, but was so far behind on e-mail i figured it was already being heavily discussed on the ml | 14:06 |
| *** amitgandhinz has joined #openstack-infra | 14:06 | |
| fungi | looked maybe oslo.db related | 14:06 |
| sdague | yeh, so there is a tox revert, as well as a real proposed fix | 14:06 |
| sdague | honestly, probably need someone from the neutron team more engaged here to decide which is the right one | 14:06 |
| amuller | the right one is the proper fix | 14:07 |
| *** Piet has quit IRC | 14:07 | |
| amuller | sdague: can you bump that to the front of the gate queue? | 14:07 |
| amuller | sdague: https://review.openstack.org/#/c/217379/ | 14:07 |
| jordanP | Cinder is hit by a serious bug here : https://bugs.launchpad.net/cinder/+bug/1488690?comments=all and it has a "fix" here: https://review.openstack.org/#/c/217882/ Are you aware of this one ? | 14:08 |
| openstack | Launchpad bug 1488690 in Cinder "test_create_backing_from_stream_optimized_file fails with "AttributeError: '_SentinelObject' object has no attribute 'readlines'"" [Undecided,In progress] - Assigned to Jon Bernard (jbernard) | 14:08 |
| jordanP | there's a logstack query in the bug report | 14:10 |
| *** yamamoto has joined #openstack-infra | 14:10 | |
| fungi | amuller: i think we need a better likelihood of 217379 actually passing jobs before we can promote it. looks like last round it failed both the functional and dvr jobs on the current patchset | 14:10 |
| *** zeih has quit IRC | 14:10 | |
| fungi | amuller: a super-patch with the fixes for whatever's breaking the other two jobs squashed together would probably stand more of a chance of making it in | 14:11 |
| amuller | fungi: The neutron functional job has had a consistently highish (15-20%) failure rate for the past month, it's not likely to change today | 14:11 |
| fungi | that seems too high to remain voting | 14:12 |
| amuller | fungi: you might be right | 14:12 |
| amuller | fungi: I would be OK with making the job non-voting until we can get some fixes in there | 14:12 |
| amuller | fungi: it's around 5 different failures at this point, together they cause a pretty high failure rate | 14:13 |
| amuller | fungi: the DVR failure I'm not up to date yet, there's a discussion starting on openstack-neutron now, that looks like a new issue | 14:13 |
| *** Kennan has joined #openstack-infra | 14:14 | |
| fungi | amuller: it sounds like multiple new sources of nondeterminism have been recheck-spammed into neutron in recent days/weeks, so yes i would recommend declaring a state of emergency within your project, getting your fellow core reviewers to stop approving new neutron changes, switch the jobs which aren't consistently passing to non-voting, take those jobs out of the gate pipeline, and focus on merging | 14:16 |
| fungi | only stability fixes until you can get those jobs working consistently enough to be reenabled | 14:16 |
| amuller | fungi: I'll push a patch to project-config to make the functional job non-voting | 14:17 |
| *** yfried_ has joined #openstack-infra | 14:17 | |
| sdague | amuller: and the dvr one, right? | 14:17 |
| amuller | merge that in, and push the py34 fix to the top of the gate queue | 14:17 |
| sdague | because I think both are causing a bunch of fails | 14:17 |
| sdague | I think if those both go non voting, we can then promote this, and feel more confident it will land | 14:17 |
| amuller | sdague: do you have the failure rate on that one on hand? If not I can grab it from somewhere | 14:17 |
| sdague | amuller: I don't, it just seemed to show up as problematic in the thread | 14:18 |
| fungi | the trick there is to avoid having people approve other changes to make the situation worse while those jobs are disabled | 14:18 |
| sdague | and it definitely bounced that patch | 14:18 |
| sdague | fungi: well, we could pull them from the gate queue and leave them in check | 14:18 |
| fungi | yep | 14:18 |
| sdague | that means that they'd still add friction to neutron, but not other projects | 14:19 |
| fungi | that would be a reasonable middle-ground | 14:19 |
| amuller | that sounds good | 14:19 |
| amuller | it'll add more incentive to other neutron cores to take this seriously since it'll still screw up our own queue :) | 14:19 |
| sdague | amuller: :) | 14:20 |
| sdague | yeh, so get that project config patch up, and we can get that in pretty quick | 14:20 |
| fungi | but yeah, it'll be tough since this is heading into feature freeze, so there's a lot of temptation for feature changes to get rammed through no matter the cost (which is probably partly how things have gotten this bad in recent days) | 14:20 |
| sdague | fungi: so, honestly, I think that's been happening regardless | 14:20 |
| fungi | wouldn't surprise me | 14:20 |
| amuller | sdague: fungi: DVR failure rate - holy crap: http://goo.gl/8xjtro | 14:21 |
| amuller | yeah time to make that non-voting as well | 14:21 |
| *** zhenguo has joined #openstack-infra | 14:22 | |
| fungi | yeeowch | 14:22 |
| amuller | or knock it off the gate queue at least | 14:22 |
| amuller | oh I know exactly what happened with the DVR failure rate: It's a new tempest test | 14:23 |
| amuller | https://review.openstack.org/#/c/209949/ | 14:24 |
| amuller | that test actually has a conditional skip for DVR but it looks like it's not skipping properly | 14:26 |
| openstackgerrit | Merged openstack-infra/elastic-recheck: Update query for ebs test bug 1489581 https://review.openstack.org/218276 | 14:27 |
| openstack | bug 1489581 in tempest "test_create_ebs_image_and_check_boot is race failing" [High,In progress] https://launchpad.net/bugs/1489581 - Assigned to Andrey Pavlov (apavlov-e) | 14:27 |
| amuller | fungi: sdague: mtreinish: I think the proper fix for the DVR failure rate is to actually fix that tempest test so that it skips it for DVR properly | 14:27 |
| *** yamahata has joined #openstack-infra | 14:28 | |
| *** yamamoto has quit IRC | 14:28 | |
| amuller | mtreinish: I notice that the DVR job is not running on tempest changes, which is one reason why that patch got it in when it should not have | 14:29 |
| anteaya | amuller: I think you need to bring mestery armax or dougwig into this conversation | 14:29 |
| *** haleyb has joined #openstack-infra | 14:29 | |
| anteaya | as they are the neutron infra liasons | 14:29 |
| amuller | *sighs* so many fires I don't know where to start | 14:29 |
| mtreinish | amuller: I thought it was running on tempest | 14:29 |
| anteaya | and are the ones we have been asked to co-ordinate with in times like this | 14:29 |
| mtreinish | amuller: its on check, not gate | 14:29 |
| regXboi | amuller: gate-tempest-dsvm-neutron-dvr passed on that patch | 14:29 |
| amuller | anteaya: do you want me to not help and go away? | 14:29 |
| anteaya | as the point of having liasons in the first place is to avoid having to go back and undo things after the fact | 14:30 |
| *** andersonvom has quit IRC | 14:30 | |
| anteaya | amuller: I would like you to respect decsions others have made | 14:30 |
| amuller | anteaya: I can push a patch to add myself as an infra liason if that will make you happy | 14:30 |
| anteaya | neuton made this decison | 14:30 |
| sdague | anteaya: yeh, unfortunately, the liasons haven't gotten engaged here as of yet | 14:30 |
| amuller | in the mean time I'm trying to help | 14:30 |
| anteaya | it is up to infra to honor it | 14:30 |
| anteaya | sdague: were they pinged or asked? | 14:30 |
| sdague | we had a whole ML thread which kicked off this set of actions this morning | 14:31 |
| anteaya | discussion is good and so are opinions | 14:31 |
| anteaya | but we had to go through a whole lot of reverting patches a while back | 14:31 |
| * regXboi checking the tempest DVR job on the tempest patch | 14:31 | |
| sdague | and I've yet to see any of the folks listed on that thread | 14:31 |
| anteaya | which was the motivation for setting up liasons in the first place | 14:31 |
| *** burgerk has joined #openstack-infra | 14:32 | |
| *** hdd has quit IRC | 14:32 | |
| * fungi hopes the current neutron ci job status is not a surprise to neutron's infra liaisons | 14:32 | |
| regXboi | so... I'll poing out that while the DVR job passed on that tempest change, it didn't pass in the correct way | 14:33 |
| anteaya | the thread is 2.5 hours old | 14:33 |
| regXboi | http://logs.openstack.org/49/209949/6/check/gate-tempest-dsvm-neutron-dvr/b03c367/console.html.gz | 14:33 |
| anteaya | I don't know if mestery armax and dougwig are online | 14:33 |
| regXboi | it didn't skip the job, it gave an OK on it | 14:33 |
| regXboi | anteaya: mestery is on his way | 14:33 |
| regXboi | but I come back to the fact that the tempest job didn't do what it was supposed to :( | 14:33 |
| anteaya | regXboi: thank you | 14:34 |
| sdague | anteaya: right, so the issue is the liason model works fine when the liasons are keeping on top of the issues. This does not seem to be the case here given the composite failure rates we got. So I'm happy for anyone that's trying to help fix things. | 14:35 |
| anteaya | sdague: well the postions are 2 weeks old | 14:35 |
| anteaya | so I think they are figuring it out what it means to be liason | 14:35 |
| fungi | and so may still need adjusting | 14:35 |
| anteaya | so I don't think jumping over their heads and not even pinging them is going to help them understand what is needed here | 14:36 |
| regXboi | fungi, sdague, anteaya: while mestery catches up, I'll point folks at the URL I pasted in above | 14:36 |
| anteaya | regXboi: as you see fit | 14:36 |
| regXboi | anteaya: they've been pinged in -neutron | 14:36 |
| sdague | regXboi: right, so there is a fix there | 14:36 |
| openstackgerrit | Assaf Muller proposed openstack-infra/project-config: Remove Neutron functioal job from the gate queue https://review.openstack.org/218302 | 14:36 |
| *** burgerk_ has joined #openstack-infra | 14:36 | |
| sdague | but I don't think we can land it until we get some of these other bits in | 14:36 |
| anteaya | sdague: I'm not saying there isn't an issue, I'm saying we also need to learn how to work with the neutron liasons | 14:37 |
| * regXboi would strongly prefer to make the dvr job NV, and not remove it | 14:37 | |
| *** dprince has joined #openstack-infra | 14:37 | |
| openstackgerrit | Assaf Muller proposed openstack-infra/project-config: Remove Neutron functional job from the gate queue https://review.openstack.org/218302 | 14:37 |
| fungi | regXboi: i think what was being suggested was to leave the dvr, py34 and functional jobs in the check pipeline but remove them from the gate pipeline (temporarily) while the fixes for those jobs get merged | 14:37 |
| regXboi | fungi: ack | 14:38 |
| sdague | regXboi: well, we were only considering removing it from the "gate" queue, it would still be voting on check | 14:38 |
| fungi | and then put them back if they're confirmed stabilized | 14:38 |
| *** sigmavirus24 is now known as sigmavirus24_awa | 14:38 | |
| *** sigmavirus24_awa is now known as sigmavirus24 | 14:38 | |
| regXboi | fungi: I missed that part of the conversation, but that makes more sense | 14:38 |
| amuller | mtreinish: if it's possible to fix that conditional skip trivially, that would be a better solution than to take the DVR job off the gate queue | 14:38 |
| *** nfedotov has quit IRC | 14:38 | |
| fungi | problem being there are several sources of nondeterminism in several jobs, and the fixes for each are somewhat blocked on the failure rates caused by the others | 14:39 |
| *** glauco_ has quit IRC | 14:39 | |
| amuller | mtreinish: I don't have enough experience with Tempest to fix that myself, so if you're willing to do that, great, if not tell me and let's take the DVR job off the gate queue as well | 14:39 |
| *** burgerk has quit IRC | 14:39 | |
| *** arxcruz has quit IRC | 14:39 | |
| regXboi | so... mestery has +1'd https://review.openstack.org/#/c/218302/ | 14:39 |
| regXboi | (he is on a phone call and not able to get over here) | 14:40 |
| mtreinish | amuller: kevinbenton already has a fix proposed | 14:40 |
| sdague | fungi: ok so mestery is +1 on - https://review.openstack.org/#/c/218302 | 14:40 |
| sdague | can you +A that | 14:40 |
| sdague | that removes one layer | 14:40 |
| sdague | mtreinish: where's the tempest patch? | 14:41 |
| mestery | sdague fungi regXboi amuller: Yes, apologies, I'm juggling too many things but that looks good, I've read the scrollback here and I'm caught up | 14:41 |
| amuller | https://review.openstack.org/#/c/218242/ | 14:41 |
| mtreinish | sdague: waiting on check, it's got my +2 | 14:41 |
| *** glauco has joined #openstack-infra | 14:41 | |
| anteaya | mestery: thank you | 14:41 |
| amuller | ok, so we need these three patches merged: https://review.openstack.org/#/c/218242/ (skip failing dvr test), https://review.openstack.org/#/c/218302/ (knock functional job off the gate queue), and https://review.openstack.org/#/c/217379/ (fix neutron py34 job) | 14:42 |
| amuller | in that order | 14:42 |
| amuller | are we good? | 14:42 |
| sdague | mtreinish: ok, I'll +2 as well, but I agree we need results on it | 14:42 |
| amuller | or not in that order actually =D | 14:42 |
| amuller | kevinbenton: thanks for the Tempest patch | 14:42 |
| regXboi | wait | 14:42 |
| regXboi | isn't that logic backwards? | 14:42 |
| regXboi | we want to skip if distributed is True | 14:43 |
| fungi | sdague: mtreinish: some of the tempest jobs for 218242 have already completed (successfully) | 14:43 |
| *** julim has quit IRC | 14:43 | |
| fungi | what results specifically are you looking for--is there a particular job whose logs would be most useful? | 14:43 |
| kevinbenton | regXboi: that's what we do | 14:44 |
| sdague | fungi: the dvr job | 14:44 |
| amuller | regXboi: naw that conditional is correct, the False is the default value in case 'distributed' is not part of the router object | 14:44 |
| sdague | this should only impact the dvr job | 14:44 |
| sdague | so that's the one we want to see green | 14:44 |
| fungi | sdague: ahh, yep, still running, eta 24 minutes according to jenkins | 14:44 |
| *** mtanino has joined #openstack-infra | 14:44 | |
| haleyb | regXboi: i think that gettattr() call is just defaulting to False, but will raise if True | 14:44 |
| *** bexelbie has quit IRC | 14:45 | |
| * regXboi remains somewhat unconvinced, but the check pipeline *should* make me happy | 14:45 | |
| *** teran has joined #openstack-infra | 14:45 | |
| *** FallenPegasus has joined #openstack-infra | 14:46 | |
| *** amrith is now known as _amrith_ | 14:47 | |
| * regXboi needs to go run a separate meeting at the top of the hour, but will keep his fingers crossed | 14:47 | |
| *** vikram has quit IRC | 14:47 | |
| fungi | yeah, top of the hour is roughly when we should expect to have an indication of whether the tempest change dtrt | 14:48 |
| jeblair | yes, the third argument to getattr is the value to return if there is no such named attribute. | 14:48 |
| *** doude has quit IRC | 14:48 | |
| regXboi | amuller, haleyb: assuming that 218242 merges, I think we should run a recheck on some of the jobs that failed in the neutron heck pipeline just to double check | 14:48 |
| regXboi | amuller, haleyb: what do you think? | 14:49 |
| fungi | https://docs.python.org/2/library/functions.html#getattr | 14:49 |
| *** fitoduarte has quit IRC | 14:49 | |
| haleyb | regXboi: sure, there's plenty to choose from :) | 14:49 |
| *** doude has joined #openstack-infra | 14:49 | |
| regXboi | haleyb: ack - I have the logstash list and I was thinking of hitting all of them in turn | 14:49 |
| * regXboi was looking for so map reduce help :) | 14:50 | |
| regXboi | er for *some* map | 14:50 |
| *** thiagop_ has quit IRC | 14:54 | |
| *** thiagop_ has joined #openstack-infra | 14:54 | |
| *** bhunter71 has joined #openstack-infra | 14:54 | |
| *** sergmelikyan has joined #openstack-infra | 14:55 | |
| *** ihrachys_ has joined #openstack-infra | 14:56 | |
| haleyb | regXboi: you just want to hit a subset? | 14:57 |
| *** ihrachyshka has quit IRC | 14:57 | |
| *** ihrachys has quit IRC | 14:57 | |
| regXboi | haleyb: I won't get to it until 60 minutes - have a meeting to run | 14:58 |
| regXboi | haleyb: will ping you then (earlier if I get lucky) | 14:58 |
| *** _amrith_ is now known as amrith | 15:00 | |
| regXboi | folks: that patched job just failed on router_reschedule :( | 15:00 |
| *** sergmelikyan has quit IRC | 15:00 | |
| regXboi | https://jenkins03.openstack.org/job/gate-tempest-dsvm-neutron-dvr/816/consoleFull | 15:00 |
| haleyb | regXboi: well, zuul is showing the gate queue many hours long, so it will be a while | 15:00 |
| regXboi | haleyb: I'm talking the check pipeline for 218242 | 15:01 |
| kevinbenton | regXboi: ugh, apparently that router isn't distributed | 15:01 |
| *** skolekonov has quit IRC | 15:02 | |
| regXboi | kevinbenton: I'd have to go verify with the code, but I think you may be right | 15:02 |
| *** julim has joined #openstack-infra | 15:02 | |
| *** skraynev has quit IRC | 15:03 | |
| *** mtanino has quit IRC | 15:03 | |
| *** hdd has joined #openstack-infra | 15:04 | |
| regXboi | kevinbenton: yeah, the code says it isn't a DVR | 15:04 |
| *** skraynev has joined #openstack-infra | 15:05 | |
| *** skolekonov has joined #openstack-infra | 15:05 | |
| *** tonytan4ever has joined #openstack-infra | 15:07 | |
| *** flepied1 has quit IRC | 15:07 | |
| sdague | kevinbenton: so... is it a configuration issue with the job setup? | 15:09 |
| *** dizquierdo has joined #openstack-infra | 15:09 | |
| *** ihrachyshka has joined #openstack-infra | 15:09 | |
| rcarrillocruz | hi folks, may i please get reviews for https://review.openstack.org/#/c/218121/ ? i need to make the job non voting, no changes can gate | 15:10 |
| *** pblaho has quit IRC | 15:10 | |
| kevinbenton | sdague: it may have actually found a bug with DVR and non-distributed routers | 15:10 |
| *** naggappan has quit IRC | 15:11 | |
| kevinbenton | sdague: i'm confirming | 15:11 |
| kevinbenton | sdague: if that's the case, then we just need to skip the job for now with a FIXME | 15:11 |
| kevinbenton | sdague: er, that test | 15:11 |
| kevinbenton | sdague: or revert the patch that recently added it | 15:11 |
| sdague | fungi: so, we're going to have to reenquee that tempest change, the large-ops job failed because of a timeout | 15:12 |
| *** armax has joined #openstack-infra | 15:12 | |
| sdague | because the node is *so* slow | 15:12 |
| sdague | or just merge the patch, because everything else passed on it | 15:12 |
| fungi | sdague: no problem, as long as it looks like the change is not at fault and otherwise does what is needed, i can just enqueue it into the gate pipeline directly | 15:13 |
| *** geoffarnold has joined #openstack-infra | 15:13 | |
| regXboi | kevinbenton: I think we should file a bug, skip that test with a refer to the bug number and then get the bug fixed | 15:13 |
| sdague | fungi: this is the one you already promoted | 15:13 |
| fungi | sdague: oh? | 15:13 |
| fungi | grr | 15:13 |
| sdague | fungi: 217804,3 | 15:13 |
| *** boris-42 has joined #openstack-infra | 15:13 | |
| fungi | thought you meant the one for dvr, but yeah i see that's still churning | 15:13 |
| *** zhenguo has quit IRC | 15:14 | |
| *** glauco has quit IRC | 15:14 | |
| amuller | kevinbenton: no need to revert the test addition, I'd much rather just skip the test entirely | 15:14 |
| kevinbenton | regXboi, amuller: doing that | 15:14 |
| amuller | kevinbenton++ | 15:14 |
| *** mase_x200 has joined #openstack-infra | 15:14 | |
| sdague | fungi: so, I'm sure that fix is valid, I'd suggest ninja merge, otherwise we're going to be babysitting stuff all day | 15:14 |
| regXboi | kevinbenton: thx | 15:14 |
| fungi | sdague: oh, i've got it back at the top of the gate again | 15:15 |
| sdague | fungi: yeh, it's just another hour delay for no real win, it's passed all the things that it should have fixed | 15:15 |
| *** sdake_ has quit IRC | 15:15 | |
| *** sdake has joined #openstack-infra | 15:16 | |
| *** amrith is now known as _amrith_ | 15:16 | |
| *** doude has quit IRC | 15:16 | |
| *** doude_ has joined #openstack-infra | 15:16 | |
| openstackgerrit | Pranav Salunke proposed openstack-infra/project-config: Create new project labs https://review.openstack.org/218345 | 15:16 |
| *** sputnik13 has joined #openstack-infra | 15:17 | |
| dougwig | morning, catching up. if we haven't already, can we make the two offending jobs non-voting while the other fixes merge? | 15:17 |
| fungi | dougwig: that's basically what's happening | 15:17 |
| amuller | dougwig: I'm summarizing this party on openstack-dev as we speak | 15:18 |
| dougwig | fungi: ok, any patches need pushing, or is it all being handled? in other words, what can i do to help? | 15:18 |
| amuller | dougwig: it's all done I think | 15:18 |
| *** skolekonov has quit IRC | 15:19 | |
| *** gabriel-bezerra is now known as gabrielb | 15:19 | |
| kevinbenton | sdague, mtreinish: https://review.openstack.org/#/c/218242/ | 15:19 |
| *** mase_x200 has quit IRC | 15:19 | |
| kevinbenton | just skip it for now while i figure out what is happening | 15:19 |
| sdague | kevinbenton: works for me | 15:19 |
| *** mase_x200 has joined #openstack-infra | 15:20 | |
| *** gabrielb is now known as gabriel-bezerra | 15:20 | |
| openstackgerrit | Merged openstack-infra/project-config: Remove Neutron functional job from the gate queue https://review.openstack.org/218302 | 15:20 |
| amuller | that's 1 out of 3 | 15:21 |
| fungi | amuller: well, it'll still be ~30 minutes before that takes effect | 15:21 |
| mtreinish | kevinbenton: I'm fine with the skip, but should probably use the skip_because decorator. One sec I'll respin it and just fast approve it after | 15:22 |
| *** gabriel-bezerra is now known as gbezerra | 15:22 | |
| *** gbezerra is now known as gabezerra | 15:22 | |
| *** gabezerra is now known as gbezerra_ | 15:23 | |
| *** gbezerra_ is now known as gabrielb | 15:23 | |
| *** enikanorov has quit IRC | 15:23 | |
| *** jaypipes has joined #openstack-infra | 15:23 | |
| *** gabrielb is now known as gabriel-bezerra | 15:23 | |
| *** jaypipes is now known as leakypipes | 15:24 | |
| dougwig | sdague: regarding your comments on not being on top of this, i think this wedge is mostly the py34 bug, which has had a fix working its way through since yesterday morning. we were specifically asked not to request ninja merges, so i'm not sure what more you wanted us to do there? we (neutron) also seem to set jobs voting with ~10-12% failure rates, which | 15:24 |
| dougwig | is way too high in my opinion. what's the norm there? | 15:24 |
| *** enikanorov has joined #openstack-infra | 15:25 | |
| anteaya | dougwig: sdague and I have talked | 15:25 |
| anteaya | I'll share some thoughts with you when you have a moment | 15:25 |
| anteaya | I have a few suggestions which we can discuss | 15:25 |
| *** bharathm has quit IRC | 15:25 | |
| *** sabeen has joined #openstack-infra | 15:25 | |
| sdague | dougwig: definitely not that high, if it's going to be in gate, < 5% for sure. | 15:25 |
| anteaya | and perhaps discuss a bit of a new schedule implimentation for infra/neutron liasions | 15:25 |
| yolanda | here, fungi, clarkb, jeblair... i'd like to have more feedback about the puppet-httpd issue. Are we ok on the short term goals, fixing with the creation of the httpd::mod type? | 15:26 |
| sdague | otherwise it impacts a bunch of other projects | 15:26 |
| *** enikanorov has quit IRC | 15:26 | |
| dougwig | sdague: good, i'll get that written into a policy on our side. | 15:26 |
| sdague | dougwig: right, so you can get fixes promoted when they are impacting things | 15:26 |
| *** dguitarbite has quit IRC | 15:26 | |
| dougwig | sdague: right, so one person's "impacting" isn't another's. so i need to figure out how to write that down in an objective manner so it's applied consistenly. something like >X% failures in gate or >N jobs stuck in gate due to neutron job, notify infra of review YYY. | 15:28 |
| *** yamamoto has joined #openstack-infra | 15:29 | |
| fungi | yolanda: i don't really have a preference, honestly. i was fine just fixing dependencies for those various bugs in the modules where they were arising, but others with stronger opinions disagreed so i'm just waiting to see what direction others are comfortable with | 15:29 |
| sdague | dougwig: so, rule of thumb, if the gate is > 5hrs deep, something is not great. That's definitely producing substantial impact to other projects progressing | 15:29 |
| *** afaranha has left #openstack-infra | 15:30 | |
| crinkle | SpamapS: https://review.openstack.org/#/c/209698 and https://review.openstack.org/#/c/213980/ are more-or-less ready to start being tested if you're at that point | 15:30 |
| sdague | anyway, I'm a bit grumpy that I had to go raise a red flag here, so it's probably better to talk about it next week instead | 15:30 |
| crinkle | SpamapS: and I'm afk till Tuesday so feel free to make change to those patches if you want | 15:31 |
| *** jeadre has quit IRC | 15:31 | |
| *** jeadre has joined #openstack-infra | 15:31 | |
| fungi | also "impact" is due a lot to compounding. if you have a job which only passes 95% of the time, that's a lot different from having 5 jobs which pass 95% of the time (because that gives your changes a one in three rate of failure) | 15:32 |
| *** hurgleburgler has left #openstack-infra | 15:32 | |
| *** puranamr has joined #openstack-infra | 15:32 | |
| fungi | er, more like one in four, but you get the idea | 15:32 |
| *** fawadk has quit IRC | 15:32 | |
| *** skraynev has quit IRC | 15:32 | |
| sdague | right, exactly. It's less of strict rules, and more about responsibility and keeping on top of these kinds of issues. | 15:32 |
| sdague | and actively trying to reduce them | 15:32 |
| fungi | 8 jobs passing 95% of the time give you a one in three failure rate | 15:33 |
| *** skylerberg has joined #openstack-infra | 15:33 | |
| *** enikanorov has joined #openstack-infra | 15:33 | |
| *** skraynev has joined #openstack-infra | 15:33 | |
| *** yamamoto has quit IRC | 15:34 | |
| *** sputnik13 has quit IRC | 15:35 | |
| *** jeadre has quit IRC | 15:36 | |
| mordred | sdague: ++ | 15:36 |
| *** jeadre has joined #openstack-infra | 15:36 | |
| *** trown is now known as trown|lunch | 15:37 | |
| dougwig | sdague, fungi: right, there are two issues here. one is how do we trend towards a sane set of jobs and keep them that way, and the other is how do we communicate to neutron folks when is the right time to make a job voting or not, and when to escalate something that is wedging everything. addressing the first makes the second irrelevant, but that's also not | 15:38 |
| dougwig | going to happen overnight. | 15:38 |
| fungi | dougwig: sure, o | 15:38 |
| fungi | er | 15:38 |
| zaro | morning | 15:38 |
| fungi | i'm not going to pretend that's easy | 15:38 |
| anteaya | morning zaro | 15:38 |
| *** sridhar_ram has joined #openstack-infra | 15:38 | |
| Clint | mordred: done conferencing? | 15:38 |
| mordred | Clint: yes. today will be plane-ing | 15:39 |
| fungi | dougwig: what tends to help the most is having strong connections between your core review team, such that you can get them to either switch to exclusively writing/reviewing/approving fixes to those sorts of bugs, or at the very least stop approving unrelated changes until the situation is back under control | 15:39 |
| *** chlong has quit IRC | 15:39 | |
| *** tjones has joined #openstack-infra | 15:39 | |
| *** skraynev has quit IRC | 15:40 | |
| Clint | mordred: fantastic, we can talk about nodepool-dib and maniphest | 15:40 |
| amuller | fungi: that comes at odd with the recent explosion of neutron cores TBH | 15:40 |
| anteaya | yes there is no substitute for trust | 15:41 |
| *** dangers_away is now known as dangers | 15:41 | |
| anteaya | amuller: perhaps that explosion is part of the problem? | 15:41 |
| anteaya | perhaps neutron is too big to manage itself | 15:41 |
| *** bdemers has quit IRC | 15:43 | |
| sdague | fungi: https://etherpad.openstack.org/p/gate-fire-2015-08-28 I'm going to stack up fixes for stuff causing gate resets here that could be promoted | 15:43 |
| fungi | might just be that their new core reviewers need more time to get familiar with those additional responsibilities and figure out more effective ways to share information and coordinate among each other | 15:43 |
| fungi | sdague: thanks! | 15:43 |
| mordred | Clint: I like talking about nodepool-dib and maniphest | 15:43 |
| dougwig | fungi: good thoughts. that's something we can address. | 15:44 |
| clarkb | not to derail the current fix the gate discussion but git cloning nova on centos7 seems to use far less memory than on centos6 | 15:44 |
| Clint | mordred: excellent | 15:44 |
| amuller | anteaya: I disagree with these general statements. I don't think that this is a situation we cannot remedy. I also recall times during the Icehouse cycle where the Neutron gate was broken for a month, so while "it's been worse" is not a great point, I think that generally we are improving. There's been a recent slip with the functional gate, and we will address that. | 15:44 |
| fungi | clarkb: ooh! that's a good sign. could improve our stability further | 15:44 |
| *** sputnik13 has joined #openstack-infra | 15:45 | |
| anteaya | amuller: well I have found no substitute for trust | 15:45 |
| fungi | clarkb: and possibly allow us to even scale back the cluster | 15:45 |
| jesusaurus | clarkb: is there a significant version bump in git across the centoses? | 15:45 |
| anteaya | if you have, than that is great for you | 15:45 |
| clarkb | fungi: yup, it is all looking pretty good so would be good to get a few of these nodes behind the load balancer | 15:45 |
| *** mase_x200 has quit IRC | 15:45 | |
| clarkb | jesusaurus: ya its 1.7.1 to 1.9.3 iirc | 15:45 |
| jesusaurus | awesome | 15:45 |
| clarkb | jesusaurus: 1.7 is ancient and 1.9 is relatively recent | 15:45 |
| jesusaurus | yeah, and there were some pretty buggy 1.7.x's as well, iirc | 15:46 |
| amuller | anteaya: I haven't, I've made public statements about some of the new cores. | 15:46 |
| amuller | Negative ones that is | 15:46 |
| sdague | I also noticed that horizon is using npm against npm package repos (non local mirrors) which fails kind of often | 15:46 |
| *** mtanino has joined #openstack-infra | 15:46 | |
| *** bdemers has joined #openstack-infra | 15:47 | |
| mordred | yeah - I think th einfra channel is th eplace we should focus on fixing th eproblem, not on thoughts on the social structure of neutron | 15:47 |
| sdague | or at least fails some | 15:47 |
| fungi | sdague: yes, i didn't count, but it appears to cause a hundred (maybe more) downloads | 15:47 |
| anteaya | amuller: you are a new core yourself, if I follow what folks I trust in neutron tell me | 15:47 |
| amuller | anteaya: Indeed I am | 15:47 |
| clarkb | huh I thought horizon was using xstatic instead of npm | 15:47 |
| *** mase_x200 has joined #openstack-infra | 15:47 | |
| sdague | it has some npm tests | 15:47 |
| sdague | gate-horizon-npm-run-test: SUCCESS | 15:47 |
| sdague | gate-horizon-npm-run-lint: | 15:47 |
| fungi | i think this was a lot of the impetus for figuring out how to mirror npm | 15:47 |
| clarkb | greghaynes: yes the two experimental jobs passed and on the node I checked it had a large / | 15:47 |
| sdague | those hit npm servers | 15:48 |
| *** vivekd has joined #openstack-infra | 15:48 | |
| clarkb | gotcha | 15:48 |
| clarkb | so its specifically testing things and not the installation of a working horizon | 15:48 |
| kevinbenton | regXboi, mtreinish, amuller: found the issue with the skip. the self.router object was a tenant's view of the router which doesn't include the distributed flag | 15:49 |
| sdague | clarkb: correct | 15:49 |
| amuller | kevinbenton: nice one | 15:49 |
| kevinbenton | https://review.openstack.org/218367 | 15:49 |
| regXboi | kevinbenton: nice catch | 15:49 |
| amuller | kevinbenton: I wonder though if it isn't possible to skip based off a configuration (Of Tempest or Neutron) instead of inspecting the router object? | 15:50 |
| *** skraynev has joined #openstack-infra | 15:50 | |
| mtreinish | amuller: it would have to be tempest configuration | 15:50 |
| sdague | kevinbenton: so, can you respin that patch to not have such a mega line :) | 15:50 |
| sdague | some intermediate variables are fine | 15:51 |
| kevinbenton | but those cost money :) | 15:51 |
| mtreinish | amuller: the only config option that is dvr specific is the api extension | 15:51 |
| sdague | honestly, I like kevinbenton's approach, because it should be definitive | 15:51 |
| kevinbenton | amuller: yeah, we chatted about that earlier, but then the non-dvr job wouldn't be able to use 'all' | 15:51 |
| kevinbenton | amuller: for the extensions | 15:51 |
| *** mrmartin has quit IRC | 15:52 | |
| sdague | and adding more tempest config is another thing where someone can misconfigure things and break for reasons they don't understand | 15:52 |
| kevinbenton | amuller: and dvr isn't really an extension... | 15:52 |
| sdague | I guess the only issue is are we guarunteed to always have admin authority? | 15:52 |
| sdague | mtreinish: ^^ | 15:52 |
| openstackgerrit | Clark Boylan proposed openstack-infra/puppet-cgit: Use systemd'd git-daemon on Centos7 https://review.openstack.org/217956 | 15:52 |
| sdague | I guess we use it other places in that test | 15:53 |
| mtreinish | sdague: yeah all the neutron scenarios are allocating admin creds for themselves: http://git.openstack.org/cgit/openstack/tempest/tree/tempest/scenario/manager.py#n549 | 15:53 |
| amuller | kevinbenton: a skipIf decorator is easily copy/pastable for new tests, I'm afraid that authors won't notice (And nor will reviewers know) that other router scheduling tests may need to be skipped in the DVR case. | 15:54 |
| mtreinish | well it's really L560 that sets that | 15:54 |
| *** john-davidge_ has quit IRC | 15:54 | |
| *** sdake_ has joined #openstack-infra | 15:55 | |
| kevinbenton | amuller: can't do it that soon. have to wait until a router is created. we could create one pre-emptively in the setup and skip like dvr tests do | 15:55 |
| *** ihrachys_ is now known as ihrachys | 15:56 | |
| amuller | kevinbenton: you could do that if you split that test to a different test case dedicated to router scheduling | 15:57 |
| *** puranamr has quit IRC | 15:57 | |
| SpamapS | crinkle: Thanks! | 15:58 |
| SpamapS | greghaynes: so, I fear I won't be able to make much progress on things today. How far did you get with the two-dnsmasq approach? | 15:59 |
| *** fitoduarte has joined #openstack-infra | 15:59 | |
| *** yfried_ has quit IRC | 15:59 | |
| *** sdake has quit IRC | 15:59 | |
| kevinbenton | amuller: looks like it will require a bit of shuffling things around into a base class for the setup helpers and another for the tests | 16:00 |
| *** Swami has joined #openstack-infra | 16:01 | |
| amuller | kevinbenton: maybe you can get yfried to do that :) | 16:01 |
| clarkb | fungi: https://review.openstack.org/#/c/188357/ and the stack at https://review.openstack.org/217956 are the changes we need for centos7 git backends | 16:01 |
| *** jordanP has quit IRC | 16:01 | |
| clarkb | I have noop applied them on centos6 and they look ok there as well | 16:01 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config: Upgrade etherpad-dev to Ubuntu Trusty https://review.openstack.org/215243 | 16:03 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config: Upgrade etherpad to Ubuntu Trusty https://review.openstack.org/215249 | 16:03 |
| *** Ala has quit IRC | 16:03 | |
| fungi | clarkb: awesome, and there's ^ the etherpad upgrade changes no longer depending on the race fixes for puppet-httpd | 16:03 |
| fungi | so that we can decouple those projects for now | 16:04 |
| clarkb | sounds good will review those shortly | 16:04 |
| fungi | same here | 16:04 |
| *** hashar has quit IRC | 16:04 | |
| clarkb | pleia2: if you are still around, should I/you/we be planning on spinning up the prod zanata server today? | 16:04 |
| *** _amrith_ is now known as amrith | 16:04 | |
| *** EricGonczer_ has joined #openstack-infra | 16:05 | |
| *** glauco has joined #openstack-infra | 16:05 | |
| *** mrmartin has joined #openstack-infra | 16:06 | |
| *** ashleighfarnham has joined #openstack-infra | 16:06 | |
| clarkb | fungi: the change to puppet-etherpad_lite must've merged then? | 16:07 |
| clarkb | git log says yes \o/ | 16:07 |
| sdague | man, why is paste.openstack.org so slow .... :( | 16:07 |
| anteaya | fungi: 215243 and 49 have almost the exact same commit message, what is the difference between them? | 16:07 |
| fungi | anteaya: the almost part | 16:07 |
| fungi | anteaya: specifically the -dev | 16:08 |
| anteaya | ah thanks | 16:08 |
| clarkb | fungi: for the change to the prod etherpad, you should update the etherpad ref to match the one that ends up being tested on -dev | 16:09 |
| clarkb | fungi: that way we keep them in sync and have known working etherpad install on prod based on dev | 16:09 |
| fungi | yolanda: clarkb: i am concerned that our puppet-httpd race on modules might crash apache on our git servers when 199798 merges. has anyone manually confirmed it's working on centos6 on the first application? | 16:09 |
| fungi | clarkb: can do | 16:09 |
| clarkb | fungi: I will leave a ocmment about that too | 16:09 |
| fungi | clarkb: thanks | 16:09 |
| clarkb | fungi: I have noop applied the child of 199798 to centos6 git08 and it seemed fine | 16:10 |
| *** skraynev has quit IRC | 16:10 | |
| *** dtantsur is now known as dtantsur|afk | 16:10 | |
| clarkb | fungi: I don't think the puppet httpd race should be a problem since we have working apache there | 16:11 |
| *** sergmelikyan has joined #openstack-infra | 16:11 | |
| clarkb | fungi: so either order should be fine | 16:11 |
| fungi | k | 16:11 |
| *** leakypipes has quit IRC | 16:11 | |
| fungi | mostly just worried if it tries to reload apache with the new vhost config before enabling the module | 16:11 |
| clarkb | fungi: on centos all the modules are enabled by default via ackage install | 16:11 |
| clarkb | so enabling modules is really a noop there | 16:12 |
| fungi | ahh, and in theory the package install will complete before the vhost is updated | 16:12 |
| clarkb | ya | 16:12 |
| yolanda | clarkb was testing that yesterday right? i have this working on centos6 and centos7, but seemed that my local patch was a bit different than the one i sent upstream | 16:12 |
| *** bharathm has joined #openstack-infra | 16:12 | |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck: Add query for cinder py27 sentinel bug 1488690 https://review.openstack.org/218378 | 16:12 |
| openstack | bug 1488690 in Cinder "test_create_backing_from_stream_optimized_file fails with "AttributeError: '_SentinelObject' object has no attribute 'readlines'"" [Undecided,In progress] https://launchpad.net/bugs/1488690 - Assigned to Jon Bernard (jbernard) | 16:12 |
| greghaynes | SpamapS: Much, I am battling a bug where I cant update configdrives, but I think once I get past that it should 'just work' | 16:13 |
| *** kiran-r has joined #openstack-infra | 16:13 | |
| *** ramashri has joined #openstack-infra | 16:13 | |
| clarkb | yolanda: I was predominantly testing against centos7, I did do a noop puppet run agains tcentos6 | 16:13 |
| greghaynes | SpamapS: theres two dnsmasq's running on bastion01 now, one of them I kicked off manually | 16:13 |
| fungi | sdague: looks like 217804,3 merged, so we're set for me to promote the next three in the etherpad, right? | 16:13 |
| sdague | fungi: yeh | 16:14 |
| *** ramashri has quit IRC | 16:14 | |
| sdague | I think in the order listed | 16:14 |
| pleia2 | clarkb: worth handling this -1? https://review.openstack.org/#/c/217787/4/modules/openstack_project/manifests/translate.pp | 16:14 |
| pleia2 | clarkb: and we need an ssl certificate created, not sure who has those powers these days | 16:14 |
| fungi | sdague: yep, looks like the most likely order to succeed | 16:14 |
| *** ramashri has joined #openstack-infra | 16:14 | |
| sdague | yeh, the cinder top of queue patch after the tempest one probably is worth letting finish | 16:15 |
| clarkb | pleia2: fungi typically does them, I can work with him to sort that out | 16:15 |
| sdague | as it's close | 16:15 |
| *** sergmelikyan has quit IRC | 16:15 | |
| pleia2 | clarkb: ok, I think other than those two things we're ready once this patch lands | 16:15 |
| clarkb | pleia2: oh ya we should probably go ahead and bubble server up into the node {} def | 16:15 |
| fungi | sdague: fine with me. i'm building up the command in the meantime | 16:15 |
| clarkb | pleia2: do you have time to do that or should I take over the change? | 16:15 |
| sdague | cool, I'll keep an eye on it and poke you when it's ready | 16:15 |
| fungi | pleia2: clarkb: i'm happy to order the cert today if you're ready for it | 16:15 |
| clarkb | fungi: I think we are would be for translate.openstack.org | 16:16 |
| pleia2 | clarkb: I haven't been paying too close attention to our work on that spec, so I don't actually know what the change needs to be (and the comment doesn't help much) | 16:16 |
| pleia2 | clarkb: is it just moving the parameters over to the site.pp? | 16:16 |
| clarkb | pleia2: if you look in manifests/site.pp you will ifnd that openstack_project::server is in the node def | 16:16 |
| pleia2 | clarkb: ok, I'll whip up a change and see if I got it right | 16:17 |
| clarkb | pleia2: instead of being nested into the application specific manifest | 16:17 |
| pleia2 | right, got it | 16:17 |
| yolanda | clarkb, were you able to spin git-daemon on selinux enforcing? | 16:17 |
| *** dannywilson has joined #openstack-infra | 16:17 | |
| clarkb | yolanda: yes see https://review.openstack.org/217956 | 16:17 |
| yolanda | oh cool | 16:18 |
| clarkb | yolanda: the problem before is git_system_t did not have access to git_port_t because systemd is expected to do socket activation | 16:18 |
| clarkb | yolanda: so I set it up to use systemd socket activation on centos7 | 16:18 |
| *** claudiub has quit IRC | 16:18 | |
| *** unicell has quit IRC | 16:18 | |
| yolanda | reviewing that,this will allow us to use selinux enforcing | 16:19 |
| fungi | clarkb: curious why you inverted the conditional match for the initscript/unit file | 16:20 |
| openstackgerrit | Merged openstack-infra/system-config: Start using jfryman/puppet-selinux https://review.openstack.org/188357 | 16:20 |
| *** shayneburgess has joined #openstack-infra | 16:20 | |
| clarkb | fungi: I think becuase I started with if redhat and release > 7 or not redhat then realized that was silly and simplified the logic instead of flipping the blocks around | 16:21 |
| fungi | clarkb: as in why you used "if !a||!b then x else y" instead of "if a&&b then y else x" | 16:21 |
| clarkb | fungi: I can flip the blocks around if you will find that more readable | 16:21 |
| mriedem | clarkb: my favorite https://review.openstack.org/#/c/218380/ | 16:21 |
| fungi | clarkb: not a big deal, just making sure it wasn't hiding something more subtle there | 16:22 |
| *** chlong has joined #openstack-infra | 16:22 | |
| clarkb | fungi: there is one subtle thing, the maj release check only applies to redhat | 16:22 |
| yolanda | clarkb, semana for port 9418/29418 worked fine for you ? i tried to run that from command line at some point, and i was getting an error,about port already exisiting on that rules | 16:23 |
| *** skraynev has joined #openstack-infra | 16:23 | |
| clarkb | yolanda: yes see selinux.pp's update | 16:23 |
| yolanda | clarkb, yes, i tried the same at some point, i had a chance for that, but i abandoned, i was hitting that duplication | 16:23 |
| yolanda | maybe it was for some old run | 16:23 |
| clarkb | yolanda: the issue is that git_system_t did not have access to git_port_t | 16:24 |
| fungi | clarkb: well, "if ($::osfamily == 'RedHat' and $::operatingsystemmajrelease >= '7') {" would match the same things as long as you reversed the conditional order | 16:24 |
| *** yamamoto has joined #openstack-infra | 16:24 | |
| clarkb | yolanda: so if you used the old init script or ran git-daemon direclty it didn't have access to any git_port_t whether 9418 or 29418 | 16:24 |
| clarkb | fungi: right I could flip the blocks around, why don't I do that | 16:24 |
| clarkb | yolanda: so the real trick there is to use systemd socket activation | 16:25 |
| fungi | clarkb: i don't think it's necessary, but up to you | 16:25 |
| *** kiran-r has quit IRC | 16:25 | |
| clarkb | yolanda: and just make sure that whatever port you use is in git_port_t | 16:25 |
| yolanda | clarkb, much appreciated, we were struggling with that | 16:25 |
| openstackgerrit | Jesse Pretorius proposed openstack-infra/project-config: Rename stackforge/os-ansible-deployment{,specs} https://review.openstack.org/200730 | 16:26 |
| yolanda | clarkb, there is also that problem with iptables vs firewall on centos 7, did you hit that issue? | 16:26 |
| SpamapS | greghaynes: should be easy enough to turn the manually kicked off one into an upstart job and be done with it. | 16:26 |
| *** fawadkhaliq has joined #openstack-infra | 16:26 | |
| greghaynes | SpamapS: Yep | 16:27 |
| clarkb | yolanda: yes I fixed it too in puppet-iptables, fix already merged | 16:27 |
| SpamapS | greghaynes: ok, I don't have much I can add right now... | 16:27 |
| yolanda | clarkb nice | 16:27 |
| openstackgerrit | Elizabeth K. Joseph proposed openstack-infra/system-config: Add manifest for translate.openstack.org https://review.openstack.org/217787 | 16:27 |
| yolanda | we should update that from our side | 16:27 |
| pleia2 | yolanda: does this address your concern? ^^ | 16:27 |
| greghaynes | SpamapS: Yea, its going fine... just need to figure out why these config drives arent updating :( | 16:27 |
| yolanda | pleia2, sure | 16:28 |
| greghaynes | TheJulia: Any protips on where I should look for when the config drive is added to the disk? | 16:28 |
| *** e0ne has quit IRC | 16:28 | |
| *** samuelBartel has quit IRC | 16:28 | |
| greghaynes | TheJulia: I mounted the one in /httpboot, and it looks like I expect it which is not what im getting when I redeploy the node | 16:28 |
| fungi | clarkb: why is the selinux rule for the git daemon port subscribing to httpd.conf? | 16:29 |
| *** doug-fish has joined #openstack-infra | 16:29 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/puppet-cgit: Use systemd'd git-daemon on Centos7 https://review.openstack.org/217956 | 16:29 |
| clarkb | fungi: is it? thats a bug | 16:29 |
| clarkb | let me fix | 16:29 |
| yolanda | pleia2, i see that require => [ class['openstack_project::server'] inside translate.pp . If we decouple it, that require should be in the upper manifests/site.pp , when you call the openstack_project::translate | 16:30 |
| clarkb | fungi: bad copy pasta | 16:30 |
| fungi | figured | 16:30 |
| *** doug-fish has quit IRC | 16:30 | |
| openstackgerrit | Claudiu Belu proposed openstack-infra/project-config: Adds os-win repository https://review.openstack.org/209204 | 16:30 |
| *** annegentle has quit IRC | 16:31 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/puppet-cgit: Use systemd'd git-daemon on Centos7 https://review.openstack.org/217956 | 16:31 |
| clarkb | fungi: ^ that should be better | 16:31 |
| *** bmjen has left #openstack-infra | 16:32 | |
| pleia2 | yolanda: so the site.pp should have a ::zanata section too? | 16:32 |
| openstackgerrit | Merged openstack-infra/puppet-cgit: New Updated config for CentOS 7 and Apache 2.4 https://review.openstack.org/199798 | 16:32 |
| yolanda | mm, no, let me add that on the review, should be easier to explain | 16:32 |
| pleia2 | thanks | 16:32 |
| sdague | man, this stack of ipv6 tests that take 100s each are really killing me | 16:33 |
| sdague | fungi: still waiting for the neutron job to finish on that cinder patch | 16:33 |
| *** claudiub has joined #openstack-infra | 16:33 | |
| yolanda | pleia2, review sent | 16:34 |
| sdague | mtreinish: are we seeding testr here ? I'm surprised these are all stacking up in one worker | 16:34 |
| *** tonytan4ever has quit IRC | 16:34 | |
| sdague | it looks like all the rest of the workers are done https://jenkins02.openstack.org/job/gate-tempest-dsvm-neutron-full/11201/consoleFull and now we're going to spend 10 minutes in the last worker running these tests | 16:34 |
| pleia2 | yolanda: I'm confused, that doesn't seem to be the case on other ones in site.pp, looking at eavesdrop, paste, planet... | 16:35 |
| pleia2 | yolanda: why is this one different? | 16:35 |
| sdague | oh... we're only splitting by class. So these will always be in one worker together. | 16:35 |
| *** jistr has quit IRC | 16:35 | |
| haleyb | sdague: which stack of ipv6 tests? | 16:35 |
| yolanda | pleia2, it raised my attention, maybe because i'm reviewing that from scratch, and the others were refactors | 16:36 |
| sdague | haleyb: | 16:36 |
| sdague | 2015-08-28 16:29:28.379 | {0} tempest.scenario.test_network_v6.TestGettingAddress.test_dualnet_multi_prefix_dhcpv6_stateless [119.106058s] ... ok | 16:36 |
| sdague | 2015-08-28 16:31:09.336 | {0} tempest.scenario.test_network_v6.TestGettingAddress.test_dualnet_multi_prefix_slaac [100.943439s] ... ok | 16:36 |
| sdague | 2015-08-28 16:32:42.386 | {0} tempest.scenario.test_network_v6.TestGettingAddress.test_dualnet_slaac_from_os [93.035435s] ... ok | 16:36 |
| sdague | 2015-08-28 16:34:09.734 | {0} tempest.scenario.test_network_v6.TestGettingAddress.test_multi_prefix_dhcpv6_stateless [87.344107s] ... ok | 16:36 |
| pleia2 | yolanda: I don't understand | 16:36 |
| *** derekh has quit IRC | 16:36 | |
| sdague | haleyb: and there are a few others iirc | 16:36 |
| *** salv-orlando has quit IRC | 16:36 | |
| sdague | test_network_v6 has a ton of tests which start 2 guests | 16:36 |
| *** austin81 has joined #openstack-infra | 16:36 | |
| yolanda | pleia2, i mean, i'm reviewing that manifest from the start | 16:36 |
| sdague | in serial | 16:36 |
| yolanda | and for the others, i didn't review, or they were already like that | 16:37 |
| *** mrunge has quit IRC | 16:37 | |
| sdague | haleyb: you can see - https://jenkins02.openstack.org/job/gate-tempest-dsvm-neutron-full/11201/consoleFull running them at the moment | 16:37 |
| pleia2 | yolanda: ok, what I'm asking is why this manifest requires it and the other ones don't | 16:37 |
| *** tqtran_ has joined #openstack-infra | 16:37 | |
| *** esker has joined #openstack-infra | 16:37 | |
| yolanda | pleia2, so others may require that as well, but i didn't have a chance to review, if i had seen that, i'd have said the same | 16:38 |
| sdague | haleyb: I think there are 8 tests that do this, they all run > 1 minute (sometimes > 2 minutes), so they are probably worth looking into potentially optimizing | 16:38 |
| pleia2 | yolanda: happy to remove it from translate.pp, but I don't understand what I'm supposed to put in the site.pp | 16:38 |
| pleia2 | yolanda: but they seem to be running on... | 16:38 |
| pleia2 | s/on/ok | 16:38 |
| clarkb | sdague: odd that they need 2 VMs | 16:38 |
| *** mriedem is now known as mriedem_lunch | 16:38 | |
| *** MarkAtwood has joined #openstack-infra | 16:38 | |
| yolanda | pleia2, removing from the translate.pp and not adding in site.pp is fine | 16:38 |
| *** esker has quit IRC | 16:38 | |
| *** yamahata has quit IRC | 16:38 | |
| sdague | clarkb: they test ipv6 between the guests | 16:38 |
| clarkb | should be sufficient to check IPv6 details against a single VM then update the inter VM connectivity test to do ipv6 checks too in that one instance | 16:38 |
| clarkb | sdague: I know but you only have to do that once | 16:39 |
| clarkb | sdague: and there is an existing test that sinks that cost | 16:39 |
| yolanda | in case you needed the require, it's better done on site.pp, but if you don't need the require for anything, you can drop it | 16:39 |
| sdague | clarkb: yeh, probably | 16:39 |
| *** esker has joined #openstack-infra | 16:39 | |
| sdague | like I said, this is ripe for optimization for anyone interested | 16:39 |
| sdague | I only noticed this yesterday | 16:39 |
| haleyb | sdague: so each one boots two vms, that's pretty bad | 16:39 |
| sdague | haleyb: yep | 16:39 |
| openstackgerrit | Elizabeth K. Joseph proposed openstack-infra/system-config: Add manifest for translate.openstack.org https://review.openstack.org/217787 | 16:39 |
| sdague | I think it's copy / paste from a base pattern | 16:39 |
| fungi | sdague: been keeping it open in a separate tab. looks like it's wrapping up now | 16:39 |
| sdague | fungi: yep, uploading logs now | 16:40 |
| *** annegentle has joined #openstack-infra | 16:40 | |
| sdague | haleyb: if you wanted to take a stab on optimizing, I'm sure many people would love you | 16:40 |
| fungi | promoting 218242,4 217882,1 217379,6 as soon as those two changes at the top of the gate report | 16:40 |
| *** earlephilhower has joined #openstack-infra | 16:40 | |
| yolanda | pleia2 , +1 | 16:40 |
| *** mase_x200 has quit IRC | 16:41 | |
| pleia2 | yolanda: thanks | 16:41 |
| *** dprince has quit IRC | 16:41 | |
| sdague | fungi: cool | 16:41 |
| sdague | also, static.openstack.org got restarted? | 16:41 |
| sdague | https://jenkins04.openstack.org/job/gate-neutron-lbaas-docs/197/console | 16:41 |
| sdague | or something else happened to make that docs job fail. It will get a second chance because of the promote | 16:42 |
| haleyb | amuller: ^^ on the tempest ipv6 tests as an FYI. Not trying to throw it at you but this way one of us will remember. I can file a bug | 16:42 |
| fungi | sdague: up 59 days | 16:42 |
| clarkb | and disks are not full | 16:42 |
| clarkb | thats probabl more likely a network fail | 16:42 |
| sdague | yeh, I meant apache restarting sometimes triggers a fail like that | 16:42 |
| amuller | haleyb: a bug would be a good start | 16:43 |
| clarkb | sdague: that shouldn't touch apache at all, its just copying files to disk over an scp connection | 16:43 |
| sdague | fungi: ok, should be good to go | 16:43 |
| clarkb | sshd has been running since the 19th | 16:43 |
| sdague | clarkb: oh, over scp | 16:43 |
| sdague | ok, never mind | 16:43 |
| fungi | yep, just realized i needed to enqueue 218242,4 into the gate first | 16:43 |
| fungi | and zuul's still chewing through its events/results for a minute | 16:44 |
| sdague | fungi: also, there are 2 fails in the gate further down which are fixed by 2 of the things being promoted | 16:44 |
| fungi | nice | 16:44 |
| sdague | so, I'm hopefully this will grease the wheels quite a bit | 16:44 |
| *** chlong has quit IRC | 16:45 | |
| fungi | grr, 217379,6 wasn't in the gate yet either | 16:45 |
| sdague | fungi: yeh, well, delays :) | 16:45 |
| fungi | the grr was at me for not checking and enqueuing them earlier | 16:45 |
| *** unicell has joined #openstack-infra | 16:46 | |
| fungi | anyway, all three are at the top of the gate now | 16:46 |
| mtreinish | sdague: yes we should be pre seeding testr, but it only has a small effect | 16:46 |
| *** devvesa has quit IRC | 16:46 | |
| mtreinish | also it uses the 10 most recent samples from the db when nodepool is made | 16:46 |
| mtreinish | so we might end up missing something | 16:46 |
| *** amitgandhinz has quit IRC | 16:46 | |
| sdague | mtreinish: gotcha | 16:46 |
| *** tiswanso has joined #openstack-infra | 16:47 | |
| fungi | sdague: clarkb: also no exciting filesystem-related errors in dmesg | 16:47 |
| mtreinish | sdague: did you have a run you were looking at? | 16:47 |
| sdague | mtreinish: https://jenkins02.openstack.org/job/gate-tempest-dsvm-neutron-full/11201/consoleFull | 16:47 |
| *** cdent has quit IRC | 16:47 | |
| sdague | mtreinish: the last 10 minutes of the run was spent in 1 worker | 16:48 |
| mtreinish | sdague: http://logs.openstack.org/15/217615/5/gate/gate-tempest-dsvm-neutron-full/5b4875e//console.html#_2015-08-28_15_52_22_480 | 16:48 |
| *** thiagop_ has quit IRC | 16:48 | |
| sdague | mtreinish: which test streams are those? | 16:48 |
| sdague | because the list of tests run for neutron is much different than non neutron | 16:49 |
| sdague | and how are skips accounted for | 16:49 |
| mtreinish | sdague: it's an imperfect mechanism that doesnt do much, but I was limited by testr | 16:49 |
| mtreinish | sdague: it's whatever the last 10 added to the db were when nodepool ran | 16:49 |
| *** jogo is now known as flashgordon | 16:49 | |
| sdague | mostly, I wonder if because these tests never get accounted for because the config is different. | 16:50 |
| *** hdd has quit IRC | 16:50 | |
| *** tiswanso has quit IRC | 16:50 | |
| mtreinish | then the test time for the last loaded run of a test is what is used by the scheduler | 16:50 |
| *** tiswanso has joined #openstack-infra | 16:51 | |
| mtreinish | sdague: it might be, it also could be the grouping throwing things off a bit too | 16:51 |
| *** pradk has quit IRC | 16:52 | |
| TheJulia | greghaynes: sounds like a bug when built with the dib element instead of coreos, if you can file a bug with the details, I can investigate and try to reproduce | 16:52 |
| *** tjones has left #openstack-infra | 16:52 | |
| *** dzamboni has quit IRC | 16:52 | |
| openstackgerrit | Fausto Marzi proposed openstack-infra/project-config: Splitting stackforge/freezer on multiple repos https://review.openstack.org/193125 | 16:52 |
| *** sputnik13 has quit IRC | 16:53 | |
| mtreinish | sdague: that slow worker at the end was all the v6 scenarios and those can't be split up at all | 16:53 |
| sdague | mtreinish: sure | 16:53 |
| sdague | but I wonder if it's got bad data so it assumes those are all cheap | 16:53 |
| sdague | I wouldn't expect them to be split, I'd just expect that worker to have a lot less stuff in it | 16:53 |
| *** enikanorov has quit IRC | 16:56 | |
| haleyb | sdague: so i just thought of something - all these tests are in parallel, so this isn't as easy as i first imagined | 16:57 |
| *** enikanorov has joined #openstack-infra | 16:57 | |
| mtreinish | sdague: the algorithm just sorts the list of all groups based on time and then goes through them putting them in different workers: https://github.com/testing-cabal/testrepository/blob/master/testrepository/testcommand.py#L432 | 16:58 |
| *** dprince has joined #openstack-infra | 16:59 | |
| sdague | mtreinish: right, I'm just surprised it was given so much other expensive work | 16:59 |
| *** mtanino has quit IRC | 17:00 | |
| openstackgerrit | Igor Shishkin proposed openstack-infra/lodgeit: Turning NullPool to avoid MySQL server gone away https://review.openstack.org/218224 | 17:01 |
| mtreinish | sdague: I think it's just that 1 test class takes >10mins to execute that it throws off whatever is going to get it | 17:02 |
| mtreinish | I don't think any other group comes that close | 17:02 |
| mtreinish | *comes close to 10min | 17:02 |
| *** davidjc has joined #openstack-infra | 17:03 | |
| mtreinish | haleyb: tests are serialized in a class, the test classes run in parallel to each other | 17:03 |
| *** ifarkas has quit IRC | 17:03 | |
| *** austin81 has left #openstack-infra | 17:03 | |
| *** jerryz has joined #openstack-infra | 17:04 | |
| haleyb | sdague: ok, thanks for the info | 17:04 |
| mtreinish | sdague: but I'll push up a test patch again to extract the times.dbm file from a run that'll at least show us what testr has | 17:04 |
| *** ihrachys has quit IRC | 17:04 | |
| sdague | mtreinish: yeh, that would be useful, we can at least see | 17:05 |
| regXboi | kevinbenton, haleyb: sorry, I'm back now - where are we with respect to the rescheduling patch? | 17:05 |
| sdague | this worker was given an expensive testbasic ops group, as well as live migration (which was skipped), so knowing what it thinks it's feeding it would be goood | 17:05 |
| *** rossella_s has quit IRC | 17:05 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/devstack-gate: DO NOT MERGE COPY ALL OF .TESTREPOSITORY to log server https://review.openstack.org/149719 | 17:05 |
| *** julim has quit IRC | 17:06 | |
| *** rossella_s has joined #openstack-infra | 17:06 | |
| mtreinish | sdague: ^^^ | 17:07 |
| mtreinish | sdague: yeah, that kinda thing it doesn't acount for, it just goes 1 for you, next to you, etc. So after 4 it's gonna get a slow group again. | 17:07 |
| *** amitgandhinz has joined #openstack-infra | 17:07 | |
| sdague | mtreinish: I thought it did the handout by cost though | 17:08 |
| *** dizquierdo has quit IRC | 17:08 | |
| *** mjturek1 has quit IRC | 17:08 | |
| sdague | oh, wait, it round robins by cost? It doesn't pack by cost? | 17:08 |
| openstackgerrit | Grant Murphy proposed openstack-infra/jeepyb: Make SMTP connection configurable https://review.openstack.org/214733 | 17:08 |
| openstackgerrit | Igor Shishkin proposed openstack-infra/lodgeit: Turning NullPool to avoid MySQL server gone away https://review.openstack.org/218224 | 17:09 |
| *** julim has joined #openstack-infra | 17:09 | |
| mtreinish | sdague: I could be reading the code wrong, but look at that link I pasted before | 17:09 |
| mtreinish | that's the method which is doing the paritioning for timed groups | 17:09 |
| sdague | yeh, honestly, this code is too dense for me, especially on a friday. | 17:10 |
| haleyb | regXboi: i see it in the change queue, showing 1:45 remaining | 17:10 |
| *** erhudy has joined #openstack-infra | 17:10 | |
| *** madhu_ak has joined #openstack-infra | 17:11 | |
| regXboi | haleyb: yeah found it - sitting on the dvr job console output now | 17:11 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/devstack-gate: DO NOT MERGE COPY ALL OF .TESTREPOSITORY to log server https://review.openstack.org/149719 | 17:11 |
| *** trown|lunch is now known as trown | 17:11 | |
| *** salv-orlando has joined #openstack-infra | 17:12 | |
| *** ivar-lazzaro has joined #openstack-infra | 17:14 | |
| pleia2 | clarkb: https://review.openstack.org/#/c/217787 has jenkins +1, if you and another core want to land it I can launch the server in a couple hours (need to run to the vet now with my cat, because right I leave for a trip is obviously the best time to get sick! ugh, cats) | 17:14 |
| *** samueldmq has joined #openstack-infra | 17:14 | |
| mtreinish | sdague: heh, fair enough. Actually I was conflating it with the round robin for un timed things | 17:14 |
| *** yamahata has joined #openstack-infra | 17:14 | |
| clarkb | ok will review in a bit my hands are a bit full at home currently too | 17:14 |
| fungi | i guess i need to order the ssl cert now in that case. will have it shortly | 17:15 |
| mtreinish | sdague: it goes through the list of groups sorted by time and inserts the group into the partition with the shortest time | 17:15 |
| * pleia2 nods | 17:15 | |
| fungi | pleia2: the first part of my morning was also spent taking our cat to the vet on short notice. fun times! | 17:16 |
| openstackgerrit | Merged openstack/diskimage-builder: Fix ironic-image pkg-map https://review.openstack.org/217129 | 17:17 |
| mtreinish | sdague: it is kinda dense because its sorting the list of lists to put the parition with quickest in the first entry | 17:17 |
| pleia2 | fungi: oof, hope all is well on your end! | 17:17 |
| *** davidjc has quit IRC | 17:18 | |
| *** e0ne has joined #openstack-infra | 17:18 | |
| kevinbenton | regXboi: two patches now. one to just skip it to stop the bleeding | 17:18 |
| *** kfox1111 has joined #openstack-infra | 17:19 | |
| kevinbenton | regXboi: https://review.openstack.org/#/c/218242/4 | 17:19 |
| kevinbenton | regXboi: second fixes the logic i was trying to implement before https://review.openstack.org/#/c/218367/ | 17:19 |
| *** sdake_ is now known as sdake | 17:19 | |
| fungi | pleia2: yeah, turned out to be a relatively minor procedure and the cat is resting comfortably now | 17:19 |
| pleia2 | glad to hear it | 17:20 |
| *** ihrachys has joined #openstack-infra | 17:21 | |
| *** sridhar_ram has quit IRC | 17:21 | |
| *** wendar_ is now known as wendar | 17:23 | |
| regXboi | kevinbenton: I saw the first one | 17:25 |
| regXboi | oh lord | 17:26 |
| regXboi | the first one is failing on postgress-full-{kilo/juno} | 17:26 |
| *** ivar-lazzaro has quit IRC | 17:27 | |
| *** sputnik13 has joined #openstack-infra | 17:28 | |
| *** ivar-lazzaro has joined #openstack-infra | 17:28 | |
| *** sridhar_ram has joined #openstack-infra | 17:29 | |
| regXboi | kevinbenton: on 218242: ^^^^^^ | 17:30 |
| regXboi | :( | 17:30 |
| *** sdake_ has joined #openstack-infra | 17:30 | |
| kevinbenton | regXboi: that's the check queue | 17:31 |
| sdague | regXboi: we already jumped it to the gate so it shouldn't matter | 17:31 |
| kevinbenton | regXboi: it's been expressed to the front of the gate | 17:31 |
| regXboi | ah ok | 17:31 |
| regXboi | it is emergency time | 17:31 |
| regXboi | ok ... it looks like 218367 can go into the merge queue: | 17:32 |
| regXboi | 2015-08-28 17:31:06.843 | {3} tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_router_rescheduling ... SKIPPED: Rescheduling test does not apply to distributed routers. | 17:32 |
| *** sputnik13 has quit IRC | 17:32 | |
| regXboi | from https://jenkins07.openstack.org/job/gate-tempest-dsvm-neutron-dvr/860/consoleFull | 17:32 |
| fungi | yeah, the failure in the check pipeline shouldn't impact the run that's already in the gate | 17:32 |
| fungi | sdague: mtreinish: 218242,4 just failed grenade. should i requeue? | 17:33 |
| regXboi | anyway, I need to grab some lunch and a replacement laptop drive - be back in 45 to 60 | 17:33 |
| *** changbl has quit IRC | 17:33 | |
| *** e0ne has quit IRC | 17:33 | |
| sdague | fungi: let me look | 17:33 |
| fungi | sdague: mtreinish: http://logs.openstack.org/42/218242/4/gate/gate-grenade-dsvm/1a82a54/ | 17:34 |
| *** sdake has quit IRC | 17:34 | |
| mtreinish | sdague: it looks like the ping thing | 17:34 |
| sdague | fungi: yeh, that's this guests stop being pingable thing that I've been trying to get to the bottom of for months | 17:34 |
| fungi | okay, so unrelated to the commit. i'll reset the jobs | 17:34 |
| sdague | so please reenqueue | 17:34 |
| sdague | yep | 17:35 |
| fungi | done | 17:35 |
| fungi | tonight we can all drink to the great gate fire of liberty-3 | 17:36 |
| sdague | heh | 17:37 |
| clarkb | sdague broken dhcp? | 17:37 |
| sdague | yeh, that grenade thing, I am so freaking baffled by | 17:37 |
| sdague | clarkb: I'm pretty sure not | 17:37 |
| *** ihrachys has quit IRC | 17:37 | |
| sdague | clarkb: it was pingable here - http://logs.openstack.org/42/218242/4/gate/gate-grenade-dsvm/1a82a54//logs/grenade.sh.txt.gz#_2015-08-28_17_19_58_428 | 17:38 |
| sdague | and here - http://logs.openstack.org/42/218242/4/gate/gate-grenade-dsvm/1a82a54//logs/grenade.sh.txt.gz#_2015-08-28_17_21_18_194 | 17:38 |
| sdague | and here - http://logs.openstack.org/42/218242/4/gate/gate-grenade-dsvm/1a82a54//logs/grenade.sh.txt.gz#_2015-08-28_17_21_31_819 | 17:38 |
| sdague | but not here - http://logs.openstack.org/42/218242/4/gate/gate-grenade-dsvm/1a82a54//logs/grenade.sh.txt.gz#_2015-08-28_17_31_31_615 | 17:39 |
| sdague | I've got iptables detailed dumps, but can't figure out anything from them | 17:39 |
| docaedo | fungi: do we have to wait 'till tonight? Catching up on backlog makes me think you all deserve a hoppy noon-time toast :) | 17:40 |
| fungi | sdague: is iptables set to log session creation/rejection? if not, that might make debugging easier | 17:40 |
| *** mriedem_lunch is now known as mriedem | 17:40 | |
| sdague | my next bit of plan of attack was going to be to put really detailed iptables logging in | 17:40 |
| sdague | fungi: I don't know | 17:40 |
| fungi | docaedo: for all you know, i'm three sheets in already | 17:40 |
| sdague | but that's going to wait until we're past L3 | 17:41 |
| docaedo | fungi: hah | 17:41 |
| sdague | also, if anyone else wants to stare at these things and throw ideas out there, that would be great. I'm definitely pushing the boundaries of my iptables intuition here. | 17:41 |
| fungi | sdague: i will see what i can come up with | 17:42 |
| *** EricGonczer_ has quit IRC | 17:42 | |
| sdague | fungi: to try to help track things down I've got a worlddump with all the iptables info at earlier phases as well | 17:42 |
| fungi | though i tend to get frustrated with iptables. pf is so much easier to configure and debug | 17:42 |
| *** amuller is now known as amuller_afk | 17:42 | |
| sdague | so there should be some progressive slices there | 17:42 |
| sdague | to see what world was working, and what world was not | 17:43 |
| * fungi wishes there were a usable port of pf to linux, but alas... and no time for me to even think about writing one either | 17:43 | |
| fungi | it's too closely tied to the bsd kernel and network stack anyway | 17:43 |
| *** ybathia has joined #openstack-infra | 17:43 | |
| docaedo | maybe it's time to port the stack to freebsd? | 17:44 |
| fungi | docaedo: i've done a little test-driving debian/kfreebsd but it's still rough around the edges | 17:44 |
| *** ivar-lazzaro has quit IRC | 17:45 | |
| *** hdd has joined #openstack-infra | 17:45 | |
| docaedo | yeah I was (mostly) joking. I haven't really touched it, but there's that part of me from so long ago that continues to love freebsd :) | 17:45 |
| fungi | i'd rather have openbsd personally, but freebsd is a little more flexible on its userspace | 17:46 |
| amrith | folks, can anyone in the infra team help expedite change set 217943 through the gate. It has been there for over 16 hours and it is a revert of a previous change that is holding up all of trove changes. I see other changes are being advanced to the head of the queue, could I request your considering moving this forward please. | 17:46 |
| *** jaypipes has joined #openstack-infra | 17:46 | |
| amrith | i.e. this change reverts a regression. | 17:46 |
| amrith | would you please help move it forward in the list. thanks. | 17:46 |
| *** glauco has quit IRC | 17:47 | |
| fungi | amrith: the other changes were promoted to reduce the current carnage within the gate pipeline. if trove changes aren't making it into the gate then they're not contributing to the current openstack-wide problem | 17:47 |
| amrith | oh, so that's the reference to drinking about the great gate fire of l3 | 17:48 |
| *** ihrachys has joined #openstack-infra | 17:48 | |
| amrith | well, I'm sure that having this sit in the queue is contributing to the problem ;) | 17:49 |
| amrith | so you should help expedite it ... | 17:49 |
| *** tiswanso has quit IRC | 17:49 | |
| amrith | thanks for any help, we're also trying to make other changes to reduce our burden on the CI at this stage. | 17:49 |
| amrith | and all of that is gated behind reverting this change. | 17:49 |
| fungi | amrith: infra basically follows the qa team's recommendations on what changes need to get prioritized to improve overall health of the integrated gate queue, so if you can convince them this will get everyone else's (not just trove's) changes merging faster then please get them involved | 17:50 |
| amrith | thx fungi, will do that. | 17:51 |
| *** ihrachyshka has quit IRC | 17:51 | |
| sdague | fungi: damn, that cinder unit test fix exploded on a different cinder unit test error - https://jenkins04.openstack.org/job/gate-cinder-python27/1436/console :( | 17:52 |
| *** thiagop has quit IRC | 17:52 | |
| sdague | well, I'm about to call it a day | 17:52 |
| clarkb | sdague: http://logs.openstack.org/42/218242/4/gate/gate-grenade-dsvm/1a82a54//logs/syslog.txt.gz#_Aug_28_17_25_13 | 17:52 |
| clarkb | sdague: I can't easily correlate that to the fial because its private net vs floating IP | 17:52 |
| clarkb | sdague: but something would've had a bad day getting working IP | 17:52 |
| fungi | yeah, i feel like we're missing a piece of the puzzle without translation mapping to correlate | 17:53 |
| clarkb | can probalby go look at nova api logs to figure out what IP that floating IP was attached to | 17:53 |
| fungi | oh, right! | 17:53 |
| sdague | clarkb: oh... wait | 17:53 |
| sdague | so, the dhcp lease is short enough that if it tries to renew during service down we're toast? | 17:53 |
| clarkb | but knowing what I know of nova's broken dhcp I would not be surprised if that is the problem here | 17:53 |
| sdague | 25:13 is likely in the middle of the upgrade | 17:54 |
| *** sergmelikyan has joined #openstack-infra | 17:54 | |
| clarkb | sdague: either that or the other thing that seems to happen is the node boots and dhcp requests before dnsmasq knows about it | 17:54 |
| sdague | the node booted 8 minutes before then | 17:54 |
| sdague | and was pingable | 17:54 |
| clarkb | ok then it would be a renew in this case | 17:54 |
| clarkb | like I said need to do more correlation but having that show up in your logs is bad for business | 17:55 |
| *** e0ne has joined #openstack-infra | 17:55 | |
| clarkb | in particular the NAK that should tell the client to stop using that ip right away | 17:55 |
| sdague | although, that's a different node | 17:55 |
| clarkb | ok that is what I couldn't easily check on | 17:55 |
| sdague | 10.1.0.2 is the node that failed | 17:55 |
| clarkb | sdague: did you look in the api log to see that? | 17:55 |
| sdague | http://logs.openstack.org/42/218242/4/gate/gate-grenade-dsvm/1a82a54//logs/grenade.sh.txt.gz#_2015-08-28_17_19_58_428 | 17:56 |
| sdague | I looked back to the first ping | 17:56 |
| clarkb | sdague: http://logs.openstack.org/42/218242/4/gate/gate-grenade-dsvm/1a82a54//logs/syslog.txt.gz#_Aug_28_17_24_47 | 17:56 |
| clarkb | sdague: affects that node too | 17:56 |
| *** achanda has joined #openstack-infra | 17:56 | |
| *** mvandijk has left #openstack-infra | 17:56 | |
| sdague | ok | 17:56 |
| sdague | so at 24:47 we're building lxml - http://logs.openstack.org/42/218242/4/gate/gate-grenade-dsvm/1a82a54//logs/grenade.sh.txt.gz#_2015-08-28_17_23_51_325 | 17:57 |
| sdague | nova is definitely still offline then | 17:57 |
| clarkb | sdague: I wonder if nova removes the leases from dnsmasq when it turns off? | 17:57 |
| clarkb | or if nova has to actively renew leases with dnsmasq? | 17:57 |
| *** pelix has quit IRC | 17:57 | |
| sdague | yeh, that's a really good question | 17:57 |
| clarkb | but its very clearly a NAK which means dnsmasq is saying you cannot have this address because it is for someone else | 17:58 |
| sdague | so, I'm still going to call it a day, but this is a whole new direction to explore | 17:58 |
| sdague | clarkb: thanks much for exposing it | 17:58 |
| sdague | I'll dive in once we get past L3 with this idea in mind | 17:58 |
| *** unicell has quit IRC | 17:58 | |
| *** unicell1 has joined #openstack-infra | 17:58 | |
| sdague | yeh, we'll want to get all the mac addr dumps as well, I hope that something wonky like mac addr didn't get scrambled in this process | 17:59 |
| dougwig | is it multiple dnsmasq's, by any chance? | 18:00 |
| *** bknudson has quit IRC | 18:00 | |
| sdague | dougwig: shouldn't be | 18:01 |
| *** esker has quit IRC | 18:01 | |
| clarkb | dougwig: 28754 appears to be the only one logging | 18:01 |
| clarkb | so I don't think so | 18:01 |
| *** sridhar_ram1 has joined #openstack-infra | 18:01 | |
| *** ivar-lazzaro has joined #openstack-infra | 18:02 | |
| clarkb | it does log successful leases too | 18:02 |
| clarkb | earlier on | 18:02 |
| *** glauco has joined #openstack-infra | 18:02 | |
| *** sridhar_ram has quit IRC | 18:02 | |
| *** sergmelikyan has quit IRC | 18:03 | |
| *** mjturek1 has joined #openstack-infra | 18:04 | |
| fungi | so the theory here is that dnsmasq is running but nova controller has been stopped for upgrade, and then an instance's dhcp client attempts to renew in that timeframe and gets a nack? | 18:04 |
| clarkb | fungi: yes because all dhcp is completely managed by nova. You do not get a lease unless nova has told dnsmasq to give you one | 18:05 |
| *** sergmelikyan has joined #openstack-infra | 18:05 | |
| fungi | if so, that seems like it could similarly impact production implementations during upgrade | 18:05 |
| clarkb | yup | 18:05 |
| clarkb | I think generally compute node upgrades are fast, just restart nova net and nova cpu | 18:06 |
| *** pelix has joined #openstack-infra | 18:06 | |
| clarkb | so probably not usually a problem, but grenade doesn't do the upgrade that way instead it stops, updates then starts iirc | 18:07 |
| *** pelix has quit IRC | 18:07 | |
| clarkb | which is good for catching things like this | 18:07 |
| openstackgerrit | Sahdev Zala proposed openstack/requirements: Add tosca-parser to global requirements https://review.openstack.org/217991 | 18:07 |
| *** pelix has joined #openstack-infra | 18:08 | |
| smcginnis | Hey infra folks. | 18:08 |
| clarkb | fungi: when you go into hiera for the ssl cert can you check if the translate mysql host/password keys are in there as well? | 18:08 |
| clarkb | fungi: if you are already done iwth hiera I can do it | 18:08 |
| fungi | clarkb: i'll check the whole list of keys in that change | 18:08 |
| smcginnis | In Cinder we have a patch to skip a troublesome unit test that's blocking some things. | 18:09 |
| clarkb | fungi: great, thank you | 18:09 |
| fungi | i haven't pulled up namecheap just yet but doing that now | 18:09 |
| smcginnis | sdague "promoted that in the gate queue" to help it go through faster, but we had to recheck it. | 18:09 |
| smcginnis | Anyone that can do the magic again to get it moving? | 18:09 |
| clarkb | smcginnis: looking | 18:10 |
| smcginnis | clarkb: Thanks! | 18:10 |
| clarkb | smcginnis: 217882? | 18:10 |
| smcginnis | clarkb: Yep, that'd be the one. | 18:10 |
| *** tjones has joined #openstack-infra | 18:11 | |
| clarkb | so its still in the gate (recheck is currently a noop) | 18:11 |
| clarkb | when it kicks out I can requeue it, but first need to make sure it didn't fail for the reason it is trying to fix | 18:11 |
| smcginnis | I see it in both gate and check queues. | 18:11 |
| clarkb | python27 did fail | 18:11 |
| *** mmedvede has quit IRC | 18:11 | |
| clarkb | smcginnis: can someone look into the fail and make sure that it failed for unrelated reasons? | 18:11 |
| *** nicodemos has quit IRC | 18:11 | |
| smcginnis | clarkb: Yeah, I took a look at the logs. It was something odd in the db handling. Not at all related to the issue it's trying to get by. | 18:11 |
| clarkb | smcginnis: but could it be related to the change, in other words could thi schange have caused the failure? | 18:12 |
| *** marzif has joined #openstack-infra | 18:12 | |
| smcginnis | clarkb: I hadn't seen that particular failure before, but pretty confident it's not a normal problem or related. | 18:12 |
| tbarron | clarkb: all the change does is skip a failing unit test | 18:12 |
| clarkb | ok so likely not at fault then | 18:13 |
| tbarron | clarkb: can't see any way | 18:13 |
| clarkb | once the current gate state resets I can queue it back in | 18:13 |
| smcginnis | clarkb: 99% sure. :) | 18:13 |
| smcginnis | clarkb: Excellent. Thanks for your help! | 18:13 |
| clarkb | gonna give the other 2 that are pssing a chance at going through this round | 18:13 |
| marzif | here please any one can help with: https://review.openstack.org/#/c/193125/ | 18:13 |
| marzif | fungi, jeblair, anteaya ^^ | 18:14 |
| greghaynes | clarkb: so you said the upload for new ubuntu-trusty failed? | 18:14 |
| clarkb | greghaynes: it worked when I did it by hand | 18:14 |
| clarkb | greghaynes: growroot happened and tests passed | 18:15 |
| *** smoser has quit IRC | 18:15 | |
| asselin_ | has anyone seen these stack traces with nodepool trying to talk to jenkins? http://paste.openstack.org/show/431191/ | 18:15 |
| clarkb | asselin_: check if your jenkins master is healthy | 18:16 |
| clarkb | that appears to be like it is dropping connections and spitting out bad http | 18:16 |
| clarkb | greghaynes: so I think it owrked | 18:16 |
| *** glauco has quit IRC | 18:17 | |
| clarkb | greghaynes: once we get the automated uploads working ocnsistently we can probably consider switching the jobs over | 18:17 |
| greghaynes | clarkb: oh wow | 18:17 |
| greghaynes | clarkb: awesome@ | 18:17 |
| asselin_ | clarkb, yeah, well hard to say. I've see it really get overloaded, then there's slaves that are offline / out of sync with what nodepool says. | 18:17 |
| *** glauco has joined #openstack-infra | 18:18 | |
| asselin_ | is jenkins known to leak threads? | 18:18 |
| *** glauco__ has joined #openstack-infra | 18:19 | |
| clarkb | asselin_: yes | 18:19 |
| marzif | asselin, that would be a jvm issue probably? | 18:19 |
| fungi | asselin_: we see ever-increasing stale threads in jenkins causing it to accumulate offline slaves which nodepool thinks are deleted, ready slaves which never get jobs assigned to them, et cetera | 18:19 |
| clarkb | asselin_: and when it does you have to restart as even ninja sniping the threads is no good | 18:19 |
| clarkb | marzif: no its a jenkins bug in the ssh slave plugin | 18:19 |
| *** tiswanso has joined #openstack-infra | 18:19 | |
| clarkb | the garbage collector goes crazy and you run out of stack allocation space | 18:20 |
| clarkb | see https://issues.jenkins-ci.org/browse/JENKINS-27514 | 18:20 |
| marzif | clarkb, ok ty | 18:20 |
| clarkb | and it looks like I need to leave a comment | 18:20 |
| *** jeadre has quit IRC | 18:20 | |
| clarkb | people do not read the bugs very well | 18:20 |
| marzif | clarkb, leave a -1 :) | 18:20 |
| fungi | asselin_: basic process we follow (when it's at least responsive enough to be able to interact with the ui) is to put it into prepare for shutdown until no jobs are running, then stop the service and delete the slave entries from the global config.xml file | 18:21 |
| clarkb | well they keep thinking we are using a cloud plugin :| | 18:21 |
| clarkb | we don't we just hit the api to add and remove slaves | 18:21 |
| *** jeadre has joined #openstack-infra | 18:21 | |
| asselin_ | fungi, clarkb thanks for the info. looking at the bug now | 18:21 |
| asselin_ | cbader, ^^ | 18:21 |
| marzif | fungi, clarkb please if you have couple of min, we need this https://review.openstack.org/#/c/193125/ | 18:22 |
| fungi | asselin_: also while jenkins is in prepare for shutdown, generally a good idea to do one or two passes deleting any ready nodes for that master in nodepool since some were inevitably building before that | 18:22 |
| marzif | we are splitting the repos. | 18:22 |
| *** glauco has quit IRC | 18:22 | |
| *** mtanino has joined #openstack-infra | 18:23 | |
| clarkb | ok updated with sanity | 18:23 |
| greghaynes | TheJulia: https://bugs.launchpad.net/bifrost/+bug/1489990 | 18:24 |
| openstack | Launchpad bug 1489990 in bifrost "Config-drive is not updated on redeploy" [Undecided,New] | 18:24 |
| *** jaosorior has quit IRC | 18:25 | |
| *** smoser has joined #openstack-infra | 18:26 | |
| *** pvaneck has joined #openstack-infra | 18:27 | |
| asselin_ | how long does http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2015-08-28.log.html take to refresh? | 18:28 |
| anteaya | about 10 minutes | 18:28 |
| anteaya | asselin_: remove the .html | 18:28 |
| anteaya | then it is realtime | 18:28 |
| *** pradk has joined #openstack-infra | 18:29 | |
| asselin_ | ok, thanks I remember it being real-time | 18:29 |
| anteaya | yeah | 18:29 |
| fungi | clarkb: do you think there's a chance https://github.com/jenkinsci/ssh-slaves-plugin/commit/84caa2c may fix it? | 18:29 |
| *** ihrachys has quit IRC | 18:29 | |
| fungi | i wonder if we're sometimes deleting instances before the cleanup completes there | 18:30 |
| clarkb | oh maybe | 18:30 |
| clarkb | I did upgrade to the latest version available when I filed the bug to test and that was still affected but that commit is much newer | 18:30 |
| fungi | that might be worth an experiment if they have an updated plugin release | 18:30 |
| clarkb | we cna also build it ourselves to test | 18:30 |
| clarkb | definitely worth a shot | 18:30 |
| *** amotoki has quit IRC | 18:31 | |
| *** ihrachys has joined #openstack-infra | 18:31 | |
| *** teran has quit IRC | 18:31 | |
| *** ihrachys has quit IRC | 18:32 | |
| *** fawadkhaliq has quit IRC | 18:33 | |
| *** ihrachys has joined #openstack-infra | 18:33 | |
| openstackgerrit | Sridhar Ramaswamy proposed openstack-infra/system-config: Add #tacker channel to eavesdrop irclogs https://review.openstack.org/218406 | 18:34 |
| pleia2 | fungi: when you have a chance, can you have a look at https://review.openstack.org/#/c/217787/ ? we'll also need a real ssl certificate (hoping to get this all launched in a few hours so one less thing clarkb has to do while I'm gone!) | 18:35 |
| fungi | pleia2: yes, i'm working on that now | 18:35 |
| pleia2 | \o/ | 18:35 |
| nibalizer | /go starc | 18:35 |
| nibalizer | >_> | 18:36 |
| clarkb | yup fungi said he would also hceck the other hiera keys | 18:36 |
| pleia2 | oh great | 18:36 |
| pleia2 | I'll add the mysql stuff for trove in hiera soon | 18:37 |
| fungi | i can stick it in there unless you already generated it | 18:38 |
| pleia2 | not yet | 18:38 |
| sridhar_ram1 | infra-team: need help - for the stackforge/tacker project we have need to create a stable/kilo branch ... what are the steps ? I've never done this within openstack and assuming it will be different from typical git repo branch creation. Thanks | 18:39 |
| openstackgerrit | Ju Lim proposed openstack-infra/system-config: new mailing list for openstack-ux https://review.openstack.org/218409 | 18:40 |
| *** jaypipes has quit IRC | 18:40 | |
| *** skraynev has quit IRC | 18:41 | |
| *** e0ne has quit IRC | 18:42 | |
| *** tonytan4ever has joined #openstack-infra | 18:42 | |
| *** skraynev has joined #openstack-infra | 18:42 | |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck: Add query for tempest + neutron dvr route scheduler bug 1489929 https://review.openstack.org/218410 | 18:43 |
| openstack | bug 1489929 in tempest "test_router_rescheduling breaks on DVR" [Undecided,In progress] https://launchpad.net/bugs/1489929 - Assigned to Kevin Benton (kevinbenton) | 18:43 |
| *** glauco__ has quit IRC | 18:43 | |
| *** achanda has quit IRC | 18:44 | |
| openstackgerrit | Ju Lim proposed openstack-infra/irc-meetings: add new meeting for OpenStack-UX team https://review.openstack.org/218413 | 18:45 |
| greghaynes | TheJulia: Also, I think were using IPA, not ramdisk element | 18:45 |
| greghaynes | or, theres an ipa.initramfs in our httpboot dir | 18:45 |
| *** ramashri has quit IRC | 18:46 | |
| fungi | to repeat my mischan from -qa, looks like namecheap will do sni for up to 100 domain names for about 3x what i've been paying for single domain certs, so not bad if we want to give that a shot on static.o.o | 18:47 |
| *** amuller_afk is now known as amuller | 18:48 | |
| *** pelix has quit IRC | 18:48 | |
| fungi | as an alternative to splitting tarballs.o.o (and possibly other vhosts) off to their own separate instances | 18:48 |
| pleia2 | fungi: adding mysql info to hiera | 18:49 |
| fungi | pleia2: thanks! | 18:50 |
| *** skath has joined #openstack-infra | 18:50 | |
| openstackgerrit | Merged openstack-infra/elastic-recheck: Add query for cinder py27 sentinel bug 1488690 https://review.openstack.org/218378 | 18:50 |
| openstack | bug 1488690 in Cinder "test_create_backing_from_stream_optimized_file fails with "AttributeError: '_SentinelObject' object has no attribute 'readlines'"" [Undecided,In progress] https://launchpad.net/bugs/1488690 - Assigned to Jon Bernard (jbernard) | 18:50 |
| clarkb | fungi: 3x isn't that much if I remember my prices and considering how many things we host three should be cheaper than individual certs | 18:50 |
| clarkb | fungi: I say go for it | 18:50 |
| fungi | yeah, i just need to work out the csr details for that, so not today | 18:51 |
| *** tiswanso has quit IRC | 18:52 | |
| *** tiswanso has joined #openstack-infra | 18:52 | |
| openstackgerrit | Merged openstack-dev/pbr: Have pbr egg_info.writer check PBR is enabled https://review.openstack.org/217528 | 18:52 |
| *** xyang1 has joined #openstack-infra | 18:52 | |
| *** ramashri has joined #openstack-infra | 18:53 | |
| *** glauco has joined #openstack-infra | 18:53 | |
| *** hashar has joined #openstack-infra | 18:55 | |
| *** hodos has joined #openstack-infra | 18:55 | |
| *** tiswanso has quit IRC | 18:57 | |
| marzif | fungi, we need a small Tsunami of +2 +A here https://review.openstack.org/#/c/193125/ : ) | 18:58 |
| openstackgerrit | Merged openstack-infra/elastic-recheck: Add query for tempest + neutron dvr route scheduler bug 1489929 https://review.openstack.org/218410 | 18:58 |
| openstack | bug 1489929 in tempest "test_router_rescheduling breaks on DVR" [Critical,In progress] https://launchpad.net/bugs/1489929 - Assigned to Kevin Benton (kevinbenton) | 18:58 |
| openstackgerrit | Merged openstack-infra/project-config: Do not ignore the exit status creating Zanata XML https://review.openstack.org/218087 | 18:58 |
| *** jcoufal has quit IRC | 18:59 | |
| *** claudiub has quit IRC | 18:59 | |
| *** vivekd_ has joined #openstack-infra | 18:59 | |
| *** tjones has left #openstack-infra | 19:00 | |
| *** vivekd has quit IRC | 19:01 | |
| *** vivekd_ is now known as vivekd | 19:01 | |
| *** sdake_ is now known as sdake | 19:01 | |
| *** sridhar_ram1 has quit IRC | 19:02 | |
| *** sdake_ has joined #openstack-infra | 19:03 | |
| *** sigmavirus24 is now known as sigmavirus24_awa | 19:05 | |
| *** ivar-laz_ has joined #openstack-infra | 19:05 | |
| mtreinish | sdague: http://logs.openstack.org/19/149719/16/check/gate-tempest-dsvm-neutron-full/cb31e8e/logs/testrepository/times.dbm | 19:06 |
| *** ashleighfarnham has quit IRC | 19:06 | |
| *** ramashri has quit IRC | 19:07 | |
| *** sdake has quit IRC | 19:07 | |
| *** ramashri has joined #openstack-infra | 19:07 | |
| *** ivar-lazzaro has quit IRC | 19:08 | |
| *** stevemar has joined #openstack-infra | 19:08 | |
| *** dannywilson has quit IRC | 19:09 | |
| *** sergmelikyan has quit IRC | 19:09 | |
| *** achanda has joined #openstack-infra | 19:11 | |
| *** teran has joined #openstack-infra | 19:11 | |
| *** sridhar_ram has joined #openstack-infra | 19:12 | |
| *** hdd has quit IRC | 19:13 | |
| *** teran_ has joined #openstack-infra | 19:15 | |
| jeblair | fungi: is this so we can have https://governance, etc? | 19:17 |
| *** dannywilson has joined #openstack-infra | 19:17 | |
| *** teran has quit IRC | 19:18 | |
| fungi | jeblair: and also https://tarballs | 19:20 |
| *** sputnik13 has joined #openstack-infra | 19:20 | |
| fungi | at least that's the one which has been requested the most | 19:20 |
| * greghaynes requests .openstack tld | 19:21 | |
| clarkb | fungi: the neutron change behind the cinder change failed | 19:22 |
| clarkb | on dsvm-api | 19:22 |
| fungi | namecheap is taking way longer than usual to issue the translate.o.o cert after confirming the approval to hostmaster | 19:22 |
| fungi | clarkb: joyous | 19:22 |
| *** sputnik13 has quit IRC | 19:22 | |
| greghaynes | TheJulia: hrm, I suspect this might be more PEBKAC than bug | 19:23 |
| fungi | tripped over the api job this time, looks like | 19:23 |
| *** sputnik13 has joined #openstack-infra | 19:23 | |
| clarkb | jeblair: not sure if you saw but I got a centos7 git bakcend host up and running, 2001:4800:7818:101:be76:4eff:fe04:811 23.253.247.221 if you want to test. The nova repo is the only one populated | 19:23 |
| clarkb | jeblair: the only major change is using systemd socket activation to run git-daemon (think inetd but systemdy) | 19:23 |
| fungi | clarkb: have you looked into how/why the shallow clones are working via smart http? | 19:24 |
| clarkb | fungi: I have not | 19:24 |
| clarkb | let me dig up the trusty bug | 19:24 |
| clarkb | the daemon should be running as nobody and shouldn't have any special perms to the repos | 19:25 |
| clarkb | oh wait thats for git-daemon | 19:25 |
| clarkb | httpd runs as apache so similar situation | 19:25 |
| *** yamahata has quit IRC | 19:26 | |
| *** sputnik13 has quit IRC | 19:26 | |
| clarkb | fungi: https://git.kernel.org/cgit/git/git.git/tree/Documentation/RelNotes/1.9.2.txt#n57 i sthe only thing I see that may be related in the release notes | 19:27 |
| *** hdd has joined #openstack-infra | 19:28 | |
| *** hodos|2 has joined #openstack-infra | 19:28 | |
| *** john-davidge has joined #openstack-infra | 19:29 | |
| greghaynes | SpamapS: So ive hit an issue - I think theres an issue where mellanox isnt properly ipxeing, but VSP + mellanox ipxe doesnt work to debug (I think the line discipline is off)... any ideas on how to get textcons on one of these hosts? | 19:29 |
| mtreinish | sdague: from the times.dbm: http://paste.openstack.org/show/431266/ | 19:29 |
| clarkb | I am cloning git, it is slow | 19:30 |
| fungi | ran a few tests but couldn't easily spot where the temp packfiles are being generated | 19:31 |
| mtreinish | sdague: so it looks like testr is getting data from previous runs (at least for today's nodepool images) | 19:31 |
| *** hodos has quit IRC | 19:31 | |
| *** sergmelikyan has joined #openstack-infra | 19:32 | |
| clarkb | fungi: looking at the full release notes content with grep isn't helping me much | 19:34 |
| clarkb | fungi: do you happen to remembe rwhere the broken behavior was documented? | 19:34 |
| *** andrey-mp has joined #openstack-infra | 19:35 | |
| fungi | you mean in terms of what errors we were seeing? | 19:35 |
| *** amrith is now known as _amrith_ | 19:36 | |
| jeblair | clarkb: i agree the server appears to work for http or git, shallow or deep; neat! (?) | 19:36 |
| *** madhu_ak has quit IRC | 19:37 | |
| *** changbl has joined #openstack-infra | 19:37 | |
| *** devlaps has joined #openstack-infra | 19:37 | |
| fungi | while a curious part of me wants to know why it's working on centos 7 and not on ubuntu trusty, i'm okay pressing forward with the upgrade when we're ready | 19:38 |
| fungi | possible red hat have backported the fix to their source tree or something | 19:38 |
| clarkb | reading git commit logs on git itself it says "shallow clones sometimes write tempfiles" | 19:38 |
| clarkb | so its possible we just arne't in the correct set of circumstances | 19:38 |
| clarkb | I am about to dive into the C to see what temporary_shallow.buf is set to | 19:39 |
| jeblair | 2015-03-21T20:02:02 <mordred> http://git.661346.n2.nabble.com/PATCH-upload-pack-allow-shallow-fetching-from-a-read-only-repository-td7604326.html | 19:39 |
| jeblair | 2015-03-21T20:01:47 <jeblair> https://bugs.launchpad.net/linaro-infrastructure-misc/+bug/1253139 | 19:39 |
| openstack | Launchpad bug 1253139 in Linaro Infrastructure Misc "staging.git; --depth=1 doesn't seem to work with "smart http"" [Medium,Fix released] - Assigned to Milo Casagrande (milo) | 19:39 |
| clarkb | git_path("shallow_XXXXXX") | 19:39 |
| jeblair | 2015-03-21T20:08:53 <mordred> https://github.com/git/git/blob/master/shallow.c#L237 | 19:40 |
| andrey-mp | hello, can someone check my two small reviews? one is the small patchset for devstack - https://review.openstack.org/#/c/216403/ (it allows to use another ec2/s3 services). and second for project-config https://review.openstack.org/#/c/215766/ (it will allow additional testing of swift3, and swift/swift3 team doesn't have any objections) | 19:40 |
| jeblair | and then we started talking about scrapmetal | 19:41 |
| jeblair | that was my fault. sorry. (not really) | 19:41 |
| jeblair | http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2015-03-21.log | 19:41 |
| *** Piet has joined #openstack-infra | 19:42 | |
| smcginnis | clarkb: 217882 made it through. Thanks again for your help. | 19:42 |
| *** achanda has quit IRC | 19:43 | |
| *** eharney has quit IRC | 19:43 | |
| *** stevemar has quit IRC | 19:44 | |
| *** ramashri has quit IRC | 19:44 | |
| clarkb | jeblair: ok jeff kings commit is what I was looking at | 19:44 |
| *** stevemar has joined #openstack-infra | 19:44 | |
| *** andrey-mp has quit IRC | 19:45 | |
| greghaynes | TheJulia: ok, so whats going on is that bifrost is using ansible to figure out what the ip for things like agent_config template, we actually dont want our default gw interface ip to be the ip used in these places | 19:45 |
| openstackgerrit | Tim Buckley proposed openstack-infra/project-config: Add new OpenStack QA project StackViz. https://review.openstack.org/217416 | 19:45 |
| *** rfolco has quit IRC | 19:47 | |
| *** achanda has joined #openstack-infra | 19:47 | |
| clarkb | jeblair: but Duy's commit desn't appear to be present | 19:48 |
| *** stevemar has joined #openstack-infra | 19:49 | |
| *** stevemar has quit IRC | 19:49 | |
| *** sdake_ is now known as sdake | 19:51 | |
| *** teran has joined #openstack-infra | 19:51 | |
| *** hashar has quit IRC | 19:52 | |
| *** hashar has joined #openstack-infra | 19:52 | |
| *** jcoufal has joined #openstack-infra | 19:52 | |
| *** cfarquhar has quit IRC | 19:52 | |
| *** teran_ has quit IRC | 19:54 | |
| *** stevemar has joined #openstack-infra | 19:55 | |
| *** cfarquhar has joined #openstack-infra | 19:55 | |
| *** cfarquhar has quit IRC | 19:55 | |
| *** cfarquhar has joined #openstack-infra | 19:55 | |
| *** vivekd_ has joined #openstack-infra | 19:56 | |
| openstackgerrit | Dan Prince proposed openstack-infra/tripleo-ci: Update DELOREAN_URL to one from 2015-08-26 https://review.openstack.org/214591 | 19:57 |
| jeblair | clarkb: i don't see any opens for a tempfile when stracing apache during a shallow clone | 19:57 |
| clarkb | jeblair: I wonder if they just removed that path from the receive-pack shallow clone process completely | 19:58 |
| clarkb | er upload-pack | 19:58 |
| *** sputnik13 has joined #openstack-infra | 19:58 | |
| *** sshnaidm has quit IRC | 19:59 | |
| *** sigmavirus24_awa is now known as sigmavirus24 | 19:59 | |
| *** tjones has joined #openstack-infra | 19:59 | |
| *** vivekd has quit IRC | 19:59 | |
| *** vivekd_ is now known as vivekd | 19:59 | |
| clarkb | jeblair: the upload-pack code does if shallow_nr then it gets temporary_shallow_file | 19:59 |
| clarkb | jeblair: which seems to use git_path("shallow_XXXXXX") which should go in the git repo | 20:01 |
| clarkb | so I think I need to figure out what a shallow_nr is | 20:01 |
| marzif | all: we are blocked on https://review.openstack.org/#/c/193125/ please help | 20:01 |
| openstackgerrit | Assaf Muller proposed openstack-infra/elastic-recheck: Add query for Neutron + functional bug 1478847 https://review.openstack.org/218455 | 20:01 |
| openstack | bug 1478847 in neutron "test_udp_output_port fails with " AssertionError: RuntimeError not raised."" [High,In progress] https://launchpad.net/bugs/1478847 - Assigned to Jakub Libosvar (libosvar) | 20:01 |
| amuller | mriedem: ^ | 20:01 |
| *** bknudson has joined #openstack-infra | 20:01 | |
| *** _amrith_ is now known as amrith | 20:02 | |
| mriedem | amuller: ok, i was looking at that earlier today, couldn't find a good fingerprint | 20:02 |
| amuller | mriedem: gonna push a few more soon, we have a bunch of unclassified bugs in the neutron functional job, time to change that... | 20:03 |
| mriedem | this isn't great :) we should use the new test_ids stuff | 20:03 |
| *** pc_m has quit IRC | 20:03 | |
| mriedem | yeah, the test_ids stuff was added for the neutron functional jobs i think | 20:03 |
| amuller | mriedem: new test_ids stuff? sounds shiny, is there a doc for that? | 20:03 |
| mriedem | there is | 20:03 |
| mriedem | sec | 20:03 |
| *** angela-s has joined #openstack-infra | 20:03 | |
| mtreinish | mriedem: it won't work on functional tests, the subunit2sql db only contains tempest gate runs | 20:03 |
| mriedem | http://docs.openstack.org/infra/elastic-recheck/readme.html#queries | 20:03 |
| mriedem | mtreinish: gdi | 20:03 |
| mtreinish | mriedem: it's a simple config change to start collecting other results | 20:04 |
| jeblair | clarkb: number of shallow commits requested? | 20:04 |
| mtreinish | although there are questions about data purity | 20:04 |
| mtreinish | also if we open up the flood gates I imagine that little trove node will explode before too long | 20:04 |
| *** madhu_ak has joined #openstack-infra | 20:05 | |
| clarkb | jeblair: I think its number of objects in order to fullfill the shallow clone | 20:06 |
| clarkb | it increments that counter for every object with flags that don't match CLIENT_SHALLOW|SHALLOW | 20:06 |
| mriedem | mtreinish: it's fine | 20:06 |
| clarkb | er CLIENT_SHALLOW|NOT_SHALLOW | 20:06 |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Update DELOREAN_URL to one from 2015-08-26 https://review.openstack.org/214591 | 20:06 |
| mriedem | amuller: remove the check job from that query | 20:06 |
| *** sputnik13 has quit IRC | 20:07 | |
| amuller | mriedem: problem is today we removed the neutron functional job from the gate queue | 20:07 |
| mriedem | amuller: when the only thing we have is the test itself that's failing in a unit/functional job, we only want to query those in the gate | 20:07 |
| mriedem | amuller: so it's non-voting now? | 20:07 |
| mriedem | or just check queue? | 20:07 |
| amuller | mriedem: it was removed from the gate queue entirely, and it's voting in the neutron check queue | 20:07 |
| mriedem | yikes | 20:07 |
| mtreinish | mriedem: it's a problem we're going to have to sort out before too long | 20:07 |
| mriedem | well, that was probably a long time coming | 20:07 |
| amuller | mriedem: aye | 20:08 |
| clarkb | jeblair: if I had to make a guess, that kicks in when the side being cloned from is shallow itself | 20:08 |
| mtreinish | mriedem: because the subunit2sql data will probably be a big part of the test results dashboard | 20:08 |
| mriedem | amuller: in that case, i'd really prefer a change goes up to neutron to add more specific details in the assertion message when that specific test fails | 20:08 |
| *** john-davidge has quit IRC | 20:08 | |
| mriedem | mtreinish: test results dashboard?! do you have a newsletter than i can subscribe to? | 20:08 |
| clarkb | jeblair: and since we have complete repos we are serving we are fine. That makes sense since cloning from a shallow repo likely has to make up some lies in a temporary spot in order to pretent to a client that it is legit | 20:08 |
| clarkb | I am going to test this theory by making a shallow clone then serving that up | 20:09 |
| mtreinish | mriedem: nope, but if you were coming to the code sprint you could learn all about it | 20:09 |
| jeblair | clarkb: k | 20:09 |
| amuller | mriedem: just so I understand, what is the issue with the logstash query I posted? | 20:09 |
| mriedem | amuller: it's so generic that no one can tell why that test failed | 20:10 |
| *** john-davidge has joined #openstack-infra | 20:10 | |
| mriedem | it could be for bug 1441361 or it could be for some new bug that someone has festering in their change that isn't approved yet | 20:10 |
| openstack | bug 1441361 in OpenStack Compute (nova) "Don't trace on InstanceInfoCacheNotFound when refreshing network info_cache" [Low,In progress] https://launchpad.net/bugs/1441361 - Assigned to Chuck Carmack (chuckcarmack75) | 20:10 |
| mriedem | sorry, wrong bug | 20:10 |
| *** jeadre has quit IRC | 20:10 | |
| mriedem | bug 1478847 | 20:10 |
| openstack | bug 1478847 in neutron "test_udp_output_port fails with " AssertionError: RuntimeError not raised."" [High,In progress] https://launchpad.net/bugs/1478847 - Assigned to Jakub Libosvar (libosvar) | 20:10 |
| amuller | mriedem: is there a way to reference a test name without the test_ids stuff (That only works for Tempest currently?) | 20:11 |
| *** puranamr has joined #openstack-infra | 20:11 | |
| *** jeadre has joined #openstack-infra | 20:11 | |
| mriedem | mtreinish: should probably update the http://docs.openstack.org/infra/elastic-recheck/readme.html#queries docs about the restriction on which jobs that works on | 20:12 |
| mriedem | amuller: just what you have | 20:12 |
| clarkb | jeblair: fatal: Could not read from remote repository. | 20:13 |
| openstackgerrit | Tim Buckley proposed openstack-infra/project-config: Add new OpenStack QA project StackViz. https://review.openstack.org/217416 | 20:13 |
| mtreinish | mriedem: :/ I'll do it when I get home | 20:13 |
| clarkb | jeblair: I set up project-config that way | 20:13 |
| clarkb | jeblair: now to see if I can find the logs from when it breaks | 20:13 |
| clarkb | AH01215: fatal: attempt to fetch/clone from a shallow repository hrm maybe thats normal | 20:13 |
| *** jtomasek has quit IRC | 20:15 | |
| jeblair | clarkb: yeah, that sounds like an unsupported thing :) | 20:16 |
| *** puranamr has quit IRC | 20:17 | |
| *** rlandy has quit IRC | 20:17 | |
| clarkb | jeblair: it looks like you can do it in some circumstances but not just like that | 20:17 |
| *** sergmelikyan has quit IRC | 20:17 | |
| *** amrith has left #openstack-infra | 20:18 | |
| *** glauco has quit IRC | 20:18 | |
| *** kgiusti has left #openstack-infra | 20:18 | |
| *** puranamr has joined #openstack-infra | 20:20 | |
| annegentle | hey infra, I'm around if you have questions about http://lists.openstack.org/pipermail/openstack-infra/2015-August/003122.html the server request | 20:20 |
| *** sputnik13 has joined #openstack-infra | 20:21 | |
| *** puranamr has quit IRC | 20:21 | |
| amuller | mriedem: there's a whole bunch of tests in that test class all using the same function with different parameters, I'm not sure if there's an effective way to make the assertions more specific so they would differentiate between the tests, apart from just looking at the test name: https://github.com/openstack/neutron/blob/master/neutron/tests/functional/agent/linux/test_iptables.py#L121 | 20:22 |
| mriedem | amuller: so it fails here? https://github.com/openstack/neutron/blob/master/neutron/tests/functional/agent/linux/test_iptables.py#L89 | 20:23 |
| fungi | grumble... and now my internet at the house is down, so i'm on wireless modem once more | 20:23 |
| *** sputnik13 has quit IRC | 20:24 | |
| amuller | mriedem: exactly | 20:24 |
| clarkb | jeblair: Duys fix did eventually get in, but git describe for it isn't quite making sense | 20:24 |
| mriedem | amuller: https://github.com/testing-cabal/testtools/blob/master/testtools/testcase.py#L923 | 20:25 |
| jeblair | clarkb: link? | 20:25 |
| clarkb | jeblair: supposedly it went in in 1.9.1 but thats the version on precise | 20:25 |
| clarkb | jeblair: give me a minute was doing it locally but can find it on github for you | 20:25 |
| mriedem | amuller: construct a msg param for ExpectedException which will make it specific for the test case that's failing | 20:25 |
| mriedem | given the params passed in to the common test code | 20:25 |
| clarkb | jeblair: b790e0f67cd97f29b72cb9007632b0329e5eebec is the sha1 | 20:25 |
| mriedem | amuller: like using fw_manager, direction, port, protocol | 20:25 |
| fungi | clarkb: could it be that we need to repack the repo to expose the issue we saw before? | 20:25 |
| *** julim has quit IRC | 20:25 | |
| *** sputnik13 has joined #openstack-infra | 20:26 | |
| clarkb | fungi: no I think it was actually fixed in b790e0 just have to convince myself that we are running that code now | 20:26 |
| amuller | mriedem: gotcha, I'll do that np | 20:26 |
| jeblair | annegentle: i know mordred has started looking into swagger; i'm not sure he's quite done wrapping his head around it yet | 20:26 |
| fungi | clarkb: oh, i see. so maybe the version on trusty isn't actually running what we thought | 20:26 |
| fungi | annegentle: a naive question on my part, but can't the content be precompiled and uploaded? does it really need to be treated as dynamic content? | 20:27 |
| clarkb | oh! I bet that 1.9 and 2.0 are on separate branches | 20:27 |
| clarkb | so the fix would've been in 2.0 upstream | 20:27 |
| fungi | clarkb: i believe they are, yes | 20:27 |
| annegentle | fungi: well I suppose it could be, but the current way is it's querying the json to serve the content | 20:28 |
| annegentle | fungi: not really creating html but serving objects | 20:28 |
| clarkb | so vanilla 1.9.1 and 1.9.3 don't have it | 20:28 |
| clarkb | but 2.0 should and posibly whatever version of 1.9.3 centos cooks up would have it | 20:28 |
| *** achanda has quit IRC | 20:28 | |
| fungi | annegentle: it just seems odd to re-render that on demand rather than render it once on a job worker and put it on a webserver | 20:28 |
| annegentle | fungi: it's the only way to eventually get to a sandbox with trystack behind it, where users could enter test requests | 20:29 |
| *** vivekd has quit IRC | 20:29 | |
| annegentle | serving requests and responses not just html | 20:29 |
| fungi | annegentle: oh! so this is an interactive interface then, not just documentation | 20:29 |
| jeblair | clarkb: but we're running something called 1.8.3.1...? | 20:29 |
| clarkb | jeblair: oh I thought it was 1.9.3 | 20:30 |
| clarkb | jeblair: in that case centos7 probably is too old to have the bug | 20:30 |
| clarkb | and I should go see an optometrist | 20:30 |
| clarkb | jeblair: ya thats it | 20:30 |
| annegentle | fungi: right that's the intent -- eventually services could just have this in their wsgi config, I think. | 20:30 |
| clarkb | jeblair: breaking change went into 1.8.4 | 20:30 |
| jeblair | clarkb: oh, ha! | 20:30 |
| clarkb | er 1.8.5 | 20:31 |
| mordred | annegentle: I have thoughts, but am about to take off | 20:31 |
| openstackgerrit | Ramy Asselin proposed openstack-infra/system-config: Force gzip to run on log archives https://review.openstack.org/218462 | 20:31 |
| mordred | annegentle: I'll ping in 20 | 20:31 |
| annegentle | mordred: no prob | 20:31 |
| mordred | annegentle: also, I've learned a bunch and have other swagger uses in mind now :) | 20:32 |
| openstackgerrit | Ramy Asselin proposed openstack-infra/system-config: Force gzip to run on log archives https://review.openstack.org/218462 | 20:32 |
| fungi | i'm more stagger than swagger i'm afraid | 20:32 |
| clarkb | fwiw while having an API sandbox would be cool, I don't think that api documentation should rely on services running somewhere | 20:33 |
| clarkb | so my suggestion would be to decouple the two, but maybe mordred has better ideas | 20:33 |
| annegentle | heh fungi I like that | 20:33 |
| *** ramashri has joined #openstack-infra | 20:33 | |
| clarkb | (I should be able to read the docs offline behind my firewall without any problems) | 20:33 |
| annegentle | clarkb: it is an interesting conundrum for docs, right, because we dont' necessarily get to design the apis which is more what a spec like swagger is for | 20:33 |
| mordred | clarkb: I mostly agree, but also disagree a little ... and can't type it all from my phone | 20:34 |
| fungi | i sort of like the idea that api documentation can be rendered, packaged and redistributed, which is hard to accomplish if you need a full-featured web framework to serve it interactively | 20:34 |
| annegentle | clarkb: but decoupling the docs has meant 200+ issues | 20:34 |
| clarkb | annegentle: you can still render them from the real thing | 20:34 |
| *** smcginnis is now known as smcginnis_afk | 20:34 | |
| clarkb | annegentle: but make them renderable so that they can be shipped around | 20:34 |
| annegentle | fungi: I think the first round is literally going to be files, not a wsgi config | 20:34 |
| annegentle | clarkb: yeah something more like that | 20:35 |
| clarkb | mordred: I await for flight tking off | 20:35 |
| annegentle | could work | 20:35 |
| annegentle | heh we can go on and on :) | 20:35 |
| fungi | annegentle: thanks. mostly i was just trying to figure out what it was about the api documentation that needed a dynamic backend. the request makes a little more sense now | 20:35 |
| clarkb | my ideal would be to have `man openstack-api` do the correct thing | 20:35 |
| *** tjones has quit IRC | 20:36 | |
| fungi | yeah, the nice thing about sphinx is that it can generate manpages from your application's documentation | 20:36 |
| clarkb | then if I want to play with the sandbox I can load up a browser and hit api-sandbox.openstack.org or something | 20:36 |
| *** b10n1k_ has joined #openstack-infra | 20:37 | |
| pleia2 | annegentle: I was vacationing when you mentioned it, but Grace Hopper! That's awesome! | 20:37 |
| annegentle | I am super hesitant about it because it does mean maintenance of "one more thing" | 20:37 |
| annegentle | pleia2: oh yeah, can you believe it? 12,000 attendees is still boggling my mind | 20:37 |
| pleia2 | yeah, it'll be huge | 20:37 |
| clarkb | jeblair: can get review for https://review.openstack.org/#/c/217956/ ? its the last change we need for centos7 git backends | 20:37 |
| *** tjones has joined #openstack-infra | 20:38 | |
| clarkb | jeblair: fungi oh also the switch to system sort of breaks the launch script what appears to happen is launch node issues the reboot command and the ssh connection is killed ungracefully returning an error on the launch script side | 20:38 |
| clarkb | I haven't quite sorted out how best to deal with that yet | 20:38 |
| *** mrmartin has quit IRC | 20:38 | |
| clarkb | *switch to systemd | 20:38 |
| clarkb | but once that change merges I will make a new git08 using the prod puppet env and swap out the old git08 | 20:39 |
| fungi | clarkb: perhaps we could reboot&&exit | 20:39 |
| clarkb | fungi: I think systemd is faster than bash | 20:39 |
| *** stevemar has quit IRC | 20:39 | |
| clarkb | fungi: basically init is killing things quick | 20:40 |
| *** gordc has quit IRC | 20:40 | |
| fungi | rub lennart blood on bash to make it go faster | 20:40 |
| jeblair | fungi, clarkb: or wrap that reboot call with something that expects an error? | 20:40 |
| jeblair | (in the launch script) | 20:40 |
| clarkb | the reason is that sshd uses systemd's socket activation | 20:40 |
| clarkb | so you tell systemd to reboot and it just drops all those sockets | 20:40 |
| clarkb | jeblair: ya that may be the best thing I think return code 255 is what we get | 20:41 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/elastic-recheck: Elaborate on the limitations of test_id filtering https://review.openstack.org/218464 | 20:41 |
| clarkb | can special case that error | 20:41 |
| *** MarkAtwood has quit IRC | 20:41 | |
| jeblair | or can we do something like "shutdown -r 5s" ? | 20:41 |
| fungi | if the reboot call is via a separate ssh fork then that's probably trivial. if not, we could probably split it out fairly easily | 20:41 |
| fungi | or, yes, background a delayed shutdown | 20:41 |
| *** regXboi has quit IRC | 20:41 | |
| clarkb | its is own ssh command | 20:41 |
| fungi | and then manually make sure it worked afterward | 20:42 |
| clarkb | hrm ya doing a delayed reboot would also work | 20:42 |
| clarkb | that might be best | 20:42 |
| clarkb | I will test that with the rebuild that puts centos7 git08 in production | 20:42 |
| *** cburroughs has quit IRC | 20:43 | |
| fungi | but really, ssh reboot and trap exit code 255 seems fine to me | 20:43 |
| fungi | if it's expected behavior on at least some platforms, why fight it | 20:44 |
| fungi | also, namecheap is worrying me. i successfully approved the certificate issuance a couple hours ago and their interface still indicates it's pending and they have yet to e-mail me the cert either | 20:46 |
| jeblair | fungi: i think i have had relatively quick success with their web chat thingy in the past | 20:47 |
| *** sputnik13 has quit IRC | 20:47 | |
| fungi | yeah, i will most likely resort to contacting their helpdesk as soon as my broadband access returns | 20:47 |
| clarkb | jeblair: I should note the reason we provide our own git-daemon systemd unit file instead of using the one from the distro is that we don't want to run with homedir git repos being served and we want to customize the port | 20:48 |
| clarkb | and we fix a bug in the unit files on systemd that was fixed in fedora but never backported apparently | 20:48 |
| mordred | annegentle: ok - so - I basically love any steps that get us to some swagger. I share clarkb's concerns about running a server - and I'm also concerned about relying on trystack for the rest service in the future - mainly because if it's out of date we're not solving the problem and running CD from master is, I hear, hard | 20:48 |
| mordred | annegentle: HOWEVER | 20:48 |
| annegentle | yah | 20:48 |
| mordred | the tool to do the thing clarkb wants does not exist currently, and the service thing does | 20:48 |
| mordred | I think the world would be a better place if clarkb's tool exists | 20:49 |
| clarkb | jeblair: specifically since we use socket activation you have to use =- with exec start to have systemd clean up stale git-daemon processes | 20:49 |
| clarkb | the distro only uses = which assumes you are running a single process | 20:49 |
| mordred | but I'm not sure that running a thing is _that_ hard that we should block on a new tool (I might be wrong) | 20:49 |
| clarkb | (but its sytemd so its al la bit magic to me) | 20:49 |
| jeblair | clarkb: wow i uh. | 20:49 |
| mordred | annegentle: my main initial concern is - how is the api representation in the docs tool going to stay up to date | 20:49 |
| mordred | ? | 20:49 |
| *** teran has quit IRC | 20:50 | |
| mordred | annegentle: because it looks like the cinder api code is copied in and stubbed out - which _also_ doesn't solve the "up to date" problem | 20:50 |
| clarkb | jeblair: also the @ in the service name means start multiples of these for each connection | 20:50 |
| clarkb | jeblair: in the service name! | 20:50 |
| annegentle | mordred: so the original idea was to run a Swagger maker as a wsgi application | 20:50 |
| jeblair | clarkb: i don't see the @ | 20:50 |
| annegentle | mordred: each service could be configured to generate swagger or raml or whatever | 20:51 |
| *** krtaylor has quit IRC | 20:51 | |
| annegentle | then we'd use those generated files to serve docs, compare if docs drift, that sort of thing | 20:51 |
| fungi | clarkb: wow. you mangle the service name to get independent forks per connection? like, how inetd would have naturally forked? | 20:51 |
| clarkb | jeblair: https://review.openstack.org/#/c/217956/5/manifests/init.pp line 205 | 20:51 |
| jeblair | clarkb: oh, in the path to /usr/lib...yeah | 20:51 |
| clarkb | fungi: ya | 20:51 |
| annegentle | mordred: so the cinder example is for now just to prove we can do this | 20:51 |
| jeblair | clarkb: he wrote this from scratch and came up with that? | 20:51 |
| mordred | annegentle: yah - so it seems that we could also just serve those generated docs - and do the generation during a devstack run | 20:51 |
| mordred | annegentle: or something | 20:51 |
| clarkb | jeblair: apparently and never wouldve figured it out if it werent for the fedora bug | 20:52 |
| fungi | so far the only unit files i've written have been for listening daemons which did their own socket management, so hadn't had chance to encounter that bit of madness | 20:52 |
| clarkb | * I never would've | 20:52 |
| clarkb | the @ makes it a template which means you can run multiples | 20:52 |
| annegentle | mordred: or something, yeah. http://specs.openstack.org/openstack/docs-specs/specs/liberty/api-site.html has details, not that I expect you to read it all, but we know the docs can't stay updated across 30+ services. | 20:52 |
| clarkb | it can also pass in some data from the calling systemd daemon or something but we don't use that functionality | 20:52 |
| mordred | annegentle: so I think my tl;dr is "highly supportive, would like to riff a bit more on the specific impl with folks" | 20:52 |
| clarkb | like %i gets interpolated as the port or something | 20:52 |
| mordred | annegentle: becuase I _think_ we might be able to make it easier | 20:53 |
| jeblair | mordred: ++ | 20:53 |
| clarkb | annegentle: so I read that and I couldn't reall figure out where it said "we hvae to run a service against live apis" | 20:53 |
| clarkb | annegentle: but I also skimmed because its large | 20:53 |
| *** amuller has quit IRC | 20:53 | |
| annegentle | mordred: clarkb: Heh I need to reformat this: #. Do a git clone command to get a copy of the project’s repo. #. Run the automation script with some config info in the api-paste.ini file to create Swagger. #. Repeat and test for each project. | 20:54 |
| annegentle | clarkb: what I had sort of hoped is that we'd just use the upstream Swagger display stuff | 20:54 |
| mordred | clarkb: yah - that's because swagger-ui itself, the existing tool, expects to be pointed at a REST service - I think we the rest service is just the quickest way to get from annegentle's steps above to a working thing | 20:54 |
| clarkb | mordred: annegentle in that case maybe the spec needs updating? | 20:55 |
| annegentle | mordred: right I had hoped to use swagger-ui directly but it's not great for our apis | 20:55 |
| clarkb | mordred: annegentle because to me thats a critical portion of this and its completely glossed over | 20:55 |
| *** jaypipes has joined #openstack-infra | 20:55 | |
| clarkb | but maybe I am not the intended audience and my ignorance is expected | 20:55 |
| annegentle | clarkb: ayup I didn't realize it until we had the proof of concept | 20:55 |
| mordred | annegentle: I'm thinking we might want to make a patch to upstream swagger to allow it to point to a local file instead of a rest service | 20:55 |
| annegentle | clarkb: not at all it's a good point | 20:55 |
| annegentle | mordred: hm! | 20:56 |
| *** samuel-dmq has joined #openstack-infra | 20:56 | |
| mordred | annegentle: because that way if we do the generatoin step you mention, we can just update the local file copy | 20:56 |
| annegentle | mordred: would certainly help with our docs use case. But, it's also what Russell is already thinking, right, make JSON, serve JSON? | 20:56 |
| mordred | annegentle: that's right! | 20:56 |
| *** ivar-laz_ has quit IRC | 20:56 | |
| mordred | I think we just need to skip the part where the only way to serve the json is via a REST call - since hte json is actually static | 20:56 |
| fungi | if the bit of swagger that does the live discovery can be separated out, we could in theory just run that in a job and generate data to pass into the part which generates the content | 20:56 |
| mordred | fungi: exactly | 20:57 |
| fungi | yeah, lagging here because wireless modem | 20:57 |
| mordred | fungi: dude. I'm in a plane | 20:57 |
| mordred | my lag is amaze | 20:57 |
| * fungi shakes fist at terrible cable company | 20:57 | |
| fungi | mordred: yeah, and you're still beating me. it's sad, sad, sad | 20:57 |
| mordred | annegentle: so maybe sometime next week we can chat with russell and make a plan or something (I'll be on not-planes next week) | 20:57 |
| *** samueldmq has quit IRC | 20:57 | |
| *** esker has joined #openstack-infra | 20:58 | |
| annegentle | mordred: awesome. I'll email you me Russell | 20:58 |
| *** esker has quit IRC | 20:58 | |
| annegentle | mordred: he's at Aptira in Australia | 20:58 |
| *** amuller has joined #openstack-infra | 20:58 | |
| *** esker has joined #openstack-infra | 20:59 | |
| fungi | alternatively, we could just run swagger in a devstack job to generate the content? | 20:59 |
| *** MarkAtwood has joined #openstack-infra | 20:59 | |
| *** dprince has quit IRC | 21:00 | |
| annegentle | fungi: it seems like it, but I don't know enough about the doc object model, etc. | 21:00 |
| * fungi has no idea if that's as insane as it sounds | 21:00 | |
| openstackgerrit | Merged openstack-infra/puppet-cgit: Use systemd'd git-daemon on Centos7 https://review.openstack.org/217956 | 21:00 |
| annegentle | fungi: mordred: tell me if this is accurate: angularjs:json apache:html? | 21:00 |
| *** dangers is now known as dangers_away | 21:01 | |
| *** tjones has left #openstack-infra | 21:01 | |
| annegentle | fungi: so it's sorta like I'm asking for a non-apache web server | 21:01 |
| annegentle | fungi: we'd build the json files, serve the json files | 21:02 |
| jeblair | ETOOHOTTOTHINK | 21:02 |
| annegentle | fungi: but I might be wrong | 21:02 |
| annegentle | jeblair: heh it's 97 here, uhh where are you? | 21:02 |
| fungi | hrm, yeah if the rendering is in javascript and the data is in json, then just serving those would in theory let the browser turn them into content | 21:02 |
| *** puranamr has joined #openstack-infra | 21:02 | |
| fungi | so wouldn't necessarily need any special sort of web server | 21:02 |
| mordred | fungi: right. .. so | 21:03 |
| mordred | the big question is if swagger-ui is using any verbs other than "GET" to request the json files" | 21:03 |
| fungi | ahh, right | 21:03 |
| EmilienM | hello, I have a small patch for puppet logs script in project-config, https://review.openstack.org/#/c/218245/ | 21:03 |
| mordred | or if it's using parameters | 21:03 |
| mordred | if it's using parameters, we need to be cleverer | 21:03 |
| *** krtaylor has joined #openstack-infra | 21:03 | |
| mordred | but if it's just GETing a path - we can fake that with static apache | 21:04 |
| jeblair | annegentle: it's 85 here. with no ac or breeze. | 21:04 |
| jeblair | like, it's 85 indoors | 21:04 |
| fungi | because people building houses in the bay area never plan for 85f with no breeze | 21:04 |
| annegentle | jeblair: oh that's the _worst_ | 21:04 |
| *** amuller has quit IRC | 21:05 | |
| *** aysyd has quit IRC | 21:05 | |
| annegentle | jeblair: our AC went out twice this summer and even at 82-84 indoors it's hard to sleep/eat/function | 21:05 |
| *** hashar has quit IRC | 21:05 | |
| *** MarkAtwood has quit IRC | 21:05 | |
| *** rossella_s has quit IRC | 21:05 | |
| *** amuller has joined #openstack-infra | 21:05 | |
| annegentle | ok, for sure I'm gonna update the spec, and set up the meeting for next week | 21:06 |
| jeblair | annegentle: that's what i thought! but i wasn't sure. :) | 21:06 |
| *** rossella_s has joined #openstack-infra | 21:06 | |
| annegentle | crazy what we get used to :) | 21:06 |
| annegentle | mordred: and to be clear, it's not swagger-ui exactly... | 21:06 |
| jeblair | fungi: yeah, we get an actual summer for 2 weeks in september. | 21:06 |
| jeblair | so, i guess it's september. | 21:07 |
| fungi | jeblair: guess it came early this year | 21:07 |
| annegentle | mordred: though he did make it in a poc http://fairy-slipper.russellsim.org/swagger-ui/ | 21:07 |
| *** amuller has quit IRC | 21:07 | |
| *** amuller_ has joined #openstack-infra | 21:07 | |
| jeblair | fortunately, my new house is actually cooler! | 21:07 |
| clarkb | jeblair: we have had a really warm summer (most days above 90F ever recorded in one year so far) and the house would get up to ~82F those days | 21:07 |
| *** amuller_ is now known as amuller | 21:07 | |
| annegentle | mordred: anyway, poke around all you like, I've gotta go pick up school stuff | 21:07 |
| *** trown is now known as trown|outttypeww | 21:08 | |
| mordred | annegentle: woot! | 21:08 |
| jeblair | fungi, clarkb: my current place manages to typically be _warmer_ than outside; the new place cooler (better insulation, shade, site, etc). | 21:08 |
| mordred | jeblair: woot! | 21:11 |
| *** b10n1k_ has quit IRC | 21:12 | |
| kfox1111 | any status on the gate? | 21:13 |
| fungi | jeblair: yeah, our new place has ceiling fans in basically every room (the living room even has two!) so i'm able to comfortably keep the air conditioner not running much | 21:14 |
| clarkb | kfox1111: looks like all but one of the fixes have merged | 21:14 |
| clarkb | fungi: ^ we need to requeue the last assuming it didn't fail due to the change itself? | 21:14 |
| *** ldnunes has quit IRC | 21:15 | |
| *** samuel-dmq is now known as samueldmq | 21:15 | |
| *** puranamr has quit IRC | 21:15 | |
| fungi | clarkb: oh? i thought i already did | 21:15 |
| kfox1111 | k. thanks. | 21:15 |
| *** sthillma has joined #openstack-infra | 21:15 | |
| clarkb | fungi: maybe my grep failed | 21:15 |
| clarkb | fungi: I don't see 217379 | 21:16 |
| fungi | clarkb: i can't load the zuul status over this connection but it's possible it failed a second time after being requeued | 21:16 |
| clarkb | ah gotcha | 21:16 |
| fungi | if zuul reported a -2 verify on it in the past hour then it must have | 21:16 |
| *** burgerk_ has quit IRC | 21:16 | |
| clarkb | oh it did merge nevermind | 21:16 |
| clarkb | I wa sgoing off the therpad and its just behind | 21:17 |
| *** b10n1k_ has joined #openstack-infra | 21:17 | |
| clarkb | kfox1111: so all 4 have merged | 21:17 |
| fungi | also namecheap came through, _finally_ so i can get the cert added to hiera in moments | 21:17 |
| fungi | clarkb: yeah, i was the one updating that etherpad and the change presumably merged after my broadband fell over | 21:17 |
| *** stevemar has joined #openstack-infra | 21:18 | |
| pleia2 | eep, I need lunch | 21:23 |
| pleia2 | bbiab | 21:23 |
| *** mjturek1 has quit IRC | 21:23 | |
| openstackgerrit | Assaf Muller proposed openstack-infra/elastic-recheck: Add query for Neutron functional failure bug 1477860 https://review.openstack.org/218475 | 21:24 |
| openstack | bug 1477860 in neutron "TestAsyncProcess.test_async_process_respawns fails with TimeoutException" [High,Confirmed] https://launchpad.net/bugs/1477860 | 21:24 |
| *** FallenPe_ has joined #openstack-infra | 21:24 | |
| *** b10n1k_ has quit IRC | 21:24 | |
| *** hdd has quit IRC | 21:25 | |
| *** alivigni has quit IRC | 21:25 | |
| *** puranamr has joined #openstack-infra | 21:25 | |
| *** tlian has quit IRC | 21:25 | |
| *** hdd has joined #openstack-infra | 21:25 | |
| *** FallenPe_ has quit IRC | 21:26 | |
| docaedo | Is the gate unwedged enough for me to nudge you on https://review.openstack.org/#/c/205882/ ? | 21:27 |
| *** FallenPegasus has quit IRC | 21:27 | |
| docaedo | (it's very minor, only impacts app catalog, and merging would help further the efforts on the horizon plugin for the app-catalog) | 21:27 |
| *** eharney has joined #openstack-infra | 21:28 | |
| *** gordc has joined #openstack-infra | 21:29 | |
| *** puranamr has quit IRC | 21:29 | |
| *** tjones has joined #openstack-infra | 21:29 | |
| *** gordc has quit IRC | 21:29 | |
| *** ihrachys has quit IRC | 21:30 | |
| *** sridhar_ram1 has joined #openstack-infra | 21:32 | |
| *** sridhar_ram has quit IRC | 21:35 | |
| *** mriedem has quit IRC | 21:37 | |
| *** dimsum__ has quit IRC | 21:38 | |
| openstackgerrit | Merged openstack-infra/elastic-recheck: Elaborate on the limitations of test_id filtering https://review.openstack.org/218464 | 21:39 |
| clarkb | launching new git08 with suggested shutdown -r 5s update to launch node | 21:39 |
| openstackgerrit | Assaf Muller proposed openstack-infra/elastic-recheck: Add query for Neutron functional failure bug 1478190 https://review.openstack.org/218484 | 21:39 |
| openstack | bug 1478190 in neutron "neutron.tests.functional.test_server.TestWsgiServer.test_restart_wsgi_on_sighup_multiple_workers failed with psutil._error.NoSuchProcess" [Undecided,New] https://launchpad.net/bugs/1478190 - Assigned to Elena Ezhova (eezhova) | 21:39 |
| clarkb | if this all works I will replace the old git08 in the rotation | 21:40 |
| *** cbader has quit IRC | 21:40 | |
| *** amitgandhinz has quit IRC | 21:40 | |
| *** amitgandhinz has joined #openstack-infra | 21:41 | |
| *** hdd has quit IRC | 21:41 | |
| *** glauco has joined #openstack-infra | 21:43 | |
| *** john-davidge has quit IRC | 21:43 | |
| *** glauco has quit IRC | 21:44 | |
| Clint | fungi: what's the status of "get the non-devstack jobs able to run on devstack-nodes" workstream ? | 21:44 |
| *** samueldmq has quit IRC | 21:45 | |
| *** amitgandhinz has quit IRC | 21:46 | |
| *** annegentle has quit IRC | 21:47 | |
| *** jaypipes has quit IRC | 21:47 | |
| *** annegentle has joined #openstack-infra | 21:47 | |
| *** mwagner_lap has joined #openstack-infra | 21:48 | |
| *** achanda has joined #openstack-infra | 21:48 | |
| *** hdd has joined #openstack-infra | 21:49 | |
| *** hdd has quit IRC | 21:49 | |
| *** jaypipes has joined #openstack-infra | 21:49 | |
| anteaya | jeblair: yay for shade | 21:50 |
| anteaya | I swear by it | 21:50 |
| *** tiswanso has joined #openstack-infra | 21:51 | |
| clarkb | looks like shutdown only takes integer increments of minutes | 21:51 |
| anteaya | :( | 21:51 |
| clarkb | so shutdown -r 5s doesn't work and neither does +5s | 21:51 |
| clarkb | Clint: I believe its waiting partially on the consolidated image types to be built and yesterday we got growroot working on trusty | 21:52 |
| fungi | Clint: i have an example job which does it for nova unit tests. it's mostly blocked on getting the images in better shape (growroot, more stable glance uploads to rax) | 21:52 |
| openstackgerrit | Assaf Muller proposed openstack-infra/elastic-recheck: Add query for Neutron functional failure bug 1490037 https://review.openstack.org/218488 | 21:53 |
| *** jaypipes has quit IRC | 21:53 | |
| openstack | bug 1490037 in neutron "test_port_vlan_tags failing wait_until_ports_state" [High,Confirmed] https://launchpad.net/bugs/1490037 | 21:53 |
| *** lascii is now known as alaski | 21:53 | |
| fungi | Clint: though in theory it would work for non-dib nodes too if we need, just have to port the bindep package cache implementation to nodepool prep scripts for snapshot-based images | 21:53 |
| *** jaypipes has joined #openstack-infra | 21:53 | |
| jeblair | clarkb: *headdesk* | 21:55 |
| *** dkranz has quit IRC | 21:56 | |
| *** dimsum__ has joined #openstack-infra | 21:56 | |
| *** tjones has quit IRC | 21:59 | |
| *** sigmavirus24 is now known as sigmavirus24_awa | 21:59 | |
| *** sigmavirus24_awa is now known as sigmavirus24 | 21:59 | |
| *** dims_ has joined #openstack-infra | 22:00 | |
| *** xyang1 has quit IRC | 22:00 | |
| Clint | clarkb: fungi: what else does growroot need? | 22:00 |
| *** bauwser is now known as bauzas | 22:00 | |
| fungi | Clint: i think just some working image uploads as of what i saw greghaynes and clarkb discussing a few hours ago | 22:01 |
| clarkb | yup | 22:01 |
| clarkb | so the images work now we just have to get them in the clouds reliably | 22:01 |
| fungi | there was stuff that needed updating and releasing in dib which just happened like yesterdaying | 22:01 |
| fungi | yesterdayish | 22:02 |
| *** dimsum__ has quit IRC | 22:02 | |
| * Clint nods. | 22:03 | |
| Clint | fungi: and what's the status of the bindep stuff? | 22:04 |
| fungi | Clint: fully functional, but the example job implementation is just in nova's experimental pipeline and needs a bit of a project plan for how we can convert the current set of tox-based jobs to leveraging it without (hopefully) breaking them | 22:05 |
| *** tjones has joined #openstack-infra | 22:06 | |
| *** ybathia has quit IRC | 22:06 | |
| fungi | i think we can just turn some of what's in the current job macros into a slave script or two, or maybe work it into the current run-tox.sh slave script | 22:07 |
| openstackgerrit | K Jonathan Harker proposed openstack-infra/project-config: Create a job to ease manpage reviews https://review.openstack.org/218493 | 22:07 |
| fungi | also one thing i haven't implemented yet is more robust package retrieval (retries for package index updates and package installs) | 22:07 |
| *** thorst1 has quit IRC | 22:08 | |
| mordred | I know many don't care | 22:08 |
| mordred | but twitter seems to be HARD DOWN and has been for a little bit | 22:08 |
| mordred | I'm reporting not because it's important | 22:08 |
| *** dkranz has joined #openstack-infra | 22:08 | |
| mordred | but because bit sites rarely go this down for this long | 22:08 |
| anteaya | it is interesting | 22:08 |
| mordred | big | 22:08 |
| *** jaypipes has quit IRC | 22:08 | |
| fungi | Clint: we have some retry logic we can steal from devstack-gate, so shouldn't be tough to splice in | 22:08 |
| *** tjones has left #openstack-infra | 22:09 | |
| anteaya | mordred: it is up for me | 22:09 |
| fungi | mordred: wow. and here i thought my internet connection was down, but it's really that i just always assume twitter is up (ha!) | 22:09 |
| Clint | Something is technically wrong. | 22:09 |
| Clint | Thanks for noticing—we're going to fix it up and have things back to normal soon. | 22:09 |
| jesusaurus | mordred: it appears to be up for me, and @doctorow tweeted about 10 minutes ago | 22:09 |
| mordred | Clint: yup. that's what I get | 22:10 |
| Clint | it's a targeted attack against new yorkers | 22:10 |
| mordred | Clint: ++ | 22:10 |
| mordred | Clint: we are a shifty bunch | 22:10 |
| * Clint shifts. | 22:10 | |
| mordred | Clint: also, you just implied I'm a New Yorker, which makes me warm inside | 22:10 |
| clarkb | mordred: maybe they decided you were doing something illegal | 22:10 |
| anteaya | mordred: I thought you were in a plane still | 22:10 |
| clarkb | mordred: I hear they shut down accounts for that now or something | 22:10 |
| clarkb | oh also united inflight internets were caught filtering a bunch of sites | 22:10 |
| mordred | clarkb: wow | 22:12 |
| Clint | i bet you neither of us is on united | 22:12 |
| anteaya | mordred: I'm going with delta | 22:12 |
| clarkb | but they all use the same gogo inflight right? | 22:12 |
| jeblair | stopgo | 22:13 |
| Clint | wake me up before you stopgo | 22:13 |
| clarkb | wait they put mordred and leakypipes on the same plane? what were they thinking | 22:13 |
| anteaya | ha ha ha | 22:14 |
| *** dimtruck is now known as zz_dimtruck | 22:14 | |
| mordred | clarkb: it was fun | 22:14 |
| anteaya | cat got herself half through the loop handle of a paper gift bag and then took down a lamp, it fell on me and then the shade protected the 35 dollar led bulb, whew | 22:15 |
| anteaya | she got herself into a safe place and let me remove the bag | 22:15 |
| anteaya | thus saving other breakable items from breaking | 22:16 |
| anteaya | first time she has ever done that | 22:16 |
| mordred | anteaya: bwahahahaha | 22:16 |
| anteaya | she normally doesn't do the silly things | 22:16 |
| anteaya | yeah, figured you would enjoy that | 22:16 |
| anteaya | part of me was laughing | 22:16 |
| anteaya | and part of me was not wanting broken things | 22:17 |
| mordred | anteaya: that's two different mentions of the word "shade" from you in the last few minutes neither of which had to do with the library | 22:17 |
| *** ivar-lazzaro has joined #openstack-infra | 22:17 | |
| mordred | I was confused for a second both times | 22:17 |
| anteaya | do I get points? | 22:17 |
| mordred | anteaya: yes! | 22:17 |
| anteaya | ha ha ha | 22:17 |
| anteaya | trigger words | 22:17 |
| fungi | oh yeah, one of my cats got his head stuck through the handle of a shopping bag once. mass destruction ensued | 22:17 |
| anteaya | ha ha ha | 22:17 |
| anteaya | part of you was laughing on the inside | 22:17 |
| mordred | anteaya: you also get this mildly happy bag of iguanas | 22:17 |
| anteaya | but not the part that had to clean up the things | 22:17 |
| *** b10n1k_ has joined #openstack-infra | 22:17 | |
| anteaya | let me tell you my iguana story | 22:18 |
| anteaya | I have one | 22:18 |
| mordred | HAHAHAHAHAHAHAHAHAHAHA | 22:18 |
| fungi | he was running all around, trying to run away from the bag. it really, REALLY wasn't working | 22:18 |
| anteaya | this was in university | 22:18 |
| anteaya | fungi: ha ha ha | 22:18 |
| anteaya | I'm in fourth year, we had the most awesome house | 22:18 |
| anteaya | another time I will tell you about the guy who I shared the top floor with | 22:18 |
| anteaya | he was the best | 22:18 |
| anteaya | but this is the iguana story | 22:19 |
| anteaya | the woman who had the room off the kitchen (7 of us in the house) | 22:19 |
| anteaya | was an art student (very awesome) | 22:19 |
| anteaya | and had an iguana as she was allergic to everything else | 22:19 |
| anteaya | she often had the door to her room open as the iguana for most of the year moved in centimeter units | 22:20 |
| *** tiswanso has quit IRC | 22:20 | |
| mordred | anteaya: they do that | 22:20 |
| anteaya | usually it sat on the branch in the empty fish tank that waas its home | 22:20 |
| anteaya | yes, that was the expected behaviour | 22:20 |
| *** tiswanso has joined #openstack-infra | 22:20 | |
| anteaya | one night I was alone in the kitchen cooking, I think she was upstairs having a shower and random housemates were in the living room doing things | 22:21 |
| anteaya | watching tv I think | 22:21 |
| anteaya | all of a sudden iguana is out of her room and pinwheeling agains the wall opposite her door | 22:21 |
| anteaya | balancing on its tail | 22:21 |
| mordred | oh my | 22:21 |
| anteaya | all four legs going in circles | 22:21 |
| *** ivar-lazzaro has quit IRC | 22:21 | |
| *** dkranz has quit IRC | 22:22 | |
| anteaya | getting no purchase on the wall and not stopping | 22:22 |
| anteaya | I just turned and staired | 22:22 |
| mordred | hehehe | 22:22 |
| anteaya | stared | 22:22 |
| anteaya | roommates came to also stare from the other end of the hall | 22:22 |
| anteaya | someone got her out of the shower, none of the rest of us knew what to do and the thing wouldn't stop | 22:22 |
| anteaya | she got it into the room and closed the door | 22:23 |
| *** rlandy has joined #openstack-infra | 22:23 | |
| anteaya | after that, iguana never moved again | 22:23 |
| anteaya | but an exciting 5 minutes | 22:23 |
| anteaya | etched in my memory | 22:23 |
| anteaya | thought I would share | 22:24 |
| mordred | I had no idea iguanas could run | 22:24 |
| anteaya | neither did I | 22:24 |
| mordred | or, really, do anything other that sit and look at you | 22:24 |
| anteaya | but apparently should they choose they don't have much by way of evaluation of prior decision | 22:24 |
| openstackgerrit | Assaf Muller proposed openstack-infra/elastic-recheck: Add query for Neutron functional failure bug 1490043 https://review.openstack.org/218499 | 22:25 |
| openstack | bug 1490043 in neutron "test_keepalived_respawns fails when trying to kill -15" [High,Confirmed] https://launchpad.net/bugs/1490043 | 22:25 |
| *** bnemec has quit IRC | 22:27 | |
| *** sridhar_ram1 has quit IRC | 22:27 | |
| clarkb | I am updating dns now for git08 | 22:35 |
| clarkb | haproxy uses ips so it won't go in until we merge the change I need to push | 22:35 |
| clarkb | gonna make sure gerrit replicates to it cleanly first | 22:36 |
| clarkb | and wait for dns to cycle out TTLs | 22:36 |
| fungi | okay, new ssl cert/key/chain _finally_ in hiera for translate.o.o | 22:36 |
| fungi | checking the other hiera keys are present now | 22:36 |
| clarkb | I also have a patch to check for ssh fails on the reboot command | 22:37 |
| clarkb | but doing dns first as its an hour wait | 22:37 |
| *** ddieterly has joined #openstack-infra | 22:38 | |
| jeblair | clarkb: i think there may be haproxy commands you can do to disable servers | 22:38 |
| *** sridhar_ram has joined #openstack-infra | 22:38 | |
| clarkb | jeblair: there are but its unnecessary as it uses ip addrs | 22:39 |
| clarkb | jeblair: will be unaffected by the dns changes | 22:39 |
| *** bhunter71 has quit IRC | 22:39 | |
| jeblair | clarkb: yeah, though just in case you wanted to disable it during the window when gerrit stops replicating to it but before haproxy changes | 22:39 |
| clarkb | oh good point let me do that | 22:39 |
| ddieterly | where can i find a list of packages and versions that zuul uses to build and test python code? specifically, i'm looking for the influxdb-python versions. | 22:39 |
| jeblair | clarkb: are you familiar with 'socat /var/lib/haproxy/stats readline' and 'prompt'? | 22:40 |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder: added support for jdepend plugin https://review.openstack.org/201533 | 22:40 |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config: Swap out git08 to new centos7 node https://review.openstack.org/218504 | 22:41 |
| clarkb | jeblair: no, do those make socat interactive? | 22:42 |
| clarkb | jeblair: I have a handful of commands for socat stashed in my command history that I will be running now | 22:42 |
| jeblair | clarkb: yep; run 'socat /var/lib/haproxy/stats readline' from shell; and then type "prompt" | 22:42 |
| jeblair | clarkb: then you can type 'help', etc. | 22:42 |
| *** dkranz has joined #openstack-infra | 22:43 | |
| fungi | ddieterly: it's potentially job-specific. do you have a sample job you're inquiring about? | 22:44 |
| fungi | ddieterly: worth noting, zuul doesn't really test python code, it's merely a (very complex and flexible) job scheduler | 22:44 |
| jeblair | clarkb: i discovered that while working on haproxy -> statsd (WIP here: https://review.openstack.org/213952) | 22:44 |
| ddieterly | fungi: ok, here's what i'm trying to get to pass the tests. at least these are the results. | 22:45 |
| jeblair | i think that's ready actually, just need to write an init script for it. | 22:45 |
| ddieterly | Build failed (check pipeline). For information on how to proceed, see http://docs.openstack.org/infra/manual/developers.html#automated-testing | 22:45 |
| ddieterly | - gate-monasca-api-pep8 http://logs.openstack.org/95/218495/3/check/gate-monasca-api-pep8/ae97fa6/ : SUCCESS in 1m 05s | 22:45 |
| ddieterly | - gate-monasca-api-python27 http://logs.openstack.org/95/218495/3/check/gate-monasca-api-python27/851ac6b/ : FAILURE in 1m 13s | 22:45 |
| ddieterly | - gate-monasca-api-build http://logs.openstack.org/95/218495/3/check/gate-monasca-api-build/3495f42/ : SUCCESS in 4m 01s | 22:45 |
| clarkb | jeblair: it broke the pipe after the command shoul dI take that as an indication it failed? | 22:45 |
| fungi | ddieterly: in most cases, there should be a pip freeze in the console log of those jobs. if not, then the script comprising the job commands could be amended to add one | 22:46 |
| *** ybathia has joined #openstack-infra | 22:46 | |
| *** wenlock has quit IRC | 22:46 | |
| openstackgerrit | Merged openstack-infra/system-config: Add manifest for translate.openstack.org https://review.openstack.org/217787 | 22:46 |
| fungi | ddieterly: for example http://logs.openstack.org/95/218495/3/check/gate-monasca-api-pep8/ae97fa6/console.html#_2015-08-28_22_21_31_380 | 22:47 |
| *** eharney has quit IRC | 22:47 | |
| fungi | ddieterly: that claims that influxdb==2.8.0 was installed in the environment where those tests ran | 22:47 |
| ddieterly | yea, i see that. weird. it looks like the older version of influxdb was used. | 22:48 |
| jeblair | clarkb: i think we want "set timeout cli 2m" or something :) | 22:48 |
| jeblair | clarkb: cause i think it defaults to 10 seconds | 22:48 |
| clarkb | oh fun | 22:48 |
| jeblair | i don't know if that persists | 22:48 |
| clarkb | I just did it with echo | socat | 22:48 |
| ddieterly | well, the freeze has the right version, but the error seems to indicte that the older version was used. | 22:48 |
| clarkb | jeblair: and we only have one haproxy node now according to dns | 22:49 |
| jeblair | clarkb: i think it's per-session. so i guess you'd want to start a session with "prompt;set timeout cli 2m" | 22:49 |
| fungi | ddieterly: it's possible you have some conflicting requirements expressed. http://git.openstack.org/cgit/stackforge/monasca-api/tree/requirements.txt#n17 claims that monasca-api should use influxdb==2.3.0 | 22:49 |
| *** ybathia has quit IRC | 22:49 | |
| jeblair | clarkb: 1 haproxy matches my understanding of the current topology. :) | 22:49 |
| fungi | ddieterly: oh, i see. the change you're testing is raising that | 22:50 |
| ddieterly | yea | 22:50 |
| *** sabeen has quit IRC | 22:50 | |
| fungi | ddieterly: as i said, environments are job-specific. if the failing gate-monasca-api-python27 is the one you're interested in, it's possible something is overriding it in that job | 22:51 |
| *** ybathia has joined #openstack-infra | 22:52 | |
| openstackgerrit | Assaf Muller proposed openstack-infra/elastic-recheck: Add query for Neutron functional failure bug 1490051 https://review.openstack.org/218506 | 22:52 |
| *** stevemar has quit IRC | 22:52 | |
| openstack | bug 1490051 in neutron "test_assert_pings_during_br_int_setup_not_lost fails with oslo_rootwrap.wrapper.NoFilterMatched" [High,Confirmed] https://launchpad.net/bugs/1490051 | 22:52 |
| fungi | ddieterly: though the log for it similarly shows pip installing 2.8.0 at the start of the job, and then at the end it indicates 2.8.0 is still installed | 22:53 |
| *** stevemar has joined #openstack-infra | 22:53 | |
| fungi | ddieterly: have you tried running it locally the same way? (git clean -dfx; tox -e py27) | 22:55 |
| ddieterly | yea, let me try that. might be something with the tests. | 22:55 |
| *** tonytan4ever has quit IRC | 22:55 | |
| clarkb | ok ssh host key accepted for new git08 on review.o.o and replication triggered | 22:57 |
| *** stevemar has quit IRC | 22:57 | |
| *** yamahata has joined #openstack-infra | 22:58 | |
| *** ramashri has quit IRC | 22:59 | |
| *** jamielennox is now known as jamielennox|away | 23:00 | |
| *** dkranz has quit IRC | 23:01 | |
| *** dkranz has joined #openstack-infra | 23:02 | |
| clarkb | hrm it still doesn't like the host key | 23:02 |
| *** megm has quit IRC | 23:03 | |
| clarkb | I am able to do it by hand and the key was accepted | 23:03 |
| clarkb | lets see if it gets better now | 23:03 |
| *** megm has joined #openstack-infra | 23:03 | |
| clarkb | oh! maybe gerrit has its own dns cache and its hitting the old host | 23:04 |
| *** ashleighfarnham has joined #openstack-infra | 23:04 | |
| clarkb | in the future I will try to preserve host keys I didn't realize it would be such a pain | 23:04 |
| *** annegentle has quit IRC | 23:04 | |
| *** dannywilson has quit IRC | 23:05 | |
| *** dkranz has quit IRC | 23:07 | |
| *** hodos|2 has quit IRC | 23:08 | |
| clarkb | fungi: ^ do you know if that is the case? | 23:08 |
| clarkb | a thread for gerrit 2.4 says you have to restart gerrit :/ | 23:09 |
| clarkb | you think I should do that or grab th ekeys off the old git08 and put them back in known_hosts? | 23:09 |
| clarkb | I might as well start that since restarting gerrit is sad times | 23:10 |
| *** baoli has quit IRC | 23:10 | |
| *** ramashri has joined #openstack-infra | 23:11 | |
| *** rlandy has quit IRC | 23:11 | |
| anteaya | :( | 23:13 |
| openstackgerrit | Merged openstack/requirements: Updated oslo_middleware to 2.8.0 https://review.openstack.org/217921 | 23:14 |
| *** tiswanso has quit IRC | 23:15 | |
| *** ramashri has quit IRC | 23:15 | |
| *** tiswanso has joined #openstack-infra | 23:15 | |
| *** ddieterly has quit IRC | 23:16 | |
| clarkb | ok that seems to hvae fixed it, I moved the rsa host key over | 23:17 |
| clarkb | lesson learned | 23:17 |
| clarkb | and I have readded the key to the known_hosts file for gerrit2 to be the old rsa key again | 23:17 |
| clarkb | so next gerrit reload will get the correct info | 23:17 |
| *** tonytan4ever has joined #openstack-infra | 23:19 | |
| *** tiswanso has quit IRC | 23:19 | |
| clarkb | ok replication completed | 23:20 |
| *** dkranz has joined #openstack-infra | 23:21 | |
| *** mtanino has quit IRC | 23:21 | |
| clarkb | https://review.openstack.org/218504 is ready when you are, though if you don't want that going in over a weekend that cool too | 23:21 |
| *** david-lyle has quit IRC | 23:24 | |
| *** tiswanso has joined #openstack-infra | 23:25 | |
| pleia2 | fungi: thanks re: certs | 23:25 |
| pleia2 | clarkb: I'm going to go ahead and give standing up this zanata instance a try | 23:26 |
| clarkb | pleia2: ok I am still here | 23:26 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/project-config: Add docs publish docs for grenade https://review.openstack.org/218513 | 23:26 |
| *** tjones has joined #openstack-infra | 23:27 | |
| anteaya | mtreinish: the link you added as a comment to 218513 points to 218513 | 23:28 |
| mtreinish | yes, see the follow up comment :) | 23:29 |
| anteaya | ah | 23:29 |
| mtreinish | anteaya: heh, I had the wrong tab open, the review pages looked the same | 23:31 |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config: Handle ssh failures with reboot in launch-node.py https://review.openstack.org/218514 | 23:31 |
| clarkb | and thats the updated launch-node.py to handle systemd | 23:32 |
| anteaya | mtreinish: it's all a blur | 23:33 |
| mtreinish | anteaya: I blame the beer | 23:35 |
| pleia2 | fungi: hit a quota limit, is there anything in DFW we can delete? (I see two ask.o.o servers) | 23:40 |
| *** amuller has quit IRC | 23:40 | |
| pleia2 | actually, getting to delete one mordred created | 23:40 |
| pleia2 | but FYI in general about hitting the quota limit, 100 of 100 created in DFW | 23:41 |
| *** stevemar has joined #openstack-infra | 23:42 | |
| *** ybathia has quit IRC | 23:43 | |
| pleia2 | now ram quota \o/ | 23:44 |
| clarkb | and I would like to keep the old centos6 git08 just in case we have to revert soon | 23:44 |
| pleia2 | might need to get rid of the old ask server afterall | 23:44 |
| *** sridhar_ram has quit IRC | 23:45 | |
| *** salv-orlando has quit IRC | 23:45 | |
| *** ajmiller has quit IRC | 23:46 | |
| *** sridhar_ram has joined #openstack-infra | 23:46 | |
| openstackgerrit | pangwa proposed openstack-infra/git-review: fix encoding issue on Windows https://review.openstack.org/218167 | 23:47 |
| anteaya | mtreinish: it helps I hear | 23:48 |
| pleia2 | yeah the ask servers are 4G nodes, so getting rid of the extra one will get us enough ram to launch translate.o.o | 23:48 |
| *** dimsum__ has joined #openstack-infra | 23:52 | |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder: Add support for zulip notifications. https://review.openstack.org/206200 | 23:53 |
| mordred | pleia2: what's our total ram count? | 23:53 |
| *** tjones has quit IRC | 23:54 | |
| pleia2 | mordred: Forbidden: Quota exceeded for ram: Requested 8192, but already used 1018880 of 1024000 ram (HTTP 403) | 23:54 |
| mordred | nice. | 23:55 |
| *** dims_ has quit IRC | 23:55 | |
| pleia2 | if it wasn't 15:55 on a friday before an airplane, I would be more amused by hitting both quota and ram limits independently on the same server deployment :) | 23:55 |
| *** tjones has joined #openstack-infra | 23:56 | |
| pleia2 | er 16:55 | 23:56 |
| *** Swami has quit IRC | 23:57 | |
| *** tjones has left #openstack-infra | 23:57 | |
| clarkb | I can always get it done on mondat | 23:57 |
| pleia2 | I'll be around for couple more hours, so should be able to manage | 23:58 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!