Wednesday, 2016-07-06

mtreinishclarkb: off the top of my head no, I'd have to dig it up in the nova code00:00
*** dfflanders_ has joined #openstack-infra00:00
*** Jeffrey4l_ has joined #openstack-infra00:00
*** adrian_otto has quit IRC00:01
*** adrian_otto has joined #openstack-infra00:01
mtreinishSpamapS: heh, that's also approaching the biggest python stack trace I've seen. It's only topped by ones with osprofiler that call itself a dozen or so times in the middle00:01
clarkbmtreinish: I can do the grepping. Such fun00:02
*** thorst has quit IRC00:02
*** ddieterly has joined #openstack-infra00:02
*** pahuang_ has joined #openstack-infra00:02
*** csomerville has quit IRC00:03
*** amitgandhinz has quit IRC00:03
*** aeng has quit IRC00:03
*** pahuang has quit IRC00:04
SpamapSmtreinish: indeed!00:04
*** kzaitsev_mb has quit IRC00:04
* SpamapS wonders why he feels proud of that00:04
*** sflanigan has joined #openstack-infra00:05
*** adrian_otto has quit IRC00:06
*** Sukhdev has quit IRC00:06
*** darvon has joined #openstack-infra00:06
clarkbnetwork_data.json appears to be more correct00:06
*** camunoz has quit IRC00:06
*** dingyichen has joined #openstack-infra00:07
clarkbI had confused vendor* with network*00:07
clarkbI wonder if network_info.json is a rax thing then nova used network_data.json00:08
*** csomerville has joined #openstack-infra00:08
*** salv-orlando has joined #openstack-infra00:08
*** harlowja has quit IRC00:10
clarkbsambetts|afk: ok I see now that everything gets stuff into a dict there anyways which is then passed along and sorted later00:12
clarkbwe should clean up that comment and all the sorted things I think00:12
clarkbto make it clearer they aren't required, but of course I have no idea if that will somehow break stuff00:12
mgagneclarkb: I think network_info.json is used by bifrost00:15
*** xenogear has quit IRC00:15
*** aeng has joined #openstack-infra00:15
mgagneclarkb: https://github.com/openstack/bifrost/blob/master/playbooks/roles/bifrost-configdrives-dynamic/tasks/main.yml#L4100:15
mgagneunfortunately, the official name found in Nova is "network_data.json"00:16
mgagneto be more precise: https://github.com/openstack/bifrost/blob/master/playbooks/roles/bifrost-configdrives-dynamic/tasks/main.yml#L53-L5700:16
clarkbmgagne: ya that is what I am discovering00:16
mgagnevendor data was used by RAX when Nova didn't have it yet00:16
clarkbmgagne: I think that may be because rax used network_info to start too?00:16
clarkboh00:17
mgagneclarkb: it was vendor data00:17
*** EricGonczer_ has quit IRC00:17
clarkbso we have 3 files all doing the same thing?00:17
mgagnewith a networkinfo key or something00:17
* clarkb bows down to the interface gods and asks forgiveness00:17
SpamapSmtreinish: whoa that's.. weird00:17
mgagnefun fact, with RAX, you will find the VLAN mac_address in the ethernet_mac_address field while with nova, it's in vlan_mac_address. :D00:18
*** Sukhdev has joined #openstack-infra00:18
mgagneI discovered it the hard way today :-/00:18
clarkbmgagne: maybe we should buy a goat and sacrifice it to get better interfaces00:19
clarkbor do a raindance or something00:19
clarkbhogepodge: ^ totally a defcore thing00:19
mgagneand it was supposed to be a "standard" interface for network config, now we have 3 standards :D00:20
mgagnebut yea, vendora_data.json is supposed to be RAX specific so I wouldn't count it. as for bifrost... ¯\_(ツ)_/¯00:21
anteaya+1 raindance00:21
openstackgerritEric Brown proposed openstack-infra/project-config: Enable py35 voting for Bandit project  https://review.openstack.org/33786200:21
*** salv-orlando has quit IRC00:22
notmorganmtreinish: lol00:22
mtreinishSpamapS: yeah it looked weird from my quick glance at it00:22
notmorganmtreinish: "just copy"00:22
notmorganhah00:22
mtreinishnotmorgan: well ignoring the migration specific parts :)00:22
*** camunoz has joined #openstack-infra00:23
notmorganmtreinish: yeah i was just going with something like that. just a little simplier00:23
*** mtanino has quit IRC00:27
pabelangerEmilienM: mtreinish: I haven't touch stackviz DIB element, it should be installed last I checked (few weeks ago)00:27
pabelangerclarkb: I am not sure about the spike00:27
pabelangerclarkb: about to test our puppet-iptables fix now00:27
clarkbpabelanger: kk00:28
*** ddieterly has quit IRC00:30
mtreinishpabelanger: ok, I wasn't sure. Like I said I wasn't watching it too closely. I just knew you were fighting issues with it a while ago00:31
openstackgerritMorgan Fainberg proposed openstack-infra/nodepool: Add alembic migration support  https://review.openstack.org/13658900:31
notmorganclarkb, jeblair: ^ need to add the "check modles vs schema" bit but that is most of it now.00:31
notmorganoh erm... missing something doh*00:32
notmorgan[and not tested locally yet] so i'll mark it wip00:32
notmorgan(doc change needed too)00:33
*** lvdongbing has joined #openstack-infra00:33
pabelangermtreinish: EmilienM: I'll follow up in the morning see what is going on00:33
pabelangerbut nodepool.o.o should show what is happening00:34
mtreinishEmilienM: oh, I knoew what it probably is, do you do the stackviz post processing in the puppet jobs?00:34
*** Apoorva has quit IRC00:34
mtreinishEmilienM: there is a devstack-gate step that builds the json output and copies the built js from nodepool to the log dir at the end of the run00:35
*** Sukhdev has quit IRC00:38
openstackgerritClark Boylan proposed openstack-infra/glean: Use constant for mac addr permanent type  https://review.openstack.org/33793100:38
openstackgerritClark Boylan proposed openstack-infra/glean: Sort output data once before we write it  https://review.openstack.org/33793200:38
openstackgerritClark Boylan proposed openstack-infra/glean: Explain difference between config drive files  https://review.openstack.org/33793300:38
openstackgerritClark Boylan proposed openstack-infra/glean: Explain vlan setup with interface names  https://review.openstack.org/33793400:38
clarkbgreghaynes: cinerama mgagne ^ I think I am making sense of some of the hard to read code there00:38
greghaynesclarkb: nice. I basically have to reread all of glean to remember what any of it does :(00:39
clarkbgreghaynes: ya thats what I am learning too, so hoping this is a start at making it readable00:40
clarkbits definitely not making everything understandable but I have to grok each peice as I go00:40
*** tonytan4ever has joined #openstack-infra00:42
*** mriedem_away has quit IRC00:45
clarkband hacking bites again00:46
* clarkb proposes we disable hacking on glean00:46
*** tonytan4ever has quit IRC00:47
*** piet has joined #openstack-infra00:47
*** salv-orlando has joined #openstack-infra00:48
openstackgerritClark Boylan proposed openstack-infra/glean: Explain vlan setup with interface names  https://review.openstack.org/33793400:50
openstackgerritClark Boylan proposed openstack-infra/glean: Explain difference between config drive files  https://review.openstack.org/33793300:50
*** csomerville has quit IRC00:50
openstackgerritClark Boylan proposed openstack-infra/glean: Sort output data once before we write it  https://review.openstack.org/33793200:50
openstackgerritClark Boylan proposed openstack-infra/glean: Use constant for mac addr permanent type  https://review.openstack.org/33793100:50
openstackgerritClark Boylan proposed openstack-infra/glean: Ignore hacking rules  https://review.openstack.org/33793700:50
*** amotoki has joined #openstack-infra00:50
*** pvaneck has quit IRC00:51
*** Goneri has joined #openstack-infra00:53
clarkbhrm maybe its just H not H*00:53
*** dberube has joined #openstack-infra00:53
*** dkehn__ has joined #openstack-infra00:53
* clarkb caves and runs tests locally like he should00:53
*** thorst has joined #openstack-infra00:54
*** thorst has quit IRC00:55
*** thorst has joined #openstack-infra00:56
*** dkehn_ has quit IRC00:56
pabelangerfungi: pleia2: xenial mirror rebuilt, vos release mirror.ubuntu -localauth in progress00:57
openstackgerritClark Boylan proposed openstack-infra/glean: Ignore hacking rules  https://review.openstack.org/33793700:58
openstackgerritClark Boylan proposed openstack-infra/glean: Explain vlan setup with interface names  https://review.openstack.org/33793400:58
openstackgerritClark Boylan proposed openstack-infra/glean: Explain difference between config drive files  https://review.openstack.org/33793300:58
openstackgerritClark Boylan proposed openstack-infra/glean: Sort output data once before we write it  https://review.openstack.org/33793200:58
openstackgerritClark Boylan proposed openstack-infra/glean: Use constant for mac addr permanent type  https://review.openstack.org/33793100:58
clarkbcentos image is almost done building, I will queue up the uploads as soon as it finishes00:59
*** amitgandhinz has joined #openstack-infra01:00
EmilienMmtreinish: https://review.openstack.org/#/c/337898/01:00
EmilienMI changed something01:01
*** kzaitsev_mb has joined #openstack-infra01:02
*** dberube has left #openstack-infra01:02
*** salv-orlando has quit IRC01:03
*** Goneri has quit IRC01:04
*** thorst has quit IRC01:04
*** amitgandhinz has quit IRC01:05
*** shashank_hegde has joined #openstack-infra01:07
*** harlowja has joined #openstack-infra01:09
*** Goneri has joined #openstack-infra01:09
openstackgerritPaul Belanger proposed openstack-infra/bindep: Do not merge  https://review.openstack.org/33794401:10
*** EricGonczer_ has joined #openstack-infra01:10
openstackgerritMerged openstack-infra/jenkins-job-builder: doc/definition.rst: add missing '/' into include path  https://review.openstack.org/33627401:11
*** shashank_hegde has quit IRC01:11
clarkbimage conversions so slow01:13
*** piet has quit IRC01:13
openstackgerritMerged openstack-infra/jenkins-job-builder: scm.py: correct indention to fix sphinx-docs warning  https://review.openstack.org/33627501:14
*** thorst has joined #openstack-infra01:15
openstackgerritKien Ha proposed openstack-infra/jenkins-job-builder: Add valid set of options parameters to convert_xml  https://review.openstack.org/33738101:17
openstackgerritKien Ha proposed openstack-infra/jenkins-job-builder: Add valid options for video-quality options to TestFairy  https://review.openstack.org/33794501:17
*** EricGonc_ has joined #openstack-infra01:17
openstackgerritHenry Gessau proposed openstack-infra/project-config: [WIP] Switch from py34 to py35 for all of neutron  https://review.openstack.org/33794601:18
*** thorst has quit IRC01:18
clarkband image uplaods for centos7 are queued01:19
clarkbshould start seeing happier jobs there in an hour or so01:19
*** EricGonczer_ has quit IRC01:20
*** tqtran has quit IRC01:22
pabelangerhttps://review.openstack.org/#/c/337889/01:24
*** rajinir has quit IRC01:24
pabelangerclarkb: that works locally as expected ^01:25
*** EricGonc_ has quit IRC01:25
pabelangerjhesketh: If you don't mind reviewing too^01:25
pabelangerthat will fix our firewall rules on centos-701:25
*** EricGonczer_ has joined #openstack-infra01:27
pabelangerand bindep is failing as expected now: http://logs.openstack.org/44/337944/1/check/gate-bindep-bindep-fallback-ubuntu-xenial/bbcf7da/console.html01:27
openstackgerritHenry Gessau proposed openstack-infra/project-config: [WIP] Switch from py34 to py35 for all of neutron  https://review.openstack.org/33794601:27
*** mtanino has joined #openstack-infra01:28
*** salv-orlando has joined #openstack-infra01:29
*** karthikp_ has joined #openstack-infra01:30
openstackgerritMadhumitha Viswanathan proposed openstack-infra/jenkins-job-builder: Add partial support for maven-release-plugin  https://review.openstack.org/31319601:32
openstackgerritEric Brown proposed openstack-infra/project-config: Enable py35 voting on oslo.vmware  https://review.openstack.org/33794901:32
*** Daisy has joined #openstack-infra01:33
*** Goneri has quit IRC01:37
*** weshay has quit IRC01:38
*** karthikp_ has quit IRC01:38
*** karthikp_ has joined #openstack-infra01:38
openstackgerritMerged openstack-infra/project-config: Raise ubuntu-xenial min-ready  https://review.openstack.org/33783801:38
*** zhurong has joined #openstack-infra01:39
*** chlong has quit IRC01:40
*** sarob has quit IRC01:40
*** sarob has joined #openstack-infra01:41
*** yanyanhu has joined #openstack-infra01:43
openstackgerritMerged openstack-infra/project-config: Make py35 jobs voting on tempest  https://review.openstack.org/33774801:44
*** salv-orlando has quit IRC01:45
*** tphummel has quit IRC01:48
*** Daisy has quit IRC01:49
*** Daisy has joined #openstack-infra01:49
*** Daisy_ has joined #openstack-infra01:50
*** moravec has quit IRC01:51
openstackgerritMerged openstack-infra/project-config: Remove php5 packages  https://review.openstack.org/33781601:52
pabelangerbindep fixed on xenial ^01:52
*** chlong has joined #openstack-infra01:53
*** Daisy has quit IRC01:54
*** moravec has joined #openstack-infra01:55
*** thorst has joined #openstack-infra01:56
*** thorst has quit IRC01:57
*** thorst has joined #openstack-infra01:58
*** esberglu has quit IRC01:59
*** Daisy_ has quit IRC02:00
*** Daisy has joined #openstack-infra02:00
*** karthikp_ has quit IRC02:00
*** Daisy has quit IRC02:01
*** Daisy has joined #openstack-infra02:01
*** elo has joined #openstack-infra02:01
*** karthikp_ has joined #openstack-infra02:02
*** tonytan4ever has joined #openstack-infra02:02
*** asselin_ has joined #openstack-infra02:05
*** thorst has quit IRC02:07
*** tonytan4ever has quit IRC02:07
*** karthikp_ has quit IRC02:08
*** swat30 has quit IRC02:08
pabelangerrcarrillocruz: jhesketh: AJaeger: if you don't mind landing https://review.openstack.org/#/c/337889/ before DIBs tomorrow, that would be great. It will fix our firewall issues on centos-702:08
*** karthikp_ has joined #openstack-infra02:08
pabelangerand fix our centos-7-2-node images02:09
*** asselin__ has quit IRC02:09
*** amotoki has quit IRC02:09
*** amotoki has joined #openstack-infra02:10
*** salv-orlando has joined #openstack-infra02:11
*** camunoz has quit IRC02:12
*** amitgandhinz has joined #openstack-infra02:13
openstackgerritfumihiko kakuma proposed openstack-infra/devstack-gate: Add neutron-dynamic-routing to PROJECT list  https://review.openstack.org/33795802:15
*** tphummel has joined #openstack-infra02:17
*** amitgandhinz has quit IRC02:18
jheskethpabelanger: looking02:20
*** aviau has quit IRC02:20
*** karthikp_ has quit IRC02:20
*** aviau has joined #openstack-infra02:21
*** tqtran has joined #openstack-infra02:21
*** rhallisey has quit IRC02:21
*** karthikp_ has joined #openstack-infra02:22
*** rfolco has quit IRC02:23
*** swat30 has joined #openstack-infra02:24
*** EricGonczer_ has quit IRC02:26
*** salv-orlando has quit IRC02:26
*** tqtran has quit IRC02:26
*** liusheng has quit IRC02:30
*** tonytan4ever has joined #openstack-infra02:31
*** aeng has quit IRC02:33
*** amoralej|pto has quit IRC02:33
*** sflanigan has quit IRC02:33
*** amoralej has joined #openstack-infra02:33
openstackgerritfumihiko kakuma proposed openstack-infra/project-config: Update tempest job for neutron-dynamic-routing  https://review.openstack.org/33796402:38
*** roxanagh_ has joined #openstack-infra02:43
*** Daisy_ has joined #openstack-infra02:46
*** sflanigan has joined #openstack-infra02:46
*** aeng has joined #openstack-infra02:47
*** roxanagh_ has quit IRC02:48
*** tonytan4ever has quit IRC02:48
*** solumuser__ has joined #openstack-infra02:48
*** Daisy has quit IRC02:49
*** tonytan4ever has joined #openstack-infra02:49
*** yuanying has quit IRC02:50
*** Daisy_ has quit IRC02:50
*** Daisy has joined #openstack-infra02:51
*** gyee has quit IRC02:52
*** salv-orlando has joined #openstack-infra02:52
*** tphummel has quit IRC02:54
*** Daisy has quit IRC02:56
*** PollyZ has joined #openstack-infra02:57
*** woodster_ has quit IRC02:59
*** kien-ha has joined #openstack-infra02:59
*** dfflanders_ has quit IRC03:02
*** karthikp_ has quit IRC03:03
*** karthikp_ has joined #openstack-infra03:04
*** thorst has joined #openstack-infra03:05
*** yamamoto has joined #openstack-infra03:05
*** roxanagh_ has joined #openstack-infra03:06
*** tonytan_brb has joined #openstack-infra03:06
*** salv-orlando has quit IRC03:07
*** coreyob has quit IRC03:08
*** PollyZ_ has joined #openstack-infra03:09
*** tonytan4ever has quit IRC03:09
*** coreyob has joined #openstack-infra03:10
*** thorst has quit IRC03:11
*** PollyZ has quit IRC03:11
*** asselin__ has joined #openstack-infra03:12
*** amitgandhinz has joined #openstack-infra03:14
*** asselin_ has quit IRC03:16
*** amitgandhinz has quit IRC03:18
*** julim has quit IRC03:21
*** karthikp_ has quit IRC03:21
*** amotoki has quit IRC03:26
openstackgerritKien Ha proposed openstack-infra/jenkins-job-builder: Update TestFairy plugin  https://review.openstack.org/33794503:26
*** karthikp_ has joined #openstack-infra03:27
*** roxanagh_ has quit IRC03:28
*** ayoung has quit IRC03:30
*** arif-ali has quit IRC03:32
*** Daisy has joined #openstack-infra03:32
*** salv-orlando has joined #openstack-infra03:33
*** zhurong has quit IRC03:35
*** amotoki has joined #openstack-infra03:35
*** karthikp_ has quit IRC03:36
*** zhurong has joined #openstack-infra03:36
*** karthikp_ has joined #openstack-infra03:37
*** Daisy has quit IRC03:37
*** sflanigan has quit IRC03:40
*** Daisy has joined #openstack-infra03:44
*** kzaitsev_mb has quit IRC03:44
*** DaiDV has joined #openstack-infra03:48
*** Daisy has quit IRC03:48
*** DaiDV has quit IRC03:48
*** yuanying has joined #openstack-infra03:49
*** karthikp_ has quit IRC03:49
*** salv-orlando has quit IRC03:49
*** karthikp_ has joined #openstack-infra03:50
*** Daisy has joined #openstack-infra03:51
*** PollyZ_ has quit IRC03:51
*** itisha has joined #openstack-infra03:52
*** arif-ali has joined #openstack-infra03:52
*** amotoki has quit IRC03:55
*** Daisy has quit IRC03:56
*** Daisy has joined #openstack-infra03:58
*** ramishra has quit IRC04:01
*** roxanagh_ has joined #openstack-infra04:01
*** Daisy has quit IRC04:02
*** ramishra has joined #openstack-infra04:02
*** aeng has quit IRC04:03
*** Daisy has joined #openstack-infra04:03
openstackgerritKhai Do proposed openstack-infra/puppet-gerrit: Allow puppet to configure gerrit commitmessage parameters  https://review.openstack.org/31617504:05
openstackgerritKhai Do proposed openstack-infra/puppet-gerrit: Allow puppet to configure gerrit commitmessage parameters  https://review.openstack.org/31617504:07
*** karthikp_ has quit IRC04:08
*** adrian_otto has joined #openstack-infra04:08
openstackgerritYAMAMOTO Takashi proposed openstack-infra/project-config: networking-midonet: switch to python-db-jobs  https://review.openstack.org/33555104:08
*** Daisy has quit IRC04:08
*** thorst has joined #openstack-infra04:10
*** dimtruck is now known as zz_dimtruck04:11
*** karthikp_ has joined #openstack-infra04:12
*** amotoki has joined #openstack-infra04:14
*** Daisy has joined #openstack-infra04:14
*** salv-orlando has joined #openstack-infra04:15
*** thorst has quit IRC04:16
*** solumuser__ has quit IRC04:18
*** Daisy has quit IRC04:19
*** aeng has joined #openstack-infra04:19
*** karthikp_ has quit IRC04:20
*** netsinn has quit IRC04:20
*** karthikp_ has joined #openstack-infra04:21
*** Daisy has joined #openstack-infra04:21
*** sputnik13 has quit IRC04:22
*** netsin has joined #openstack-infra04:23
*** tqtran has joined #openstack-infra04:23
*** amotoki has quit IRC04:25
*** zhurong has quit IRC04:25
*** Daisy has quit IRC04:26
*** Daisy has joined #openstack-infra04:26
*** tqtran has quit IRC04:28
*** asselin_ has joined #openstack-infra04:28
*** salv-orlando has quit IRC04:30
*** Daisy has quit IRC04:31
*** amitgandhinz has joined #openstack-infra04:31
*** amotoki has joined #openstack-infra04:31
*** asselin__ has quit IRC04:32
*** shashank_hegde has joined #openstack-infra04:36
*** amitgandhinz has quit IRC04:36
*** adrian_otto has quit IRC04:37
*** Daisy has joined #openstack-infra04:38
*** karthikp_ has quit IRC04:38
*** shashank_hegde has quit IRC04:38
*** shashank_hegde has joined #openstack-infra04:40
*** kzaitsev_mb has joined #openstack-infra04:41
*** karthikp_ has joined #openstack-infra04:42
*** psachin has joined #openstack-infra04:42
*** Daisy has quit IRC04:44
*** shashank_hegde has quit IRC04:44
*** zhurong has joined #openstack-infra04:45
*** Shrews has quit IRC04:45
*** kzaitsev_mb has quit IRC04:46
*** links has joined #openstack-infra04:48
openstackgerritKien Ha proposed openstack-infra/jenkins-job-builder: Update support for BuildResultTrigger plugin  https://review.openstack.org/32161904:50
*** mrmartin has joined #openstack-infra04:50
*** Daisy has joined #openstack-infra04:51
*** karthikp_ has quit IRC04:52
*** roxanagh_ has quit IRC04:53
*** karthikp_ has joined #openstack-infra04:54
*** Daisy has quit IRC04:56
*** salv-orlando has joined #openstack-infra04:57
openstackgerritMerged openstack-infra/project-config: Make ovn-native-l3 check job voting  https://review.openstack.org/33562404:57
*** Daisy has joined #openstack-infra04:57
*** exploreshaifali has joined #openstack-infra04:58
fricklerpabelanger: does https://review.openstack.org/337816 still need new xenial images being built before it takes effect? python35 jobs seem broken currently, probably since the xenial mirror rebuild04:59
fricklerhttp://logs.openstack.org/78/337878/1/check/gate-ceilometer-python35-nv/5535bf8/console.html#_2016-07-06_04_17_26_81685005:00
openstackgerritKien Ha proposed openstack-infra/jenkins-job-builder: Update FSTrigger plugin  https://review.openstack.org/32845605:00
*** salv-orlando has quit IRC05:01
*** karthikp_ has quit IRC05:04
*** salv-orlando has joined #openstack-infra05:06
*** DaiDV has joined #openstack-infra05:07
*** Daisy has quit IRC05:08
*** Shrews has joined #openstack-infra05:12
*** thorst has joined #openstack-infra05:13
openstackgerritKien Ha proposed openstack-infra/jenkins-job-builder: Update JaCoCo plugin  https://review.openstack.org/32075805:13
*** yamamoto has quit IRC05:14
*** Daisy has joined #openstack-infra05:15
openstackgerritKien Ha proposed openstack-infra/jenkins-job-builder: Update JaCoCo plugin  https://review.openstack.org/32075805:17
*** Daisy has quit IRC05:20
*** thorst has quit IRC05:21
*** Daisy has joined #openstack-infra05:22
*** aaltman_ has joined #openstack-infra05:22
*** aaltman has quit IRC05:22
*** Daisy_ has joined #openstack-infra05:24
*** sree has joined #openstack-infra05:25
*** oanson has joined #openstack-infra05:26
*** Daisy has quit IRC05:26
*** Nikolay_St has joined #openstack-infra05:28
*** dizquierdo has joined #openstack-infra05:29
*** Daisy has joined #openstack-infra05:29
*** asettle has joined #openstack-infra05:29
*** savihou has joined #openstack-infra05:30
*** Daisy_ has quit IRC05:32
*** amitgandhinz has joined #openstack-infra05:32
*** Daisy_ has joined #openstack-infra05:32
*** cbader has quit IRC05:33
*** cbader has joined #openstack-infra05:33
*** mtanino has quit IRC05:34
*** asettle has quit IRC05:34
*** Daisy has quit IRC05:34
*** amitgandhinz has quit IRC05:37
*** solumuser__ has joined #openstack-infra05:37
*** garyk1 has joined #openstack-infra05:38
*** garyk1 has quit IRC05:38
*** garyk1 has joined #openstack-infra05:39
*** sputnik13 has joined #openstack-infra05:40
*** mrmartin has quit IRC05:41
*** jswarren__ has quit IRC05:45
*** asselin__ has joined #openstack-infra05:47
*** zhenguo_ has joined #openstack-infra05:48
*** asselin_ has quit IRC05:51
*** armax has quit IRC05:54
*** yamamoto has joined #openstack-infra05:57
*** Daisy_ has quit IRC05:57
*** Daisy has joined #openstack-infra05:57
*** itisha has quit IRC06:00
*** Daisy has quit IRC06:00
*** dchen has joined #openstack-infra06:03
*** pcaruana has joined #openstack-infra06:05
*** aeng has quit IRC06:07
*** dingyichen has quit IRC06:07
*** rcernin has joined #openstack-infra06:09
*** asselin_ has joined #openstack-infra06:10
*** Daisy has joined #openstack-infra06:11
openstackgerritOpenStack Proposal Bot proposed openstack-infra/project-config: Normalize projects.yaml  https://review.openstack.org/33800406:11
*** asselin has joined #openstack-infra06:13
AJaegerfrickler: 337816 needs new images build, so tomorrow around this time.06:13
*** asselin__ has quit IRC06:13
*** jak-jak has joined #openstack-infra06:14
*** tonytan_brb has quit IRC06:14
jak-jakjoin ##chat06:14
AJaegerfrickler: indeed, that's now broken ;( we need to wait 24 hours roughly for new images.06:14
*** asselin__ has joined #openstack-infra06:14
AJaegerOr is any infra-root around that can rebuild ubuntu-xenial only and get those uploaded?06:14
*** andreas_s has joined #openstack-infra06:15
*** asselin_ has quit IRC06:16
*** aeng has joined #openstack-infra06:18
*** asselin has quit IRC06:18
*** thorst has joined #openstack-infra06:19
*** dingyichen has joined #openstack-infra06:19
*** nmagnezi has joined #openstack-infra06:22
AJaegerfrickler: that means that all python 3.5 jobs will fail, correct?06:23
AJaegerYeah ;(06:24
* AJaeger sends a note06:24
*** sree has quit IRC06:24
AJaeger#status notice All python 3.5 jobs are failing today, we need to build new xenial images first.06:26
openstackstatusAJaeger: sending notice06:26
*** thorst has quit IRC06:26
gmannAJaeger: does it mean, it will fail till tomorrow we have new image06:26
gmannAJaeger: most of the projects moved from nv to voting on that06:27
AJaegergmann: It will work tomorrow again once we have new images.06:27
-openstackstatus- NOTICE: All python 3.5 jobs are failing today, we need to build new xenial images first.06:27
AJaegergmann: Only one project moved already, others just proposed.06:27
gmannAJaeger: tempest one got merged06:28
gmannhttps://review.openstack.org/#/c/337748/06:28
*** jswarren has joined #openstack-infra06:28
jheskethAJaeger: I can rebuild images06:28
AJaegergmann: and that is the only one ;)06:28
jheskethAJaeger: do you know why they are failing? 337816 doesn't look related (or I'm missing something)06:28
AJaegerjhesketh: that would be great!06:28
gmannAJaeger: i see :)06:28
openstackstatusAJaeger: finished sending notice06:29
AJaegerjhesketh: See http://logs.openstack.org/78/337878/1/check/gate-ceilometer-python35-nv/5535bf8/console.html#_2016-07-06_04_17_26_81685006:29
fricklerjhesketh: there was an old php5-* package in the xenial mirror, that got cleaned up tonight06:29
*** sarob has quit IRC06:29
AJaegerjhesketh: : and bindep fallback list had this old php5 package in it.06:29
AJaegerSo, now we get "Package php5-cli is not available, but is referred to by another package."06:29
*** sarob has joined #openstack-infra06:30
AJaegerand the job exits06:30
AJaegerjhesketh: 337816 removes the package from the bindep fallback list and now everything should be fine again.06:30
AJaegergmann: if python 3.5 voting on tempest is a severe blocker, we can revert...06:31
gmannAJaeger: i was thinking about that, let's check with oomichi about that06:31
*** cshastri has joined #openstack-infra06:32
gmannoomichi: is it ok till tomorrow or should we revert 35 job to nv ?06:32
AJaegerlet'S see how long we need to update images, jhesketh is going to rebuild them now06:32
*** asselin_ has joined #openstack-infra06:32
*** sarob has quit IRC06:32
*** abregman has joined #openstack-infra06:33
*** amitgandhinz has joined #openstack-infra06:33
gmannAJaeger: ok, it can be soon(by today itself) then should be fine06:33
jheskethAJaeger: ah I see, okay will take al ook06:33
AJaegerthanks, gmann06:33
AJaegerjhesketh: so, xenial images only06:34
jheskethyep06:34
*** harlowja has quit IRC06:34
gmannAJaeger:  we have 5-6 patches with +A (mostly doc one )so we can wait06:35
*** asselin__ has quit IRC06:36
AJaegergmann: please tell people to wait with rechecking ;)06:37
*** amitgandhinz has quit IRC06:37
gmannAJaeger: yea06:38
*** isaacb has joined #openstack-infra06:38
*** asselin__ has joined #openstack-infra06:39
openstackgerritMerged openstack-infra/project-config: Normalize projects.yaml  https://review.openstack.org/33800406:41
*** Daisy has quit IRC06:42
*** Daisy has joined #openstack-infra06:42
*** kzaitsev_mb has joined #openstack-infra06:43
*** asselin_ has quit IRC06:43
*** hosanai has quit IRC06:44
*** rdx_matt has joined #openstack-infra06:45
*** tlian has quit IRC06:45
*** kien-ha has quit IRC06:48
*** kzaitsev_mb has quit IRC06:48
*** bryan_att has quit IRC06:53
*** dizquierdo has quit IRC06:54
*** Daisy has quit IRC06:55
*** Daisy has joined #openstack-infra06:56
*** oferby has quit IRC06:57
*** maishsk has joined #openstack-infra07:00
*** Daisy has quit IRC07:00
*** yanyanhu has quit IRC07:02
*** ccamacho has joined #openstack-infra07:03
*** amotoki has quit IRC07:03
*** yanyanhu has joined #openstack-infra07:04
*** pahuang_ has quit IRC07:06
*** shashank_hegde has joined #openstack-infra07:08
*** dtardivel has joined #openstack-infra07:09
*** salv-orlando has quit IRC07:10
*** amotoki has joined #openstack-infra07:12
jheskethhmm, the build ran for a while and then failed with no obvoius reason... http://paste.openstack.org/show/526510/07:13
jheskeththere's plenty of disk space, so I'm going to try again07:13
*** tonytan4ever has joined #openstack-infra07:14
*** slaweq has quit IRC07:18
*** mrmartin has joined #openstack-infra07:18
*** pahuang_ has joined #openstack-infra07:19
*** bexelbie has quit IRC07:19
*** zeih has joined #openstack-infra07:19
*** tonytan4ever has quit IRC07:20
*** tesseract- has joined #openstack-infra07:21
*** bexelbie has joined #openstack-infra07:21
*** Daisy has joined #openstack-infra07:24
*** thorst has joined #openstack-infra07:25
*** jlanoux has joined #openstack-infra07:26
*** shardy has joined #openstack-infra07:26
*** jlanoux has quit IRC07:26
*** flepied has joined #openstack-infra07:27
*** jlanoux has joined #openstack-infra07:27
openstackgerritGhanshyam Mann proposed openstack-infra/project-config: Revert "Make py35 jobs voting on tempest"  https://review.openstack.org/33805907:27
*** arif-ali has quit IRC07:28
openstackgerritGhanshyam Mann proposed openstack-infra/project-config: Revert "Make py35 jobs voting on tempest"  https://review.openstack.org/33805907:28
gmannAJaeger: can you check this - https://review.openstack.org/#/c/338059/07:28
openstackgerritXiang Chen proposed openstack/diskimage-builder: Give a more clear definition abount vm element in README  https://review.openstack.org/33806007:29
gmannAJaeger: i think its better to revert the tempest one and we can revert revert once new image is ready07:29
*** Daisy has quit IRC07:29
*** mikelk has joined #openstack-infra07:30
*** yolanda has joined #openstack-infra07:30
openstackgerritJin Jing Lin proposed openstack-infra/project-config: Enable Neutron VPNaaS API CI tests  https://review.openstack.org/33701107:31
*** thorst has quit IRC07:31
yolandagood morning07:31
*** hferenc has joined #openstack-infra07:33
gmannyolanda: good morning07:33
gmannyolanda:  https://review.openstack.org/#/c/338059/07:33
*** amitgandhinz has joined #openstack-infra07:33
gmannyolanda: can you please check this.07:34
yolandahi gmann07:35
yolanda+2... i see it's a blocker07:35
gmannyolanda: Thanks.07:36
*** Daisy has joined #openstack-infra07:36
*** eezhova has joined #openstack-infra07:36
*** dingyichen has quit IRC07:37
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Replace rcarrillocruz ssh key  https://review.openstack.org/33806707:37
*** dchen has quit IRC07:37
*** dizquierdo has joined #openstack-infra07:37
rcarrillocruzyolanda: mind reviewing ^ pls07:37
*** yolanda has quit IRC07:38
*** yolanda has joined #openstack-infra07:38
*** Daisy has quit IRC07:38
*** Daisy_ has joined #openstack-infra07:38
*** amitgandhinz has quit IRC07:39
*** ifarkas has joined #openstack-infra07:39
*** jpich has joined #openstack-infra07:41
*** zeih has quit IRC07:41
*** bexelbie has quit IRC07:42
*** yolanda has quit IRC07:43
*** bexelbie has joined #openstack-infra07:43
AJaegeryolanda, could you review https://review.openstack.org/#/c/337889/ , please? pabelanger asked for it07:45
AJaegergmann: will approve ...07:45
AJaegerjhesketh: that's sad ;( I have to leave now, so won't be around the next few hours...07:46
gmannAJaeger: Thanks. yolanda just +A07:46
*** zeih has joined #openstack-infra07:47
*** amotoki has quit IRC07:49
*** yolanda has joined #openstack-infra07:49
openstackgerritMerged openstack-infra/project-config: Revert "Make py35 jobs voting on tempest"  https://review.openstack.org/33805907:50
AJaegeryolanda, could you review https://review.openstack.org/#/c/337889/ , please? pabelanger asked for it07:51
yolandaAJaeger, sure07:52
AJaegerthanks, yolanda.07:52
AJaegeryolanda: FYI: We need new xenial images, all xenial builds are broken due to outdated bindep list and cleaned up xenial mirror. jhesketh is currently building images.07:52
*** kzaitsev_mb has joined #openstack-infra07:53
* AJaeger will be back later07:54
*** amotoki has joined #openstack-infra07:54
yolandaAJaeger, ok. I will be away for a bit as well, reinstalling SO07:54
*** slaweq has joined #openstack-infra07:54
*** eezhova has quit IRC07:54
*** eezhova has joined #openstack-infra07:56
*** zeih has quit IRC07:56
*** zeih has joined #openstack-infra07:57
*** abregman is now known as abregman|mtg07:58
*** garyk1 has quit IRC07:59
*** yolanda has quit IRC07:59
*** zzzeek has quit IRC08:00
*** zzzeek has joined #openstack-infra08:00
*** Daisy_ has quit IRC08:01
*** Daisy has joined #openstack-infra08:01
*** nbloom has joined #openstack-infra08:02
*** DaiDV has quit IRC08:03
openstackgerritGiulio Fidente proposed openstack-infra/tripleo-ci: Add Ceph to the OVB job  https://review.openstack.org/33808808:05
*** Daisy has quit IRC08:06
jheskeththe dib build failed again, trying to debug08:07
*** Daisy has joined #openstack-infra08:07
rcarrillocruzjhesketh: i'm seeing failures when cloning puppet-openstackid08:07
rcarrillocruzlooks like a mirror doesn't have that repo sane08:08
rcarrillocruzhttp://paste.openstack.org/show/526515/08:08
*** garyk1 has joined #openstack-infra08:08
rcarrillocruzwhich would explain failures like http://logs.openstack.org/67/338067/1/check/gate-infra-puppet-apply-ubuntu-precise/4c4c6c2/console.html08:08
jheskethrcarrillocruz: hmm, it seems to be missing refs for some reason http://git.openstack.org/cgit/openstack-infra/puppet-openstackid/refs/08:09
rcarrillocruzif you reload , you may eventually get refs08:10
*** zeih has quit IRC08:10
rcarrillocruzfirst time i clicked showed none, refreshed and now show refs08:10
rcarrillocruzi still have to figure out how afs mirrors work in infra upstream, but looks to me as a mirror being sick08:10
jheskethah, that indicates it's a git slave having trouble08:10
rcarrillocruz++08:11
*** zeih has joined #openstack-infra08:11
*** Daisy has quit IRC08:11
*** dizquierdo has quit IRC08:13
*** chem has joined #openstack-infra08:13
*** Daisy has joined #openstack-infra08:14
*** sorantis has joined #openstack-infra08:15
*** hashar has joined #openstack-infra08:15
*** csatari has quit IRC08:17
*** csatari has joined #openstack-infra08:17
*** abregman|mtg is now known as abregman08:19
*** DaiDV has joined #openstack-infra08:20
*** kzaitsev_mb has quit IRC08:22
*** zeih has quit IRC08:22
*** derekh has joined #openstack-infra08:24
*** rubasov has left #openstack-infra08:24
*** lucas|afk is now known as lucasagomes08:25
*** shashank_hegde has quit IRC08:26
*** hferenc has quit IRC08:27
*** zeih has joined #openstack-infra08:27
*** thorst has joined #openstack-infra08:29
*** hferenc has joined #openstack-infra08:30
*** BobBall_AWOL is now known as BobBall08:32
*** sree_ has joined #openstack-infra08:34
*** dizquierdo has joined #openstack-infra08:34
*** sree_ is now known as Guest4952808:34
*** Daisy_ has joined #openstack-infra08:34
*** amitgandhinz has joined #openstack-infra08:34
*** asettle has joined #openstack-infra08:35
*** asettle has quit IRC08:36
*** asettle has joined #openstack-infra08:36
*** thorst has quit IRC08:36
*** Daisy has quit IRC08:37
*** Guest49528 has quit IRC08:39
*** priteau has joined #openstack-infra08:39
*** amitgandhinz has quit IRC08:39
fricklerjhesketh: there was some disk error affecting the rax-iad mirror yesterday, maybe that git slave issue is related08:40
jheskethfrickler: thanks08:43
jheskethit appears that the repo is corrupt on a few git slaves, but for some reason I can't get gerrit replicate a fresh repo across... still looking into it08:43
*** asettle has quit IRC08:44
*** Daisy has joined #openstack-infra08:46
*** asettle has joined #openstack-infra08:47
*** zeih has quit IRC08:47
*** zeih has joined #openstack-infra08:48
*** dizquierdo has quit IRC08:48
*** Daisy_ has quit IRC08:49
*** yanyanhu has quit IRC08:52
*** ramishra has quit IRC08:52
openstackgerritGiulio Fidente proposed openstack-infra/tripleo-ci: Add Ceph to the OVB job  https://review.openstack.org/33808808:53
*** yanyanhu has joined #openstack-infra08:54
*** ramishra has joined #openstack-infra08:54
*** dizquierdo has joined #openstack-infra08:57
*** Daisy has quit IRC08:58
*** Daisy has joined #openstack-infra08:59
openstackgerritArtur Zarzycki proposed openstack-infra/project-config: Add fuel-ccp-ironic and fuel-ccp-searchlight  https://review.openstack.org/33721608:59
*** electrofelix has joined #openstack-infra09:00
*** yolanda has joined #openstack-infra09:03
*** Daisy has quit IRC09:03
*** Daisy has joined #openstack-infra09:04
*** mhickey has joined #openstack-infra09:06
penguinologhttp://logs.openstack.org/98/338098/1/check/gate-fuel-devops-python35-nv/4000336/console.html -- oops, packages install again?09:08
*** Daisy has quit IRC09:08
*** sputnik13 has quit IRC09:09
*** acoles_ is now known as acoles09:09
*** Daisy has joined #openstack-infra09:10
*** makowals has joined #openstack-infra09:10
*** sputnik13 has joined #openstack-infra09:11
openstackgerritamrith proposed openstack-infra/irc-meetings: Update trove meeting information  https://review.openstack.org/33812709:13
*** dfflanders has quit IRC09:15
*** sshnaidm|afk is now known as sshnaidm09:15
*** john-davidge has joined #openstack-infra09:15
openstackgerritDimitri Mazmanov proposed openstack-infra/project-config: Add python-kingbirdclient to the project list  https://review.openstack.org/33725409:15
*** dfflanders has joined #openstack-infra09:16
*** tonytan4ever has joined #openstack-infra09:16
*** john-dav_ has joined #openstack-infra09:17
openstackgerritRob Cresswell proposed openstack-infra/irc-meetings: Update Horizon Meeting times  https://review.openstack.org/33813009:17
*** john-dav_ is now known as john-davidge_09:17
*** eezhova has quit IRC09:18
openstackgerritamrith proposed openstack-infra/irc-meetings: Proposing OpenStack SWG Meeting  https://review.openstack.org/33813409:19
*** john-davidge has quit IRC09:20
*** Daisy has quit IRC09:20
*** tonytan4ever has quit IRC09:21
*** jianghuaw has joined #openstack-infra09:22
*** DaiDV has quit IRC09:22
*** huanxie has joined #openstack-infra09:22
*** SamYaple has quit IRC09:23
jheskethrcarrillocruz, frickler: okay git01 and git06 are repaired... let me know how you go cloning/using puppet-openstackid09:23
*** eezhova has joined #openstack-infra09:24
*** amotoki has quit IRC09:24
rcarrillocruzcool, let me recheck my changes09:24
*** esikachev has joined #openstack-infra09:24
*** tqtran has joined #openstack-infra09:25
*** Daisy has joined #openstack-infra09:26
openstackgerritamrith proposed openstack-infra/irc-meetings: Proposing OpenStack SWG Meeting  https://review.openstack.org/33813409:26
*** tqtran has quit IRC09:29
*** Hal has joined #openstack-infra09:30
*** Hal is now known as Guest2562709:30
*** kzaitsev_mb has joined #openstack-infra09:30
*** sambetts|afk is now known as sambetts09:30
*** Daisy has quit IRC09:31
openstackgerritMerged openstack-infra/release-tools: add a "ptl" command to show info about how to contact a PTL  https://review.openstack.org/33346209:34
*** thorst has joined #openstack-infra09:35
*** amitgandhinz has joined #openstack-infra09:35
*** ociuhandu has joined #openstack-infra09:36
*** Daisy has joined #openstack-infra09:38
*** mgould|afk is now known as mgould09:38
*** DaiDV has joined #openstack-infra09:38
*** amitgandhinz has quit IRC09:40
AJaegerpenguinolog: see my email on openstack-dev and the previous notice about needing new xenial images.09:41
*** thorst has quit IRC09:42
*** Daisy has quit IRC09:42
*** lvdongbing has quit IRC09:43
*** Daisy has joined #openstack-infra09:43
*** zhurong has quit IRC09:44
*** Daisy has quit IRC09:48
openstackgerritSam Betts proposed openstack-infra/project-config: Add Ironic multinode job  https://review.openstack.org/33249009:48
*** ociuhandu has quit IRC09:48
AJaegerjhesketh: did the xenial images build fine now?09:48
*** Daisy has joined #openstack-infra09:49
*** daemontool has joined #openstack-infra09:49
jheskethAJaeger: no, just debugging them.. they are failing fetching bulkify so I suspect it's a mirror issue09:49
jheskethAJaeger: http://paste.openstack.org/show/526524/ if you're interested09:50
AJaegerjhesketh: was it perhaps removed? You know that pabelanger recreated the mirror yesterday since we had a mirror of the beta tree and kept old packages in it.09:50
AJaegerAh, that's an external package ;(09:50
jheskethit's on the mirrors and I'm not sure why it resolves to an external link there.. (which works for me)09:51
AJaegerworks for me as well09:51
*** eezhova has quit IRC09:51
*** tonytan4ever has joined #openstack-infra09:52
openstackgerritMerged openstack-infra/project-config: Add new project Fuel CCP  https://review.openstack.org/33558409:54
*** sorantis has quit IRC09:56
rcarrillocruzpabelanger , clarkb : i've sorted out with the NET team access to baremetal0009:57
rcarrillocruzlet me know later if you can access ( you should )09:57
*** nwkarsten has joined #openstack-infra09:57
*** daemontool has quit IRC09:58
openstackgerritMerged openstack-infra/project-config: Publish os-performance-tools docs  https://review.openstack.org/31700509:58
*** daemontool has joined #openstack-infra09:58
*** Daisy has quit IRC09:59
*** zeih has quit IRC09:59
openstackgerritMerged openstack-infra/project-config: Add channel openstack-swg  https://review.openstack.org/33789910:00
yolandaAJaeger, can you check https://review.openstack.org/#/c/337105/2 ?10:00
*** links has quit IRC10:00
*** nwkarsten has quit IRC10:01
*** zeih has joined #openstack-infra10:02
*** notnownikki has joined #openstack-infra10:02
openstackgerritOleksii Zamiatin proposed openstack-infra/project-config: Turn on additional configurations for zmq f-tests  https://review.openstack.org/33710510:03
*** Daisy has joined #openstack-infra10:05
jamespageyolanda, ta :-)10:08
yolandajamespage, np10:09
openstackgerritMerged openstack-infra/project-config: Add cursive library  https://review.openstack.org/33770910:10
*** Daisy has quit IRC10:10
*** hashar has quit IRC10:11
*** yanyanhu has quit IRC10:11
*** Guest25627 has quit IRC10:12
*** sorantis has joined #openstack-infra10:12
*** sputnik13 has quit IRC10:13
openstackgerritMerged openstack-infra/project-config: Run identity v3 only job in ceilometer  https://review.openstack.org/31522410:16
*** Daisy has joined #openstack-infra10:17
AJaegeryolanda: good catch, looks fine now IMHO.10:17
openstackgerritMerged openstack-infra/project-config: Run Ceilometer functional tests in Identity v3-only  https://review.openstack.org/31507710:17
openstackgerritMerged openstack-infra/system-config: Replace hpuseast name for chocolate  https://review.openstack.org/33058010:18
*** sshnaidm is now known as sshnaidm|afk10:18
yolandaAJaeger, approved10:18
openstackgerritMerged openstack-infra/project-config: Add install-guide job to Aodh  https://review.openstack.org/33735710:19
openstackgerritMerged openstack-infra/project-config: Add Magnum functional test for k8s ironic  https://review.openstack.org/33738610:19
openstackgerritMerged openstack-infra/project-config: Add CI jobs for murano-plugin-networking-sfc  https://review.openstack.org/33711310:19
rcarrillocruzjhesketh: the two changes i was getting failures cloning puppet-openstackid went thru , thx10:21
*** DaiDV has quit IRC10:21
rcarrillocruznow i hoping i can merge my new ssh key and root access changes10:21
*** huanxie has quit IRC10:21
*** Daisy has quit IRC10:22
*** Daisy has joined #openstack-infra10:22
*** jianghuaw has left #openstack-infra10:23
openstackgerritAdam Coldrick proposed openstack-infra/storyboard: Allow superusers to delete comments  https://review.openstack.org/33816410:26
openstackgerritAdam Coldrick proposed openstack-infra/storyboard: Allow editing and deletion of comments to be disabled  https://review.openstack.org/33816510:26
openstackgerrityolanda.robla proposed openstack-infra/project-config: Start using infra element instead of puppet  https://review.openstack.org/29516810:26
openstackgerrityolanda.robla proposed openstack-infra/shade: Add new test with betamax for create flavors  https://review.openstack.org/30219210:26
openstackgerrityolanda.robla proposed openstack-infra/shade: Update delete flavor test to work with betamax  https://review.openstack.org/30345910:26
openstackgerrityolanda.robla proposed openstack-infra/shade: Add new test for deleting non-existing flavors  https://review.openstack.org/30351510:26
jheskethrcarrillocruz: cool, sounds like it's fixed10:27
openstackgerrityolanda.robla proposed openstack-infra/shade: Add test for delete flavor exception with betamax  https://review.openstack.org/30352910:27
openstackgerrityolanda.robla proposed openstack-infra/shade: Update test for setting flavor specs to use betamax  https://review.openstack.org/30406410:27
openstackgerrityolanda.robla proposed openstack-infra/shade: Update test_unset_flavor_specs to use betamax  https://review.openstack.org/30407310:27
openstackgerrityolanda.robla proposed openstack-infra/shade: Add shade call to get flavor access list  https://review.openstack.org/30417110:27
*** Daisy has quit IRC10:27
openstackgerrityolanda.robla proposed openstack-infra/shade: Add new test for list flavors based on betamax  https://review.openstack.org/30571810:27
*** Daisy has joined #openstack-infra10:28
openstackgerritMerged openstack-infra/project-config: Fuel-mirror project is deprecated  https://review.openstack.org/33586810:28
openstackgerritMerged openstack-infra/project-config: Add repositories for Charm layers  https://review.openstack.org/33488010:28
openstackgerritMerged openstack-infra/project-config: Add repositories for Charm interfaces  https://review.openstack.org/33450210:29
openstackgerritMerged openstack-infra/project-config: Fix branches for reno jobs of oslo projects  https://review.openstack.org/33712810:29
openstackgerritFerenc Horváth proposed openstack-infra/project-config: Add job which builds test images with watchdog support  https://review.openstack.org/32983210:29
*** salv-orlando has joined #openstack-infra10:31
*** Daisy has quit IRC10:32
*** Daisy has joined #openstack-infra10:34
*** mrmartin has quit IRC10:36
*** amitgandhinz has joined #openstack-infra10:36
*** Daisy has quit IRC10:38
*** rossella_s has quit IRC10:39
*** rossella_s has joined #openstack-infra10:39
*** Daisy has joined #openstack-infra10:39
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Add compute090.chocolate metadata for better tracking  https://review.openstack.org/33759510:40
*** thorst has joined #openstack-infra10:40
*** amitgandhinz has quit IRC10:41
*** jed56 has joined #openstack-infra10:41
*** salv-orlando has quit IRC10:42
*** Daisy has quit IRC10:44
*** Daisy has joined #openstack-infra10:45
*** salv-orlando has joined #openstack-infra10:45
*** thorst has quit IRC10:46
*** vsaienko1 has left #openstack-infra10:47
*** Daisy has quit IRC10:49
*** Daisy has joined #openstack-infra10:50
openstackgerritMerged openstack-infra/project-config: Turn on additional configurations for zmq f-tests  https://review.openstack.org/33710510:51
*** yamamoto has quit IRC10:54
*** ociuhandu has joined #openstack-infra10:55
*** amotoki has joined #openstack-infra10:55
*** Daisy has quit IRC10:55
*** john-davidge_ is now known as john-davidge10:55
*** eezhova has joined #openstack-infra10:55
*** zeih has quit IRC10:56
*** Daisy has joined #openstack-infra10:56
*** tpsilva has joined #openstack-infra10:56
*** amotoki has quit IRC10:56
*** asettle has quit IRC10:57
*** asettle has joined #openstack-infra10:57
*** zeih has joined #openstack-infra10:57
*** Daisy has quit IRC11:00
*** asettle has quit IRC11:02
*** zeih has quit IRC11:02
fricklerjhesketh: do we have a snapshot of the old xenial mirror? maybe reverting the mirror state would be a feasible workaround until the new image builds are succeeding11:03
*** coolsvap has joined #openstack-infra11:05
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Add rcarrillocruz as infra-root  https://review.openstack.org/33769211:11
rcarrillocruzyolanda: ^ pls11:11
*** weshay has joined #openstack-infra11:14
*** lucasagomes is now known as lucas|hungry11:17
*** sshnaidm|afk is now known as sshnaidm11:19
*** erlon has joined #openstack-infra11:19
openstackgerritMerged openstack-infra/system-config: Replace rcarrillocruz ssh key  https://review.openstack.org/33806711:22
*** links has joined #openstack-infra11:24
*** zeih has joined #openstack-infra11:25
*** asettle has joined #openstack-infra11:25
*** swat30 has quit IRC11:26
openstackgerritLiam Young proposed openstack-infra/irc-meetings: Add Juju Openstack Charms Meeting  https://review.openstack.org/33818711:26
*** swat30 has joined #openstack-infra11:27
yolandarcarrillocruz, sure11:28
*** claudiub has joined #openstack-infra11:28
*** zeih has quit IRC11:29
openstackgerritMerged openstack-infra/tripleo-ci: Fix tempest configuration  https://review.openstack.org/33199711:33
*** ldnunes has joined #openstack-infra11:34
openstackgerritMarkus Zoeller (markus_z) proposed openstack-infra/release-tools: update the script to expire old bug reports  https://review.openstack.org/33764211:36
*** esikachev has quit IRC11:36
rcarrillocruzthx11:40
*** rodrigods has quit IRC11:41
*** rodrigods has joined #openstack-infra11:41
*** mrmartin has joined #openstack-infra11:41
*** kzaitsev_mb has quit IRC11:42
*** yamamoto has joined #openstack-infra11:42
*** esikachev has joined #openstack-infra11:47
*** deadnull has joined #openstack-infra11:49
*** thorst has joined #openstack-infra11:50
coolsvapcan someone help kick a spam guest user in #openstack-meeting11:52
*** thorst_ has joined #openstack-infra11:52
coolsvapwe have requirements meeting in 8 minutes11:52
*** thorst__ has joined #openstack-infra11:52
*** zeih has joined #openstack-infra11:53
*** markusry has joined #openstack-infra11:53
*** aysyd has joined #openstack-infra11:54
*** thorst has quit IRC11:54
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: Update overcloud log retrieval of rh2  https://review.openstack.org/33820911:55
*** thorst_ has quit IRC11:56
*** rfolco has joined #openstack-infra11:56
*** Nikolay_St has quit IRC11:56
*** kzaitsev_mb has joined #openstack-infra11:57
*** hrybacki|afk is now known as hrybacki11:58
*** zeih has quit IRC11:58
*** salv-orl_ has joined #openstack-infra11:59
*** Daisy has joined #openstack-infra11:59
*** esikachev has quit IRC12:00
*** amoralej is now known as amoralej|lunch12:01
*** EricGonczer_ has joined #openstack-infra12:01
*** salv-orlando has quit IRC12:01
*** permalac has joined #openstack-infra12:02
*** exploreshaifali has quit IRC12:02
*** yamamoto has quit IRC12:02
*** EricGonc_ has joined #openstack-infra12:03
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: Remove Fedora support from CI scripts  https://review.openstack.org/33821012:03
*** salv-orl_ has quit IRC12:03
*** DaiDV has joined #openstack-infra12:03
*** weshay is now known as weshay_mtg12:04
*** DaiDV has quit IRC12:04
*** Daisy has quit IRC12:04
*** savihou has quit IRC12:05
*** DaiDV has joined #openstack-infra12:05
*** Daisy has joined #openstack-infra12:05
*** kzaitsev_mb has quit IRC12:06
*** EricGonczer_ has quit IRC12:07
*** dprince has joined #openstack-infra12:08
*** markusry has quit IRC12:08
*** markusry has joined #openstack-infra12:08
*** psilvad has joined #openstack-infra12:08
*** Daisy has quit IRC12:09
*** vgridnev has quit IRC12:10
*** yamamoto has joined #openstack-infra12:10
*** Daisy has joined #openstack-infra12:10
*** leifmadsen_ is now known as leifmadsen12:12
*** ldnunes_ has joined #openstack-infra12:12
*** raildo-afk is now known as raildo12:14
*** Daisy has quit IRC12:15
*** ldnunes has quit IRC12:15
*** dprince has quit IRC12:16
*** dprince has joined #openstack-infra12:16
*** Daisy has joined #openstack-infra12:16
*** rhallisey has joined #openstack-infra12:16
openstackgerritArie Bregman proposed openstack-infra/shade: Allow to register nodes from file  https://review.openstack.org/33821612:18
*** yamamoto has quit IRC12:18
*** yamamoto has joined #openstack-infra12:19
*** nmagnezi has quit IRC12:19
*** Daisy has quit IRC12:20
*** lucas|hungry is now known as lucasagomes12:21
*** Daisy has joined #openstack-infra12:22
*** vgridnev has joined #openstack-infra12:22
*** ccamacho has quit IRC12:24
*** amitgandhinz has joined #openstack-infra12:25
*** ccamacho has joined #openstack-infra12:25
*** Daisy has quit IRC12:26
*** kzaitsev_mb has joined #openstack-infra12:27
*** tonytan4ever has quit IRC12:27
*** savihou has joined #openstack-infra12:27
*** hashar has joined #openstack-infra12:27
*** Daisy has joined #openstack-infra12:27
*** berendt has joined #openstack-infra12:27
openstackgerritNeil Jerram proposed openstack-infra/project-config: Add experimental CI test for networking-calico  https://review.openstack.org/33822412:28
*** amitgandhinz has quit IRC12:29
*** thorst__ is now known as thorst12:30
*** yamamoto has quit IRC12:31
*** yamamoto has joined #openstack-infra12:31
*** Daisy has quit IRC12:32
*** yamamoto has quit IRC12:32
*** Daisy has joined #openstack-infra12:33
*** yamamoto has joined #openstack-infra12:34
openstackgerritArie Bregman proposed openstack-infra/shade: Allow to register nodes from file  https://review.openstack.org/33821612:34
*** Daisy has quit IRC12:38
*** tonytan4ever has joined #openstack-infra12:38
*** Daisy has joined #openstack-infra12:38
*** zz_dimtruck is now known as dimtruck12:39
*** esikachev has joined #openstack-infra12:39
openstackgerritLiam Young proposed openstack-infra/irc-meetings: Add Juju Openstack Charms Meeting  https://review.openstack.org/33818712:40
AJaegerjhesketh: I fear that xenial builds failed again since we merged some repos, didn't it?12:40
*** zeih has joined #openstack-infra12:41
*** ramishra has quit IRC12:43
*** Daisy has quit IRC12:43
*** bryan_att has joined #openstack-infra12:43
*** rlandy has joined #openstack-infra12:43
openstackgerritLiam Young proposed openstack-infra/irc-meetings: Add Juju Openstack Charms Meeting  https://review.openstack.org/33818712:43
*** ramishra has joined #openstack-infra12:44
*** cshastri has quit IRC12:49
*** pradk has joined #openstack-infra12:49
*** eharney has joined #openstack-infra12:49
*** zeih has quit IRC12:51
savihouHello Core Reviewers, apologies again for nudging - would apreciate your help with my problematic kid - https://review.openstack.org/#/c/263026/ , Thanks!12:53
*** kushal has joined #openstack-infra12:53
*** kushal has joined #openstack-infra12:53
*** kzaitsev_mb has quit IRC12:54
*** zeih has joined #openstack-infra12:55
*** Daisy has joined #openstack-infra12:55
openstackgerritAdam Coldrick proposed openstack-infra/storyboard-webclient: Allow comments to be edited by their owner  https://review.openstack.org/33341812:55
*** kzaitsev_mb has joined #openstack-infra12:56
AJaegersavihou: for which repository is your review?12:57
*** tlian has joined #openstack-infra12:57
savihouAJaeger: Apologies, wrong channel :)12:58
AJaegerAh...12:58
*** Daisy has quit IRC12:59
openstackgerritNeil Jerram proposed openstack-infra/project-config: Add experimental CI test for networking-calico  https://review.openstack.org/33822413:00
*** yamamoto has quit IRC13:00
*** Goneri has joined #openstack-infra13:01
openstackgerritLiam Young proposed openstack-infra/irc-meetings: Add Juju Openstack Charms Meeting  https://review.openstack.org/33818713:01
jheskethAJaeger: it was failing to get that package still but I haven't figured out why.. what merged repos are you refering to?13:01
*** abe_music has joined #openstack-infra13:02
*** mdrabe has joined #openstack-infra13:03
*** isaacb has quit IRC13:03
*** weshay_mtg is now known as weshay13:04
openstackgerritAdam Coldrick proposed openstack-infra/storyboard: Allow editing and deletion of comments to be disabled  https://review.openstack.org/33816513:04
*** AL34N1X has joined #openstack-infra13:04
*** woodster_ has joined #openstack-infra13:04
AJaegerjhesketh: looking at the log file, it seems it failed since it tried to access new repos while those were just beeing created - the usual problem if we approve new repos while images are building.13:04
*** cshastri has joined #openstack-infra13:04
AJaegerOh, I just see the log file is continuing...13:05
AJaegerbut very large today - 18 M compared to yesterdays 5 M13:05
jheskethAJaeger: oh right.. I just kicked it off again so it might be that one13:05
jheskethAJaeger: yeah I had a few runs all failing at various networks or repos13:06
AJaegerAh, so it'S appending to the same log? Ok...13:06
*** Daisy has joined #openstack-infra13:06
AJaegerOk, let's see what happens. Thanks, jhesketh13:06
jheskethyep, all appended... I could rotate them manually13:07
jheskethyep13:07
*** trown|outtypewww is now known as trown13:07
*** kzaitsev_mb has quit IRC13:07
*** jtomasek_ has joined #openstack-infra13:08
*** dane-fichter has joined #openstack-infra13:09
*** Daisy has quit IRC13:10
dane-fichterHey if anyone on the infra team has a sec, could you add me (dane-fichter) to the cursive-core and cursive-release Gerrit groups?13:10
dane-fichterAJaeger: ^13:11
*** dfflanders has quit IRC13:11
AJaegerdane-fichter: I don't have those permissions, an infra-root needs to do this.13:11
*** jcoufal has joined #openstack-infra13:11
*** Daisy has joined #openstack-infra13:12
dane-fichterAJaeger: How do I get in touch with someone on that team?13:12
yolandahi dane-fichter , i can help13:13
dane-fichterhi yolanda13:13
dane-fichterthanks13:13
AJaegerdane-fichter: just asking here is enough as you see ;)13:13
yolandadane-fichter, do you have the change that you sent when you created that project handy?13:13
*** abe_music has left #openstack-infra13:14
*** amoralej|lunch is now known as amoralej13:14
dane-fichteryolanda: https://review.openstack.org/#/c/337709/13:14
gnuoyCould someone help me understand why Jenkins is nack'ing https://review.openstack.org/#/c/338187 ? If I follow the link to the Jenkins run it seems to show a beutifully formatted page showing details of the meeting but I can't see an error. I'm sure I'm missing something obvious.13:15
yolandadane-fichter,  i see that you have two accounts. One with the email in caps, another with email in lower case, what's prefered one?13:15
dane-fichteryolanda: Oh god this again. hold on let me check13:15
*** solumuser__ has quit IRC13:16
AJaegergnuoy: go to http://logs.openstack.org/87/338187/4/check/gate-irc-meetings-tox-ical/c1f72dd/console.html to get the log file13:16
dane-fichteryolanda: lower case please.13:16
*** Daisy has quit IRC13:16
*** kgiusti has joined #openstack-infra13:16
gnuoyAJaeger, thanks.13:16
*** mrmartin has quit IRC13:16
yolandadone13:17
dane-fichteryolanda: thanks so much13:17
yolandaanytime13:17
dane-fichterfor what it's worth y'all are extremely responsive and awesome13:17
*** zhurong has joined #openstack-infra13:18
*** inc0 has joined #openstack-infra13:20
fungiAJaeger: that job used to only publish index.html if it passed, i thought13:20
fungii wonder if we somehow screwed it up in redoing the publishers for zuul v2.513:21
*** _ari_ has joined #openstack-infra13:22
*** inc0_ has joined #openstack-infra13:22
*** sigmavirus_away is now known as sigmavirus13:24
*** mhickey has quit IRC13:24
*** inc0 has quit IRC13:25
*** amitgandhinz has joined #openstack-infra13:25
*** zeih has quit IRC13:25
*** tonytan4ever has quit IRC13:26
*** tqtran has joined #openstack-infra13:26
*** tonytan4ever has joined #openstack-infra13:27
*** tonytan_brb has joined #openstack-infra13:27
*** kzaitsev_mb has joined #openstack-infra13:27
*** njohnston_ has joined #openstack-infra13:27
*** tonytan4ever has quit IRC13:28
AJaegerfungi: ah - let's review it.13:28
*** rhallisey has quit IRC13:29
*** bexelbie has quit IRC13:29
AJaegerfungi, the job has "copy-after-failure: true"13:29
*** amitgandhinz has quit IRC13:30
*** yamamoto has joined #openstack-infra13:30
*** tqtran has quit IRC13:30
*** rhallisey has joined #openstack-infra13:30
AJaegerfungi, could you double check this? In infra.yaml13:30
*** isaacb has joined #openstack-infra13:30
AJaegerfungi, did you read backscroll about xenial images? OR do you want a short summary?13:30
*** amitgandhinz has joined #openstack-infra13:31
*** njohnston_ is now known as njohnston13:32
*** Daisy has joined #openstack-infra13:32
*** esberglu has joined #openstack-infra13:32
puiterwijkjeblair: hi, just checking in if you had any updates or questions about Ipsilon?13:33
*** dkehn__ is now known as dkehn_13:34
fungiAJaeger: yeah, i caught up on the xenial discussion13:35
rcarrillocruzhey folks, i'll be off my afternoon/evening, my son is sick and i'll be taking care of him13:35
rcarrillocruzfungi: got an email from a HPE program manager to have a call about the infracloud hw move13:36
*** esikachev has quit IRC13:36
rcarrillocruzi'll try to move it tomorrow and ask him for people from here to attend13:36
rcarrillocruzall good?13:36
fungipuiterwijk: smarcet should be back around this week, and we needed to find out what extensions to openid were implemented in openstackid which foundation web properties might be relying on13:36
fungircarrillocruz: yep, works for me. thanks!13:36
puiterwijkfungi: okay, sounds good. Feel free to let him ping me if he wants to discuss about it or anything13:37
AJaegerrcarrillocruz: all the best for your son, don't get ill yourself!13:37
*** asettle has quit IRC13:37
*** cardeois has joined #openstack-infra13:37
*** Daisy has quit IRC13:37
*** hashar has quit IRC13:37
rcarrillocruzindeed, cos when toddlers get sick *somehow* parents get sick multiplied by two or three :S13:37
*** amotoki has joined #openstack-infra13:37
*** asettle has joined #openstack-infra13:37
*** ayoung has joined #openstack-infra13:38
*** coreycb` has quit IRC13:38
*** hashar has joined #openstack-infra13:38
openstackgerritMerged openstack-infra/irc-meetings: Update trove meeting information  https://review.openstack.org/33812713:38
*** oanson has quit IRC13:38
*** coreycb has joined #openstack-infra13:38
*** solumuser__ has joined #openstack-infra13:38
openstackgerritAdam Coldrick proposed openstack-infra/storyboard-webclient: Allow comments to be edited by their owner  https://review.openstack.org/33341813:38
*** mhickey has joined #openstack-infra13:39
*** itisha has joined #openstack-infra13:39
*** pt_15 has joined #openstack-infra13:40
AJaegerrcarrillocruz: been there often enough ;(13:40
*** kzaitsev_mb has quit IRC13:41
openstackgerritMerged openstack-infra/irc-meetings: Update Horizon Meeting times  https://review.openstack.org/33813013:41
*** EricGonc_ has quit IRC13:42
ShrewsTheJulia: you may want to review 33821613:42
*** nwkarsten has joined #openstack-infra13:42
*** dimtruck is now known as zz_dimtruck13:43
*** kzaitsev_mb has joined #openstack-infra13:44
*** kushal has quit IRC13:44
*** Daisy has joined #openstack-infra13:44
openstackgerritRaildo Mascena proposed openstack-infra/project-config: Adds ironic v3 only job  https://review.openstack.org/30430613:47
jheskethAJaeger, fungi: looks like xenial is stuck on doing the npm install again (it's about to fail)13:48
AJaegersad panda ;(13:48
*** Daisy has quit IRC13:48
*** r-mibu has quit IRC13:49
*** mrmartin has joined #openstack-infra13:49
*** r-mibu has joined #openstack-infra13:49
jheskethI don't think I'm going to have time to debug further sorry13:49
*** Daisy has joined #openstack-infra13:50
fricklerfungi: is there any chance to get the xenial mirror back to the state where it had the php5 pkgs? that should get jobs running until fixed images are available13:50
AJaegerfungi, could you take over from jhesketh, please?13:53
*** Daisy has quit IRC13:54
*** eezhova has quit IRC13:54
*** mriedem has joined #openstack-infra13:54
fricklerfungi: if you are missing the packages, it seems I have them in some snapshots on my local xenial mirror13:54
TheJuliaShrews: thank you13:54
*** jlanoux has quit IRC13:54
*** xyang1 has joined #openstack-infra13:54
*** Daisy has joined #openstack-infra13:55
*** eezhova has joined #openstack-infra13:56
*** esikachev has joined #openstack-infra13:57
*** links has quit IRC13:57
AJaegerfungi, jhesketh : Can we reach the https://registry.npmjs.org/bulkify/-/bulkify-1.1.1.tgz from the nodepool server itself?13:57
*** nbloom has quit IRC13:58
*** xarses has quit IRC13:58
*** garyk1 has quit IRC13:58
fungifrickler: the problem with that idea is that what we were testing wasn't really xenial anyway13:58
fungiAJaeger: i think krot_vaca_jul19 said something about npm being semi-broken on xenial13:59
*** rbrndt has joined #openstack-infra13:59
AJaeger;(13:59
*** Daisy has quit IRC13:59
fungifrequent timeouts retrieving packages or something13:59
jheskethAJaeger: yes14:00
fungii can wget that url fine from nodepool.o.o but the npm version in the xenial chroot might be choking on it still14:00
AJaegerSo, how can we then build images with stackviz? Disable stackviz completely - or only for xenial?14:00
AJaegerthanks for confirming that wget works14:00
*** Daisy has joined #openstack-infra14:01
fungiwe could probably move the stackviz element up into the nodepool config (no longer depend on it in other elements) and then not include it on xenial14:01
fungior we could add a conditional to no-op in its element14:01
fungior... there are really countless options to make that happen14:01
timothyb89this may be an issue I can fix in stackviz itself?14:03
*** AL34N1X has quit IRC14:03
fungitimothyb89: perhaps. you could make it not use npm?14:03
*** julim has joined #openstack-infra14:03
fungidunno how integral javascript is to stackviz's design14:03
timothyb89ah, well, that may be more difficult... but if its one failing dependency I may be able to work around it14:04
AJaegertimothyb89: failure is http://paste.openstack.org/show/526524/14:04
timothyb89sorry, still catching up on the issue14:04
*** savihou has quit IRC14:04
*** PollyZ has joined #openstack-infra14:04
AJaegertimothyb89: so, npn imstall gets 502 on installing bulkify14:04
*** savihou has joined #openstack-infra14:04
fungitimothyb89: i think the issue is that npm on ubuntu 16.04 hits arbitrary timeouts at random (or semi-random)14:04
fricklerfungi: iiuc your xenial mirror was including pre-release packages of php5*, that got removed for final xenial release and now the jobs trying to install them are broken. I can offer you a snapshot of xenial repos from 20160329 that still has the packages. you could use these until the patch to not install php5 on xenial can take effect14:05
*** Daisy has quit IRC14:05
*** zz_dimtruck is now known as dimtruck14:05
fungifrickler: sure, but that's not xenial. might as well just run jobs on trusty and install some newer packages14:05
fungiit was a preview of xenial with old packages14:06
timothyb89fungi: it looks like its getting http errors from the server, it may be some degradation on npm's end, but probably not a timeout on ours14:06
*** Daisy has joined #openstack-infra14:06
fungiwe should have never started relying on it in that state, but we didn't know14:06
fungijhesketh: what's the npm error we're getting on that lib?14:08
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Disable stackviz on xenial for now  https://review.openstack.org/33827614:08
AJaegerfungi, I do not see a dependency on stackviz but I'm not a nodepool expert. What do you think of the change above? ^14:08
*** SamYaple has joined #openstack-infra14:08
timothyb89fungi, AJaeger: if it's just the one job failure, we may just be able to retry the image build? I'm not seeing any signs its anything but a random failure on npm's end14:09
fungiAJaeger: perfect. the other half of what i was suggesting was apparently already the case14:09
*** jamesmcarthur has joined #openstack-infra14:09
AJaegertimothyb89: jhesketh has retried several times today already ;(14:09
*** thiagop has joined #openstack-infra14:09
fungitimothyb89: yeah, retries are apparently still hitting that error14:09
timothyb89oh, sorry, didn't see those logs14:10
AJaegertimothyb89: see http://nodepool.openstack.org/dib.ubuntu-xenial.log for logs of retries14:10
AJaegertimothyb89: that paste just contained the essence.14:10
AJaegertimothyb89: it takes 90 mins to build an image - and it aborts after one hour or so ;(14:10
*** Daisy has quit IRC14:10
fungii said "timeouts" earlier but really i don't recall much of the issue krot_vaca_jul19 was tracking other than npm is unreliable on xenial. i'm looking for the details on that now14:11
AJaegertimothyb89: and right now we have xenial completely broken for non-dsvm jobs - so we need to fix it.14:11
*** cbader has quit IRC14:11
*** dangers_away has quit IRC14:11
*** electrofelix has quit IRC14:11
timothyb89AJaeger: alright, makes sense. I suspect all we can do is wait for that npm mirror to update its cache14:11
*** solumuser__ has quit IRC14:12
*** dangers_away has joined #openstack-infra14:12
*** cbader has joined #openstack-infra14:12
*** fernnest_ has joined #openstack-infra14:12
*** Daisy has joined #openstack-infra14:12
*** jtomasek_ has quit IRC14:12
*** caowei has joined #openstack-infra14:12
jeblairfungi, AJaeger: would reverting to yesterday's xenial images help?14:12
timothyb89AJaeger, fungi: would it make sense to change the behavior of the stackviz script so it doesn't cause the whole job to fail? I think that should be an easy enough change14:12
*** isaacb has quit IRC14:12
*** hashar_ has joined #openstack-infra14:12
AJaegerjeblair: not at all. We recreated the xenial mirror14:12
*** hashar_ has quit IRC14:13
AJaegerAnd our old images (all of them) have in the bindep list a reference to packages in the xenial mirror.14:13
*** AL34N1X has joined #openstack-infra14:13
AJaegerSo, the only alternative is adding those broken packages to our mirror.14:13
fungijeblair: nope. the problem is that the bindep fallback list embedded in images up to now expect packages which are no longer ()and never should have been) on the mirror14:13
*** hashar_ has joined #openstack-infra14:13
jeblairgotcha14:13
*** fernnest has quit IRC14:14
*** hashar has quit IRC14:14
*** hongbin has joined #openstack-infra14:14
fungiapparently we synced the mirror before xenial was released, and ubuntu deleted packages from xenial between then and the release but reprepro doesn't remove them in that situation14:14
fungiand so we had jobs installing packages which ubuntu didn't officially include in xenial14:15
fungiand didn't realize it until yesterday14:15
*** amotoki has quit IRC14:15
AJaegerwe should have first removed the entries from bindep; create new images and then recreated the mirror. But overlooked that first recreating the mirrors broke everything.14:16
AJaegerAFAIK it's only python 3.5 and those jobs are non-voting...14:16
AJaegerThe only voting change was reverted already so that tempest can continue to merge.14:17
fungii think some projects move to xenial for their jobs already for newer versions of other things14:17
fungier, moved14:17
*** Daisy has quit IRC14:17
AJaegerfungi: yeah - but those are using devstack, not bindep fallback I guess.14:17
* AJaeger didn't check14:17
fungiahh, fair point most of them _are_ devstack-based i think14:17
*** Daisy has joined #openstack-infra14:17
AJaegerAt least there was nobody here yet complaining ;)14:18
fungiwell, frickler seems pretty urgent about something14:18
funginot sure if it's over a voting non-py35 job14:19
fungior something else14:19
mtreinishAJaeger: I was gonna before the revert :)14:19
jheskethfungi: 502 (I can pull up the logs if you need)14:19
*** mdrabe has quit IRC14:20
AJaegerfungi: he showed nova and ceilometer logs for py35. But might be others he's interested in as well...14:20
fungijhesketh: nah, i was going to try to hunt down any npm bugs that might explain that error14:20
fungibut now my workstation is severely overheating. i may need to drop offline for a while and try to hack a replacement fan into it somehow14:21
*** sorantis has quit IRC14:21
*** Daisy has quit IRC14:22
AJaeger;(14:23
fungidownside to these ultra-compact computer models. unless they rely exclusively on passive cooling they include impossible to find super--tiny fans14:23
mtreinishfungi: sounds like it's time for you to water cool it14:24
*** zhurong has quit IRC14:24
fungithat's so messy14:24
*** xarses has joined #openstack-infra14:25
*** isaacb has joined #openstack-infra14:25
fungifor many years my younger brother was into active fluid cooling for overclocking purposes. it's a serious pain to maintain14:25
mtreinishfungi: heh, not as messy as sticking it in a fish tank filled with mineral oil, which is another option :)14:25
openstackgerritTim Buckley proposed openstack-infra/project-config: Ignore errors during stackviz build  https://review.openstack.org/33828814:26
timothyb89fungi, AJaeger, jhesketh, mtreinish: ^^^14:26
abregmanI was asked to "add a reno release note" in shade with one of my patches. I have no idea what is the process to do that..can anyone explain?14:26
timothyb89that patch should help work around future errors like this14:26
*** mriedem1 has joined #openstack-infra14:26
AJaegerabregman: let me get you a link...14:26
mtreinishabregman: http://docs.openstack.org/developer/reno/usage.html#creating-new-release-notes14:26
*** mriedem has quit IRC14:26
abregmanmtreinish, AJaeger: thank you :)14:26
AJaegerthanks, mtreinish ! You're too fast ;)14:27
fungii have a super-tiny fan i took out of a radio transmitter that didn't need it because i'm operating it in a very low-power mode, and though it's not the same size as the dead fan in this machine and i probably won't be able to reassemble the case fully with it in there, it's probably better than the present situation14:27
*** aarefiev has quit IRC14:28
mtreinishAJaeger: heh, it was just in my recent browser history. I've had to share it a few time the past few weeks14:28
*** mriedem1 is now known as mriedem14:29
mtreinishfungi: yeah, it's probably better than nothing14:29
fungieven having closed out all non-essential applications and just running some xterms and simple console tools the cores have auto-scaled down to ~200mhz and all the thermocouples are reading 100c/alarm crit14:29
fungiload average is only ~0.25 right now14:29
*** psachin has quit IRC14:30
*** aarefiev has joined #openstack-infra14:30
* fungi will be afk for 30 minutes or so14:30
*** gordc has joined #openstack-infra14:31
*** mdrabe has joined #openstack-infra14:31
AJaegertimothyb89: thanks for the change. Let's see what others say to it. Once that patch is in, we can revert mine. I'm not really happy about silent fails as you change does.14:31
fungioh, before i disappear... is devstack-gate relying on stackviz presence on images now? if so, rebuilding xenial without stackviz might break all devstack-based jobs on xenial?14:32
timothyb89AJaeger: neither am I, but npm is depressingly unreliable :(14:32
timothyb89fungi: devstack-gate will simply not build stackviz if /opt/stackviz doesn't exist14:33
AJaegerthanks for confirming, timothyb8914:33
fungitimothyb89: okay, so we'll fail gracefully without stackviz on images. good to know14:33
*** sdake has joined #openstack-infra14:33
*** AL34N1X has quit IRC14:33
*** amotoki has joined #openstack-infra14:34
Shrewsabregman: you should read http://docs.openstack.org/infra/shade/coding.html14:34
*** Daisy has joined #openstack-infra14:34
*** psachin has joined #openstack-infra14:35
openstackgerritEvgeny Sikachev proposed openstack-infra/project-config: Added missed exports to sahara-cli job  https://review.openstack.org/33829214:36
mtreinishfungi: https://www.pugetsystems.com/submerged.php :)14:36
*** adrian_otto has joined #openstack-infra14:37
*** Daisy has quit IRC14:38
openstackgerritTim Buckley proposed openstack-infra/project-config: Ignore errors during stackviz build  https://review.openstack.org/33828814:38
*** rossella_s has quit IRC14:39
timothyb89fungi, AJaeger: ^^^ a few more safety checks while we're at it14:39
*** rossella_s has joined #openstack-infra14:39
*** yamahata has joined #openstack-infra14:39
dulekHi, can I get some infra cores feedback on devstack-gate patch: https://review.openstack.org/#/c/327619/ ?14:41
AJaegerjeblair: still happy with 338288?14:41
*** Goneri has quit IRC14:42
jeblairyep14:42
*** exploreshaifali has joined #openstack-infra14:42
*** weshay is now known as weshay_afk14:43
*** exploreshaifali has quit IRC14:44
AJaegerSo, should we take timothyb89's change - and abandon my https://review.openstack.org/338276 for rebuild?14:44
* AJaeger needs to leave now - I'm happy with whatever you all decide on doing - and will check in later tonight again14:45
timothyb89AJaeger: it may still be best to put yours through if we need images rebuilt ASAP14:45
*** Daisy has joined #openstack-infra14:45
*** caowei has quit IRC14:45
*** PollyZ has quit IRC14:46
*** exploreshaifali has joined #openstack-infra14:46
*** PollyZ has joined #openstack-infra14:46
*** hashar_ is now known as hashar14:49
*** Daisy has quit IRC14:50
*** EricGonczer_ has joined #openstack-infra14:50
*** armax has joined #openstack-infra14:51
*** Daisy has joined #openstack-infra14:51
*** mhickey has quit IRC14:54
openstackgerritMerged openstack-infra/project-config: Disable stackviz on xenial for now  https://review.openstack.org/33827614:55
*** Daisy has quit IRC14:56
*** trown is now known as trown|brb14:56
*** Daisy has joined #openstack-infra14:56
*** adrian_otto has quit IRC14:57
*** slaweq has quit IRC14:57
*** maishsk has quit IRC14:58
*** Swami has joined #openstack-infra14:58
*** Swami_ has joined #openstack-infra14:58
*** trown|brb is now known as trown15:00
*** pilgrimstack has quit IRC15:00
*** njohnston has left #openstack-infra15:00
*** Julien-zte has joined #openstack-infra15:00
*** Daisy has quit IRC15:01
*** eezhova has quit IRC15:01
*** karthikp_ has joined #openstack-infra15:01
*** isaacb has quit IRC15:01
*** Daisy has joined #openstack-infra15:02
*** eezhova has joined #openstack-infra15:03
*** nwkarsten has quit IRC15:04
openstackgerritArie Bregman proposed openstack-infra/shade: Allow to register nodes from file  https://review.openstack.org/33821615:04
*** nwkarsten has joined #openstack-infra15:04
openstackgerritAdam Coldrick proposed openstack-infra/storyboard: Allow editing and deletion of comments to be disabled  https://review.openstack.org/33816515:04
*** ddieterly has joined #openstack-infra15:05
*** pcaruana has quit IRC15:06
ShrewsSpamapS: since you had some comments on https://review.openstack.org/319395, will let you decide if it's ready to go15:06
*** mhickey has joined #openstack-infra15:06
*** Daisy has quit IRC15:07
*** harlowja has joined #openstack-infra15:07
*** Daisy has joined #openstack-infra15:07
*** tphummel has joined #openstack-infra15:08
*** PollyZ has quit IRC15:08
openstackgerritArie Bregman proposed openstack-infra/shade: Allow to register nodes from file  https://review.openstack.org/33821615:08
*** nwkarsten has quit IRC15:09
*** cbader has quit IRC15:09
*** nmagnezi has joined #openstack-infra15:10
*** psachin has quit IRC15:10
*** rcernin has quit IRC15:10
*** cshastri has quit IRC15:10
*** psachin has joined #openstack-infra15:10
*** Daisy has quit IRC15:12
*** Daisy has joined #openstack-infra15:13
slaglepabelanger: is there a way to test experimental jobs with this? https://review.openstack.org/#/c/337889/ or do we need to just wait for it to merge?15:13
slaglei wasn't sure if a depends-on would work15:14
openstackgerritMerged openstack-infra/tripleo-ci: Update overcloud log retrieval of rh2  https://review.openstack.org/33820915:14
*** gsilvis_ is now known as gsilvis15:14
*** Swami__ has joined #openstack-infra15:18
EmilienMShrews: is your async work merged in ansible ? I was not sure15:18
*** Daisy has quit IRC15:18
*** rajinir has joined #openstack-infra15:18
*** dprince has quit IRC15:18
*** kzaitsev_mb has quit IRC15:18
*** tonytan_brb is now known as tonytan4ever15:18
*** Daisy has joined #openstack-infra15:19
*** yamamoto has quit IRC15:19
*** kzaitsev_mb has joined #openstack-infra15:20
*** yamamoto has joined #openstack-infra15:20
ShrewsEmilienM: neither mine nor mordred's have merged yet (mordred: you should add a 'ready_for_review' comment on yours to get the needs_rebase tag off)15:20
EmilienMShrews: ack15:21
EmilienMShrews: /me waiting for https://review.openstack.org/#/c/334464/15:21
pabelangermorning15:21
*** Swami has quit IRC15:21
pabelangerjust catching up on backscroll now15:21
*** esikachev has quit IRC15:22
*** Daisy has quit IRC15:23
*** PollyZ has joined #openstack-infra15:23
pabelangerAJaeger: Right, we rebuilt of xenial mirror last night as a results the php5 packages were removed.  Which would break python35 jobs using bindep fallback15:24
*** Daisy has joined #openstack-infra15:24
therveWe're getting failures in heat when trying to boot up a qcow image in integration tests15:24
thervehttp://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Unable%20to%20convert%20image%20to%20raw.%5C%2215:25
*** inc0_ has quit IRC15:25
therveStarted a lot around 2 days ago15:25
ShrewsEmilienM: there is an ansible core team meeting tomorrow. if i can remember, i'll bring those PRs up in the meeting15:25
*** Kiall has quit IRC15:25
*** inc0 has joined #openstack-infra15:25
pabelangerslagle: need to wait, I'll refresh our centos-7 image too today15:25
*** mtanino has joined #openstack-infra15:26
*** weshay_afk is now known as weshay15:26
*** rhallisey has quit IRC15:27
therveDoes any other project test booting qcow images?15:27
*** Kiall has joined #openstack-infra15:28
slaglepabelanger: ack, thx15:28
*** andreas_s has quit IRC15:30
*** nmagnezi has quit IRC15:31
silehtdansmith, dtroyer , Hi, can I get your though about grenade stuffs ? https://review.openstack.org/#/c/335363/15:31
openstackgerritTim Buckley proposed openstack-infra/project-config: Ignore errors during stackviz build  https://review.openstack.org/33828815:33
openstackgerritGiulio Fidente proposed openstack-infra/tripleo-ci: Add hyperconverged Ceph to the HA job  https://review.openstack.org/33808815:33
clarkbtherve: the nodepool tests do, not sure if tempest is using qcow2 cirroa15:34
greghaynestherve: theres a handful of projects which do15:34
*** salv-orlando has joined #openstack-infra15:34
*** Daisy has quit IRC15:34
therveclarkb, greghaynes: OK, thanks. Weird that we're the only ones seeing that, I'm not sure what's specific about the Heat test15:35
therveMaybe a slightly bigger image15:35
clarkbor format mismatches somehow15:35
pabelangerfungi: AJaeger: I restarted the ubuntu-xenial builds15:35
clarkbsince qcow2 has multiple incompatible versions15:36
greghayneshaha, I really hope that is gone15:36
therveclarkb, Which would depend on what? qemu tools version?15:36
*** watersoul_ has quit IRC15:36
therveIt's not a systematic error, unfortunately or not15:36
*** watersoul has joined #openstack-infra15:37
clarkbtherve: qemu-img iirc15:37
pabelangerfungi: AJaeger: So, I am thinking for stackviz, we ignore errors in the elements and continue regardless if stackviz installs.15:37
*** mriedem1 has joined #openstack-infra15:37
pabelangerfungi: AJaeger: Then, more the build process outside of DIBs into a periodic job where we can download the bits from tarballs.o.o15:37
*** makowals has quit IRC15:38
openstackgerritMerged openstack-infra/puppet-iptables: Ensure service logic run regardless of using chroot  https://review.openstack.org/33788915:38
*** vhosakot has joined #openstack-infra15:39
mtreinishclarkb: http://logs.openstack.org/09/337809/5/check/gate-tempest-dsvm-neutron-full/bc63d52/logs/devstacklog.txt.gz#_2016-07-06_10_41_00_119 it's not qcow in the gate15:39
*** mriedem has quit IRC15:39
*** mriedem1 is now known as mriedem15:39
*** Daisy has joined #openstack-infra15:41
clarkbmtreinish: that is sadness15:41
*** Goneri has joined #openstack-infra15:41
* clarkb is not aware of any clouds that use that image format15:41
mtreinishclarkb: amazon does...15:41
*** rhallisey has joined #openstack-infra15:41
clarkbmtreinish: openstack clouds :P15:41
clarkbwe use a bunch of clouds and its all raw, vhd, or qcow215:41
mtreinishI didn't realize that we used it either. I knew we had a bunch of legacy stuff in tempest for uploading images in that format (which I think I removed all of)15:42
*** watersoul_ has joined #openstack-infra15:42
*** watersoul has quit IRC15:42
clarkbpabelanger: AJaeger timothyb89 what can I do to help get new xenial images built?15:43
*** amitgandhinz has quit IRC15:43
*** adrian_otto has joined #openstack-infra15:43
*** amitgandhinz has joined #openstack-infra15:43
*** jogo has quit IRC15:43
mtreinishclarkb: if there are published qcow or raw cirros images it should be pretty straightforward to pivot devstack to use those instead15:44
pabelangerclarkb: AJaeger: I'm working on the uploads now15:44
*** mriedem1 has joined #openstack-infra15:44
*** mikelk has quit IRC15:44
clarkbpabelanger: doesn't look like we have disabled stackviz in the builds. Does this mean you got an image to build with stackviz?15:44
jeblairpabelanger: agree.  this makes us ignore stackviz failures: https://review.openstack.org/33828815:45
therveclarkb, I meant qcow2 btw, sorry15:45
jeblairpabelanger, clarkb: this has merged and should disable stackviz: https://review.openstack.org/33827615:45
*** jogo has joined #openstack-infra15:45
pabelangerjeblair: ah, great. I was just working on that too15:45
*** Daisy has quit IRC15:46
clarkbtherve: yes, we definitely have jobs that boot qcow2's. Nodepools integration testing, dibs image build testing and so on. I thought that tempest might also use qcow2 but was wrong about that15:46
*** Sukhdev has joined #openstack-infra15:46
clarkbjeblair: thanks15:46
therveclarkb, Do you build image on the fly for those tests?15:47
clarkbtherve: yes15:47
therveWe're using a pre built image, that may be where the issue resides15:47
*** mriedem has quit IRC15:47
therveIf there is some kind of version mismatch15:47
clarkbtherve: is it a consistent 100% of the time fail or less consistent?15:47
therveclarkb, No much less than that15:48
therveAbout 40 times in the 2 days, no idea how many tests run we have, but more than that15:48
clarkblooks like it is happening on multiple clouds so not a cloud specific issue15:49
therveYeah. More on ovh though15:49
*** sambetts is now known as sambetts|afk15:49
*** rcernin has joined #openstack-infra15:49
*** dane-fichter has quit IRC15:49
openstackgerritAnita Kuno proposed openstack-infra/storyboard: Add example commands for the Tasks api  https://review.openstack.org/33787915:49
*** asettle has quit IRC15:51
*** Daisy has joined #openstack-infra15:52
*** dizquierdo has quit IRC15:52
greghaynestherve: octavia is another one who I know does this15:53
openstackgerritDavid TARDIVEL proposed openstack-infra/project-config: Add check-requirement to watcher-dashboard project  https://review.openstack.org/33836615:53
openstackgerritArie Bregman proposed openstack-infra/shade: Allow to register nodes from file  https://review.openstack.org/33821615:53
*** mriedem1 is now known as mriedem15:54
clarkbhrm no world dump on that failure?15:54
clarkbtherve: http://logs.openstack.org/11/338211/2/check/gate-heat-dsvm-functional-orig-mysql-lbaasv2/2f90ffe/logs/screen-n-cpu.txt.gz?level=TRACE#_2016-07-06_13_52_17_000 is the error from nova compute15:54
openstackgerritAnita Kuno proposed openstack-infra/storyboard: Add example commands for the Timeline api  https://review.openstack.org/33785415:55
openstackgerritVictor Ryzhenkin proposed openstack-infra/project-config: Add openstack/fuel-plugin-murano-tests project  https://review.openstack.org/33215115:55
clarkbits definitely an io error of some sort15:55
clarkbpossibly run out of disk space?15:55
*** roxanagh_ has joined #openstack-infra15:56
*** Daisy has quit IRC15:56
*** trown is now known as trown|lunch15:56
therveThe df output looks normal...15:57
clarkbtherve: where is the df output?15:57
therveclarkb, http://logs.openstack.org/98/338198/2/check/gate-heat-dsvm-functional-convg-mysql-lbaasv2/5fbda19/logs/df.txt.gz15:57
*** Rockyg has joined #openstack-infra15:57
clarkboh its got its own file15:58
clarkbI was looking for the world dump log15:58
*** Daisy has joined #openstack-infra15:58
clarkbtherve: I think that df may be from before everything runs not after15:58
therveRight15:58
greghaynesor, is it writing out to tmpfs?15:58
clarkbtherve: nope it is at the end but thats after things stop15:58
*** ddieterly is now known as ddieterly[away]15:59
*** abregman has quit IRC16:00
*** ayoung has quit IRC16:00
pleia2good morning16:01
*** yamahata has quit IRC16:02
clarkbgreghaynes: maybe?16:02
*** Daisy has quit IRC16:02
clarkbI think qemu-img by default writes to current dir16:02
clarkbeven for its temp file stuff16:03
greghaynesyes16:03
*** Daisy has joined #openstack-infra16:03
*** jamesmcarthur has quit IRC16:04
clarkbtherve: you might try reproducing that conversion locally to see if you can reproduce16:04
clarkbits not cloud or region specific so in theory it should be reproduceable16:04
greghayneswhere does the image build happen in those logs?16:05
clarkbgreghaynes: it doesn't, they grab an image from elsewhere16:05
*** esikachev has joined #openstack-infra16:05
greghaynesoh16:05
clarkband nova converts it to raw to boot16:05
greghaynesah16:05
*** eliqiao has quit IRC16:05
*** ddieterly[away] is now known as ddieterly16:05
greghaynes2016-07-06 13:47:41.213 4563 ERROR nova.compute.manager [instance: 17e7569c-53bc-41f2-8be9-89e6483f25eb] Stderr: u'qemu-img: error while reading sector 1073024: Input/output error\n'16:06
greghaynesso yea, that really sounds like disk full16:06
*** maishsk has joined #openstack-infra16:06
clarkbjeblair: pabelanger note that the builder daemon will need to be restarted to pick up that change iirc16:06
greghaynesactually, 2016-07-06 13:47:41.213 4563 ERROR nova.compute.manager [instance: 17e7569c-53bc-41f2-8be9-89e6483f25eb] ImageUnacceptable: Image eacca1fb-87bd-4184-a454-0727624ef544 is unacceptable: Unable to convert image to raw: Image /opt/stack/data/nova/instances/_base/f6259c6d83681935fb73c3811e68cdc527bb5381.part16:06
therveclarkb, Yep, will do16:06
greghaynes.part?16:06
therveThanks for the hints16:06
*** yamamoto has quit IRC16:06
clarkbgreghaynes: I think thats a qemu-img temp file16:07
*** eliqiao has joined #openstack-infra16:07
*** Daisy has quit IRC16:08
*** zeih has joined #openstack-infra16:08
clarkbxenial dib build is calculating image size so I think we may be getting a good image this time around16:08
cineramacool16:09
pabelangerclarkb: you are correct, however in this case stackviz actually built correct16:09
clarkbpabelanger: yup16:09
pabelangerclarkb: I'll restart it before we do uploads16:09
clarkbpabelanger: keep in mind that will kill all the other uploads for today16:10
pabelangerclarkb: ya, I think that already happen actually16:10
*** jamesmcarthur has joined #openstack-infra16:10
pabelangerI am not sure why16:10
*** matbu is now known as matbu|brb16:10
*** cbader has joined #openstack-infra16:10
mtreinishclarkb: err, I hadn't followed through with the removal from tempest yet, just deprecated them: https://review.openstack.org/#/c/338377/116:10
greghaynesclarkb: nope, thats the input to qemu-img16:10
mtreinishclarkb: still doesn't fix the devstack side, that's next16:10
clarkbgreghaynes: so maybe a nova/glance issue?16:11
clarkbmtreinish: cool16:12
*** ddieterly is now known as ddieterly[away]16:12
greghaynesclarkb: yea, it reads to me like nova didnt pull down the full image - if you note that i/o error its failing on a read too, not a write16:12
openstackgerritMerged openstack-infra/project-config: Ignore errors during stackviz build  https://review.openstack.org/33828816:12
clarkbtherve: ^ thats a theory16:12
greghaynesclarkb: http://logs.openstack.org/98/338198/2/check/gate-heat-dsvm-functional-convg-mysql-lbaasv2/5fbda19/logs/screen-n-cpu.txt.gz#_2016-07-06_13_47_41_21316:12
clarkbgreghaynes: I wonder if glance logs show anything funny16:12
*** ifarkas has quit IRC16:13
greghaynesa lot of image not founds, that could just be a red herring though16:13
clarkbya though thats odd, why would the tests be requesting a bunch of images that don't exist16:13
clarkbtherve: ^16:13
clarkbgreghaynes: http://logs.openstack.org/98/338198/2/check/gate-heat-dsvm-functional-convg-mysql-lbaasv2/5fbda19/logs/screen-g-api.txt.gz?level=INFO#_2016-07-06_13_48_23_107 is where it gets the actual image that is failing I think16:14
clarkbgreghaynes: therve 192824414 is the byte count reported by wsgi, I wonder how that compares against the actual image16:14
clarkbtherve: where is the source image?16:15
therveclarkb, http://tarballs.openstack.org/heat-test-image/fedora-heat-test-image.qcow216:15
*** alex_xu has quit IRC16:15
*** ayoung has joined #openstack-infra16:15
clarkb580737024 is the byte count reported by wget16:16
clarkbtherve: ^ I think that may give you your answer, now you need to figure out if the image put into glance was short or if glance returned less data for some reason16:16
therveclarkb, I can reproduce on a local devstack \o/16:17
therveSo that should give me some help16:17
clarkbtherve: not sure how much stuff you are putting in that image but I think my local dib builds of fedora are ~250MB if I don't add anything in16:18
therveThe image source seems shorter16:18
therveclarkb, Probably too much stuff...16:18
clarkbits been a while since I built one though16:18
*** mrmartin has quit IRC16:18
therveWe have a good chunk of tools installed though, but I don't know the details16:19
openstackgerritAdam Coldrick proposed openstack-infra/storyboard: Allow editing and deletion of comments to be disabled  https://review.openstack.org/33816516:19
*** Julien-zte has quit IRC16:19
therveSo it looks like the upload in glance is wrong16:19
greghaynesyea, I bet youre uploading a .part16:19
*** ianychoi has joined #openstack-infra16:19
*** Daisy has joined #openstack-infra16:20
thervegreghaynes, We're uploading on the fly the curl output?16:20
therveNot even sure how that works in fact16:20
greghaynesI dunno, just a theory16:20
* greghaynes applies occams razor16:21
*** gyee has joined #openstack-infra16:21
clarkbfound the issue16:21
clarkbhttp://logs.openstack.org/11/338211/2/check/gate-heat-dsvm-functional-orig-mysql-lbaasv2/2f90ffe/console.html#_2016-07-06_13_40_28_03538616:21
clarkbyou should fail at that point if the image is required16:21
*** psachin has quit IRC16:22
clarkbcurl is saying "I failed to get all the bytes" then the image is uploaded anyways16:22
therveAahah16:23
therveclarkb, Was on it myself :)16:23
*** bryan_att has quit IRC16:23
clarkbI think that is devstack that needs updating to better catch the error16:23
clarkbas for ixing the issue, you are going to have much more reliable tests if they don't try to download 554MB images across the atlantic ocean16:24
therveYeah not sure devstack can do much, it's in our test hooks script16:24
*** tphummel has quit IRC16:24
* therve nodes16:24
*** Daisy has quit IRC16:24
clarkbidentifying what you actually need in your image and potentially using the common image may help. (I don't know what the requirments for this test are)16:24
therveclarkb, greghaynes: thanks a ton. I have a bunch of things to try now.16:24
*** yamahata has joined #openstack-infra16:25
clarkbpabelanger: ya I don't see any other builder activity, gonna look at its logs now16:25
clarkbpabelanger: a whole lot of 'DIB failed creating' errors, maybe they ran into the same npm issues?16:28
*** DaiDV has quit IRC16:28
clarkbah no was new git repo causing fails16:29
pabelangerclarkb: possible, I can look more into it after some food16:29
clarkbso ya we should be fine to restart the builder once this xenial image is built16:29
pabelangerYup, we are compressing images now16:29
clarkbI need to upload the new centos-7 image to one region that failed last night too, otherwise we should have mostly working images later today16:29
openstackgerritAnita Kuno proposed openstack-infra/storyboard: Add example commands for the Tasks api  https://review.openstack.org/33787916:30
*** david-lyle_ has joined #openstack-infra16:30
*** Daisy has joined #openstack-infra16:31
*** david-lyle_ is now known as david-lyle16:32
*** dprince has joined #openstack-infra16:32
*** PollyZ has quit IRC16:32
*** vhosakot has quit IRC16:35
openstackgerritMerged openstack-infra/tripleo-ci: Log stderr when creating and destroying envs  https://review.openstack.org/33777516:35
*** ddieterly[away] is now known as ddieterly16:35
*** Daisy has quit IRC16:36
*** vhosakot has joined #openstack-infra16:36
*** kzaitsev_mb has quit IRC16:37
*** yamamoto has joined #openstack-infra16:37
*** david-lyle has quit IRC16:39
*** jamesmcarthur has quit IRC16:40
*** Apoorva has joined #openstack-infra16:40
*** hashar has quit IRC16:42
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: Remove sleeps if testenv create/destroy fails  https://review.openstack.org/33839416:42
*** Swami__ has quit IRC16:42
*** Swami_ has quit IRC16:43
*** jamesmcarthur has joined #openstack-infra16:43
*** nmagnezi has joined #openstack-infra16:44
*** trown|lunch is now known as trown16:44
*** gyee has quit IRC16:45
*** kzaitsev_mb has joined #openstack-infra16:45
*** Daisy has joined #openstack-infra16:48
*** vhosakot has quit IRC16:48
*** vhosakot has joined #openstack-infra16:48
*** yamamoto has quit IRC16:48
*** amoralej is now known as amoralej|off16:49
*** tesseract- has quit IRC16:49
*** yamahata has quit IRC16:49
*** markusry has quit IRC16:49
*** markusry has joined #openstack-infra16:50
openstackgerritAnita Kuno proposed openstack-infra/storyboard: Add example commands for the Branches api  https://review.openstack.org/33839616:50
*** PollyZ has joined #openstack-infra16:50
*** yamahata has joined #openstack-infra16:51
*** tphummel has joined #openstack-infra16:51
*** rdx_matt has quit IRC16:51
*** matbu|brb is now known as matbu16:51
*** rdx_matt has joined #openstack-infra16:52
fungiwoah! confirmed this fan actually _is_ on the market. now just to find it somewhere not quite so overpriced... http://www.newegg.com/Product/Product.aspx?Item=9SIA6252GS266416:52
fungilooks like it was also used in some lenovo systems16:53
clarkbfungi: thats the refurb price too :)16:53
*** Daisy has quit IRC16:53
fungitell me about it16:53
*** matt-borland has joined #openstack-infra16:53
pleia2yeesh16:53
fungifound it on pchub for us$26 new16:54
fungithink i'll get a few16:54
anteayafungi: that is better than 7616:54
fungiindeed16:54
timothyb89fungi: I'd bet you can find those on ebay for $5-$1516:54
fungigood call16:55
*** lucasagomes is now known as lucas|afk16:55
fungiindeed i can16:55
* fungi forgets ebay exists sometimes16:55
*** PollyZ has quit IRC16:55
timothyb89:)16:55
*** inc0 has quit IRC16:56
*** rdx_matt has quit IRC16:57
*** inc0 has joined #openstack-infra16:57
clarkbmtreinish: https://review.openstack.org/338399 is a start16:57
pabelangerclarkb: ubuntu-xenial build finished, starting uploads now16:57
*** karthikp_ has quit IRC16:57
clarkbpabelanger: ok16:57
*** Swami has joined #openstack-infra16:57
clarkbpabelanger: were you going to restart hte builder first?16:57
*** exploreshaifali has quit IRC16:57
clarkbpabelanger: I have one centos-7 straggler to upload too, but can wiat for when xenial is done16:58
mtreinishclarkb: lgtm, heh but lets see if it passes the gate :)16:58
anteayaummm it appears that for some reason my python 34-db job couldn't find pretty-tox.sh, would I be reading this log correctly? http://logs.openstack.org/79/337879/3/check/gate-storyboard-python34-db/538cd37/console.html#_2016-07-06_16_47_10_07791116:58
clarkbmtreinish: yes, I admit to no tknowing if all 3 keys in that file are required16:58
openstackgerritAaron Rosen proposed openstack-infra/project-config: Add non voting python34/35 checks for vmware-nsx  https://review.openstack.org/33840216:59
*** shashank_hegde has joined #openstack-infra16:59
mtreinishmriedem: ^^^ who would be good to review clarkb's config drive checking patch? mikal?16:59
*** gyee has joined #openstack-infra17:00
*** derekh has quit IRC17:00
*** chem has quit IRC17:00
mriedemmtreinish: yeah17:01
pabelangerclarkb: ugh, I forgot17:01
*** piet has joined #openstack-infra17:01
pabelangerclarkb: lets do it, I can restart uploads17:01
clarkbpabelanger: its fine iether way17:01
clarkbwe can get it when xenial is done too17:01
mriedemmtreinish: probably artom also since he just wrote code for that in tempest17:01
pabelangerclarkb: okay, restarted17:01
*** karthikp_ has joined #openstack-infra17:01
*** Sukhdev has quit IRC17:01
*** DaiDV has joined #openstack-infra17:01
pabelangerand uploads restarted17:02
clarkbpabelanger: cool, can you queue up an upload for rax-dfw centos7 too?17:02
clarkbpabelanger: that one didn't succeed last night and is the last centos-7 image with the weird virtualenv devstack situation going on17:03
*** adrian_otto has quit IRC17:04
*** PollyZ has joined #openstack-infra17:04
fungitimothyb89: thanks for the ebay suggestion. for us$40 i got 5 of them from a domestic source with free shipping17:04
timothyb89fungi: nice! glad to have helped :)17:04
*** rdxnoam has joined #openstack-infra17:06
rdxnoamsmcginnis: still here?17:06
*** ayoung has quit IRC17:06
mtreinishfungi: heh, I like that newegg advertises that as "First from ASIA"17:07
anteayafungi: wow17:08
*** BobBall is now known as BobBall_AWOL17:08
*** adrian_otto has joined #openstack-infra17:08
*** PollyZ has quit IRC17:08
jeblairfungi: or possibly 5 bobcats17:09
jeblairlet us know17:09
mtreinishfungi: I still thinking dunking your computer is your best option for heat mgt :)17:10
mtreinishs/thinking/think/g17:10
*** timrc has quit IRC17:10
fungijeblair: ooh, i could use a few bobcats to keep the seagulls in check17:11
clarkbI have a fanless amd apu. Pretty sure the heat kills it when running freebsd due to poor thermal management17:11
*** jlk has quit IRC17:11
fungiclarkb: interesting. i have a couple of those as well and no heat issues under openbsd, though i also don't heavily load them17:11
*** timrc has joined #openstack-infra17:11
*** shashank_hegde has quit IRC17:12
clarkbfungi: it runs fine with linux. But it would crash and die about once a day during hot summer days with freebsd17:12
*** jlk has joined #openstack-infra17:12
*** jlk has quit IRC17:12
*** jlk has joined #openstack-infra17:12
fungiapu1d4 with the metal case acting as an extension of the heat sunk17:12
*** ajmiller has joined #openstack-infra17:12
clarkbno kernel panics or anything logged by the operating system so pretty sure the hardware was saving itself17:12
openstackgerritClint 'SpamapS' Byrum proposed openstack-infra/glean: Add coverage generated files to .gitignore  https://review.openstack.org/33840917:12
fungiclarkb: yeah, that sounds like thermally-induced shutoff. you should have support for the on-chip thermocouple in kernel and could stream readings from it to somewhere17:13
*** ddieterly is now known as ddieterly[away]17:13
bkeroI wonder if FreeBSD has proper p/c state support for downclocking it17:13
clarkbalsoit was consistently during the hottest part of the day ~4pm ish17:14
SpamapSgreghaynes: hey, am I crazy or does glean not actually record that it has run successfully anywhere?17:14
greghaynesSpamapS: it doesnt17:14
*** acoles is now known as acoles_17:14
SpamapSdoh!17:14
SpamapSgreghaynes: we need to fix that17:14
SpamapSthings reboot17:14
greghaynesSpamapS: I think it doesn't overwrite existing interface config, though17:14
SpamapSconfigs change17:14
fungii wonder if there's a good place it could write a flagfile17:14
*** karthikp_ has quit IRC17:15
greghaynesSpamapS: which is not ideal but is a stopgap for that reason17:15
SpamapSit needs it's own home17:15
clarkbgreghaynes: ya it is supposed to leave any interface files alone and ssh authorized keys as is if they exist iirc17:15
SpamapS/var/lib/glean is the proper place17:15
SpamapSwhat about hostname?17:15
clarkbnot sure about hostname17:15
clarkbI wasn't digging in that code yesterday17:15
greghaynesyes, theres also the larger issue of 'what if the config files dont match what config drive says'17:15
fungiagreed, /var/lib or /var/log would be sane options as long as /var can be assumed to be mounted when glean runs17:15
greghaynesreally it needs to hash the config drive data it ran with and save that on success17:15
clarkbfungi: can't be if /var is network mounted, can be otherwise17:16
SpamapSlooks like it's bullish on hostname17:16
anteayathe storyboard tox file calls pretty_tox: http://git.openstack.org/cgit/openstack-infra/storyboard/tree/tox.ini#n13 which exists in the storyboard repo: http://git.openstack.org/cgit/openstack-infra/storyboard/tree/tools/pretty_tox.sh17:16
SpamapSgreghaynes: nooo17:16
SpamapSit should do it's job17:16
clarkb(network mounting /var is probably a bad idea but hey we had nfs ounted /usr/local at one place)17:16
SpamapSand get the heck out of the way17:16
anteayaSpamapS: can we ease back on the swearing please?17:16
SpamapSanteaya: bullish is a swear word?17:16
clarkbnibalizer: were you around for network mounted /usr/local? its good stuff17:16
nibalizerclarkb: ah yes17:17
anteayahas anything changed in the python24-db job that would affect its ability to process the tox file17:17
nibalizersolairs boxes, high efficiency17:17
greghaynesSpamapS: Yea, I'd be fine with that actually - just a 'dont run again' flag17:17
fungianteaya: python27-db?17:17
*** gnuoy is now known as jamespage_shadow17:17
fungianteaya: have an example failure?17:17
anteayafungi: sorry python34-db17:17
AJaegerpabelanger: thanks for building the xenial images17:17
SpamapSgreghaynes: the flag should be the instance ID17:17
anteayahttp://logs.openstack.org/79/337879/3/check/gate-storyboard-python34-db/538cd37/console.html#_2016-07-06_16_47_10_07791117:17
*** ddieterly[away] is now known as ddieterly17:17
*** xenogear has joined #openstack-infra17:18
*** PollyZ has joined #openstack-infra17:18
greghaynesSpamapS: is that in config-drive?17:18
*** daemontool has quit IRC17:18
clarkbSpamapS: so I disagree a little bit on that because that is config drives behavior nad it totally breaks in some cases17:18
clarkber cloud init17:18
clarkblike when all my ssh host keys were removed17:18
greghaynesclarkb: you mean the dont re-run even if config-drive changes?17:18
clarkband cloud init lolled at me wanting them to be recreated17:18
anteayaSpamapS: yes17:18
SpamapSclarkb: well I still think it shouldn't overwrite stuff.17:18
clarkbgreghaynes: no I mean rerun if necessary files are non existant17:18
*** jamespage_shadow is now known as gnuoy17:18
SpamapSanteaya: It's a stock market term.17:18
pabelangerAJaeger: uploading now which re-up any failures too17:18
clarkbSpamapS: yes not overwriting is fine, but cloud inits behavior is to do absolutely nothing when that host id matches17:18
anteayait is a conveyance of agression17:18
anteayaand I don't feel comfortable around it17:19
*** mhickey has quit IRC17:19
clarkbSpamapS: but there are important cases where you want it to do the right thing which it can safely do eg with ssh host keys17:19
pabelangerclarkb: sure, let me do a rebuilt of centos-7 so we pickup our iptables fixes too17:19
pabelangerclarkb: if you don't mind17:19
*** ayoung has joined #openstack-infra17:19
clarkb+1 to not overwriting, -1 to ignoring everything if a single flag matches17:19
clarkbpabelanger: thats fine17:19
*** sarob has joined #openstack-infra17:19
fungianteaya: that failure message is secondary. 8 tests failed17:19
anteayafungi: okay thank you17:20
fungianteaya: scroll up to where the tests are running and you can see output from them17:20
SpamapSanteaya: Apologies, I was not aware that it was offensive to anyone. I won't use it anymore.17:20
*** jpich has quit IRC17:20
fungianteaya: there are captured logging and tracebacks from each test failure17:20
pabelangercentos-7 image-build started17:20
anteayaSpamapS: I appreciate your understanding, thank you17:21
anteayafungi: I see fixtures._fixtures.timeout.TimeoutException17:21
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Revert "Disable stackviz on xenial for now"  https://review.openstack.org/33841217:21
anteayafungi: at the end of all the tracebacks I see17:21
AJaegertimothyb89's change merged quickly - so we can revert the stackviz disabling again ^17:22
fungianteaya: yep, that's what i'm seeing too. those tests may take too long for testr's default test timeout17:22
*** kzaitsev_mb has quit IRC17:22
clarkbfungi: I don't think testr hasa default timeout, we build timeouts into our test frameworksthough17:22
anteayaI see a lot of INFO storyboard.plugin.scheduler [-] Scheduler is not enabled. too17:22
fungiclarkb: oh, right, that's in ostestr17:22
greghaynesclarkb: does that just apply to missing interface config files and ssh host keys or are there other cases? For the hostname it seems like we only ever should set that once17:23
fungianteaya: yeah, probably best to compare against a working job log since i have no idea whether that's expected/benign17:23
*** gnuoy is now known as jamespage_shadow17:23
SpamapSclarkb: so, I'm not sure I have a good response to the cloud-init situation you had. cloud-init's behavior was at least predictable.17:23
anteayafungi: patchset 2 passed 30 minutes prior: https://review.openstack.org/#/c/337879/317:23
*** PollyZ has quit IRC17:24
SpamapSclarkb: what I worry about is that glean has no way to know if my config-management renamed the config files it wrote out, and replaced them with better ones / different netconfig methods/etc.17:24
fungianteaya: entirely possible you hit an under-performing node17:24
anteayaI'll try again then I guess, thanks for looking at it with me17:25
*** jamespage_shadow is now known as gnuoy17:25
mriedemmtreinish: clarkb: the network metadata change seems fine17:25
clarkbgreghaynes: if you have no hostname set in /etc/hosts /etc/hostname shouldn't it reset it?17:25
clarkbgreghaynes: SpamapS the problem is predictable isn't always useful and cloud inits behavior very much was not useful17:25
clarkbif we are going to go through the trouble of reinventing the wheel we should at least make it more useful17:26
clarkbotherwise just use cloud init17:26
greghaynesclarkb: SpamapS Maybe the way forward is - only do networking once ever based on machine id, the other things always assert they exist (but only assert contents first boot)?17:26
SpamapSwell a wheel with no python dependencies is still better than cloud-init ;)17:26
greghaynessince the other things are less disruptive17:26
AJaegerclarkb: question for you on https://review.openstack.org/337946 - do we remove python34 testing from old branches or keep it?17:26
SpamapSgreghaynes: hostname could be _super_ disruptive17:27
clarkbSpamapS: the specific case I had was my ssh host keys were removed by a bad dib patch. glean and cloud init both generate those on boot if they do their things because the ssh service on debuntu does not generate those if they are missing17:27
clarkbSpamapS: yes because nova is silly :)17:27
greghaynesSpamapS: hrm, youre thinking someone changes it post-first-boot and then wants it to stick?17:27
greghaynesSpamapS: actually though, thats what I mean by dont assert contents17:27
greghaynesSpamapS: only re-assert if its unset17:27
SpamapSIf I've reset the box's hostname, and reboot it later, and it comes back as the old one, and suddenly my logs all have the wrong hostname and my quorum system is mad because there are duplicates or missing hosts.... I'm very annoyed.17:27
clarkbgreghaynes: ya I think that is the situation, like if you repurpose a host17:27
*** tqtran has joined #openstack-infra17:27
clarkbI am not suggestin we change any hostnames though17:28
AJaegermtreinish: is my WIP on https://review.openstack.org/333461 correct?17:28
clarkbonly set it if it isn't set17:28
clarkbwhich is different17:28
fungianteaya: compare the job runtimes too. on that last patchset the job went 20 minutes17:28
*** berendt has quit IRC17:28
anteayafungi: ah thank you, I will look17:28
greghaynesthe downside there is you cant intentionally have an unset hostname17:28
greghaynesnot sure thats a thing we need to support?17:28
SpamapSgreghaynes: that's a good mode actually17:28
*** berendt has joined #openstack-infra17:29
*** gnuoy is now known as jamespage_shadow17:29
clarkbgreghaynes: but you can unintentionally unset it17:29
clarkbits trivial to rm hostname and remove lines from /etc/hosts17:29
greghaynesclarkb: oh, so use empty file = unset17:29
greghaynesyea, that works17:29
notmorganjeblair, clarkb, pabelanger: are our images using mariadb or mysql-server for tests?17:29
notmorgansubtle differences, but I'm curious which one we are testing with.17:30
clarkbnotmorgan: mariadb on red hat things and mysql on ubuntu. Not sure about debian17:30
notmorganclarkb: wfm.17:30
*** Goneri has quit IRC17:30
notmorganclarkb: i'll still with mysql-server on my ubuntu env then.17:30
notmorganstick*17:30
AJaegerclarkb: mysql on dpkg in general17:30
*** jamespage_shadow is now known as gnuoy17:30
notmorgani was having issues getting mariadb to play nice in ubuntu for some reason.17:30
notmorganwhich is why i was asking17:31
clarkbnotmorgan: are there mariadb packages for ubuntu now?17:31
greghaynesclarkb: SpamapS So were fine with assert contents on first boot, later boots only set if there is nothing set for hostname, ssh host keys, root ssh key. For networking only re-assert if machine-id doesnt match or hasnt been written out?17:31
notmorganclarkb: have been for a while. i'm on xenial17:31
notmorganso yes.17:31
notmorganbut it ... gets cranky in some cases.17:32
notmorganmostly it's because i've been trying to do super cheap (effort wise) installs. since it's just for local unit tests.17:32
notmorganthough I guess i could docker-compose this now.17:32
pabelangerAJaeger: osic-cloud1 uploaded first17:32
pabelangerAJaeger: followed by ovh-gra117:33
notmorganand get isolation from my local system.17:33
clarkbnotmorgan: I have a mysql docker instance for this17:33
AJaegerpabelanger: great! Let's see once first python35 jobs succeed again...17:33
*** berendt has quit IRC17:34
notmorganclarkb: yeah. i think i'm going to use docker-compose and build a simple "make tests" which spins it up/run tests and dumps out results.17:34
notmorgan(fwiw, i did docker for building dpkgs recently, was amazing)17:34
clarkbnotmorgan: because as running kde once taught me running mysql as a part of your actual desktop install is insanity17:34
notmorganLOL17:34
notmorgangod. i am using KDE on my laptop... and realize, it isn't stable enough =/17:35
greghaynesnotmorgan: haha17:35
SpamapSgreghaynes: yeah, I'm still a little fuzzy on the contents vs. empty vs. missing though.17:35
fungiclarkb: notmorgan: mariadb-server packages were present on trusty for that matter17:35
greghaynesnotmorgan: so there was supposed to be an ORM layer to allow !mysql17:35
notmorganfungi: the mariadb on trusty just didn't work for me when i tried i.17:35
notmorgangreghaynes: LOL right.17:35
*** gnuoy is now known as jamespage_shadow17:35
greghaynesnotmorgan: then by the time someone tried writing another backend it was realized that making a !mysql driver was going to be impossible17:35
greghaynesso no one ever did it17:35
notmorgangreghaynes: well plasma5 is better17:35
clarkbnotmorgan: kde 4 I think it was intsalled mysql for indexing stuff or something. It made everything sadness17:36
notmorgangreghaynes: but i have to send my laptop to lenovo to get repaired... annnnnnnd that means i have to basically re-image it since they want access.17:36
*** jamespage_shadow is now known as gnuoy17:36
*** Rockyg has quit IRC17:36
AJaegerclarkb: used mainly for kmail - and everything related to it. akonadi wsa the component.17:36
notmorganclarkb: latest kde4 thing didn't have myql on it until i installed it directly. but i purged kmail.17:36
greghaynesSpamapS: thatll be for hostname, ssh host keys, and root pubkey - for those its if no_prev_run: assert_file elif not file_exists: assert_file17:36
AJaegerclarkb: and it was for me also one of the reasons to switch to gnome17:37
notmorganAJaeger: what, no enlightenment? :P17:37
* notmorgan ducks.17:37
greghaynesclarkb: it was for the personal information manager17:37
greghaynesclarkb: basically there was a big push from kde to make a shared information manager that both kde and gnome would use, then it really fizzled out and gnome said ew17:37
* notmorgan might go back to ubuntu or debian on the personal laptop.17:37
greghaynesthe idea being you could boot gnome and kde and have a lot of the same settings/info automagically there17:38
notmorgani like tumbleweed, but KDE isn't stable enough and i'm annoyed by gnome3/mate/xfce right now.17:38
*** eezhova has quit IRC17:38
greghaynesheh17:38
greghaynesYea, I dont use it anymore17:39
*** Jeffrey4l_ has quit IRC17:39
openstackgerritAdam Coldrick proposed openstack-infra/storyboard-webclient: Complex priorities UI in stories  https://review.openstack.org/31266617:39
*** savihou has quit IRC17:40
*** jamesmcarthur has quit IRC17:40
clarkbAJaeger: I am glad I wasnt the only one :)17:41
mtreinishAJaeger: yes it is. Sry I missed your earlier comment asking for that17:41
*** sree_ has joined #openstack-infra17:41
*** sree_ is now known as Guest3255817:42
AJaegermtreinish: do you want to WIP yourself? Then I can remove mine and you tell us once it's ready to merge...17:42
mtreinishAJaeger: we install subunit2sql from pip not git there, so until the depends on is included in the next release nodepool won't have access to it17:42
mtreinishAJaeger: meh, doesn't bug me one way or the other, I can always rebase it to remove yours :)17:42
*** Goneri has joined #openstack-infra17:42
AJaegermtreinish: ok, then I'll leave mine. Just comment and ping me  or rebase ;)17:43
openstackgerritBen Nemec proposed openstack-infra/tripleo-ci: [WIP] Update to use quintupleo  https://review.openstack.org/33630917:43
*** Jeffrey4l_ has joined #openstack-infra17:44
AJaegerwe have two openstack-ansible jobs at the top of the check queue - one for nearly 7h, the other for over 4h - is there a problem?17:44
*** Qiming has quit IRC17:45
*** PollyZ has joined #openstack-infra17:45
*** gnuoy is now known as jamespage_shadow17:45
*** sarob has quit IRC17:46
*** Guest32558 has quit IRC17:46
*** makowals has joined #openstack-infra17:46
krot_vaca_jul19What's going on with the npm jobs?17:46
jeblairAJaeger: maybe they are stuck in a loop?17:47
AJaegermight be - didn't watch them long enough for that ;)17:47
jeblairthe top running job is telnet://172.99.106.217:808817:47
jeblairfor 330994,617:47
pabelangerAJaeger: only rax-ord, rax-iad and ovh-bhs1 left for uploads17:47
jeblairlet's check back in 30 mins and see if it has re-launched17:48
*** jamespage_shadow is now known as gnuoy17:48
AJaegergood idea17:48
AJaegerpabelanger: cool...17:48
*** Qiming has joined #openstack-infra17:49
*** ddieterly is now known as ddieterly[away]17:49
*** sarob has joined #openstack-infra17:50
*** PollyZ has quit IRC17:50
*** mikelk has joined #openstack-infra17:54
*** rossella_s has quit IRC17:54
fungikrot_vaca_jul19: not the npm jobs specifically, but 504 errors from npm building stackviz for our images17:54
fungikrot_vaca_jul19: but only on xenial it seems17:55
*** shashank_hegde has joined #openstack-infra17:55
*** sdake has quit IRC17:55
*** sdake has joined #openstack-infra17:56
AJaegerfungi: 50217:57
AJaegersee http://paste.openstack.org/show/526524/17:57
fungiahh, yes sorry, 50217:57
*** gouthamr_ has joined #openstack-infra17:57
AJaegerand that was on xenial17:57
*** shardy has quit IRC17:58
*** john-davidge has quit IRC17:58
*** pvaneck has joined #openstack-infra17:59
*** gouthamr has quit IRC18:00
pabelangeractually, we can revert the stackviz patch AJaeger did. We should no longer fail DIBs if stackviz errors18:01
fungithough devstack will still try to complete the stackviz build, right? or just ignore the incomplete stackviz installation?18:01
timothyb89fungi: it will ignore it18:02
*** PollyZ has joined #openstack-infra18:02
timothyb89fungi, pabelanger: also, AJaeger already put up a patch to revert: https://review.openstack.org/33841218:02
*** sdake has quit IRC18:02
*** M-docaedo_vector has quit IRC18:03
pabelangertimothyb89: great, thanks!18:03
AJaegercurrently py35 running on ubuntu-xenial-internap-nyj01-2289461 - bindep passed ;)18:04
pabelangermissed that18:04
pabelanger\o/18:04
pabelangeronly 2 uploads left18:04
clarkbAJaeger: did the change for groups other-requirements merge?18:06
clarkbsince they will need that now I think18:07
*** jamesmcarthur has joined #openstack-infra18:07
*** sdake has joined #openstack-infra18:08
*** sdake has quit IRC18:08
*** sdake has joined #openstack-infra18:08
*** notnownikki has quit IRC18:09
AJaegerclarkb: yes, it did18:09
*** jamesmcarthur has quit IRC18:09
AJaegerclarkb: change Ic175744997ac01b3c89793c78d501d43044507e818:09
*** jamesmcarthur has joined #openstack-infra18:09
*** permalac has quit IRC18:09
AJaegerSo, py35 job for ironic finished - failed the testsuite but that looks unrelated to our setup.18:09
sshnaidmclarkb, hi, can you please re-review in your time https://review.openstack.org/#/c/334452/ ? thank you18:11
openstackgerritMerged openstack-infra/project-config: Revert "Disable stackviz on xenial for now"  https://review.openstack.org/33841218:11
*** salv-orlando has quit IRC18:12
*** deadnull is now known as _deadnull18:13
AJaegerjeblair: the top-most ansible job failed finally...18:13
AJaegerjeblair: so, looks fine now18:14
pabelangerEmilienM: Hows the UCA mirrors working? Have you switched?18:15
EmilienMlet me check18:15
EmilienMwe had a blocker and I think it was solved18:15
AJaegeryeah, first python35 success: http://logs.openstack.org/30/308930/51/check/gate-manila-python35-db-nv/f9437c4/18:16
EmilienMpabelanger: rechecking your patch18:16
EmilienMpabelanger: I'll let you know18:16
EmilienMbut +2 it18:16
pabelangerEmilienM: nice, thanks18:16
EmilienMpabelanger: no gpg issue?18:17
*** jamesmcarthur has quit IRC18:17
pabelangerEmilienM: not sure. I think they should work18:17
*** jamesmcarthur has joined #openstack-infra18:18
EmilienMpabelanger: it should be good18:18
*** inc0 has quit IRC18:18
pabelangerya, I think so too18:18
EmilienMpabelanger: the extras patch is about to merge18:18
*** karthikp_ has joined #openstack-infra18:18
clarkbsshnaidm: ok looks good. The only thing I wonder is if we should allow each of those to be individually configureable? I think as is you must provide all 3 values since None isn't a valid setsockopt param value?18:18
EmilienMpabelanger: can you backport it to stable/mitaka & liberty? so we can use it in our stable CI too18:18
*** inc0 has joined #openstack-infra18:18
pabelangerEmilienM: sure18:18
*** M-docaedo_vector has joined #openstack-infra18:19
*** pgadiya has joined #openstack-infra18:19
sshnaidmclarkb, yes, I've added defaults in cli - I'm not sure we should duplicate them in function also18:19
*** Jeffrey4l_ has quit IRC18:19
*** Nikolay_St has joined #openstack-infra18:19
*** pgadiya has quit IRC18:19
clarkbsshnaidm: well gear is both a command line tool and a library18:19
EmilienMpabelanger: also please do https://review.openstack.org/#/c/335106/ for master18:20
sshnaidmclarkb, ok, so duplicate them in function as well?18:20
*** _sarob has joined #openstack-infra18:20
EmilienMpabelanger: I think we can use the mirror on master, we're using updates repo18:20
clarkbsshnaidm: ya I think we probably should that way people running a gearman server via python and not the command can individually select those18:20
sshnaidmclarkb, ok, will do18:21
clarkbsshnaidm: sorry I didn't think of that until now.18:21
sshnaidmclarkb, np, I didn't know either it's used as a library :)18:21
openstackgerritSagi Shnaidman proposed openstack-infra/gear: Support TCP keepalives in geard  https://review.openstack.org/33445218:23
*** sarob has quit IRC18:23
*** yonglihe has quit IRC18:24
*** aorourke has joined #openstack-infra18:25
*** jed56 has quit IRC18:25
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Revert "Revert "Make py35 jobs voting on tempest""  https://review.openstack.org/33846618:26
AJaegermtreinish, gmann, we should be fine to merge the change above - do you want this now or wait a day? ^18:26
*** piet has quit IRC18:26
*** markusry has quit IRC18:27
jeblairsshnaidm: lgtm, but can you add docstrings for the new valus?  i left an anline comment pointing to the location18:28
jeblair'inline'18:29
*** Goneri has quit IRC18:30
sshnaidmjeblair, sure, will add18:32
*** ddieterly[away] is now known as ddieterly18:34
*** ociuhandu has quit IRC18:34
*** adrian_otto has quit IRC18:34
*** javeriak has joined #openstack-infra18:35
*** ayoung has quit IRC18:35
*** Guest_94857 has joined #openstack-infra18:38
Guest_94857Allah is doing18:38
Guest_94857sun is not doing Allah is doing18:39
openstackgerritSagi Shnaidman proposed openstack-infra/gear: Support TCP keepalives in geard  https://review.openstack.org/33445218:40
anteayacan an infra root help our Guest find the door?18:40
*** sshnaidm is now known as sshnaidm|afk18:42
Guest_94857moon is doing Allah is doing18:42
anteayainfra-root ^^18:42
*** Guest_94857 has left #openstack-infra18:42
*** Goneri has joined #openstack-infra18:43
fungii'18:44
fungier18:44
fungii've added that nick to the akick list for this channel in case it returns18:45
openstackgerritJames Slagle proposed openstack-infra/tripleo-ci: Add mulitnode CI job support to tripleo-ci  https://review.openstack.org/32477718:45
notmorganoh joy... looks like on xenial NO_AUTO_CREATE_USER is automagically set... somewhere...18:46
anteayafungi: thanks18:47
anteayanotmorgan: what command is failing for you?18:48
*** trown is now known as trown|brb18:49
notmorgananteaya: uhm... creating a new user in unit tests18:51
openstackgerritsebastian marcet proposed openstack-infra/openstackid-resources: Upgrade Laravel and ORM  https://review.openstack.org/32230718:52
AJaegernotmorgan: do you have a log file?18:52
notmorganAJaeger: it's that the server has NO_AUTOCREATE enabled, this is a local mysql install18:52
notmorgantrying to find it because it might hit us in the gate as well.18:53
notmorganAJaeger: i'm getting "no user row matched" error.18:53
*** piet has joined #openstack-infra18:53
*** esumerfd has joined #openstack-infra18:53
mtreinishAJaeger: I'm fine whenever you think things are ready18:53
pabelangerclarkb: centos-7 uploads started18:54
AJaegermtreinish: ok, thanks. We should be ready. Let's see when others review...18:54
anteayanotmorgan: :(18:54
openstackgerritJens Rosenboom proposed openstack-infra/project-config: Fix ceilometer testing  https://review.openstack.org/33849218:54
notmorgananteaya: yeah it's ... annoying18:55
AJaegernotmorgan: thanks for looking into it.18:55
anteayayeah18:55
mat128hey guys, anyone has time to look at this change? https://review.openstack.org/#/c/336278/18:55
mat128we (Ironic) are ready to merge it, Jim +1d18:55
notmorganAJaeger: i think it's something we might need to just include in /etc/mysql/mysql.conf.d or some such18:56
notmorganAJaeger: i'll report as soon as i find it. [this matters for nodepool and other such things that are doing opertunistic testing]18:56
notmorganit might also just be something wrong in my setup [ which i hope ]18:56
*** ddieterly is now known as ddieterly[away]18:57
funginotmorgan: in our jobs we have macros which do basic account setup in mysql. would that help?18:58
*** mikelk has quit IRC18:58
notmorganfungi: it's a server level issue18:58
*** doffm has joined #openstack-infra18:58
notmorganfungi: the account is fine locally (checked), some option is being globally set that prevents the opportunistic dynamic users from working18:58
asselin__rcarrillocruz, how is the 'ansible master' vm setup?18:59
*** Sukhdev has joined #openstack-infra18:59
funginotmorgan: oh, got it. i think most jobs just use the admin account we create, and then if necessary use it to create other accounts directly19:00
notmorganfungi: but i just lost my local scratch volume soooo will get back to you when that is fixed.19:00
*** asettle has joined #openstack-infra19:00
notmorganfungi: nodepool dynamically creates users.19:00
notmorganfungi: and databases.19:00
notmorganfungi: and it looks like these dynamic users are sans password, which is broken by this option.19:00
fungii honestly didn't even know mysql had the ability to autocreate users for you19:00
fungibut yeah, behavior of passwordless accounts seems to have changed on xenial's packaged version19:01
notmorganfungi: "grant <options> on db.tables" etc19:01
notmorganthat creates the user.19:01
fungiahh. we do rely on grants adding accounts, yes19:01
notmorganwith this option "passwordless" isn't autocreated.19:01
fungii was misunderstanding what you were saying was disabled19:01
notmorganso you mus use 'identified by "XXXX"' syntax19:02
fungifor the accounts we create in our job macros, we set passwords. there was some complication on xenial using the built-in root account while passwordless19:02
notmorganyep. that too.19:02
*** ddieterly[away] is now known as ddieterly19:02
notmorganthe root account is... "special"19:02
*** _deadnull is now known as deadnull19:02
notmorganbut this is looking like something they ... compiled in?19:03
notmorganas a default... maybe.19:03
notmorganbecause i'm not seeing the option in any files.19:03
notmorganbut i might just be missing it :P19:03
fungithis is the mechanism we at least try to make sure works in our jobs on all platforms: http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/macros.yaml#n69419:04
fungiand that's supposedly working for us on xenial now19:05
slaglepabelanger: did the centos7 image uploads finish?19:05
clarkbslagle: no they are still all in progress19:06
*** kien-ha has joined #openstack-infra19:06
*** savihou has joined #openstack-infra19:06
EmilienMpabelanger: https://review.openstack.org/335101 merged, you can backport it now19:07
*** DaiDV has quit IRC19:07
*** inc0 has quit IRC19:07
*** thorst has quit IRC19:08
pabelangerslagle: they are coming online now19:08
pabelangerEmilienM: great19:08
slaglek19:08
*** ociuhandu has joined #openstack-infra19:09
*** thorst has joined #openstack-infra19:10
*** rockyg has joined #openstack-infra19:10
pabelangerEmilienM: do you usually use the web interface for puppet modules or command lne?19:10
*** esumerfd has left #openstack-infra19:10
pabelangerline*19:10
*** _ari_ has left #openstack-infra19:10
pabelangerhappy to do either, what ever makes things easier19:10
EmilienMpabelanger: what web interface?19:10
pabelangerEmilienM: gerrit19:11
EmilienMpabelanger: I use gertty19:11
pabelangergreat19:11
EmilienMbut webapp works too19:11
*** deadnull has quit IRC19:13
*** thorst has quit IRC19:14
krot_vaca_jul19AJaeger: Hrm.19:14
krot_vaca_jul19AJaeger: Ok, so, I personally think the issue is that the xenial npm package is not staying up to date with LTS backports.19:15
krot_vaca_jul19AJaeger: There was a big issue a few weeks ago where the xenial package threw up once npm switched their hosting to gzip-by-default19:16
krot_vaca_jul19Anyway19:16
*** trown|brb is now known as trown19:16
krot_vaca_jul19We may just want to switch back to using the nodesource repo19:16
fungikrot_vaca_jul19: yeah, i have a feeling we should continuously consume language-ecosystem-specific package managers19:16
*** dtardivel has quit IRC19:17
sridhar_raminfra-team: a general question, related to which is the correct way to pull in a project's python client library - should we use requirements.txt vs test-requirements.txt ?19:17
fungikrot_vaca_jul19: so maybe not distro packages but just unpack into /usr/local or whatever the distro-agnostic installation mechanism is for npm19:17
sridhar_ramThis is in the context of https://review.openstack.org/336305 where we added python-tackerclient to tacker's test-requirements instead of requirements.txt. But I see many projects pulling in their client using requirements.txt. Perhaps this is for devstack install purpose ? In general is there a suggested guideline or convention ?19:17
openstackgerritMerged openstack-infra/project-config: Bump VM memory for latest stable CoreOS IPA  https://review.openstack.org/33627819:17
pabelangerEmilienM: odd, gertty doesn't list stable/mitaka as an available branch to propose change too19:18
AJaegerkrot_vaca_jul19: please check https://review.openstack.org/#/c/337481/19:18
pabelangerneed to check my configs19:18
krot_vaca_jul19AJaeger: That... is now how I expected him to do that.19:19
krot_vaca_jul19*not19:19
AJaegersridhar_ram: do you need the lib for testing - or for general running? Answer that and you know what to do...19:19
rcarrillocruzsridhar_ram: requirements are the libraries neeeded to run your project, test is for test19:19
rcarrillocruzwhen you run tox19:19
rcarrillocruzit installs both19:19
rcarrillocruzrequirements and test-requirements19:19
sridhar_ramAJaeger: we need it only for test19:19
rcarrillocruzfungi: i got an invite for tomorrow to have a call about infra-cloud19:19
rcarrillocruzsridhar_ram: then just put it on test-requirements19:19
AJaegersridhar_ram: then add it to test-requirements.19:20
rcarrillocruzfungi: i'll fwd shortly19:20
AJaegerkrot_vaca_jul19: could you comment, please?19:20
*** asettle has quit IRC19:20
krot_vaca_jul19AJaeger: Willdo19:20
sridhar_ramrcarrillocruz: fair enough, but we might pre-create few things as part of devstack install in the future.. then we need to move it to requirements.txt (seems logical) ?19:21
sridhar_ramsimilar to neutron pre-creating few initial networks19:21
mtreinishclarkb: heh, looks like my first crack at it didn't quite work: https://review.openstack.org/33843519:21
openstackgerritMerged openstack-infra/project-config: Introduce experimental job to test TinyIPA with less RAM  https://review.openstack.org/32799419:22
AJaegerclarkb: could you comment on https://review.openstack.org/337946, please? do we remove python34 testing from old branches or keep it?19:22
krot_vaca_jul19AJaeger, fungi: To use shell variable replacements in a jjb macro, I have to double escape it somehow, right?19:22
* AJaeger waves good bye for today19:22
mtreinishoh there is also gonna be a race with nodepool precaching the images and the gate jobs using them19:23
sridhar_ramAJaeger: rcarrillocruz: thanks for the clarification!19:23
pabelangerEmilienM: can you confirm, in gertty using 335101, if I CTRL-X (propose change to branch) I don't see stable/mitaka19:23
AJaegerkrot_vaca_jul19: for a template, you have to do it - use {{ instead of {, same for }}19:23
EmilienMpabelanger: weird19:24
*** PollyZ_ has joined #openstack-infra19:25
pabelangerjeblair: do you have a moment to review my comment about gertty^ and offer a suggestion on how to fix my branch issues?19:27
*** thorst has joined #openstack-infra19:27
openstackgerritMichael Krotscheck proposed openstack-infra/project-config: Always install nodejs from nodesource  https://review.openstack.org/33850819:27
krot_vaca_jul19fungi: Note, I haven't tested this manually. On vacation. ^^19:28
*** PollyZ has quit IRC19:28
anteayakrot_vaca_jul19: would you be willing to field a question on the storyboard branches api, specifically the feature to create a new branch?19:28
krot_vaca_jul19anteaya: I'm not contributing directly to storyboard right now.19:29
anteayakrot_vaca_jul19: I'll take that as no, thanks anyway19:29
*** asettle has joined #openstack-infra19:29
*** PollyZ has joined #openstack-infra19:30
*** vhosakot has quit IRC19:31
jeblairpabelanger: are you subscribed to the job?19:31
jeblairer19:31
jeblairpabelanger: are you subscribed to the project?19:31
*** PollyZ_ has quit IRC19:32
pabelangerjeblair: I am not. I used the search feature to find the review19:32
jeblairpabelanger: i think gertty may only know about branches for subscribed projects19:32
pabelangerokay, let me try that19:32
jeblairperhaps project branches are light-weight enough that we could sync all of them19:33
*** xenogear has quit IRC19:35
*** asettle has quit IRC19:36
krot_vaca_jul19AJaeger: Commented19:36
*** vhosakot has joined #openstack-infra19:44
*** ddieterly is now known as ddieterly[away]19:44
rcarrillocruzpabelanger , clarkb : have the centos dibs been uploaded?19:45
rcarrillocruzi believe greghaynes pushed a new release of dib with the revert from yesterday19:45
openstackgerritgreghaynes proposed openstack-infra/glean: Use blkid to grab config drive fs type  https://review.openstack.org/33555919:45
clarkbrcarrillocruz: yes all cloud regions should have that fix now19:45
clarkbrcarrillocruz: but there may be a few straggler nodes that nodepool needs to cycle out19:46
pabelangerrcarrillocruz: doing that now19:46
rcarrillocruzexcellent, let me recheck my change then19:46
rcarrillocruzalso, should I +A https://review.openstack.org/#/c/337692/19:46
rcarrillocruz?19:46
clarkbpabelanger: ya but the fix rcarrillocruz needs was everywhere last night except for rax-dfw19:46
*** jesusaur has quit IRC19:46
clarkband rax-dfw just finished today's upload so should be good19:46
pabelangeronly rax-ord and ovh-bhs1 left to upload19:46
pabelangerclarkb: ah, perfect19:46
fricklercould another core take a look at https://review.openstack.org/338492 and help fix ceilometer gate, pls?19:46
clarkbrcarrillocruz: should be fine, maybe double check with fungi if worried about it19:47
rcarrillocruzfungi: ^, the change to add myself as root19:47
clarkbfrickler: ya one sec19:48
clarkbfrickler: done19:48
fricklerclarkb: thx19:48
*** jesusaur has joined #openstack-infra19:49
*** zul has quit IRC19:51
*** ramishra has quit IRC19:51
pabelangerrcarrillocruz: +2, again19:52
*** ramishra has joined #openstack-infra19:52
*** esikachev has quit IRC19:53
pabelangerclarkb: slagle: centos-7-2-node looks to be online: telnet://104.239.167.66:808819:54
openstackgerritAnita Kuno proposed openstack-infra/storyboard: Add example commands for the Milestones api  https://review.openstack.org/33852219:54
*** njohnston has joined #openstack-infra19:54
fungircarrillocruz: approvinated19:55
openstackgerritMerged openstack-infra/project-config: Fix ceilometer testing  https://review.openstack.org/33849219:56
rcarrillocruzthx19:56
*** ayoung has joined #openstack-infra19:56
slaglepabelanger: cool, will keep an eye on it19:57
*** alex_xu has joined #openstack-infra19:58
*** asettle has joined #openstack-infra19:59
*** piet has quit IRC20:01
*** julim has quit IRC20:01
*** jamesmcarthur has quit IRC20:01
notmorganAJaeger: turns out it's a built-in now, the way to allow "passwordless" users is to change sql_mode - which would also impact root20:03
notmorganAJaeger: it's a change (default change) in mysql 5.7 and later20:03
*** Hal has joined #openstack-infra20:03
notmorganwhich.. xenial i think was the upgrade that pushed me over that version.20:04
*** Hal is now known as Guest7543720:04
therveclarkb, So you mentioned earlier that downloading from tarballs.openstack.org was crossing the ocean20:04
therveclarkb, Would it be possible to have dc specific locations for download?20:04
clarkbtherve: for ovh it does20:04
therveI guess like the pypi mirror20:04
*** julim has joined #openstack-infra20:05
clarkbtherve: we could possibly put that stuff into the afs system and potentially serve it that way20:05
mordredyah. we've talked about doing that before20:05
jeblair++20:05
clarkbI do still think it would be good to understand what the actual requirements are reather than just download a 550MB image every job20:05
mordred++20:05
therveclarkb, If we make the effort to make it smaller? :)20:05
openstackgerritMerged openstack-infra/system-config: Add rcarrillocruz as infra-root  https://review.openstack.org/33769220:05
*** piet has joined #openstack-infra20:05
*** rcernin_ has joined #openstack-infra20:06
*** hashar has joined #openstack-infra20:06
openstackgerritMerged openstack/diskimage-builder: Clear up "already provided" message  https://review.openstack.org/29096820:06
therveclarkb, Yeah from what I gathered we're in the middle of the road for no great reasons. The image still doesn't have everything we need to test everything that we wants, and it's still to big to be reliably fetched20:06
therveSo for the short term we'll try to get back to a simple fedora image I think20:07
therveFedora 24 has the good taste to be under 200M, so that'd be better than the 500 that we fetch right now20:07
*** ddieterly[away] is now known as ddieterly20:08
*** Apoorva has quit IRC20:09
*** Apoorva has joined #openstack-infra20:10
*** jamesmcarthur has joined #openstack-infra20:10
*** rcernin_ has quit IRC20:10
pabelangerspeaking of fedora-24, should see if I nodepool can build it20:12
*** sdake has quit IRC20:12
*** pvaneck has quit IRC20:13
*** pvaneck has joined #openstack-infra20:13
*** Apoorva has quit IRC20:14
*** bryan_att has joined #openstack-infra20:15
*** rcernin_ has joined #openstack-infra20:15
*** javeriak has quit IRC20:16
*** kzaitsev_mb has joined #openstack-infra20:17
*** makowals has quit IRC20:18
openstackgerritRaildo Mascena proposed openstack-infra/project-config: Adds ironic v3 only job  https://review.openstack.org/30430620:18
*** rcernin_ has quit IRC20:19
*** sdake has joined #openstack-infra20:19
*** rcernin_ has joined #openstack-infra20:19
*** rcernin_ has quit IRC20:20
*** Nikolay_St has quit IRC20:22
openstackgerritAnita Kuno proposed openstack-infra/storyboard: Add example commands for the Tags api  https://review.openstack.org/33853220:22
nibalizerrcarrillocruz: welcome !20:22
openstackgerritgordon chung proposed openstack-infra/project-config: re-enable voting on ceilometer+mongodb tempest tests  https://review.openstack.org/33853320:22
timrcrcarrillocruz: Woohoo!20:22
openstackgerritBen Nemec proposed openstack-infra/tripleo-ci: Don't try to get stack details if the stack doesn't exist  https://review.openstack.org/31933720:24
openstackgerritBen Nemec proposed openstack-infra/tripleo-ci: TEST: Delete the overcloud when finished  https://review.openstack.org/29732820:24
openstackgerritBen Nemec proposed openstack-infra/tripleo-ci: Stop piping yes to Heat  https://review.openstack.org/31773020:24
openstackgerritBen Nemec proposed openstack-infra/tripleo-ci: Test overcloud deletion in periodic job  https://review.openstack.org/29676520:24
*** rfolco_ has joined #openstack-infra20:24
notmorganAJaeger: ok i have a fix that will allow auto user creation, it's just dropping a config into my.cnf under [mysqld]20:24
*** rcernin has quit IRC20:24
*** rcernin has joined #openstack-infra20:25
*** eharney has quit IRC20:25
*** adrian_otto has joined #openstack-infra20:25
*** rcernin has quit IRC20:25
*** rcernin has joined #openstack-infra20:25
*** vhosakot has quit IRC20:26
*** rfolco has quit IRC20:26
openstackgerritBen Nemec proposed openstack-infra/tripleo-ci: TEST: Run undercloud idempotency test on all jobs  https://review.openstack.org/33611220:26
openstackgerritBen Nemec proposed openstack-infra/tripleo-ci: Add undercloud idempotency test  https://review.openstack.org/27921820:26
openstackgerritMehdi Abaakouk (sileht) proposed openstack-infra/project-config: ceilometer: Reenable mongodb gate job  https://review.openstack.org/33853820:26
openstackgerritMehdi Abaakouk (sileht) proposed openstack-infra/project-config: ceilometer: make correct mongodb job non-voting  https://review.openstack.org/33853820:27
*** vhosakot has joined #openstack-infra20:27
funginotmorgan: how much additional hassle is it to have the test fixture for the user also track a password?20:28
fungithat way the implementation would be portable across default configs of older and newer mysql20:28
notmorganfungi: not sure. just trying to determine what changed first locally so i know why20:28
notmorganfungi: ideally it should be easy.20:28
notmorganfungi: but... i'll need to go poke at it20:28
notmorgani just greatly dislike when things magically change20:29
fungii'm just hesitant to end up with "and if you're running jobs on xenial make sure you use this non-default database configuration"20:29
*** cody-somerville has joined #openstack-infra20:29
notmorgansure.20:29
* notmorgan goes off down another rabbit hole for a moment20:30
*** Apoorva has joined #openstack-infra20:31
openstackgerritBen Nemec proposed openstack-infra/tripleo-ci: Record Heat resource deployment times to Graphite  https://review.openstack.org/31430820:33
*** ddieterly is now known as ddieterly[away]20:33
slaglepabelanger: looks like it might be stuck where it should be doing a pip install. wonder if the right mirror is configured20:34
openstackgerritKien Ha proposed openstack-infra/jenkins-job-builder: Clean up import helpers in publishers  https://review.openstack.org/33854420:35
*** esikachev has joined #openstack-infra20:35
*** ddieterly[away] is now known as ddieterly20:36
*** piet has quit IRC20:37
*** piet has joined #openstack-infra20:38
openstackgerritChristian Fetzer proposed openstack-infra/jenkins-job-builder: Update Gitlab to >=1.2.4  https://review.openstack.org/33311320:38
*** Guest_84847 has joined #openstack-infra20:38
*** rcernin has quit IRC20:38
*** rdxnoam_ has joined #openstack-infra20:38
*** rcernin has joined #openstack-infra20:39
*** _sarob is now known as sarob20:39
*** Guest_84847 has quit IRC20:39
*** PollyZ has quit IRC20:39
*** PollyZ has joined #openstack-infra20:39
rdxnoam_Hello. I've setup CI and want to push to ci-sandbox to see if my gerrit-zuul-jenkins configuration is correct, but encounter a permissions error. Can anyone assist?20:40
pabelangerslagle: not sure, looks like delorean hung trying to setup the virtualenv20:40
pabelangerwonder if this is the issue rcarrillocruz was reporting20:40
rcarrillocruzpabelanger: i doubt it, i rechecked  change that was failing consistently on running virtualenv and passed20:41
rdxnoam_Or please point me in the right direction if this is not the correct channel...20:41
*** njohnston has left #openstack-infra20:41
rcarrillocruzcan i see the error slagle ?20:41
slaglercarrillocruz: we don't have one yet...just watching a running job20:42
pabelangerslagle: it is rax-iad, maybe a network issue too20:42
*** Guest_84755 has joined #openstack-infra20:42
slagleit appears hung creating a virtualenv, or on the next step, which is a pip install in the venv20:42
rcarrillocruzthe issue that happened yesterday was about centos nodes not being able to run virtualenv, i.e. command not found20:42
rcarrillocruzso doesn't look like your issue (hung virtualenv)20:42
slagleok, i think we're past that20:42
Guest_84755Allah is doing20:42
Guest_84755sun is not doing Allah is doing20:42
Guest_84755moon is not doing Allah is doing20:43
Guest_84755stars are not doing Allah is doing20:43
*** ChanServ sets mode: +b Guest_84755!*@*20:43
*** Guest_84755 was kicked by ChanServ (Banned: spamming is off-topic)20:43
mordredI honestly wonder what the person who runs that bot thinks they are accomplishing20:44
fungithe will of their deity, no doubt20:44
nibalizeri have somewhat memorized it, so its almost effective20:44
mgagne"bots are not doing Allah is doing" ? =)20:44
bkeroagain?20:44
mtreinishnibalizer: well, not until you start preaching the good word20:45
mordrednibalizer: yah - but that's only effective if the intent is for you to memorize a set of words :)20:45
slaglepabelanger: ok, i see the error now. i think it's trying to use the mirror server we use in the tripleo cloud. i thought i had unconfigured that, but will double check20:45
slagleoh, it's $http_proxy rather. i'll need to unset that for the multinode job20:46
mordrednibalizer: wow, it goes on a long time before it repeats: https://openstack.nimeyo.com/78412/openstack-dev-somebody-posted-this-on-%23openstack-meetng-420:47
mordrednibalizer: if you have that entire thing memorized, i'm _very_ impressed20:47
openstackgerritJames Slagle proposed openstack-infra/tripleo-ci: Add mulitnode CI job support to tripleo-ci  https://review.openstack.org/32477720:48
nibalizerbut it always starts with sun and moon20:48
nibalizeralso lol, nimeyo :(20:48
rdxnoam_can anyone help?20:48
nibalizerfungi: pleased you were able to get them to stop using the infringing logo20:48
funginibalizer: they also set the service to no longer post to our mailing lists. it's just a read-only archive now20:49
fungithough that took unsubscribing them before they'd pay attention to my hails and actually get back to me20:49
*** dprince has quit IRC20:49
*** mtanin___ has joined #openstack-infra20:50
*** mtanino has quit IRC20:51
pabelangerslagle: okay20:51
*** raildo is now known as raildo-afk20:51
*** kgiusti has left #openstack-infra20:52
*** sdake has quit IRC20:53
*** mtanin___ has quit IRC20:54
*** dfflanders has joined #openstack-infra20:54
pabelangerslagle: also, now that tripleo-test-cloud-rh2 is running AFS, you should be able to drop your proxies and start to rely more on AFS20:54
*** esikachev has quit IRC20:54
*** Nikolay_St has joined #openstack-infra20:55
EmilienMpabelanger: I guess we'll need to sed jenkins/zuul in puppet CI ? https://review.openstack.org/#/c/337821/20:57
*** Goneri has quit IRC20:57
EmilienMpabelanger: nevermind, we don't use hardcode the user20:58
*** acoles_ is now known as acoles20:58
*** dfflanders has quit IRC20:58
*** dfflanders has joined #openstack-infra20:59
*** jamesmcarthur has quit IRC21:01
*** rockyg has quit IRC21:01
EmilienMpabelanger: btw, it works fine: https://review.openstack.org/#/c/335106/ - thanks for this work!21:01
*** matt-borland has quit IRC21:02
EmilienMpabelanger: could you submit to master and liberty?21:02
*** sdake has joined #openstack-infra21:03
*** ajmiller_ has joined #openstack-infra21:03
*** rcernin has quit IRC21:03
*** rcernin has joined #openstack-infra21:04
fungircarrillocruz: what's the timezone on that conference call invite?21:04
*** trown is now known as trown|outtypewww21:04
*** lucas|afk has quit IRC21:05
rcarrillocruzfungi: that got 'translated' by outlook to my timezone, cest21:05
fungircarrillocruz: thanks, so utc+2 right now?21:06
*** karthikp_ has quit IRC21:06
rcarrillocruzfungi: yep21:06
*** ajmiller has quit IRC21:07
*** jcoufal has quit IRC21:07
fungiawesome. so call is for 14:30-15:00 utc i guess21:07
*** julim has quit IRC21:07
*** rcernin has quit IRC21:07
*** rcernin has joined #openstack-infra21:07
fungidoesn't conflict with anything on my schedule for tomorrow21:07
*** derekh has joined #openstack-infra21:08
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: Nothing to see here  https://review.openstack.org/11101121:08
*** jamesmcarthur has joined #openstack-infra21:10
rcarrillocruz++21:10
*** Nikolay_St has quit IRC21:10
*** lucasagomes has joined #openstack-infra21:12
*** asettle has quit IRC21:13
*** aysyd has quit IRC21:14
*** ldnunes_ has quit IRC21:15
*** dmsimard is now known as dmsimard|afk21:16
*** rhallisey has quit IRC21:16
*** aeng has quit IRC21:17
*** gouthamr_ has quit IRC21:20
*** dingyichen has joined #openstack-infra21:20
*** gouthamr has joined #openstack-infra21:21
rdxnoam_Help needed! I am trying to fill out contact information at https://review.openstack.org/#/settings/contact and receive a 'Server Error' message21:21
rdxnoam_can anyone help?21:21
*** aeng has joined #openstack-infra21:22
mordredfungi: I will be on a plane to asia, so I will not be able to make the call21:22
mordredfungi: although I can make occasional snarky comments in IRC if it's helpful21:23
*** karthikp_ has joined #openstack-infra21:23
fungirdxnoam_: were you following the instructions at http://docs.openstack.org/infra/manual/developers.html#account-setup ?21:24
rdxnoam_fungi: I have a launchpad account and ssh keys added and my zuul installation is connected properly. when i try to 'git review' it says that i need to fill out the contact information. Does this answer the question?21:25
openstackgerritK Jonathan Harker proposed openstack-infra/puppet-log_processor: Give each job's console log its own crm classifier  https://review.openstack.org/33855821:26
*** priteau has quit IRC21:27
fungirdxnoam_: it sounds like you're trying to test against the openstack-dev/sandbox repo, which requires a contributor agreement with contact information. if you're testing a ci system you should likely instead test against the openstack-dev/ci-sandbox repo which does not have those requirements21:29
rdxnoam_fungi: you are correct. I will try testing with ci-sandbox. thanks!21:29
pabelangerEmilienM: sure, I'll do that in a bit21:30
* fungi grumbles about new server he's trying to launch not puppeting far enough to create his account so that he can log in and find the remaining puppet errors in syslog21:30
*** rfolco_ has quit IRC21:34
fungioh, i think this is the "our groups aren't getting expanded" bug again21:34
openstackgerritAnita Kuno proposed openstack-infra/storyboard: Add example commands for the Subscriptions api  https://review.openstack.org/33856121:35
mordredfungi: gah21:36
fungiaha, nope, i see why!21:36
mordredfungi: sorry - I was afk for _way_ more of today than I was originally hoping to be - but I'm fairly captive all of tomorrow21:36
*** acoles is now known as acoles_21:37
*** rcernin has quit IRC21:37
openstackgerritJeremy Stanley proposed openstack-infra/system-config: Put signing01.ci in the signing group  https://review.openstack.org/33856221:38
fungimordred: 'twas MY fault! ^21:38
mordredfungi: oh good21:38
*** larainema has quit IRC21:39
fungialready had it in modules/openstack_project/files/puppetmaster/groups.txt but didn't also set the group in manifests/site.pp'21:39
mordredfungi: that actually shouldn't matter21:40
mordredfungi: oh - wait21:40
mordredno, you're right21:40
mordredit's still important21:40
*** ddieterly is now known as ddieterly[away]21:40
fungiat least i assume that's why the errors i'm seeing are about it not finding hiera keys i've defined in its group-specific hiera file21:41
mordredfungi: I'm about to send out the release naming polls. this time I added "you are receiving this because you are a foundation member" up at the top so that maybe I won't get quite as many "unsubscribe" emails :)21:41
fungimordred: heh, g'luck!21:41
mordredfungi: your supposition makes sense to me21:41
*** esberglu has quit IRC21:42
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: Remove the rewrite of TOCI_JOBTYPE  https://review.openstack.org/33856621:42
*** derekh has quit IRC21:43
fungithis workstation feels like a brand new system now that i've breathed life back into its cpu fan so it's not downclocking itself by an order of magnitude21:44
*** EricGonczer_ has quit IRC21:44
*** yamahata has quit IRC21:44
*** jamesmcarthur has quit IRC21:44
*** gordc has quit IRC21:44
stevebakerhey, heat's integration tests use this custom built image http://tarballs.openstack.org/heat-test-image/ but we'd like to switch to a vanilla centos or fedora image instead. I'd like to know if we should just use upstream download URLs or some infra cached url21:44
*** piet has quit IRC21:45
fungistevebaker: it's possible we bake one into our images already? i know for a while we were downloading a fedora image and stashing it on all our devstack images (back when we had separate images for devstack jobs)21:45
*** yamahata has joined #openstack-infra21:46
dstufftmordred: https://s.caremad.io/Hh9QXtE5cL/21:46
fungiclarkb: pabelanger: ^ do you know if that's still the case with our base images?21:46
clarkbI dont think it is21:46
*** ajmiller_ has quit IRC21:46
mordreddstufft: never21:46
*** ajmiller_ has joined #openstack-infra21:46
fungistevebaker: if memory serves, it was painful for _us_ because fedora's download site was unreliable and it would bomb our image updates when we couldn't get that file21:47
fungistevebaker: so i can only imagine how terrible it would be to try to retrieve it from them in a frequently-run job21:48
*** rvba has quit IRC21:48
anteayafungi: congratulations21:48
anteayaso that's what electronics need, fungi to breath on them21:48
anteayaif I get some new hard drives can I get them shipped to you so you can breath on them?21:49
stevebakerfungi: yes, thats right. we'd probably prefer centos anyway. I'll check but I think the RDO jobs rely on centos infra anyway so maybe we should just rely on http://cloud.centos.org/centos/7/images/21:49
fungistevebaker: we were discussing earlier with therve that it might make sense to start providing some larger files like that on our per-provider mirrors so that you don't end up having to retrieve them over the broader internet21:49
openstackgerritsebastian marcet proposed openstack-infra/openstackid-resources: Upgrade Laravel and ORM  https://review.openstack.org/32230721:50
*** nmagnezi has quit IRC21:50
stevebakerfungi: how about I start by trying IMAGE_URLS+=http://cloud.centos.org/centos/7/images/CentOS-7-x86_64-GenericCloud.qcow2 and see how that goes?21:50
fungistevebaker: it's a self-testing change i think so worth seeing what happens21:51
openstackgerritAnita Kuno proposed openstack-infra/storyboard: Add example commands for the Users api  https://review.openstack.org/33857021:51
*** savihou has quit IRC21:51
*** Guest75437 has quit IRC21:51
*** sigmavirus is now known as sigmavirus_away21:52
anteayafungi: correct me if I am wrong the only one that can create a modify a storyboard user is a storyboard admin, correct?21:52
anteayacreate and modify*21:53
*** vhosakot has quit IRC21:53
fungianteaya: i think storyboard users are auto-created on first login21:53
anteayaoh yes, that is true, hmmmm, okay thanks21:54
fungianteaya: i don't know of any other way to do it, but it's possible admin features have been added to allow creating/modifying users21:54
*** gyee has quit IRC21:54
anteayaI've made comments in the patch, hopefully I can get some direction21:55
anteayathank you21:55
anteayaif anyone feels like reviewing some storyboard api documentation example commands, I'd be grateful for the reviews: https://review.openstack.org/#/q/topic:expand-api-docs21:57
anteayathank you21:57
*** larainema has joined #openstack-infra21:57
*** salv-orlando has joined #openstack-infra21:57
*** mriedem is now known as mriedem_afk21:58
*** ddieterly[away] is now known as ddieterly22:03
openstackgerritMerged openstack-infra/system-config: Put signing01.ci in the signing group  https://review.openstack.org/33856222:04
*** cardeois_ has joined #openstack-infra22:05
*** tphummel has quit IRC22:05
*** tphummel has joined #openstack-infra22:07
*** cardeois has quit IRC22:09
*** cardeois_ has quit IRC22:09
*** thiagop has quit IRC22:10
openstackgerritSumit Naiksatam proposed openstack-infra/project-config: Add experimental testing job for GBP  https://review.openstack.org/33857722:10
clarkbjhesketh: any chance you can review https://review.openstack.org/#/c/336311/ as it will help us do the xenial job registrations22:11
*** mdrabe has quit IRC22:12
clarkbgreghaynes: hrm the glean sort cleanup seems to have tripped over something gentoo related22:13
clarkbgreghaynes: oh I think I see the issue, gentoo has other side effecting calls there22:14
*** Goneri has joined #openstack-infra22:15
*** kzaitsev_mb has quit IRC22:19
*** burgerk has quit IRC22:19
*** xyang1 has quit IRC22:19
*** kzaitsev_mb has joined #openstack-infra22:20
*** david-lyle_ has joined #openstack-infra22:21
*** cody-somerville has quit IRC22:21
*** dkehn__ has joined #openstack-infra22:21
*** dkehn_ has quit IRC22:24
*** Julien-zte has joined #openstack-infra22:25
*** david-lyle__ has joined #openstack-infra22:26
jheskethMorning22:26
jheskethclarkb: looking22:26
jheskethfungi: was the npm issue resolved?22:27
fungijhesketh: resolved by making stackviz ignore npm errors22:27
jheskethfungi: is that package missing on the slaves then??22:27
jhesketh(woops, didn't mean the anxious double '?')22:27
*** kzaitsev_mb has quit IRC22:28
fungijhesketh: stackviz is probably broken on xenial slaves that build from this point forward, at least until npm gets better on xenial22:28
fungibut incomplete stackviz installation should no longer impact our ability to create those images22:29
jheskethokay that's fair enough22:29
*** kzaitsev_mb has joined #openstack-infra22:30
openstackgerritKien Ha proposed openstack-infra/jenkins-job-builder: Update Grails plugin to use convert xml  https://review.openstack.org/33858322:30
*** amitgandhinz has quit IRC22:30
jheskethclarkb: with that change how will jobs that need a specific slave get the right node type22:30
clarkbjhesketh: you will create a job per node type22:31
jesusaurinfra-core: when you get a chance can you take a look at https://review.openstack.org/320728 and https://review.openstack.org/320729? I'd like to get some rudimentary testing in place on the logstash-filters project22:31
clarkbjhesketh: https://review.openstack.org/337905 is the rough idea22:31
clarkbprometheanfire: any idea why removing the sorts from https://review.openstack.org/#/c/337932/3/glean/cmd.py line 277 causes the tests to fail? we are editing a dict there which is not sorted information22:32
timothyb89fungi, jhesketh: stackviz was actually building properly again as of the last xenial build22:32
timothyb89it looks like the bad npm mirror fixed itself22:33
prometheanfirelooking22:33
*** amitgandhinz has joined #openstack-infra22:33
prometheanfireclarkb: tests expect an order, since dict's are unordered that needs to be 'fixed' on way is to sort on the actual code side22:34
prometheanfirethe real fix would be to have the tests not expect an order22:34
clarkbprometheanfire: ya you'll see at the end of that diff we have a single sort on the output side22:35
*** david-lyle_ is now known as david-lyle22:35
clarkbprometheanfire: whcich seems to work for red hat and debuntu tests but not gentoo22:35
clarkbprometheanfire: trying to simplify the glean code a little as it can be hard to follow right now22:36
*** Apoorva_ has joined #openstack-infra22:36
prometheanfireah22:36
prometheanfireya, it can be22:36
*** david-lyle__ has quit IRC22:36
prometheanfirethe ordering in the file counts for gentoo22:36
clarkbit does look like gentoo functions may write as it goes which is fine I can add the sorts back to it there, but line 277 is before that and only modifies that dict22:37
clarkbso I am confused22:37
prometheanfirecant have config eth0_42 without eth0 being set up iirc22:37
clarkband sorting iteration on a dict that only modifies a dict should be a noop sort22:37
clarkbbecaues it doesn't matter if I add foo to dict then bar or bar to dict then foo because dicts are unordred22:37
prometheanfireI'm not totally sure22:38
clarkbmaybe I should rebase and change the order of those cahnges and figure out the sorts later22:38
clarkb(but this sort (hahaha) of confusion is what I am trying to remove from glean22:39
*** Apoorva has quit IRC22:39
openstackgerritClark Boylan proposed openstack-infra/glean: Explain vlan setup with interface names  https://review.openstack.org/33793422:40
openstackgerritClark Boylan proposed openstack-infra/glean: Explain difference between config drive files  https://review.openstack.org/33793322:40
openstackgerritClark Boylan proposed openstack-infra/glean: Sort output data once before we write it  https://review.openstack.org/33793222:40
prometheanfireya, I think I coppied redhat's setup22:40
prometheanfireso, blame them22:40
clarkbyes debian does it too, but for red hat and debian it doesn't appear to be necessary there which is what I expect. They fall through to the sort at the end of that diff and work fine22:40
prometheanfirethe tests are the same?22:41
clarkbno they all assert different files and contents22:41
prometheanfireya22:43
prometheanfireit's been a while22:43
*** salv-orl_ has joined #openstack-infra22:43
*** dimtruck is now known as zz_dimtruck22:43
*** hashar has quit IRC22:45
*** salv-orlando has quit IRC22:46
*** tpsilva has quit IRC22:48
openstackgerritKien Ha proposed openstack-infra/jenkins-job-builder: Update Grails plugin to use convert xml  https://review.openstack.org/33858322:48
openstackgerritKien Ha proposed openstack-infra/jenkins-job-builder: Update Grails plugin to use convert xml  https://review.openstack.org/33858322:50
jheskethclarkb: (sorry was on a call). so okay, you're not going to turn this new option on until all the jobs are renamed appropriately?22:50
clarkbjhesketh: well it will work today as is if we turn it one becuase none of our jobs use ZUUL_NODE parameter22:50
clarkbjhesketh: but yes plan is to bifurcate things to handle trusty and xenial at the same time using job per image/label22:51
jheskethclarkb: what about nodepool demand calculations... don't they use the node label ?22:51
*** ddieterly is now known as ddieterly[away]22:52
*** ddieterly[away] is now known as ddieterly22:52
*** ddieterly has quit IRC22:52
clarkboh hrm good question22:52
*** Sukhdev has quit IRC22:53
*** rbrndt has quit IRC22:55
clarkbjhesketh: it does seem to accomodate it, not sure if it is necessary yet22:56
clarkbya if I am reading getNeededWorkers() properly we will stop satisfying demand if we go this route :(22:58
jeblairif we register no ":label" jobs, then demand for those jobs will be ignored22:58
jeblair(so it will fall back on min-ready)22:58
clarkbjeblair: ya22:58
*** tlian2 has joined #openstack-infra22:59
clarkbso things will "work" just not as dynamically22:59
fungioh, er yeah that wasn't a side effect i considered previously22:59
clarkbsomething to chew on :/23:00
jeblairclarkb: nodepool does have the information to map job names to workers though23:00
clarkbjeblair: even if the jobs don't register with the labels?23:00
jeblairclarkb: geard will tell it "ubuntu-trusty-1234" can run "build:foo", and nodepool knows than "ubuntu-trusty-1234" is an ubuntu-trusty node type (since nodepool built it and has a current node table record for it)23:01
jeblairclarkb: so we should be able to have it make the connection that "build:foo" is demand for "ubuntu-trusty"23:01
*** tlian has quit IRC23:02
clarkbgotcha23:02
fungigood point23:02
jeblaira fix like that is kind of nice in that it works regardless of whether you use the new switch in zuul-launcher23:02
clarkbya23:02
* clarkb pokes at getNeededNodes()23:03
*** ociuhandu has quit IRC23:03
jeblairclarkb: i *think* that will mean also issuing a geard 'workers' admin request23:03
clarkboh wait maybe it already infers that?23:03
clarkbthere is handling for 'unspecified_jobs'23:04
jeblairclarkb: that's to map demand for "build:foo" to a node, but it requires having previously seen "build:foo:trusty"23:04
jeblairs/node/image/23:04
jeblairclarkb: otherwise, it will hit the "if not workers: continue" path23:05
clarkbya  Isee that now, need to populate the job_workers_map dict23:05
*** hongbin has quit IRC23:05
clarkbI think I can see how to add this though23:06
jeblaircool23:06
openstackgerritVictor Ryzhenkin proposed openstack-infra/project-config: Add openstack/fuel-plugin-murano-tests project  https://review.openstack.org/33215123:06
*** Apoorva has joined #openstack-infra23:06
*** Apoorva_ has quit IRC23:07
*** kien-ha has quit IRC23:11
*** rdxnoam_ has quit IRC23:11
*** thorst has quit IRC23:16
*** thorst has joined #openstack-infra23:16
openstackgerritClark Boylan proposed openstack-infra/nodepool: Infer node demand from workers listing  https://review.openstack.org/33859823:20
clarkbjeblair: ^ something like that. Still very much WIP and needs test(s) but I think thats the general shape of it23:20
clarkbjeblair: for the names coming back do the workers register themselves with their label or their hostname or their fqdn?23:21
*** rlandy has quit IRC23:21
clarkbI think zuul-launcher sets that value when it registers as a worker23:21
jeblairclarkb: 272 104.130.136.141 ubuntu-xenial-rax-dfw-2296650 : build:gate-puppet-oslo-puppet-beaker-rspec-ubuntu-xenial23:22
jeblaireasiest way to be sure :)23:22
*** HeOS has quit IRC23:23
*** HeOS has joined #openstack-infra23:23
*** thorst_ has joined #openstack-infra23:23
*** zul has joined #openstack-infra23:24
*** arosen12 has joined #openstack-infra23:24
*** thorst has quit IRC23:25
*** tphummel has quit IRC23:25
arosen12Hi, I'm wondering why I'm getting this error: Job gate-vmware-nsx-python35-nv not defined  here: https://review.openstack.org/#/c/338402/1   ?  Would someone mind pointing out what I'm missing?23:25
*** asselin__ has quit IRC23:26
arosen12It seems like gate-networking-ovn-python35-nv is defined the same way.23:26
clarkbjeblair: ok so its the hostname, which  Ican map to a label type23:26
jeblairclarkb: yeah, i think it's intended to be the node name23:27
clarkbarosen12: you have to define it in jenkins/jobs/projects.yaml should be plenty of examples there if you grep for 35-nv23:27
*** roxanagh_ has quit IRC23:27
jeblairclarkb: as in, i think it's this value:23:27
jeblair    def assignViaGearman(self):23:27
jeblair        args = dict(name=self.node.nodename,23:27
*** thorst_ has quit IRC23:28
clarkbjeblair: oh good that value is all nodepoo lmanaged so should be 1:1 in all the places we pass it around23:28
arosen12clarkb:  ah i see. Thanks.23:29
jeblairclarkb: left a suggestion on the change23:29
openstackgerritAaron Rosen proposed openstack-infra/project-config: Add non voting python34/35 checks for vmware-nsx  https://review.openstack.org/33840223:29
clarkbjeblair: good point23:30
jeblairclarkb: i double checked zuul-launcher -- that is the value that's passed down and used as the gearman worker name, so we should be gtg23:30
openstackgerritAaron Rosen proposed openstack-infra/project-config: Update gitignore .swp -> .sw?  https://review.openstack.org/33860223:31
openstackgerritArmando Migliaccio proposed openstack-infra/project-config: Add job configuration for unit testing against neutron-lib source  https://review.openstack.org/33860323:31
clarkbjeblair: I need to map ubuntu-xenial-rax-dfw-2296650 to ubuntu-xenial though right?23:31
openstackgerritJames Slagle proposed openstack-infra/tripleo-ci: Add mulitnode CI job support to tripleo-ci  https://review.openstack.org/32477723:31
clarkbjeblair: and actually record foo-job: ubuntu-xenial23:31
pabelangerfungi: stevebaker: We do have some images mirrored to AFS: http://mirror.dfw.rax.openstack.org/centos/7/os/x86_64/ I am not sure if that works for you23:31
pabelangerfungi: stevebaker: otherwise, I am sure we add things to our rsync mirror script if needed23:32
jheskethclarkb: I don't think your nodepool change is correct... I've lfet a comment23:32
clarkbjhesketh: ya thats what we were just dicsussing23:33
docaedoAre any of the infra servers using a swift store for anything? We're going to need swift in the not distant future for the app catalog...23:33
clarkbjhesketh: and what my todo comment is for23:33
jheskethah, sorry23:33
jeblairclarkb, jhesketh: yep, sorry i glossed over that.23:33
*** PollyZ has quit IRC23:33
clarkbjhesketh: np glad that people are catching the issues :)23:34
jheskethso I'm not sure if geard knows anything about a noode type... it can map a job to a node, but that's the individual node... further that individual node is a zuul-launcher so depending on which slaves zuul-launcher has connected it could be managing multiple node types23:34
jeblairso it will need a db connection so it can query each of the nodes to find out its type.  alternatively, it could be ugly and parse the name.23:34
clarkbjeblair: I think I can also drop the unspecified handling entirely23:35
clarkbsince everythin will be either directly specified or infered23:35
clarkbjhesketh: ^23:35
jeblairjhesketh: zuul-launcher behaves like jenkins gearman plugin in that it has a connection per attached node -- the gearman identifier used by that connection is the nodepool node name23:36
*** pt_15 has quit IRC23:36
jheskethjeblair: ah, of course23:36
*** adrian_otto has quit IRC23:36
stevebakerpabelanger: ok, thanks. Looks like we could add some qcow images to the rsync if what I'm doing works out23:36
jeblairclarkb: i think so too23:36
*** rfolco_ has joined #openstack-infra23:36
jheskethjeblair: that'll still require parsing the node name most likely23:36
pabelangerstevebaker: yup, shouldn't be to hard to do just that23:37
jheskethor now I see what you mean by querying the db23:37
jeblairjhesketh: the *right* way would be to do a db lookup by node name to get the node record which also has the image type for the node23:37
*** karthikp_ has quit IRC23:37
jheskeththat doesn't seem too bad surely?23:37
jheskethor difficult etc23:37
jeblairjhesketh: no... the thing i'm worried about is whether we would have lock contention...23:38
jeblairwe might be okay though23:38
jheskethwhat would the contention be on?23:39
jeblair(i mean, it's read only, so it's no worse than issuing a 'nodepool list', which usually succeeds)23:39
jeblairjhesketh: probably nothing, now that i've talked myself through that :)23:39
jhesketh:-)23:39
jeblairso yeah, i lean toward that and against parsing the field :)23:40
*** itisha has quit IRC23:40
*** binbincong has quit IRC23:41
fungidocaedo: technically gerrit does. we have a script in jeepyb that updates an rss feed of reviews for some projects and publishes it to a swift container (no idea if it still works or if anyone uses it). however for the app store case i think puppet can just be made to put some swift credentials in a file on disk and the app store backend could just know to use those? we'd create the container by hand23:41
*** binbincong has joined #openstack-infra23:41
fungiand put the dedicated account for it in hiera so puppet would have it23:41
*** Julien-zte has quit IRC23:41
openstackgerritMatt Mulsow proposed openstack-infra/glean: Added support for setting bond-lacp-rate and bond-xmit_hash_policy  https://review.openstack.org/33860723:42
*** sdake has quit IRC23:43
docaedofungi: sounds like a good plan - part of using glare for the backend will require it to have a place to put the binaries23:43
docaedofungi: and just dumping them on disk of that VM would not be smart, but it handles access to swift nicely and because of the magic of replication and so on, that seems a better choice23:44
fungidocaedo: cool, so presumably we make the glare config an erb template with variable expansion for the swift details/creds23:44
fungidocaedo: or... how do you configure glare to access swift if not in a config file?23:45
docaedofungi: it would be a config file that puppet would handle, so we are on the same page23:45
docaedofungi: we'll also need a database, but I hear that's pretty easy and it'll just be a trove thing, so we'll coordinate those creds as well23:45
fungiyeah, that's what i meant by "put some swift credentials in a file on disk"23:45
fungiand yes, we can trove that23:46
docaedofungi: I'm vaguely hopeful that this will not be terribly difficult to add the the server, and I'll be doing it all through an update to the puppet-apps_site repo23:46
fungidocaedo: well, all except for a minor patch to system-config to plumb the hiera bits through23:47
fungijust a few lines added to manifests/site.pp there23:47
docaedofungi: ah yes - don't worry I'll be pestering you fine folks for feedback and details pretty soon23:48
fungiwe're here!23:48
docaedothank goodness for that!23:48
openstackgerritMerged openstack/diskimage-builder: Make ubuntu-core support releases  https://review.openstack.org/29414923:49
jeblairdoesn't ubuntu already support their releases?  for a few years anyway?23:49
* jeblair likes the idea of landing patches that make people support things23:50
fungii support that23:51
cineramahee hee23:51
openstackgerritClark Boylan proposed openstack-infra/nodepool: Infer node demand from workers listing  https://review.openstack.org/33859823:52
clarkbjeblair: jhesketh ^ I think that is much much closer now23:52
clarkbjeblair: jhesketh still cimpletely untested but I got so much good feedback I figured I would push early :)23:52
*** roxanagh_ has joined #openstack-infra23:54
openstackgerritJames Slagle proposed openstack-infra/tripleo-ci: Add mulitnode CI job support to tripleo-ci  https://review.openstack.org/32477723:55
*** xarses has quit IRC23:56
clarkbjeblair: jhesketh but also its getting late in the date and I just had 3 trees get trimmed so should get utside and see what they did (protip fruit trees are apparently terribleness)23:57
jeblairclarkb: i just planted 3!  (or 13 depending on how you count).  what kind?23:58
fungiwe have fig trees. not too terrible23:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!