Monday, 2017-01-09

*** sflanigan has joined #openstack-infra00:00
*** sflanigan has joined #openstack-infra00:00
*** masayukig_ is now known as masayukig00:00
*** sdake has joined #openstack-infra00:01
*** dingyichen has joined #openstack-infra00:05
*** sdake has quit IRC00:08
*** aspiers has joined #openstack-infra00:08
*** salv-orlando has quit IRC00:15
*** rbuzatu has joined #openstack-infra00:23
*** sdake has joined #openstack-infra00:24
*** sdake has quit IRC00:25
*** rbuzatu_ has joined #openstack-infra00:25
*** rbuzatu has quit IRC00:27
openstackgerritJames Slagle proposed openstack-infra/tripleo-ci: Implement major upgrade for Newton to Ocata  https://review.openstack.org/40483100:27
*** wolverineav has joined #openstack-infra00:30
*** timrc has quit IRC00:32
*** wolverineav has quit IRC00:35
*** jlk has quit IRC00:35
*** sdake has joined #openstack-infra00:38
*** Julien-zte has joined #openstack-infra00:39
*** ianychoi has quit IRC00:40
*** Julien-zte has quit IRC00:40
*** yujunz has joined #openstack-infra00:41
*** Julien-zte has joined #openstack-infra00:41
*** jamielennox is now known as jamielennox|away00:41
*** ianychoi has joined #openstack-infra00:44
*** kjorgensen has joined #openstack-infra00:54
*** kjorgensen has quit IRC00:58
*** jamielennox|away is now known as jamielennox00:58
*** sdake_ has joined #openstack-infra01:00
*** sdake has quit IRC01:01
*** yujunz has quit IRC01:02
*** yujunz has joined #openstack-infra01:02
*** Julien-zte has quit IRC01:05
*** jog0 has joined #openstack-infra01:05
*** jog0 has quit IRC01:05
*** jog0 has joined #openstack-infra01:05
*** yujunz has quit IRC01:06
*** Julien-zte has joined #openstack-infra01:06
*** Julien-zte has quit IRC01:06
*** Julien-zte has joined #openstack-infra01:08
*** Julien-zte has quit IRC01:08
*** yujunz has joined #openstack-infra01:08
*** Julien-zte has joined #openstack-infra01:09
*** yujunz has quit IRC01:09
*** Julien-zte has quit IRC01:10
*** Julien-zte has joined #openstack-infra01:11
*** yujunz has joined #openstack-infra01:11
*** Julien-zte has quit IRC01:12
*** dirk has quit IRC01:12
*** dougwig has quit IRC01:12
*** rattboi has quit IRC01:12
*** fbo has quit IRC01:12
*** cschwede has quit IRC01:12
*** mattymo has quit IRC01:12
*** mrunge has quit IRC01:12
*** jcook has quit IRC01:12
*** llu has quit IRC01:12
*** aarefiev_afk has quit IRC01:12
*** erlon has quit IRC01:12
*** jesusaur has quit IRC01:12
*** rosmaita has quit IRC01:12
*** med_ has quit IRC01:12
*** tobiash_ has quit IRC01:12
*** Qiming has quit IRC01:12
*** skraynev has quit IRC01:12
*** kambiz has quit IRC01:12
*** mattfarina has quit IRC01:12
*** comstud has quit IRC01:12
*** jak-jak has quit IRC01:12
*** kfox1111 has quit IRC01:12
*** russellb has quit IRC01:12
*** ryotagami has quit IRC01:12
*** morgabra has quit IRC01:12
*** patrickeast has quit IRC01:12
*** lewo has quit IRC01:12
*** calebb has quit IRC01:12
*** mcarden has quit IRC01:12
*** asaprykin has quit IRC01:12
*** jbryce has quit IRC01:12
*** mgagne has quit IRC01:12
*** vkmc has quit IRC01:12
*** mordred has quit IRC01:12
*** mattoliverau has quit IRC01:12
*** khappone_ has quit IRC01:12
*** DinaBelova has quit IRC01:12
*** jhesketh has quit IRC01:12
*** dims has quit IRC01:12
*** xhku has quit IRC01:12
*** csmart has quit IRC01:12
*** darrenc has quit IRC01:12
*** Daviey has quit IRC01:12
*** olaph has quit IRC01:12
*** yanyanhu has joined #openstack-infra01:12
*** kfox1111 has joined #openstack-infra01:12
*** csmart has joined #openstack-infra01:12
*** matt6434 has joined #openstack-infra01:12
*** rosmaita has joined #openstack-infra01:12
*** morgabra_ has joined #openstack-infra01:12
*** Daviey has joined #openstack-infra01:12
*** cschwede has joined #openstack-infra01:12
*** xhku has joined #openstack-infra01:12
*** med_ has joined #openstack-infra01:12
*** mordred has joined #openstack-infra01:12
*** Julien-zte has joined #openstack-infra01:12
*** darrenc_ has joined #openstack-infra01:12
*** lewo_ has joined #openstack-infra01:12
*** skraynev has joined #openstack-infra01:12
*** comstud has joined #openstack-infra01:12
*** jbryce_ has joined #openstack-infra01:12
*** calebb has joined #openstack-infra01:12
*** DinaBelova has joined #openstack-infra01:12
*** olaph has joined #openstack-infra01:13
*** russellb has joined #openstack-infra01:13
*** med_ is now known as Guest3422001:13
*** mattymo has joined #openstack-infra01:13
*** mrunge has joined #openstack-infra01:13
*** kambiz has joined #openstack-infra01:13
*** yujunz has quit IRC01:13
*** jbryce_ is now known as jbryce01:13
*** lewo_ is now known as lewo01:13
*** aarefiev_afk has joined #openstack-infra01:13
*** jhesketh has joined #openstack-infra01:13
*** jesusaur has joined #openstack-infra01:13
*** rattboi has joined #openstack-infra01:13
*** ryotagami has joined #openstack-infra01:13
*** khappone has joined #openstack-infra01:13
*** mcarden has joined #openstack-infra01:13
*** vkmc has joined #openstack-infra01:13
*** vkmc has quit IRC01:13
*** vkmc has joined #openstack-infra01:13
*** Guest96631 has joined #openstack-infra01:13
*** tobiash has joined #openstack-infra01:13
*** jak-jak has joined #openstack-infra01:13
*** fbo has joined #openstack-infra01:13
*** darrenc_ is now known as darrenc01:14
*** Qiming has joined #openstack-infra01:14
*** dims has joined #openstack-infra01:14
*** jcook has joined #openstack-infra01:15
*** llu has joined #openstack-infra01:15
*** dougwig has joined #openstack-infra01:15
*** Julien-zte has quit IRC01:15
*** asaprykin has joined #openstack-infra01:16
*** Julien-zte has joined #openstack-infra01:16
*** erlon has joined #openstack-infra01:16
*** mriedem has quit IRC01:16
*** patrickeast has joined #openstack-infra01:17
*** Julien-zte has quit IRC01:17
*** mattfarina has joined #openstack-infra01:18
*** dirk has joined #openstack-infra01:18
*** timrc has joined #openstack-infra01:19
*** Julien-zte has joined #openstack-infra01:19
*** yujunz has joined #openstack-infra01:19
*** Julien-zte has quit IRC01:20
*** Julien-zte has joined #openstack-infra01:20
*** yujunz-zte has joined #openstack-infra01:21
*** Julien-zte has quit IRC01:21
*** liujiong has joined #openstack-infra01:21
*** Julien-zte has joined #openstack-infra01:22
*** yujunz has quit IRC01:22
*** Julien-zte has quit IRC01:22
*** yujunz-zte has quit IRC01:22
*** yujunz_ has joined #openstack-infra01:22
*** Julien-zte has joined #openstack-infra01:23
*** Mmike has quit IRC01:23
*** xgerman has quit IRC01:23
*** sweston has quit IRC01:23
*** Hunner has quit IRC01:23
*** uberjay has quit IRC01:23
*** bmjen has quit IRC01:23
*** timrc_ has quit IRC01:23
*** briancurtin has quit IRC01:23
*** johnsom has quit IRC01:23
*** ogelbukh has quit IRC01:23
*** YorikSar has quit IRC01:23
*** jeckersb has quit IRC01:23
*** ansiwen has quit IRC01:23
*** dhellmann has quit IRC01:23
*** szaher has quit IRC01:23
*** dangers_away has quit IRC01:23
*** akscram has quit IRC01:23
*** davidlenwell has quit IRC01:23
*** warpc has quit IRC01:23
*** edwarnicke has quit IRC01:23
*** knikolla has quit IRC01:23
*** jistr has quit IRC01:23
*** morgan has quit IRC01:23
*** lbragstad has quit IRC01:23
*** lin_yang has quit IRC01:23
*** berendt has quit IRC01:23
*** sridhar_ram has quit IRC01:23
*** dasm has quit IRC01:23
*** JayF has quit IRC01:23
*** zigo has quit IRC01:23
*** michaelxin has quit IRC01:23
*** lcastell has quit IRC01:23
*** antwash has quit IRC01:23
*** panda|pto has quit IRC01:23
*** jmccrory has quit IRC01:23
*** jpmaxman has quit IRC01:23
*** kevinshort has quit IRC01:23
*** dtroyer has quit IRC01:23
*** amit213 has quit IRC01:23
*** fmccrthy has quit IRC01:23
*** auggy has quit IRC01:23
*** igorbelikov has quit IRC01:23
*** sslypushenko_ has quit IRC01:23
*** niedbalski has quit IRC01:23
*** melwitt has quit IRC01:23
*** verdurin has quit IRC01:23
*** zerick has quit IRC01:23
*** redrobot has quit IRC01:23
*** hemna has quit IRC01:23
*** yujunz_ has quit IRC01:23
*** niedbalski has joined #openstack-infra01:23
*** bmjen has joined #openstack-infra01:23
*** zigo has joined #openstack-infra01:23
*** lbragstad_ has joined #openstack-infra01:23
*** jmccrory_ has joined #openstack-infra01:23
*** YorikSar has joined #openstack-infra01:23
*** melwitt has joined #openstack-infra01:23
*** hemna has joined #openstack-infra01:23
*** zerick has joined #openstack-infra01:23
*** dangers_away has joined #openstack-infra01:23
*** dasm has joined #openstack-infra01:23
*** panda|pto has joined #openstack-infra01:24
*** jeckersb has joined #openstack-infra01:24
*** lcastell has joined #openstack-infra01:24
*** akscram has joined #openstack-infra01:24
*** timrchavez_ has joined #openstack-infra01:24
*** melwitt is now known as Guest8661901:24
*** JayF has joined #openstack-infra01:24
*** antwash has joined #openstack-infra01:24
*** uberjay has joined #openstack-infra01:24
*** Hunner has joined #openstack-infra01:24
*** Hunner has quit IRC01:24
*** Hunner has joined #openstack-infra01:24
*** yujunz has joined #openstack-infra01:24
*** dasm is now known as Guest368401:24
*** verdurin has joined #openstack-infra01:24
*** redrobot has joined #openstack-infra01:24
*** Julien-zte has quit IRC01:24
*** redrobot is now known as Guest5979201:24
*** jistr has joined #openstack-infra01:24
*** ansiwen has joined #openstack-infra01:24
*** Mmike has joined #openstack-infra01:25
*** Julien-zte has joined #openstack-infra01:25
*** jmccrory_ is now known as jmccrory01:25
*** kevinshort has joined #openstack-infra01:25
*** dhellmann has joined #openstack-infra01:25
*** szaher has joined #openstack-infra01:25
*** Julien-zte has quit IRC01:25
*** Julien-zte has joined #openstack-infra01:26
*** edwarnicke has joined #openstack-infra01:26
*** warpc has joined #openstack-infra01:26
*** Julien-zte has quit IRC01:26
*** jlk has joined #openstack-infra01:26
*** jlk has quit IRC01:26
*** jlk has joined #openstack-infra01:26
*** lin_yang has joined #openstack-infra01:26
*** dtroyer has joined #openstack-infra01:28
*** Julien-zte has joined #openstack-infra01:28
*** jpmaxman has joined #openstack-infra01:28
*** Julien-zte has quit IRC01:28
*** briancurtin has joined #openstack-infra01:28
*** auggy has joined #openstack-infra01:28
*** igorbelikov has joined #openstack-infra01:28
*** Julien-zte has joined #openstack-infra01:29
*** fmccrthy has joined #openstack-infra01:29
*** ogelbukh has joined #openstack-infra01:29
*** sweston has joined #openstack-infra01:29
*** davidlenwell has joined #openstack-infra01:29
*** sslypushenko_ has joined #openstack-infra01:29
*** amit213 has joined #openstack-infra01:29
*** Julien-zte has quit IRC01:30
*** Julien-zte has joined #openstack-infra01:32
*** rkukura has joined #openstack-infra01:32
*** Julien-zte has quit IRC01:32
*** xgerman has joined #openstack-infra01:33
*** Julien-zte has joined #openstack-infra01:33
*** Julien-zte has quit IRC01:34
*** Julien-zte has joined #openstack-infra01:34
*** michaelxin has joined #openstack-infra01:35
*** tuanluong has joined #openstack-infra01:35
*** morgan has joined #openstack-infra01:36
*** Julien-zte has quit IRC01:36
*** Julien-zte has joined #openstack-infra01:37
*** ihrachys has quit IRC01:38
*** berendt has joined #openstack-infra01:39
*** johnsom has joined #openstack-infra01:39
*** sridhar_ram has joined #openstack-infra01:39
*** thorst has joined #openstack-infra01:39
*** ameade has quit IRC01:42
*** dtroyer has quit IRC01:43
*** Julien-zte has quit IRC01:43
*** ameade has joined #openstack-infra01:43
*** thorst has quit IRC01:44
*** Julien-zte has joined #openstack-infra01:44
*** dtroyer has joined #openstack-infra01:45
*** Julien-zte has quit IRC01:46
*** tomhambleton has quit IRC01:46
*** salv-orlando has joined #openstack-infra01:46
*** Julien-zte has joined #openstack-infra01:46
*** tomhambleton has joined #openstack-infra01:47
*** matt6434 is now known as mattoliverau01:48
*** salv-orlando has quit IRC01:50
*** lewo has quit IRC01:51
*** lewo has joined #openstack-infra01:51
*** sslypushenko_ has quit IRC01:54
*** sslypushenko_ has joined #openstack-infra01:55
*** thorst has joined #openstack-infra01:56
*** ildikov has quit IRC01:57
*** ildikov has joined #openstack-infra02:00
openstackgerritDavanum Srinivas (dims) proposed openstack-infra/devstack-gate: Remove USE_PYTHON3 usage  https://review.openstack.org/41772002:03
*** Julien-zte has quit IRC02:04
*** Julien-zte has joined #openstack-infra02:05
*** thorst has quit IRC02:09
*** ihrachys has joined #openstack-infra02:13
*** cody-somerville has quit IRC02:17
*** booleanfour has quit IRC02:17
*** hyakuhei has quit IRC02:19
*** xgerman has quit IRC02:20
*** hyakuhei has joined #openstack-infra02:22
*** csatari has quit IRC02:28
*** csatari has joined #openstack-infra02:28
*** liusheng has joined #openstack-infra02:32
*** Julien-zte has quit IRC02:37
*** Julien-zte has joined #openstack-infra02:38
*** yujunz has quit IRC02:38
*** yujunz has joined #openstack-infra02:39
*** sdake_ has quit IRC02:40
*** makowals has quit IRC02:41
*** yujunz has quit IRC02:41
*** dave-mccowan has joined #openstack-infra02:41
*** yujunz has joined #openstack-infra02:42
*** makowals has joined #openstack-infra02:42
*** samueldmq has quit IRC02:42
*** ashtokolov has quit IRC02:42
*** aglarendil has quit IRC02:42
*** jraim has quit IRC02:42
*** larainema has quit IRC02:42
*** skath has quit IRC02:42
*** fkautz has quit IRC02:42
*** dham1 has quit IRC02:42
*** erlon has quit IRC02:42
*** samueldmq has joined #openstack-infra02:42
*** zigo has quit IRC02:43
*** puiterwijk|cld has quit IRC02:43
*** fkautz has joined #openstack-infra02:43
*** erlon has joined #openstack-infra02:43
*** larainema has joined #openstack-infra02:43
*** xnox has quit IRC02:43
*** sdake has joined #openstack-infra02:44
*** khappone_ has joined #openstack-infra02:44
*** khappone has quit IRC02:44
*** jroll has quit IRC02:44
*** alexm2 has quit IRC02:44
*** adam_g has quit IRC02:44
*** lVathan[m] has quit IRC02:44
*** calebb has quit IRC02:45
*** adam_g has joined #openstack-infra02:45
*** alexm2 has joined #openstack-infra02:45
*** jroll has joined #openstack-infra02:45
*** xnox has joined #openstack-infra02:45
*** calebb has joined #openstack-infra02:47
*** dham1 has joined #openstack-infra02:47
*** salv-orlando has joined #openstack-infra02:47
*** lVathan[m] has joined #openstack-infra02:48
*** zigo has joined #openstack-infra02:48
*** jraim has joined #openstack-infra02:49
*** ashtokolov has joined #openstack-infra02:49
*** skath has joined #openstack-infra02:50
*** gildub has quit IRC02:51
*** salv-orlando has quit IRC02:51
*** aglarendil_ has joined #openstack-infra02:52
*** david-lyle has quit IRC02:55
*** claudiub has quit IRC02:57
*** ihrachys has quit IRC02:59
jianghuawhi mordred.02:59
*** ihrachys has joined #openstack-infra02:59
jianghuawI moved the network setup from nodepool.yaml to clouds.yaml. But I met the following errors when boot server:03:00
jianghuawpenStackCloudException: Error in creating instance (Inner Exception: Multiple possible networks found, use a Network ID to be more specific. (HTTP 409) (Request-ID: req-fd5530b3-94df-4b04-a54f-ec55f9b4d2b1))03:00
*** EricGonc_ has joined #openstack-infra03:00
jianghuawsee the configuration setup at: http://paste.openstack.org/show/594249/03:01
jianghuawIt looks like nodepool doesn't use the network setup from clouds.yaml.03:01
*** andymccr has quit IRC03:01
*** andymccr has joined #openstack-infra03:02
*** EricGonczer_ has quit IRC03:03
jianghuawmordred, are you around?03:03
*** dtroyer has quit IRC03:03
*** erikmwilson has quit IRC03:03
openstackgerritrajiv proposed openstack-infra/subunit2sql: H402 hacking have been deprecated  https://review.openstack.org/41772703:04
*** dtroyer has joined #openstack-infra03:06
*** EricGonc_ has quit IRC03:07
*** saibarspeis has quit IRC03:07
*** xarses has joined #openstack-infra03:07
*** erikmwilson has joined #openstack-infra03:07
*** saibarspeis has joined #openstack-infra03:08
*** xarses_ has joined #openstack-infra03:10
*** ramishra has quit IRC03:10
*** xarses_ has quit IRC03:10
*** xarses_ has joined #openstack-infra03:11
*** ramishra has joined #openstack-infra03:12
*** gongysh has joined #openstack-infra03:13
*** xarses has quit IRC03:13
*** yanyanhu has quit IRC03:13
*** hongbin has joined #openstack-infra03:15
*** gildub has joined #openstack-infra03:16
*** gongysh has quit IRC03:17
*** yanyanhu has joined #openstack-infra03:18
*** Jeffrey4l_ has quit IRC03:18
*** dimtruck has quit IRC03:20
openstackgerritChangBo Guo(gcb) proposed openstack-infra/project-config: Add functional-py35 non-voting job for Nova  https://review.openstack.org/41010503:23
*** dimtruck has joined #openstack-infra03:23
*** yujunz has quit IRC03:23
*** yujunz has joined #openstack-infra03:24
openstackgerritHongbin Lu proposed openstack-infra/project-config: Fuxi: Add python35 non-voting jobs  https://review.openstack.org/41773403:25
*** lewo has quit IRC03:27
*** yujunz has quit IRC03:27
*** lewo has joined #openstack-infra03:27
*** Guest86619 is now known as melwitt03:27
mordredjianghuaw: can you try again leaving what you've put in clouds.yaml (that looks great) but also uncomment lines 4-6 ?03:28
mordredjianghuaw: I need to write better docs on the difference between these two things03:29
mordredI also need to add one more network setting to clouds.yaml I think ...03:29
mordredbut I'll have to do that tomorrow03:29
*** yujunz has joined #openstack-infra03:30
jianghuawmordred, the error is:03:31
jianghuawOpenStackCloudException: unable to create floating IP for net None: Unable to find an external network in this cloud which makes getting a floating IP impossible03:31
mordredjianghuaw: ok. you may need to keep doing your patch for now - I'm not smart enough this late to figure out a fix - but I may want to ask you for some info tomorrow ...03:34
*** gongysh has joined #openstack-infra03:34
jianghuawmordred, thanks and Sorry for having bothered you so late for you.03:35
*** Ng has quit IRC03:36
*** yujunz has quit IRC03:37
*** Ng has joined #openstack-infra03:37
*** yujunz has joined #openstack-infra03:37
*** askb has quit IRC03:38
*** rbuzatu_ has quit IRC03:41
*** askb has joined #openstack-infra03:42
*** lewo has quit IRC03:44
*** lewo has joined #openstack-infra03:47
*** mariojv has quit IRC03:47
*** salv-orlando has joined #openstack-infra03:48
*** mariojv has joined #openstack-infra03:48
*** EricGonczer_ has joined #openstack-infra03:51
*** salv-orlando has quit IRC03:52
*** yamahata has joined #openstack-infra03:56
*** dimtruck is now known as zz_dimtruck03:58
*** xarses_ has quit IRC04:00
*** links has joined #openstack-infra04:02
*** yujunz is now known as yujunz[away]04:03
openstackgerritHongbin Lu proposed openstack-infra/project-config: Fuxi: Add python35 non-voting jobs  https://review.openstack.org/41773404:04
*** yujunz[away] has quit IRC04:05
*** kjorgensen has joined #openstack-infra04:05
*** kjorgens_ has joined #openstack-infra04:07
*** EricGonczer_ has quit IRC04:08
*** kjorgensen has quit IRC04:10
*** psachin has joined #openstack-infra04:13
*** xarses_ has joined #openstack-infra04:16
*** xgerman has joined #openstack-infra04:17
*** andreww has joined #openstack-infra04:17
*** andreww has quit IRC04:18
*** andreww has joined #openstack-infra04:18
*** rkukura has quit IRC04:21
*** xarses_ has quit IRC04:21
*** sdake has quit IRC04:25
*** dave-mccowan has quit IRC04:28
*** zul has quit IRC04:37
openstackgerritHongbin Lu proposed openstack-infra/project-config: Fuxi: Add python35 dsvm non-voting jobs  https://review.openstack.org/41774104:38
*** kjorgens_ has quit IRC04:39
*** puiterwijk|cld has joined #openstack-infra04:40
*** rbuzatu has joined #openstack-infra04:42
*** yamahata has quit IRC04:45
*** cloudnull has quit IRC04:45
*** rbuzatu has quit IRC04:46
*** cloudnull has joined #openstack-infra04:48
*** salv-orlando has joined #openstack-infra04:49
*** hongbin has quit IRC04:50
*** salv-orlando has quit IRC04:54
*** makowals has quit IRC05:00
*** ihrachys has quit IRC05:00
*** bhavik1 has joined #openstack-infra05:01
*** ihrachys has joined #openstack-infra05:01
*** spligak has quit IRC05:01
*** kjorgensen has joined #openstack-infra05:03
*** bhavik1 has quit IRC05:03
*** ijw has joined #openstack-infra05:07
*** kjorgensen has quit IRC05:07
*** sdake has joined #openstack-infra05:12
*** adriant has quit IRC05:12
*** claudiub has joined #openstack-infra05:13
*** rbuzatu has joined #openstack-infra05:21
openstackgerritHongbin Lu proposed openstack-infra/project-config: Fuxi: Add python35 dsvm non-voting jobs  https://review.openstack.org/41774105:22
*** rbuzatu has quit IRC05:26
*** nadya has joined #openstack-infra05:32
*** sdake has quit IRC05:34
*** ihrachys has quit IRC05:36
*** aviau has quit IRC05:37
*** aviau has joined #openstack-infra05:37
*** yujunz has joined #openstack-infra05:38
*** thorst has joined #openstack-infra05:38
*** adrian_otto has joined #openstack-infra05:40
*** thorst has quit IRC05:43
*** nadya has quit IRC05:49
*** adrian_otto has quit IRC05:53
*** tnovacik has joined #openstack-infra06:02
*** flepied has quit IRC06:03
*** nadya has joined #openstack-infra06:07
*** nadya has quit IRC06:08
*** flepied has joined #openstack-infra06:17
*** itisha has quit IRC06:22
*** rbuzatu has joined #openstack-infra06:22
*** rbuzatu has quit IRC06:27
*** salv-orlando has joined #openstack-infra06:27
*** ediardo has quit IRC06:32
*** ediardo has joined #openstack-infra06:32
*** flepied has quit IRC06:33
*** booleanfour has joined #openstack-infra06:40
*** Guest99613 has quit IRC06:42
AJaegerjhesketh, ianw : Could either of you +A 417679, please? Let's merge it to fix docs publishing to the AFS site.06:48
jheskethAJaeger: I'm out at the moment sorry. Can take a look later tonight if ianw doesn't beat me06:50
*** gildub has quit IRC06:53
AJaegerjhesketh: thanks06:54
*** AJaeger has quit IRC06:54
*** ijw_ has joined #openstack-infra06:55
*** makowals has joined #openstack-infra06:55
*** ijw has quit IRC06:58
*** AJaeger has joined #openstack-infra07:00
*** spligak has joined #openstack-infra07:02
*** pgadiya has joined #openstack-infra07:03
*** tnovacik has quit IRC07:04
*** bhavik1 has joined #openstack-infra07:04
*** florianf has joined #openstack-infra07:05
*** rcernin has joined #openstack-infra07:08
*** tesseract has joined #openstack-infra07:12
*** cody-somerville has joined #openstack-infra07:15
*** cody-somerville has quit IRC07:15
*** cody-somerville has joined #openstack-infra07:15
openstackgerritJulia Varlamova proposed openstack-infra/devstack-gate: Add MULTI_KEYSTONE flag and new role  https://review.openstack.org/39489507:17
*** yujunz is now known as yujunz[away]07:23
*** yujunz[away] is now known as yujunz07:23
*** rbuzatu has joined #openstack-infra07:23
*** yujunz is now known as yujunz[away]07:24
*** yujunz[away] is now known as yujunz07:24
*** yujunz is now known as yujunz[away]07:24
*** yujunz[away] is now known as yujunz07:25
*** nherciu has joined #openstack-infra07:29
*** armax has joined #openstack-infra07:33
*** nherciu_ has joined #openstack-infra07:35
*** markus_z has joined #openstack-infra07:36
AJaegermorning yolanda, could you +2A  417679, please? Let's merge it to fix docs publishing to the AFS site.07:37
yolandahi, good morning07:37
yolandasure07:37
AJaegerthanks07:37
*** rbuzatu has quit IRC07:38
*** nherciu has quit IRC07:38
*** thorst has joined #openstack-infra07:39
*** pcaruana has joined #openstack-infra07:41
openstackgerritMerged openstack-infra/project-config: Fuxi: Add python35 non-voting jobs  https://review.openstack.org/41773407:41
openstackgerritMerged openstack-infra/system-config: Publish to read-write docs volume  https://review.openstack.org/41767907:43
*** hashar has joined #openstack-infra07:43
openstackgerritMerged openstack-infra/project-config: Rename usage of USE_PYTHON3 to DEVSTACK_GATE_USE_PYTHON3  https://review.openstack.org/41745607:43
*** thorst has quit IRC07:43
*** kaisers has left #openstack-infra07:44
openstackgerritYuval Brik proposed openstack-infra/project-config: kuryr-kubernetes: add dragonflow to PROJECTS  https://review.openstack.org/41776807:44
openstackgerritMerged openstack-infra/project-config: glance rally test with python3.5  https://review.openstack.org/41722807:44
openstackgerritMerged openstack-infra/project-config: Enable Python3x DSVM in Octavia deliverables  https://review.openstack.org/41712007:46
openstackgerritMerged openstack-infra/project-config: Enable registry upload on successfull merge  https://review.openstack.org/41746007:47
openstackgerritMerged openstack-infra/project-config: Add ansible-role-sudoers project  https://review.openstack.org/41747207:48
*** erikmwilson has quit IRC07:48
*** dtardivel has joined #openstack-infra07:49
openstackgerritMerged openstack-infra/project-config: Add experimental python3 dsvm job for ironic  https://review.openstack.org/41720207:49
*** erikmwilson has joined #openstack-infra07:50
*** flepied has joined #openstack-infra07:51
openstackgerritMerged openstack-infra/project-config: Make networking-cisco OpenStack version tests voting  https://review.openstack.org/40754607:51
*** andreas_s has joined #openstack-infra07:54
AJaegeryolanda: please also review https://review.openstack.org/#/c/417630 and https://review.openstack.org/#/c/414714/ https://review.openstack.org/#/c/416919/ https://review.openstack.org/#/c/417110/ https://review.openstack.org/#/c/417464/ if you have time.07:55
openstackgerritYuval Brik proposed openstack-infra/project-config: kuryr-kubernetes: add dragonflow to PROJECTS  https://review.openstack.org/41776807:55
*** thorst has joined #openstack-infra08:00
*** salv-orlando has quit IRC08:01
*** wolverineav has joined #openstack-infra08:02
*** Hal has joined #openstack-infra08:03
*** d0ugal_ has quit IRC08:03
*** Hal is now known as Guest3452808:03
*** d0ugal has joined #openstack-infra08:03
*** d0ugal has quit IRC08:03
*** d0ugal has joined #openstack-infra08:03
*** bhavik1 has quit IRC08:03
*** thorst has quit IRC08:04
*** ccamacho has joined #openstack-infra08:05
*** wolverineav has quit IRC08:06
*** cody-somerville has quit IRC08:07
ianwthanks yolanda :)  sorry just checking in quickly08:08
*** nmagnezi has joined #openstack-infra08:09
*** gouthamr has quit IRC08:11
openstackgerritDougal Matthews proposed openstack-infra/tripleo-ci: Use a more specific feed for my blog  https://review.openstack.org/41168708:14
openstackgerritOpenStack Proposal Bot proposed openstack-infra/project-config: Normalize projects.yaml  https://review.openstack.org/41777508:16
*** ralonsoh has joined #openstack-infra08:18
*** openstackgerrit has quit IRC08:18
*** nadya has joined #openstack-infra08:24
*** mptacekx has joined #openstack-infra08:25
*** nadya has quit IRC08:25
*** openstackgerrit has joined #openstack-infra08:28
openstackgerritFedor Zhadaev proposed openstack-infra/project-config: Add fuel-plugin-congress  https://review.openstack.org/41512608:28
*** pgadiya has quit IRC08:28
*** panda|pto is now known as panda08:28
*** ijw_ has quit IRC08:29
*** ijw has joined #openstack-infra08:30
openstackgerritCarlos Camacho proposed openstack-infra/tripleo-ci: Remove the current CI matrix from tripleo-ci  https://review.openstack.org/40969708:33
*** rbuzatu has joined #openstack-infra08:34
openstackgerritMerged openstack-infra/project-config: Normalize projects.yaml  https://review.openstack.org/41777508:35
*** rbuzatu has quit IRC08:38
*** nadya has joined #openstack-infra08:43
*** pgadiya has joined #openstack-infra08:44
*** jpich has joined #openstack-infra08:47
*** dingyichen has quit IRC08:50
*** Mmike has quit IRC08:52
*** Mmike has joined #openstack-infra08:52
*** yujunz is now known as yujunz[away]08:53
*** salv-orlando has joined #openstack-infra08:53
*** yujunz[away] is now known as yujunz08:54
*** yujunz is now known as yujunz[away]08:54
*** yujunz[away] is now known as yujunz08:54
*** flaper87 has joined #openstack-infra08:54
*** flaper87 has quit IRC08:55
*** flaper87 has joined #openstack-infra08:55
*** yujunz is now known as yujunz[away]08:55
*** yujunz[away] is now known as yujunz08:55
*** ekuris has joined #openstack-infra08:55
*** yujunz is now known as yujunz[away]08:55
*** yujunz[away] is now known as yujunz08:56
*** yujunz is now known as yujunz[away]08:57
*** zzzeek has quit IRC09:00
*** zzzeek has joined #openstack-infra09:00
*** yujunz[away] is now known as yujunz09:00
*** yujunz is now known as yujunz[away]09:01
*** umeshksingla has joined #openstack-infra09:01
*** yujunz[away] is now known as yujunz09:01
*** yujunz is now known as yujunz[away]09:01
*** yujunz[away] is now known as yujunz09:04
*** dtantsur|afk is now known as dtantsur09:07
*** markus_z has quit IRC09:09
*** shardy has joined #openstack-infra09:10
*** akscram has quit IRC09:11
*** yujunz is now known as yujunz[away]09:11
*** markus_z has joined #openstack-infra09:12
*** ekuris has quit IRC09:14
*** yujunz[away] has quit IRC09:16
AJaegerjeblair, mordred, infra-root, 417679 was merged 90 mins ago - and I still see access to /afs/openstack.org/docs - see http://logs.openstack.org/d8/d86ff308a1c5cdf104e7390609e6a97a50589201/post/openstack-manuals-releasenotes/c5852d4/_zuul_ansible/ansible_log.txt . Do we need to restart some zuul services for 417679 to be in use?09:26
*** wolverineav has joined #openstack-infra09:26
*** ekuris has joined #openstack-infra09:28
AJaegerI guess, we need to restart zuul-launcher for this change. Could anybody confirm - and do it, please?09:30
*** florianf has quit IRC09:31
*** rbuzatu has joined #openstack-infra09:35
*** Julien-zte has quit IRC09:36
*** rbuzatu_ has joined #openstack-infra09:37
*** saibarspeis has quit IRC09:37
*** derekh has joined #openstack-infra09:38
*** arxcruz has quit IRC09:40
*** rbuzatu has quit IRC09:40
*** john-davidge has joined #openstack-infra09:42
*** roeyc has joined #openstack-infra09:44
*** florianf has joined #openstack-infra09:44
*** roeyc has left #openstack-infra09:45
*** mhickey has joined #openstack-infra09:57
*** e0ne has joined #openstack-infra09:59
*** sambetts|afk is now known as sambetts10:00
*** thorst has joined #openstack-infra10:00
*** dizquierdo has joined #openstack-infra10:01
wznoinskhi infra, is there any timeline for zuul v3?10:06
*** thorst has quit IRC10:07
*** claudiub has quit IRC10:07
*** yanyanhu has quit IRC10:07
*** chem has joined #openstack-infra10:08
*** umeshksingla has quit IRC10:18
*** salv-orl_ has joined #openstack-infra10:22
*** salv-orlando has quit IRC10:25
*** akscram has joined #openstack-infra10:29
*** gongysh has quit IRC10:31
*** bauwser is now known as bauzas10:32
*** arxcruz has joined #openstack-infra10:37
*** ociuhandu has quit IRC10:39
*** liujiong has quit IRC10:39
*** tobasco has quit IRC10:40
*** asettle has joined #openstack-infra10:41
*** dtantsur is now known as dtantsur|brb10:45
*** hyakuhei has quit IRC10:49
*** hyakuhei has joined #openstack-infra10:49
*** hyakuhei has quit IRC10:49
*** hyakuhei has joined #openstack-infra10:49
*** rfolco has joined #openstack-infra10:52
*** ldnunes has joined #openstack-infra10:56
dimsyolanda : you already +2A'ed https://review.openstack.org/#/c/417504/ however that needs https://review.openstack.org/#/c/417630/ :) please take a look when you get a chance10:57
*** claudiub has joined #openstack-infra10:58
*** jordanP has joined #openstack-infra10:58
openstackgerritJulie Pichon proposed openstack-infra/tripleo-ci: Remove references to Liberty (now EOL'd)  https://review.openstack.org/41781410:59
*** nicolasbock has joined #openstack-infra11:04
*** tobasco has joined #openstack-infra11:05
openstackgerritJulie Pichon proposed openstack-infra/tripleo-ci: Remove references to Liberty (now EOL'd)  https://review.openstack.org/41781411:09
*** nicolasbock has quit IRC11:11
*** ldnunes has quit IRC11:11
*** ociuhandu has joined #openstack-infra11:12
*** tuanluong has quit IRC11:12
*** ldnunes has joined #openstack-infra11:13
*** salv-orl_ has quit IRC11:14
*** salv-orlando has joined #openstack-infra11:15
openstackgerritJulie Pichon proposed openstack-infra/tripleo-ci: Use the Newton node management commands  https://review.openstack.org/41698211:15
yolandahi dims , i was out on doctors. Resuming the review queue now11:16
dimsyolanda : much appreciated :) no worries11:17
dimshope all's well yolanda11:17
yolandayep, thanks :)11:17
*** vgridnev- has quit IRC11:18
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci: TripleO containers job with quickstart  https://review.openstack.org/41625911:18
*** vgridnev has joined #openstack-infra11:18
*** nicolasbock has joined #openstack-infra11:23
*** nbloom has joined #openstack-infra11:25
openstackgerritMerged openstack-infra/project-config: Switch order of special and node for heat functional tests  https://review.openstack.org/41763011:26
*** dtantsur|brb is now known as dtantsur11:27
openstackgerritMerged openstack-infra/project-config: python3 functional test for heat  https://review.openstack.org/41750411:28
openstackgerritMerged openstack-infra/project-config: Introduce TripleO multinode (x4) experimental jobs with scenario005  https://review.openstack.org/41458511:29
openstackgerritMerged openstack-infra/project-config: Add grenade testing for Zaqar  https://review.openstack.org/41679011:30
openstackgerritMerged openstack-infra/project-config: releasenotes not part of ceilometermiddleware mitaka  https://review.openstack.org/41674511:31
openstackgerritMerged openstack-infra/project-config: Remove redundant gerrit-git-prep from kolla jobs  https://review.openstack.org/41746411:33
openstackgerritMerged openstack-infra/project-config: Add experimental jobs for tripleoclient ramdisks  https://review.openstack.org/41718311:33
openstackgerritMerged openstack-infra/project-config: Make gate-tripleo-buildimage-overcloud-full-centos-7 voting  https://review.openstack.org/41718411:33
openstackgerritMerged openstack-infra/project-config: oslo.messaging: switch to tox functional test  https://review.openstack.org/41711011:33
openstackgerritMerged openstack-infra/project-config: Disable novnc service for dsvm-heat-apache job  https://review.openstack.org/41691911:33
AJaegerwznoinsk: there's a #zuul channel and a weekly meeting for zuul. You might need to wait for some US folks to wake up for an answer...11:35
*** ramishra has quit IRC11:40
*** sdake has joined #openstack-infra11:41
openstackgerritMerged openstack-infra/project-config: kuryr-kubernetes: add dragonflow to PROJECTS  https://review.openstack.org/41776811:42
*** ramishra has joined #openstack-infra11:43
*** sdague has joined #openstack-infra11:59
wznoinskAJaeger, thanks12:00
*** ekuris has quit IRC12:04
*** rbuzatu has joined #openstack-infra12:07
*** rtheis has joined #openstack-infra12:08
*** rbuzatu_ has quit IRC12:10
*** drifterza has joined #openstack-infra12:11
*** hashar is now known as hasharLunch12:12
*** ekuris has joined #openstack-infra12:17
*** dkehn has quit IRC12:24
*** thorst has joined #openstack-infra12:24
*** thorst has quit IRC12:24
*** ramishra has quit IRC12:24
*** thorst has joined #openstack-infra12:25
*** nadya has quit IRC12:25
*** nbloom has quit IRC12:26
*** ramishra has joined #openstack-infra12:27
*** dkehn has joined #openstack-infra12:31
*** zul has joined #openstack-infra12:31
openstackgerritDavanum Srinivas (dims) proposed openstack-infra/project-config: Switch off glance->swift in rally+py35 tests  https://review.openstack.org/41785712:32
*** ramishra has quit IRC12:34
*** _Cyclone_ has joined #openstack-infra12:34
*** Julien-zte has joined #openstack-infra12:34
*** _Cyclone_ has quit IRC12:34
*** ramishra has joined #openstack-infra12:36
*** sreeram_v has joined #openstack-infra12:37
*** gongysh has joined #openstack-infra12:38
*** e0ne has quit IRC12:39
*** zul has quit IRC12:39
*** zul has joined #openstack-infra12:40
*** e0ne has joined #openstack-infra12:40
*** e0ne has quit IRC12:41
*** lucasagomes is now known as lucas-hungry12:41
*** e0ne has joined #openstack-infra12:41
*** zul has quit IRC12:41
*** e0ne has quit IRC12:41
*** zul has joined #openstack-infra12:42
*** e0ne has joined #openstack-infra12:42
*** e0ne has quit IRC12:42
*** e0ne has joined #openstack-infra12:43
*** e0ne has quit IRC12:43
*** e0ne has joined #openstack-infra12:43
*** e0ne has quit IRC12:44
*** rhallisey has joined #openstack-infra12:47
*** nadya has joined #openstack-infra12:48
dimsAJaeger : yolanda : i switched off swift explicitly in https://review.openstack.org/#/c/417228/11/jenkins/jobs/projects.yaml, but swift *STILL* shows up in the job http://logs.openstack.org/18/417818/1/check/gate-rally-dsvm-py35-glance-nv/abe069b/logs/localrc.txt.gz .12:49
dimsAJaeger : yolanda : so here's a review to explicitly disable glance->swift via the config option https://review.openstack.org/#/c/417857/12:50
*** wolverineav has quit IRC12:52
*** gongysh has quit IRC12:53
*** timello has quit IRC12:54
*** flepied has quit IRC12:56
openstackgerritsebastian marcet proposed openstack-infra/openstackid-resources: Fix on team json format  https://review.openstack.org/41786412:57
openstackgerritMerged openstack-infra/openstackid-resources: Fix on team json format  https://review.openstack.org/41786412:59
*** ramishra has quit IRC12:59
*** dprince has joined #openstack-infra12:59
*** ramishra has joined #openstack-infra13:01
*** ekuris has quit IRC13:02
*** timello has joined #openstack-infra13:05
*** ggnel_t has joined #openstack-infra13:06
*** pgadiya has quit IRC13:07
*** trown|outtypewww is now known as trown13:08
*** weshay_afk is now known as weshay13:09
*** tensionlanguage has joined #openstack-infra13:13
*** flepied has joined #openstack-infra13:13
*** umeshksingla has joined #openstack-infra13:14
*** booleanfour has quit IRC13:15
*** ekuris has joined #openstack-infra13:15
openstackgerritMerged openstack-infra/openstackid-resources: Update readme file  https://review.openstack.org/41744913:20
*** salv-orlando has quit IRC13:21
*** rlandy has joined #openstack-infra13:24
*** kgiusti has joined #openstack-infra13:26
*** mdrabe_ has joined #openstack-infra13:27
openstackgerritsebastian marcet proposed openstack-infra/openstackid-resources: Fix on team json format  https://review.openstack.org/41787513:27
*** pradk has joined #openstack-infra13:28
openstackgerritMerged openstack-infra/openstackid-resources: Fix on team json format  https://review.openstack.org/41787513:28
*** benj_ has quit IRC13:30
*** gouthamr has joined #openstack-infra13:31
*** salv-orlando has joined #openstack-infra13:32
*** benj_ has joined #openstack-infra13:34
*** fguillot has joined #openstack-infra13:34
*** fguillot has quit IRC13:34
*** fguillot has joined #openstack-infra13:34
*** umeshksingla has quit IRC13:34
*** asettle has quit IRC13:35
*** asettle has joined #openstack-infra13:36
*** jamielennox is now known as jamielennox|away13:39
openstackgerritJohn Trowbridge proposed openstack-infra/tripleo-ci: WIP: Add OOOQ multinode test support (follow-up)  https://review.openstack.org/41749413:40
*** ramishra has quit IRC13:43
*** ramishra has joined #openstack-infra13:44
*** srobert has joined #openstack-infra13:45
*** thorst has quit IRC13:45
*** thorst has joined #openstack-infra13:46
*** nbloom has joined #openstack-infra13:46
*** gordc has joined #openstack-infra13:46
*** ihrachys has joined #openstack-infra13:47
nbloomHi everyone :) can someone take a look at this log and perhaps tell me why the job fails? http://logs.openstack.org/74/417674/2/check/gate-vitrage-dashboard-docs-ubuntu-xenial/67f1c96/console.html13:48
*** jcoufal has joined #openstack-infra13:48
*** rafaels has joined #openstack-infra13:49
AJaegernbloom: broken docutils 2.13.1 or so possible - let me grab you a link...13:49
*** jheroux has joined #openstack-infra13:50
AJaegernbloom: try http://lists.openstack.org/pipermail/openstack-dev/2016-December/108742.html13:50
*** thorst has quit IRC13:50
*** lucas-hungry is now known as lucasagomes13:53
*** Goneri has joined #openstack-infra13:53
*** xyang1 has joined #openstack-infra13:55
*** ramishra has quit IRC13:55
*** ramishra has joined #openstack-infra13:55
*** ldnunes has quit IRC13:56
*** ihrachys has quit IRC13:57
*** ramishra has quit IRC13:57
*** ihrachys has joined #openstack-infra13:57
nbloomAJaeger: Thanks :)13:57
*** wolverineav has joined #openstack-infra13:58
*** srobert_ has joined #openstack-infra14:00
*** drifterza has quit IRC14:00
*** knikolla has joined #openstack-infra14:00
*** srobert_ has quit IRC14:01
*** srobert has quit IRC14:01
*** srobert has joined #openstack-infra14:01
*** ramishra has joined #openstack-infra14:02
*** nbloom has quit IRC14:02
*** wolverin_ has joined #openstack-infra14:03
*** thorst has joined #openstack-infra14:05
*** wolverineav has quit IRC14:06
*** nadya has quit IRC14:06
*** lindycoder has joined #openstack-infra14:07
*** Guest3684 is now known as dasm14:08
*** ldnunes has joined #openstack-infra14:09
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: Revert "temprevert neutron "Don't create HA resources until needed" patch"  https://review.openstack.org/41788814:09
*** krtaylor has joined #openstack-infra14:11
*** hongbin has joined #openstack-infra14:12
*** nadya has joined #openstack-infra14:14
*** baoli has joined #openstack-infra14:15
*** eharney has quit IRC14:15
*** umeshksingla has joined #openstack-infra14:15
*** aleph1 is now known as agarner14:18
*** tlian has joined #openstack-infra14:18
*** lbragstad_ is now known as lbragstad14:19
*** john-davidge has quit IRC14:19
*** john-dav_ has joined #openstack-infra14:19
*** sreeram_v has quit IRC14:23
*** jaosorior has joined #openstack-infra14:23
openstackgerritIhar Hrachyshka proposed openstack-infra/irc-meetings: Added Artur as a co-chair for the neutron upgrades meeting  https://review.openstack.org/41789714:24
*** sreeram_v has joined #openstack-infra14:24
*** mdrabe_ has quit IRC14:27
*** mdrabe_ has joined #openstack-infra14:27
*** baoli has quit IRC14:28
*** sreeram_v has quit IRC14:29
*** superdan is now known as dansmith14:30
*** salv-orlando has quit IRC14:31
*** jaosorior has quit IRC14:31
*** john-dav_ has quit IRC14:31
*** john-davidge has joined #openstack-infra14:32
*** jordanP has quit IRC14:32
openstackgerritBen Kero proposed openstack-infra/tripleo-ci: WIP: Add OOOQ multinode test support  https://review.openstack.org/41611014:33
*** ramishra has quit IRC14:33
*** dave-mccowan has joined #openstack-infra14:33
*** matt-borland has joined #openstack-infra14:35
*** jordanP has joined #openstack-infra14:35
*** jaosorior has joined #openstack-infra14:35
*** Goneri has quit IRC14:36
*** asselin has joined #openstack-infra14:36
*** ramishra has joined #openstack-infra14:37
*** eharney has joined #openstack-infra14:37
*** trown is now known as trown|afk14:38
*** asselin has quit IRC14:39
*** rbrndt has joined #openstack-infra14:39
fungiAJaeger: i can get launcher restarts going if nobody else is working on that yet (i didn't see anyone here saying they were)14:40
*** psachin has quit IRC14:41
*** baoli has joined #openstack-infra14:41
*** baoli has quit IRC14:41
*** rbrndt has quit IRC14:41
AJaegerfungi: thanks! Nobody else said anything...14:42
fungiwznoinsk: the short answer is "when it's ready" (hopefully in a few more months?). there's also a weekly zuul dev meeting mondays (today!) at 22:00 utc in #openstack-meeting-alt if you're interested14:42
*** tonytan4ever has joined #openstack-infra14:42
fungiit's coming along quite quickly14:42
AJaegerwould be good to give this priority since I asked release team to not release anythign - otherwise content is missing in AFS14:43
*** umeshksingla has quit IRC14:43
*** eharney has quit IRC14:43
openstackgerritDoug Hellmann proposed openstack-infra/release-tools: remove aclmanager.py  https://review.openstack.org/41790014:45
openstackgerritDoug Hellmann proposed openstack-infra/project-config: set up ACLs for ocata stable branches before release candidate phase  https://review.openstack.org/41790114:45
*** rafaels has quit IRC14:46
fungihas infracloud-chocolate been disabled in nodepool yet?14:47
AJaegerfungi: I doubt it14:47
fungii see tons of gate failures being caused by it14:47
*** ramishra has quit IRC14:47
*** asselin has joined #openstack-infra14:48
*** Goneri has joined #openstack-infra14:48
*** ramishra has joined #openstack-infra14:49
*** links has quit IRC14:50
*** panda is now known as panda|brb14:51
wznoinskfungi, thanks for coming back on this14:51
*** baoli has joined #openstack-infra14:51
AJaegerfungi, let me do a patch for chocolate disabling14:51
fungiAJaeger: thanks, i've manually disabled first and am deleting remaining nodes there14:52
*** tonytan4ever has quit IRC14:52
fungii'll undo my emergency puppet block once the change lands14:52
*** tonytan4ever has joined #openstack-infra14:52
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Disable chocolate until mirror is up again  https://review.openstack.org/41790414:53
AJaegerfungi, ^14:53
AJaegerfungi, have started with restarting the zuul-launchers already?14:54
fungiAJaeger: yep, i'm up to zl0314:54
jianghuawmordred, is it the proper time for you to have a chat about the float IP issue?14:54
*** eharney has joined #openstack-infra14:56
fungiwe're down to 0 nodepool nodes in chocolate now14:57
*** cody-somerville has joined #openstack-infra14:57
*** cody-somerville has quit IRC14:57
*** cody-somerville has joined #openstack-infra14:57
fungiso that should stop the bleeding there while we find time to investigate14:57
AJaegerfungi, thanks14:59
fungiAJaeger: restarts are done now. i see a openstack-ansible-openstack_openrc-docs-ubuntu-xenial job queued for 41d71da in post which should hopefully dtrt now?15:00
AJaegerlet's see...15:00
* AJaeger watches15:00
*** zz_dimtruck is now known as dimtruck15:01
*** yamahata has joined #openstack-infra15:02
*** baoli has quit IRC15:02
*** lindycoder has quit IRC15:03
AJaeger... takes some time until we have a node for that docs job ;(15:03
openstackgerritJuan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: Add "undecloud" or extra node's details to OVB's nodes' json  https://review.openstack.org/41354815:04
openstackgerritJuan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: TLS everywhere job introduction  https://review.openstack.org/39173815:04
openstackgerritJuan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: Add temporary key to OVB stack if undercloud is created  https://review.openstack.org/41347715:04
*** nmagnezi has quit IRC15:04
*** csomerville has joined #openstack-infra15:05
*** panda|brb is now known as panda15:06
fungi#status log All zuul-launcher services have been emergency restarted so that zuul.conf change https://review.openstack.org/417679 will take effect.15:08
openstackstatusfungi: finished logging15:08
openstackgerritBen Kero proposed openstack-infra/tripleo-ci: WIP: Add OOOQ multinode test support  https://review.openstack.org/41611015:08
*** cody-somerville has quit IRC15:08
cloudnullmornings15:08
fungi#status log Nodepool use of Infra-cloud's chocolate region has been disabled with https://review.openstack.org/417904 while nova host issues impacting its mirror instance are investigated.15:09
openstackstatusfungi: finished logging15:09
openstackgerritMerged openstack-infra/project-config: Disable chocolate until mirror is up again  https://review.openstack.org/41790415:10
fungiAJaeger: it seems to be running now15:10
fungicaught up pretty quickly15:10
*** mmandel has joined #openstack-infra15:11
*** ramishra has quit IRC15:12
AJaegerfungi, releasenotes publishing worked! So, we should be green again. LEt's wait one or two more jobs before giving the all clear15:12
*** EricGonczer_ has joined #openstack-infra15:13
fungihttp://logs.openstack.org/41/41d71da4d991e1ad3b9a40042993fe317a60be6c/post/openstack-ansible-openstack_openrc-docs-ubuntu-xenial/2765a6f/_zuul_ansible/post_playbook shows target: /afs/.openstack.org/docs/developer/openstack-ansible-openstack_openrc now too15:13
*** mptacekx has quit IRC15:14
AJaegerfungi, could you review https://review.openstack.org/417701 and https://review.openstack.org/415901 for infra-manual, please?15:14
AJaegerfungi: good!15:14
fungisure15:14
*** Goneri has quit IRC15:16
*** baoli has joined #openstack-infra15:16
*** jianghuaw-m has joined #openstack-infra15:16
*** umeshksingla has joined #openstack-infra15:18
openstackgerritMonty Taylor proposed openstack-infra/shade: DNM HTTP Tracing to track down networking gate issue  https://review.openstack.org/41792115:18
*** ramishra has joined #openstack-infra15:19
pabelangerAJaeger: fungi: I'm surprised we don't see the mirror failures in our configure_mirror.sh script, I'll have to poke into why15:20
AJaegergood idea, pabelanger - thanks.15:21
*** jianghuaw-m has left #openstack-infra15:21
AJaegerpabelanger: could you push 417701 and 415901 on your review queue as well, please?15:21
AJaegerthanks, fungi and pabelanger !15:22
*** baoli has quit IRC15:22
openstackgerritMonty Taylor proposed openstack-infra/shade: DNM Testing a devstack revert patch  https://review.openstack.org/41792515:22
fungii'll go ahead and take nodepool.o.o back out of the disabled list in the emergency file now15:22
fungisince the config change merged15:23
*** trown|afk is now known as trown15:23
*** jianghuaw-m has joined #openstack-infra15:23
*** mdrabe_ is now known as mdrabe15:23
*** markvoelker has joined #openstack-infra15:24
*** ramishra has quit IRC15:25
openstackgerritMerged openstack-infra/infra-manual: Improve retiring documentation  https://review.openstack.org/41770115:25
openstackgerritMerged openstack-infra/infra-manual: add note to run git review -s before deleting repo  https://review.openstack.org/41590115:25
fungimordred: not running for a board seat this time?15:25
*** ramishra has joined #openstack-infra15:26
*** hongbin has quit IRC15:26
mordredjianghuaw: normally yes - but I've got a gate failure issue I'm tracking down . in the mean time, could you run : http://paste.openstack.org/show/594297/ and paste the output for me?15:27
mordredfungi: nope. decided I'd let someone else have a turn15:27
evrardjphey15:27
fungimordred: cool, just making sure there wasn't a problem with my ballot ;)15:28
evrardjpdid you see issues with infracloud-chocolate? like apt packages missing?15:28
odyssey4mehappy new year to all15:28
odyssey4meseeing failures for the xenial mirror in infracloud: http://logs.openstack.org/39/417039/3/check/gate-openstack-ansible-os_keystone-linters-ubuntu-xenial/0d28c98/console.html#_2017-01-09_13_59_05_21880415:28
evrardjpindeed happy new year :)15:28
*** pradk- has joined #openstack-infra15:28
AJaegerevrardjp: we just shut down chocolate because of problems with the mirrors15:29
fungievrardjp: the mirror there is down. we've taken it out of rotation for the moment https://wiki.openstack.org/wiki/Infrastructure_Status15:29
evrardjpok cool15:29
evrardjpthanks15:29
fungievrardjp: you shouldn't see any more failures like that for now15:29
odyssey4meawesome, thanks15:29
evrardjp\o/15:29
*** matt-borland has quit IRC15:29
*** tlian has quit IRC15:29
*** knikolla has quit IRC15:29
*** asettle has quit IRC15:29
*** bnemec has quit IRC15:29
*** matt-borland has joined #openstack-infra15:29
*** tlian has joined #openstack-infra15:30
*** knikolla has joined #openstack-infra15:30
fungihopefully an infra-root with more intimate knowledge of infra-cloud host management will have time to look into what happened to those nova servers15:30
*** asettle has joined #openstack-infra15:30
*** Goneri has joined #openstack-infra15:31
*** ramishra has quit IRC15:31
*** jianghuaw-m has quit IRC15:32
pabelangerfungi: yes, looking now15:32
fungiawesome, thanks pabelanger!15:32
pabelangerfungi: do you know which compute node fell over?15:32
fungiat a minimum, the one where the mirror instance was running15:32
fungii'll see if anyone said in scrollback15:32
*** rafaels has joined #openstack-infra15:33
*** bnemec has joined #openstack-infra15:33
pabelangerthat is okay, I can find out15:33
fungipabelanger: jhesketh surmised that all of chocolate may be offline since he couldn't find a separate bifrost server for it and the one for vanilla doesn't have any of chocolate in its inventory (but was unsure he was checking correctly)15:34
pabelangerIIRC, we only have a single bifrost server now15:35
pabelangerfor both regions15:35
*** Jeffrey4l_ has joined #openstack-infra15:35
*** ociuhandu has quit IRC15:35
fungicould be he was looking in the wrong inventory15:36
*** marst has quit IRC15:36
*** drifterza has joined #openstack-infra15:36
*** mmandel has quit IRC15:39
AJaegerfungi, before giving the release team an all-clear, I wanted to see one openstack-manuals change merge - but that takes too long now and I'll soon be offline for two hours. Since the first tests were fine, I'll give them an all-clear now.15:40
*** jistr is now known as jistr|biab15:40
*** sdake_ has joined #openstack-infra15:40
pabelangerlooks like 11 compute nodes are not responding15:41
openstackgerritMonty Taylor proposed openstack-infra/shade: Log request ids when debug logging is enabled  https://review.openstack.org/41793115:42
*** ramishra has joined #openstack-infra15:43
*** rbrndt has joined #openstack-infra15:43
*** sdake has quit IRC15:43
*** jistr|biab is now known as jistr15:43
*** d0ugal has quit IRC15:44
*** john-davidge has quit IRC15:44
*** d0ugal has joined #openstack-infra15:44
*** d0ugal has quit IRC15:44
*** d0ugal has joined #openstack-infra15:44
*** d0ugal has quit IRC15:44
*** markvoelker has quit IRC15:44
*** d0ugal has joined #openstack-infra15:45
*** d0ugal has quit IRC15:45
*** d0ugal has joined #openstack-infra15:45
*** john-davidge has joined #openstack-infra15:45
*** marst has joined #openstack-infra15:45
*** jeckersb is now known as jeckersb_gone15:45
*** baoli has joined #openstack-infra15:45
fungiAJaeger: sounds good. thanks for checking so thoroughly15:46
fungipabelanger: maybe they had a power incident in one rack or something15:46
*** jeckersb_gone is now known as jeckersb15:47
*** john-davidge has quit IRC15:47
*** john-dav_ has joined #openstack-infra15:47
fungiwendar is probably our only contact with access to jira at this point, and she doesn't seem to be around at the moment15:47
*** jaosorior has quit IRC15:47
*** ramishra has quit IRC15:48
*** hasharLunch is now known as hashar15:48
*** baoli has quit IRC15:48
*** ramishra has joined #openstack-infra15:49
mugsiefungi: you looking for HPE Jira access ?15:50
*** ociuhandu has joined #openstack-infra15:51
fungimugsie: potentially, if pabelanger doesn't have much luck figuring out why these servers are down and trying to remotely reboot them15:51
*** john-dav_ has quit IRC15:51
AJaegerfungi, could you discuss with ianw or pabelanger mirroring docs.o.o again, please? I guess once ianw is awake, I'll be asleep ;)15:51
fungiAJaeger: yep. ianw was out all last week, pabelanger said he'd take a look sometime today, so between the two of them hopefully it can get tackled15:52
pabelangerAJaeger: fungi: I have time to talk about that too15:52
*** baoli has joined #openstack-infra15:53
mordredinfra-root: shade's gate (and non-voting nodepool devstack jobs) are broken. I've tosed up a couple of debug related patches and have added a hold job in nodepool to grab a failed node15:53
AJaegerpabelanger: great! ianw documented at https://etherpad.openstack.org/p/afs-docs - see line 41 - what he did15:53
openstackgerritBen Nemec proposed openstack-infra/tripleo-ci: Remove neutron temprevert  https://review.openstack.org/41793715:53
mordredjianghuaw: also - are you running the latest version of shade with your nodepool?15:54
AJaegerpabelanger: so, you need to rerun the lftp mirror command with proper parameters to update all the content15:54
pabelangerAJaeger: okay, let me do that15:54
*** rbuzatu has quit IRC15:55
*** deadnull has joined #openstack-infra15:55
*** rbuzatu has joined #openstack-infra15:55
*** ramishra has quit IRC15:57
AJaegerpabelanger: use mirror -e -n (the -n means only transfer newer files)15:57
AJaegerpabelanger: so, we avoid mirroring old content15:57
AJaegerpabelanger: still it will take several hours15:57
openstackgerritSharat Sharma proposed openstack-infra/project-config: Add translation-jobs to mistral  https://review.openstack.org/41794215:57
AJaegerand add -P615:57
dimsAJaeger : one job ran into a problem - Fix : https://review.openstack.org/#/c/417857/ (Details :  i switched off swift explicitly in https://review.openstack.org/#/c/417228/11/jenkins/jobs/projects.yaml, but swift *STILL* shows up in the job http://logs.openstack.org/18/417818/1/check/gate-rally-dsvm-py35-glance-nv/abe069b/logs/localrc.txt.gz)16:00
*** nadya has quit IRC16:00
*** beagles is now known as beagles_brb16:01
*** ekuris has quit IRC16:01
AJaegerdims, I'll be offline for the next two+ hours. Best to ask fungi and pabelanger to review for now, please ^16:02
* AJaeger will be back later16:02
*** ramishra has joined #openstack-infra16:04
*** jordanP has quit IRC16:05
*** Swanson has quit IRC16:05
*** jordanP has joined #openstack-infra16:05
kfox1111could use a second +2.  very quick review. https://review.openstack.org/#/c/417635/16:07
* mordred would like to express how much nicer dealing with ipv6 hosts is since he got the he.net set up in his router16:08
*** ramishra has quit IRC16:09
*** mtanino has joined #openstack-infra16:09
*** florianf has quit IRC16:10
*** baoli has quit IRC16:10
pabelangerAJaeger: fungi: lftp mirror command running now16:10
*** ramishra has joined #openstack-infra16:11
mugsiemordred: I am still finding bits of the internet that are semi broken on IPv6 :/16:14
*** andreas_s has quit IRC16:14
mordredmugsie: I had to set up a firewall reject rule for ipv6 for netflix, because even though they have ipv6 addresses, they don't let you stream over ipv6 because they 'can't tell what location you're coming from'16:15
*** rafaels has quit IRC16:15
mugsie-_-16:15
mordredoh - sorry - because I'm coming from a tunnelbroker16:15
mordredif I had ipv6 direct from my isp they would let me stream through that16:15
mugsieso am i, but my provider is here in ireland16:15
mugsieso it seems to work16:15
fungiinterestingly, i've had no trouble netflixing from my roku on a v6-enabled network, but maybe the netflix app there is smart enough to reroute over v4 or provides location info some other way16:16
mordredI think their larger concern is non-us people using proxies to access us-only content16:16
clarkbpabelanger: fungi: jhesketh mentioned that he couldn't tell if any hosts were registered with the baremetal00 server in vanilla yesterday. pabelanger is baremetal00 in vanilla the ironic server for both clouds? and if so is that the correct place to query for server enrollment etc?16:16
mordredof course, my broker's pop is in dallas16:16
mugsieI can actually get better speed over IPv6 depending on the far end16:16
mugsiemine is 2 hops from my ISP, which is handy16:16
openstackgerritMerged openstack-infra/tripleo-ci: Fix REPO_PREFIX handling of ceph repos  https://review.openstack.org/40981416:17
fungii route through he's iad pop, which both my broadband provider and rackspace have peering with (and most of my virtual servers are in that rackspace pop as well)16:18
fungiand it's geographically pretty near here16:18
*** lVathan[m] has quit IRC16:18
*** M-docaedo_vector has quit IRC16:19
pabelangerclarkb: fungi: jhesketh: just checked my notes, yes we only have a single bifrost node for both vanilla and chocolate: ssh barmetal00; source /opt/stack/bifrost/env-vars; ironic node-list16:19
pabelangerclarkb: I am not sure the current status of server enrollments, I see a mixture of things that are offline and deploy failed16:20
*** rafaels has joined #openstack-infra16:21
fungircarrillocruz: ^ if you happen to be around, were you tracking any of that?16:21
*** Jeffrey4l_ has quit IRC16:22
*** baoli has joined #openstack-infra16:22
*** adrian_otto has joined #openstack-infra16:22
*** flepied has quit IRC16:22
*** hongbin has joined #openstack-infra16:23
*** mriedem has joined #openstack-infra16:23
openstackgerritClark Boylan proposed openstack-infra/system-config: More bifrost details for infracloud  https://review.openstack.org/41795316:23
clarkbpabelanger: fungi ^16:24
fungiooh, thanks!16:24
*** Jeffrey4l_ has joined #openstack-infra16:24
pabelanger+216:25
*** florianf has joined #openstack-infra16:25
pabelangerI'm trying to see which server our mirror lives on, and compare to ironic16:25
clarkbpabelanger: jhesketh was able to determine that it was on one of the hypervisors that was done16:25
clarkbs/done/down/16:25
clarkbpabelanger: nova show as admin on the mirror instance should tell you what hypervisor it is on16:26
*** ramishra has quit IRC16:28
*** jascott1_ has joined #openstack-infra16:28
pabelangercompute017.chocolate.ic.openstack.org is our server16:29
pabelangercurrently power off16:29
pabelangerlet me try to turn it back on16:29
*** jascott1_ has quit IRC16:29
*** kjorgensen has joined #openstack-infra16:29
dimsfungi : pabelanger : easy one when you get a chance please. https://review.openstack.org/#/c/417857/16:30
*** deadnull has quit IRC16:31
*** Diziet_ is now known as Diziet16:31
*** panda is now known as panda|bbl16:32
openstackgerritJohn L. Villalovos proposed openstack-infra/project-config: Run tempest smoke tests for ironic-.*-multinode job  https://review.openstack.org/41795916:32
*** markvoelker has joined #openstack-infra16:32
fungidims: does that need a todo to remind you to unwind it later when swift gets py3k support working? or are you tracking it on your own?16:34
*** kjorgensen has quit IRC16:34
clarkbdims: fungi also that echo for $GLANCE_API_CONF looks really funny16:34
*** Qiming has quit IRC16:34
clarkb[[post-config|/$GLANCE_API_CONF]] is what you get out of it, is that / desired?16:34
dimsfungi : i will be reviewing all USE_PYTHON3 in project-config and devstack* periodically. tracking stuff on etherpad as well https://etherpad.openstack.org/p/support-python3.5-functional-tests16:35
fungiyeah, i was wondering about that. is it supposed to be a bash eval, or is [[]] also some different level of heading for a special ini subformat?16:35
*** Jeffrey4l__ has joined #openstack-infra16:35
openstackgerritJulie Pichon proposed openstack-infra/tripleo-ci: Remove references to Liberty (now EOL'd)  https://review.openstack.org/41781416:36
*** Guest59792 is now known as redrobot16:36
dimsclarkb : fungi : i used the pattern already used here http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/neutron.yaml#n14916:37
dimsi guess we need the $ to pass through and end up on the file on disk16:37
fungilooks like a configuration heading level: http://git.openstack.org/cgit/openstack-dev/devstack/tree/doc/source/configuration.rst#n5116:37
fungi"meta-sections"16:38
clarkbdims: ya I think it is being evaluated later within the context of devstack itself when reading that file16:38
*** pcaruana has quit IRC16:38
clarkbthe / is what has me confused16:38
clarkboh but thats documented too /me reads more of fungi's link16:38
dimsclarkb : there's only job where the if condition is True, and that's what i want am working on16:38
*** rcernin has quit IRC16:38
fungioh, got it. wondering if it should be |\$ instead of |/\$16:38
*** Jeffrey4l_ has quit IRC16:39
dimsso if i don't get what i need, will quickly propose another rev :)16:39
fungibut the [[post-config|/$Q_PLUGIN_CONF_FILE]] example on line 76 of that document implies that's a special syntax16:39
clarkbdims: fungi at least according to the devstack doc Q_PLUGIN_CONF_FILE is unique and needs the / prefix16:39
clarkbits possible that glance also needs it too16:39
clarkbin any case //some/path/here is valid so meh16:40
dimsclarkb : hang on then16:40
*** Qiming has joined #openstack-infra16:40
dimslet me remove the /16:40
clarkbdims: well it won't hurt16:40
clarkbit may just be unnecessary extra16:40
clarkbdims: cd //home/dims :)16:40
*** flepied has joined #openstack-infra16:40
fungii guess the end result is that it ensures the path is absolute even if teh path in the raw variable is missing a leading /16:41
*** jascott1_ has joined #openstack-infra16:41
clarkbfungi: ya16:41
openstackgerritDavanum Srinivas (dims) proposed openstack-infra/project-config: Switch off glance->swift in rally+py35 tests  https://review.openstack.org/41785716:41
*** jaosorior has joined #openstack-infra16:41
*** jaosorior has quit IRC16:42
*** mhickey has quit IRC16:42
*** jaosorior has joined #openstack-infra16:43
dimsthanks clarkb, fungi ^16:43
openstackgerritMerged openstack-infra/project-config: Add kolla-kubernetes for helm operator  https://review.openstack.org/41763516:44
*** vhosakot has joined #openstack-infra16:46
*** M-docaedo_vector has joined #openstack-infra16:46
clarkbdims: related, is https://review.openstack.org/#/c/417720/1 ready to merge? the one project-config change that switched var names that I know of did merge16:47
rcarrillocruzhi fungi16:48
* rcarrillocruz reads scrollback16:48
*** nadya has joined #openstack-infra16:48
clarkbdims: and if you have a link to a recent USE_PYTHON3=True test run I'd like to double check that the reproduce.sh script is generated properly for you if you have link to those logs16:48
* clarkb checks etherpad16:48
*** Guest34528 has quit IRC16:48
rcarrillocruzclarkb, fungi : yeah, baremetal00 serves as the bifrost server for both vanilla and chocolate16:49
*** kukacz has joined #openstack-infra16:49
dimsclarkb : y, https://review.openstack.org/#/c/417720 can merge now.16:50
*** dizquierdo has quit IRC16:50
dimsclarkb : i can see DEVSTACK_GATE_USE_PYTHON3 in  http://logs.openstack.org/33/417533/2/check/gate-keystone-dsvm-py35-functional-v3-only-ubuntu-xenial-nv/f193103/logs/reproduce.sh16:50
openstackgerritBen Kero proposed openstack-infra/tripleo-ci: WIP: Add OOOQ multinode test support  https://review.openstack.org/41611016:50
clarkbdims: perfect thanks (for both items)16:51
*** Jeffrey4l__ has quit IRC16:51
*** tonytan4ever has quit IRC16:52
clarkbsdague: you had asked for review on https://review.openstack.org/#/c/334504/ late last week. I left a comment there (there is a problem where screen logs aren't captured properly due to a path getting misset)16:52
*** tonytan4ever has joined #openstack-infra16:52
*** beagles_brb is now known as beagles16:53
sdagueclarkb: ok, just that one fix needed?16:54
openstackgerritBen Kero proposed openstack-infra/tripleo-ci: WIP: Add OOOQ multinode test support  https://review.openstack.org/41611016:54
openstackgerritSean Dague proposed openstack-infra/devstack-gate: Add save_* functions for clearer saving of artifacts  https://review.openstack.org/33450416:54
clarkbsdague: yes I think that is it16:54
*** drifterza has quit IRC16:55
*** makowals has quit IRC16:56
openstackgerritKen Giusti proposed openstack-infra/project-config: Deprecate the amqp1 dsvm-functional test on centos 7  https://review.openstack.org/41797116:56
sdagueclarkb: ok, lets see if we can get that bit sorted, then I'll work on the local.conf patch on top16:56
*** ijw_ has joined #openstack-infra16:56
*** mriedem has quit IRC16:56
*** makowals_ has joined #openstack-infra16:56
*** david-lyle has joined #openstack-infra16:57
*** nadya has quit IRC16:57
openstackgerritMerged openstack-infra/system-config: More bifrost details for infracloud  https://review.openstack.org/41795316:57
openstackgerritMerged openstack-infra/project-config: Switch off glance->swift in rally+py35 tests  https://review.openstack.org/41785716:58
*** mrtenio has joined #openstack-infra16:59
*** tesseract has quit IRC16:59
*** ijw has quit IRC16:59
clarkbdims: sdague and then sort of related to the python3 work is https://review.openstack.org/#/c/417555/ which should help us continue to ensure that reproduce.sh doesn't need python2 installed if python3 is available (though I think we need to mess with the actual invocation/shebang line of test-matrix.py for that to actually happen)16:59
clarkblooks like dims already +1'd16:59
dims++ clarkb17:00
clarkboh but the depends on is what needs review17:00
*** markus_z has quit IRC17:01
*** kjorgensen has joined #openstack-infra17:04
*** kjorgens_ has joined #openstack-infra17:04
pabelangermordred: q: does novaclient support clouds.yaml?17:05
fungipabelanger: i believe it does not, but openstackclient will17:05
fungipabelanger: i don't see it depending on os-client-config in http://git.openstack.org/cgit/openstack/python-novaclient/tree/requirements.txt17:06
rcarrillocruzi believe not17:07
rcarrillocruzneutronclient does17:07
rcarrillocruzbut memory tells me nova didn't17:07
rcarrillocruz(happy new year everyone btw, been off for quite a while!)17:07
*** kjorgensen has quit IRC17:08
dimsclarkb : +1 on https://review.openstack.org/#/c/417554/17:09
fungircarrillocruz: pabelanger: indeed, that looks likely based on http://git.openstack.org/cgit/openstack/python-neutronclient/tree/requirements.txt#n1317:11
fungircarrillocruz: happy new year to you as well!17:11
mordredpabelanger: what fungi and rcarrillocruz is correct. novaclient does not. openstackclient is your better bet17:12
mordredpabelanger: I added support to neutronclient for it because osc didn't have good neutron support for a while17:12
pabelangerk17:12
pabelangerissue is, I need to run nova reset-state --active mirror.regionone.infracloud-chocolate.openstack.org17:12
pabelangerbut, openstackclient doesn't support reset-state17:12
mordredah - I have no idea if osc can do that or not17:12
mordredah17:13
*** jamesdenton has joined #openstack-infra17:13
mordredwell piddle17:13
fungiadmin function support in osc has a tendency to lag behind17:13
mordredyou could do a direct rest call using os_client_config.make_rest_client('compute', cloud= ...) ...17:13
*** asettle has quit IRC17:14
*** trown is now known as trown|lunch17:14
openstackgerritClark Boylan proposed openstack-infra/devstack-gate: Run test-matrix.py under python3 if present  https://review.openstack.org/41798217:15
clarkbdims: and something like ^ to actually just use python3 there17:15
stevemarpabelanger: hmm, it doesn't?17:16
clarkbpabelanger: does that imply you were able to successfully reboot the hypervisor using ironic?17:16
pabelangerclarkb: yes, server back online, but VM in bad state. Trying to fix that17:16
stevemarpabelanger: `openstack server set <server_name_or_id> --state <active, error>17:16
pabelangerstevemar: Oh...17:17
stevemarhttp://docs.openstack.org/developer/python-openstackclient/command-objects/server.html#server-set17:17
*** inc0 has joined #openstack-infra17:18
*** tnovacik has joined #openstack-infra17:19
dimsclarkb : python3-PyYAML - you confirmed on a fedora system? LGTM otherwise17:19
clarkbdims: https://apps.fedoraproject.org/packages/python3-PyYAML and we should enable epel I think17:20
clarkbdims: change is self testing though so should fail on that stuff if I got it wrong17:20
*** tonytan4ever has quit IRC17:20
*** tonytan4ever has joined #openstack-infra17:21
pabelangerstevemar: thanks, was looking in the wrong place17:21
dimsclarkb : works for me!17:21
*** emerson has quit IRC17:23
fungistepping out to run a quick errand, brb17:23
pabelangerhttp://mirror.regionone.infracloud-chocolate.openstack.org/17:26
pabelangerstevemar: thanks for the help17:27
*** mriedem has joined #openstack-infra17:28
openstackgerritPaul Belanger proposed openstack-infra/project-config: Revert "Disable chocolate until mirror is up again"  https://review.openstack.org/41798517:29
pabelangerfungi: clarkb: ^17:29
*** jpich has quit IRC17:31
*** umeshksingla has quit IRC17:31
*** hashar is now known as hasharAway17:32
openstackgerritMonty Taylor proposed openstack-infra/shade: Log request ids when debug logging is enabled  https://review.openstack.org/41793117:32
*** jheroux has quit IRC17:32
*** jheroux has joined #openstack-infra17:38
openstackgerritBen Nemec proposed openstack-infra/tripleo-ci: Cherry-pick neutron ha fix  https://review.openstack.org/41798917:39
*** jascott1_ has quit IRC17:39
*** liusheng has quit IRC17:40
*** liusheng has joined #openstack-infra17:41
*** toabctl has quit IRC17:42
*** dtantsur has quit IRC17:42
*** dtantsur has joined #openstack-infra17:44
*** yamahata has quit IRC17:45
*** jascott1_ has joined #openstack-infra17:45
openstackgerritBen Kero proposed openstack-infra/tripleo-ci: WIP: Add OOOQ multinode test support  https://review.openstack.org/41611017:46
*** dtantsur is now known as dtantsur|afk17:46
pabelangermordred: So, we are still getting some username / password exceptions from RAX glance API. Do you have some times to poke at it for nodepool?17:46
*** liusheng has quit IRC17:47
*** liusheng has joined #openstack-infra17:47
mordredsure.17:48
mordredpabelanger: where ya seeing them/17:48
mordred?17:48
*** tonytan4ever has quit IRC17:48
pabelangermordred: nb01.o.o /var/log/nodepool/builder-debug.log17:49
*** tonytan4ever has joined #openstack-infra17:49
openstackgerritMerged openstack-infra/nodepool: Fix some doc typos  https://review.openstack.org/41257117:49
openstackgerritDavid Shrewsbury proposed openstack-infra/nodepool: Create per-provider ZK work threads  https://review.openstack.org/41799417:50
*** nadya has joined #openstack-infra17:50
mordredpabelanger: that should be fixed/different error with latest shade17:51
pabelangermordred: we haven't deleted images in quiet some time from rax, I think we're pushing around 400 leaked images17:51
pabelangermordred: okay, great17:51
mordredpabelanger: nod17:51
mordredpabelanger: perhaps we should restart ... EXCEPT17:51
*** Guest96631 is now known as mgagne17:51
mordredlet's hold off, since it's just a leak for now17:51
*** mgagne has quit IRC17:51
*** mgagne has joined #openstack-infra17:51
mordredpabelanger: because the shade gate is broken so landing an emergency fix if there is a problem uncovered by the upgrade will be "hard"17:51
pabelanger++17:51
mordredactively investigating the gate break currently with sc`17:52
mordredgah17:52
mordredactively investigating the gate break currently with sc86cal17:52
*** bhunter71 has joined #openstack-infra17:54
openstackgerritsebastian marcet proposed openstack-infra/openstackid-resources: Fix on missing using added missing using on DoctrineChatTeamInvitationRepository  https://review.openstack.org/41799617:56
*** dtantsur|afk has quit IRC17:57
*** csaba has quit IRC17:58
*** pblaho has quit IRC17:58
*** pblaho has joined #openstack-infra17:58
*** derekh has quit IRC17:58
*** itisha has joined #openstack-infra17:58
*** csaba has joined #openstack-infra17:58
*** lucasagomes is now known as lucas-afk17:59
*** dtantsur has joined #openstack-infra17:59
*** tonytan4ever has quit IRC18:00
*** tonytan4ever has joined #openstack-infra18:00
openstackgerritMerged openstack-infra/openstackid-resources: Fix on missing using added missing using on DoctrineChatTeamInvitationRepository  https://review.openstack.org/41799618:00
openstackgerritMerged openstack-infra/tripleo-ci: Add temporary key to OVB stack if undercloud is created  https://review.openstack.org/41347718:03
*** bhunter71 has quit IRC18:04
openstackgerritMerged openstack-infra/tripleo-ci: Remove the current CI matrix from tripleo-ci  https://review.openstack.org/40969718:04
*** ihrachys_ has joined #openstack-infra18:04
openstackgerritMerged openstack-infra/project-config: Revert "Disable chocolate until mirror is up again"  https://review.openstack.org/41798518:05
*** mriedem has quit IRC18:05
*** ralonsoh has quit IRC18:05
*** woodster_ has joined #openstack-infra18:06
*** ihrachys has quit IRC18:06
*** spzala has joined #openstack-infra18:08
*** bhunter71 has joined #openstack-infra18:08
openstackgerritDavid Shrewsbury proposed openstack-infra/nodepool: Remove --no-launches nodepoold option  https://review.openstack.org/41800118:09
*** trown|lunch is now known as trown18:11
*** jascott1_ has quit IRC18:13
*** nadya has quit IRC18:14
clarkbpabelanger: you didn't reboot all the down hosts in chocolate did you? if you didn't I'd like to do that (just to get experience with it)18:15
*** palendae has joined #openstack-infra18:15
clarkbpabelanger: and if you did, oh well I am srue there will be another opportunity :)18:15
palendaeIs this the channel to get help on customizing setup.{cfg,py} for pbr?18:16
clarkbpalendae: it is probably one of the better locations (oslo's channel being the other)18:16
*** lVathan[m] has joined #openstack-infra18:16
pabelangerclarkb: not all, I'm still working though them.  If you want to finish up, I'm okay with that18:16
clarkbpabelanger: if that is ok with you I'd like to do that just to get hands on with infra cloud a bit more18:17
pabelangerclarkb: sure, np18:17
clarkbI am gonna grab caffiene and breakfast then get to it18:17
palendaeI'm looking to modify the setup.cfg/py for openstack-ansible (http://git.openstack.org/cgit/openstack/openstack-ansible/tree/) such that the 'lib' directory is the root for the package18:17
*** drifterza has joined #openstack-infra18:18
mordredpalendae: hi!~18:19
palendaeo/ mordred18:19
mordredpalendae: that is not supported in pbr, nor will it be18:19
palendaeOk, that's what I thought...18:19
palendaemordred: So is there a way to write our own setup.py and get global requirements updates?18:19
palendaeafaik the only reason openstack-ansible has a setup.py/pbr right now is for that18:20
mordredpalendae: we'd like to avoid that - let me look at your tree real quick18:20
mordredpalendae: oh - you don't actually use your setup.py for anything?18:20
palendaeNot other than updates18:20
palendaeThe majority of the stuff in there is ansible playbooks in YAML18:21
palendaeBut we would like to take the python parts and make them installable18:21
mordredthose are python ansible modules right?18:21
sigmavirusmordred: in lib? no18:22
palendaeIn lib, no - they're modules used by the playbooks/inventory/dynamic_inventory.py script. But we'd like to, eventually, make them pluggable via stevedore18:22
*** yamahata has joined #openstack-infra18:22
palendaeThey're not given directly to ansible; ansible calls the dynamic_inventory.py script via subprocess18:22
sigmavirusalso "that is not supported in pbr, nor will it be" is rather a horrible opinion to have of an increasingly common way of structuring repositories and having setuptools discover the modules (inside a separate directory)18:22
mordredgotcha. so - in that case, honestly, I'd rename lib to openstack_ansible so that they can get installed into site-packages18:22
sigmavirusThat attitude will ensure that only OpenStack will ever use PBR going forward18:22
palendaeI'm in agreement with sigmavirus there...but I'll try renaming the dir18:23
sigmavirusI don't think that was ever the intent behind the project18:23
AJaegerrakhmerov, are you around to discuss i18n for mistral?18:23
*** Jeffrey4l_ has joined #openstack-infra18:23
palendaeUnfortunately the pbr opinions block off a lot of functionality18:23
mordredsigmavirus: well, there is no specific reason one _must_ put things into a lib subdir. it does not add functionality to do so18:23
mordredit is a way people can choose to organize things18:23
sigmavirusmordred: I'm not saying pbr should enforce that. But it shouldn't prevent it either18:24
palendaemordred: Right; it's a way people can choose to organize, and pbr's explicitly preventing it18:24
*** gyee has joined #openstack-infra18:24
mordredpbr's entire purpose in life is to remove choice so that people can just do things one way and ahve that way work every time. setuptools is available if someone wants things to be more complex18:24
palendaeSo if I simply use setuptools, do I lose global requirements updating?18:25
mordredplease do not do that if you are an openstack project18:25
palendae...18:25
*** rafaelss has joined #openstack-infra18:25
mordredthe reason that pbr is so opininate is that we have 1500 python repos and the flexibility of choice offered by setuptools is too large18:25
sigmavirusmordred: if that were true, passing through all options to setuptools shouldn't be a thing pbr does, because that sounds contrary to its purpose and it seems to not be internally consistent18:25
palendaeThat seems like a very poor stance18:25
*** Hal has joined #openstack-infra18:26
*** Hal is now known as Guest6562918:26
palendaemordred: So you're saying if pbr restricts a project too much, feel free to use setuptools...however if that project is hosted in openstack repos, don't use setuptools18:26
mordredso - the thing I would imagine you're trying to do by having 6 bare python files in a lib dir is to get those installed into site-packages directly. that would result in top level modules named "filesystem" and "ip" and "tools" - which is a pretty dirty thing to do to a system. is there a reason to _not_ put them into a module that is related to the project they are in?18:27
clarkbpabelanger: looks like I want to touch all the instances that are in a state of power off + active (eg ignore deploy failed nodes)?18:27
mordredpalendae: I'm saying for non-openstack projects, there is nothing stopping people from using setuptools18:27
mordredI'm saying for openstack projects the consistency provided by reduction of flexibilty is valuable18:27
mordredit's two different problems18:27
palendaeyeah18:28
palendaeWell, we're an optenstack project that would appreciate it18:28
mordredtotally understand18:28
palendaeAnd I would put them in a module18:28
palendaeIf I put them in lib/openstack_ansible that's fine18:28
mordredyah - so why not just put them in a model?18:28
openstackgerritDavid Shrewsbury proposed openstack-infra/nodepool: Remove --no-builder nodepoold option  https://review.openstack.org/41801318:28
mordredjust rename lib to openstack_ansible and it'll just work18:28
mordredwhy do you need the lib dir?18:28
*** sdake_ has quit IRC18:28
*** rafaels has quit IRC18:28
*** sdake has joined #openstack-infra18:28
palendaeInitially that was there as a place to put refactored code18:29
palendaeIt was all in a single .py file18:29
palendaeAnd it was 'lib' to differentiate from most of the code being ansible playbooks18:29
mordrednod18:29
pabelangerclarkb: yes, I set node-set-maintenance off then node-set-power-state reboot for those nodes after some time, the physical server came back online18:29
palendaeI'll try that18:29
clarkbpabelanger: and they automagically enter maintenance when ironic has trouble with them?18:30
*** thorst has quit IRC18:30
clarkbpabelanger: but then I should be able to go over to the chocolate controller and see that those nodes have reconnected to nova ya? /me sets that up too18:30
*** thorst has joined #openstack-infra18:31
mordredpalendae: I really didn't/don't mean to be negative - if there is a use case that you need that we're not handling, I very much want to support you if at all possible - I just want to start with seeing if we can do so without divergence first18:31
pabelangerclarkb: looks that way, ironic node-show will list a maintenance_reason18:31
clarkbpabelanger: tyty18:31
*** harlowja_ has joined #openstack-infra18:31
*** harlowja has quit IRC18:31
*** sdake_ has joined #openstack-infra18:31
palendaemordred: I understand; changing the directory name's going to be the path of least resistance18:32
palendaeI'm just frustrated by the "oh, well just use setuptools...but only use pbr if you're an openstack project" stance18:32
palendaeI'd rather just hear "we don't support setuptools without pbr", personally18:32
mordredpalendae: yes - sorry - that's what I should have said - I got things confused by also trying to answer the question about wider adoption of pbr in the same conversation18:33
palendaeGotcha18:33
mordredpalendae: the stance actually is "we do not support setuptools without pbr in openstack"18:33
*** sdake has quit IRC18:33
palendaeOk18:33
palendaeAnd, I assume, pbr's not supported outside of openstack?18:33
mordredit is18:33
palendaeI can't imagine using it otherwise18:33
mordredthere are plenty of non-openstack projects using it18:34
mordredbut that's not its primary goal18:34
palendaeSure18:34
*** thorst has quit IRC18:35
palendaeanyway, that addresses the main questions, so thanks mordred!18:35
clarkbpabelanger: I am noticing that ironic things compute024 is powered off but it is up (I can ssh to it and nova says it is there) so I am going to skip that one for now and do some of the others18:36
clarkbpabelanger: but maybe you know what is going on with 024?18:36
*** sambetts is now known as sambetts|afk18:36
mordredpalendae: you're welcome! sorry for the confusion and/or tone18:36
pabelangerclarkb: sure, let me look18:38
*** xnox has quit IRC18:38
*** tonytan4ever has quit IRC18:38
*** pradk- has quit IRC18:39
*** erlon is now known as erlon_pto18:39
*** tonytan4ever has joined #openstack-infra18:39
openstackgerritPaul Belanger proposed openstack-infra/nodepool: Remove the ability for nodepoold to launch a builder  https://review.openstack.org/41093918:39
*** sdake has joined #openstack-infra18:39
*** nadya has joined #openstack-infra18:39
openstackgerritMerged openstack-infra/zuul: Fix test_rerun_on_abort  https://review.openstack.org/41142618:40
pabelangerclarkb: ya, looks like ironic shut it off, when it was actually still online (network issue?)18:40
*** xnox has joined #openstack-infra18:40
*** drifterza has quit IRC18:41
*** rbrndt has quit IRC18:41
clarkbI have 042 rebooting now18:41
pabelangerclarkb: if we take it out of maintenance, will ironic poll it again to get the correct state?18:41
clarkbpabelanger: maybe?18:42
clarkbpabelanger: probably worht trying18:42
pabelangerreading docs now to see18:42
*** sdake_ has quit IRC18:42
clarkbworst case we reboot it like the others and zuul restarts any jobs on VMs there18:42
pabelangerya18:42
clarkbpabelanger: it is normal for reboot to take a while before nova sees the hypervisor/ssh works ya?18:43
clarkbguessing this hardware needs like 5-10 minutes to round trip on that18:43
pabelangerclarkb: Ya, took some time18:43
*** e0ne has joined #openstack-infra18:46
clarkbok, 042 seems happy again. Hopefully last question, do I need to clean up VMs that were running on 042 before whatever happened?18:46
clarkbpabelanger: ^18:46
clarkbserver list --all-projects shows every VM is active so I guess not18:47
pabelangerclarkb: So, I haven't done any VM cleanup, yet18:47
pabelangerfigured nodepool would seem them a leaked nodes18:47
clarkbpabelanger: ya I think that is what happened, at least querying as admin I only see happy active VMs18:48
*** thorst has joined #openstack-infra18:49
openstackgerritsebastian marcet proposed openstack-infra/openstackid-resources: Fix on get my invitations  https://review.openstack.org/41802118:49
*** rtheis has quit IRC18:52
openstackgerritsebastian marcet proposed openstack-infra/openstackid-resources: Fix on get my invitations  https://review.openstack.org/41802118:53
pabelangerclarkb: I know I mentioned it before, but it would have been neat for nodepool to realise our VM for the AFS mirror was online, attempt to destroy it, and launch a replacement.18:53
pabelangers/online/offline18:53
fungipabelanger: yep, in the past we've discussed various auto(re)deployment schemes, some of which might integrate with a future state/feature of nodepool18:54
fungipabelanger: what makes it a bit of a challenge to generalize across providers right now is lack of consistent ip addressing necessitating automated dns changes18:55
clarkbya, I just wonder if nodepool is the right place for that since mirrors aren't inherently a nodepool thing18:55
clarkb(heat for example can do all that for you I think)18:55
*** john-davidge has joined #openstack-infra18:55
pabelangerfungi: Ya, dns is a thing to deal with for sure18:55
clarkb(or even ansible with os_server? and a loop?)18:55
mordredwe could totally do it with launcher18:56
*** rbuzatu_ has joined #openstack-infra18:56
openstackgerritBen Kero proposed openstack-infra/tripleo-ci: WIP: Add OOOQ multinode test support  https://review.openstack.org/41611018:56
pabelangermordred: which launcher :)18:56
fungipabelanger: what is a lot cheaper though is if nodepool failed to bring nodes into service (or didn't even try to boot them?) when the mirror is unreachable18:56
mordredpabelanger: hte ansible cloud launcher - ricky's thing18:56
openstackgerritJuan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: Add failures output of ovb stack to TE_DATAFILE  https://review.openstack.org/41802818:56
mordredpabelanger: we could probably pretty safely make a playbook that defines all of the AFS mirrors and ensures they stay around18:57
clarkbmordred: ya that and add "convergence" by putting an invocation in a cron loop somewhere18:57
clarkbas fungi points out the tricky thing is likely dns though18:57
pabelangerI think the missing piece right now is the ansible-role-puppet bit18:58
fungiright, i think we will need to un-table our back-burner dns automation ideas as a prerequisite18:58
pabelangerfor cloud-launcher18:58
*** rbuzatu has quit IRC18:59
*** nadya has quit IRC19:00
*** makowals_ has quit IRC19:03
*** dtardivel has quit IRC19:04
*** makowals has joined #openstack-infra19:05
openstackgerritJuan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: Add stack-show output of ovb stack to TE_DATAFILE  https://review.openstack.org/41802819:05
*** claudiub|2 has joined #openstack-infra19:05
*** bhunter71 has quit IRC19:06
clarkbaround 2017-01-08 19:33:43 a number of our logstash worker processes stopped getting gearman events19:07
clarkbits weird because that timestamp shows up on many of the servers as when they went down on polling the connection19:07
clarkbso not an OOM issue I don't think19:08
clarkbpossible network problems? in any case restarting those workers so they do work again19:08
*** claudiub has quit IRC19:09
openstackgerritJuan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: Add stack-show output of ovb stack to TE_DATAFILE  https://review.openstack.org/41802819:12
mikalShall I assume the apt mirror issues with python35 tests are a known issue?19:13
clarkbmikal: if the job ran in infracloud then yes19:14
fungimikal: are they for the infra-cloud "chocolate" region? if so that's been resolved for a while19:14
mikalIs that the chocolate thing?19:14
mikalYeah, that19:14
clarkbmikal: yup19:14
mikalOk, I'll recheck19:14
*** ianychoi has quit IRC19:14
clarkbpabelanger: did you try setting 024 to maintenance off? I see that it is currently still True so if you did that didn't help. If you didn't I can go ahead and give it a try19:15
*** rcernin has joined #openstack-infra19:15
fungimikal: we took it out of service around 15:00 utc so should only be impacting results from then or earlier19:15
*** rbrndt has joined #openstack-infra19:15
mikalYep, these are failures from about 12 hours ago19:15
fungipabelanger: clarkb: feel free to #status log a followup to my last entry at the top of https://wiki.openstack.org/wiki/Infrastructure_Status19:16
clarkbpabelanger: also 064 is supposedly powered on but nova sees it as off and I cannot ssh to it. So I am going to reboot it via ironic as well19:16
clarkbfungi: will do as soon as these last few items are cleared up19:16
clarkbpabelanger: also 002 so rebooting that one too19:17
*** pradk has quit IRC19:17
pabelanger++19:19
pabelangerclarkb: not yet, I can do it now19:19
pabelangerclarkb: okay, off now19:19
pabelangerlets see what happens19:19
*** dizquierdo has joined #openstack-infra19:20
*** jordanP has quit IRC19:20
openstackgerritMerged openstack-infra/project-config: add networking-sfc tempest tests into periodic jobs.  https://review.openstack.org/40995519:22
clarkb#status log using ironic node-set-maintenance $node off && ironic node-set-power-state $node reboot infracloud hypervisors that had disappeared were brought back to life. The mirror VM was then reenabled with openstack server set $vm_name active.19:23
openstackstatusclarkb: finished logging19:23
clarkbfungi: ^ that help?19:23
*** aviau has quit IRC19:23
*** aviau has joined #openstack-infra19:24
clarkbstill trying to get into 002 and 06419:24
*** ihrachys_ has quit IRC19:25
clarkbpabelanger: 2017-01-09 19:15:56.484 17967 ERROR oslo.messaging._drivers.impl_rabbit [-] AMQP server on controller00.chocolate.ic.openstack.org:5671 is unreachable: [Errno -2] Name or service not known. Trying again in 32 seconds. is the problem with 06419:27
clarkboh wait thats older logs19:28
*** pradk has joined #openstack-infra19:28
*** pradk has quit IRC19:28
*** david-lyle has quit IRC19:28
fungiclarkb: thanks, i guess one can infer from it that we've brought chocolate back into service for nodepool19:32
clarkbI can make that connection via nc and it seems to be fine. Also hasn't logged it in a bit but controller still says that compute host is down.19:32
clarkbif I grep for the controller on the compute host I don't get much. Do we not log debug?19:32
pabelangerclarkb: no debug by default19:33
clarkbgrep says no... I wonder if we need that19:33
clarkbpabelanger: for 064 I am going to try force up the service via compute service set --up compute064.chocolate.ic.openstack.org19:34
clarkbpabelanger: since I see nothing else broken19:34
pabelangerclarkb: ack19:35
*** pradk has joined #openstack-infra19:35
clarkbCompute service nova-compute of host compute064.chocolate.ic.openstack.org failed to set.19:36
*** MasterOfBugs has joined #openstack-infra19:36
*** cgfielde has joined #openstack-infra19:36
clarkbbut that generated an error in the compute log so progress. Looks like nova-compute can't connect to libvirt socket19:36
openstackgerritMerged openstack-infra/devstack-gate: Remove USE_PYTHON3 usage  https://review.openstack.org/41772019:38
*** bhunter71 has joined #openstack-infra19:38
clarkbsuccess, I had to restart the libvirt-bin process then the nova-compute process and it updated itself19:39
*** dprince has quit IRC19:40
clarkb002 is still unresponsive19:41
clarkbpabelanger: 024 shows power on now, but it also shows it went back to maintenance :/19:42
*** john-davidge has quit IRC19:43
openstackgerritBen Kero proposed openstack-infra/tripleo-ci: WIP: Add OOOQ multinode test support  https://review.openstack.org/41611019:45
openstackgerritJames Slagle proposed openstack-infra/tripleo-ci: Implement major upgrade for Newton to Ocata  https://review.openstack.org/40483119:45
clarkbpabelanger: reading the maintenance reason it appears it will flip that toggle any time it discovers an unexpected state so I think it is safe to set it back to maintenance off. Doing that now19:46
clarkband now it is ahppy19:47
openstackgerritBen Kero proposed openstack-infra/tripleo-ci: WIP: Add OOOQ multinode test support  https://review.openstack.org/41611019:48
clarkb\o/ only node that is out of whack is 002 and that may require a ticket? I can't get it to talk to me though ironic seems to think it is active19:48
clarkbrcarrillocruz: if you have a moment at some point in the future poking at chocolate compute 002 would be helpful19:48
*** umeshksingla has joined #openstack-infra19:48
pabelangerclarkb: cool, did you do anything for 024?19:49
clarkbpabelanger: I set it back to maintenance off19:50
*** pcaruana has joined #openstack-infra19:50
pabelangerand reboot?19:50
AJaegermriedem, sdague, dims: Could you review this nova py35 change, please? https://review.openstack.org/41010519:50
clarkbpabelanger: basically anytime you set maintenance off and it finds a state it doesnt expect it set maintenance on. So when you set it off and it transition from powered off -> on without an actual reboot it set maintenance back on immediatlely beacuse that was an unexpected state19:50
pabelangerk19:50
clarkbpabelanger: so I set it back to maintenance off again and this time it had no state transition beacuse it was powered on and discovered it on and so stayed that way. No reboot19:50
*** dimtruck is now known as zz_dimtruck19:51
rcarrillocruzclarkb: that's going to be tricky, if there are HW issues i cannot open tickets19:51
rcarrillocruzfrom today, I no longer work at HP, just started at Ansible19:51
rcarrillocruzasselin may have access to the ticketing system19:52
asselinhi....what's up?19:52
dimsAJaeger : lgtm19:53
*** umeshksingla has quit IRC19:53
openstackgerritClark Boylan proposed openstack-infra/project-config: Run fedora platform job against devstack-gate  https://review.openstack.org/41808019:53
AJaegerthanks, sdague and dims19:53
clarkbAJaeger: ianw can has review on 418080?19:54
*** umeshksingla has joined #openstack-infra19:54
clarkbrcarrillocruz: mugsie earlier had asked about it. I guess my question is more what can we do on our end to debug or do we have to file a ticket at this point?19:54
AJaegerclarkb: on it...19:54
rcarrillocruzasselin: it seems the infracloud machines have hw issues19:54
AJaegerany takers on reviewing https://review.openstack.org/#/c/417693 - step 1 of retiring fairy-slipper - ,please?19:54
rcarrillocruzbut i cannot access Jira anymore19:54
asselinrcarrillocruz, ok...I did see an e-mail about power interruption in houston am219:55
rcarrillocruzoh19:55
rcarrillocruznice19:55
rcarrillocruzclarkb: ^19:55
*** e0ne has quit IRC19:55
rcarrillocruzthat might be it19:55
rcarrillocruzthat's the DC where the infracloud is19:55
mugsiethey did claim there was no impact, but it could be the issue19:56
clarkbyup, and we rebooted all the other instances that died and got them back in but 002 is still non responsive19:56
asselinOn 01/09/2017 at 6:44 am (CST),    Utility power was restored at 7:42 am (CST).19:56
clarkboh no this happened like 2 days ago19:56
clarkbon the 7th19:56
rcarrillocruzhaha, it wouldn't suprirse me if there were issues 2 days ago19:56
*** e0ne has joined #openstack-infra19:56
rcarrillocruzand today the had a schduled maintenance window to address power or go figure19:57
clarkbanyways we are 95% back to normal. One host does not come back to us if asked to reboot but ironic thinks it is powered up19:57
clarkbthat one host is infracloud chocolate compute 00219:57
*** e0ne has quit IRC19:57
clarkbrcarrillocruz: so more generally is there any other debugging you/we can do on our end before filing a ticket?19:57
*** e0ne has joined #openstack-infra19:57
pabelangerwe should be able to get onto console, if ironic sees compute002.chocolate.ic.openstack.org online19:57
rcarrillocruzi can look in a bit, about to jump in a call19:57
rcarrillocruzbut normally i just get into iLO and see what's up19:58
*** e0ne has quit IRC19:58
*** umeshksingla has quit IRC19:58
*** e0ne has joined #openstack-infra19:58
clarkband ilo is located at the ipmi addr and uses the impi passwd?19:59
openstackgerritMerged openstack-infra/openstackid-resources: Fix on get my invitations  https://review.openstack.org/41802119:59
*** umeshksingla has joined #openstack-infra19:59
*** umeshksingla has quit IRC19:59
*** umeshksingla has joined #openstack-infra19:59
*** e0ne has quit IRC20:00
openstackgerritMerged openstack-infra/project-config: Add functional-py35 non-voting job for Nova  https://review.openstack.org/41010520:00
openstackgerritsebastian marcet proposed openstack-infra/openstackid-resources: Fix on create/update team desc validation now team description if set to empty or null is reseted  https://review.openstack.org/41808320:00
openstackgerritMerged openstack-infra/openstackid-resources: Fix on create/update team desc validation now team description if set to empty or null is reseted  https://review.openstack.org/41808320:02
*** panda|bbl is now known as panda20:02
*** e0ne has joined #openstack-infra20:03
*** e0ne has quit IRC20:03
rcarrillocruzclarkb: yah, that's right20:04
clarkbnow to find where the ipmi password is stashed20:04
jrollclarkb: driver_info20:05
jrolland in more recent ironic, hidden with policy stuff20:05
*** salv-orlando has joined #openstack-infra20:05
pabelangerclarkb: /opt/stack/baremetal.json I believe20:05
*** bhunter71 has quit IRC20:06
clarkbjroll: thanks, bifrost also has it in the inventory file which is ^20:06
*** gouthamr has quit IRC20:06
jrollah right20:06
*** salv-orlando has quit IRC20:10
*** andreww has quit IRC20:11
clarkbthe last recorded entry in /system1/log1 is from july of 2016 so nothing there20:13
clarkbrcarrillocruz: pabelanger any other hints at what I should be looking at?20:13
rcarrillocruzin a call, can look later, sorry20:13
pabelangerclarkb: no, I've never used the iLO before20:13
*** ldnunes has quit IRC20:15
*** bknudson has left #openstack-infra20:16
*** nherciu_ has quit IRC20:18
*** salv-orlando has joined #openstack-infra20:18
*** eranrom has quit IRC20:18
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: Add scenario005 job (4nodes)  https://review.openstack.org/41809820:19
*** flepied has quit IRC20:20
clarkbalright I need to break for lunch but will be back in a bit to poke at it more. Haven't seen anything obviously wrong though20:20
openstackgerritFrédéric Guillot proposed openstack-infra/project-config: Disable identity v2 for Almanach gate  https://review.openstack.org/41810420:24
*** bknudson has joined #openstack-infra20:26
serverascodewould anyone be willing to allow this change to add a new project in project-config? https://review.openstack.org/#/c/417522/ I have a ops team meeting in a couple days and it would be great to have that repo available for us to start working with20:27
*** ricardoas has quit IRC20:27
AJaegerserverascode: do you have any existing code that you want to import?20:28
*** mrtenio has quit IRC20:28
serverascodenope we don't20:28
AJaegerserverascode: could you remove some extra whitespace at https://review.openstack.org/#/c/417522/3/zuul/layout.yaml , please?20:28
serverascodeAJaeger ok will do20:28
AJaegerserverascode: thanks - I'll +2 afterwards20:29
*** zz_dimtruck is now known as dimtruck20:31
openstackgerritCurtis Collicutt proposed openstack-infra/project-config: Add ops-telecom-nfv repository  https://review.openstack.org/41752220:32
AJaegerserverascode: leave an empty line but without whitespace on it20:34
*** eranrom has joined #openstack-infra20:34
serverascodeAJaeger oh ok sorry20:34
*** apetrich has quit IRC20:35
*** apetrich has joined #openstack-infra20:36
*** david-lyle has joined #openstack-infra20:37
openstackgerritCurtis Collicutt proposed openstack-infra/project-config: Add ops-telecom-nfv repository  https://review.openstack.org/41752220:38
AJaegerserverascode: LGTM, thanks for updating20:39
serverascodeAJaeger thank you20:39
AJaegerpabelanger, rcarrillocruz, clarkb could you review https://review.openstack.org/#/c/417693 - to retire fairyslipper, please? Also, a review for a nit: https://review.openstack.org/41766620:41
*** baoli has quit IRC20:41
*** rbuzatu_ has quit IRC20:43
*** srobert_ has joined #openstack-infra20:46
*** srobert_ has quit IRC20:48
*** srobert_ has joined #openstack-infra20:48
*** srobert has quit IRC20:49
openstackgerritJohn Trowbridge proposed openstack-infra/tripleo-ci: WIP: Add OOOQ multinode test support  https://review.openstack.org/41611020:49
*** mrtenio has joined #openstack-infra20:50
openstackgerritMerged openstack-infra/project-config: Don't use single item list for node in heat  https://review.openstack.org/41766620:50
*** ricardoas has joined #openstack-infra20:51
*** eranrom has quit IRC20:51
ianwclarkb: looking20:51
openstackgerritMerged openstack-infra/tripleo-ci: Add "undecloud" or extra node's details to OVB's nodes' json  https://review.openstack.org/41354820:52
*** srobert has joined #openstack-infra20:53
*** shardy has quit IRC20:53
*** srobert has quit IRC20:53
*** hichihara has joined #openstack-infra20:54
*** dizquierdo has quit IRC20:55
*** srobert_ has quit IRC20:56
*** umeshksingla has quit IRC20:56
*** lindycoder has joined #openstack-infra20:57
*** umeshksingla has joined #openstack-infra20:57
openstackgerritMerged openstack-infra/project-config: Run fedora platform job against devstack-gate  https://review.openstack.org/41808020:59
openstackgerritDavid Shrewsbury proposed openstack-infra/nodepool: Handle provider removal from config  https://review.openstack.org/41811221:00
*** jamielennox|away is now known as jamielennox21:00
*** baoli has joined #openstack-infra21:00
*** timrc has quit IRC21:01
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Reduce our 3-node labels to min-ready 0  https://review.openstack.org/41471421:02
AJaegerpabelanger: rebased it manually for you ^21:02
*** shamail has joined #openstack-infra21:02
*** jaosorior has quit IRC21:03
AJaegerianw, can you +2A https://review.openstack.org/414714 and https://review.openstack.org/417693 , please? Nothing urgent - whenever you have time...21:03
*** jaosorior has joined #openstack-infra21:03
*** jlk has quit IRC21:06
*** xarses has joined #openstack-infra21:06
* AJaeger waves good night21:06
*** bhunter71 has joined #openstack-infra21:08
*** flepied has joined #openstack-infra21:08
ianwAJaeger: will do after some morning coffee :)  good night21:10
*** rafaels has joined #openstack-infra21:10
*** rafaelss has quit IRC21:13
openstackgerritKen Giusti proposed openstack-infra/project-config: Deprecate the amqp1 dsvm-functional test on centos 7  https://review.openstack.org/41797121:14
*** denismakogon has joined #openstack-infra21:15
*** tonytan4ever has quit IRC21:15
shamailHi infra-team, is there a way to take over as chair of a meeting if the initial user got disconnected?21:16
*** tonytan4ever has joined #openstack-infra21:16
fungishamail: only if they added you as an additional chair first21:16
shamailfungi: they did not (this is for the meeting in openstack-3 right now)21:17
shamailerr21:17
shamailopenstack-alt21:17
shamailsorry.21:17
fungishamail: options are to keep talking in hopes they show back up, or wait quietly for them to show back up. once an hour has elapsed from #startmeeting anyone (not only the chairs) can issue a #endmeeting21:18
*** dimtruck is now known as zz_dimtruck21:18
fungiyou won't be able to officially change the topic, but can probably get by21:18
pabelangerif CarolBarrett_ renames, should it work again?21:19
pabelangerback to CarolBarrett21:19
fungiyeah, all meetbot cares about is that the chair nick matches21:19
pabelangershamail: have the chair rename back to their original IRC nick21:19
pabelangercommands should work again21:19
*** tonytan4ever has quit IRC21:20
shamail thanks fungi and pabelanger, ill ask Carol to change her nick21:20
*** tonytan4ever has joined #openstack-infra21:20
*** dfflanders has joined #openstack-infra21:21
*** hasharAway is now known as hashar21:26
*** kgiusti has left #openstack-infra21:27
shamailThat did the trick, thanks!21:27
fungiglad to help21:27
*** wolverineav has joined #openstack-infra21:29
*** wolverin_ has quit IRC21:30
openstackgerritUmesh Kumar Singla proposed openstack-infra/jenkins-job-builder: Update Gradle plugin to use convert_mapping_to_xml function  https://review.openstack.org/41637721:30
*** flepied has quit IRC21:30
*** ccamacho has quit IRC21:32
openstackgerritZara proposed openstack-infra/storyboard: Remove contentless operator manual  https://review.openstack.org/41812221:32
*** Jeffrey4l__ has joined #openstack-infra21:34
*** Jeffrey4l_ has quit IRC21:35
openstackgerritJohn Trowbridge proposed openstack-infra/tripleo-ci: Use quickstart-extras checked out from zuul refs  https://review.openstack.org/41812321:36
openstackgerritMerged openstack-infra/project-config: Retire fairy-slipper (1/2)  https://review.openstack.org/41769321:36
*** rbrndt has quit IRC21:37
*** rbrndt has joined #openstack-infra21:37
clarkbianw: thanks for the review 41808021:38
clarkbianw: I have rechecked https://review.openstack.org/#/c/417982/ to make sure it doesn't break on our fedora images21:39
*** tonytan4ever has quit IRC21:39
*** adriant has joined #openstack-infra21:43
*** fguillot has quit IRC21:44
*** zz_dimtruck is now known as dimtruck21:44
*** fguillot has joined #openstack-infra21:45
*** eharney has quit IRC21:48
*** rbuzatu has joined #openstack-infra21:48
*** timrc has joined #openstack-infra21:49
clarkbianw: and it looks like it worked fine so thats good to know21:50
ianwcool21:51
*** jlk has joined #openstack-infra21:51
*** jlk has quit IRC21:51
*** jlk has joined #openstack-infra21:51
ianwyeah, f25 is on my today todo list21:51
*** rbuzatu has quit IRC21:53
*** jaosorior has quit IRC21:53
*** ihrachys has joined #openstack-infra21:54
*** john-davidge has joined #openstack-infra21:55
jheskethMorning21:55
clarkbjhesketh: good morning. I status logged the tl;dr on fixing infra chocolate21:55
*** burgerk has joined #openstack-infra21:55
jheskethclarkb: yep just reading the scroll back. Thanks for that21:56
openstackgerritBen Nemec proposed openstack-infra/tripleo-ci: Install openstackclient on te-broker  https://review.openstack.org/41812921:58
*** john-davidge has quit IRC21:59
*** fguillot has quit IRC22:01
jeblairit's zuul meeting time in #openstack-meeting-alt22:01
*** shamail has quit IRC22:02
openstackgerritPaul Belanger proposed openstack-infra/nodepool: Source glean installs in simple-init  https://review.openstack.org/41466222:02
*** hichihara has quit IRC22:03
*** trown is now known as trown|outtypewww22:04
*** jcoufal has quit IRC22:05
*** Goneri has quit IRC22:09
*** adrian_otto has quit IRC22:10
*** jheroux has quit IRC22:11
*** rcernin has quit IRC22:12
openstackgerritPaul Belanger proposed openstack-infra/nodepool: Remove the ability for nodepoold to launch a builder  https://review.openstack.org/41813722:18
openstackgerritPaul Belanger proposed openstack-infra/nodepool: Remove the ability for nodepoold to launch a builder  https://review.openstack.org/41813722:19
*** mixos has joined #openstack-infra22:19
ianwpabelanger: speaking of builders, i'd like to get push notifications going again for builds, if you get a chance to look at https://review.openstack.org/#/c/413874/ (json output for build status) that would help.  won't be a stretch to script that into a quick cron checker22:20
*** mattoliverau has quit IRC22:21
*** matt6434 has joined #openstack-infra22:21
*** baoli has quit IRC22:21
*** matt6434 is now known as mattoliverau22:22
*** mriedem has joined #openstack-infra22:22
*** salv-orl_ has joined #openstack-infra22:23
*** darrenc is now known as darrenc_afk22:23
*** salv-orlando has quit IRC22:26
*** wolverineav has quit IRC22:26
*** duartea__ has quit IRC22:27
*** wolverineav has joined #openstack-infra22:27
*** duartea__ has joined #openstack-infra22:27
*** wolverineav has quit IRC22:27
*** matt-borland has quit IRC22:28
*** wolverineav has joined #openstack-infra22:31
*** mixos has quit IRC22:33
*** spzala has quit IRC22:34
*** wolverineav has quit IRC22:35
*** salv-orl_ has quit IRC22:36
*** salv-orlando has joined #openstack-infra22:37
*** denismakogon has quit IRC22:38
*** jamesdenton has quit IRC22:41
*** adrian_otto has joined #openstack-infra22:42
*** wolverineav has joined #openstack-infra22:43
*** tnovacik has quit IRC22:46
*** wolverineav has quit IRC22:47
jeblairianw: +2 with comment22:50
*** xyang1 has quit IRC22:50
*** burgerk_ has joined #openstack-infra22:52
ianwjeblair: would you think a direct pass-through of toDict() to basically json?  i'm thinking how we generate the "id" and that maybe it's more flexible to have that shim layer22:54
*** Swami has joined #openstack-infra22:54
*** Swami_ has joined #openstack-infra22:55
*** burgerk has quit IRC22:55
*** umeshksingla has quit IRC22:56
Shrewsianw: i left a -1 with a comment  :)22:57
*** panda is now known as panda|zZ22:57
*** burgerk_ has quit IRC22:57
jeblairianw: yeah, i think that along with having the presentation decoupled from the internal data structure so that we aren't surprised by exposing new fields in the future are good reasons for the shim (otoh, we don't automatically expose new fields :)22:57
ianwShrews: cool, i'll look at that22:58
*** david-lyle has quit IRC22:59
*** newmember has joined #openstack-infra22:59
Shrewsianw: i could be off base on what you want, but when i say "won't have certain attributes", i meant "they could have no values". if that's ok, i'll retract my -122:59
Shrewsi was thinking you only wanted valid builds23:00
*** darrenc_afk is now known as darrenc23:00
*** wolverineav has joined #openstack-infra23:00
jeblairShrews, ianw: yeah, i think we want the whole set there -- it does the same thing as the prettytable version above23:00
jeblairwhole set == all of the builds regardless of state23:01
ianwShrews: well, what I used to do was get the latest build log on nodepool.o.o for each build, download it and scrape the output looking for failure/success23:01
ianwit worked better than you might think :)23:01
*** baoli has joined #openstack-infra23:01
*** baoli has quit IRC23:01
Shrewsjeblair: ianw: ah, well if you want the entire set, that's fine. but we have api methods for getting the most recent builds (which is what you mention in the commit msg)23:01
ianwyeah, it's actually mostly about failures, so i get my daily "everything's on fire" message :023:02
Shrewsianw: sounds like you need everything. i'll change my vote then  :)23:02
*** florianf has quit IRC23:03
jeblairianw: https://simpsonswiki.com/wiki/Everything's_OK_alarm23:03
*** rafaels has quit IRC23:05
ianwheh, it's kind of the dream to have that going off :)23:05
*** jaosorior has joined #openstack-infra23:07
ianwi'm thinking with this that we can ship a nice generic-ish bootstrap html page with nodepool (just keep it in a contrib/ type location) which we can deploy as the landing page for nodepool.openstack.org23:07
*** gildub has joined #openstack-infra23:08
ianwrather than having nodepool serving it all up23:08
fungii'm also fond of the inverse, the "PROBLEM" light from http://venturebrothers.wikia.com/wiki/Careers_in_Science23:09
*** EricGonczer_ has quit IRC23:10
*** pradk has quit IRC23:10
fungi(extent of the diagnostic interface on the space station was a singular blinking light)23:10
*** sdake has quit IRC23:11
*** mixos has joined #openstack-infra23:11
*** asettle has joined #openstack-infra23:12
ianwsounds like the ancient ibm mainframe for the billing department of a job i had long ago23:12
jeblairNote: Should the 'PROBLEM' light activate, the condition can be resolved by performing corrective action.23:13
ianwi remember some poor operator sitting there in the freezing ac with a giant folder on his lap trying to decode some problem light23:13
jeblair(see also: "Check engine soon")23:14
*** mixos has quit IRC23:15
*** lindycoder has quit IRC23:16
*** pradk has joined #openstack-infra23:21
*** mriedem has quit IRC23:22
*** asettle has quit IRC23:22
*** wolverineav has quit IRC23:23
*** asettle has joined #openstack-infra23:23
*** sdake has joined #openstack-infra23:26
*** vhosakot has quit IRC23:26
*** asettle has quit IRC23:27
*** sdake has quit IRC23:28
*** sdake has joined #openstack-infra23:29
openstackgerritMerged openstack-infra/devstack-gate: Add save_* functions for clearer saving of artifacts  https://review.openstack.org/33450423:31
*** thorst has quit IRC23:32
*** aeng has quit IRC23:32
*** thorst has joined #openstack-infra23:32
*** wolverineav has joined #openstack-infra23:35
*** salv-orlando has quit IRC23:35
*** thorst has quit IRC23:37
*** Julien-zte has quit IRC23:38
*** gordc has quit IRC23:38
*** wolverineav has quit IRC23:39
*** spzala has joined #openstack-infra23:40
*** harlowja_ has quit IRC23:40
*** jaosorior has quit IRC23:40
*** harlowja has joined #openstack-infra23:41
*** marst has quit IRC23:42
*** lihi has quit IRC23:43
*** lihi has joined #openstack-infra23:43
*** umeshksingla has joined #openstack-infra23:44
*** stakeda has joined #openstack-infra23:44
*** mminesh has quit IRC23:45
*** mminesh has joined #openstack-infra23:45
*** aeng has joined #openstack-infra23:46
*** salv-orlando has joined #openstack-infra23:48
*** ianychoi has joined #openstack-infra23:49
*** mdrabe has quit IRC23:51
*** inc0 has quit IRC23:52
*** salv-orlando has quit IRC23:53
clarkbsdague: ^ there you go23:54
*** wolverineav has joined #openstack-infra23:54
*** palendae has left #openstack-infra23:56
*** rbrndt has quit IRC23:58
openstackgerritHongbin Lu proposed openstack-infra/project-config: Fuxi: get python35 jobs voting  https://review.openstack.org/41815723:58
*** wolverineav has quit IRC23:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!