| *** oanson has joined #openstack-infra | 00:01 | |
| jeblair | that's weird. we only have threads for one connection (which is the expected situation) | 00:01 |
|---|---|---|
| jeblair | so i wonder what the other connection is. do the threads we have correspond with the flapping connection, or the stable one? is the stable connection a ghost? | 00:01 |
| jeblair | oh tcpdump can help answer that | 00:02 |
| clarkb | the thread we have must correspond with the flapping connection because we see it flap in logs and via netstat output (I think that is a correct interpretation) | 00:02 |
| clarkb | but yes tcpdump should show us | 00:03 |
| *** masber has joined #openstack-infra | 00:03 | |
| jeblair | yeah, i'm seeing the local port numbers change | 00:04 |
| *** aeng has joined #openstack-infra | 00:04 | |
| *** gildub has joined #openstack-infra | 00:06 | |
| *** Apoorva_ has quit IRC | 00:07 | |
| *** Apoorva has joined #openstack-infra | 00:08 | |
| jeblair | clarkb: i'm baffled. i'm going to tcpdump to files for a few minutes so that if we have any ideas later, we can investigate, then i'll be ready to restart. | 00:10 |
| clarkb | jeblair: ok | 00:10 |
| clarkb | sounds reasonable | 00:10 |
| *** gmann has quit IRC | 00:11 | |
| *** gmann has joined #openstack-infra | 00:11 | |
| *** jkilpatr has quit IRC | 00:11 | |
| *** Apoorva has quit IRC | 00:12 | |
| clarkb | hrm looks like we also have post failures? | 00:14 |
| clarkb | from the jobs that are running | 00:14 |
| jeblair | clarkb: link? | 00:14 |
| clarkb | possible the zuul launchers are unhappy too? I'm going to try and find the zl logs for some of these to understand why they are unhappy | 00:14 |
| clarkb | http://logs.openstack.org/31/476631/19/gate/gate-nova-pep8-ubuntu-xenial/76c6c13/ is example | 00:14 |
| jeblair | clarkb: okay. i'll restart nodepool now while you do that | 00:15 |
| *** makowals has quit IRC | 00:15 | |
| clarkb | looks like ^ ran on zl05 | 00:16 |
| *** gongysh has quit IRC | 00:17 | |
| jeblair | nodepool performed a status request sucessfully and is spamming clouds | 00:17 |
| clarkb | http://paste.openstack.org/show/614971/ | 00:17 |
| clarkb | I thnik those job failures must be unrelated as it is ssh failure to the remote and job timeouts | 00:17 |
| clarkb | perhaps because nodepool gearman and zuul were out of sync, eg nodepool thought it could delete the node? | 00:18 |
| clarkb | that seems unlikely though | 00:18 |
| *** tjones has joined #openstack-infra | 00:18 | |
| *** tjones has left #openstack-infra | 00:19 | |
| *** rlandy is now known as rlandy|bbl | 00:19 | |
| jeblair | clarkb: maybe possible... maybe if the assign job got far enough on the zuul-launcher side before the gearman connection died and returned a failure to nodepool? | 00:20 |
| jeblair | clarkb: we can check that node id in the log | 00:20 |
| jeblair | 2017-07-10 22:35:01,385 ERROR nodepool.NodeLauncher: NoConnectedServersError launching node id: 9738807 in provider: ovh-bhs1 error: | 00:20 |
| jeblair | clarkb: i think that might actually be it | 00:20 |
| clarkb | interesting but ya I agree | 00:21 |
| *** gcb has quit IRC | 00:21 | |
| jeblair | nodepool has gone through several demand calcs now | 00:21 |
| jeblair | some clouds are unhappy | 00:22 |
| clarkb | fwiw this is similar to the other thing with zookeeper where I think kill -9 is hurting us | 00:22 |
| mordred | jeblair: yah - I was about to walk away from the keyboard then saw those | 00:22 |
| jeblair | clarkb, mordred: i don't know that it would help *this* situation, but we should still merge this: https://review.openstack.org/425248 | 00:23 |
| *** liujiong has joined #openstack-infra | 00:24 | |
| mordred | jeblair: those cloud errors are the result of the cloud returning an ERROR status | 00:24 |
| clarkb | ya I think in this case gracefully killing processes so that python can close sockets and do the tcp fin fin ack would help | 00:24 |
| mordred | well - no - one of them is | 00:24 |
| jeblair | it's now been a few minutes since the last demand calc. that's either because the problem has resurfaced, or just normal thread starvation. we should know soon. | 00:25 |
| mordred | jeblair: we're getting nodes generally - so I don't think it's a system failure in launchind nodes | 00:26 |
| clarkb | jeblair: there is a large number of building instances (which could lead to thread starvation) | 00:26 |
| jeblair | mordred: yeah, probably just hitting stress limits on some clouds because we we went straight to max | 00:26 |
| mordred | yah | 00:26 |
| clarkb | 1060 according to wc -l | 00:26 |
| * mordred needs to update the shade we're running - we'll get better errors on some of these errors with the latest | 00:27 | |
| *** stakeda has joined #openstack-infra | 00:27 | |
| mordred | not all of them - some of the errors are just the errors that we'll get | 00:27 |
| jeblair | that's enough threads for me to think that even if we see a timeout on the status command, it may not be a resurfacing of the problem. | 00:27 |
| clarkb | jeblair: ya | 00:27 |
| clarkb | it is falling under 1k now | 00:27 |
| clarkb | and a portion of them appear to be transitioning to used at least | 00:28 |
| *** tosky has quit IRC | 00:28 | |
| clarkb | I'll try to make an effrot later this week after doing gerrit 2.13 things to look into updating more of our init scripts to try sig term first then fallback on kill after a timeout | 00:28 |
| *** annegentle has joined #openstack-infra | 00:28 | |
| *** dfflanders has joined #openstack-infra | 00:29 | |
| *** makowals has joined #openstack-infra | 00:30 | |
| jeblair | clarkb: mordred also wanted to do some init scripts stuff. and in zuulv3, at least, i'd like to implement command sockets for all the commands, not just zuul-executor, so we can do 'zuul-foo stop'. | 00:30 |
| clarkb | tehre is likely more changes necessary like the ones we had for the builder that will need to happen first | 00:31 |
| clarkb | and those will be necessary for command sockets or signal handling | 00:31 |
| jeblair | ya. it's time for a refresh for sure. | 00:31 |
| clarkb | building down to 700 or so with 250 or so used | 00:33 |
| clarkb | seems to be properly flipping things over | 00:34 |
| mordred | clarkb: let's sync on init scripts when we get to it | 00:34 |
| clarkb | mordred: ok | 00:34 |
| clarkb | mordred: are you going full systemd? | 00:34 |
| jeblair | we're suddenly getting a lot of ssh errors | 00:34 |
| jeblair | i wonder if those could be manifestations of thread starvation somehow | 00:35 |
| jeblair | mordred, clarkb: let's all sync on that, since there are currently problems in zuul start/stop that can't be solved by init scripts :) | 00:36 |
| mordred | jeblair, clarkb: yes - agree- I think we need to fix the start/stop problems that can't be done with init scripts and then we need to make sure the init scripts work too | 00:37 |
| jeblair | ++ | 00:37 |
| jeblair | i'm really looking forward to multiple launchers in v3 | 00:37 |
| clarkb | we have almost none building now, ~450 used and another 200 or so ready | 00:39 |
| *** makowals has quit IRC | 00:39 | |
| clarkb | so we converted at least ~60% of the building nodes? | 00:39 |
| clarkb | probably not too bad considering we just asked all the clouds to build all the nodes | 00:40 |
| jeblair | based on the "deleting server" calls i'm seeing, that doesn't surprise me | 00:40 |
| clarkb | jeblair: mordred another quick fix may be to separate the geard server from zuul proper? | 00:40 |
| jeblair | the main loop is speeeding up. i think it may be about ready to cycle. | 00:40 |
| jeblair | clarkb: yeah, we should probably do that, in both branches. | 00:43 |
| jeblair | nodepool did another demand calc without error. i think we're good now. | 00:43 |
| clarkb | cool | 00:43 |
| jeblair | i'm going to work on dinner now | 00:43 |
| jeblair | clarkb, mordred: thanks for your help | 00:43 |
| clarkb | me too and thank you | 00:44 |
| *** thorst has quit IRC | 00:45 | |
| *** oanson has quit IRC | 00:54 | |
| *** priteau has joined #openstack-infra | 00:54 | |
| *** cuongnv has joined #openstack-infra | 00:55 | |
| *** oanson has joined #openstack-infra | 00:56 | |
| *** priteau has quit IRC | 00:59 | |
| *** baoli has quit IRC | 01:08 | |
| *** zzzeek_ has quit IRC | 01:09 | |
| *** vhosakot has quit IRC | 01:10 | |
| *** zzzeek_ has joined #openstack-infra | 01:11 | |
| *** bobh has quit IRC | 01:12 | |
| *** sdague has quit IRC | 01:17 | |
| *** camunoz has quit IRC | 01:18 | |
| *** makowals has joined #openstack-infra | 01:18 | |
| *** Apoorva has joined #openstack-infra | 01:18 | |
| *** annegentle has quit IRC | 01:19 | |
| *** cshastri has joined #openstack-infra | 01:20 | |
| *** thorst has joined #openstack-infra | 01:28 | |
| *** thorst has quit IRC | 01:28 | |
| *** vhosakot has joined #openstack-infra | 01:28 | |
| *** baoli has joined #openstack-infra | 01:32 | |
| *** zzzeek_ has quit IRC | 01:33 | |
| *** zzzeek_ has joined #openstack-infra | 01:34 | |
| *** zzzeek_ has quit IRC | 01:36 | |
| *** annegentle has joined #openstack-infra | 01:38 | |
| *** annegentle has quit IRC | 01:39 | |
| *** LindaWang has joined #openstack-infra | 01:46 | |
| *** zzzeek_ has joined #openstack-infra | 01:47 | |
| *** Apoorva has quit IRC | 01:49 | |
| *** tonytan_brb has joined #openstack-infra | 01:49 | |
| *** Apoorva has joined #openstack-infra | 01:49 | |
| *** yamahata has quit IRC | 01:50 | |
| *** tonytan4ever has quit IRC | 01:52 | |
| *** iyamahat has quit IRC | 01:52 | |
| *** Apoorva has quit IRC | 01:54 | |
| *** hongbin has joined #openstack-infra | 01:56 | |
| *** mat128 has joined #openstack-infra | 01:57 | |
| *** Sukhdev has quit IRC | 01:57 | |
| *** liusheng has quit IRC | 01:59 | |
| *** liusheng has joined #openstack-infra | 02:00 | |
| *** zhurong has joined #openstack-infra | 02:03 | |
| *** gcb has joined #openstack-infra | 02:03 | |
| *** ramishra has joined #openstack-infra | 02:07 | |
| *** ramishra has quit IRC | 02:11 | |
| *** ramishra has joined #openstack-infra | 02:11 | |
| *** bobh has joined #openstack-infra | 02:26 | |
| *** sshnaidm has quit IRC | 02:28 | |
| *** bobh has quit IRC | 02:29 | |
| *** hongbin has quit IRC | 02:30 | |
| *** hongbin_ has joined #openstack-infra | 02:30 | |
| *** hongbin_ has quit IRC | 02:31 | |
| *** hongbin has joined #openstack-infra | 02:32 | |
| *** makowals has quit IRC | 02:36 | |
| *** makowals has joined #openstack-infra | 02:39 | |
| *** sshnaidm has joined #openstack-infra | 02:41 | |
| *** ramishra has quit IRC | 02:46 | |
| *** ramishra has joined #openstack-infra | 02:48 | |
| *** makowals has quit IRC | 02:54 | |
| *** priteau has joined #openstack-infra | 02:55 | |
| *** tonytan_brb has quit IRC | 02:58 | |
| *** yamamoto has joined #openstack-infra | 02:58 | |
| *** priteau has quit IRC | 03:00 | |
| *** aeng has quit IRC | 03:03 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: executor: add support for custom ansible_port https://review.openstack.org/468710 | 03:04 |
| *** aeng has joined #openstack-infra | 03:04 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/storyboard master: Better handle pika connection errors https://review.openstack.org/482391 | 03:05 |
| *** namnh has joined #openstack-infra | 03:07 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Add TenantProjectConfig object https://review.openstack.org/479073 | 03:07 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Permit config shadowing https://review.openstack.org/479084 | 03:11 |
| *** rlandy|bbl is now known as rlandy | 03:15 | |
| *** rlandy has quit IRC | 03:16 | |
| *** tjones has joined #openstack-infra | 03:18 | |
| *** tjones has left #openstack-infra | 03:18 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Fix race in test_queue_rate_limiting_dependent https://review.openstack.org/481824 | 03:18 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/storyboard master: Add show attribute to the Task model https://review.openstack.org/482393 | 03:21 |
| *** gyee has quit IRC | 03:23 | |
| *** makowals has joined #openstack-infra | 03:24 | |
| *** tonytan4ever has joined #openstack-infra | 03:28 | |
| *** thorst has joined #openstack-infra | 03:29 | |
| *** Kevin_Zheng has joined #openstack-infra | 03:30 | |
| openstackgerrit | Amrith Kumar proposed openstack/diskimage-builder master: Allow users to specify partition type in the MBR PTE https://review.openstack.org/482396 | 03:32 |
| *** vhosakot has quit IRC | 03:33 | |
| *** thorst has quit IRC | 03:34 | |
| *** links has joined #openstack-infra | 03:41 | |
| *** winggundamth has joined #openstack-infra | 03:42 | |
| *** sshnaidm has quit IRC | 03:53 | |
| *** jamesmcarthur has joined #openstack-infra | 03:54 | |
| *** makowals has quit IRC | 03:56 | |
| *** tonytan4ever has quit IRC | 03:57 | |
| *** jamesmcarthur has quit IRC | 03:58 | |
| *** psachin has joined #openstack-infra | 04:00 | |
| *** dave-mccowan has quit IRC | 04:00 | |
| *** makowals has joined #openstack-infra | 04:04 | |
| *** isaacb has joined #openstack-infra | 04:08 | |
| *** sshnaidm has joined #openstack-infra | 04:09 | |
| *** isaacb has quit IRC | 04:10 | |
| *** baoli has quit IRC | 04:10 | |
| *** 07IAAFLCM has joined #openstack-infra | 04:13 | |
| *** 7GHABGTVE has joined #openstack-infra | 04:13 | |
| *** 7GHABGTVE has quit IRC | 04:13 | |
| *** aeng has quit IRC | 04:13 | |
| *** adreznec has quit IRC | 04:13 | |
| *** adreznec has joined #openstack-infra | 04:14 | |
| *** hongbin has quit IRC | 04:16 | |
| *** makowals has quit IRC | 04:18 | |
| *** mdrabe has quit IRC | 04:18 | |
| *** makowals has joined #openstack-infra | 04:25 | |
| *** mwarad has quit IRC | 04:26 | |
| *** jrist has quit IRC | 04:33 | |
| *** slaweq has joined #openstack-infra | 04:34 | |
| *** ykarel has joined #openstack-infra | 04:39 | |
| *** slaweq has quit IRC | 04:39 | |
| *** makowals has quit IRC | 04:41 | |
| *** Julien-zte has quit IRC | 04:41 | |
| *** Julien-zte has joined #openstack-infra | 04:42 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/storyboard master: Add show attribute to the Task model https://review.openstack.org/482393 | 04:44 |
| *** priteau has joined #openstack-infra | 04:56 | |
| *** makowals has joined #openstack-infra | 05:00 | |
| *** priteau has quit IRC | 05:00 | |
| *** sree has joined #openstack-infra | 05:09 | |
| openstackgerrit | Merged openstack-infra/irc-meetings master: Add OpenStack Jenkins Job Builder team meeting https://review.openstack.org/481664 | 05:14 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul feature/zuulv3: Add zuul.d configuration split documentation https://review.openstack.org/482405 | 05:20 |
| AJaeger | yolanda: once you review, please include my final round of mitaka-eol changes https://review.openstack.org/#/q/status:open+project:openstack-infra/project-config+branch:master+topic:mitaka-eol | 05:20 |
| *** pratik has joined #openstack-infra | 05:20 | |
| *** d0ugal has joined #openstack-infra | 05:22 | |
| *** zaneb has quit IRC | 05:29 | |
| *** zaneb has joined #openstack-infra | 05:29 | |
| *** thorst has joined #openstack-infra | 05:30 | |
| *** thorst has quit IRC | 05:34 | |
| *** amotoki_away is now known as amotoki | 05:35 | |
| *** slaweq has joined #openstack-infra | 05:35 | |
| *** eranrom has quit IRC | 05:37 | |
| *** armax has quit IRC | 05:39 | |
| *** ianychoi has joined #openstack-infra | 05:40 | |
| *** slaweq has quit IRC | 05:41 | |
| *** rcernin has joined #openstack-infra | 05:41 | |
| *** dhajare has joined #openstack-infra | 05:43 | |
| *** katyafervent_awa has quit IRC | 05:44 | |
| *** katyafervent_awa has joined #openstack-infra | 05:44 | |
| *** mwarad has joined #openstack-infra | 05:47 | |
| *** d0ugal has quit IRC | 05:52 | |
| *** mat128 has quit IRC | 05:54 | |
| *** nicolasbock has joined #openstack-infra | 05:55 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul feature/zuulv3: Prevent loading zuul.(yaml|d) in untrusted-projects https://review.openstack.org/482411 | 06:02 |
| *** pcaruana has joined #openstack-infra | 06:04 | |
| *** jamesmcarthur has joined #openstack-infra | 06:10 | |
| *** dingyichen has quit IRC | 06:13 | |
| *** dhajare has quit IRC | 06:13 | |
| *** jamesmcarthur has quit IRC | 06:14 | |
| *** claudiub has joined #openstack-infra | 06:16 | |
| *** cshen has joined #openstack-infra | 06:23 | |
| openstackgerrit | Merged openstack-infra/project-config master: jenkins: jobs: ansible-role-jobs: Always enable the EPEL repo https://review.openstack.org/478095 | 06:25 |
| *** gtrxcb has joined #openstack-infra | 06:25 | |
| *** 07IAAFLCM has quit IRC | 06:27 | |
| *** slaweq has joined #openstack-infra | 06:29 | |
| *** dhajare has joined #openstack-infra | 06:31 | |
| *** slaweq_ has joined #openstack-infra | 06:36 | |
| *** zhurong has quit IRC | 06:40 | |
| *** slaweq_ has quit IRC | 06:41 | |
| *** zhurong has joined #openstack-infra | 06:45 | |
| *** cuongnv has quit IRC | 06:52 | |
| *** namnh has quit IRC | 06:52 | |
| openstackgerrit | Jamie Lennox proposed openstack-infra/zuul feature/zuulv3: Cleanup on ssh-agent failure https://review.openstack.org/481344 | 06:54 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul feature/zuulv3: Prevent loading zuul.(yaml|d) in untrusted-projects https://review.openstack.org/482411 | 06:55 |
| *** priteau has joined #openstack-infra | 06:57 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/infra-specs master: Move zuul_split to completed https://review.openstack.org/482422 | 07:00 |
| *** namnh has joined #openstack-infra | 07:01 | |
| *** priteau has quit IRC | 07:01 | |
| *** cuongnv has joined #openstack-infra | 07:01 | |
| *** tesseract has joined #openstack-infra | 07:02 | |
| *** florianf has joined #openstack-infra | 07:05 | |
| *** shardy has joined #openstack-infra | 07:11 | |
| *** dfflanders has quit IRC | 07:18 | |
| *** jpena|off is now known as jpena | 07:19 | |
| openstackgerrit | Merged openstack-infra/project-config master: Create tempest-tripleo project https://review.openstack.org/481158 | 07:24 |
| AJaeger | yolanda: could you include my final round of mitaka-eol changes https://review.openstack.org/#/q/status:open+project:openstack-infra/project-config+branch:master+topic:mitaka-eol , please? | 07:24 |
| yolanda | hi AJaeger , suer | 07:25 |
| yolanda | sure | 07:25 |
| *** zhurong has quit IRC | 07:30 | |
| *** mriedem has quit IRC | 07:30 | |
| *** thorst has joined #openstack-infra | 07:31 | |
| AJaeger | thanks | 07:32 |
| *** openstackgerrit has quit IRC | 07:33 | |
| *** cshastri has quit IRC | 07:35 | |
| *** openstackgerrit has joined #openstack-infra | 07:35 | |
| openstackgerrit | Merged openstack-infra/project-config master: New project for Castellan UI plugin https://review.openstack.org/475835 | 07:35 |
| openstackgerrit | Merged openstack-infra/project-config master: Move old ovb-updates job to use quickstart https://review.openstack.org/478924 | 07:35 |
| openstackgerrit | Merged openstack-infra/project-config master: Update branches for gerritbot notifications for Packstack https://review.openstack.org/481354 | 07:35 |
| openstackgerrit | Merged openstack-infra/project-config master: Do some openstack-ansible job housekeeping https://review.openstack.org/481568 | 07:36 |
| *** thorst has quit IRC | 07:36 | |
| *** zaneb has quit IRC | 07:36 | |
| *** slaweq_ has joined #openstack-infra | 07:37 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add non-voting py35 jobs for ARA https://review.openstack.org/481823 | 07:38 |
| openstackgerrit | Merged openstack-infra/project-config master: horizon: Run py27dj18 job for master https://review.openstack.org/479019 | 07:40 |
| openstackgerrit | Merged openstack-infra/project-config master: Build openSUSE 42.3 nodepool images https://review.openstack.org/481752 | 07:40 |
| openstackgerrit | Rabi Mishra proposed openstack-infra/project-config master: Fix check and gate jobs for python-heatclient https://review.openstack.org/482434 | 07:41 |
| *** slaweq_ has quit IRC | 07:42 | |
| *** aarefiev_pto is now known as aarefiev | 07:44 | |
| *** namnh has quit IRC | 07:45 | |
| *** cuongnv has quit IRC | 07:45 | |
| *** gtrxcb has quit IRC | 07:46 | |
| *** markvoelker_ has quit IRC | 07:47 | |
| *** jpich has joined #openstack-infra | 07:47 | |
| *** ralonsoh has joined #openstack-infra | 07:48 | |
| *** dhajare has quit IRC | 07:48 | |
| *** zaneb has joined #openstack-infra | 07:48 | |
| *** dizquierdo has joined #openstack-infra | 07:49 | |
| *** egonzalez has joined #openstack-infra | 07:57 | |
| *** zzzeek has quit IRC | 08:00 | |
| *** zzzeek has joined #openstack-infra | 08:00 | |
| *** rossella_s has joined #openstack-infra | 08:01 | |
| *** namnh has joined #openstack-infra | 08:01 | |
| *** cuongnv has joined #openstack-infra | 08:01 | |
| *** dhajare has joined #openstack-infra | 08:02 | |
| *** openstackgerrit has quit IRC | 08:03 | |
| *** openstackgerrit has joined #openstack-infra | 08:07 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Add symlink test for resolv.conf restore https://review.openstack.org/480805 | 08:07 |
| *** shu-mutou-AWAY is now known as shu-mutou | 08:08 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Check ret for None in zuul_run_command https://review.openstack.org/482442 | 08:10 |
| openstackgerrit | Merged openstack/diskimage-builder master: pip-and-virtualenv: Install python3 on openSUSE https://review.openstack.org/478648 | 08:11 |
| *** raissa has joined #openstack-infra | 08:12 | |
| *** jascott1 has quit IRC | 08:13 | |
| *** jascott1 has joined #openstack-infra | 08:14 | |
| *** jascott1 has quit IRC | 08:18 | |
| *** rossella_s has quit IRC | 08:18 | |
| *** e0ne has joined #openstack-infra | 08:19 | |
| *** rossella_s has joined #openstack-infra | 08:20 | |
| *** ccamacho has joined #openstack-infra | 08:20 | |
| *** gildub has quit IRC | 08:21 | |
| *** isaacb has joined #openstack-infra | 08:23 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Check ret for None in zuul_run_command https://review.openstack.org/482442 | 08:25 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Check ret for None in zuul_run_command https://review.openstack.org/482442 | 08:27 |
| *** zhurong has joined #openstack-infra | 08:27 | |
| openstackgerrit | Rabi Mishra proposed openstack-infra/project-config master: Use python-jobs template for python-heatclient https://review.openstack.org/482434 | 08:27 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Check ret for None in zuul_run_command https://review.openstack.org/482442 | 08:28 |
| *** sambetts|afk is now known as sambetts | 08:30 | |
| *** efoley has joined #openstack-infra | 08:31 | |
| openstackgerrit | Alfredo Moralejo proposed openstack-infra/tripleo-ci master: Update promotion jobs with new ovb-1ctlr_1comp_1ceph-featureset024 https://review.openstack.org/482447 | 08:32 |
| *** lucas-afk is now known as lucasagomes | 08:32 | |
| openstackgerrit | Lucas Alvares Gomes proposed openstack-infra/project-config master: Add networking-ovn job to ovsdbapp gate https://review.openstack.org/481650 | 08:35 |
| *** dtantsur|afk is now known as dtantsur | 08:35 | |
| *** priteau has joined #openstack-infra | 08:36 | |
| *** slaweq_ has joined #openstack-infra | 08:38 | |
| *** winggundamth has quit IRC | 08:39 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: Update new job name of previous updates job https://review.openstack.org/482449 | 08:41 |
| *** winggundamth has joined #openstack-infra | 08:42 | |
| *** sree has quit IRC | 08:43 | |
| *** slaweq_ has quit IRC | 08:43 | |
| openstackgerrit | yolanda.robla proposed openstack/diskimage-builder master: [WIP] LVM support for dib-block-device https://review.openstack.org/472065 | 08:44 |
| openstackgerrit | Anton Arefiev proposed openstack-infra/project-config master: Skip non-doc jobs for doc-only python-ironic-inspector-client changes https://review.openstack.org/482451 | 08:45 |
| *** sree has joined #openstack-infra | 08:46 | |
| *** vtapia has quit IRC | 08:47 | |
| *** dizquierdo has quit IRC | 08:50 | |
| *** abelur_ has joined #openstack-infra | 08:53 | |
| *** tosky has joined #openstack-infra | 08:59 | |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack-infra/project-config master: jenkins: jobs: ansible-role-jobs: Check if yum-config-manager is present https://review.openstack.org/482465 | 09:02 |
| *** kjackal_ has joined #openstack-infra | 09:02 | |
| *** mwarad has quit IRC | 09:09 | |
| Jeffrey4l | infra-root is there any guys could check this? wget https://tarballs.openstack.org/kolla/images/centos-source-registry-ocata.tar.gz Get 403 Forbidden | 09:11 |
| *** pgadiya has joined #openstack-infra | 09:13 | |
| *** baoli has joined #openstack-infra | 09:14 | |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack-infra/project-config master: jenkins: jobs: ansible-role-jobs: Check if yum-config-manager is present https://review.openstack.org/482465 | 09:15 |
| *** electrofelix has joined #openstack-infra | 09:15 | |
| *** winggundamth has quit IRC | 09:16 | |
| *** abelur_ has quit IRC | 09:17 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/project-config master: Simplify keystone OSA upgrade job config https://review.openstack.org/482474 | 09:18 |
| *** baoli has quit IRC | 09:19 | |
| AJaeger | Jeffrey4l: yes, this was disabled yesterday since the images produced massive load. fungi started putting up a caching proxy for these. | 09:20 |
| Jeffrey4l | roger. thanks. | 09:20 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/project-config master: Simplify keystone OSA upgrade job config https://review.openstack.org/482474 | 09:20 |
| *** neiljerram has joined #openstack-infra | 09:20 | |
| AJaeger | Jeffrey4l: it looks like those were not enabled yet (didn't follow discussion completely). Best ask fungi and inc0 once they're available later today (US time) | 09:21 |
| Jeffrey4l | ok. i will. | 09:21 |
| *** jtomasek_ has joined #openstack-infra | 09:23 | |
| *** raissa has quit IRC | 09:25 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/project-config master: Simplify keystone OSA upgrade job config https://review.openstack.org/482474 | 09:25 |
| odyssey4me | AJaeger do you have a moment to advise on ^ based on what I'm trying to achieve | 09:26 |
| *** vtapia has joined #openstack-infra | 09:26 | |
| *** zhurong has quit IRC | 09:27 | |
| *** neiljerram has quit IRC | 09:29 | |
| *** e0ne has quit IRC | 09:31 | |
| *** thorst has joined #openstack-infra | 09:32 | |
| *** e0ne has joined #openstack-infra | 09:34 | |
| openstackgerrit | Zhenguo Niu proposed openstack-infra/project-config master: Export IRONIC_USE_RESOURCE_CLASSES for Mogan https://review.openstack.org/482484 | 09:36 |
| *** links has quit IRC | 09:37 | |
| *** thorst has quit IRC | 09:37 | |
| *** gildub has joined #openstack-infra | 09:38 | |
| *** shu-mutou is now known as shu-mutou-AWAY | 09:38 | |
| *** slaweq_ has joined #openstack-infra | 09:39 | |
| *** links has joined #openstack-infra | 09:39 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/project-config master: Simplify keystone OSA upgrade job config https://review.openstack.org/482474 | 09:41 |
| *** dizquierdo has joined #openstack-infra | 09:43 | |
| *** links has quit IRC | 09:44 | |
| *** slaweq_ has quit IRC | 09:45 | |
| *** jtomasek_ has quit IRC | 09:46 | |
| *** markvoelker has joined #openstack-infra | 09:48 | |
| *** links has joined #openstack-infra | 09:49 | |
| *** d0ugal has joined #openstack-infra | 09:52 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/project-config master: Simplify keystone OSA upgrade job config https://review.openstack.org/482474 | 09:56 |
| AJaeger | odyssey4me: sorry, no. I like the idea;) | 09:57 |
| AJaeger | I can check later tonight... | 09:57 |
| odyssey4me | I think that will do it. :) | 09:57 |
| *** d0ugal has quit IRC | 09:59 | |
| AJaeger | ;) | 09:59 |
| *** iyamahat has joined #openstack-infra | 10:01 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/project-config master: Simplify keystone OSA upgrade job config https://review.openstack.org/482474 | 10:02 |
| *** priteau has quit IRC | 10:03 | |
| *** yamamoto has quit IRC | 10:03 | |
| *** sdague has joined #openstack-infra | 10:05 | |
| *** cuongnv has quit IRC | 10:08 | |
| *** namnh has quit IRC | 10:08 | |
| *** jamesmcarthur has joined #openstack-infra | 10:11 | |
| *** gcb has quit IRC | 10:13 | |
| *** links has quit IRC | 10:13 | |
| *** jamesmcarthur has quit IRC | 10:15 | |
| *** baoli has joined #openstack-infra | 10:16 | |
| *** iyamahat has quit IRC | 10:17 | |
| *** lucasagomes has quit IRC | 10:20 | |
| *** liujiong has quit IRC | 10:20 | |
| *** baoli has quit IRC | 10:20 | |
| *** markvoelker has quit IRC | 10:22 | |
| *** lucasagomes has joined #openstack-infra | 10:23 | |
| *** raissa has joined #openstack-infra | 10:25 | |
| openstackgerrit | Dmitry Burmistrov proposed openstack-infra/jeepyb master: Make use of Gerrit CLI to retrieve group UUID https://review.openstack.org/284843 | 10:29 |
| openstackgerrit | Gael Chamoulaud proposed openstack-infra/tripleo-ci master: WIP: Enable tripleo-validations tests https://review.openstack.org/481080 | 10:30 |
| *** yolanda has quit IRC | 10:32 | |
| *** gildub has quit IRC | 10:33 | |
| *** gouthamr has quit IRC | 10:33 | |
| *** yolanda has joined #openstack-infra | 10:33 | |
| *** yolanda_ has joined #openstack-infra | 10:33 | |
| *** yolanda_ has quit IRC | 10:33 | |
| *** sree has quit IRC | 10:36 | |
| openstackgerrit | Gergely Csatari proposed openstack-dev/cookiecutter master: Update to the Pike document structure https://review.openstack.org/482504 | 10:37 |
| *** florianf has quit IRC | 10:38 | |
| *** katkapilatova has joined #openstack-infra | 10:40 | |
| *** slaweq_ has joined #openstack-infra | 10:41 | |
| *** gouthamr has joined #openstack-infra | 10:41 | |
| *** slaweq_ has quit IRC | 10:46 | |
| *** slaweq_ has joined #openstack-infra | 10:48 | |
| *** slaweq has quit IRC | 10:48 | |
| *** wolverineav has joined #openstack-infra | 10:50 | |
| *** thorst has joined #openstack-infra | 10:50 | |
| *** florianf has joined #openstack-infra | 10:53 | |
| *** slaweq_ has quit IRC | 10:55 | |
| *** yamamoto has joined #openstack-infra | 11:04 | |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci master: WIP: containers periodic test https://review.openstack.org/475747 | 11:05 |
| *** gildub has joined #openstack-infra | 11:07 | |
| *** slaweq has joined #openstack-infra | 11:08 | |
| *** dave-mccowan has joined #openstack-infra | 11:09 | |
| *** yamamoto has quit IRC | 11:11 | |
| *** Julien-zte has quit IRC | 11:12 | |
| *** Julien-zte has joined #openstack-infra | 11:13 | |
| *** baoli has joined #openstack-infra | 11:16 | |
| *** markvoelker has joined #openstack-infra | 11:20 | |
| *** baoli has quit IRC | 11:20 | |
| openstackgerrit | Sorin Sbârnea proposed openstack-infra/shade master: Added useful links to README https://review.openstack.org/482521 | 11:23 |
| pratik | Hi openstack-infra team, can anyone please review https://review.openstack.org/#/c/475300/? | 11:28 |
| *** yamamoto has joined #openstack-infra | 11:30 | |
| *** pratik has quit IRC | 11:32 | |
| chandankumar | AJaeger: thanks, i learned how to add tox job, replaced tempest plugin sanity job https://review.openstack.org/#/c/474168/ | 11:38 |
| robcresswell | Any chance a project-config core could give a second +2 to https://review.openstack.org/#/c/480612 ? It's a trivial change, but the IRC channel spam is getting frustrating :) | 11:40 |
| *** data has joined #openstack-infra | 11:40 | |
| *** data is now known as Guest68477 | 11:40 | |
| *** slaweq_ has joined #openstack-infra | 11:41 | |
| *** lucasagomes is now known as lucas-hungry | 11:45 | |
| *** jkilpatr has joined #openstack-infra | 11:45 | |
| *** slaweq_ has quit IRC | 11:46 | |
| *** links has joined #openstack-infra | 11:46 | |
| AJaeger | robcresswell: if nobody does in the next 20 hours, tell me and I'll single approve it... | 11:47 |
| *** Guest68477 has quit IRC | 11:48 | |
| robcresswell | Thanks AJaeger :) | 11:49 |
| *** ldnunes has joined #openstack-infra | 11:52 | |
| *** katkapilatova has left #openstack-infra | 11:53 | |
| *** markvoelker has quit IRC | 11:53 | |
| AJaeger | project-config cores, could you review https://review.openstack.org/#/c/474168/ and https://review.openstack.org/#/q/status:open+project:openstack-infra/project-config+branch:master+topic:mitaka-eol , please? | 11:53 |
| *** kjackal_ has quit IRC | 11:57 | |
| *** links has quit IRC | 11:59 | |
| *** raissa_ has joined #openstack-infra | 12:00 | |
| *** raissa has quit IRC | 12:00 | |
| *** felipemonteiro_ has joined #openstack-infra | 12:08 | |
| *** felipemonteiro__ has joined #openstack-infra | 12:09 | |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci master: Add settings for ovb in rdocloud https://review.openstack.org/480950 | 12:12 |
| *** felipemonteiro_ has quit IRC | 12:12 | |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci master: WIP: containers periodic test https://review.openstack.org/475747 | 12:14 |
| *** mwarad has joined #openstack-infra | 12:14 | |
| *** jpena is now known as jpena|lunch | 12:16 | |
| openstackgerrit | Akihiro Motoki proposed openstack-dev/hacking master: Turn on warning-is-error in doc build https://review.openstack.org/482535 | 12:19 |
| openstackgerrit | Akihiro Motoki proposed openstack-dev/hacking master: Switch from oslosphinx to openstackdocstheme https://review.openstack.org/482536 | 12:19 |
| *** markvoelker has joined #openstack-infra | 12:19 | |
| openstackgerrit | Akihiro Motoki proposed openstack-dev/hacking master: rearrange existing documentation to fit the new standard layout https://review.openstack.org/482537 | 12:19 |
| openstackgerrit | Akihiro Motoki proposed openstack-dev/hacking master: Update the doc URL in the documents https://review.openstack.org/482538 | 12:19 |
| openstackgerrit | Arx Cruz proposed openstack-infra/tripleo-ci master: Fix stackviz collect data https://review.openstack.org/474559 | 12:23 |
| *** kjackal_ has joined #openstack-infra | 12:24 | |
| *** rlandy has joined #openstack-infra | 12:29 | |
| *** trown|outtypewww is now known as trown | 12:30 | |
| *** mat128 has joined #openstack-infra | 12:32 | |
| openstackgerrit | Akihiro Motoki proposed openstack-infra/project-config master: build-releasenotes: Add language index to all files https://review.openstack.org/481850 | 12:36 |
| *** yamamoto has quit IRC | 12:38 | |
| openstackgerrit | Merged openstack-infra/project-config master: gerritbot: Do not notify horizon plugin changes to #-horizon https://review.openstack.org/480612 | 12:42 |
| robcresswell | Thanks for merging that^^ :) | 12:42 |
| *** slaweq_ has joined #openstack-infra | 12:42 | |
| fungi | robcresswell: it was an easy review--thanks for the solid explanation! | 12:44 |
| robcresswell | it was amotoki who put up the patch, so he deserves the thanks there. I was just nagging about review :) | 12:45 |
| *** rossella_s has quit IRC | 12:46 | |
| *** gordc has joined #openstack-infra | 12:46 | |
| *** slaweq_ has quit IRC | 12:47 | |
| *** pgadiya has quit IRC | 12:47 | |
| *** openstackgerrit has quit IRC | 12:47 | |
| *** sree has joined #openstack-infra | 12:48 | |
| *** mdrabe has joined #openstack-infra | 12:54 | |
| *** dizquierdo has quit IRC | 12:56 | |
| *** LindaWang has quit IRC | 12:56 | |
| *** funzo has quit IRC | 12:58 | |
| *** funzo has joined #openstack-infra | 12:59 | |
| *** esberglu has joined #openstack-infra | 13:00 | |
| *** burgerk has joined #openstack-infra | 13:01 | |
| *** lucas-hungry is now known as lucasagomes | 13:02 | |
| *** rhallisey has joined #openstack-infra | 13:02 | |
| *** rossella_s has joined #openstack-infra | 13:02 | |
| *** Goneri has joined #openstack-infra | 13:03 | |
| fungi | oh, indeed so it was. thanks amotoki! | 13:04 |
| *** pradk has quit IRC | 13:04 | |
| *** openstackgerrit has joined #openstack-infra | 13:07 | |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci master: WIP: containers periodic test https://review.openstack.org/475747 | 13:07 |
| *** rossella_s has quit IRC | 13:11 | |
| *** LindaWang has joined #openstack-infra | 13:11 | |
| *** dprince has joined #openstack-infra | 13:12 | |
| *** baoli has joined #openstack-infra | 13:13 | |
| *** dhajare has quit IRC | 13:15 | |
| openstackgerrit | Merged openstack-infra/shade master: Added useful links to README https://review.openstack.org/482521 | 13:15 |
| *** ykarel has quit IRC | 13:16 | |
| *** jascott1 has joined #openstack-infra | 13:16 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul feature/zuulv3: Handle aborted websocket client connection cleanly https://review.openstack.org/482560 | 13:19 |
| *** jascott1 has quit IRC | 13:21 | |
| openstackgerrit | Jiri Stransky proposed openstack-infra/project-config master: TripleO: make container upgrade job voting https://review.openstack.org/479198 | 13:22 |
| *** rossella_s has joined #openstack-infra | 13:23 | |
| *** yamamoto has joined #openstack-infra | 13:25 | |
| *** baoli has quit IRC | 13:26 | |
| *** baoli has joined #openstack-infra | 13:27 | |
| *** jpena|lunch is now known as jpena | 13:27 | |
| pabelanger | morning | 13:27 |
| pabelanger | I've given up on backscroll for the last 3 weeks :) | 13:27 |
| *** bobh has joined #openstack-infra | 13:28 | |
| AJaeger | welcome back, pabelanger ! How are you? | 13:29 |
| *** jamesdenton has quit IRC | 13:29 | |
| *** alexchadin has joined #openstack-infra | 13:29 | |
| *** jamesdenton has joined #openstack-infra | 13:31 | |
| *** askb has quit IRC | 13:32 | |
| *** gcb has joined #openstack-infra | 13:32 | |
| pabelanger | AJaeger: good thanks. Family loved traveling in UK and france | 13:33 |
| *** dhajare has joined #openstack-infra | 13:39 | |
| *** slaweq_ has joined #openstack-infra | 13:43 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Add fedora-26 support for nodepool dsvm jobs https://review.openstack.org/482566 | 13:44 |
| openstackgerrit | Lee Yarwood proposed openstack-infra/shade master: router: Ignore L3 HA ports when listing interfaces https://review.openstack.org/482153 | 13:45 |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool feature/zuulv3: Add support to test fedora-26 https://review.openstack.org/482568 | 13:45 |
| *** rossella_s has quit IRC | 13:46 | |
| *** burgerk has quit IRC | 13:47 | |
| openstackgerrit | Paul Belanger proposed openstack/diskimage-builder master: Bump fedora-minimal DIB_RELEASE 26 https://review.openstack.org/482570 | 13:48 |
| *** slaweq_ has quit IRC | 13:48 | |
| *** eharney has joined #openstack-infra | 13:48 | |
| *** mat128 has quit IRC | 13:49 | |
| *** mat128 has joined #openstack-infra | 13:50 | |
| *** baoli has quit IRC | 13:52 | |
| *** ykarel has joined #openstack-infra | 13:52 | |
| *** baoli has joined #openstack-infra | 13:53 | |
| *** d0ugal has joined #openstack-infra | 13:54 | |
| *** dklyle has quit IRC | 13:55 | |
| AJaeger | pabelanger: hope you did as well ;) | 13:55 |
| *** david-lyle has joined #openstack-infra | 13:55 | |
| pabelanger | AJaeger: ya, I need vacation from my PTO now :D | 13:56 |
| AJaeger | On your own? ;) | 13:58 |
| *** jistr is now known as jistr|call | 13:58 | |
| AJaeger | pabelanger, mordred, fungi, clarkb : yolanda was happy with my final mitaka-eol changes, if any of you can review them as well, it would be great: https://review.openstack.org/#/q/status:open+project:openstack-infra/project-config+branch:master+topic:mitaka-eol | 13:58 |
| AJaeger | pabelanger: we retired most of mitaka - and I did a good "summer" cleanup of jobs. Now the log of the layout job has shrunk from 18MB to 14MB due to the many removed jobs... | 13:59 |
| pabelanger | AJaeger: maybe :) | 14:00 |
| pabelanger | AJaeger: execellent | 14:00 |
| *** d0ugal has quit IRC | 14:00 | |
| *** rossella_s has joined #openstack-infra | 14:01 | |
| *** hongbin has joined #openstack-infra | 14:03 | |
| *** baoli has quit IRC | 14:03 | |
| *** baoli has joined #openstack-infra | 14:04 | |
| AJaeger | pabelanger: thanks for reviewing | 14:07 |
| *** rbrndt has joined #openstack-infra | 14:08 | |
| pabelanger | np | 14:08 |
| *** gildub has quit IRC | 14:09 | |
| *** ihrachys has joined #openstack-infra | 14:10 | |
| *** baoli has quit IRC | 14:10 | |
| fungi | pabelanger: you may be interested in this stack for increasing capacity of our per-provider caching proxy implementation: https://review.openstack.org/482236 | 14:14 |
| *** baoli has joined #openstack-infra | 14:14 | |
| fungi | i'd also be thrilled if someone wants to volunteer for rebuilding the mirror instances in infra-cloud if/when 482236 is approved | 14:14 |
| fungi | i've already done the legwork to add cinder volumes to all the others | 14:15 |
| openstackgerrit | Merged openstack-infra/project-config master: Cleanup templates https://review.openstack.org/481804 | 14:15 |
| openstackgerrit | Merged openstack-infra/project-config master: Add back trusty jobs for pbr https://review.openstack.org/481953 | 14:15 |
| openstackgerrit | Merged openstack-infra/project-config master: Remove unused puppet syntax jobs https://review.openstack.org/481956 | 14:15 |
| openstackgerrit | Merged openstack-infra/project-config master: mitaka-eol: Remove py34 from regexes https://review.openstack.org/481918 | 14:15 |
| *** pradk has joined #openstack-infra | 14:16 | |
| openstackgerrit | Merged openstack-infra/project-config master: Remove trusty docs publish job https://review.openstack.org/481832 | 14:16 |
| openstackgerrit | Merged openstack-infra/project-config master: Do not define trusty localrepo-upload jobs https://review.openstack.org/481952 | 14:16 |
| *** LindaWang1 has joined #openstack-infra | 14:17 | |
| rlandy | are there any docs for people providing nodepool host clouds? we like to compare any configuration | 14:17 |
| ykarel | AJaeger, hi | 14:18 |
| fungi | rlandy: you mean https://docs.openstack.org/infra/system-config/contribute-cloud.html ? | 14:18 |
| *** camunoz has joined #openstack-infra | 14:18 | |
| fungi | rlandy: or something else? | 14:18 |
| ykarel | AJaeger, where to find ocata docs(URL?) for any project | 14:18 |
| *** zzzeek_ has quit IRC | 14:18 | |
| ykarel | For example: https://docs.openstack.org/magnum/latest/ it's magnum latest, what's the url for ocata or newton | 14:19 |
| AJaeger | ykarel: let's discuss on #openstack-doc such topics, please | 14:20 |
| ykarel | AJaeger, Ok | 14:20 |
| rlandy | fungi: we started running multinode jobs on rdocloud (through software factory) and are facing issue with the overcloud node connecting back to the undercloud. The admins suggested we adjust the mtus. I was wondering if I could see what configuration upstream had done - if any - so we could match that | 14:20 |
| *** LindaWang has quit IRC | 14:20 | |
| *** LindaWang1 is now known as LindaWang | 14:20 | |
| *** zzzeek_ has joined #openstack-infra | 14:21 | |
| openstackgerrit | Merged openstack-infra/project-config master: Remove unused ansible job definitions https://review.openstack.org/481960 | 14:22 |
| openstackgerrit | Merged openstack-infra/project-config master: Remove unused neutron templates https://review.openstack.org/481829 | 14:22 |
| openstackgerrit | Merged openstack-infra/project-config master: Fix python27 regexes https://review.openstack.org/481919 | 14:22 |
| openstackgerrit | Merged openstack-infra/project-config master: Remove mitaka from gerritbot https://review.openstack.org/481834 | 14:22 |
| openstackgerrit | Merged openstack-infra/project-config master: mitaka-eol: Remove mitaka for openstack-ansible https://review.openstack.org/481813 | 14:22 |
| fungi | rlandy: oh, if it's not documented in http://git.openstack.org/cgit/openstack-infra/tripleo-ci/tree/docs/TripleO-ci.rst then i'm not sure. my understanding of the tripleo-ci architecture is tentative at best | 14:22 |
| openstackgerrit | Merged openstack-infra/project-config master: mitaka-eol: Further cleanup https://review.openstack.org/481814 | 14:22 |
| openstackgerrit | Merged openstack-infra/project-config master: mitaka-eol Cleanup grenade https://review.openstack.org/481812 | 14:23 |
| fungi | rlandy: the tripleo test cloud isn't really built or maintained by the infra team, so the operators responsible for it likely have their own notes | 14:23 |
| *** yamamoto has quit IRC | 14:24 | |
| rlandy | fungi: ok - thanks .. will check with people running on rh1 | 14:25 |
| *** d0ugal has joined #openstack-infra | 14:25 | |
| pabelanger | fungi: oh, nice. looking | 14:25 |
| fungi | pabelanger: that stack of changes is ultimately in service of being able to add a caching proxy for tarballs.o.o so that we stop having ci jobs eat up all its available bandwidth and knock it offline for long periods of time | 14:26 |
| *** spzala has joined #openstack-infra | 14:27 | |
| pabelanger | fungi: I can handle infra-cloud mirrors today too | 14:27 |
| *** xyang1 has joined #openstack-infra | 14:27 | |
| *** gyee has joined #openstack-infra | 14:27 | |
| fungi | pabelanger: oh! if you can take that piece off my plate i'd be ever so grateful | 14:27 |
| fungi | i sort of ended up going down a rathole with tarballs.o.o outage mitigation all yesterday and so other priorities on my to do list are suffering as a result | 14:28 |
| *** d0ugal has quit IRC | 14:31 | |
| *** marst has joined #openstack-infra | 14:31 | |
| *** dizquierdo has joined #openstack-infra | 14:31 | |
| *** alexchadin has quit IRC | 14:32 | |
| *** armax has joined #openstack-infra | 14:33 | |
| pabelanger | fungi: np! | 14:37 |
| pabelanger | Also starting to look into fedora-26 DIBs. Hopefully won't be much work to bring them online now | 14:37 |
| fungi | awesome | 14:37 |
| *** camunoz has quit IRC | 14:37 | |
| *** armax has quit IRC | 14:37 | |
| *** rcernin has quit IRC | 14:38 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Split bindep package install into tasks https://review.openstack.org/482584 | 14:39 |
| *** yamamoto has joined #openstack-infra | 14:40 | |
| *** rcernin has joined #openstack-infra | 14:40 | |
| tosky | fungi: hi, did you mention an outage of tarballs.o.o? I've noticed various failures in some sahara jobs whie accessing artifacts from there; should I wait before rechecks? | 14:41 |
| fungi | tosky: there was an issue for an hour or two yesterday, which should have resolved itself by about 15:30 utc (about 23.5 hours ago) | 14:42 |
| tosky | fungi: uh, thanks; I will try a recheck | 14:43 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Add phase and count to variables passed in to playbooks https://review.openstack.org/481942 | 14:43 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/project-config master: Add gating jobs to quickstart extras repo https://review.openstack.org/482588 | 14:44 |
| sshnaidm | EmilienM, ^^ | 14:44 |
| EmilienM | sshnaidm: thanks, looking | 14:45 |
| *** slaweq_ has joined #openstack-infra | 14:45 | |
| *** armax has joined #openstack-infra | 14:45 | |
| *** jpena is now known as jpena|off | 14:46 | |
| *** xarses has joined #openstack-infra | 14:47 | |
| *** openstackgerrit has quit IRC | 14:48 | |
| *** jtomasek_ has joined #openstack-infra | 14:49 | |
| *** slaweq_ has quit IRC | 14:49 | |
| *** mdrabe has quit IRC | 14:49 | |
| *** openstackgerrit has joined #openstack-infra | 14:50 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Create fedora-26 mirror in openstack-infra https://review.openstack.org/482592 | 14:50 |
| *** camunoz has joined #openstack-infra | 14:50 | |
| pabelanger | fungi: btw: what happen to bluebox cloud? Did we end up losing it? | 14:51 |
| fungi | pabelanger: yes, ibm decided they were done providing it and pulled it down | 14:51 |
| pabelanger | sadness | 14:52 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Create configure-mirrors role https://review.openstack.org/482593 | 14:52 |
| fungi | clouds come and go. i've got another in the discussion pipeline now, maybe two, with much higher capacity than bluebox anyway | 14:52 |
| pabelanger | Ya, that is the great thing. We always seem to be in a good place for replacements | 14:53 |
| *** slaweq has quit IRC | 14:53 | |
| AJaeger | team, I have two changes for infra-manual to switch to openstackdocstheme and fix some problems. they have a lready a +2 from fungi - anybody to +2A, please? https://review.openstack.org/#/c/479182/ https://review.openstack.org/#/c/479181/ | 14:56 |
| fungi | pabelanger: i don't know that it will always be the case, and have to operate on the assumption that it won't, but for now it's a pretty nice situation anyway | 14:56 |
| *** vhosakot has joined #openstack-infra | 14:56 | |
| *** jamesmcarthur has joined #openstack-infra | 14:56 | |
| *** mdrabe has joined #openstack-infra | 14:56 | |
| clarkb | rlandy: fungi but for devstack testing we create a shared layer two between the multinode test nodes using vxlan then assign an mtu appopriate to those interface based on vxlan overhead then configure neutron to respect that in its dhcp server when configuring instances | 14:56 |
| pabelanger | fungi: agree | 14:56 |
| clarkb | rlandy: fungi a lot of that mtu work happened on the neutron side to ensure all of neutrons managed interfaces and dhcp server did the right thing | 14:57 |
| fungi | clarkb: great point, i wasn't picking up on the original statement that their issues were multi-node job related | 14:58 |
| fungi | so yes, it's possible they're hitting some sort of fragmentation issues with the vxlan tunnels between nodes | 14:58 |
| clarkb | fungi: assuming that tripleo jobs use a similar setup. They may also be hitting routing problems which is why we introduced the overlay in the devstack jobs | 14:59 |
| *** isaacb has quit IRC | 14:59 | |
| clarkb | without an overlay your nodes won't necessarily know how to talk to each other over the cloud managed ip addrs | 14:59 |
| clarkb | with an overlay you have to deal with mtus | 14:59 |
| *** mattmceuen has joined #openstack-infra | 15:00 | |
| fungi | ahh, right, they're not using devstack-gate for these i guess | 15:00 |
| mordred | fwiw, I'm going to be picking up the ansible-ify devstack-gate work soon, so maybe even pre-v3 there will be a thing we can point folks to for a "run this on your mutli-node things if you're ont already using devstack-gate" | 15:01 |
| mordred | (I've got about three things on my list before I hit devstack-gate - but I expect to be able to start hacking on that this week) | 15:01 |
| *** iyamahat has joined #openstack-infra | 15:03 | |
| *** Julien-zte has quit IRC | 15:03 | |
| AJaeger | fungi, clarkb : Thanks to reviews by pabelanger and yolanda all the mitaka cleanups are in. Feel free to double check for anything that I missed. | 15:04 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Create configure-mirrors role https://review.openstack.org/482593 | 15:05 |
| *** jamesdenton has quit IRC | 15:05 | |
| rlandy | clarkb: thanks for that extra info. | 15:05 |
| AJaeger | project-config cores, here're a few small cleanups of mine that I'd like to be reviewed https://review.openstack.org/#/c/481806/ https://review.openstack.org/#/c/481892/ https://review.openstack.org/#/c/481951/ https://review.openstack.org/#/c/482117/ (and that's my last ping for these for today;) | 15:06 |
| mordred | AJaeger: looing | 15:07 |
| mordred | AJaeger: oh - also, I'm going to mention this in today' infra meeting- but that's a time when you should normally be offline ... | 15:07 |
| *** jamesdenton has joined #openstack-infra | 15:08 | |
| mordred | AJaeger: we're getting close to the primetime with v3 - so reviewing patches to zuul-jobs isn't a _complete_ waste of your time at this point - although it is still early, so no pressure or anything - mostly wanted to let you know that the jobs are actually starting to take shape for real | 15:09 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Add tox-docs jobs https://review.openstack.org/482596 | 15:09 |
| *** jamesdenton has joined #openstack-infra | 15:09 | |
| rlandy | clarkb: is there any link you could share wrt multinode work? what did you set the mtus at? | 15:10 |
| clarkb | rlandy: its set at host interface mtu - 50 iirc | 15:10 |
| clarkb | rlandy: then we tell neutron what that value is | 15:10 |
| *** MatthewLi has quit IRC | 15:11 | |
| clarkb | https://git.openstack.org/cgit/openstack-infra/devstack-gate/tree/multinode_setup_info.txt | 15:11 |
| rlandy | trown: ^^ | 15:11 |
| *** dklyle has joined #openstack-infra | 15:11 | |
| *** david-lyle has quit IRC | 15:11 | |
| rlandy | clarkb: thank you | 15:11 |
| *** zzzeek_ has quit IRC | 15:12 | |
| openstackgerrit | Merged openstack-infra/infra-manual master: Switch to openstackdocstheme https://review.openstack.org/479181 | 15:12 |
| openstackgerrit | Merged openstack-infra/infra-manual master: Fix Sphinx build https://review.openstack.org/479182 | 15:12 |
| *** baoli has quit IRC | 15:13 | |
| *** jpena|off is now known as jpena | 15:13 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Add openstack-doc-build job for documentation testing https://review.openstack.org/482596 | 15:13 |
| *** dhajare has quit IRC | 15:13 | |
| openstackgerrit | Jose Luis Franco proposed openstack-infra/project-config master: Make centos7-undercloud-upgrades job vote. https://review.openstack.org/480230 | 15:14 |
| *** dklyle is now known as david-lyle | 15:15 | |
| *** psachin has quit IRC | 15:18 | |
| andreas_s | Hi did anybody do a cleanup of the http://tarballs.openstack.org ? | 15:18 |
| *** rcernin has quit IRC | 15:18 | |
| andreas_s | Seems like the trove mysql image got removed http://tarballs.openstack.org/trove/images/ubuntu/mysql.qcow2 | 15:18 |
| *** baoli has joined #openstack-infra | 15:18 | |
| andreas_s | but devstack is still referencing it in stackrc which causes the diskimagebuilder to fail :/ | 15:18 |
| pabelanger | I know there has been work on tarballs.o.o recently cc fungi ^ | 15:19 |
| pabelanger | but not sure about mysql.qcow2 | 15:19 |
| pabelanger | FWIW: I think we cache that file on our DIBs | 15:20 |
| clarkb | pabelanger: ya I think that caching is why dib builds are failing | 15:20 |
| pabelanger | do not sure why devstack would be accessing tarballs.o.o | 15:20 |
| pabelanger | Oh | 15:20 |
| pabelanger | I see what you mean | 15:20 |
| andreas_s | https://github.com/openstack-dev/devstack/blob/master/stackrc#L741 | 15:20 |
| *** jamesmcarthur has quit IRC | 15:20 | |
| pabelanger | andreas_s: ya, that seems wrong | 15:20 |
| AJaeger | mordred: understood about zuul-jobs - will try to catch up slowly there. Thanks | 15:20 |
| AJaeger | andreas_s: there should be a README on tarballs./trove - that was indeed removed | 15:21 |
| AJaeger | andreas_s: http://tarballs.openstack.org/trove/images/README | 15:21 |
| *** sree has quit IRC | 15:21 | |
| andreas_s | the cache-devstack dib element (project-config) seems to parse the stackrc... | 15:22 |
| clarkb | andreas_s: ya it runs a script in devstack that emits a list of images to cache based on the stackrc data | 15:22 |
| *** jistr|call is now known as jistr | 15:22 | |
| pabelanger | ya, it looks like nodepool-builder is failing because of missing trove image | 15:22 |
| *** sree has joined #openstack-infra | 15:22 | |
| fungi | andreas_s: those were removed at amrith's request. i saved them and _can_ put them back temporarily if needed | 15:22 |
| clarkb | fungi: andreas_s pabelanger AJaeger I think we just need to remove that image from devstack on all valid branches | 15:23 |
| *** isaacb has joined #openstack-infra | 15:23 | |
| pabelanger | yup, think so too | 15:23 |
| AJaeger | clarkb: agreed | 15:23 |
| pabelanger | I can work up a patch quickly | 15:23 |
| fungi | note the readme file i added to that directory with the message amrith drafted as well | 15:23 |
| andreas_s | sounds like the best approach | 15:23 |
| AJaeger | bbl | 15:23 |
| clarkb | on the plus side that will make our images slightly smaller :) | 15:23 |
| pabelanger | ++ | 15:23 |
| pabelanger | however, they are down in size recently too | 15:24 |
| *** jtomasek_ has quit IRC | 15:24 | |
| pabelanger | I think 6GB now | 15:24 |
| fungi | indeedy. it also shaved ~5gib of utilization off the tarballs fs | 15:24 |
| *** MatthewLi has joined #openstack-infra | 15:24 | |
| fungi | though i also reclaimed the old mirror.o.o logical volume we haven't used for a couple years and extended the tarballs lv by another 150gib to buy us more room | 15:24 |
| *** annegentle has joined #openstack-infra | 15:24 | |
| clarkb | pabelanger: except for rax where they are vhd and still gigantic. But 6gb qcow is not too bad | 15:25 |
| pabelanger | remote: https://review.openstack.org/482600 Remove mysql.qcow2 from stackrc | 15:26 |
| *** sree has quit IRC | 15:26 | |
| pabelanger | clarkb: Ya, even with vhd our image builds are about 30mins now, last I checked a few weeks ago | 15:29 |
| *** rderose has joined #openstack-infra | 15:29 | |
| clarkb | pabelanger: that will need backports to stable branches roo | 15:30 |
| clarkb | *too | 15:30 |
| pabelanger | clarkb: ya, did via gerritUI | 15:30 |
| *** zzzeek_ has joined #openstack-infra | 15:30 | |
| *** rderose has left #openstack-infra | 15:31 | |
| fungi | devstack reviewers may be picky about making sure backports all share the same change-id as the master branch change | 15:31 |
| fungi | gerrit's backport button ends up creating new change-ids for each of them instead | 15:31 |
| pabelanger | Ah, k. lets see what they say | 15:32 |
| fungi | no, i stand corrected. it seems to preserve the change-id after all | 15:32 |
| fungi | https://review.openstack.org/#/q/I2034e8ebc530704d6e63a231056f92e14a8654e4,n,z | 15:32 |
| *** e0ne has quit IRC | 15:34 | |
| clarkb | once I've got caffeine I'm going to approve change to bump review-dev up to 2.13.9 | 15:34 |
| clarkb | once that is up and running I think we will be ready to set up zuul testing | 15:34 |
| *** jcoufal has joined #openstack-infra | 15:37 | |
| mnaser | i feel like even if this is biased but its probably good to hear thoughts: why pick diskimage-builder over packer (https://www.packer.io/) ? | 15:40 |
| * mnaser is looking for a way to build a set of application specific images | 15:40 | |
| mordred | mnaser: becaus packer specializes existing cloud images | 15:40 |
| mordred | mnaser: and we explicitly don't wnatthat - we want to start with either an image that is not already in a cloud or that we have locally, specialize that, and then upload that same image to more than one cloud | 15:41 |
| mnaser | mordred so arguably one of the benefits of dib is multicloud single image, vs packer is multi image at each cloud | 15:41 |
| mnaser | that's fair, im just trying to decide which one would work better. though i feel being able to write our own elements and just referencing them might make things very reusable | 15:42 |
| mordred | mnaser: yah - also, packer has a hashicorp style DSL - whereas dib uses bash | 15:42 |
| clarkb | * dib uses anything that you can run on linux | 15:42 |
| mordred | ++ | 15:42 |
| mordred | mnaser: so, from my side, the packer files drive me batty - but that's just a personal preference | 15:43 |
| mordred | I also find vagrant COMPLETELY unusable, but I know it works for other people | 15:43 |
| clarkb | you could write perl scripts or lisp or whatever as long as that runtime is available to the build | 15:43 |
| clarkb | dib ends up using a lot of bash because it is available | 15:43 |
| mnaser | that's cool. that might come in useful | 15:43 |
| tosky | mnaser: well, if you look for something that can customize an existing image, you may check libguestfs and its tools as well (like virt-customize) | 15:44 |
| *** ccamacho has left #openstack-infra | 15:44 | |
| *** andreas_s has quit IRC | 15:44 | |
| tosky | with binding in whatever language you want | 15:44 |
| clarkb | mnaser: mordred infra used puppet for a long time until recently | 15:44 |
| clarkb | so ya fairly flexibly | 15:44 |
| mnaser | tosky: we've had demand for app specific images so we want something that can operate within our CI smoothly so we can constantly ship them | 15:45 |
| mnaser | i know dib gives that because infra is building images all the time with it | 15:45 |
| *** slaweq has joined #openstack-infra | 15:45 | |
| tosky | mnaser: nothing I wrote above implies the opposite - but this is flamey topic | 15:46 |
| mnaser | tosky in my experience virt-customize + libguestfs in general needs a lot of plumbing to get working, but i dont discredit it | 15:47 |
| tosky | mnaser: uhm, virt-customize does not require setup | 15:48 |
| *** mhayden has quit IRC | 15:48 | |
| fungi | virt-customize can work via chroot into a normal loopback-mounted filesystem? | 15:48 |
| *** mhayden has joined #openstack-infra | 15:48 | |
| fungi | or do you need a hypervisor/container layer? | 15:49 |
| mnaser | i thought that it used qemu to start it in that specific image | 15:49 |
| *** trown is now known as trown|lunch | 15:49 | |
| mnaser | so without qemu, no luck, so that means we cant do CI and use it in our VMs | 15:49 |
| *** slaweq has quit IRC | 15:50 | |
| tosky | it works against an existing image, not a loopback-mounted filesystem | 15:50 |
| *** tjones has joined #openstack-infra | 15:51 | |
| tosky | I don't see how it brings it to not being able to use it in a CI | 15:51 |
| mnaser | tosky we have jobs that build images. the "builders" are one-time use slaves which are VMs on our cloud | 15:52 |
| tosky | mnaser: yes, we have in an internal CI as well - and virt-customize is executed in some jobs | 15:52 |
| mnaser | we dont allow nested kvm on our cloud which means we cant use qemu (or if we can, it will be pretty slow) | 15:52 |
| mnaser | if your builders are physical hardware you're good, but virtual will be problematic | 15:53 |
| mnaser | anyways, probably a discussion to have elsewhere rather than -infra :-p | 15:53 |
| tosky | sure, I just pointed out another solution (and our slaves are virtual too) | 15:54 |
| tosky | to be precise, without nested virtualization, you can't use kvm-accelerated qemu, but you can use qemu | 15:54 |
| openstackgerrit | Sergey Kulanov proposed openstack-infra/jeepyb master: Ensure acl_config has absolute path https://review.openstack.org/482612 | 15:54 |
| pabelanger | mtreinish: sdague: do you mind adding https://review.openstack.org/#/c/482600/ to your review pipeline? That will help fix nodepool-builder, we'll also need to backport into stable branches once you are happy with the code | 15:55 |
| *** aarefiev is now known as aarefiev_afk | 15:56 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool feature/zuulv3: EOL ubuntu-precise for dsvm job https://review.openstack.org/482613 | 15:58 |
| *** yamamoto has quit IRC | 15:58 | |
| fungi | c | 15:59 |
| fungi | wait, you're not my mutt window! | 15:59 |
| *** dizquierdo has quit IRC | 16:00 | |
| *** eharney has quit IRC | 16:00 | |
| *** krtaylor_ has quit IRC | 16:00 | |
| *** krtaylor has joined #openstack-infra | 16:01 | |
| *** jascott1 has joined #openstack-infra | 16:01 | |
| *** dizquierdo has joined #openstack-infra | 16:01 | |
| *** egonzalez has quit IRC | 16:02 | |
| *** slaweq has joined #openstack-infra | 16:03 | |
| *** ykarel has quit IRC | 16:04 | |
| clarkb | I've approved https://review.openstack.org/#/c/453323/ once, that is merged and ansible puppet finishes its existing run through I will remove review-dev from the emergency host file | 16:05 |
| *** ykarel has joined #openstack-infra | 16:05 | |
| *** jamesmcarthur has joined #openstack-infra | 16:06 | |
| *** jascott1 has quit IRC | 16:06 | |
| *** jascott1 has joined #openstack-infra | 16:06 | |
| *** jamesmcarthur has quit IRC | 16:06 | |
| *** jamesmcarthur has joined #openstack-infra | 16:07 | |
| *** Apoorva has joined #openstack-infra | 16:08 | |
| bkero | Any clue why nodepool would throw a HTTP 405 against Jenkins when trying to create new nodes in Jenkins? http://paste.openstack.org/show/615044/ | 16:09 |
| bkero | clarkb: ^ | 16:09 |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci master: WIP: containers periodic test https://review.openstack.org/475747 | 16:09 |
| fungi | bkero: permissions on the jenkins side for the api account nodepool is using? | 16:09 |
| bkero | fungi: Jenkins is in 'valid user? all the perms!' mode | 16:10 |
| *** mriedem has joined #openstack-infra | 16:11 | |
| clarkb | bkero: fungi I think someone mentioned that jenkins changed and API from a GET to a POST last week | 16:11 |
| fungi | i want to say enforcing cors on the http api was also a problem, but it's been so long now i don't remember clearly | 16:11 |
| clarkb | so python-jenkins needs updating, pretty sure there is/was a change for that not sure where it has gone though | 16:12 |
| fungi | bkero: how new is your jenkins? | 16:12 |
| pabelanger | sounds familiar | 16:12 |
| *** MatthewLi has quit IRC | 16:12 | |
| bkero | fungi: 2.67 | 16:12 |
| fungi | is 2.67 recent? | 16:12 |
| *** jascott1 has quit IRC | 16:12 | |
| *** slaweq has quit IRC | 16:13 | |
| bkero | 2.69 is latest | 16:13 |
| *** annegentle has quit IRC | 16:13 | |
| bkero | I'm using nodepool 0.4.0 and python-jenkins 0.4.14 | 16:13 |
| openstackgerrit | Merged openstack-infra/system-config master: upgrade review-dev.o.o with gerrit v2.13.9 https://review.openstack.org/453323 | 16:13 |
| clarkb | https://review.openstack.org/#/c/305882/ maybe? | 16:13 |
| *** amotoki is now known as amotoki_away | 16:14 | |
| bkero | FWIW nodepool 0.4.1 totally rewrote the jenkins manager since there's a lot of duplicate code | 16:14 |
| bkero | I'm unclear if I can run nodepool 0.4.1 without a zuul though | 16:15 |
| *** dhajare has joined #openstack-infra | 16:15 | |
| clarkb | bkero: iirc that is an artifact from when python-jenkins aws a mostly dead lp hosted project? so nodepool fixed things in subclass then python-jenkins got moved to our gerrit and was updated and may now be on life support again? though changes merged to it a couple weeks ago | 16:15 |
| *** isaacb has quit IRC | 16:16 | |
| clarkb | bkero: so yes nodepool has a bunch of subclass redundant code and there is a change to clean that up but last time we approved it it broke everything iirc | 16:16 |
| *** jascott1 has joined #openstack-infra | 16:16 | |
| bkero | clarkb: looks like it's broken again? I'm not sure what to do. | 16:16 |
| bkero | I'm back to dumping params and attempting to match against the Jenkins REST API | 16:17 |
| *** annegentle has joined #openstack-infra | 16:17 | |
| clarkb | bkero: jenkins has made non backward compatible changes to their api, things that talk to that api python-jenkins/nodepool will need to be updated | 16:17 |
| *** jascott1 has quit IRC | 16:17 | |
| bkero | So jenkins downgrade or python-jenkins/nodepool update | 16:17 |
| *** shu-mutou-AWAY has quit IRC | 16:17 | |
| *** armax has quit IRC | 16:17 | |
| *** jascott1 has joined #openstack-infra | 16:17 | |
| clarkb | https://review.openstack.org/#/c/480601/ | 16:17 |
| clarkb | which points to https://review.openstack.org/#/c/259157 | 16:18 |
| *** pabelanger has quit IRC | 16:18 | |
| *** pabelanger has joined #openstack-infra | 16:18 | |
| clarkb | looks like we went ahead and merged the cleanup code again. so I think you can just updte nodepool? | 16:18 |
| bkero | clarkb: can I run 0.4.1 without a zuul? | 16:18 |
| clarkb | bkero: yes, but you need a zookeeper for image builds | 16:19 |
| *** ykarel has quit IRC | 16:19 | |
| bkero | That's fine | 16:19 |
| clarkb | mordred: fwiw ^ this appears to be another case of nodepool without zuul being useful? | 16:19 |
| *** sshnaidm is now known as sshnaidm|afk | 16:21 | |
| mordred | clarkb: yah - folks who are still using jenkins | 16:22 |
| clarkb | mordred: sure, I'm not sure we'd expect them to all switch to zuul necessarily. Especially since there is potentially significant effort required to do so | 16:23 |
| mordred | clarkb: I believe that once we release v3 it would be a great idea for someone (not us) to write a jenkins provider plugin that can talk to the nodepool zk api | 16:23 |
| mordred | clarkb: I obviously have no interest in doing so, and I thnk doing it before that api is fully documneted and solidified would be problematic - but I think it would be a really smart thing for folks to write | 16:24 |
| *** jpich has quit IRC | 16:25 | |
| *** ralonsoh has quit IRC | 16:30 | |
| pabelanger | okay, going to start poking at mirrors in infracloud | 16:33 |
| fungi | pabelanger: the flavor change hasn't merged yet | 16:33 |
| pabelanger | oh | 16:33 |
| fungi | i rechecked it a few minutes ago--one of its jobs failed on a dns lookup error | 16:33 |
| pabelanger | ya, see that now | 16:33 |
| fungi | and a giant stack of heat changes has us a bit starved for nodes in check at the moment so could be a few minutes still | 16:35 |
| fungi | a stack of 30 heat changes apparently | 16:36 |
| pabelanger | eep | 16:36 |
| fungi | so not "giant" but fairly sizable nonetheless | 16:37 |
| *** spzala has quit IRC | 16:37 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul feature/zuulv3: Add docs for web server component https://review.openstack.org/482630 | 16:42 |
| *** baoli has quit IRC | 16:43 | |
| *** baoli has joined #openstack-infra | 16:43 | |
| clarkb | the bug I filed years ago against jenkins about the ssh slaves plugin thread leak OOMing the server? it is still affecting jenkins users and still being updated :/ | 16:44 |
| fungi | but no longer affecting us! | 16:45 |
| mtreinish | clarkb: at least they're working on it :) | 16:45 |
| clarkb | mtreinish: well working on it means being incredulous at their users for doing such a thing and constantly telling people that cloud slave plugin drivers are broken by design despite being told over and over that this isn't a cloud slave plugin and instead adding and removing instances via the rest api | 16:46 |
| mordred | fungi: can we cut a release of bindep? the currently released version has a depend on pbr < 1 - it's fixed in master | 16:46 |
| fungi | mordred: seems like a good idea to me. are there other bug fixes we should review and merge before tagging? | 16:47 |
| dims | dirk : "[12:36:46] <fungi>a stack of 30 heat changes apparently" | 16:48 |
| dims | fungi : clarkb : requirements jobs are not bothering gate, are they? | 16:48 |
| *** shardy has quit IRC | 16:49 | |
| pabelanger | dims: they seem to be failing right now, looks like neutron-full is the issue? | 16:49 |
| mordred | fungi: https://review.openstack.org/#/c/467803 | 16:50 |
| dims | pabelanger : just wanted to be sure we did not wedge the gate | 16:50 |
| pabelanger | but ya, gate is up to 75 jobs atm | 16:50 |
| fungi | dims: not as far as i'm aware. the heat changes i was referring to was zaneb's series of 30 at https://review.openstack.org/481757 | 16:50 |
| pabelanger | dims: ya, keeping an eye on it today. gate looks to be busy | 16:50 |
| mordred | pabelanger, fungi: https://review.openstack.org/#/c/468433/ and https://review.openstack.org/#/c/468432/2 seem like a good idea | 16:50 |
| dims | ack fungi thanks! | 16:50 |
| dirk | dims: pabelanger : it looks like there are gate resets due to some of the requirements jobs are failing, so it starts all over again? | 16:50 |
| mtreinish | clarkb: yep, thats working on it by someone's definition :p | 16:51 |
| pabelanger | dirk: right, once top most job fails, gate will reset | 16:51 |
| *** dprince has quit IRC | 16:51 | |
| pabelanger | right now upto almost 6hour | 16:51 |
| mordred | fungi, pabelanger: also https://review.openstack.org/#/c/468071 looks like a good idea | 16:52 |
| *** lucasagomes is now known as lucas-afk | 16:52 | |
| pabelanger | mordred: +2 on all 3 | 16:52 |
| *** jamesmcarthur has quit IRC | 16:52 | |
| *** jamesmcarthur has joined #openstack-infra | 16:53 | |
| mordred | the others are refactors or have a -1 - so I don't think landing them real quick is a great call | 16:53 |
| *** dizquierdo has quit IRC | 16:54 | |
| *** dprince has joined #openstack-infra | 16:55 | |
| *** slaweq has joined #openstack-infra | 16:55 | |
| fungi | mordred: thanks, i've approved the set you mentioned | 16:55 |
| fungi | once those land, i guess we need to decide on the next version number based on what's changed | 16:56 |
| openstackgerrit | Merged openstack-infra/bindep master: Update tox.ini with py35 https://review.openstack.org/467803 | 16:57 |
| *** sshnaidm|afk has quit IRC | 16:58 | |
| *** yamamoto has joined #openstack-infra | 16:59 | |
| hwoarang | Hello. any opinions on https://review.openstack.org/#/c/476851/ which is sitting there for a few weeks now? TIA :) | 16:59 |
| pabelanger | clarkb: there is a large number of held nodes on nodepool.o.o, can I start to clean them up? | 16:59 |
| pabelanger | clarkb: I only see your name | 17:00 |
| openstackgerrit | Merged openstack-infra/bindep master: Fix test_main and test_depends for systems missing lsb_release https://review.openstack.org/468432 | 17:00 |
| *** pcrews has joined #openstack-infra | 17:00 | |
| pabelanger | infra-root: if you have a held node on nodepool, let me know. Otherwise, I plan on start deleting some. A few are from 30+ days | 17:00 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Improve logging on reconfiguration https://review.openstack.org/482635 | 17:01 |
| *** LindaWang has quit IRC | 17:01 | |
| openstackgerrit | Merged openstack-infra/bindep master: Log an error when lsb_release cannot be executed https://review.openstack.org/468433 | 17:02 |
| openstackgerrit | Merged openstack-infra/bindep master: Handle dotted versions in platform selectors https://review.openstack.org/468071 | 17:02 |
| fungi | pabelanger: i don't seem to have any held | 17:02 |
| *** spzala has joined #openstack-infra | 17:02 | |
| fungi | so no objection from me | 17:02 |
| *** baoli has quit IRC | 17:02 | |
| *** spzala has quit IRC | 17:03 | |
| mordred | fungi: oh - crappit. I may be all smoke and mirrors | 17:03 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul feature/zuulv3: Cleanup components doc https://review.openstack.org/482636 | 17:03 |
| jeblair | pabelanger: feel free to delete mine. and i always think any held node without a name is fair game to delete. :) | 17:03 |
| *** spzala has joined #openstack-infra | 17:03 | |
| fungi | mordred: you mean like https://git.openstack.org/cgit/openstack-infra/bindep/tree/requirements.txt ? | 17:03 |
| fungi | pbr>=2.0.0 | 17:04 |
| mordred | fungi: we already released it | 17:04 |
| mordred | fungi: I ran in to "I had editable version installed and pip did weird things when I tried to do pip install -U" | 17:04 |
| fungi | doesn't hurt to flush some bindep changes and do a new release anyway | 17:04 |
| mordred | but yah | 17:04 |
| fungi | that was on my to do list along with git-review | 17:04 |
| *** yamamoto has quit IRC | 17:04 | |
| *** baoli has joined #openstack-infra | 17:04 | |
| fungi | one of those to do list entries i seem to keep not getting to due to lack of time | 17:04 |
| *** slaweq has quit IRC | 17:05 | |
| bkero | clarkb, fungi: Great. master nodepool and master python-jenkins fixed it | 17:08 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul feature/zuulv3: Cleanup components doc https://review.openstack.org/482636 | 17:08 |
| *** jcoufal has quit IRC | 17:12 | |
| clarkb | I have removed review-dev from the ansible emergency hosts list | 17:13 |
| fungi | thanks clarkb! | 17:13 |
| *** jpena is now known as jpena|off | 17:13 | |
| *** eranrom has joined #openstack-infra | 17:14 | |
| *** jpena|off has quit IRC | 17:14 | |
| clarkb | that should automatically update from 2.13.8 to 2.13.9 and then gerrit will do an online reindex | 17:14 |
| clarkb | then we'll be in a good spot to do proper testing of gerrit | 17:14 |
| *** iyamahat has quit IRC | 17:15 | |
| *** efoley has quit IRC | 17:15 | |
| *** armax has joined #openstack-infra | 17:17 | |
| *** esberglu has quit IRC | 17:19 | |
| *** sambetts is now known as sambetts|afk | 17:21 | |
| *** baoli has quit IRC | 17:27 | |
| fungi | clarkb: planning to provide a status update on the gerrit upgrade during our meeting today? just making sure i get it on the agenda if so | 17:27 |
| clarkb | fungi: sure. | 17:27 |
| clarkb | I am waiting for puppet to restart the server, hasn't happened yet | 17:28 |
| fungi | cool, i'm adding it on the agenda now | 17:28 |
| clarkb | I think the 1730 run will do it | 17:28 |
| *** krtaylor has quit IRC | 17:30 | |
| *** iyamahat has joined #openstack-infra | 17:31 | |
| *** slaweq has joined #openstack-infra | 17:31 | |
| *** baoli has joined #openstack-infra | 17:33 | |
| *** baoli has quit IRC | 17:33 | |
| *** jcoufal has joined #openstack-infra | 17:34 | |
| bkero | clarkb: Finally got nodepool putting nodes in to jenkins, and am now dealing with the Jenkins SSH Slaves plugin. I didn't realize that it would require a java agent on every build slave. Is that correct, or did infra do it another way? | 17:35 |
| clarkb | bkero: that is correct, jenkins will self install it though, you just need a jvm on the slaves iirc | 17:35 |
| bkero | Yeah. I'll have to add an element for that. | 17:36 |
| fungi | really it's just a distro package to get the jvm on your images, so if you already have an element where you manage your package additions i'd just stick it in your packagemap there | 17:39 |
| *** baoli has joined #openstack-infra | 17:39 | |
| *** jamesmcarthur has quit IRC | 17:40 | |
| *** eharney has joined #openstack-infra | 17:40 | |
| *** tosky has quit IRC | 17:41 | |
| *** trown|lunch is now known as trown | 17:43 | |
| bkero | fungi: the slave.jar doesn't support jre-7 anymore | 17:44 |
| bkero | So trusty needs a ppa and a bump, whee | 17:44 |
| fungi | oh, ick | 17:44 |
| clarkb | review-dev is updating now. It appears to be doing an offline reindex so it isn't going as I expected it to, but it should be fine | 17:44 |
| fungi | so basically the ssh slave agent has dropped production support for ubuntu trusty (any any other distros which don't have java 8 by default)? | 17:45 |
| bkero | I guess so | 17:45 |
| clarkb | worst case we manually stop gerrit and rerun an offline reindex if the one puppet is running stalls out or doesn't complete | 17:45 |
| *** jrniemijr has joined #openstack-infra | 17:45 | |
| bkero | and the SSH Slave plugin won't blindly trust hostkeys anymore, and there's no knob for that | 17:45 |
| fungi | bkero: rather unfortunate | 17:45 |
| bkero | fungi: https://jenkins.io/blog/2017/04/10/jenkins-has-upgraded-to-java-8/ | 17:45 |
| *** zzzeek_ has quit IRC | 17:46 | |
| fungi | clarkb: yeah, i thought we hadn't properly fixed it to avoid offline reindexing yet based on my prior tests | 17:46 |
| bkero | fungi: what did you all do for hostkeys on slaves? | 17:46 |
| clarkb | bkero: I think it didn't check them | 17:47 |
| bkero | Just the runtime-ssh-host-keys element? | 17:47 |
| fungi | bkero: in zuul 2.x we just don't check them | 17:47 |
| bkero | awesome | 17:47 |
| fungi | in zuul v3 there is a bunch of discussion around it | 17:47 |
| *** yamahata has joined #openstack-infra | 17:47 | |
| clarkb | bkero: https://issues.jenkins-ci.org/browse/JENKINS-43062 that implies there is a non verifying strategy that you can use | 17:48 |
| fungi | but generally, tofu for single-use slaves is probably reasonable when relying exclusively on rsa auth | 17:48 |
| clarkb | bkero: however its possible that nodepool would need an update to specify that when it adds the slaves to jenkins | 17:48 |
| bkero | clarkb: Yes, there is. It can't be selected as default behavior though. | 17:48 |
| bkero | you have to go into the form and click 'save' | 17:48 |
| bkero | ._. | 17:48 |
| clarkb | bkero: you can't set it when adding the slave? | 17:48 |
| *** zzzeek_ has joined #openstack-infra | 17:49 | |
| *** slaweq has quit IRC | 17:49 | |
| *** raissa_ has quit IRC | 17:49 | |
| bkero | clarkb: I haven't looked at the API for it yet | 17:49 |
| clarkb | Jul 11 17:43:48 review-dev puppet-user[4650]: (/Stage[main]/Gerrit/Exec[gerrit-init]) /etc/init.d/gerrit stop; /usr/bin/java -jar /home/gerrit2/review_site/bin/gerrit.war init -d /home/gerrit2/review_site --batch --no-auto-start returned 128 instead of one of [0] | 17:50 |
| clarkb | I'm going to rerun the init command to see what ^ is about | 17:50 |
| EmilienM | hey folks, could we get review on a devstack-gate patch please? https://review.openstack.org/#/c/475788 | 17:51 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Use package module to install bindep packages https://review.openstack.org/482584 | 17:52 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Simplify bindep logic removing fallback support https://review.openstack.org/482650 | 17:52 |
| *** rbrndt has quit IRC | 17:53 | |
| clarkb | looks to be a permissions issue on the review_site/lib contents, it wants to rename the libs in there and can't beacuse of perms | 17:53 |
| EmilienM | clarkb: fyi, https://review.openstack.org/#/c/481731 is our latest effort to reduce logsize for container jobs | 17:53 |
| * clarkb double checking that I'm not supposed to run that as root | 17:53 | |
| EmilienM | weshay: not sure we have more efforts in progress, maybe you know better | 17:53 |
| clarkb | puppet says run as gerrit2, let me chown things | 17:54 |
| weshay | EmilienM, bah.. just trying to get the gate to merge that patch | 17:54 |
| EmilienM | weshay: it should merge now | 17:54 |
| EmilienM | weshay: we fixed our gate issue this morning | 17:54 |
| weshay | EmilienM, that reduces the undercloud-containers job to about 75mb, also have https://review.openstack.org/#/c/476242/ | 17:56 |
| weshay | so I can then reduce the logs further | 17:56 |
| EmilienM | weshay: really cool the last one | 17:57 |
| * EmilienM approving patches | 17:57 | |
| *** annegentle has quit IRC | 17:58 | |
| clarkb | ok the reason for that failure is that there were preexisting .bcp*.jar.backup files in review_site/lib. I manually deleted them but we should add cleanup of those libs to our puppet manifests | 17:58 |
| clarkb | the next puppet run should fix the libs for us and download the versions we want, then I will reindex and start gerrit | 17:58 |
| *** annegentle has joined #openstack-infra | 17:59 | |
| clarkb | #status log small hiccup in review-dev gerrit 2.13.8 -> 2.13.9 upgrade. Will be offline temporarily while we wait on puppet to curate lib installations | 18:00 |
| openstackstatus | clarkb: finished logging | 18:00 |
| *** pvaneck has joined #openstack-infra | 18:00 | |
| *** baoli has quit IRC | 18:00 | |
| *** baoli has joined #openstack-infra | 18:01 | |
| *** jamesmcarthur has joined #openstack-infra | 18:01 | |
| *** yamamoto has joined #openstack-infra | 18:02 | |
| *** electrofelix has quit IRC | 18:02 | |
| *** esberglu has joined #openstack-infra | 18:02 | |
| *** esberglu has quit IRC | 18:04 | |
| *** esberglu has joined #openstack-infra | 18:05 | |
| *** eharney has quit IRC | 18:05 | |
| *** tosky has joined #openstack-infra | 18:07 | |
| *** yamamoto has quit IRC | 18:07 | |
| pabelanger | clarkb: was it okay to delete you held nodepool nodes? | 18:11 |
| clarkb | I think so let me double check | 18:11 |
| *** jamesmcarthur has quit IRC | 18:12 | |
| openstackgerrit | Arx Cruz proposed openstack-infra/tripleo-ci master: Fix stackviz collect data https://review.openstack.org/474559 | 18:12 |
| clarkb | pabelanger: yes they can be deleted. I will do that now (I still need to test more of the apt/pip stuff through the caching proxy but I don't have time right this moment and can hold new nodes for that | 18:12 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: WIP: Enable verify_host for synchronize https://review.openstack.org/482655 | 18:13 |
| pabelanger | clarkb: ack, thanks | 18:13 |
| clarkb | pabelanger: done | 18:14 |
| *** spzala has quit IRC | 18:14 | |
| pabelanger | clarkb: ty | 18:14 |
| *** yamahata has quit IRC | 18:14 | |
| *** yamahata has joined #openstack-infra | 18:15 | |
| clarkb | pabelanger: idea is get https://review.openstack.org/#/c/476289/ in then use that to test performance and reliabilty of pip installs via caching reverse proxy instead of via afs cache from places like europe | 18:15 |
| clarkb | then based on that data maybe we switch to using that caching method rather than afs? its mostly still in the we need more data phase | 18:16 |
| *** krtaylor has joined #openstack-infra | 18:16 | |
| *** spzala has joined #openstack-infra | 18:16 | |
| openstackgerrit | Merged openstack-infra/system-config master: Increase Infra-cloud mirror flavor disk to 250GiB https://review.openstack.org/482236 | 18:17 |
| *** eharney has joined #openstack-infra | 18:18 | |
| *** ggillies_ has quit IRC | 18:19 | |
| *** spzala has quit IRC | 18:21 | |
| *** ggillies has joined #openstack-infra | 18:21 | |
| *** spzala has joined #openstack-infra | 18:24 | |
| clarkb | puppet failed again because it ran the batch which I didn't expect it to. The reason for this is the gerrit.config file updated. I'm guessing on whitespace changes? we'll want to fix that | 18:25 |
| *** tesseract has quit IRC | 18:25 | |
| clarkb | but considering I ran a successful batch earlier I'm just going to go ahead and reindex and start the service | 18:25 |
| *** jamesmcarthur has joined #openstack-infra | 18:26 | |
| *** spzala_ has joined #openstack-infra | 18:26 | |
| openstackgerrit | Ronelle Landy proposed openstack-infra/tripleo-ci master: DO NOT MERGE: Adjust mtus for vxlan networking https://review.openstack.org/481222 | 18:26 |
| clarkb | I don't know why gerrit got so complicated, it doesn't seem to have really added features, just ops mess | 18:27 |
| *** spzala has quit IRC | 18:27 | |
| clarkb | I guess its always rerwitten its config file? | 18:27 |
| fungi | yep | 18:27 |
| clarkb | oh wait it happened because it changed the war file permissions | 18:27 |
| clarkb | we only subscribe to the war file updating and while the file contents didn't update the mode did which was enough to make it go rerun init and fail again :/ | 18:28 |
| *** Swami has joined #openstack-infra | 18:28 | |
| clarkb | so I think the todo here is to add puppet tidy rule cleanups for .bcp*.jar.backup | 18:29 |
| clarkb | in review_site/lib/ | 18:29 |
| clarkb | then we won't fail if we have to rerun init | 18:29 |
| clarkb | I'm running foreground reindex then will start the service | 18:30 |
| clarkb | I think we should also consider removing support for very old gerrit in the gerrit puppetry in order to simplify the manifest so that it is understandable | 18:34 |
| clarkb | because right now it seems like it never works and is quite complicated (granted gerrit deployment got quite complicated) | 18:35 |
| *** jcoufal has quit IRC | 18:35 | |
| *** GregHouse has quit IRC | 18:35 | |
| *** Goneri has quit IRC | 18:36 | |
| *** lathiat_ has quit IRC | 18:36 | |
| *** Goneri has joined #openstack-infra | 18:36 | |
| AJaeger | clarkb: here's a debian unbound change - could you put this on your review queue for later, please? https://review.openstack.org/476442 | 18:38 |
| *** EricGonczer_ has joined #openstack-infra | 18:38 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Create configure-mirrors role https://review.openstack.org/482593 | 18:38 |
| *** annegentle has quit IRC | 18:39 | |
| AJaeger | fungi, https://review.openstack.org/#/c/479057 adds a diskimage-builder-ci ACL - do you want to review this? | 18:41 |
| *** lathiat has joined #openstack-infra | 18:41 | |
| AJaeger | EmilienM: please review https://review.openstack.org/#/c/476089/3/zuul/layout.yaml | 18:41 |
| EmilienM | AJaeger: sure | 18:41 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Improve logging on reconfiguration https://review.openstack.org/482635 | 18:41 |
| EmilienM | AJaeger: everytime we change the regex, we break it :D | 18:42 |
| *** rbrndt has joined #openstack-infra | 18:42 | |
| AJaeger | EmilienM: so, let's not merge that one ;) | 18:43 |
| EmilienM | AJaeger: let me 5 min | 18:43 |
| *** makowals has quit IRC | 18:44 | |
| *** esberglu_ has joined #openstack-infra | 18:44 | |
| AJaeger | EmilienM: I can evven wait 5 days ;) | 18:45 |
| EmilienM | thanks | 18:45 |
| clarkb | https://review-dev.openstack.org/#/q/status:open is now running 2.13.9 | 18:47 |
| *** esberglu has quit IRC | 18:47 | |
| *** dtroyer has quit IRC | 18:47 | |
| AJaeger | yeah | 18:47 |
| clarkb | and the change id comment links work again | 18:47 |
| *** krtaylor has quit IRC | 18:48 | |
| clarkb | will have to watch puppet and make sure it is steady state too | 18:48 |
| *** dtroyer has joined #openstack-infra | 18:49 | |
| *** nicolasbock has quit IRC | 18:50 | |
| *** jcoufal has joined #openstack-infra | 18:52 | |
| *** jamesmcarthur has quit IRC | 18:54 | |
| *** krtaylor has joined #openstack-infra | 18:56 | |
| dmsimard | clarkb: is the cirros image no longer cached on the devstack image ? grep "cirros" yields nothing in diskimage-builder or in project-config/nodepool | 18:56 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Create configure-mirrors role https://review.openstack.org/482593 | 18:56 |
| fungi | dmsimard: we get it via devstack's list of things it uses, if memory serves | 18:59 |
| *** sdague has quit IRC | 19:00 | |
| EmilienM | AJaeger: lgtm. Let's give it a try | 19:00 |
| clarkb | yup, we don't explicitly say what we cache anywhere, instead devstack has a script we run that tells us | 19:00 |
| clarkb | dmsimard: fungi ^ | 19:00 |
| *** pvaneck has quit IRC | 19:00 | |
| fungi | dmsimard: we don't define it directly in project-config, it's just handled by the cache-devstack element | 19:00 |
| openstackgerrit | Clark Boylan proposed openstack-infra/puppet-gerrit master: Remove backed up gerrit lib files https://review.openstack.org/482670 | 19:00 |
| clarkb | fungi: ^ fix for bcp backup files | 19:00 |
| fungi | it's that (weekly infra team meeting) time again! find us in #openstack-meeting for the next hour | 19:00 |
| dmsimard | fungi: ohhh | 19:01 |
| *** wolverineav has quit IRC | 19:01 | |
| *** baoli has quit IRC | 19:02 | |
| *** baoli has joined #openstack-infra | 19:02 | |
| *** rockyg has joined #openstack-infra | 19:03 | |
| *** wolverineav has joined #openstack-infra | 19:04 | |
| *** spzala_ has quit IRC | 19:06 | |
| *** sslypushenko_ has joined #openstack-infra | 19:07 | |
| clarkb | review-dev puppet just reran and it appears to be fairly steady state other than the config file changes | 19:08 |
| clarkb | so that lgtm | 19:08 |
| *** annegentle has joined #openstack-infra | 19:10 | |
| *** wolverineav has quit IRC | 19:10 | |
| *** makowals has joined #openstack-infra | 19:12 | |
| *** annegentle has quit IRC | 19:13 | |
| *** jamesmcarthur has joined #openstack-infra | 19:14 | |
| openstackgerrit | Ihar Hrachyshka proposed openstack-infra/project-config master: Removed periodic -pg- job from neutron gate https://review.openstack.org/482676 | 19:14 |
| *** annegentle has joined #openstack-infra | 19:15 | |
| dmsimard | clarkb, fungi: so I *think* we're caching a 126MB Ubuntu 12.04 image due to https://github.com/openstack-infra/project-config/blob/ed7836dd564fc61a1935b640b77c525aaa509322/nodepool/elements/cache-devstack/extra-data.d/55-cache-devstack-repos#L81 | 19:17 |
| fungi | dmsimard: that's fun! | 19:17 |
| dmsimard | and https://github.com/openstack-dev/devstack/blob/master/stackrc#L680 | 19:17 |
| dmsimard | http://paste.openstack.org/raw/615064/ | 19:17 |
| clarkb | dmsimard: only if the non legacy find images code doesn't run | 19:17 |
| clarkb | which it should run | 19:17 |
| jeblair | fungi, pabelanger: a quick google search for sl390s rail kits suggests we may be in for a challenge. | 19:18 |
| clarkb | because all branches of devstack have image_list.sh | 19:18 |
| dmsimard | clarkb: dunno, noticed this on an image built outside infra from the upstream elements and config | 19:18 |
| dmsimard | clarkb: don't have a way to check if it reproduces inside infra | 19:18 |
| clarkb | dmsimard: if tools/image_list.sh says cache 12.04 then that is more likely the cause than the code you linked | 19:18 |
| * dmsimard looks | 19:18 | |
| fungi | jeblair: yeah, i was afraid that might be the case :/ | 19:19 |
| openstackgerrit | Merged openstack-infra/project-config master: Stop running puppet-* jobs on tempest unittests/doc patches https://review.openstack.org/476089 | 19:19 |
| dmsimard | clarkb: yeah I don't know where that image comes from. | 19:20 |
| clarkb | dmsimard: devstack/tools/image_list.sh is what we run first and if that doesn't return any data we run that legacy lookup function you linked | 19:21 |
| clarkb | dmsimard: the fix here is likely editing devstack to stop emitting that image when image_list.sh runs | 19:21 |
| dmsimard | clarkb: I just don't know where it could be from tbh, http://codesearch.openstack.org/?q=ubuntu-12.04-x86_64.tar.gz&i=nope&files=&repos= yields nothing useful | 19:22 |
| *** rocky_g has joined #openstack-infra | 19:22 | |
| clarkb | dmsimard: on a stable branch? | 19:22 |
| *** rockyg has quit IRC | 19:23 | |
| dmsimard | clarkb: I would have to guess I'm using master from project-config nodepool elements which likely picks up the master of everything else | 19:23 |
| clarkb | dmsimard: we cache the images that devstack tells us for all available stable branches of devstack | 19:23 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Check ret for None in zuul_run_command https://review.openstack.org/482442 | 19:23 |
| clarkb | dmsimard: since stable jobs run on these hosts too | 19:24 |
| dmsimard | clarkb: ohhhhh | 19:24 |
| fungi | yeah, our cache element(s) iterate over all stable branches to decide what to preinstall/precache | 19:24 |
| fungi | though we do a lot less of that now than we used to | 19:24 |
| dmsimard | clarkb: oh wow devstack goes all the way back to stable/kilo | 19:24 |
| dmsimard | time for a cleanup ? | 19:25 |
| dmsimard | http://paste.openstack.org/raw/615066/ | 19:28 |
| dmsimard | Looks like 12.04 was returned by image_list.sh even in stable/ocata | 19:28 |
| dmsimard | then it's not in stable/newton | 19:28 |
| dmsimard | ¯\_(ツ)_/¯ | 19:28 |
| dmsimard | then it's in stable/liberty | 19:28 |
| dmsimard | and mitaka and kilo :( | 19:28 |
| fungi | wacky | 19:29 |
| jlk | fungi: hi there, is the project membership levels backed by source control anywhere? Core members and such? | 19:31 |
| *** rockyg has joined #openstack-infra | 19:31 | |
| jlk | (asking because I'd like to show something to my manager given you added me to a core group today :) | 19:31 |
| *** rocky_g has quit IRC | 19:32 | |
| AJaeger | jlk: which project you're core on? Let me show you something... | 19:33 |
| inc0 | fungi: http://tarballs.openstack.org/kolla/images/ <- ubuntu tarballs still throws 403 | 19:33 |
| AJaeger | jlk: so, no source control - just showing current status | 19:34 |
| inc0 | centos seems to be working...most of centos at least | 19:34 |
| AJaeger | jlk: https://review.openstack.org/#/settings/group-memberships - should have taken screenshot yesterday ;) | 19:34 |
| inc0 | would you guys be kind enough to chmod them back until we get cache going?:) our gates are red until we do this:( | 19:35 |
| AJaeger | jlk: and https://review.openstack.org/#/admin/groups/908,members is zuul-core | 19:35 |
| *** armax has quit IRC | 19:38 | |
| jlk | awesome, thanks! | 19:38 |
| *** slaweq has joined #openstack-infra | 19:40 | |
| *** makowals has quit IRC | 19:41 | |
| fungi | jlk: or you can do https://review.openstack.org/#/admin/groups/zuul-core as a shortcut | 19:43 |
| *** jamesmcarthur has quit IRC | 19:43 | |
| fungi | inc0: chairing the infra meeting at the moment, and yes any images built overnight got their permissions reset to readable but i can take care of the others in a bit | 19:43 |
| fungi | inc0: and pabelanger is working through the last piece of the puzzle for the caching prerequisites (infra-cloud mirror instance replacements with a larger disk flavor) | 19:44 |
| *** makowals has joined #openstack-infra | 19:45 | |
| inc0 | cool, thanks fungi, sry for interrupting | 19:46 |
| *** jamesden_ has joined #openstack-infra | 19:48 | |
| *** jamesdenton has quit IRC | 19:49 | |
| *** jamesden_ has quit IRC | 19:49 | |
| *** jamesdenton has joined #openstack-infra | 19:51 | |
| *** esberglu_ is now known as esberglu | 19:53 | |
| openstackgerrit | Lucas Alvares Gomes proposed openstack-infra/project-config master: Add networking-ovn job to ovsdbapp gate https://review.openstack.org/481650 | 19:54 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Enable verify_host for synchronize https://review.openstack.org/482655 | 19:55 |
| *** jamesmcarthur has joined #openstack-infra | 19:58 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Avoid using http://keyserver.ubuntu.com for GPG keys https://review.openstack.org/482683 | 19:59 |
| *** wolverineav has joined #openstack-infra | 20:00 | |
| *** jamesmcarthur has quit IRC | 20:00 | |
| *** xarses has quit IRC | 20:05 | |
| *** marst has quit IRC | 20:08 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Avoid using http://keyserver.ubuntu.com for GPG keys https://review.openstack.org/482683 | 20:10 |
| *** jkilpatr has quit IRC | 20:11 | |
| *** dprince has quit IRC | 20:12 | |
| *** slaweq has quit IRC | 20:12 | |
| *** annegentle has quit IRC | 20:12 | |
| *** slaweq has joined #openstack-infra | 20:13 | |
| *** armax has joined #openstack-infra | 20:13 | |
| *** makowals has quit IRC | 20:17 | |
| fungi | inc0: i've switched all the remaining files to world readable again, but will be keeping a close eye on bandiwdth utilization while we're finishing getting the proxy cache upfit completed. obviously your jobs will be just as broken as everyone else's if they knock tarballs.o.o offline again | 20:22 |
| *** sree has joined #openstack-infra | 20:23 | |
| openstackgerrit | Samantha Blanco proposed openstack-infra/project-config master: Set Tempest concurrency for non-multinode Patrole jobs https://review.openstack.org/482693 | 20:23 |
| *** annegentle has joined #openstack-infra | 20:24 | |
| *** askb has joined #openstack-infra | 20:26 | |
| *** sree has quit IRC | 20:27 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Avoid using http://keyserver.ubuntu.com for GPG keys https://review.openstack.org/482683 | 20:29 |
| *** jrniemijr has quit IRC | 20:31 | |
| *** jkilpatr has joined #openstack-infra | 20:32 | |
| *** rhallisey has quit IRC | 20:33 | |
| *** annegentle has quit IRC | 20:33 | |
| *** thorst has quit IRC | 20:35 | |
| *** adisky__ has quit IRC | 20:36 | |
| inc0 | fungi: understandable, it was fine for few weeks so I think we just pushed more stuff at same time | 20:41 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Avoid using apt-add-repository https://review.openstack.org/482683 | 20:42 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Add openstack-doc-build job for documentation testing https://review.openstack.org/482596 | 20:43 |
| *** wolverineav has quit IRC | 20:44 | |
| *** krtaylor has quit IRC | 20:46 | |
| *** zzzeek_ has quit IRC | 20:49 | |
| *** camunoz has quit IRC | 20:49 | |
| dirk | pabelanger: fungi: can we merge https://review.openstack.org/#/c/468117/ ? the corresponding nodepool builder project-config change has merged earlier today, so I think it makes sense to have the logging config in place | 20:54 |
| fungi | inc0: looking at the graph we likely had a few brief outages related to this in recent weeks as well, but none as lengthy as yesterday | 20:55 |
| fungi | and so nobody really had time to track down the cause | 20:55 |
| AJaeger | fungi, karbor translations jobs work fine again - thanks for your help. | 20:56 |
| fungi | AJaeger: thanks for finding an unrelated issue across a bunch of our persistent job nodes | 20:57 |
| *** mat128 has quit IRC | 20:57 | |
| openstackgerrit | Dirk Mueller proposed openstack/diskimage-builder master: opensuse-minimal: install glibc-locale https://review.openstack.org/473959 | 20:57 |
| inc0 | fungi: sorry :( | 20:59 |
| *** camunoz has joined #openstack-infra | 21:01 | |
| *** priteau has joined #openstack-infra | 21:01 | |
| *** camunoz has quit IRC | 21:03 | |
| *** zzzeek_ has joined #openstack-infra | 21:11 | |
| *** dhajare has quit IRC | 21:11 | |
| *** florianf has quit IRC | 21:12 | |
| *** jascott1 has quit IRC | 21:13 | |
| openstackgerrit | Dirk Mueller proposed openstack-infra/project-config master: Add grafana devstack failure rate graph for openSUSE https://review.openstack.org/482706 | 21:13 |
| *** jascott1 has joined #openstack-infra | 21:13 | |
| *** jascott1 has quit IRC | 21:14 | |
| openstackgerrit | Dirk Mueller proposed openstack-infra/project-config master: Add grafana devstack failure rate graph for openSUSE https://review.openstack.org/482706 | 21:15 |
| *** ldnunes has quit IRC | 21:17 | |
| *** makowals has joined #openstack-infra | 21:20 | |
| *** pcaruana has quit IRC | 21:23 | |
| openstackgerrit | Dirk Mueller proposed openstack/diskimage-builder master: Remove additional Bumblebee repository for opensuse element https://review.openstack.org/482708 | 21:24 |
| *** ihrachys has quit IRC | 21:25 | |
| openstackgerrit | Dirk Mueller proposed openstack-infra/project-config master: Update zuul-status rendering for the Tempest/Grenade graphs https://review.openstack.org/482711 | 21:30 |
| *** jascott1 has joined #openstack-infra | 21:32 | |
| *** rockyg has quit IRC | 21:34 | |
| openstackgerrit | Merged openstack-dev/cookiecutter master: Update to the Pike document structure https://review.openstack.org/482504 | 21:34 |
| *** thorst has joined #openstack-infra | 21:35 | |
| openstackgerrit | Merged openstack/os-testr master: Turn on warning-is-error in sphinx build https://review.openstack.org/478329 | 21:40 |
| *** slaweq has quit IRC | 21:40 | |
| openstackgerrit | Merged openstack/os-testr master: Switch from oslosphinx to openstackdocstheme https://review.openstack.org/478433 | 21:40 |
| openstackgerrit | Merged openstack/os-testr master: rearrange content to fit the new standard layout https://review.openstack.org/478546 | 21:41 |
| openstackgerrit | Merged openstack/os-testr master: update doc URL to the new location in the readme https://review.openstack.org/478550 | 21:41 |
| *** thorst has quit IRC | 21:42 | |
| clarkb | I can push changes to review-dev and new patchsets. | 21:43 |
| clarkb | I have confirmed that the changeid comment link works now | 21:44 |
| *** makowals has quit IRC | 21:45 | |
| *** eharney has quit IRC | 21:47 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Remove tox-py35-on-zuul https://review.openstack.org/482717 | 21:47 |
| *** dfflanders has joined #openstack-infra | 21:48 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/openstack-zuul-jobs master: Add tox-py35-on-zuul job definition https://review.openstack.org/482718 | 21:48 |
| *** rloo has joined #openstack-infra | 21:49 | |
| *** Apoorva has quit IRC | 21:50 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Add tox-py35-on-zuul to zuul-jobs check pipeline https://review.openstack.org/482719 | 21:50 |
| *** sslypushenko_ has quit IRC | 21:57 | |
| *** gordc has quit IRC | 21:57 | |
| openstackgerrit | Ronelle Landy proposed openstack-infra/tripleo-ci master: WIP: Adjust MTUs in multinode workflow on RDO Cloud https://review.openstack.org/481222 | 22:02 |
| *** mdrabe has quit IRC | 22:02 | |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Remove tox-py35-on-zuul https://review.openstack.org/482717 | 22:04 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Add tox-py35-on-zuul job definition https://review.openstack.org/482718 | 22:04 |
| *** Apoorva has joined #openstack-infra | 22:04 | |
| *** felipemonteiro__ has quit IRC | 22:04 | |
| mtreinish | mordred, rcarrillocruz: want to do me a favor and take a look at: https://github.com/ansible/ansible/pull/23769 we might be able to remove our local infra copy if that ever merges | 22:09 |
| *** baoli has quit IRC | 22:10 | |
| *** baoli has joined #openstack-infra | 22:11 | |
| *** baoli has quit IRC | 22:11 | |
| clarkb | leaving comments on changes both inline and cover seem to work well as does search (implying the secondary index is happy) | 22:13 |
| clarkb | You cannot leave inline comments on mobile browsers, at least not on my device | 22:13 |
| clarkb | but thats not really a regression | 22:13 |
| *** dave-mccowan has quit IRC | 22:14 | |
| *** priteau has quit IRC | 22:15 | |
| clarkb | I think thats the vast majority of my usage. There are some cosmetic differences people will need to get used to as well, but they seem to be improvements | 22:16 |
| *** markvoelker has quit IRC | 22:16 | |
| clarkb | when you are on an old patchset for example it shows up in bright yellow/orange | 22:16 |
| *** markvoelker has joined #openstack-infra | 22:17 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add tox-py35-on-zuul to zuul-jobs check pipeline https://review.openstack.org/482719 | 22:19 |
| fungi | that's a nice improvement | 22:19 |
| fungi | i should try pointing gertty at it | 22:19 |
| fungi | but not tonight | 22:20 |
| *** bnemec has quit IRC | 22:24 | |
| *** jcoufal has quit IRC | 22:24 | |
| *** bnemec has joined #openstack-infra | 22:25 | |
| *** jbadiapa_ has joined #openstack-infra | 22:29 | |
| *** pradk has quit IRC | 22:29 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Handle aborted websocket client connection cleanly https://review.openstack.org/482560 | 22:31 |
| *** jbadiapa has quit IRC | 22:31 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Fix error loading projects out of order https://review.openstack.org/482725 | 22:32 |
| *** MatthewLi has joined #openstack-infra | 22:32 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Fix error loading projects out of order https://review.openstack.org/482725 | 22:33 |
| *** bobh has quit IRC | 22:34 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: WIP Add job's project as implicit role project https://review.openstack.org/482726 | 22:37 |
| *** slaweq has joined #openstack-infra | 22:41 | |
| *** slaweq has quit IRC | 22:46 | |
| *** rbrndt has quit IRC | 22:46 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Cleanup on ssh-agent failure https://review.openstack.org/481344 | 22:46 |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: Updated Eventbrite permission model https://review.openstack.org/482730 | 22:54 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Fix error loading projects out of order https://review.openstack.org/482725 | 22:56 |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Use finishJob funtion to clean up on failure https://review.openstack.org/482731 | 22:57 |
| *** claudiub has quit IRC | 22:58 | |
| *** jaypipes has quit IRC | 23:00 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Move ssh-agent cleanup into existing exception handler https://review.openstack.org/482735 | 23:01 |
| *** tosky has quit IRC | 23:13 | |
| *** Goneri has quit IRC | 23:14 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Fail execution if a role repo is a bare collection of roles https://review.openstack.org/482327 | 23:14 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: WIP Add job's project as implicit role project https://review.openstack.org/482726 | 23:15 |
| *** apetrich has quit IRC | 23:16 | |
| *** apetrich has joined #openstack-infra | 23:18 | |
| *** ihrachys has joined #openstack-infra | 23:25 | |
| *** MatthewLi has quit IRC | 23:26 | |
| *** makowals has joined #openstack-infra | 23:27 | |
| *** hongbin has quit IRC | 23:29 | |
| *** d0ugal has joined #openstack-infra | 23:36 | |
| *** ramishra has quit IRC | 23:41 | |
| *** slaweq has joined #openstack-infra | 23:43 | |
| *** thorst has joined #openstack-infra | 23:47 | |
| *** slaweq has quit IRC | 23:47 | |
| *** krtaylor has joined #openstack-infra | 23:50 | |
| *** mattmceuen has quit IRC | 23:58 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!