Monday, 2017-08-28

*** dingyichen has joined #openstack-infra00:01
*** felipemonteiro_ has quit IRC00:02
clarkbkong: how are tou creating a VM?00:03
clarkbcould be your hypervisor doesnt like qcow200:04
clarkbor can only boot old versions of qcow2 (does dib not set that option?)00:04
kongclarkb: i created vm in an openstack deployment00:04
kongwith KVM as hypervisor00:04
*** jascott1 has joined #openstack-infra00:09
*** Qiming has quit IRC00:10
*** Qiming has joined #openstack-infra00:11
*** jascott1 has quit IRC00:13
*** claudiub has quit IRC00:17
*** spzala has joined #openstack-infra00:31
*** sflanigan has joined #openstack-infra00:40
*** thorst_afk has joined #openstack-infra00:41
*** Qiming has quit IRC00:44
*** Qiming has joined #openstack-infra00:45
*** thorst_afk has quit IRC00:46
*** mwarad has quit IRC00:50
*** slaweq has joined #openstack-infra00:50
*** thorst_afk has joined #openstack-infra00:51
*** Qiming has quit IRC00:51
*** slaweq has quit IRC00:55
*** Qiming has joined #openstack-infra00:55
*** superdan is now known as dansmith00:57
*** slaweq has joined #openstack-infra01:01
*** gmann has quit IRC01:02
*** gmann has joined #openstack-infra01:02
*** hongbin has joined #openstack-infra01:02
*** Julien-zte has quit IRC01:03
*** Julien-zte has joined #openstack-infra01:04
*** thorst_afk has quit IRC01:06
*** thorst_afk has joined #openstack-infra01:07
*** slaweq has quit IRC01:07
*** thorst_afk has quit IRC01:11
*** Qiming has quit IRC01:21
*** Qiming has joined #openstack-infra01:21
*** gcb has joined #openstack-infra01:27
*** Qiming has quit IRC01:27
*** Qiming has joined #openstack-infra01:28
*** zhurong has quit IRC01:28
*** Qiming has quit IRC01:37
*** Qiming has joined #openstack-infra01:39
openstackgerritSongming Yan proposed openstack-infra/project-config master: Add qos extension to all congress dsvm jobs  https://review.openstack.org/49814401:39
*** Qiming has quit IRC01:44
chenyingping clarkb01:44
*** Qiming has joined #openstack-infra01:46
clarkbchenying: hi01:48
chenyinghi clarkb I have sent a email to openstack-dev about request to delete some unused pages from OpenStack wiki. Do you know who has the administrator permissions to the wiki? Thanks.01:49
clarkbI dont off the top of my head01:50
*** liujiong has joined #openstack-infra01:51
chenyinghi clarkb Could you help me to confirm about it?  For some reasons, I hope that these unused pages could be deleted.02:07
*** baoli has quit IRC02:07
*** baoli has joined #openstack-infra02:08
clarkbMy guess is deletion is admin only to deter defacement02:11
*** baoli has quit IRC02:13
chenyingYes, it is admin only. But I can not find the guy with admin to help me delte them. These pages were created by our team.02:16
chenyingS/delte/delete02:16
*** felipemonteiro_ has joined #openstack-infra02:16
clarkbchenying: https://wiki.openstack.org/wiki/Special:ListUsers/sysop is the list (and apparently I am on it)02:18
chenyingclarkb That's great. Could you help me to delete these pages? Thanks.02:20
clarkbchenying: I cant now. I'm on a phone, but will look at it tomorrow02:21
*** felipemonteiro_ has quit IRC02:22
chenyingclarkb  OK. Thanks.02:23
*** thorst_afk has joined #openstack-infra02:23
*** thorst_afk has quit IRC02:24
*** sree has joined #openstack-infra02:24
*** felipemonteiro has joined #openstack-infra02:27
*** felipemonteiro has quit IRC02:32
*** baoli has joined #openstack-infra02:35
*** yamamoto has joined #openstack-infra02:38
*** yamamoto has quit IRC02:38
*** zhurong has joined #openstack-infra02:39
*** mwarad has joined #openstack-infra02:40
*** LindaWang has joined #openstack-infra02:41
*** amotoki__away is now known as amotoki02:42
*** Qiming has quit IRC02:43
*** yamamoto has joined #openstack-infra02:43
*** Qiming has joined #openstack-infra02:45
*** cuongnv has joined #openstack-infra02:46
*** Qiming has quit IRC02:49
*** Qiming has joined #openstack-infra02:53
*** Qiming has quit IRC02:58
*** Qiming has joined #openstack-infra02:59
*** Qiming has quit IRC03:10
*** ykarel has joined #openstack-infra03:12
*** Qiming has joined #openstack-infra03:12
*** Qiming has quit IRC03:17
*** ramishra has joined #openstack-infra03:18
*** thorst_afk has joined #openstack-infra03:25
*** thorst_afk has quit IRC03:30
*** Qiming has joined #openstack-infra03:31
*** hongbin has quit IRC03:36
*** udesale has joined #openstack-infra03:49
*** baoli has quit IRC03:54
*** links has joined #openstack-infra04:00
*** claudiub has joined #openstack-infra04:18
*** jascott1 has joined #openstack-infra04:19
*** jascott1 has quit IRC04:23
*** thorst_afk has joined #openstack-infra04:26
*** ykarel has quit IRC04:26
*** ykarel has joined #openstack-infra04:27
*** stakeda has quit IRC04:29
*** thorst_afk has quit IRC04:31
*** zoli has quit IRC04:39
*** zoli has joined #openstack-infra04:40
*** rama_y has joined #openstack-infra04:42
*** jascott1 has joined #openstack-infra04:55
*** jascott1 has quit IRC04:59
*** pbourke has quit IRC05:08
*** pbourke has joined #openstack-infra05:10
*** adisky__ has joined #openstack-infra05:13
*** stakeda has joined #openstack-infra05:18
*** jaosorior has joined #openstack-infra05:18
*** markvoelker_ has joined #openstack-infra05:25
*** thorst_afk has joined #openstack-infra05:27
*** markvoelker has joined #openstack-infra05:29
*** markvoelker_ has quit IRC05:29
*** rcernin has joined #openstack-infra05:30
*** jamesmcarthur has joined #openstack-infra05:31
*** thorst_afk has quit IRC05:32
*** jascott1 has joined #openstack-infra05:33
*** markvoelker has quit IRC05:33
*** eumel8 has quit IRC05:34
*** markvoelker has joined #openstack-infra05:34
*** udesale has quit IRC05:35
*** udesale__ has joined #openstack-infra05:35
*** jamesmcarthur has quit IRC05:35
sshnaidmdoes anybody know why https://download.cirros-cloud.net/ is down? cirros-cloud.net is not registered anymore..05:35
calebbsshnaidm: DreamHost name servers have been getting DDoS'd, and that's where that domain gets served from iirc.05:40
sshnaidmcalebb, thanks.. do you know if I can get it by IP?05:41
AJaegercalebb, sshnaidm, looking at their status twitter bot: They're fine again. might still be a fallout05:41
calebbsshnaidm: not sure05:41
calebbAJaeger: yeah i was trying to find out if it was ongoing or if it's fallout, i think you're right05:41
AJaegerhttps://bugs.launchpad.net/cirros/+bug/1713358 is reported05:42
openstackLaunchpad bug 1713358 in CirrOS "Cirros website is no longer available" [Undecided,New]05:42
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci master: Use local cirros image for tempest tests  https://review.openstack.org/49831505:48
*** mpeterson has joined #openstack-infra05:49
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci master: Use local cirros image for tempest tests  https://review.openstack.org/49831505:53
*** ijw has joined #openstack-infra05:55
*** hashar has joined #openstack-infra05:56
*** martinkopec has joined #openstack-infra06:00
*** dhajare has joined #openstack-infra06:00
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci master: Use local cirros image for tempest tests  https://review.openstack.org/49831506:01
*** florianf has joined #openstack-infra06:07
sshnaidmdoes anybody know any mirror of centos images that can be used for now?06:07
*** slaweq has joined #openstack-infra06:08
*** pgadiya has joined #openstack-infra06:13
*** slaweq has quit IRC06:13
*** lihi has joined #openstack-infra06:16
*** pcrews has quit IRC06:19
*** wolverineav has joined #openstack-infra06:20
bkerosshnaidm: OSUOSL? http://centos.osuosl.org/7/cloud/x86_64/06:20
sshnaidmbkero, do they have cirros?06:21
sshnaidmI meant cirros when wrote centos :(06:23
*** pcrews has joined #openstack-infra06:23
bkerooh06:24
bkeromaybe06:24
*** thorst_afk has joined #openstack-infra06:28
*** thorst_afk has quit IRC06:32
AJaegersshnaidm: why are you not caching cirros always - instead of downloading it each time? Might be something for longer term as well.06:34
AJaegersshnaidm: no idea about mirror. We have them baked in our nodepool images, so an admin might be able to give you one...06:35
sshnaidmAJaeger, usually we do, but need it for 3-d party CIs,etc06:35
*** AJaeger has quit IRC06:36
*** pcaruana has joined #openstack-infra06:36
*** sree has quit IRC06:38
*** slaweq has joined #openstack-infra06:43
sshnaidmif somebody need it urgently: https://github.com/sshnaidm/cirros-mirror/raw/master/cirros-0.3.5-x86_64-disk.img06:45
*** dmsimard has quit IRC06:50
*** AJaeger has joined #openstack-infra06:55
*** eumel8 has joined #openstack-infra07:01
openstackgerritIan Wienand proposed openstack/diskimage-builder master: Use [[ for =~ matches  https://review.openstack.org/49833307:02
*** gixxer1k has joined #openstack-infra07:05
*** andreas_s has joined #openstack-infra07:14
*** therve_ is now known as therve07:18
*** dmsimard has joined #openstack-infra07:19
*** rama_y has quit IRC07:21
eumel8Good morning. We have to setup additional sync jobs for stable/pike branches in the translation server. It would be nice, if someone from the infra core team beside AJaeger can review https://review.openstack.org/#/c/498016/07:27
*** d0ugal has joined #openstack-infra07:29
*** d0ugal has quit IRC07:29
*** jpena|off is now known as jpena07:29
*** d0ugal has joined #openstack-infra07:30
*** tesseract has joined #openstack-infra07:31
*** markus_z has joined #openstack-infra07:37
*** jpich has joined #openstack-infra07:40
*** egonzalez has joined #openstack-infra07:42
*** coolsvap has joined #openstack-infra07:52
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci master: Disable update of initramfs images  https://review.openstack.org/49835507:54
*** bogdando has joined #openstack-infra07:56
*** dtantsur|sick is now known as dtantsur08:01
openstackgerritMerged openstack-infra/project-config master: Add qos extension to all congress dsvm jobs  https://review.openstack.org/49814408:02
openstackgerritMerged openstack-infra/project-config master: Add experimental opensuse-423 job for freezer  https://review.openstack.org/49146508:07
*** kjackal has joined #openstack-infra08:07
openstackgerritMerged openstack-infra/project-config master: Add SDN choice to OpenStack Helm gates.  https://review.openstack.org/49780908:08
openstackgerritMerged openstack-infra/project-config master: Add Pike translation jobs  https://review.openstack.org/49801608:09
eumel8thx :)08:09
*** slaweq_ has joined #openstack-infra08:10
*** ykarel is now known as ykarel|lunch08:12
openstackgerritMerged openstack-infra/irc-meetings master: Switch meeting chair for Cinder Queens PTL  https://review.openstack.org/49629808:14
openstackgerritMerged openstack-infra/irc-meetings master: Move searchlight meeting to biweekly (odd)  https://review.openstack.org/49746908:15
*** slaweq_ has quit IRC08:15
*** lucas-afk is now known as lucasagomes08:16
*** efoley has joined #openstack-infra08:16
*** efoley_ has joined #openstack-infra08:18
*** efoley has quit IRC08:18
*** dingyichen has quit IRC08:23
*** ralonsoh has joined #openstack-infra08:28
*** thorst_afk has joined #openstack-infra08:30
*** wolverineav has quit IRC08:33
*** thorst_afk has quit IRC08:34
*** d0ugal has quit IRC08:35
*** ykarel|lunch is now known as ykarel08:37
*** sree has joined #openstack-infra08:39
*** jaosorior has quit IRC08:39
*** e0ne has joined #openstack-infra08:43
*** sree has quit IRC08:44
bogdandohi. can we merge e-r queries https://review.openstack.org/#/c/493535/ https://review.openstack.org/#/c/493525/  to reduce unknown/uncategorised numbers?08:48
*** sree has joined #openstack-infra08:49
*** tosky has joined #openstack-infra08:49
*** jaosorior has joined #openstack-infra08:50
*** wolverineav has joined #openstack-infra09:05
*** bhavik1 has joined #openstack-infra09:08
*** claudiub|2 has joined #openstack-infra09:08
*** ykarel has quit IRC09:08
*** ykarel has joined #openstack-infra09:08
*** claudiub has quit IRC09:10
*** slaweq_ has joined #openstack-infra09:11
*** markvoelker has quit IRC09:15
*** kjackal has quit IRC09:15
*** slaweq_ has quit IRC09:16
*** wolverineav has quit IRC09:16
*** pblaho has joined #openstack-infra09:20
*** kjackal has joined #openstack-infra09:21
*** thorst_afk has joined #openstack-infra09:30
AJaegersshnaidm: see  https://review.openstack.org/498388 on one real hack for the cirros images ;(09:33
*** s-shiono has quit IRC09:33
*** thorst_afk has quit IRC09:35
sshnaidmAJaeger, nice.. although I just replaced url of it to hosted image in github09:37
*** bhavik1 has quit IRC09:40
*** alexchadin has joined #openstack-infra09:42
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci master: Fix report of promotion job in rdoproject.org  https://review.openstack.org/49827309:45
openstackgerritAntoni Segura Puimedon proposed openstack-infra/project-config master: kuryr octavia job: add barbican to PROJECTS  https://review.openstack.org/49840609:50
*** sree has quit IRC09:53
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci master: Fix report of promotion job in rdoproject.org  https://review.openstack.org/49827309:57
*** mat128 has joined #openstack-infra09:57
*** liujiong has quit IRC09:59
*** cuongnv has quit IRC10:03
*** alexchadin has quit IRC10:09
*** alexchadin has joined #openstack-infra10:09
*** wolverineav has joined #openstack-infra10:11
*** slaweq_ has joined #openstack-infra10:12
*** slaweq_ has quit IRC10:17
*** mwarad has quit IRC10:18
*** mwarad has joined #openstack-infra10:18
*** tosky has quit IRC10:21
*** SotK has quit IRC10:22
*** LindaWang has quit IRC10:22
*** SotK has joined #openstack-infra10:22
*** LindaWang1 has joined #openstack-infra10:22
*** mwarad has quit IRC10:24
*** baoli has joined #openstack-infra10:25
*** LindaWang1 has quit IRC10:27
*** lathiat has quit IRC10:27
*** lathiat has joined #openstack-infra10:28
*** baoli has quit IRC10:30
*** gixxer1k has quit IRC10:31
*** thorst_afk has joined #openstack-infra10:31
*** ykarel has quit IRC10:34
*** jkilpatr has quit IRC10:36
*** thorst_afk has quit IRC10:36
*** Julien-zte has quit IRC10:37
*** hashar is now known as hasharLunch10:38
*** baoli has joined #openstack-infra10:39
*** Julien-zte has joined #openstack-infra10:39
*** d0ugal has joined #openstack-infra10:40
*** tosky has joined #openstack-infra10:43
*** stakeda has quit IRC10:43
*** gcb has quit IRC10:51
*** jamielennox has quit IRC10:51
*** jamielennox has joined #openstack-infra10:52
*** lucasagomes is now known as lucas-hungry10:55
*** ykarel has joined #openstack-infra10:57
*** jkilpatr has joined #openstack-infra10:58
openstackgerritzhurong proposed openstack-infra/project-config master: Add test job for solum with separate tempest plugin  https://review.openstack.org/49842311:03
*** sshnaidm is now known as sshnaidm|afk11:05
*** anthonyper has quit IRC11:09
*** anthonyper has joined #openstack-infra11:09
*** slaweq_ has joined #openstack-infra11:13
*** yamamoto has quit IRC11:15
*** markvoelker has joined #openstack-infra11:15
*** yamamoto has joined #openstack-infra11:16
*** yamamoto has quit IRC11:16
*** slaweq_ has quit IRC11:18
*** jpena is now known as jpena|lunch11:22
*** udesale has joined #openstack-infra11:31
*** yamamoto has joined #openstack-infra11:31
*** dave-mccowan has joined #openstack-infra11:32
*** szaher has quit IRC11:32
*** sree has joined #openstack-infra11:32
*** thorst_afk has joined #openstack-infra11:33
*** udesale__ has quit IRC11:33
*** ldnunes has joined #openstack-infra11:34
*** szaher has joined #openstack-infra11:36
*** thorst_afk has quit IRC11:40
mhaydenclarkb: if i read your message correctly, you'd prefer to mirror some of those repos rather than reverse proxy?11:41
*** markvoelker has quit IRC11:50
*** pgadiya has quit IRC11:51
*** kjackal has quit IRC11:53
*** alexchadin has quit IRC11:57
*** alexchadin has joined #openstack-infra11:58
*** alexchadin has quit IRC12:01
*** LindaWang has joined #openstack-infra12:01
*** alexchadin has joined #openstack-infra12:01
*** kjackal has joined #openstack-infra12:02
*** udesale has quit IRC12:03
*** dprince has joined #openstack-infra12:04
*** lucas-hungry is now known as lucasagomes12:11
fungi#status log restarted ptgbot service on eavesdrop at 11:29 utc; was disconnected from freenode 2017-08-26 02:29 utc due to an irc ping timeout12:11
openstackstatusfungi: finished logging12:11
*** aspiers has quit IRC12:13
*** aspiers has joined #openstack-infra12:13
*** jcoufal has joined #openstack-infra12:14
*** slaweq_ has joined #openstack-infra12:14
*** lifeless has quit IRC12:15
*** jpena|lunch is now known as jpena12:16
*** lifeless has joined #openstack-infra12:17
*** trown|outtypewww is now known as trown12:19
*** slaweq_ has quit IRC12:20
*** eranrom has joined #openstack-infra12:21
*** zhurong has quit IRC12:23
*** gouthamr has quit IRC12:24
*** claudiub has joined #openstack-infra12:26
*** rlandy has joined #openstack-infra12:27
*** claudiub|2 has quit IRC12:29
*** sdague has joined #openstack-infra12:29
*** thorst_afk has joined #openstack-infra12:30
*** claudiub|2 has joined #openstack-infra12:33
*** claudiub has quit IRC12:36
*** dave-mccowan has quit IRC12:38
openstackgerritJan Priessnitz proposed openstack/python-jenkins master: Return variable type str from Jenkins._build_url()  https://review.openstack.org/49844812:39
*** hasharLunch is now known as hashar12:40
*** slaweq has quit IRC12:42
*** slaweq has joined #openstack-infra12:45
*** gouthamr has joined #openstack-infra12:45
*** kgiusti has joined #openstack-infra12:47
*** rhallisey has joined #openstack-infra12:48
*** dave-mccowan has joined #openstack-infra12:48
*** ykarel has quit IRC12:48
*** ykarel has joined #openstack-infra12:49
*** alexchadin has quit IRC12:50
*** alexchadin has joined #openstack-infra12:51
*** ykarel has quit IRC12:53
*** eumel8 is now known as eumel8_away12:58
pabelangerAJaeger: sshnaidm|afk: for zuulv3, I think we'll need to consider mirroring cirros images too. We've talked about removing our cached images from disk, so it make sense to either add it to tarballs.o.o reverse proxy or setup properly mirror13:00
*** bobh has joined #openstack-infra13:01
*** markvoelker has joined #openstack-infra13:01
*** dhajare has quit IRC13:03
*** mat128 has quit IRC13:03
*** abelur_ has quit IRC13:04
*** esberglu has joined #openstack-infra13:06
*** sshnaidm|afk is now known as sshnaidm13:08
sshnaidmpabelanger, yes, it would be great to have a local mirror for them13:08
pabelangerYa, if we stop caching them, we'll need to do that for sure.13:09
toskyit's only *slighly* critical, as the job explosions today show :)13:13
cloudnullmornings13:13
*** skelso has joined #openstack-infra13:13
openstackgerritMerged openstack/ptgbot master: Let room operators clean up themselves  https://review.openstack.org/49532013:13
openstackgerritMerged openstack/ptgbot master: Shut up now, ptgbot  https://review.openstack.org/49532113:14
pabelangertosky: well, they are cached today. So no job in the gate actually needs to download cirros, it lives in /opt/cache/files13:14
toskyso the issue is mostly 3rd-party CIs (like RDO)13:15
pabelangertosky: 3rdparty will also cache it, if using our nodepool elements13:15
*** slaweq_ has joined #openstack-infra13:15
toskyuhm13:15
toskysshnaidm: so I guess that the issue on oooq jobs is different, if the cached jobs can't be used?13:16
sshnaidmtosky, sorry?13:20
*** slaweq_ has quit IRC13:20
*** gixxer1k has joined #openstack-infra13:22
*** esberglu has quit IRC13:22
*** esberglu has joined #openstack-infra13:22
*** esberglu has quit IRC13:23
*** esberglu has joined #openstack-infra13:23
fungimaybe nodes there don't use our cache-devstack dib element? (which is what caches images devstack tells us are important to have)13:24
*** srobert has joined #openstack-infra13:24
toskysshnaidm: I maybe incorrectly read some patches to fix tripleo jobs which used some additional mirrors instead of using the cached image, so I thought that maybe the images do not come from nodepool13:27
pabelangerfungi: dmsimard would know for sure, they should be the same images as upstream for reasons like this.13:27
*** esberglu has quit IRC13:27
fungihttp://git.openstack.org/cgit/openstack-infra/project-config/tree/nodepool/elements/cache-devstack/extra-data.d/55-cache-devstack-repos#n107 -> http://git.openstack.org/cgit/openstack-dev/devstack/tree/tools/image_list.sh -> http://git.openstack.org/cgit/openstack-dev/devstack/tree/stackrc#n71513:28
openstackgerritAttila Darazs proposed openstack-infra/tripleo-ci master: Use shell based DLRN API client in promotion jobs  https://review.openstack.org/49827313:28
fungithat's the chain for how we get them onto our images13:28
sshnaidmtosky, fungi the rdo-ci images are not from nodepool, it's pure centos slave afaik13:28
fungigot it13:28
toskyI see13:28
pabelangerhttps://review.rdoproject.org/nodepool-log/upstream-centos-7.log is the images that would have it in RDO13:29
openstackgerritJan Priessnitz proposed openstack/python-jenkins master: Return variable type str from Jenkins._build_url()  https://review.openstack.org/49844813:29
sshnaidmpabelanger, maybe it's worth to include cirros when we build promoting images?13:29
sshnaidmpabelanger, or it's overhead..13:29
pabelangersshnaidm: I'm not sure how that is done. Do you have a log file?13:30
sshnaidmpabelanger, of building image? just a sec..13:31
*** ykarel has joined #openstack-infra13:31
sshnaidmpabelanger, http://logs.openstack.org/periodic/periodic-tripleo-ci-centos-7-ovb-ha-oooq/0c086b8/logs/undercloud/home/jenkins/overcloud-full.log.txt.gz13:32
*** mriedem has joined #openstack-infra13:33
pabelangersshnaidm: ya, so any perodic job in the gate will have the cached cirros, so if you want to in turn add that to the images you build, that would be okay. But, do you need it on overcloud image or from undercloud (I assume that is where tempest is run).13:34
sshnaidmpabelanger, need to think about it..13:38
*** sree_ has joined #openstack-infra13:38
*** esberglu has joined #openstack-infra13:38
*** sree_ is now known as Guest5067313:38
*** gcb has joined #openstack-infra13:38
*** alexchadin has quit IRC13:40
*** sshnaidm is now known as sshnaidm|bbl13:40
*** sree has quit IRC13:40
*** Goneri has joined #openstack-infra13:40
*** markvoelker has quit IRC13:40
*** lbragstad has joined #openstack-infra13:42
openstackgerritLucas Alvares Gomes proposed openstack-infra/project-config master: Add Neutron API job to networking-ovn  https://review.openstack.org/49846613:42
*** mwarad has joined #openstack-infra13:45
*** priteau has joined #openstack-infra13:47
fricklerFYI, smoser has reregistered cirros-cloud.net, so ppl can stop doing funny workarounds (at least for the next three years ;)13:47
fungiheh13:48
*** Guest50673 has quit IRC13:50
*** sree has joined #openstack-infra13:51
*** felipemonteiro has joined #openstack-infra13:51
*** trown is now known as trown|brb13:51
*** mwarad has quit IRC13:51
*** eumel8 has joined #openstack-infra13:54
*** sree has quit IRC13:55
*** kjackal has quit IRC13:55
*** kjackal has joined #openstack-infra13:56
openstackgerritMonty Taylor proposed openstack-infra/zuul feature/zuulv3: Add unittest for secrets data  https://review.openstack.org/48491113:56
*** mat128 has joined #openstack-infra13:56
*** trown|brb is now known as trown13:56
*** mat128 has quit IRC13:57
openstackgerritMerged openstack-infra/system-config master: Serialize ansible-playbooks to 10% of nodes in infracloud  https://review.openstack.org/49805413:58
openstackgerritJan Priessnitz proposed openstack/python-jenkins master: Return variable type str from Jenkins._build_url()  https://review.openstack.org/49844813:58
*** felipemonteiro has quit IRC13:58
*** hongbin has joined #openstack-infra14:00
*** efoley_ is now known as efoley14:01
openstackgerritMerged openstack-infra/system-config master: Set disablereuse=On for buildlogs.centos  https://review.openstack.org/49810414:01
pabelangerAJaeger: jeblair: I've been trying to get up to speed on our afsdocs jobs in zuulv2.5, I'm having some trouble understanding the concept of the .root-marker file. Do we document that some place?14:05
*** _ari_ is now known as _ari_|conf14:05
*** skelso has quit IRC14:06
*** ldnunes has quit IRC14:08
*** david-lyle has quit IRC14:10
jeblairpabelanger: it's explained in the afs docs spec14:10
pabelangerjeblair: great, looking now14:10
*** alexchadin has joined #openstack-infra14:13
*** skelso has joined #openstack-infra14:13
*** armax has joined #openstack-infra14:16
*** slaweq_ has joined #openstack-infra14:16
*** rbrndt has joined #openstack-infra14:18
*** jbernard has quit IRC14:20
*** slaweq_ has quit IRC14:21
*** ldnunes has joined #openstack-infra14:21
*** jbernard has joined #openstack-infra14:21
*** jbernard has quit IRC14:22
*** links has quit IRC14:22
openstackgerritAde Lee proposed openstack-infra/project-config master: Add novajoin repo for tempest tests  https://review.openstack.org/49640714:23
*** armax has quit IRC14:27
*** jbernard has joined #openstack-infra14:27
*** xyang1 has joined #openstack-infra14:29
*** beekneemech is now known as bnemec14:31
*** signed8bit has joined #openstack-infra14:31
pabelangerjeblair: AJaeger: so today we have a JJB marco called add-docs-root-marker: http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/macros.yaml#n320 Which just contains some zuul variables, but doesn't look like zuul_afs actually read the root-marker file directly, is that correct?14:31
*** skelso has quit IRC14:36
AJaegerpabelanger: the goal of the macro is only to create teh file in the filesystem14:42
pabelangerAJaeger: ya, so in zuulv3, should we just have jobs created the filter file directly or built that in the playbook / role?14:43
AJaegerpabelanger: don't know details about implmentation - but what I know is the implementation checks for existinece of that file.14:43
AJaegerpabelanger: best discuss further with jeblair14:43
*** rama_y has joined #openstack-infra14:43
*** sree has joined #openstack-infra14:45
fungipabelanger: as far as i'm aware it's just got metadata in it for the benefit of manual troubleshooting, but its real purpose is simply to act as a flag for the root of each documentation tree so that when we do recursive deletion we don't recurse into any subtrees containing one14:46
fungipretty certain the content of the file isn't used by any job, only existence of the file itself14:46
pabelangerokay, that is how I am reading zuul_afs ansible library too14:47
AJaegerfungi, you're right - content is just for troubleshoot. it is not parsed. Only existence of file is important14:47
pabelangerjust wanted to make sure it wasn't use for some other reason14:48
*** sree has quit IRC14:49
fungithat said, i wonder if the data we put it in _might_ be useful embedded into the theme somehow... for any tree it includes the name of the responsible git repo and the git revision from which it was built14:50
AJaegerfungi, openstackdocstheme codes git repo and git revision already in it. without using the root-marker14:51
fungiwe could likely embed other useful details in there like aggregate size, file count, date/time (though that last one could be obtained from the filesystem metadata too)14:51
fungiAJaeger: yeah, i agree the bits in there right now are mostly redundant from a user-facing standpoint14:52
AJaegerremember that openstack-manuals pushes several documents and index files with a single .root-marker - so there's not a relationship of one root-marker per guide14:52
AJaegerfungi, yes, they are redundant - just debugging help14:52
AJaegerwhich we haven't needed often - if at all14:53
*** rama_y has quit IRC14:53
pabelangerAJaeger: so, looking at nova-docs-unified-ubuntu-xenial is has a pretty simple .root-marker setup. Could you recommend a more complex job, that might have multiple .root-markers?14:53
AJaegerpabelanger: none has multiple root-markers14:53
AJaegerpabelanger: What I mean is: openstack-manuals creates on it's own the root-marker (in openstack-manuals repo) covering several guides14:54
AJaegerpabelanger: that's the /.root-marker file14:54
pabelangerAJaeger: okay, any content in a job that you know would be excluded? Or do all jobs have a top-level .root-marker file?14:54
fungiright, so my thinking was the .root-marker file is useful for working out which repo owns which files and any build-time details (assuming there aren't multiple repos/jobs sharing a single .root-marker file, which i wouldn't expect as that seems like it would create brokenness)14:55
AJaegerpabelanger: wait, the stable branches of openstack-manuals have several root-marker files14:55
*** xarses has joined #openstack-infra14:55
AJaegerfungi: a root-marker file is unique14:56
fungiright, seemed like it would have to be, by design14:56
*** signed8bit has quit IRC14:56
pabelangerAJaeger: okay, I'll look into those jobs14:58
AJaegerpabelanger: http://git.openstack.org/cgit/openstack/openstack-manuals/tree/tools/build-all-rst.sh?h=stable/ocata#n59 and http://git.openstack.org/cgit/openstack/openstack-manuals/tree/tools/build-all-rst.sh#n4814:59
*** ijw has quit IRC15:00
AJaegerthis is the only odd one in the tree. security-doc and api-site also generate root-marker themselves15:00
*** alexchadin has quit IRC15:02
clarkbgood morning. Once I am settled in and my meeting is over I'd like to move forward on the chocolate ssl cert replacement. Looks like things are mostly quiet this morning15:02
clarkbmhayden: ya if they allow anonymous rsync might be nice to properly mirror the rpms for mariadb too since we mirror the debs.15:03
pabelangerAJaeger: okay, so still dropped into top-level directory of said folder, but no jobs would have a root-marker under a root-marker15:03
clarkbmhayden: at least for that one set of pacakges so that we don't end up hosting it in different ways for different platforms (but if we can't rsync we may just have ot live with that)15:03
openstackgerritMonty Taylor proposed openstack-infra/nodepool feature/zuulv3: WIP: Fix broken use of pre-existing cloud images  https://review.openstack.org/49805015:03
*** skelso has joined #openstack-infra15:04
clarkbttx: thank you for helping chenying15:05
*** mat128 has joined #openstack-infra15:08
*** iyamahat has joined #openstack-infra15:09
*** mat128 has quit IRC15:11
openstackgerritMonty Taylor proposed openstack-infra/nodepool feature/zuulv3: WIP: Fix broken use of pre-existing cloud images  https://review.openstack.org/49805015:11
AJaegerpabelanger: yes, either top-level or several lower-level.15:11
*** gcb has quit IRC15:15
*** david-lyle has joined #openstack-infra15:16
*** slaweq_ has joined #openstack-infra15:17
*** eumel8 has quit IRC15:19
*** eumel8 has joined #openstack-infra15:20
*** rhagarty has quit IRC15:21
*** gmann has quit IRC15:22
*** gmann has joined #openstack-infra15:22
*** slaweq_ has quit IRC15:22
*** skelso has quit IRC15:22
openstackgerritJohn Trowbridge proposed openstack-infra/tripleo-ci master: Use local cirros image in rdocloud as well  https://review.openstack.org/49849715:23
*** jamesmcarthur has joined #openstack-infra15:23
*** eharney has joined #openstack-infra15:24
*** rhagarty has joined #openstack-infra15:24
*** armax has joined #openstack-infra15:25
*** vhosakot has joined #openstack-infra15:28
*** LindaWang has quit IRC15:28
openstackgerritMonty Taylor proposed openstack-infra/nodepool feature/zuulv3: WIP: Fix broken use of pre-existing cloud images  https://review.openstack.org/49805015:29
*** sshnaidm has joined #openstack-infra15:30
clarkbin case everyone hasn't caught up on infra ml mail I will drop this here too. https://etherpad.openstack.org/p/infra-ptg-team-dinner Trying to plan some sort of low key dinner during ptg. Please add your availability if interested15:31
*** jistr is now known as jistr|off|mtg15:33
*** jistr|off|mtg is now known as jistr15:33
*** sshnaidm|bbl has quit IRC15:33
cmurphybiergarten ftw15:33
*** links has joined #openstack-infra15:34
*** martinkopec has quit IRC15:34
*** ramishra has quit IRC15:35
*** dhajare has joined #openstack-infra15:35
*** ijw has joined #openstack-infra15:36
*** ijw has joined #openstack-infra15:36
*** eumel8 has quit IRC15:39
*** eumel8 has joined #openstack-infra15:39
fungispeaking of, need to go grab some lunch. back soon15:39
*** markus_z has quit IRC15:41
openstackgerritMerged openstack-infra/nodepool feature/zuulv3: Revert "Allow launcher to stop quicker when asked"  https://review.openstack.org/49748015:42
*** eumel8 has quit IRC15:44
*** eumel8 has joined #openstack-infra15:45
*** bogdando has quit IRC15:45
*** dhajare has quit IRC15:46
*** pcaruana has quit IRC15:47
*** rama_y has joined #openstack-infra15:52
*** jascott1 has quit IRC15:53
clarkbinfra-root can I get reviews on https://review.openstack.org/#/c/498081/ and https://review.openstack.org/#/c/498082/ I would like to get chocolate ssl updated today. Feel free to not approve I can do that and get the private hiera updated and everything once I have the reviews15:54
*** kjackal has quit IRC15:55
*** kjackal has joined #openstack-infra15:56
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Run post playbooks on job timeout  https://review.openstack.org/49827015:57
jeblairclarkb: lgtm15:58
openstackgerritMerged openstack/gertty master: Speed up loading change screen  https://review.openstack.org/49760016:00
*** lucasagomes is now known as lucas-afk16:00
*** mat128 has joined #openstack-infra16:01
clarkbpabelanger: looks like the infracloud playbook is puppeting well (glancing at logs)16:01
*** mpeterson has quit IRC16:03
*** egonzalez has quit IRC16:04
pabelangerclarkb: great!16:05
*** felipemonteiro has joined #openstack-infra16:05
*** andreas_s has quit IRC16:07
openstackgerritJames E. Blair proposed openstack-infra/devstack-gate master: Zuul v3: add a devstack-legacy job  https://review.openstack.org/49769916:09
*** rcernin has quit IRC16:10
*** felipemonteiro has quit IRC16:10
openstackgerritMonty Taylor proposed openstack-infra/project-config master: Move scripts out of jenkins dir  https://review.openstack.org/49811016:11
openstackgerritMonty Taylor proposed openstack-infra/project-config master: Extract scripts copying into a role  https://review.openstack.org/49811116:11
mordredjeblair: ^^ fixed the last two and reordred them so that the dir name one doens't need to block anything and/or we can just ignore it16:14
*** mriedem1 has joined #openstack-infra16:15
clarkbI've approved the trust infracloud certs directly change for puppetmaster. Will confirm that clouds.yaml still works with osc then move to replacing the cert entirely with the new one16:16
*** mriedem has quit IRC16:16
*** slaweq_ has joined #openstack-infra16:18
pabelangerk16:18
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Fix job timeout docs  https://review.openstack.org/49851316:20
*** eharney has quit IRC16:20
openstackgerritMonty Taylor proposed openstack-infra/zuul feature/zuulv3: Fail early if people attempt to add zuul vars or secrets  https://review.openstack.org/48400016:22
*** slaweq_ has quit IRC16:23
*** annegentle has joined #openstack-infra16:24
*** markmcd has quit IRC16:26
*** links has quit IRC16:27
*** markmcd has joined #openstack-infra16:29
*** HeOS has quit IRC16:29
*** baoli has quit IRC16:30
*** tesseract has quit IRC16:30
*** baoli has joined #openstack-infra16:37
*** e0ne has quit IRC16:37
openstackgerritMerged openstack-infra/system-config master: Trust infracloud directly using public cert  https://review.openstack.org/49808116:39
*** jpich has quit IRC16:39
*** mat128 has quit IRC16:39
*** jpena is now known as jpena|off16:39
openstackgerritMerged openstack-infra/zuul-jobs master: Add role for adding ssh key to remote nodes  https://review.openstack.org/49810916:45
*** ociuhandu has joined #openstack-infra16:45
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Run post playbooks on job timeout  https://review.openstack.org/49827016:47
*** rcernin has joined #openstack-infra16:48
*** ociuhandu has quit IRC16:51
*** dprince has quit IRC16:52
*** mriedem1 is now known as mriedem16:53
EmilienMI'm not sure if it's me but Gerrit has been quite slow this morning16:54
*** aarefiev_afk is now known as aarefiev16:57
*** ralonsoh has quit IRC17:01
*** ykarel has quit IRC17:02
*** mat128 has joined #openstack-infra17:03
*** chlong has joined #openstack-infra17:07
*** thorre has quit IRC17:08
*** ykarel has joined #openstack-infra17:09
openstackgerritJesse Keating proposed openstack-infra/zuul-jobs master: Add a role to remove an ssh private key  https://review.openstack.org/49853017:11
*** thorre has joined #openstack-infra17:11
*** efoley has quit IRC17:14
*** ykarel has quit IRC17:16
*** HeOS has joined #openstack-infra17:19
*** trown is now known as trown|lunch17:20
*** baoli has quit IRC17:21
clarkbosc seems to show the certs still work fine17:22
clarkbEmilienM: melody says memory use has increased but doesn't look terrible yet, garbage collecting isn't running often according to melody17:22
EmilienMclarkb: what is melody ? (pardon my ignorance)17:23
clarkbEmilienM: its a java/jvm introspection tool17:24
fungiclarkb: is private hiera updated for 498082 or want someone to do that still?17:24
clarkbEmilienM: shows us memory use and garbage collection time and you can kill threads and see their stack traces etc17:24
fungijust got back from lunch and finished up with storm prep17:24
clarkbfungi: its not been updated, I was going to update that as soon as 498082 merges (but wanted to get first chagne in and tested)17:24
*** Swami has joined #openstack-infra17:25
EmilienMclarkb: this ? https://github.com/javamelody/javamelody/wiki17:25
clarkbEmilienM: ya17:25
clarkbfungi: I'll go ahead and approve it now17:25
EmilienMclarkb: TIL, thanks17:25
fungik17:25
*** yamahata has quit IRC17:26
*** iyamahat has quit IRC17:26
fungiEmilienM: we'd make that monitoring interface for gerrit public, except it includes buttons to kill stuff17:27
EmilienMoops17:27
EmilienMI guess it's a no-go ;-)17:27
fungiwe've had some discussions about maybe trying to get a net-snmp backend for it so we can add it to cacti instead17:27
clarkbya you can kill threads17:28
openstackgerritJames E. Blair proposed openstack-infra/devstack-gate master: Zuul v3: add a devstack-legacy job  https://review.openstack.org/49769917:28
EmilienMI don't think everyone needs access outside of infra-cores17:28
EmilienMit's a very specific monitoring and afik folks who really need this info are already infra-core17:28
clarkbinfra-root I am taking the hiera lock now to edit chocolate's ssl cert17:29
jeblairEmilienM: this is a collectively maintained infrastructure; everyone needs to know it.  :)17:29
fungiyeah, i don't particularly enjoy having secret knowledge17:30
openstackgerritPaul Belanger proposed openstack-infra/project-config master: Add publish-docs-to-afs role  https://review.openstack.org/49853617:30
pabelangerjeblair: ^mind a quick review on kerberos / afs token logic? I have yet to add in the rsync bits17:31
EmilienMjeblair: fair enough! other option is to fork the tool and remove the button that kill threads... But not sure we have time / resources to maintain it17:31
*** baoli has joined #openstack-infra17:32
*** dprince has joined #openstack-infra17:32
clarkbEmilienM: a semi common "gerrit is slow" cause is poor ipv6 connectivity between your web browser and gerrit. We've seen that a few times too since ipv6 is at times less reliable than one would hop17:36
dhellmannI need some advice on retiring a project. The Oslo team wants to drop oslosphinx, but we need to maintain the old versions until pike is EOL. Our retiring process doc in the infra guide doesn't seem to cover this case. Should I go ahead and delete everything in master, but leave the jobs running against the other branches? Or just leave everything as it is for now?17:38
*** bhavik1 has joined #openstack-infra17:39
*** ihrachys has joined #openstack-infra17:42
fungidhellmann: i guess you also want to allow for patches to stable branches?17:42
dhellmannyeah. I mean, I don't expect any, but I don't want to prohibit it.17:42
dhellmannreleases fall into the same category17:43
dhellmannI'm thinking of saying any gate-oslosphinx-.* jobs only run on stable branches17:43
fungii would probably do the deletion and replacement readme in master, and abandon any open changes for master, but keep jobs and acl retirement steps until the stable branches reach eol17:43
dhellmannoh, just leave project-config alone?17:43
dhellmannhrm, I guess the other thing is I have to keep the release notes build in place17:44
dhellmannso some parts of master have to still work17:44
*** e0ne has joined #openstack-infra17:44
fungiyeah, any change which reverts the mass deletion in master is likely to come to the attention of core reviewers as a concern anyway17:44
dhellmannthat's fair17:44
dhellmannthanks17:44
dhellmannfungi : I won't be able to land a patch to delete the contents of master if the jobs still run against master, will I?17:45
dhellmanndocs, pep8, py*, etc. will all fail17:46
dhellmannmaybe the thing to do is just leave it alone until pike is eol17:46
dhellmannI can update the readme17:46
*** sree has joined #openstack-infra17:50
fungidhellmann: er, yeah i guess you'd need some sort of branch exclusion for all its jobs and then add a custom no-op job for master17:50
fungithat gets complicated17:50
dhellmannyeah, I'm not going to bother17:50
dhellmannhttps://review.openstack.org/498543 updates the docs and https://bugs.launchpad.net/oslosphinx/+bug/1713543 will help us remember to deal with it later17:51
openstackLaunchpad bug 1713543 in oslosphinx "obsolete library needs to be retired" [Undecided,New]17:51
*** jascott1 has joined #openstack-infra17:52
openstackgerritPaul Belanger proposed openstack-infra/zuul-jobs master: Add create / destory roles for AFS tokens  https://review.openstack.org/49854717:53
*** iyamahat has joined #openstack-infra17:53
openstackgerritMerged openstack-infra/system-config master: Update infracloud-chocolate public ssl cert  https://review.openstack.org/49808217:53
*** sree has quit IRC17:54
*** e0ne has quit IRC17:54
*** eumel8 has quit IRC17:54
*** eumel8 has joined #openstack-infra17:55
*** yamahata has joined #openstack-infra17:55
*** Rockyg has joined #openstack-infra17:59
*** bhavik1 has quit IRC18:00
*** jamesmcarthur has quit IRC18:02
jeblairdhellmann: you could leave tox.ini in master but make those job sections a no-op18:03
jeblairer "tox environments" is the term of art i should have used ^ :)18:03
dhellmannjeblair : that would work ok for everything except release notes, which only builds from master. I'm not sure how much of the python packaging bits I'd have to leave in place to make that work18:04
jeblairdhellmann: gotcha18:04
dhellmannand really it's probably not worth worrying about18:04
dhellmannI thought if there was precedent I could follow it, but I'm not really in Trailblazer mode today :-)18:05
openstackgerritMichal Jastrzebski (inc0) proposed openstack-infra/project-config master: Increase Kolla timeout on jobs  https://review.openstack.org/49855018:05
jeblairpabelanger: updated role in 498547 lgtm18:06
smcginnisAre there issues with gate? Over 5 hours for a patch to make it through the gate queue.18:06
pabelangerjeblair: thanks18:06
clarkbreleasing hiera lock now18:08
*** dtantsur is now known as dtantsur|afk18:10
*** xarses has quit IRC18:11
*** lihi has quit IRC18:11
AJaegerdhellmann: remove also any stable periodic jobs like requirements or translations - if tehy exist18:12
fungiAJaeger: well, except in this case he wants to keep stable branches open and retire master18:12
dhellmannAJaeger : I'm going to just update the README in master to say the package is obsolete and stop there for now. The rest of it is getting more complicated than is worth worrying about.18:12
dhellmannI've filed a bug. Someone can do the full retirement in 18 months or whatever18:13
AJaegerdhellmann, fungi, I see - ok, let's leave this way18:13
*** ijw has quit IRC18:20
*** sshnaidm is now known as sshnaidm|afk18:21
*** slaweq_ has joined #openstack-infra18:21
pabelangerjeblair: mordred: AJaeger: Did we want to bikeshed on zuulv3 afs docs publisher job name? Any suggestions18:22
*** slaweq_ has quit IRC18:26
* AJaeger is not joining a bikeshed - and calls it a day instead ;)18:26
jeblairpabelanger: push up a proposal18:27
pabelangerk18:27
AJaegerteam, I'll be travelling Wednesday to Sunday with little internet access...18:27
*** trown|lunch is now known as trown18:30
openstackgerritJames E. Blair proposed openstack-infra/devstack-gate master: Zuul v3: add a devstack-legacy job  https://review.openstack.org/49769918:33
openstackgerritJames E. Blair proposed openstack-infra/devstack-gate master: Remove JOB_NAME from reproduce  https://review.openstack.org/49855918:33
openstackgerritwes hayutin proposed openstack-infra/tripleo-ci master: add logs for virt-customize  https://review.openstack.org/49856218:37
clarkbthat was weird local access point turned into ~1s rtt18:40
clarkbsmcginnis: I think its just load/demand18:40
clarkbsmcginnis: at least as of last week the vast majority of the gate issues seemed to be under control18:40
*** ijw has joined #openstack-infra18:41
smcginnisclarkb: OK, good. Seems slower to me, but that's probably just my perception.18:41
clarkbsmcginnis: we've lost quite a bit of capacity in recent weeks so that is part of it18:41
smcginnisTrue. I hope that's not a long term thing.18:42
clarkbthats interesting controller00.chocolate unpings now18:43
clarkbvanilla is still up18:43
clarkbinfra-root fyi ^ chocolate is down and doesn't apear to be due to my ssl changes18:43
fungihuh18:44
jeblairchocolate is not in cacti18:44
clarkbI checked it was up and running ebfore doing ssl change and was even ssh'd in but that connection died18:44
clarkb(unsure if due to local networking or remote)18:44
fungior rising flood waters18:45
fungii'm getting a fair amount of packet loss to vanilla18:46
clarkbya perhps it was on a different pdu or similar and had to be taken down?18:46
*** ijw has quit IRC18:46
fungier, controller00.vanilla i mean18:46
pabelangercompute090.chocolate.ic.openstack.org is still up18:47
fungii have a chart with the serial number to rack mapping, lemme dig that up18:47
clarkblooking at traceroute there are two additional hops to get to vanilla that are blackholed to get to chocolate18:47
clarkbso the instance may be up and could be a networking problem18:47
*** eharney has joined #openstack-infra18:48
fungii thought they were all behind the same router though18:49
fungihttp://paste.openstack.org/show/613968/18:49
fungithough if we don't know the serial number or ilo ip/mac then i guess that table doesn't help much18:50
*** ijw has joined #openstack-infra18:51
fungii guess bifrost/ironic has the ilo address mapping?18:51
clarkbya we have it written down in system-config somewhere18:52
clarkb00:9c:02:a5:41:3c18:53
*** armax has quit IRC18:53
clarkbis the ilo mac18:53
clarkbhrm that said the last thing in the infracloud log is run puppet18:54
fungiyeah, ironic node-show just gave me ipmi_address 10.12.83.52 for it18:54
clarkbon chocolate controller0018:54
fungiwhich matches the mac you found18:55
*** ijw has quit IRC18:55
fungiSerial USE237E1LC Rack B05 U-Location 118:56
clarkbunfortunately we only have logged that puppet was running there not the puppet output18:56
clarkbI can't see how changing the ssl content would affect ssh or ping though18:57
clarkband puppet was running on that node prior (checked that this morning)18:57
clarkbso maybe it is just coincidence?18:58
clarkbanyways to the ilo18:58
fungicould tickle a processor bug i suppose18:58
clarkbI'm on the ilo19:00
clarkbI can get a login prompt to the opersating system (but we don't do local passworded accounts so thats about as far as I can get)19:01
clarkbmakes me wonder if it isn't a networking problem19:01
fungiso maybe something shot its network stack19:02
fungior, yeah, problem external to the host19:02
*** adisky__ has quit IRC19:02
dmsimardjd_: your osjenkins bot to check on the queues would be useful right about now :)19:03
fungiclarkb: if rebooting it doesn't clear the issue, we should be able to boot passwordless with a substitute init (or normal single-user if it's not set up to require the root password) and investigate via the out-of-band connection19:03
*** ijw has joined #openstack-infra19:03
jd_dmsimard: how's that?19:03
dmsimardjd_: queues are having a hard time now19:03
dmsimardjd_: I tried to look at the account but the account no longer exists :(19:04
clarkbfungi: ya I'm trying to see if the ilo will give me link state first19:04
fungidmsimard: only about a 5.5-hour backlog at the moment... we've had far, far worse in the past19:05
jd_dmsimard: https://twitter.com/OpenStackStatus ?19:05
clarkbfungi: I've found the interface but not finding a way to see link state yet19:05
openstackgerritJames E. Blair proposed openstack-infra/devstack-gate master: Zuul v3: add a devstack-legacy job  https://review.openstack.org/49769919:05
dmsimardjd_: ohhh, was it renamed ? I remember it being osjenkins before (and that's also the acconut being referenced by https://github.com/jd/osjenkins-twitter )19:05
jd_dmsimard: yeah I think I renamed it because there was no more "jenkins" around :)19:06
dmsimardfungi: oh, definitely not that bad but we're definitely seeing the ~900VM cap in the graphs19:06
dmsimardjd_: makes sense :)19:06
fungiyup19:06
openstackgerritDavid Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Set base environment as python3  https://review.openstack.org/49625119:06
*** eranrom has quit IRC19:07
*** jamesmcarthur has joined #openstack-infra19:08
*** Rockyg has quit IRC19:09
clarkbya I'm not seeing any way to determine link state from the ilo (that seems like it would be a useful feature to have)19:09
clarkbinfra-root I guess our option here is to reboot?19:10
fungiyeah, that's my vote19:10
jeblair+19:10
clarkbok it is rebooting now. Based on past experience I expect about 10 minutes to boot19:12
*** annegentle has quit IRC19:12
clarkbits up and ssh works, that was really really weird19:16
clarkbalso booted quickly19:16
*** ijw has quit IRC19:18
*** marst has joined #openstack-infra19:18
clarkbhttp://paste.openstack.org/show/619672/19:19
clarkbpuppet triggered a networking refresh which ended up taking down our interface?19:19
*** slaweq_ has joined #openstack-infra19:20
fungimaybe we have a race condition between bridges and vlan subinterfaces?19:20
clarkbopenstack services are unhappy (I think cert update only half applied due to race in getting in private side of data or due to loss of networking?)19:21
fungiyeah, i expect it'll need a followup pass from puppet19:22
fungiyou could temporarily sudo passwd and give yourself a password so you can log in through the ilom19:22
clarkboh I'm in via ssh19:23
fungiin case it goes back to broken again19:23
clarkboh19:23
fungissh won't help you much if the next puppet update takes eth3.2551 down again19:23
clarkbwell now I've got the ssl cert updated so I think puppet will noop19:23
fungiahh, wfm19:24
clarkbI'm starting services now19:24
clarkblooks like services have automatically started again with cert in place19:26
clarkbso ya I think what happened was puppet updated the cert (which it did not do on vanilla, I did it on vanilla) this then due to puppet resource deps caused a trigger on our networking which then broke19:27
fungiweird that cert updates would trigger restarting19:29
clarkbcontrol plane looks mostly good. Now we need puppet to update all the compute hosts which should get us back to normal19:29
fungithough i don't doubt that somewhere deep within our spaghetti-like dependency chain they do19:29
fungier, would trigger restarting the whole network stack i mean19:30
cmurphythat is pretty strange19:31
clarkboh19:32
clarkbits nova that triggers a networking refresh but because we ahve that weird setup it also refreshes our bridge19:32
clarkbI don't know why nova would need a networking refresh since we use neutron19:33
fungifixed>mitaka would be my bet19:34
clarkbanyways. I think the next ansible-puppet run against infracloud should work and get the computes up and running based on how vanilla went19:34
clarkband once computes are up and running again we should be set19:34
*** ijw has joined #openstack-infra19:35
clarkbkeystone does appear to be up and functioning though :) you can hit it for catalog listing19:36
clarkbfungi: ya I'm wondering if /e/n/i is not sequencing the teardown properly which results in an error and thus we don't do start up again19:40
mgagneAny encountered this bug with dib and xenial? https://bugs.launchpad.net/diskimage-builder/+bug/168408119:40
openstackLaunchpad bug 1684081 in diskimage-builder "UnicodeDecodeError with xenial on custom package-installs" [Undecided,New]19:40
mgagneit seems to crash when packages with utf-8 within their description are installed19:40
*** armax has joined #openstack-infra19:40
mgagneusing diskimage-builder==2.8.019:41
clarkbmgagne: probably a valid bug in package-installs-v219:41
openstackgerritMerged openstack-infra/nodepool master: Clarify diskimage names in docs  https://review.openstack.org/49078119:41
mgagneyes, trying to see what kind of fix I need to apply. I tried injecting PYTHONIOENCODING=utf-8 without much success. like that scientist dog, I don't know what I'm doing19:43
openstackgerritJames E. Blair proposed openstack-infra/devstack-gate master: Zuul v3: add a devstack-legacy job  https://review.openstack.org/49769919:46
openstackgerritDavid Moreau Simard proposed openstack-infra/devstack-gate master: Move network overlay configuration to Ansible playbooks  https://review.openstack.org/43593319:46
*** florianf has quit IRC19:46
clarkbmgagne: any idea if that is running under python3 or python2?19:47
clarkbmgagne: you can check by executing dib-python -v19:48
clarkboh its in the traceback its python319:48
*** e0ne has joined #openstack-infra19:48
mgagnewithin the image or outside? nodepool is running trusty but image is xenial which I suspect is running python3, I'm not sure about that one19:48
mgagnefound: apt-get --option Dpkg::Options::=--force-confold --option Dpkg::Options::=--force-confdef --assume-yes install git linux-image-generic default-jre python3-dev dkms19:48
mgagneso I suspect python3-dev19:49
mgagneexport DIB_PYTHON=python319:49
mgagneare you suggesting I downgrade to python2 to fix the problem?19:49
fungiwell, or fix the problem ;)19:50
mgagnehehe =)19:50
openstackgerritDavid Moreau Simard proposed openstack-infra/devstack-gate master: Replace inventory setup with a playbook that sets-up v3 hostvars  https://review.openstack.org/49820719:52
clarkbmgagne: the behavior is different between the two versions which is why I was curious19:53
openstackgerritDavid Moreau Simard proposed openstack-infra/devstack-gate master: Replace inventory setup with a playbook that sets-up v3 hostvars  https://review.openstack.org/49820719:53
openstackgerritDavid Moreau Simard proposed openstack-infra/devstack-gate master: Move network overlay configuration to Ansible playbooks  https://review.openstack.org/43593319:53
clarkbI'm guessing that python3 implicitly attempts an ascii conversion when reading the binary stdout pipe19:53
mgagneclarkb: yea, no problem building with trusty with same dib elements19:53
clarkbwhereas python2 would just treat it as binary the whole way19:53
clarkband it isn't until python3.6 where subprocess.popen grows and encoding parameter19:53
*** skelso has joined #openstack-infra19:54
mgagnehmmm I don't see python3.6 available on xenial, only python 3.519:55
mgagneso I extracted the function process_output in a script and ran it. command I used is cat utf8.txt with utf-8 in it. I ran it in a console, no problem.19:56
mgagnebut I had LANG=en_US.UTF-819:56
mgagneunset LANG and now it crashes19:57
clarkbI have had to restart rabbitmq-server on chocolate controller to pick up the ssl cert change19:58
clarkbpuppet seemed out to lunch and wasn't doing it19:58
clarkbmgagne: I think we can likely modify it to just operate in binary?19:58
clarkbmgagne: since it looks like all it is doing is echoing it back to the terminal that should be safe I think19:59
mgagneclarkb: I'm far from familiar with encoding stuff =)19:59
clarkbmgagne: if you give me a bit I can post a diff you can test19:59
fungimgagne: yeah, xenial predates python 3.6. there are some custom-built packages available for it i think, but it wasn't officially introduced until 17.0419:59
*** jkilpatr has quit IRC20:00
openstackgerritMonty Taylor proposed openstack-infra/nodepool feature/zuulv3: WIP: Fix broken use of pre-existing cloud images  https://review.openstack.org/49805020:00
openstackgerritJames E. Blair proposed openstack-infra/devstack-gate master: Zuul v3: add a devstack-legacy job  https://review.openstack.org/49769920:01
clarkblooks like glance-api is in a restart loop and not logging why20:02
*** mat128 has quit IRC20:03
clarkbERROR: There is a problem with your private key /etc/glance/ssl/private/controller00.chocolate.ic.openstack.org.pem.  Please verify it. OpenSSL error: []20:03
clarkbthat is from the upstart logs20:03
clarkbmaybe its permissions?20:03
*** e0ne has quit IRC20:04
*** eranrom has joined #openstack-infra20:05
clarkbbah20:05
clarkbI missed a - in the private hiera20:05
slaweq_hello devstack cores :) can You maybe take a look on my small patch: https://review.openstack.org/#/c/498252/ ?20:06
slaweq_thx in advance20:06
clarkbbut that didn't fix it either20:08
openstackgerritJames E. Blair proposed openstack-infra/project-config master: Zuul v3: Add some more projects required by devstack-gate  https://review.openstack.org/49858620:08
inc0hey guys, how can we recheck merge gates?20:08
inc0there is a change that needs to be merged asap and gates times out20:08
jeblairclarkb, mordred:   https://review.openstack.org/498586  is a blocker20:09
*** camunoz has joined #openstack-infra20:11
*** eranrom has quit IRC20:12
clarkbI've confirmed the public cert and private key match each other20:15
openstackgerritGage Hugo proposed openstack-infra/project-config master: Skip non-doc jobs in certain cases  https://review.openstack.org/49402720:18
*** felipemonteiro has joined #openstack-infra20:19
openstackgerritPaul Belanger proposed openstack-infra/zuul-jobs master: Create upload-afs role  https://review.openstack.org/49858820:19
* inc0 is thinking recheck of single job would be useful20:19
*** Goneri has quit IRC20:20
clarkbinc0: we have avoided that because it would allow you to lock in results. We know that we have already merged more than one gate breaking (and real world bug) simply due to rechecks being tried over and over until lucky20:20
clarkbinc0: I'm not sure I understand the question though, rechecks are the same regardless of change?20:20
clarkbslaweq_: devstack devs mostly hang out in #openstack-qa as that is a qa project20:21
inc0well, issue we have now is that out of 3 build jobs, one fails due to timeout20:21
inc0so we can't merge something we need for release20:21
slaweq_clarkb: thx, I will ask there :)20:21
inc0https://review.openstack.org/#/c/495555/7 <- this thing20:21
inc0I've already proposed patch to double timeout on gates until we sort it out and reliably build within 90min20:23
pabelangeris it centos that is failing?20:23
*** annegentle has joined #openstack-infra20:23
inc0oraclelinux in this case pabelanger (if you were asking me;)20:24
openstackgerritGage Hugo proposed openstack-infra/project-config master: Skip integration/non-doc jobs in certain cases  https://review.openstack.org/49401820:24
openstackgerritMerged openstack-infra/project-config master: Zuul v3: Add some more projects required by devstack-gate  https://review.openstack.org/49858620:24
pabelangerinc0: for example: http://logs.openstack.org/55/495555/7/gate/gate-kolla-dsvm-build-oraclelinux-source-centos-7/cba7241/console.html#_2017-08-28_16_34_28_19549420:24
inc0oraclelinux uses centos repos for few things20:25
pabelangerinc0: you shouldn't be downloading directly from upstream repos, we have regional reverse proxy for buildslogs.centos.org, so that will speed up things. We also have AFS mirrors for mirror.centos.org too. Locally, so that will avoid hitting upstream and downloading everything each job run20:25
inc0mitaka? hmm fun looks like we have bug there20:25
inc0ok good point pabelanger I'll double check why oraclelinux doesn't use it20:26
pabelangerinc0: do you pull any containers from docker.io during that process too?20:27
clarkbok it was the missing - in the header of the key. But we use a copy of the file in each of the service dirs20:27
clarkbI had only updated the master file not the copies...20:27
clarkbpuppet should rerun shortly and this time I expect that it will be happy20:27
inc0yes, but I'm fairly sure we use cache20:28
*** sree has joined #openstack-infra20:29
*** sree has quit IRC20:33
*** spzala has quit IRC20:36
mgagneclarkb: so dib is crashing on installation of the ca-certificates-java pkg which outputs utf-8 during installation: Adding debian:Certinomis_-_Autorité_Racine.pem20:36
inc0pabelanger: https://review.openstack.org/#/c/498592/ damned typo ;(20:36
mgagneI tried exporting LANG without much success20:37
*** srobert_ has joined #openstack-infra20:37
pabelangerinc0: k, that will help. You'll likely need to add some logic to use reverse proxy cache for trunk.rdoproject.org and buildlogs.centos.org. /etc/ci/mirror_info.sh has all the information20:38
clarkbmgagne: ya as soon as I get infracloud happy again I will work on a patch to test20:39
mgagneclarkb: yea, no rush here20:39
*** srobert has quit IRC20:41
clarkbok I restarted all controller openstack services now that the proper cert is in place20:41
clarkband puppet is puppeting on the computes20:41
clarkbI think we are seeing apis work20:41
openstackgerritPaul Belanger proposed openstack-infra/project-config master: Add afsdocs_secret secret for publishing  https://review.openstack.org/49826720:41
clarkband should see computes successfully boot instances as soon as we get computes updated20:41
*** srobert_ has quit IRC20:42
pabelangeryay20:42
pabelangerclarkb: puppet running much faster on infracloud now. 10% serial seems to be working great20:44
*** eumel8 has quit IRC20:47
clarkbpabelanger: ya it has worked well20:49
*** trown is now known as trown|outtypewww20:49
clarkbmgagne: just testing things locally I can't reproduce that exception I think because sys.getdefaultencoding() returns utf-8 not ascii20:49
clarkbmgagne: I'm testing with output of ls whcih includes utf8 snowman20:49
mgagneyep20:49
mgagnethat's what I found in my tests (with sys.getdefaultencoding)20:49
clarkbmgagne: I'm gonna try forcing sys.getdefaultencoding to ascii and see if it changes20:49
*** priteau has quit IRC20:50
clarkbbut first lunch20:50
*** ijw has quit IRC20:51
pabelangerjust see compute090.chocolate delete an instance20:51
pabelangerSSL looks to be working again20:51
*** jcoufal has quit IRC20:51
mgagneclarkb: ok, I "fixed" it but exporting LANG and LC_ALL20:53
*** chlong has quit IRC20:53
mgagneexport LANG=en_US.UTF-8 export LC_ALL=en_US.UTF-820:53
*** ijw has joined #openstack-infra20:54
*** dprince has quit IRC20:54
*** ijw has quit IRC20:56
*** ociuhandu has joined #openstack-infra20:57
*** ijw has joined #openstack-infra20:57
*** felipemonteiro has quit IRC20:58
*** felipemonteiro has joined #openstack-infra20:58
pabelangerclarkb: looking at http://status.openstack.org/elastic-recheck/#1686542 I am thinking we might want to consider bring a few more nodes of infracloud online.  infracloud-chocolate only has 5 hits in 10 days, last one being 2017-08-2621:02
pabelangerclarkb: I think vanilla can stay the say, since we are having issues with buildslogs.centos still21:03
*** tonyb has quit IRC21:04
openstackgerritMonty Taylor proposed openstack-infra/nodepool feature/zuulv3: WIP: Fix broken use of pre-existing cloud images  https://review.openstack.org/49805021:04
*** dteselkin has quit IRC21:04
*** jkilpatr has joined #openstack-infra21:06
*** dteselkin has joined #openstack-infra21:07
openstackgerritPaul Belanger proposed openstack-infra/zuul-jobs master: Create upload-afs role  https://review.openstack.org/49858821:09
*** yamamoto has quit IRC21:09
*** rockyg has joined #openstack-infra21:10
*** tonyb has joined #openstack-infra21:11
openstackgerritPaul Belanger proposed openstack-infra/project-config master: Bump infracloud-chocolate by 10 servers  https://review.openstack.org/49860321:16
*** ldnunes has quit IRC21:16
*** gixxer1k has quit IRC21:17
*** gouthamr has quit IRC21:23
*** spzala has joined #openstack-infra21:24
*** chlong has joined #openstack-infra21:26
clarkbya I think chocolate is happy now21:29
clarkbon to debugging dib21:29
*** chlong has quit IRC21:31
clarkbmgagne: interestinly my python3 is always utf-8 even when I set all the LANG and LC options to C and python2 is always ascii21:31
mgagnehmm, even when they are unset?21:32
*** thorst_afk has quit IRC21:32
*** rcernin has quit IRC21:32
openstackgerritMerged openstack-infra/zuul-jobs master: Add create / destory roles for AFS tokens  https://review.openstack.org/49854721:33
clarkbhaven't tried unset21:34
* clarkb tries21:34
*** baoli has quit IRC21:34
clarkbya even unset21:35
clarkbmy python3 is always utf821:35
clarkbI wonder if that is a change in 3.621:35
*** baoli has joined #openstack-infra21:35
*** baoli has quit IRC21:35
fungioh, and mgagne was trying with 3.521:35
mgagnestrange, I'm running python3.5, no backport or special repo here21:35
*** thorst_afk has joined #openstack-infra21:35
clarkbya sorry I'm just on my tumbleweed install21:35
*** mrunge has quit IRC21:36
*** baoli has joined #openstack-infra21:36
clarkbmgagne: can you test changing for line in iter(proc.stdout.readline, ''): to for line in iter(proc.stdout.readline, b'')21:37
mgagnewill test21:37
fungicould it be falling back on your platform's default baked-in locale?21:37
clarkbI wonder if it is converting automatically because the iteration stop variable is of string not bytes type21:37
mgagnesame21:37
fungiand maybe tumbleweed uses c.utf8 baked in if not specified in the environment?21:37
clarkbfungi: oh maybe21:37
clarkbhrm21:38
*** yamamoto_ has joined #openstack-infra21:38
fungixenial might default to c instead of c.utf821:38
mgagnehttp://paste.openstack.org/show/619700/21:38
*** mrunge has joined #openstack-infra21:38
mgagnewhere utf8.txt has utf8 content21:38
openstackgerritSam Yaple proposed openstack-infra/shade master: Allow domain_id for roles  https://review.openstack.org/49699221:39
openstackgerritSam Yaple proposed openstack-infra/shade master: Fix user role grants  https://review.openstack.org/49698021:39
*** thorst_afk has quit IRC21:40
fungii have custom-compiled local builds of 3.5 and 3.6 i can test side-by-side (on debian sid) if that helps21:40
fungiplain ol' ./configure and make21:41
fungino special patches21:41
*** baoli has quit IRC21:41
fungii use them when i want to find out what "real python" thinks21:41
clarkbaha its actually using locale.getpreferredencoding(False)21:43
clarkb(I think)21:43
pabelangerfungi: clarkb: https://review.openstack.org/498603/ bumps up infracloud-chocolate by 10 is we feel like we want to get some more nodes21:43
*** chlong has joined #openstack-infra21:43
clarkband sys and locale return different values, weird21:43
*** rockyg has quit IRC21:43
clarkbok I have reproduced21:44
mgagneI was using locale module to test encoding with above method you posted21:44
clarkbwow this seems to be a bug in python almost. If you set universal_newlines to true then you get this behavior but in python3.5 there is no way to override what the locale encoding is?21:47
fungii can recreate the issue on debian under 3.6 so it's not a 3.5 vs 3.6 issue at least21:48
mordredfungi: oh yay, maybe it's one of those lovely distro-patched-python fun times?21:49
clarkbI think I was just looking at the wrong thing21:50
clarkbwhich begs the question why are locale and sys reporting different data21:50
clarkbI guess becuase sys is system global default and locale is what your current env is21:51
clarkbso I don't think there is a great fix here. If one program outputs data encoded in a way that locale does not know about what shoudl we do? we could try and except and fall back on forcing tuf821:51
openstackgerritPaul Belanger proposed openstack-infra/project-config master: Add afsdocs_secret secret for publishing  https://review.openstack.org/49826721:51
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci master: Use local cirros image for tempest tests  https://review.openstack.org/49831521:53
openstackgerritMerged openstack-infra/project-config master: Add propose-update-constraints job  https://review.openstack.org/49772721:54
*** xyang1 has quit IRC21:54
fungiclarkb: pep 540 to the rescue?21:54
fungihttps://www.python.org/dev/peps/pep-0540/ "Add a new UTF-8 mode"21:54
mgagneso exporting LANG and LC_ALL looks to be the correct approach for now?21:55
clarkbmgagne: and maybe even file a bug against your package manager?21:56
clarkbmgagne: if lang/LC is ascii then it shouldn't output utf8 right?21:56
mgagneso they no longer output utf-8?21:56
clarkb(I don't actually know)21:56
clarkbmgagne: ya because thats just binary noise in ascii land21:56
clarkbgenerally though I expect setting locale to a utf8 locale will have better results21:56
clarkbso ++ to solving that way21:57
*** jascott1 has quit IRC21:57
clarkbfungi: well I don't think forcing utf8 is proper either because the lang could be utf16 or something silly21:57
fungiconvince your platform(s) to default to a compiled-in c.utf8 locale?21:58
clarkbfungi: in which case we'd still break21:58
fungiyeah, i expect there's no easy or even complete answer21:58
clarkbreally the input is bad (and we shouldn't trust input) and fixing that is most correct. We can force utf8 in the locale though because we control that I think (it may not have built/installed utf8 yet though?) and that gives everything a superset of workable encdoing it can use21:58
openstackgerritMerged openstack-infra/project-config master: Add a run playbook to propose-updates  https://review.openstack.org/49772921:59
fungiif rh platforms, i think you need locale packages installed21:59
clarkbprobably if we wanted to make dib extra careful it would try except on that encoding error and if hit force try using utf8 instaed21:59
clarkbbasically use what the locale.getpreferredencoding is then if that fails try again using utf821:59
*** kgiusti has left #openstack-infra22:00
*** kjackal has quit IRC22:00
mgagnethis falls outside my expertise, I will post the workaround on the bug report so someone finding the bug report can have a solution he can apply right away22:01
*** yamamoto_ has quit IRC22:04
*** felipemonteiro_ has joined #openstack-infra22:06
*** skelso has quit IRC22:07
*** felipemonteiro__ has joined #openstack-infra22:08
*** esberglu has quit IRC22:09
*** felipemonteiro has quit IRC22:10
*** esberglu has joined #openstack-infra22:10
*** esberglu has quit IRC22:10
*** hashar has quit IRC22:10
*** ijw has quit IRC22:10
*** bobh has quit IRC22:11
*** felipemonteiro_ has quit IRC22:11
*** ijw has joined #openstack-infra22:11
*** slaweq_ has quit IRC22:11
*** felipemonteiro_ has joined #openstack-infra22:12
*** slaweq_ has joined #openstack-infra22:12
*** wolverin_ has joined #openstack-infra22:12
*** wolverineav has quit IRC22:12
*** felipemonteiro has joined #openstack-infra22:14
*** ijw has quit IRC22:16
*** felipemonteiro__ has quit IRC22:16
*** felipemonteiro_ has quit IRC22:16
*** slaweq_ has quit IRC22:17
*** jascott1 has joined #openstack-infra22:17
*** jascott1 has quit IRC22:17
*** beagles has left #openstack-infra22:18
*** jascott1 has joined #openstack-infra22:18
*** jascott1 has quit IRC22:22
*** jascott1 has joined #openstack-infra22:23
*** esberglu has joined #openstack-infra22:28
*** ihrachys has quit IRC22:29
*** wolverin_ has quit IRC22:30
*** wolverineav has joined #openstack-infra22:30
*** wolverineav has quit IRC22:31
*** wolverineav has joined #openstack-infra22:31
*** marst has quit IRC22:31
*** esberglu has quit IRC22:33
*** ijw has joined #openstack-infra22:33
openstackgerritMerged openstack-infra/tripleo-ci master: Override STABLE_RELEASE when testing puppet-ceph stable/jewel  https://review.openstack.org/49812322:34
*** yamamoto_ has joined #openstack-infra22:35
*** wolverineav has quit IRC22:35
openstackgerritMerged openstack-infra/tripleo-ci master: Use shell based DLRN API client in promotion jobs  https://review.openstack.org/49827322:36
*** markvoelker has joined #openstack-infra22:42
*** vhosakot has quit IRC22:42
*** slaweq_ has joined #openstack-infra22:43
*** vhosakot has joined #openstack-infra22:43
*** yamamoto_ has quit IRC22:43
*** mriedem has quit IRC22:45
*** slaweq_ has quit IRC22:47
*** jamesmcarthur has quit IRC22:49
*** esberglu has joined #openstack-infra22:49
*** yamamoto_ has joined #openstack-infra22:49
*** felipemonteiro_ has joined #openstack-infra22:51
*** olaph1 is now known as olaph22:52
*** felipemonteiro__ has joined #openstack-infra22:53
*** felipemonteiro has quit IRC22:54
*** felipemonteiro_ has quit IRC22:56
openstackgerritJames E. Blair proposed openstack-infra/devstack-gate master: Zuul v3: add a devstack-legacy job  https://review.openstack.org/49769922:59
*** yamamoto_ has quit IRC22:59
*** rbrndt has quit IRC22:59
*** yamamoto_ has joined #openstack-infra23:06
*** hongbin has quit IRC23:07
*** vhosakot_ has joined #openstack-infra23:08
pabelangermordred: for new afs docs jobs, publish-openstack-python-docs?23:08
*** baoli has joined #openstack-infra23:08
*** rlandy is now known as rlandy|bbl23:09
*** mriedem has joined #openstack-infra23:09
*** vhosakot has quit IRC23:09
*** iyamahat has quit IRC23:09
mordredpabelanger: hrm. it's not just python docs- we publish docs to AFS for many things23:09
*** iyamahat has joined #openstack-infra23:09
mordredpabelanger: oh- but I guess the job that publishes to afs after building ...23:10
pabelangermordred: ya, so https://review.openstack.org/498267/ is the new parent of this new job23:10
mordredpabelanger: looking23:10
pabelangerI called that publish-openstack-afs23:10
pabelangerlike publish-openstack-artifact23:11
pabelangerartifacts*23:11
*** baoli has quit IRC23:12
pabelangerso, new jobs would be tox-docs + .root-marker setup, and parent to 498267 for afs publishing23:12
openstackgerritSam Yaple proposed openstack-infra/shade master: Fix switched params  https://review.openstack.org/49813223:12
openstackgerritSam Yaple proposed openstack-infra/shade master: Fix user role grants  https://review.openstack.org/49698023:12
mordredpabelanger: yah - so - I like your name23:13
mordredpabelanger: let's do that23:14
pabelangerokay23:14
*** esberglu has quit IRC23:15
openstackgerritDavid Moreau Simard proposed openstack-infra/devstack-gate master: Add the private_ipv4 hostvar to primary and sub nodes  https://review.openstack.org/49820723:15
dmsimardclarkb: ^23:15
clarkbdmsimard: and you need to rebase the other change too¿23:15
clarkbheh how did that come out upside down23:16
dmsimardclarkb: yeah, looking if I need to adjust anything23:16
clarkbdmsimard: ya actually the single quotes don't need escaping23:16
openstackgerritDavid Moreau Simard proposed openstack-infra/devstack-gate master: Move network overlay configuration to Ansible playbooks  https://review.openstack.org/43593323:16
clarkbthats my derp23:16
clarkb(the doubles do beacuse in a "" string already)23:17
mordredpabelanger: you finished off publish-to-pypi already right?23:17
openstackgerritDavid Moreau Simard proposed openstack-infra/devstack-gate master: Add the private_ipv4 hostvar to primary and sub nodes  https://review.openstack.org/49820723:17
openstackgerritDavid Moreau Simard proposed openstack-infra/devstack-gate master: Move network overlay configuration to Ansible playbooks  https://review.openstack.org/43593323:17
pabelangermordred: ya, working on testpypi23:18
pabelangermordred: just need to change to our production credentials23:18
mordredpabelanger: cool. one more job I need to convert real quick, and I think we can shift pypi publication for zuul from v2 to v323:18
dmsimardclarkb: so for paramiko, you were saying libffi might not come default on ubuntu ?23:19
clarkbdmsimard: ya pretty sure it doesn't, and I don't think we should install it by default on red hat either (though not entirely sure we do yet)23:19
clarkbdmsimard: the reason d-g works is cryptography has a generic linux whl published on pypi now23:19
clarkbdmsimard: so I think your changes are fine as that should work on any of our distros without editing anything23:20
dmsimardclarkb: not sure why it's in default DIB either23:20
clarkbinfra-root ^ any concern leaning on pypi hosted wheels for things like cryptography that we would otherwise have to build?23:20
clarkbthe risk here is that they would neglect pushing said whl next release and then we would be broken23:20
dmsimardclarkb: we probably would have the answer to that question like a year ago23:20
dmsimard(re: why they included ffi in default redhat)23:20
clarkb(because pip will prefer newer releases over wheels)23:21
fungistrangely, paramiko has a universal wheel on pypi... what does it need libffi for again?23:21
mordredwell ... it's an issue that goes away with v323:21
clarkbfungi: it doesn't, that is why it works23:21
clarkbmordred: thats a good point23:21
mordredso I have no problem leaning on it between now and then23:21
clarkbfungi: it will however need libffi if tomorrow cryprtography made a release without that whl23:21
*** sdague has quit IRC23:21
dmsimardI'm more interested in the ansible bump from 2.2 to 2.3 than paramiko, I already got surprise by a quirk that was specific to 2.223:22
dmsimardI was unpinning paramiko to figure out if it was hiding any issues23:22
clarkbdmsimard: ya I think both are safe. I have approved the ansible bump now23:22
dmsimardit had been pinned for >1 year23:22
clarkbI also think the paramiko change is safe too based on cryptography publishing that wheel23:23
dmsimardclarkb: I don't know why but I'm not sure that wheel is used in every scenario23:23
dmsimardclarkb: iirc I always need to install both libffi and openssl devel header packages before installing ansible and ara on centos boxes..23:23
fungiclarkb: ohhh, now i get it. cryptography is a dep of paramiko, not another example of a project building against cffi23:24
dmsimardoh, I have an idea, let me check something23:24
fungii was thinking you meant something else entirely there23:24
clarkbdmsimard: perhaps you are pinned to older cytpography?23:24
clarkbfungi: ya that23:24
clarkbfungi: paramiko deps on cryptography. The reason ansible installs may need libffi is paramiko deps on cryptography which without this wheel wants to compile23:25
dmsimardclarkb: this is an integration job for ara which has libffi and openssl in bindep: http://logs.openstack.org/68/495568/14/check/gate-ara-integration-py27-latest-centos-7/4ab49c6/console.html#_2017-08-28_01_46_47_91356923:25
dmsimardSee how it installs both libffi and openssl.. so I guess it's not included in the base image (DIB)?23:25
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Include the prepared projects list in zuul variables  https://review.openstack.org/49861823:25
openstackgerritKendall Nelson proposed openstack-infra/storyboard master: Adds Migration Documentation  https://review.openstack.org/49643923:25
dmsimardnow let me look at the same job on Ubuntu ..23:26
openstackgerritJames E. Blair proposed openstack-infra/project-config master: Zuul v3: Add devstack-legacy base job  https://review.openstack.org/49861923:26
clarkbdmsimard: ya and unfortunately we don't log later on which cryptography package we installed23:26
clarkbdmsimard: but I'm pretty sure we used the whl and you don't technically need those devel pacakges23:26
fungifor some reason i thought our pep8 jobs forced building without wheels to catch missing system deps, but looks like they only test building an sdist of the current project: http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/scripts/run-pep8.sh#n2123:26
dmsimardclarkb: looks like libffi and openssl are also installed on ubuntu http://logs.openstack.org/68/495568/14/check/gate-ara-integration-py35-latest-ubuntu-xenial/0ef72ac/console.html#_2017-08-28_01_46_50_75991323:27
dmsimardso I can easily test whether or not we can install ansible (paramiko with crypto) by sending a dummy patch to ara and removing both from bindep23:27
clarkbin any case I think it is safe to approve that change beacuse cryptography will install via that whl (for now at least)23:27
clarkbinfra-root ^ anyone object to that?23:27
pabelangerfungi: ya, pep8 jobs shouldn't be using wheels23:28
clarkbalso ansible doesn't even use parmiko by default right?23:28
fungidmsimard: looks like those are getting installed in the job because your project requests them: http://git.openstack.org/cgit/openstack/ara/tree/bindep.txt#n523:28
mordredclarkb: wfm23:28
clarkbso its sort a frustrating dep23:28
mordredclarkb: that's right - it uses openssh23:28
pabelangerfungi: http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/python-jobs.yaml#n7423:29
dmsimardfungi: correct, but we were saying just now that the redhat-common DIB element is supposed to install libffi by default23:29
dmsimardso what I'm saying we're probably missing something23:29
fungipabelanger: oh! that's where we hacked around it. thanks23:29
clarkbdmsimard: it doesn't all it does is set the name mapping if you request to install that package23:29
fungipabelanger: my faith in our foresight (hindsight?) is restored ;)23:29
dmsimardಠ_ಠ23:29
clarkbdmsimard: so some other element likely pulls it in and that magic in redhat-common makes it work23:29
dmsimardreally23:29
clarkbdmsimard: but its not installed by redhat-common alone23:29
*** tosky has quit IRC23:30
dmsimardoh, I understand23:30
*** bcat has joined #openstack-infra23:30
dmsimardso there's likely an element that would install it (using that mapping) but that element isn't included in the base image23:30
mordredclarkb: we COULD install with --ignore-deps23:30
clarkbdmsimard: ya exactly23:30
clarkbmordred: maybe this is good feedback for ansible next week :P23:30
clarkb(I assume you'll be there?)23:30
mordredthe actual dependencies are jinja2 and PyYAML23:30
mordredclarkb: I actually will not be - I've got to do a thing in Shanghai23:31
*** bytekitten has joined #openstack-infra23:31
* clarkb gives infra-root another minute to object to merging the paramiko unpinning before approving https://review.openstack.org/#/c/496355/123:31
dmsimardyup, confirmed that ansible installs without libffi and openssl.. wtf23:31
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Include the prepared projects list in zuul variables  https://review.openstack.org/49861823:31
dmsimardlet me test that on my centos box23:31
fungiclarkb: i do not object, no23:31
clarkbdmsimard: may be a relatively new change to their pypi publishing23:31
mordredI'm guessing it's in the requirements list because having it there is most likely to leave the most people witha working ansible23:31
fungimordred: you're being shanghaied?!?23:31
clarkbmordred: because windows?23:32
mordredfungi: yup23:32
mordredclarkb: perhaps?23:32
pabelangerclarkb: check experimental? we have both centos and fedora jobs there23:32
clarkbmordred: ebcause the env marker things shoudl handle that nicely23:32
mordredalthough if that's the case they could use markers23:32
mordredyah23:32
clarkbpabelanger: we have in check too and centos is fine23:32
clarkbpabelanger: that is what made me confused initially. And why I'm fairly certain this will just work as long as they don't release without that whl next time23:32
pabelangerclarkb: okay, cool. I can't think of anything23:33
fungimordred: clarkb: i wonder if the paramiko dep is there for improved windows support? (since people are unlikely to have a command-line ssh client available on that system)23:33
dmsimardclarkb: yeah, ansible/paramiko/crypto definitely installed fine on my centos machine without libffi and openssl.. that has got to be new23:33
clarkbdmsimard: cool I have approved the unpin change23:33
clarkbfungi: ya windows is the only thing I can think of that makes sense23:33
*** chlong has quit IRC23:34
openstackgerritMonty Taylor proposed openstack-infra/project-config master: Add openstack-infra/release-tools to fake v3 list  https://review.openstack.org/49862023:34
*** jamesmcarthur has joined #openstack-infra23:35
openstackgerritPaul Belanger proposed openstack-infra/project-config master: WIP: Add publish-openstack-python-docs jobs  https://review.openstack.org/49862123:35
openstackgerritPaul Belanger proposed openstack-infra/zuul-jobs master: Create upload-afs role  https://review.openstack.org/49858823:39
*** jamesmcarthur has quit IRC23:39
*** ijw has quit IRC23:41
*** ijw has joined #openstack-infra23:42
*** claudiub|2 has quit IRC23:43
*** slaweq_ has joined #openstack-infra23:44
openstackgerritMonty Taylor proposed openstack-infra/openstack-zuul-jobs master: Add release-announce job  https://review.openstack.org/49862323:44
openstackgerritPaul Belanger proposed openstack-infra/zuul-jobs master: Create zuul_executor_dest variable for fetch-sphinx-output  https://review.openstack.org/49862423:44
mordredjeblair, pabelanger: https://review.openstack.org/498620 is needed for https://review.openstack.org/498623 - which is hte last job we don't have for the release-to-pypi sequence23:46
*** ijw has quit IRC23:46
*** markvoelker has quit IRC23:46
*** mwarad has joined #openstack-infra23:47
openstackgerritPaul Belanger proposed openstack-infra/project-config master: WIP: Add publish-openstack-python-docs jobs  https://review.openstack.org/49862123:47
*** yamamoto_ has quit IRC23:47
*** slaweq_ has quit IRC23:48
pabelangermordred: I'll have to look in the morning, about to EOD here23:50
*** stakeda has joined #openstack-infra23:50
*** ijw has joined #openstack-infra23:52
*** gouthamr has joined #openstack-infra23:54
*** mwarad has quit IRC23:54
*** baoli has joined #openstack-infra23:55
*** mwarad has joined #openstack-infra23:55
*** ijw has quit IRC23:56
openstackgerritMerged openstack-infra/project-config master: Add openstack-infra/release-tools to fake v3 list  https://review.openstack.org/49862023:59
*** signed8bit has joined #openstack-infra23:59
openstackgerritMonty Taylor proposed openstack-infra/zuul-jobs master: Add trigger-readthedocs job  https://review.openstack.org/49862623:59

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!