| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Add job's project as implicit role project https://review.openstack.org/482726 | 00:01 |
|---|---|---|
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Remove ansible-role from implied role names https://review.openstack.org/484962 | 00:01 |
| jeblair | 2017-07-19 00:02:18,715 DEBUG zuul.web.ZuulWeb: ZuulWeb starting | 00:02 |
| jeblair | pabelanger, mordred, Shrews: ^ zuul-web is running on zuulv3.o.o | 00:02 |
| jeblair | as root | 00:03 |
| jeblair | which i think is an accident. | 00:03 |
| Diabelko | so, how harder is zuulv3 setup compared to zuulv25? | 00:03 |
| *** aeng has quit IRC | 00:03 | |
| *** sflanigan has quit IRC | 00:04 | |
| jeblair | Diabelko: hopefully a little easier. the parts fit together a little better. but if you're already using the ansible launcher, the structure is very similar. | 00:05 |
| *** aeng has joined #openstack-infra | 00:05 | |
| jeblair | Diabelko: but like i said, we don't have a full installation doc yet | 00:05 |
| Diabelko | mhm, k | 00:06 |
| Diabelko | let me try that in next few days ;) | 00:06 |
| Diabelko | thanks for all the help m8 | 00:06 |
| openstackgerrit | James E. Blair proposed openstack-infra/puppet-zuul master: Run zuul-web as zuul https://review.openstack.org/484987 | 00:06 |
| jeblair | Diabelko: there's a #zuul channel with devs and some other early adopters where, if you have questions, you may get quicker answers | 00:07 |
| jeblair | Diabelko: the docs that we do have are at https://docs.openstack.org/infra/zuul/feature/zuulv3/ | 00:07 |
| *** pahuang has quit IRC | 00:08 | |
| jeblair | Diabelko: also http://git.openstack.org/cgit/openstack-infra/zuul/tree/README.rst?h=feature%2Fzuulv3 may be helpful | 00:08 |
| jeblair | pabelanger, mordred, Shrews: https://review.openstack.org/484987 | 00:09 |
| *** esberglu has joined #openstack-infra | 00:10 | |
| *** infoprol has joined #openstack-infra | 00:12 | |
| *** gmann has quit IRC | 00:13 | |
| *** baoli has joined #openstack-infra | 00:13 | |
| *** gmann has joined #openstack-infra | 00:13 | |
| *** esberglu has quit IRC | 00:15 | |
| *** gcb has joined #openstack-infra | 00:15 | |
| *** infoprol has quit IRC | 00:16 | |
| *** sflanigan has joined #openstack-infra | 00:17 | |
| *** dchen has joined #openstack-infra | 00:24 | |
| *** gildub has joined #openstack-infra | 00:24 | |
| *** pahuang has joined #openstack-infra | 00:25 | |
| fungi | okay, i finally gave up and sent something to an ml so i could make sure i haven't broken the exim config on lists.o.o | 00:26 |
| fungi | seems to still be working fine, and in addition doing this which i wanted: http://paste.openstack.org/show/615790/ | 00:27 |
| pabelanger | jeblair: +2 | 00:27 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/puppet-exim master: Process mailman addresses last https://review.openstack.org/484989 | 00:27 |
| fungi | jeblair: ^ please see if that meets your expectations when you get a moment | 00:28 |
| *** xarses_ has joined #openstack-infra | 00:29 | |
| *** xarses has quit IRC | 00:31 | |
| openstackgerrit | Devang Doshi proposed openstack-infra/project-config master: new-project https://review.openstack.org/484990 | 00:34 |
| mordred | jeblair: +3 | 00:36 |
| *** Goneri has quit IRC | 00:41 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Fix json output appending https://review.openstack.org/484992 | 00:45 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Emit phase banner separate from play banner https://review.openstack.org/484993 | 00:45 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Remove noise both vertical and horizontal https://review.openstack.org/484994 | 00:45 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Stop printing shell commands if we have a task name https://review.openstack.org/484995 | 00:45 |
| openstackgerrit | Merged openstack-infra/puppet-zuul master: Run zuul-web as zuul https://review.openstack.org/484987 | 00:45 |
| fungi | jeblair: after a bit more reading i switched to :fail: on the outside chance that the senders are actually tracking whether the delivery is rejected or accepted at rcpt time: http://paste.openstack.org/show/615791/ | 00:46 |
| fungi | also this way legitimate users mailing that address will at least get some indication that it's not accepting e-mail right now | 00:47 |
| *** thorst has joined #openstack-infra | 00:49 | |
| *** thorst has quit IRC | 00:50 | |
| *** MatthewLi has quit IRC | 00:50 | |
| *** cuongnv has joined #openstack-infra | 00:51 | |
| *** csomerville has quit IRC | 00:56 | |
| openstackgerrit | Devang Doshi proposed openstack-infra/project-config master: new-project https://review.openstack.org/484990 | 00:58 |
| *** baoli has quit IRC | 01:01 | |
| *** cody-somerville has joined #openstack-infra | 01:03 | |
| *** Apoorva_ has quit IRC | 01:04 | |
| *** baoli has joined #openstack-infra | 01:04 | |
| *** Apoorva has joined #openstack-infra | 01:05 | |
| *** dave-mccowan has joined #openstack-infra | 01:06 | |
| *** MatthewLi has joined #openstack-infra | 01:07 | |
| *** harlowja has quit IRC | 01:13 | |
| *** zhurong has joined #openstack-infra | 01:13 | |
| openstackgerrit | Shriram Chander proposed openstack-infra/project-config master: jenkins/jobs/networking-vpp.yaml: Updating disabled_service list https://review.openstack.org/484999 | 01:14 |
| *** hongbin has joined #openstack-infra | 01:14 | |
| openstackgerrit | Shriram Chander proposed openstack-infra/project-config master: jenkins/jobs/networking-vpp.yaml: Updating disabled_service list https://review.openstack.org/484999 | 01:15 |
| *** mwarad has joined #openstack-infra | 01:18 | |
| *** bobh has quit IRC | 01:21 | |
| *** Daisy_ has joined #openstack-infra | 01:21 | |
| *** mriedem1 has joined #openstack-infra | 01:22 | |
| *** Daisy_ has quit IRC | 01:23 | |
| *** Daisy_ has joined #openstack-infra | 01:24 | |
| *** mriedem has quit IRC | 01:24 | |
| *** dougwig has quit IRC | 01:24 | |
| *** gouthamr has quit IRC | 01:25 | |
| *** liujiong has joined #openstack-infra | 01:31 | |
| *** LindaWang has joined #openstack-infra | 01:32 | |
| *** Julien-zte has quit IRC | 01:38 | |
| *** Apoorva_ has joined #openstack-infra | 01:39 | |
| *** Julien-zte has joined #openstack-infra | 01:39 | |
| *** tuanluong has joined #openstack-infra | 01:40 | |
| *** Apoorva has quit IRC | 01:42 | |
| *** yamahata has quit IRC | 01:43 | |
| *** iyamahat has quit IRC | 01:43 | |
| *** Apoorva_ has quit IRC | 01:44 | |
| *** Daisy_ has quit IRC | 01:44 | |
| *** Daisy_ has joined #openstack-infra | 01:44 | |
| *** akscram has joined #openstack-infra | 01:45 | |
| *** bobh has joined #openstack-infra | 01:45 | |
| *** aeng has quit IRC | 01:46 | |
| *** Daisy_ has quit IRC | 01:48 | |
| *** Daisy_ has joined #openstack-infra | 01:48 | |
| *** aeng has joined #openstack-infra | 01:50 | |
| *** dchen has quit IRC | 01:53 | |
| *** dchen has joined #openstack-infra | 01:54 | |
| *** mriedem1 is now known as mriedem | 01:59 | |
| *** sflanigan has quit IRC | 02:01 | |
| openstackgerrit | Boris Pavlovic proposed openstack-infra/project-config master: Fix Rally DSVM Designate job https://review.openstack.org/484954 | 02:04 |
| *** sflanigan has joined #openstack-infra | 02:15 | |
| *** gyee has quit IRC | 02:17 | |
| *** yamamoto has joined #openstack-infra | 02:17 | |
| *** gouthamr has joined #openstack-infra | 02:21 | |
| *** andreww has joined #openstack-infra | 02:21 | |
| *** xarses_ has quit IRC | 02:24 | |
| *** jerryz has quit IRC | 02:37 | |
| *** yamamoto has quit IRC | 02:38 | |
| *** sam_wan has joined #openstack-infra | 02:44 | |
| *** jamesmcarthur has joined #openstack-infra | 02:47 | |
| *** yamamoto has joined #openstack-infra | 02:48 | |
| *** thorst has joined #openstack-infra | 02:51 | |
| *** xarses_ has joined #openstack-infra | 02:51 | |
| *** andreww has quit IRC | 02:54 | |
| *** jamesmcarthur has quit IRC | 02:56 | |
| *** jamesmcarthur has joined #openstack-infra | 02:56 | |
| *** thorst has quit IRC | 02:59 | |
| *** markvoelker has quit IRC | 02:59 | |
| *** jamesmcarthur has quit IRC | 03:01 | |
| *** dchen has quit IRC | 03:01 | |
| *** dchen has joined #openstack-infra | 03:01 | |
| *** mriedem has quit IRC | 03:02 | |
| *** Daisy_ has quit IRC | 03:09 | |
| *** Daisy_ has joined #openstack-infra | 03:10 | |
| *** bobh has quit IRC | 03:11 | |
| *** sree_ has joined #openstack-infra | 03:11 | |
| *** sree_ is now known as Guest92762 | 03:11 | |
| *** Daisy_ has quit IRC | 03:14 | |
| *** Guest92762 has quit IRC | 03:15 | |
| *** mattmceuen has joined #openstack-infra | 03:29 | |
| *** sflanigan has quit IRC | 03:33 | |
| *** hongbin has quit IRC | 03:36 | |
| *** mattmceuen has quit IRC | 03:43 | |
| *** ekcs has joined #openstack-infra | 03:47 | |
| *** sflanigan has joined #openstack-infra | 03:49 | |
| *** masber has quit IRC | 03:51 | |
| *** ramishra has joined #openstack-infra | 03:52 | |
| *** yamamoto has quit IRC | 03:57 | |
| *** links has joined #openstack-infra | 03:58 | |
| *** yamamoto has joined #openstack-infra | 03:59 | |
| *** dave-mccowan has quit IRC | 04:01 | |
| *** yamamoto has quit IRC | 04:04 | |
| *** yamamoto has joined #openstack-infra | 04:05 | |
| *** pfallenop has quit IRC | 04:08 | |
| *** Dinesh_Bhor has joined #openstack-infra | 04:11 | |
| openstackgerrit | Kien Nguyen proposed openstack-infra/project-config master: Add topology to Heat job template https://review.openstack.org/482048 | 04:11 |
| *** gouthamr has quit IRC | 04:14 | |
| *** jascott1 has joined #openstack-infra | 04:14 | |
| *** yamamoto has quit IRC | 04:14 | |
| *** Dinesh_Bhor has quit IRC | 04:14 | |
| openstackgerrit | Kien Nguyen proposed openstack-infra/project-config master: Add topology flag to Heat job template https://review.openstack.org/482048 | 04:14 |
| *** Dinesh_Bhor has joined #openstack-infra | 04:15 | |
| *** yamamoto has joined #openstack-infra | 04:15 | |
| openstackgerrit | Kien Nguyen proposed openstack-infra/project-config master: Add topology flag to Heat job template https://review.openstack.org/482048 | 04:20 |
| *** andreww has joined #openstack-infra | 04:20 | |
| *** yamamoto has quit IRC | 04:21 | |
| *** psachin has joined #openstack-infra | 04:21 | |
| *** xarses_ has quit IRC | 04:23 | |
| openstackgerrit | Andreas Florath proposed openstack-infra/bindep master: Refactor: factor out subprocess and error handling stub https://review.openstack.org/455832 | 04:28 |
| *** ramishra has quit IRC | 04:30 | |
| *** harlowja has joined #openstack-infra | 04:31 | |
| *** ramishra has joined #openstack-infra | 04:32 | |
| openstackgerrit | Andreas Florath proposed openstack/diskimage-builder master: Fix race condition when caching in debootstap https://review.openstack.org/454523 | 04:33 |
| *** sshnaidm|off has quit IRC | 04:34 | |
| *** prateek has joined #openstack-infra | 04:38 | |
| *** Daisy_ has joined #openstack-infra | 04:41 | |
| *** gouthamr has joined #openstack-infra | 04:42 | |
| *** yamamoto has joined #openstack-infra | 04:44 | |
| *** yamahata has joined #openstack-infra | 04:46 | |
| *** Daisy_ has quit IRC | 04:46 | |
| *** harlowja has quit IRC | 04:47 | |
| *** jascott1 has quit IRC | 04:52 | |
| *** jascott1 has joined #openstack-infra | 04:52 | |
| *** apetrich has quit IRC | 04:54 | |
| *** zhurong has quit IRC | 04:55 | |
| *** apetrich has joined #openstack-infra | 04:56 | |
| *** jascott1 has quit IRC | 04:56 | |
| *** sree_ has joined #openstack-infra | 04:59 | |
| *** markvoelker has joined #openstack-infra | 05:00 | |
| *** mwarad has quit IRC | 05:00 | |
| *** sree_ is now known as Guest7201 | 05:00 | |
| *** Daisy_ has joined #openstack-infra | 05:00 | |
| *** Guest7201 has quit IRC | 05:04 | |
| *** gouthamr has quit IRC | 05:07 | |
| *** garyk has joined #openstack-infra | 05:08 | |
| *** nicolasbock has joined #openstack-infra | 05:11 | |
| openstackgerrit | Tomasz Trębski proposed openstack-infra/project-config master: Make Py35 gate votable for monasca-notification https://review.openstack.org/482892 | 05:14 |
| *** aeng has quit IRC | 05:14 | |
| *** armax has quit IRC | 05:14 | |
| *** armax has joined #openstack-infra | 05:15 | |
| *** aeng has joined #openstack-infra | 05:15 | |
| *** armax has quit IRC | 05:15 | |
| *** sflanigan has quit IRC | 05:16 | |
| *** armax has joined #openstack-infra | 05:16 | |
| *** armax has quit IRC | 05:16 | |
| *** armax has joined #openstack-infra | 05:17 | |
| *** armax has quit IRC | 05:17 | |
| *** Dinesh_Bhor has quit IRC | 05:17 | |
| *** dchen has quit IRC | 05:17 | |
| *** dangers_away has quit IRC | 05:17 | |
| *** armax has joined #openstack-infra | 05:17 | |
| *** dchen has joined #openstack-infra | 05:17 | |
| *** armax has quit IRC | 05:18 | |
| *** armax has joined #openstack-infra | 05:18 | |
| *** armax has quit IRC | 05:18 | |
| *** dangers_away has joined #openstack-infra | 05:20 | |
| *** frickler has quit IRC | 05:24 | |
| *** frickler has joined #openstack-infra | 05:27 | |
| *** sflanigan has joined #openstack-infra | 05:29 | |
| *** sflanigan has joined #openstack-infra | 05:29 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Add zuul.d configuration split documentation https://review.openstack.org/482405 | 05:30 |
| *** frickler has quit IRC | 05:33 | |
| *** markvoelker has quit IRC | 05:34 | |
| bkero | Who's reviewing nodepool patches these days? | 05:35 |
| *** xarses has joined #openstack-infra | 05:38 | |
| *** frickler has joined #openstack-infra | 05:38 | |
| *** andreww has quit IRC | 05:39 | |
| *** harlowja has joined #openstack-infra | 05:46 | |
| *** gildub has quit IRC | 05:48 | |
| *** zhurong has joined #openstack-infra | 05:50 | |
| *** rcernin has joined #openstack-infra | 05:50 | |
| *** Daisy_ has quit IRC | 05:52 | |
| *** Daisy_ has joined #openstack-infra | 05:53 | |
| *** pgadiya has joined #openstack-infra | 05:54 | |
| *** s-shiono has joined #openstack-infra | 05:54 | |
| *** aeng has quit IRC | 05:55 | |
| *** thorst has joined #openstack-infra | 05:56 | |
| *** Daisy_ has quit IRC | 05:57 | |
| *** cshastri has joined #openstack-infra | 05:58 | |
| *** Daisy_ has joined #openstack-infra | 05:58 | |
| *** thorst has quit IRC | 06:01 | |
| *** dhajare has joined #openstack-infra | 06:05 | |
| *** bhavik1 has joined #openstack-infra | 06:05 | |
| *** martinkopec has joined #openstack-infra | 06:06 | |
| *** mwarad has joined #openstack-infra | 06:06 | |
| *** Adri2000 has joined #openstack-infra | 06:07 | |
| *** mwarad has quit IRC | 06:09 | |
| *** jascott1 has joined #openstack-infra | 06:14 | |
| *** ccamacho has joined #openstack-infra | 06:15 | |
| *** ekcs has quit IRC | 06:20 | |
| *** aeng has joined #openstack-infra | 06:21 | |
| *** baoli has quit IRC | 06:22 | |
| *** nicolasbock has quit IRC | 06:23 | |
| *** isaacb_ has joined #openstack-infra | 06:23 | |
| *** nicolasbock has joined #openstack-infra | 06:25 | |
| *** martinkopec has quit IRC | 06:27 | |
| *** martinkopec has joined #openstack-infra | 06:29 | |
| *** aeng has quit IRC | 06:30 | |
| *** markvoelker has joined #openstack-infra | 06:31 | |
| *** ykarel has joined #openstack-infra | 06:31 | |
| openstackgerrit | Dirk Mueller proposed openstack-infra/project-config master: Adding more openSUSE 42.3 variants to existing 42.2 jobs https://review.openstack.org/484499 | 06:32 |
| *** andreas_s has joined #openstack-infra | 06:32 | |
| openstackgerrit | mathieu bultel proposed openstack-infra/project-config master: Create tripleo-upgrade project https://review.openstack.org/459762 | 06:36 |
| *** nicolasbock has quit IRC | 06:37 | |
| garyk | fungi: thank you for the help yesterday. the bot is now posting correctly!. thanks again | 06:38 |
| *** nicolasbock has joined #openstack-infra | 06:39 | |
| *** harlowja has quit IRC | 06:42 | |
| *** frickler has quit IRC | 06:42 | |
| *** frickler has joined #openstack-infra | 06:42 | |
| *** Dinesh_Bhor has joined #openstack-infra | 06:50 | |
| *** Dinesh_Bhor has quit IRC | 06:50 | |
| *** kjackal__ has quit IRC | 06:57 | |
| *** kjackal_ has quit IRC | 06:57 | |
| *** annegentle has quit IRC | 07:01 | |
| *** florianf has joined #openstack-infra | 07:01 | |
| *** isaacb_ has quit IRC | 07:04 | |
| *** markvoelker has quit IRC | 07:04 | |
| *** isaacb_ has joined #openstack-infra | 07:08 | |
| *** shardy has joined #openstack-infra | 07:11 | |
| *** tesseract has joined #openstack-infra | 07:17 | |
| *** hashar has joined #openstack-infra | 07:19 | |
| *** hashar has quit IRC | 07:19 | |
| *** hashar has joined #openstack-infra | 07:19 | |
| *** aviau has quit IRC | 07:19 | |
| *** Hal has joined #openstack-infra | 07:19 | |
| *** aviau has joined #openstack-infra | 07:19 | |
| *** adisky__ has joined #openstack-infra | 07:20 | |
| *** kjackal_ has joined #openstack-infra | 07:21 | |
| *** penguinolog has quit IRC | 07:21 | |
| *** milan has joined #openstack-infra | 07:22 | |
| *** dizquierdo has joined #openstack-infra | 07:28 | |
| *** bhavik1 has quit IRC | 07:41 | |
| *** zeih has joined #openstack-infra | 07:43 | |
| *** egonzalez has joined #openstack-infra | 07:43 | |
| *** pfallenop has joined #openstack-infra | 07:44 | |
| *** ralonsoh has joined #openstack-infra | 07:46 | |
| *** kjackal_ has joined #openstack-infra | 07:48 | |
| *** dizquierdo has quit IRC | 07:48 | |
| *** isaacb_ has quit IRC | 07:48 | |
| *** zeih has quit IRC | 07:53 | |
| *** zeih has joined #openstack-infra | 07:54 | |
| *** thorst has joined #openstack-infra | 07:57 | |
| *** kjackal__ has joined #openstack-infra | 07:58 | |
| *** isaacb_ has joined #openstack-infra | 07:58 | |
| *** isaacb_ has quit IRC | 07:59 | |
| *** kjackal___ has joined #openstack-infra | 08:00 | |
| *** kjackal___ has quit IRC | 08:00 | |
| *** kjackal_ has quit IRC | 08:01 | |
| *** kjackal___ has joined #openstack-infra | 08:01 | |
| *** zeih has quit IRC | 08:01 | |
| *** ociuhandu_ has joined #openstack-infra | 08:01 | |
| *** thorst has quit IRC | 08:02 | |
| *** garyk1 has joined #openstack-infra | 08:02 | |
| *** pfallenop has quit IRC | 08:03 | |
| *** pfalleno1 has joined #openstack-infra | 08:03 | |
| *** garyk has quit IRC | 08:03 | |
| *** kjackal__ has quit IRC | 08:03 | |
| *** zeih has joined #openstack-infra | 08:03 | |
| *** derekh has joined #openstack-infra | 08:08 | |
| *** liujiong has quit IRC | 08:09 | |
| *** dtantsur|afk is now known as dtantsur | 08:09 | |
| *** garyk1 has quit IRC | 08:15 | |
| *** garyk has joined #openstack-infra | 08:16 | |
| *** lucas-afk is now known as lucasagomes | 08:18 | |
| *** shardy is now known as shardy_afk | 08:21 | |
| *** electrofelix has joined #openstack-infra | 08:27 | |
| *** xinliang has quit IRC | 08:28 | |
| *** xinliang has joined #openstack-infra | 08:29 | |
| *** ociuhandu_ has quit IRC | 08:29 | |
| *** daidv__ has quit IRC | 08:30 | |
| *** efoley has joined #openstack-infra | 08:31 | |
| *** efoley has quit IRC | 08:32 | |
| *** efoley has joined #openstack-infra | 08:32 | |
| *** ralonsoh has quit IRC | 08:33 | |
| *** ralonsoh_ has joined #openstack-infra | 08:33 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Fix json output appending https://review.openstack.org/484992 | 08:36 |
| *** cody-somerville has quit IRC | 08:36 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Emit phase banner separate from play banner https://review.openstack.org/484993 | 08:36 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Remove noise both vertical and horizontal https://review.openstack.org/484994 | 08:36 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Stop printing shell commands if we have a task name https://review.openstack.org/484995 | 08:36 |
| *** shardy_afk is now known as shardy | 08:36 | |
| openstackgerrit | Merged openstack-infra/project-config master: Remove unused jobs for castellan-ui, nodepool, tripleo-ci https://review.openstack.org/484205 | 08:36 |
| openstackgerrit | Merged openstack-infra/project-config master: RM unused jobs: neutron-vpnaas, oslo.messaging, neutron, networking-odl https://review.openstack.org/484207 | 08:36 |
| *** cody-somerville has joined #openstack-infra | 08:36 | |
| *** cody-somerville has joined #openstack-infra | 08:36 | |
| openstackgerrit | Merged openstack-infra/project-config master: Re-enable Volume API v1 https://review.openstack.org/484921 | 08:38 |
| openstackgerrit | Merged openstack-infra/project-config master: Enable watcher in watcherclient functional job https://review.openstack.org/484865 | 08:39 |
| openstackgerrit | Merged openstack-infra/project-config master: Add "test-results" publisher to OpenStack-Ansible periodic jobs https://review.openstack.org/484843 | 08:41 |
| *** dtantsur is now known as dtantsur|brb | 08:41 | |
| *** Daisy_ has quit IRC | 08:41 | |
| *** Daisy_ has joined #openstack-infra | 08:42 | |
| *** dchen has quit IRC | 08:43 | |
| *** dingyichen has quit IRC | 08:43 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add os_tacker role to OpenStack-Ansible https://review.openstack.org/482873 | 08:43 |
| *** milan has quit IRC | 08:45 | |
| *** Daisy__ has joined #openstack-infra | 08:45 | |
| AJaeger | mordred: morning, you're up early - everything ok? | 08:45 |
| openstackgerrit | Merged openstack-infra/project-config master: Add missing jobs to ironic-python-agent-builder https://review.openstack.org/484753 | 08:46 |
| *** Daisy_ has quit IRC | 08:46 | |
| AJaeger | project-config cores, please review these small cleanups https://review.openstack.org/484663 and https://review.openstack.org/484164 | 08:47 |
| mordred | AJaeger: I'm in tokyo :) | 08:47 |
| openstackgerrit | Merged openstack-infra/project-config master: aodh: remove functional job for master branch https://review.openstack.org/483258 | 08:48 |
| AJaeger | mordred: Ah, that explains it... | 08:48 |
| *** openstackgerrit has quit IRC | 08:49 | |
| AJaeger | ihrachys, yamamoto , could you review https://review.openstack.org/#/c/484160/ , please? | 08:49 |
| *** isaacb has joined #openstack-infra | 08:49 | |
| yamamoto | AJaeger: looking | 08:50 |
| AJaeger | thanks | 08:50 |
| *** openstackgerrit has joined #openstack-infra | 08:52 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Add -m flag to setfiles for Fedora 26 https://review.openstack.org/485099 | 08:52 |
| *** kjackal_ has joined #openstack-infra | 08:55 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Bump fedora-minimal DIB_RELEASE 26 https://review.openstack.org/482570 | 08:55 |
| *** tosky has joined #openstack-infra | 08:56 | |
| *** kjackal___ has quit IRC | 08:56 | |
| *** kjackal__ has joined #openstack-infra | 08:56 | |
| AJaeger | yamamoto: should be no functional change at all... | 08:56 |
| yamamoto | AJaeger: i had just a minor comment | 08:58 |
| *** isaacb has quit IRC | 08:58 | |
| openstackgerrit | Samuel de Medeiros Queiroz proposed openstack-infra/shade master: De-client-ify Group Create https://review.openstack.org/485102 | 08:58 |
| openstackgerrit | Merged openstack-infra/project-config master: Use local_conf for castellan, oaktree, rally, tacker https://review.openstack.org/484164 | 08:58 |
| AJaeger | yamamoto: that's outside of the scope of the change - do you want to do it as followup ? | 08:58 |
| yamamoto | AJaeger: your change made it unnecessary, didn't it? anyway a followup is fine for me | 08:59 |
| *** kjackal_ has quit IRC | 09:00 | |
| openstackgerrit | Merged openstack-infra/project-config master: Remove duplicated job from openstack-ansible https://review.openstack.org/484663 | 09:00 |
| AJaeger | yamamoto: no, my change did not make it unneccessary | 09:00 |
| yamamoto | AJaeger: my understanding is if DEVSTACK_SUBNODE_CONFIG is set, DEVSTACK_LOCAL_CONFIG is not used on subnode. on the other hand local_conf section is applied to subnode unconditionally. | 09:01 |
| *** markvoelker has joined #openstack-infra | 09:02 | |
| AJaeger | yamamoto: Ah, wasn't aware of that. Will you do the change and I'll +2 - or shall I do it? Like to get this one in and iterate... | 09:02 |
| yamamoto | AJaeger: i'll change to +1 as it doesn't seem critical | 09:03 |
| *** isaacb has joined #openstack-infra | 09:04 | |
| AJaeger | thanks | 09:04 |
| *** yamamoto has quit IRC | 09:06 | |
| openstackgerrit | Samuel de Medeiros Queiroz proposed openstack-infra/shade master: De-client-ify Group List https://review.openstack.org/485108 | 09:06 |
| *** pblaho has joined #openstack-infra | 09:07 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: networking-ovn: Remove DEVSTACK_SUBNODE_CONFIG https://review.openstack.org/485110 | 09:10 |
| *** rwsu has quit IRC | 09:10 | |
| *** yamamoto has joined #openstack-infra | 09:10 | |
| *** s-shiono has quit IRC | 09:10 | |
| *** yamamoto has quit IRC | 09:10 | |
| *** mjblack has quit IRC | 09:10 | |
| *** rwsu has joined #openstack-infra | 09:12 | |
| openstackgerrit | Merged openstack-infra/project-config master: Use local_conf for jenkins/jobs/n* https://review.openstack.org/484160 | 09:14 |
| *** yamamoto has joined #openstack-infra | 09:14 | |
| *** tsuzuki has quit IRC | 09:15 | |
| *** jascott1 has quit IRC | 09:16 | |
| *** garyk has quit IRC | 09:16 | |
| *** jascott1 has joined #openstack-infra | 09:17 | |
| *** efoley_ has joined #openstack-infra | 09:21 | |
| *** jascott1 has quit IRC | 09:21 | |
| AJaeger | wow, layout check of project-config log is only 11 MB - and we have 12418 jobs defined. Before doing all the mitaka retirement and cleanup we had 18 MB of logs - and 17170 jobs. | 09:22 |
| AJaeger | mordred, jeblair: 5000 jobs less to convert to zuul v3 ;) ^ | 09:23 |
| *** efoley has quit IRC | 09:24 | |
| openstackgerrit | Samuel de Medeiros Queiroz proposed openstack-infra/shade master: De-client-ify Group Update https://review.openstack.org/485118 | 09:24 |
| *** milan has joined #openstack-infra | 09:25 | |
| *** pcrews_ has quit IRC | 09:26 | |
| openstackgerrit | Julien Danjou proposed openstack-infra/project-config master: tooz: do not run py35-etcd3* on Ocata https://review.openstack.org/484984 | 09:29 |
| openstackgerrit | Samuel de Medeiros Queiroz proposed openstack-infra/shade master: De-client-ify Group Delete https://review.openstack.org/485122 | 09:29 |
| *** markvoelker has quit IRC | 09:35 | |
| *** yamamoto has quit IRC | 09:36 | |
| *** yamamoto has joined #openstack-infra | 09:36 | |
| *** sdague has joined #openstack-infra | 09:37 | |
| openstackgerrit | Merged openstack-infra/devstack-gate master: Add ironic-inspector services to features.yaml https://review.openstack.org/483059 | 09:39 |
| openstackgerrit | Boris Pavlovic proposed openstack-infra/project-config master: Fix Rally DSVM Designate job https://review.openstack.org/484954 | 09:41 |
| *** sambetts|afk is now known as sambetts | 09:42 | |
| openstackgerrit | Gael Chamoulaud proposed openstack-infra/tripleo-ci master: WIP: Enable tripleo-validations tests https://review.openstack.org/481080 | 09:43 |
| *** panda|off is now known as panda | 09:45 | |
| *** katkapilatova has joined #openstack-infra | 09:45 | |
| *** Daisy__ has quit IRC | 09:45 | |
| *** kjackal__ has quit IRC | 09:49 | |
| *** LindaWang has quit IRC | 09:54 | |
| *** LindaWang has joined #openstack-infra | 09:55 | |
| *** LindaWang has quit IRC | 09:57 | |
| *** jamesmcarthur has joined #openstack-infra | 09:57 | |
| *** thorst has joined #openstack-infra | 09:58 | |
| *** LindaWang has joined #openstack-infra | 09:58 | |
| *** cuongnv has quit IRC | 09:59 | |
| odyssey4me | AJaeger oops, https://github.com/openstack-infra/project-config/commit/4168ea16ee2c578d68ab7e1696a3c9bd83fe6219 actually is used for the newton branch | 10:00 |
| odyssey4me | it's non voting for newton, but voting for ocata onwards | 10:00 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/project-config master: Revert "Remove duplicated job from openstack-ansible" https://review.openstack.org/485137 | 10:01 |
| *** jamesmcarthur has quit IRC | 10:01 | |
| openstackgerrit | Samuel de Medeiros Queiroz proposed openstack-infra/shade master: De-client-ify Role Create https://review.openstack.org/482245 | 10:02 |
| openstackgerrit | Samuel de Medeiros Queiroz proposed openstack-infra/shade master: De-client-ify Role List https://review.openstack.org/482246 | 10:02 |
| *** thorst has quit IRC | 10:03 | |
| *** LindaWang has quit IRC | 10:03 | |
| AJaeger | odyssey4me: Ah, that was your intention - but didn't catch it from the regexes. Sorry. | 10:04 |
| AJaeger | odyssey4me: what about adding a comment to the jobs pointing this out? | 10:04 |
| *** kjackal_ has joined #openstack-infra | 10:04 | |
| odyssey4me | AJaeger yeah, I can do that | 10:04 |
| odyssey4me | gimme a moment | 10:04 |
| openstackgerrit | Samuel de Medeiros Queiroz proposed openstack-infra/shade master: De-client-ify Role Create https://review.openstack.org/482245 | 10:05 |
| *** pgadiya has quit IRC | 10:05 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/project-config master: Revert "Remove duplicated job from openstack-ansible" https://review.openstack.org/485137 | 10:08 |
| AJaeger | odyssey4me: thanks | 10:09 |
| openstackgerrit | Samuel de Medeiros Queiroz proposed openstack-infra/shade master: De-client-ify Role List https://review.openstack.org/482246 | 10:10 |
| *** wolverineav has joined #openstack-infra | 10:15 | |
| *** isaacb has quit IRC | 10:17 | |
| *** isaacb_ has joined #openstack-infra | 10:17 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack-infra/project-config master: Normalize projects.yaml https://review.openstack.org/485142 | 10:18 |
| *** pgadiya has joined #openstack-infra | 10:21 | |
| openstackgerrit | Merged openstack-infra/project-config master: Revert "Remove duplicated job from openstack-ansible" https://review.openstack.org/485137 | 10:21 |
| *** Daisy_ has joined #openstack-infra | 10:24 | |
| openstackgerrit | Samuel de Medeiros Queiroz proposed openstack-infra/shade master: De-client-ify Role List https://review.openstack.org/482246 | 10:25 |
| *** ccamacho is now known as ccamacho|brb | 10:25 | |
| *** hashar is now known as hasharLunch | 10:27 | |
| *** Dinesh_Bhor has joined #openstack-infra | 10:27 | |
| *** sslypushenko_ has quit IRC | 10:27 | |
| openstackgerrit | Samuel de Medeiros Queiroz proposed openstack-infra/shade master: De-client-ify Role Delete https://review.openstack.org/485145 | 10:28 |
| *** Daisy_ has quit IRC | 10:29 | |
| *** yamahata has quit IRC | 10:30 | |
| *** markvoelker has joined #openstack-infra | 10:32 | |
| openstackgerrit | Merged openstack-infra/project-config master: Normalize projects.yaml https://review.openstack.org/485142 | 10:32 |
| *** kjackal_ has quit IRC | 10:34 | |
| *** kjackal__ has joined #openstack-infra | 10:34 | |
| *** jkilpatr has quit IRC | 10:36 | |
| *** raissa_ has quit IRC | 10:40 | |
| *** priteau has joined #openstack-infra | 10:41 | |
| *** LindaWang has joined #openstack-infra | 10:46 | |
| *** pgadiya has quit IRC | 10:52 | |
| *** adarazs is now known as adarazs_lunch | 10:53 | |
| *** zhurong has quit IRC | 10:53 | |
| *** yamamoto has quit IRC | 10:55 | |
| *** gcb has quit IRC | 10:55 | |
| *** stakeda has quit IRC | 10:56 | |
| *** gcb has joined #openstack-infra | 10:57 | |
| *** andreas_s has quit IRC | 10:57 | |
| *** neiljerram has joined #openstack-infra | 10:59 | |
| *** pgadiya has joined #openstack-infra | 11:04 | |
| *** tuanluong has quit IRC | 11:05 | |
| *** isaacb_ has quit IRC | 11:05 | |
| *** markvoelker has quit IRC | 11:06 | |
| *** alexchadin has joined #openstack-infra | 11:09 | |
| *** jkilpatr has joined #openstack-infra | 11:12 | |
| *** pgadiya has quit IRC | 11:16 | |
| *** dave-mccowan has joined #openstack-infra | 11:26 | |
| *** dougwig has joined #openstack-infra | 11:32 | |
| pabelanger | Tripleo gate change queue is up to almost 19hr+, from a quick look at logs, it appears they are building DIBs not using our mirrors / reverse proxy | 11:35 |
| *** wolverineav has quit IRC | 11:35 | |
| *** noorul has joined #openstack-infra | 11:38 | |
| *** hashar has joined #openstack-infra | 11:39 | |
| *** kaisers has quit IRC | 11:39 | |
| *** yamamoto has joined #openstack-infra | 11:40 | |
| *** ldnunes has joined #openstack-infra | 11:41 | |
| *** zeih has quit IRC | 11:43 | |
| *** chlong_ has quit IRC | 11:43 | |
| *** Hal has quit IRC | 11:44 | |
| panda | pabelanger: we're importing mirror_info.sh and using those variables | 11:44 |
| pabelanger | panda: are you sure? I looked at the apache mirror log for a failure and didn't see the 404 for missing package request | 11:44 |
| *** wolverineav has joined #openstack-infra | 11:45 | |
| pabelanger | panda: you'd need to do that inside your DIB elements, since they don't read yum repos from build system | 11:45 |
| *** ccamacho|brb is now known as ccamacho | 11:45 | |
| panda | pabelanger: hrm, I need to check then | 11:46 |
| *** wolverineav has quit IRC | 11:46 | |
| *** kaisers has joined #openstack-infra | 11:47 | |
| panda | pabelanger: anyway we don't build images in every job, unless we are testing a dib component | 11:50 |
| pabelanger | panda: the patches in the gate appear to be testing DIB elements, that looks to be the reason for the gate reset and 19+hr delay | 11:54 |
| pabelanger | http://status.openstack.org/elastic-recheck/data/others.html | 11:55 |
| pabelanger | top 6 jobs are tripleo ATM | 11:55 |
| pabelanger | gate-tripleo-ci-centos-7-containers-multinode being the main issue (4.3% failure rate) | 11:56 |
| pabelanger | which is building a DIB according to http://logs.openstack.org/09/484009/1/gate/gate-tripleo-ci-centos-7-containers-multinode/2b3daed/logs/undercloud/home/jenkins/undercloud_install.log.txt.gz | 11:57 |
| pabelanger | so, if mirrors are not setup properly in DIB elements, it is having a large impact on the job failures | 11:57 |
| *** wolverineav has joined #openstack-infra | 11:58 | |
| *** dprince has joined #openstack-infra | 11:58 | |
| *** adarazs_lunch is now known as adarazs | 11:58 | |
| *** kgiusti has joined #openstack-infra | 11:58 | |
| *** dizquierdo has joined #openstack-infra | 12:00 | |
| openstackgerrit | Paul Belanger proposed openstack/diskimage-builder master: Bump fedora-minimal DIB_RELEASE 26 https://review.openstack.org/482570 | 12:01 |
| *** wolverineav has quit IRC | 12:01 | |
| openstackgerrit | Paul Belanger proposed openstack/diskimage-builder master: Add -m flag to setfiles for Fedora 26 https://review.openstack.org/485099 | 12:01 |
| panda | pabelanger: I see we are setting DIB_YUM_REPO_CONF during the build, but not DIB_DISTRIBUTION_MIRROR. I don't know which wins | 12:01 |
| *** thorst has joined #openstack-infra | 12:02 | |
| *** LindaWang has quit IRC | 12:02 | |
| *** markvoelker has joined #openstack-infra | 12:03 | |
| pabelanger | panda: where are you setting that? | 12:03 |
| *** yamamoto has quit IRC | 12:03 | |
| AJaeger | pabelanger: could you review https://review.openstack.org/484476 - a nodepool change for openSUSE 42.3 -, please? | 12:04 |
| *** thorst_ has joined #openstack-infra | 12:04 | |
| pabelanger | panda: IIRC, that is used in the yum element for diskimage-builder, but I don't see the logs using that element | 12:04 |
| *** lucasagomes is now known as lucas-hungry | 12:04 | |
| AJaeger | pabelanger: could you also review this grafana change, please? | 12:05 |
| pabelanger | AJaeger: left comment, but looks good in general | 12:06 |
| *** thorst has quit IRC | 12:06 | |
| *** isaacb_ has joined #openstack-infra | 12:06 | |
| openstackgerrit | Alexandra Settle proposed openstack-infra/project-config master: Updates the acl and adds new config files for api-site https://review.openstack.org/485179 | 12:07 |
| *** LindaWang has joined #openstack-infra | 12:09 | |
| panda | pabelanger: in quickstart build role | 12:13 |
| adarazs | cmurphy: can I have your opinion on my answer to https://review.openstack.org/482210 ? | 12:14 |
| openstackgerrit | Alexandra Settle proposed openstack-infra/project-config master: Updates the acl and adds new config files for api-site https://review.openstack.org/485179 | 12:15 |
| *** ociuhandu_ has joined #openstack-infra | 12:15 | |
| openstackgerrit | Billy Olsen proposed openstack-infra/project-config master: Add charm-interface-gluster-peer for gluster peer relations https://review.openstack.org/484332 | 12:16 |
| *** trown|outtypewww is now known as trown | 12:17 | |
| pabelanger | panda: can you link the code for that, having trouble finding it | 12:17 |
| *** jascott1 has joined #openstack-infra | 12:19 | |
| pabelanger | panda: moving this to #tripleo | 12:20 |
| cmurphy | adarazs: ah sorry yes, i'll comment | 12:20 |
| adarazs | cmurphy: thanks! :) | 12:20 |
| *** yamamoto has joined #openstack-infra | 12:21 | |
| openstackgerrit | Alexandra Settle proposed openstack-infra/project-config master: Updates the acl and adds new config files for api-site https://review.openstack.org/485179 | 12:22 |
| cmurphy | adarazs: I +2'd, i'm not terribly worried about it (though curious if infra-root has any feelings about it) | 12:23 |
| *** neiljerram has quit IRC | 12:23 | |
| *** jascott1 has quit IRC | 12:23 | |
| adarazs | cmurphy: thanks! AJaeger was okay with it I think, though he didn't have time for a proper review. | 12:25 |
| adarazs | pabelanger: can you please take a look? would be nice to have the readme/howto for tripleo jobs: https://review.openstack.org/482210 | 12:25 |
| *** markvoelker has quit IRC | 12:25 | |
| *** markvoelker has joined #openstack-infra | 12:26 | |
| *** markmcd has quit IRC | 12:26 | |
| AJaeger | adarazs: I'm not involved with that repo at all... | 12:26 |
| adarazs | :) | 12:26 |
| *** eharney has joined #openstack-infra | 12:27 | |
| pabelanger | adarazs: cmurphy: this means anybody installing puppet-openstackci depends on tripleo-ci repo, is that something we want to do? cc fungi | 12:28 |
| adarazs | pabelanger: well, it just clones it and copies a static html file from it, so there's not too hard dependency. | 12:29 |
| pabelanger | adarazs: cmurphy: also, with zuulv3 coming, it might be worth holding off? since this changes a bit for jobs | 12:29 |
| adarazs | I just don't want to keep this static file in the repo to bother you with updating it. | 12:29 |
| adarazs | pabelanger: folks from TripleO really want this in, we get a lot of questions for debugging the jobs. | 12:30 |
| pabelanger | wonder if we could refactor this a bit to make it easier for projects to add them, and not have to update our control plane server to support it | 12:30 |
| *** xarses_ has joined #openstack-infra | 12:30 | |
| cmurphy | yeah i was kind of surprised that we put this module in charge of that | 12:30 |
| *** mriedem has joined #openstack-infra | 12:31 | |
| pabelanger | I mean, at first glance maybe this should live in project-config? | 12:31 |
| pabelanger | then we have a single location for things | 12:31 |
| adarazs | I wanted to have originally a generic "ReadmeName" file like "index_footer.html" so that I can just stick it in some log dir and have it there. | 12:31 |
| adarazs | but there were some security concerns for that. | 12:31 |
| *** baoli has joined #openstack-infra | 12:31 | |
| *** ralonsoh_ has quit IRC | 12:32 | |
| *** xarses has quit IRC | 12:33 | |
| *** baoli_ has joined #openstack-infra | 12:33 | |
| pabelanger | adarazs: ya, I think in zuulv3 jobs we might be able to do that. But need to think about it for a moment | 12:33 |
| *** noorul has quit IRC | 12:34 | |
| adarazs | pabelanger: meanwhile I'd appreciate if we could have this. :) | 12:34 |
| pabelanger | lets see what fungi says, I'm concerned about now depending on tripleo-ci | 12:36 |
| *** baoli has quit IRC | 12:37 | |
| *** dtantsur|brb is now known as dtantsur | 12:40 | |
| *** rlandy has joined #openstack-infra | 12:41 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Add html based websocket client for console stream https://review.openstack.org/483503 | 12:42 |
| *** isaacb_ has quit IRC | 12:44 | |
| adarazs | I added him to the review as well. | 12:48 |
| openstackgerrit | Rosario Di Somma proposed openstack-infra/shade master: Don't remove top-container element in the adapter https://review.openstack.org/484178 | 12:51 |
| *** Goneri has joined #openstack-infra | 12:52 | |
| *** raissa_ has joined #openstack-infra | 12:52 | |
| *** gcb has quit IRC | 12:54 | |
| *** rdearauj has joined #openstack-infra | 12:57 | |
| *** raissa_ has quit IRC | 12:57 | |
| *** gouthamr has joined #openstack-infra | 12:58 | |
| openstackgerrit | Dirk Mueller proposed openstack/diskimage-builder master: doc: supported_distros: Add openSUSE Leap 42.2/3 and Tumbleweed https://review.openstack.org/421301 | 12:59 |
| *** trown is now known as trown|brb | 13:01 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Add html based websocket client for console stream https://review.openstack.org/483503 | 13:01 |
| *** jamesdenton has joined #openstack-infra | 13:01 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Add html based websocket client for console stream https://review.openstack.org/483503 | 13:02 |
| *** aviau has quit IRC | 13:02 | |
| *** aviau has joined #openstack-infra | 13:02 | |
| openstackgerrit | Dirk Mueller proposed openstack-infra/project-config master: Add openSUSE 42.3 as non-voting check to devstack* https://review.openstack.org/485204 | 13:06 |
| *** jcoufal has joined #openstack-infra | 13:06 | |
| *** esberglu has joined #openstack-infra | 13:09 | |
| *** isaacb has joined #openstack-infra | 13:09 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Add html based websocket client for console stream https://review.openstack.org/483503 | 13:12 |
| *** wolverineav has joined #openstack-infra | 13:14 | |
| *** trown|brb is now known as trown | 13:15 | |
| *** dhajare has quit IRC | 13:17 | |
| *** mriedem is now known as mriedem_meeting | 13:17 | |
| *** ralonsoh has joined #openstack-infra | 13:19 | |
| *** ralonsoh has quit IRC | 13:20 | |
| *** ralonsoh has joined #openstack-infra | 13:20 | |
| *** efoley_ has quit IRC | 13:23 | |
| openstackgerrit | Alexandra Settle proposed openstack-infra/project-config master: Updates the acl and adds new config files for api-site https://review.openstack.org/485179 | 13:24 |
| fungi | #status log the lists.o.o server is temporarily in emergency disable mode pending merger of https://review.openstack.org/484989 | 13:24 |
| openstackstatus | fungi: finished logging | 13:24 |
| *** EricGonc_ has joined #openstack-infra | 13:32 | |
| *** EricGonczer_ has quit IRC | 13:33 | |
| *** sree has joined #openstack-infra | 13:34 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Fix zuul.(yaml|d) loading order and add tests https://review.openstack.org/482411 | 13:35 |
| *** bobh has joined #openstack-infra | 13:37 | |
| *** askb has quit IRC | 13:39 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Add html based websocket client for console stream https://review.openstack.org/483503 | 13:40 |
| *** Daisy_ has joined #openstack-infra | 13:40 | |
| *** ykarel has quit IRC | 13:43 | |
| *** isaacb has quit IRC | 13:44 | |
| *** Daisy_ has quit IRC | 13:45 | |
| *** rwsu has quit IRC | 13:49 | |
| *** hongbin has joined #openstack-infra | 13:49 | |
| *** rdearauj is now known as raissa | 13:51 | |
| *** jamesmcarthur has joined #openstack-infra | 13:51 | |
| *** raissa is now known as Guest11014 | 13:52 | |
| *** Guest11014 is now known as raissa_ | 13:52 | |
| *** jrist has quit IRC | 13:53 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Add html based websocket client for console stream https://review.openstack.org/483503 | 13:54 |
| *** camunoz has joined #openstack-infra | 13:54 | |
| sbezverk | good morning infra team. fungi, do you have a second for qq? | 13:56 |
| *** petevg has joined #openstack-infra | 13:56 | |
| *** jamesden_ has joined #openstack-infra | 13:56 | |
| *** rwsu has joined #openstack-infra | 13:57 | |
| *** jamesdenton has quit IRC | 13:58 | |
| petevg | Hello. Does anyone online at present know where I can find the gpg key for the apt package repositories at http://mirror.mtl01.internap.openstack.org? (I've poked around in the directory hierarchy on the server, and haven't found it.) | 13:58 |
| pabelanger | petevg: we don't sign our deb packages today, so no gpg key | 13:58 |
| pabelanger | petevg: because we regenerate indexes in our mirrors, we cannot use the keys from upstream too | 13:59 |
| petevg | pabelanger: Got it. Thank you. If I want to sudo apt install something in a gating test, is the current best practice to simply disable the check for the key? | 13:59 |
| *** armax has joined #openstack-infra | 13:59 | |
| pabelanger | petevg: Yup, that's what we do today by default | 13:59 |
| *** LindaWang has quit IRC | 14:00 | |
| pabelanger | we have discussed getting gpg keys working again, but would require updates to our mirror process | 14:00 |
| fungi | also, it discourages general use of those mirrors outside of our ci system | 14:00 |
| fungi | which is a bit of a feature | 14:00 |
| pabelanger | but, since these images are just for testing, we don't really mind about gpg | 14:00 |
| petevg | Got it. Thank you for the info. | 14:01 |
| *** alexchadin has quit IRC | 14:02 | |
| *** markmcd has joined #openstack-infra | 14:02 | |
| sbezverk | fungi pabelanger is there any way to request rebuilding a tarball without merging PS? | 14:02 |
| pabelanger | sbezverk: not today | 14:02 |
| *** isaacb has joined #openstack-infra | 14:02 | |
| fungi | well, you can request it in here, depending on the circumstances. if the job failed and just needs to be retriggered i can do that | 14:03 |
| pabelanger | with DAG jobs in zuulv3, it should make artifact handling easier | 14:03 |
| pabelanger | Oh | 14:03 |
| pabelanger | I read that as upload to tarballs | 14:03 |
| pabelanger | sorry | 14:03 |
| *** markmcd has quit IRC | 14:04 | |
| fungi | sbezverk: was there a nondeterministic failure which broke tarball building/uploading? do you have a link to the failure log for that? | 14:04 |
| sbezverk | fungi I do not have failure :( but I see the change which was merged yesterday noonish is missing in tarball I got 10 minutes ago.. | 14:05 |
| fungi | or if you can let us know the repository and branch i can look it up | 14:05 |
| sbezverk | fungi: kolla ubuntu source master | 14:05 |
| fungi | sbezverk: have you looked to see if the branch tarball job for it is still pending in the post pipeline on the zuul status page? | 14:05 |
| sbezverk | fungi: I have not as I do not know how.. | 14:06 |
| *** jrist has joined #openstack-infra | 14:06 | |
| fungi | the job generally gets enqueued when a change merges, and then waits in the post pipeline for an available node to run it | 14:06 |
| fungi | so it may not show up on tarballs.o.o instantly | 14:06 |
| *** markmcd has joined #openstack-infra | 14:06 | |
| *** rbrndt has joined #openstack-infra | 14:07 | |
| fungi | sbezverk: look for kolla commits in the post pipeline at http://status.openstack.org/zuul/ | 14:07 |
| fungi | there are 50+ commits waiting in post so it may be one of those | 14:08 |
| *** felipemonteiro__ has joined #openstack-infra | 14:08 | |
| fungi | about a 6-hour backlog | 14:08 |
| *** yamamoto has quit IRC | 14:09 | |
| sbezverk | fungi: one of our kolla comrades spotted issue: | 14:09 |
| sbezverk | [ ]ubuntu-binary-registry-.tar.gz2017-07-19 09:492.7G | 14:09 |
| sbezverk | newly generated tarball has wrong name | 14:09 |
| fungi | oh, that looks pretty broken | 14:09 |
| fungi | yeah | 14:09 |
| sbezverk | it is missing branch | 14:10 |
| fungi | i thought those registry tarballs ran once a day in the periodic pipeline | 14:10 |
| fungi | i wonder if the lack of implicit branches in the periodic pipeline wasn't taken into consideration | 14:11 |
| *** felipemonteiro_ has joined #openstack-infra | 14:11 | |
| fungi | i thought you were talking about source tarballs, not registry tarballs | 14:11 |
| fungi | hence my confusion | 14:11 |
| *** jaypipes has joined #openstack-infra | 14:12 | |
| sbezverk | fungi: sorry I was not clear.. yes it is registry | 14:13 |
| *** felipemonteiro__ has quit IRC | 14:14 | |
| *** jamesmcarthur has quit IRC | 14:15 | |
| fungi | okay, i recommend double-checking that your registry tarball build job template explicitly set the branch via a parameter so that you're creating branch-specific jobs to assign to the periodic pipeline | 14:15 |
| fungi | i gather these started out as experimental pipeline jobs, and the experimental pipeline gets a branch communicated from zuul based on the context of the change where check experimental was called, but jobs in the periodic pipeline have no context (change, commit) from which to infer a branch and so need to know on their own what branch to checkout before running | 14:17 |
| *** links has quit IRC | 14:18 | |
| openstackgerrit | Rob Cresswell proposed openstack-infra/project-config master: Make Horizon & D_O_A Django 1.11 jobs voting https://review.openstack.org/485220 | 14:19 |
| *** xyang1 has joined #openstack-infra | 14:19 | |
| *** milan has quit IRC | 14:20 | |
| openstackgerrit | Rob Cresswell proposed openstack-infra/project-config master: Make Horizon & D_O_A Django 1.11 jobs voting https://review.openstack.org/485220 | 14:22 |
| sbezverk | fungi based on your exaplanation this then will not work properly, right? BRANCH=$(echo "$ZUUL_BRANCH" | cut -d/ -f2) | 14:22 |
| pabelanger | ianw: https://review.openstack.org/#/c/482570/ passed now, had to update one of your patches for a typo | 14:24 |
| pabelanger | ianw: well, nodepool passed | 14:24 |
| pabelanger | ianw: guess we need to update mirror for fedora-minimal | 14:25 |
| pabelanger | will look into that | 14:25 |
| pabelanger | ianw: actually, maybe it is just cold afs cache. will recheck to see | 14:25 |
| openstackgerrit | Tomasz Trębski proposed openstack-infra/project-config master: Make Py35 gate votable for monasca-notification https://review.openstack.org/482892 | 14:26 |
| fungi | sbezverk: correct, in periodic pipelines there is no $ZUUL_BRANCH so the job template needs to take a job parameter for the branch and then you instantiate multiple branch-specific jobs for that in jenkins/jobs/projects.yaml | 14:26 |
| fungi | (or at least that's how it's most commonly done) | 14:26 |
| *** sam_wan has quit IRC | 14:29 | |
| sbezverk | fungi: got it, thanks a lot for explanation. | 14:29 |
| *** sam_wan has joined #openstack-infra | 14:29 | |
| fungi | sbezverk: you should be able to find some examples in there of periodic dsvm and pyNN jobs which run on declared branches as bitrot checks | 14:31 |
| fungi | if you have trouble let me know and i can take a minute to dig some up, just not right this moment | 14:32 |
| sbezverk | fungi: sure, I understand.. | 14:32 |
| *** annegentle has joined #openstack-infra | 14:34 | |
| *** marst has joined #openstack-infra | 14:35 | |
| *** gordc has joined #openstack-infra | 14:35 | |
| *** ianychoi has joined #openstack-infra | 14:35 | |
| *** cshastri has quit IRC | 14:39 | |
| *** gyee has joined #openstack-infra | 14:39 | |
| *** abadger1999 has quit IRC | 14:39 | |
| robcresswell | Thanks for the swift review AJaeger | 14:39 |
| AJaeger | fungi, pabelanger : Is proposal node down? The translation post jobs are not running | 14:43 |
| pabelanger | AJaeger: let me check | 14:44 |
| pabelanger | AJaeger: its running jobs | 14:46 |
| *** panda is now known as panda|bbl | 14:46 | |
| pabelanger | nc proposal.slave.openstack.org 19885 | 14:46 |
| AJaeger | pabelanger: ah... | 14:46 |
| AJaeger | running on second oldest change, not oldest - interesting | 14:47 |
| AJaeger | thanks | 14:47 |
| *** lucas-hungry is now known as lucasagomes | 14:47 | |
| *** isaacb has quit IRC | 14:48 | |
| pabelanger | ya, see that oo | 14:48 |
| pabelanger | too* | 14:48 |
| *** LindaWang has joined #openstack-infra | 14:48 | |
| AJaeger | not worth digging into that - I saw the 5h delay and nothing obvious... | 14:48 |
| AJaeger | but I'm happy now ;) | 14:48 |
| *** isaacb has joined #openstack-infra | 14:49 | |
| AJaeger | pabelanger: could you also review this grafana change, please - https://review.openstack.org/#/c/484185/ ? | 14:51 |
| *** rossella_ has joined #openstack-infra | 14:55 | |
| *** rossella_s has quit IRC | 14:56 | |
| *** aviau_ has joined #openstack-infra | 14:57 | |
| *** rloo has joined #openstack-infra | 14:58 | |
| *** aviau_ has joined #openstack-infra | 14:58 | |
| fungi | looks like the oldest change in post is has jobs running now | 14:58 |
| Shrews | pabelanger: qq... what's the relationship between puppet-nodepool and puppet-openstackci in regards to nodepool setup? | 14:59 |
| *** iyamahat has joined #openstack-infra | 14:59 | |
| AJaeger | fungi: indeed | 14:59 |
| openstackgerrit | Merged openstack-infra/project-config master: Merge devstack and tempest views and fix graph references https://review.openstack.org/484185 | 14:59 |
| openstackgerrit | Merged openstack-infra/project-config master: Add openSUSE 42.3 tempest run graph https://review.openstack.org/484481 | 15:00 |
| pabelanger | Shrews: openstack-infra depends on puppet-openstackci, which then depends on puppet-nodepool. So we usually need to make changes to both projects when proposing new changes | 15:01 |
| pabelanger | Shrews: puppet-openstackci is also used by 3rd-party ci systems | 15:01 |
| *** aviau has quit IRC | 15:01 | |
| fungi | Shrews: put another way, puppet-nodepool is specifically for just running nodepool, puppet-openstackci is for running a ci system similar to ours, including nodepool. system-config is our entrypoint to puppet-openstackci setting the specific variables we want in _our_ exact deployment | 15:02 |
| Shrews | gotcha. thx | 15:03 |
| *** mriedem_meeting is now known as mriedem | 15:03 | |
| *** rkukura has quit IRC | 15:04 | |
| *** electrofelix has quit IRC | 15:05 | |
| *** electrofelix has joined #openstack-infra | 15:06 | |
| fungi | the evolution there is that it was originally all in one repository, then we split out separate modules for each service into their own repositories giving us a two-layer design, but then there was still a lot of grouping/organization in the top layer which needed extensive copying and modifying to run a whole ci system so the middle layer was introduced to encapsulate that while filtering out our | 15:07 |
| fungi | deployment-specific settings | 15:07 |
| *** psachin has quit IRC | 15:08 | |
| *** milan has joined #openstack-infra | 15:09 | |
| *** yamamoto has joined #openstack-infra | 15:09 | |
| *** yamahata has joined #openstack-infra | 15:16 | |
| *** rkukura has joined #openstack-infra | 15:17 | |
| *** felipemonteiro_ has quit IRC | 15:18 | |
| *** yamamoto has quit IRC | 15:19 | |
| *** jascott1 has joined #openstack-infra | 15:22 | |
| *** jascott1 has quit IRC | 15:26 | |
| *** gcb has joined #openstack-infra | 15:32 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/puppet-nodepool master: Add support for python 3 https://review.openstack.org/485245 | 15:32 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/puppet-openstackci master: Add support for python 3 for nodepool https://review.openstack.org/485246 | 15:32 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/system-config master: Move nl01.o.o to py3 https://review.openstack.org/485247 | 15:32 |
| Shrews | pabelanger: i'm sure there's likely all sorts of things wrong with those 3 ^^^^. Awaiting you to tell me where I went wrong. | 15:32 |
| dmsimard | Zuul queues seem abnormally long right now, anything odd going on ? | 15:32 |
| smcginnis | Lead up to the milestone 3 crunch probably. | 15:35 |
| *** xarses_ has quit IRC | 15:35 | |
| *** vhosakot has joined #openstack-infra | 15:35 | |
| pabelanger | dmsimard: tripleo jobs have a high failure rate, we suspect it is not using AFS mirror | 15:36 |
| *** vhosakot has quit IRC | 15:36 | |
| *** vhosakot has joined #openstack-infra | 15:36 | |
| *** xarses_ has joined #openstack-infra | 15:36 | |
| pabelanger | integrated change queue looks pretty good this morning | 15:36 |
| pabelanger | dmsimard: we are also down on some cloud resources also, so that is also a factor | 15:37 |
| dmsimard | pabelanger: ack, thanks | 15:37 |
| pabelanger | dmsimard: basically, I think instack needs to be patches to use mirrors and panda|bbl is working on it | 15:37 |
| *** Apoorva has joined #openstack-infra | 15:38 | |
| *** Apoorva_ has joined #openstack-infra | 15:38 | |
| *** Sukhdev_ has joined #openstack-infra | 15:39 | |
| *** gyee has quit IRC | 15:40 | |
| *** dougwig has quit IRC | 15:42 | |
| *** pcrews has joined #openstack-infra | 15:42 | |
| *** Apoorva has quit IRC | 15:42 | |
| *** gyee has joined #openstack-infra | 15:42 | |
| inc0 | good morning, I have question about https://review.openstack.org/#/c/476989/ | 15:43 |
| *** EricGonc_ has quit IRC | 15:43 | |
| inc0 | is this enough to make jobs periodic or should I do something more? And where can we tweak frequency of runs? | 15:43 |
| pabelanger | inc0: peroidic jobs only run every 24 hours atm, mostly to save resources. But shouldn't be an issue to move a job into that pipeline | 15:45 |
| pabelanger | tripleo does something like this for there images | 15:45 |
| inc0 | yeah, that's what this change is about | 15:45 |
| inc0 | although I think I'll make this experimental to run it at least once before it's periodic | 15:46 |
| pabelanger | right, easier to debug when experimental | 15:46 |
| *** EricGonczer_ has joined #openstack-infra | 15:46 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Support UUID as builder identifier https://review.openstack.org/484414 | 15:46 |
| inc0 | so for future, is this enough to make it periodic? https://review.openstack.org/#/c/476989/3/zuul/layout.yaml | 15:46 |
| pabelanger | yes | 15:47 |
| inc0 | cool thank you | 15:47 |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack-infra/project-config master: Jobs to move temporary tarballs to regularc https://review.openstack.org/476989 | 15:50 |
| *** rbrndt has quit IRC | 15:50 | |
| *** gyee has quit IRC | 15:52 | |
| *** Rockyg has joined #openstack-infra | 15:52 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/puppet-nodepool master: Add support for python 3 https://review.openstack.org/485245 | 15:52 |
| *** rbrndt has joined #openstack-infra | 15:52 | |
| *** isaacb has quit IRC | 15:54 | |
| *** gcb has quit IRC | 15:54 | |
| *** tjones has joined #openstack-infra | 15:55 | |
| *** gcb has joined #openstack-infra | 15:55 | |
| *** trown is now known as trown|lunch | 15:57 | |
| *** tjones has quit IRC | 15:57 | |
| *** LindaWang has quit IRC | 15:57 | |
| *** jascott1 has joined #openstack-infra | 15:58 | |
| inc0 | #startmeeting kolla | 15:59 |
| openstack | Meeting started Wed Jul 19 15:59:59 2017 UTC and is due to finish in 60 minutes. The chair is inc0. Information about MeetBot at http://wiki.debian.org/MeetBot. | 16:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 16:00 |
| openstack | The meeting name has been set to 'kolla' | 16:00 |
| *** jamesmcarthur has joined #openstack-infra | 16:00 | |
| inc0 | #topic w00t for kolla | 16:00 |
| pabelanger | inc0: wrong room | 16:00 |
| inc0 | o/ everyone, show the love | 16:00 |
| inc0 | ah | 16:00 |
| inc0 | :( | 16:00 |
| inc0 | sorry | 16:00 |
| vhosakot | inc0: wrong room :) | 16:00 |
| inc0 | #endmeeting kolla | 16:00 |
| openstack | Meeting ended Wed Jul 19 16:00:58 2017 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 16:01 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/kolla/2017/kolla.2017-07-19-15.59.html | 16:01 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/kolla/2017/kolla.2017-07-19-15.59.txt | 16:01 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/kolla/2017/kolla.2017-07-19-15.59.log.html | 16:01 |
| inc0 | sorry guys, I pressed arrow once time too much:/ | 16:01 |
| *** kiennt has joined #openstack-infra | 16:01 | |
| *** dmellado_ has quit IRC | 16:03 | |
| *** dizquierdo has quit IRC | 16:03 | |
| *** dtantsur is now known as dtantsur|afk | 16:03 | |
| *** katkapilatova has left #openstack-infra | 16:04 | |
| *** aarefiev is now known as aarefiev_afk | 16:04 | |
| *** ccamacho has quit IRC | 16:04 | |
| *** felipemonteiro has joined #openstack-infra | 16:05 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Add html based websocket client for console stream https://review.openstack.org/483503 | 16:05 |
| *** Apoorva_ has quit IRC | 16:05 | |
| *** sridhar_ram has quit IRC | 16:06 | |
| *** felipemonteiro_ has joined #openstack-infra | 16:07 | |
| jeblair | mordred, pabelanger: as mordred pointed out in #zuul, the apache config on zuulv3.o.o did not update | 16:08 |
| jeblair | i'm going to rm the file and kick puppet to see if it creates it | 16:08 |
| pabelanger | jeblair: ack | 16:08 |
| jeblair | [WARNING]: Failure using method (v2_runner_on_ok) in callback plugin (</etc/ansible/callback_plugins/mqtt.CallbackModule object at 0x7fcb8feac4d0>): Python 2.7.9 and 3.2 are the minimum supported versions for TLS. | 16:09 |
| *** rcernin has quit IRC | 16:09 | |
| jeblair | fungi, mtreinish: ^ aware of that? | 16:09 |
| *** jamesmcarthur has quit IRC | 16:09 | |
| jeblair | mordred, pabelanger: puppet recreated the file as before, without the new bits. | 16:10 |
| *** EricGonc_ has joined #openstack-infra | 16:10 | |
| *** felipemonteiro has quit IRC | 16:11 | |
| *** EricGonczer_ has quit IRC | 16:11 | |
| *** Apoorva has joined #openstack-infra | 16:11 | |
| *** isaacb has joined #openstack-infra | 16:11 | |
| *** florianf has quit IRC | 16:12 | |
| jeblair | the puppet-zuul module is up to date, and its vhost template looks correct | 16:12 |
| *** sree has quit IRC | 16:12 | |
| *** sree has joined #openstack-infra | 16:13 | |
| jeblair | mordred: oh, i see the problem. | 16:14 |
| jeblair | mordred: you only added it to the ssl version of the page. and there is no ssl version of the page on zuulv3.o.o | 16:14 |
| jeblair | mordred: you need to update the port 80 virtualhost as well | 16:15 |
| *** dizquierdo has joined #openstack-infra | 16:15 | |
| pabelanger | ya, Isee that now too | 16:15 |
| *** dmellado_ has joined #openstack-infra | 16:17 | |
| *** wolverineav has quit IRC | 16:17 | |
| *** sree has quit IRC | 16:18 | |
| fungi | jeblair: yep, those are the python versions where ssl server certificate validation began to be supported, so i guess ansible is refusing to allow you to do any tls without that capability rather than mislead anyone about the actual level of security it's providing? | 16:18 |
| *** annegentle has quit IRC | 16:19 | |
| jeblair | fungi: i think that's a mqtt error | 16:19 |
| fungi | er, it's just a warning though so i guess it's being noisy in hopes you'll be aware | 16:19 |
| jeblair | fungi: yeah, it's just that it shows up *a lot* in the puppet run all logs | 16:19 |
| fungi | oh, bubbling up from the paho-mqtt module maybe? | 16:20 |
| *** sshnaidm|off has joined #openstack-infra | 16:20 | |
| fungi | aha, raised at https://github.com/eclipse/paho.mqtt.python/blob/master/src/paho/mqtt/client.py#L679 | 16:21 |
| jeblair | fungi: yeah. and it's not clear to me how much of the ansible mqtt stuff is actually working based on that. i know it says "WARNING" but that's just ansible saying "a callback plugin failed, but it's just a callback plugin. i still ran your stuff." nonetheless, the callback plugin *did* fail. | 16:21 |
| fungi | drinking from the firehose now to see if we're getting any events from the run_all process | 16:23 |
| *** rcernin has joined #openstack-infra | 16:23 | |
| *** bobh has quit IRC | 16:24 | |
| *** lucasagomes is now known as lucas-afk | 16:24 | |
| fungi | i know it _was_ working at some point at least because we collected samples from it | 16:24 |
| *** rwsu has quit IRC | 16:24 | |
| mordred | jeblair: AHHHHHHH | 16:25 |
| fungi | introduced by https://github.com/eclipse/paho.mqtt.python/commit/0a8cccc in v1.3.0 | 16:26 |
| fungi | tagged a month ago | 16:26 |
| fungi | and uploaded to pypi at roughly the same time | 16:27 |
| fungi | mtreinish: should we cap our callback plugin dep at <1.3.0 for now and roll back to 1.2.3? | 16:28 |
| *** dangers_away has quit IRC | 16:29 | |
| *** rwsu has joined #openstack-infra | 16:29 | |
| *** kjackal__ has quit IRC | 16:29 | |
| *** kjackal_ has joined #openstack-infra | 16:29 | |
| *** bobh has joined #openstack-infra | 16:30 | |
| fungi | basically, it's broken for ubuntu trusty but when we eventually get around to upgrading the puppetmaster to xenial we can lift that cap | 16:30 |
| *** kiennt has quit IRC | 16:33 | |
| *** david-lyle has quit IRC | 16:34 | |
| *** EricGonc_ has quit IRC | 16:35 | |
| *** dangers has joined #openstack-infra | 16:35 | |
| fungi | looks like we're just installing the dep via the pip package provider in openstack_project::puppetmaster so i'll add a version pin there | 16:35 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Default tox to python3 https://review.openstack.org/485272 | 16:36 |
| *** Apoorva has quit IRC | 16:37 | |
| *** jamesmcarthur has joined #openstack-infra | 16:41 | |
| openstackgerrit | Merged openstack-infra/shade master: Don't remove top-container element in the adapter https://review.openstack.org/484178 | 16:42 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Add branch to JobDirPlaybook https://review.openstack.org/485273 | 16:42 |
| *** jamesmcarthur has quit IRC | 16:45 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Add project info to JobDirPlaybook https://review.openstack.org/485273 | 16:46 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config master: Pin paho-mqtt to 1.2.3 on the puppetmaster https://review.openstack.org/485275 | 16:47 |
| fungi | jeblair: mtreinish: ^ | 16:47 |
| amrith | hi infra (I know this may belong more realistically in -dev but I'm asking here). when using the CLI, one thing one has to do often is get a token. One of my co-workers was looking for something like ssh-agent which would handle the getting of a token without storing a password in plain text in the environment. So he came up with a simple script which would do this. Would this be something that would be useful to | 16:48 |
| amrith | others; he's willing to contribute it upstream. not sure where it'd land, is it an openstack CLI thing? Has someone already something like this? | 16:48 |
| amrith | ... asking for a friend (really) ... | 16:49 |
| *** EricGonczer_ has joined #openstack-infra | 16:49 | |
| *** gcb has quit IRC | 16:50 | |
| *** sree has joined #openstack-infra | 16:50 | |
| openstackgerrit | Corey Bryant proposed openstack-infra/project-config master: snaps: add snap-cinder definitions https://review.openstack.org/485276 | 16:51 |
| mordred | amrith: one should never get a token on the CLI | 16:51 |
| amrith | mordred ? | 16:51 |
| mordred | amrith: sorry - I responded before I read the whole thing | 16:51 |
| * amrith waits | 16:52 | |
| mordred | amrith: the problem the human is trying to solve is the "aplication credential" spec we just approved in keystone | 16:52 |
| *** david-lyle has joined #openstack-infra | 16:52 | |
| * amrith scurries to learn about application credential in keystone | 16:52 | |
| *** hashar has quit IRC | 16:53 | |
| mordred | amrith: one sec - I will get you a link | 16:53 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Install zuul to access zuul_console https://review.openstack.org/485277 | 16:53 |
| mordred | amrith: http://specs.openstack.org/openstack/keystone-specs/specs/keystone/pike/application-credentials.html | 16:53 |
| amrith | looking | 16:53 |
| mordred | amrith: the general problem being - clouds aren't very useful if you can't used them from automation | 16:53 |
| mordred | amrith: but for folks using federated identity, putting their LDAP password into config management is bad | 16:53 |
| amrith | let me read this spec, it may be similar | 16:54 |
| mordred | amrith: the tl;dr on applicatoin credentials is to be able to create an authorization - eventually scope-limited - for an application for api use | 16:54 |
| mordred | amrith: please do | 16:54 |
| *** sree has quit IRC | 16:54 | |
| amrith | at least, it may achieve the same thing | 16:55 |
| amrith | interesting, this may solve the problem | 16:55 |
| mordred | amrith: I have heard similar issues/concerns from other humans before (and a human in sweden who also coded an ssh-agent like thing :) ) | 16:55 |
| amrith | ah, ok | 16:55 |
| *** dmellado- has joined #openstack-infra | 16:55 | |
| mordred | amrith: I'll be hacking on that as my next non-infra task once I'm done with these updates to keystoneauth | 16:55 |
| amrith | ok | 16:55 |
| amrith | thanks for the link | 16:55 |
| amrith | yes, this is for our deployment automation | 16:56 |
| amrith | and this would do what we need | 16:56 |
| amrith | when can we have it? | 16:56 |
| amrith | :) | 16:56 |
| *** Goneri has quit IRC | 16:57 | |
| amrith | well, wait. this will get an application credential; does that expire? | 16:57 |
| *** dmellado_ has quit IRC | 16:58 | |
| *** derekh has quit IRC | 17:02 | |
| *** egonzalez has quit IRC | 17:02 | |
| *** EricGonc_ has joined #openstack-infra | 17:06 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Install zuul to access zuul_console https://review.openstack.org/485277 | 17:06 |
| *** isaacb has quit IRC | 17:07 | |
| *** EricGonczer_ has quit IRC | 17:07 | |
| *** jerryz has joined #openstack-infra | 17:11 | |
| mordred | amrith: hopefully you'll get it in queens :) | 17:12 |
| mordred | amrith: and it has an optional expiration thatyou can specify | 17:13 |
| mordred | amrith: so that part is up to you | 17:13 |
| *** Apoorva has joined #openstack-infra | 17:15 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Add project info to JobDirPlaybook https://review.openstack.org/485273 | 17:16 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Make playbook run meta info less fragile https://review.openstack.org/485284 | 17:16 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Add spacer after playbook stats rather than before playbook https://review.openstack.org/485285 | 17:16 |
| *** jamesmcarthur has joined #openstack-infra | 17:16 | |
| *** dizquierdo has quit IRC | 17:18 | |
| *** martinkopec has quit IRC | 17:18 | |
| *** raissa_ has quit IRC | 17:20 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Add project info to JobDirPlaybook https://review.openstack.org/485273 | 17:23 |
| *** gyee has joined #openstack-infra | 17:23 | |
| *** harlowja has joined #openstack-infra | 17:25 | |
| *** harlowja has quit IRC | 17:25 | |
| *** harlowja has joined #openstack-infra | 17:26 | |
| *** jascott1 has quit IRC | 17:28 | |
| mtreinish | fungi, jeblair: that's the first I've seen that issue | 17:28 |
| *** bobh has quit IRC | 17:29 | |
| *** rossella_ has quit IRC | 17:31 | |
| mtreinish | heh, but I don't think I ever tested the plugin on a python version < 2.7.9 | 17:31 |
| mtreinish | although I did most of my testing before paho 1.3.0 was pushed | 17:31 |
| *** bobh has joined #openstack-infra | 17:31 | |
| *** rossella_ has joined #openstack-infra | 17:31 | |
| fungi | mtreinish: right, this is new in the last month | 17:32 |
| *** tosky has quit IRC | 17:33 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Make playbook run meta info less fragile https://review.openstack.org/485284 | 17:33 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Add spacer after playbook stats rather than before playbook https://review.openstack.org/485285 | 17:33 |
| *** rbrndt has quit IRC | 17:35 | |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Install zuul to access zuul_console https://review.openstack.org/485277 | 17:36 |
| *** ekcs has joined #openstack-infra | 17:36 | |
| *** annegentle has joined #openstack-infra | 17:36 | |
| *** dprince has quit IRC | 17:37 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/shade master: Initial commit of zuulv3 jobs https://review.openstack.org/484027 | 17:39 |
| jlvillal | We want to add an experimental job to test our new Boot-From-Volume feature in Ironic | 17:41 |
| jlvillal | Reviews requested for: https://review.openstack.org/#/c/480332/ Thanks. | 17:41 |
| *** gyee has quit IRC | 17:42 | |
| *** ralonsoh has quit IRC | 17:42 | |
| AJaeger | pabelanger, dirk : How good is the openSUSE 42.3 support? Can we merge dirk's change to add more jobs or is anything missing? | 17:42 |
| *** deep-book-gk_ has joined #openstack-infra | 17:42 | |
| *** deep-book-gk_ has left #openstack-infra | 17:43 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci master: Use port-security version of OVB templates https://review.openstack.org/460748 | 17:44 |
| *** gyee has joined #openstack-infra | 17:44 | |
| *** gyee has quit IRC | 17:44 | |
| pabelanger | AJaeger: dirk: yup, +2. Confirmed diskimage-builder built the image propery | 17:45 |
| pabelanger | properly* | 17:45 |
| pabelanger | Shrews: mind a quick review on 484476 | 17:45 |
| Shrews | pabelanger: umm | 17:46 |
| Shrews | oh, had the wrong # | 17:46 |
| *** tesseract has quit IRC | 17:46 | |
| pabelanger | Shrews: your puppet changes are also good! +2 all around | 17:47 |
| Shrews | pabelanger: you missed one between the two you reviewd | 17:48 |
| pabelanger | so I did, +2 also | 17:49 |
| openstackgerrit | Monty Taylor proposed openstack-infra/puppet-zuul master: Add web proxying to non-SSL vhost too https://review.openstack.org/485294 | 17:49 |
| mordred | jeblair, pabelanger, Shrews: ^^ | 17:49 |
| pabelanger | Shrews: we should pick a time to update nl01.o.o too | 17:49 |
| Shrews | mordred: fwiw, https://review.openstack.org/#/q/topic:nodepool-py3 moves us away from py2 for nodepool in v3 land | 17:50 |
| openstackgerrit | Merged openstack-infra/nodepool master: Add support for nodepool testing of openSUSE 42.3 https://review.openstack.org/484476 | 17:52 |
| *** Rockyg has quit IRC | 17:54 | |
| *** yamahata has quit IRC | 17:54 | |
| *** iyamahat has quit IRC | 17:54 | |
| mordred | Shrews: woot! | 17:54 |
| mordred | Shrews, pabelanger: do we think we can just land the system-config change and magic will happen? or do we need to schedule that one? | 17:56 |
| Shrews | mordred: i'm not sure. i'm hoping for the magic bit, myself | 17:58 |
| pabelanger | mordred: Shrews: I think the magic will happen, if we want to do now I can help support | 17:58 |
| mordred | Shrews, pabelanger: are the builders already py3? | 17:59 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack-infra/shade master: Updated from global requirements https://review.openstack.org/483327 | 17:59 |
| Shrews | mordred: no, py2, but they are running off of master branch | 17:59 |
| pabelanger | right | 18:00 |
| Shrews | mordred: removing the py2 jobs from feature/zuulv3 branch is the logical next step | 18:00 |
| mordred | ++ | 18:00 |
| mordred | unless we wanted to switch the builders to running off feature/zuulv3 first so that we can verify those work in py3 too | 18:00 |
| mordred | Shrews: but yes, I tink we'll be set for that | 18:01 |
| pabelanger | mordred: are we okay with running production on feature/zuulv3 branch? | 18:01 |
| pabelanger | maybe we should merge master -> feature/zuulv3 first | 18:01 |
| pabelanger | then do builders? | 18:01 |
| mordred | pabelanger: maybe so? I mean, the builders code should be the same at this point mostly right? | 18:02 |
| mordred | so yes- if we merge master->feature/zuulv3, then it should be safe to run builders from v3 branch? | 18:02 |
| openstackgerrit | Merged openstack-infra/puppet-nodepool master: Add support for python 3 https://review.openstack.org/485245 | 18:02 |
| pabelanger | mordred: I think we need to add logic for nodepoolv3.yaml to builders | 18:03 |
| pabelanger | since that is a large config file change | 18:03 |
| mordred | ah. maybe let's not open that door just yet then | 18:03 |
| openstackgerrit | Merged openstack-infra/puppet-openstackci master: Add support for python 3 for nodepool https://review.openstack.org/485246 | 18:03 |
| pabelanger | mordred: ya, that is my only real concern atm | 18:03 |
| pabelanger | mordred: Shrews: however, we could bring nb01 / nb02 back online and hook them up to zuulv3.o.o? Then we'd be in a good place to cut them back into production when the time comes | 18:04 |
| *** tosky has joined #openstack-infra | 18:04 | |
| *** EricGonc_ has quit IRC | 18:04 | |
| mordred | pabelanger: wouldn't that double-build images? | 18:05 |
| *** electrofelix has quit IRC | 18:05 | |
| pabelanger | ya, but only ubuntu-xenial ATM | 18:05 |
| pabelanger | and only in infra-cloud | 18:05 |
| mordred | oh right | 18:05 |
| *** jkilpatr has quit IRC | 18:06 | |
| *** ociuhandu_ has quit IRC | 18:06 | |
| openstackgerrit | Merged openstack-infra/puppet-zuul master: Add web proxying to non-SSL vhost too https://review.openstack.org/485294 | 18:06 |
| mordred | pabelanger: yah - maybe that's a good step - it'll at least make sure there is a running copy | 18:06 |
| Shrews | i vote for keeping it as simple as possible right now and do builder migration separately and at a later point | 18:06 |
| Shrews | (like, soon, but let's not shoot for that today) :) | 18:07 |
| pabelanger | mordred: Shrews: okay, I can propose patches for nb01 / nb02 later but might be good to get that in place before ptg, just to be safe | 18:07 |
| *** EricGonczer_ has joined #openstack-infra | 18:07 | |
| Shrews | pabelanger: totally, yeah | 18:07 |
| Shrews | definitely should happen well before ptg | 18:08 |
| pabelanger | agree | 18:08 |
| *** milan has quit IRC | 18:10 | |
| pabelanger | Shrews: jeblair mordred: so did we want to upgrade to py3 for nl01.o.o now? | 18:10 |
| mordred | pabelanger: I vote yes - we're around and can watch it, and if there's a problem it's better to know now than later | 18:11 |
| *** baoli_ has quit IRC | 18:11 | |
| *** iyamahat has joined #openstack-infra | 18:12 | |
| *** EricGonc_ has joined #openstack-infra | 18:12 | |
| *** EricGonczer_ has quit IRC | 18:13 | |
| *** baoli has joined #openstack-infra | 18:15 | |
| *** milan has joined #openstack-infra | 18:15 | |
| pabelanger | wfm | 18:15 |
| *** u_nuSLASHkm8 has joined #openstack-infra | 18:16 | |
| *** boris-42___ has joined #openstack-infra | 18:17 | |
| *** u_nuSLASHkm8 has left #openstack-infra | 18:17 | |
| *** raissa_ has joined #openstack-infra | 18:18 | |
| *** ekcs has quit IRC | 18:20 | |
| *** Julien-zte has quit IRC | 18:20 | |
| *** jkilpatr has joined #openstack-infra | 18:22 | |
| *** ekcs has joined #openstack-infra | 18:22 | |
| *** Julien-zte has joined #openstack-infra | 18:23 | |
| *** dprince has joined #openstack-infra | 18:23 | |
| *** trown|lunch is now known as trown | 18:23 | |
| *** bobh has quit IRC | 18:24 | |
| *** dprince has quit IRC | 18:26 | |
| *** baoli has quit IRC | 18:27 | |
| *** rbrndt has joined #openstack-infra | 18:28 | |
| *** ociuhandu_ has joined #openstack-infra | 18:29 | |
| *** yamahata has joined #openstack-infra | 18:31 | |
| *** trown is now known as trown|mtg | 18:35 | |
| *** panda|bbl is now known as panda | 18:36 | |
| AJaeger | pabelanger: thanks. Then I'll +2 project-config changes as well... | 18:36 |
| *** bobh has joined #openstack-infra | 18:38 | |
| openstackgerrit | Corey Bryant proposed openstack-infra/project-config master: snaps: add snap-cinder definitions https://review.openstack.org/485276 | 18:47 |
| fungi | yeesh... the spam flood to openstack-owner@lists.o.o continues: we've now blocked over 2500 messages for that address in under 24 hours | 18:54 |
| *** jkilpatr has quit IRC | 18:56 | |
| *** dprince has joined #openstack-infra | 18:56 | |
| *** Swami has joined #openstack-infra | 18:57 | |
| *** jamesmcarthur has quit IRC | 18:58 | |
| *** adisky__ has quit IRC | 18:58 | |
| *** jamesmcarthur has joined #openstack-infra | 18:59 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: WIP: Check out implicit branch in timer jobs https://review.openstack.org/485329 | 19:00 |
| *** jkilpatr has joined #openstack-infra | 19:05 | |
| *** jamesmcarthur has quit IRC | 19:07 | |
| openstackgerrit | Ashwin Agate proposed openstack-infra/project-config master: Enable py35 job for monasca-transform https://review.openstack.org/484522 | 19:13 |
| *** chlong_ has joined #openstack-infra | 19:15 | |
| *** tinyurl_comSLASH has joined #openstack-infra | 19:17 | |
| *** gyee has joined #openstack-infra | 19:18 | |
| *** tinyurl_comSLASH has left #openstack-infra | 19:18 | |
| *** trown|mtg is now known as trown | 19:20 | |
| openstackgerrit | Devang Doshi proposed openstack-infra/project-config master: new-project: vmware-nsx-tempest-plugin https://review.openstack.org/484990 | 19:26 |
| AJaeger | fungi that's sad ;( | 19:28 |
| *** tinyurl_comSLASH has joined #openstack-infra | 19:28 | |
| *** tinyurl_comSLASH has left #openstack-infra | 19:29 | |
| openstackgerrit | Devang Doshi proposed openstack-infra/project-config master: new-project: vmware-nsx-tempest-plugin https://review.openstack.org/484990 | 19:30 |
| fungi | AJaeger: sadder for fifieldt who was receiving all those before we started blocking it for him | 19:30 |
| AJaeger | Oh, yeah ;( | 19:31 |
| fungi | apparently it's been hitting a lot of wmf's mailing list owners too | 19:31 |
| fungi | but they have a slightly different setup than we do | 19:31 |
| fungi | also they're trying to play whack-a-mole with combinations of sender domain and subject line | 19:31 |
| *** makowals has quit IRC | 19:32 | |
| *** eharney has quit IRC | 19:32 | |
| fungi | which seems almost as painful to keep up with | 19:32 |
| *** rossella_ has quit IRC | 19:32 | |
| *** rossella_ has joined #openstack-infra | 19:33 | |
| AJaeger | ;/ | 19:34 |
| openstackgerrit | Ashwin Agate proposed openstack-infra/project-config master: Enable py35 job for monasca-transform https://review.openstack.org/484522 | 19:37 |
| *** dangers is now known as dangers_away | 19:39 | |
| openstackgerrit | Ashwin Agate proposed openstack-infra/project-config master: Enable py35 job for monasca-transform https://review.openstack.org/484522 | 19:46 |
| *** MasterOfBugs has joined #openstack-infra | 19:46 | |
| *** pramodrj07 has joined #openstack-infra | 19:47 | |
| *** trown is now known as trown|brb | 19:49 | |
| *** dprince has quit IRC | 19:50 | |
| *** Marx314 has joined #openstack-infra | 19:51 | |
| *** coolsvap has quit IRC | 19:51 | |
| *** dougwig has joined #openstack-infra | 19:53 | |
| *** annegentle has quit IRC | 19:55 | |
| *** trown|brb is now known as trown | 19:58 | |
| *** rwsu has quit IRC | 20:00 | |
| *** chlong_ has quit IRC | 20:03 | |
| *** Daisy_ has joined #openstack-infra | 20:04 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/puppet-zuul master: Send websocket traffic to ws:// protocol https://review.openstack.org/485345 | 20:05 |
| *** dtroyer has quit IRC | 20:05 | |
| mordred | jeblair, Shrews: ^^ | 20:06 |
| mordred | fungi: don't know if you saw in #zuul - but console streaming in web browser now works | 20:06 |
| *** rwsu has joined #openstack-infra | 20:06 | |
| *** nicolasbock has quit IRC | 20:06 | |
| fungi | i did not! awesomesauce | 20:06 |
| *** thorst_ has quit IRC | 20:07 | |
| *** Daisy_ has quit IRC | 20:09 | |
| *** jcoufal_ has joined #openstack-infra | 20:09 | |
| pabelanger | fungi: clarkb: so, looking into some failures with tripleo jobs and seeing some high failures on http://mirror.sto2.citycloud.openstack.org for these jobs. Specifically it looks like we might have random failures connecting to the mirror. If I look into the apache logs for said mirror, I fail to see the request. Additionally, I see the FIP of the job in the apache logs, I wonder if we should see if | 20:09 |
| pabelanger | we can have a job use the shared tenant network and avoid sending the traffic via FIP network | 20:09 |
| *** thorst has joined #openstack-infra | 20:10 | |
| *** jcoufal has quit IRC | 20:12 | |
| clarkb | pabelanger: the fip is used because the floating IP is what is in dns so to go from $VM to mirror you have to get on the real internets | 20:12 |
| clarkb | pabelanger: are we sure that tripleo isn't breaking in their local proxy setup? | 20:13 |
| pabelanger | clarkb: ya, DNS today hits interwebs, was considering hacking /etc/hosts to see if there was a difference | 20:13 |
| pabelanger | clarkb: pretty sure they have it setup right, I can see other requests working, until 1 package fails | 20:14 |
| pabelanger | but I don't see that request | 20:14 |
| clarkb | pabelanger: I would just remove the proxy and see if it magically fixes it | 20:14 |
| *** thorst has quit IRC | 20:14 | |
| pabelanger | clarkb: which proxy are you referring too? | 20:15 |
| clarkb | pabelanger: iirc tripleo runs local squid proxy in all jobs | 20:15 |
| pabelanger | clarkb: right, this is on citycloud so no squid setup | 20:15 |
| *** makowals has joined #openstack-infra | 20:15 | |
| *** annegentle has joined #openstack-infra | 20:16 | |
| pabelanger | clarkb: and from random sample, it seems limited to sto2 | 20:16 |
| clarkb | pabelanger: does it affect not tripleo too? when I was tracking things the vast majority of these failures were related to dns resolution over ipv4 in osic | 20:16 |
| clarkb | pabelanger: with that cleaned out we still have random external resource failures but were distributed across all regions fairly evenly | 20:17 |
| pabelanger | clarkb: I am not sure, I haven't look. I was only focusing on failures in the gate | 20:17 |
| pabelanger | as they are at 20+hr backlog | 20:17 |
| *** jkilpatr has quit IRC | 20:18 | |
| *** Sukhdev_ has quit IRC | 20:19 | |
| pabelanger | panda: EmilienM: maybe you should have yum try and fail back to rdoproject.org too with mirrorlist setting in yum repo too? That way you have 2 chances to pickup the package | 20:19 |
| *** eharney has joined #openstack-infra | 20:20 | |
| *** Sukhdev has joined #openstack-infra | 20:20 | |
| clarkb | pabelanger: ya if you look at http://status.openstack.org/elastic-recheck/#1449136 that type of error is fairly well distributed across the regions | 20:21 |
| clarkb | though infracloud might be abnormally high | 20:21 |
| *** jascott1 has joined #openstack-infra | 20:23 | |
| fungi | in other jobs we wrap package download actions in a limited retry loop too, as a means of mitigating that | 20:23 |
| inc0 | fungi: so, regarding ZUUL_BRANCH thing, it seems my change broke tarballs but what I did is I copied https://github.com/openstack/kolla-ansible/blob/7333ae330fb95ae48fb24e5370c7c45110fa859a/tools/setup_gate.sh#L173 | 20:24 |
| inc0 | which works for kolla-ansible | 20:24 |
| inc0 | how come it doesn't work for kolla? | 20:24 |
| fungi | inc0: are they running that in periodic or post? | 20:24 |
| inc0 | ah...right...post | 20:24 |
| pabelanger | clarkb: http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22No%20more%20mirrors%20to%20try%5C%22%20AND%20voting%3A1 are the centos failures | 20:25 |
| pabelanger | clarkb: large amount is coming from citycloud | 20:25 |
| fungi | inc0: post pipeline has branches because the jobs are triggered by the merging of a commit to a particular branch. periodic jobs are just triggered on a timer and so have no such context | 20:25 |
| inc0 | well this is post | 20:25 |
| inc0 | and it doesn't seem to take branch as well | 20:25 |
| inc0 | https://review.openstack.org/#/c/475469/15/tools/gate_run.sh | 20:25 |
| fungi | inc0: i thought the brokenness was just on your registry tarballs which were being built from the periodic pipeline? | 20:26 |
| clarkb | pabelanger: ya though not exclusively so, possibly an issue on our end that is just made owrse in citycloud (similar to how we were using ipv4 dns in osic) | 20:26 |
| *** annegentle has quit IRC | 20:26 | |
| inc0 | they're not built in periodic yet I think | 20:26 |
| inc0 | they're trigerred by commit | 20:26 |
| pabelanger | fungi: Ya, I need to confirm but think yum tries 10 times by defulat | 20:26 |
| inc0 | we want to make them periodic | 20:26 |
| inc0 | so that's a good piece of info there but not relevant yet | 20:27 |
| pabelanger | clarkb: right, yum is smart enought to try multiple mirrors, so maybe we just configure centos jobs to try upstream direct mirror if AFS route fails | 20:28 |
| clarkb | pabelanger: any idea if those are packages that are hosted behind our caching proxy or the afs hosted mirror data? | 20:28 |
| fungi | inc0: so the tarballs at http://tarballs.openstack.org/kolla/images/ aren't being built from periodic jobs yet? | 20:28 |
| pabelanger | clarkb: seems to be a mix | 20:29 |
| clarkb | pabelanger: its possible that proxy cache could be failing when backend fails OR afs backed is failing on timeout type things when we republish the index | 20:29 |
| inc0 | nope fungi | 20:29 |
| clarkb | er XOR | 20:29 |
| clarkb | I gues sit could be both | 20:29 |
| pabelanger | clarkb: ya, I am seeing failures from AFS and reverse proxy | 20:29 |
| inc0 | fungi: https://github.com/openstack-infra/project-config/blob/master/zuul/layout.yaml#L10892 | 20:29 |
| pabelanger | clarkb: I am guessing it is a routing issue, but not confirmed | 20:30 |
| *** annegentle has joined #openstack-infra | 20:30 | |
| openstackgerrit | John L. Villalovos proposed openstack-infra/devstack-gate master: DNM/WIP: Trying to test boot-from-volume for Ironic https://review.openstack.org/485348 | 20:31 |
| fungi | inc0: thanks, probably best if i start reading at the layout entrypoint and following this through configuration | 20:31 |
| inc0 | so file I linked is most likely one that broke | 20:31 |
| *** kgiusti has left #openstack-infra | 20:31 | |
| inc0 | as this is only place that handles this thing | 20:32 |
| fungi | inc0: so it's this template? http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/kolla.yaml#n60 | 20:32 |
| inc0 | no fungi not yet | 20:32 |
| inc0 | this tpl you linked is for my new style publisher | 20:32 |
| fungi | sorry, i meant http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/kolla.yaml#n28 | 20:32 |
| inc0 | which will be periodic | 20:32 |
| fungi | kolla-dsvm-build-{distro}-{type}-{node}{suffix} | 20:33 |
| inc0 | yes, that's correct | 20:33 |
| fungi | okay, now i'll pull up a log | 20:33 |
| pabelanger | clarkb: EmilienM: panda: I would try setting retries: 0 in yum.conf and see what happens. from man: retries Set the number of times any attempt to retrieve a file should retry before returning an error. Setting this to `0' makes yum try forever. Default is `10'. | 20:34 |
| pabelanger | clarkb: EmilienM: panda: might be able to see a pattern with the test | 20:35 |
| *** annegentle has quit IRC | 20:35 | |
| *** jkilpatr has joined #openstack-infra | 20:36 | |
| fungi | inc0: so here's the most recent run for master demonstrating the issue http://logs.openstack.org/e4/e44122c782cade92f43359a9034edf0d0d6cbf1d/post/kolla-dsvm-build-centos-binary-centos-7/3d0ab5d/console.html#_2017-07-19_10_06_50_542833 | 20:37 |
| fungi | inc0: aha, yes so in post jobs you need to key on $ZUUL_REFNAME instead of $ZUUL_BRANCH (these are mostly named for corresponding parameters in the gerrit events which trigger them) http://logs.openstack.org/e4/e44122c782cade92f43359a9034edf0d0d6cbf1d/post/kolla-dsvm-build-centos-binary-centos-7/3d0ab5d/_zuul_ansible/vars.yaml | 20:38 |
| inc0 | fungi: ok, so REFNAME it is, how about periodic? | 20:38 |
| inc0 | REFNAME will work too? | 20:38 |
| fungi | inc0: periodic jobs don't have any context for a branch or ref. you need the job to spell out what to checkout | 20:39 |
| fungi | usually this is done with a job parameter in the job-template | 20:39 |
| EmilienM | pabelanger: in the ansible role that deploys repos you mean? | 20:39 |
| inc0 | mhm...ok so that's gonna be problem. yeah job template probably it'll be | 20:39 |
| fungi | inc0: right, for now this is generally done like so http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/python-bitrot-jobs.yaml | 20:40 |
| inc0 | speaking of which, what vars will I get in periodic? | 20:40 |
| *** jamesmcarthur has joined #openstack-infra | 20:41 | |
| fungi | inc0: http://logs.openstack.org/periodic/keystone-propose-translation-update/959820d/_zuul_ansible/vars.yaml | 20:41 |
| fungi | for an example | 20:41 |
| fungi | inc0: you'll be happy to see this is slated to improve in a few months as well https://review.openstack.org/485329 | 20:43 |
| inc0 | ok...let me fix our current tarballs and I'll think how to deal with this stuff | 20:44 |
| *** jamesmcarthur has quit IRC | 20:45 | |
| inc0 | I'll probably gonna do something else, publish tarballs in post | 20:45 |
| inc0 | but upload in periodic | 20:45 |
| inc0 | at least for start | 20:46 |
| *** dtroyer has joined #openstack-infra | 20:46 | |
| inc0 | pabelanger: btw I just packed all of images from ceontos-source with docker save. 12G | 20:47 |
| *** chlong_ has joined #openstack-infra | 20:48 | |
| inc0 | http://paste.openstack.org/show/615905/ | 20:48 |
| *** trown is now known as trown|outtypewww | 20:49 | |
| fungi | inc0: for an example of how the dsvm stable periodic jobs are being instantiated right now, see http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/projects.yaml#n3684 and the corresponding job groups/templates in http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/devstack-gate.yaml | 20:50 |
| pabelanger | EmilienM: ya, or start of your job | 20:50 |
| fungi | inc0: that might be more relevant since your builds are under the devstack-gate framework, since you can see how the branch overrides are passed in | 20:51 |
| EmilienM | pabelanger: I or panda will give a try - I'm on something now and I was about to leave in 10 min | 20:51 |
| inc0 | right, I need to parse what I'm reading | 20:51 |
| EmilienM | but back in evening | 20:51 |
| EmilienM | pabelanger: hard to reproduce, since we have to hit a specific cloud | 20:52 |
| EmilienM | pabelanger: also not sure about the rate of failure, do you have numbers? | 20:52 |
| pabelanger | inc0: docker save is uncompressed, you'll need to pipe to gzip: see http://git.openstack.org/cgit/openstack/dox/tree/tools/tarball.sh | 20:53 |
| panda | queue is hitting 23 hours wait now... | 20:53 |
| pabelanger | EmilienM: see logstash url I linked to clarkb above | 20:54 |
| EmilienM | ok | 20:54 |
| * EmilienM clicks | 20:54 | |
| inc0 | trying | 20:54 |
| EmilienM | pabelanger: ok yeah it's a lot | 20:55 |
| EmilienM | pabelanger: let me try this now and leave | 20:55 |
| EmilienM | weshay: ^ | 20:55 |
| pabelanger | EmilienM: panda: but setting up mirrorlist with our infra mirrors first, then real upstream mirrors 2nd might also help | 20:55 |
| EmilienM | pabelanger: I'm confused on what you're asking here ^ | 20:56 |
| EmilienM | I haven't been following all the bits today on this topic, I might have missed something | 20:56 |
| pabelanger | EmilienM: yum repo files support mirrorlist, today we have baseurl. mirror list would try a 2nd mirror, if properly seutp | 20:56 |
| pabelanger | EmilienM: I'm would try failing back to actually repo if our local mirror failed | 20:57 |
| pabelanger | yum would do that automagically, if configured | 20:57 |
| panda | pabelanger: what's the url for the infra mirrorlist ? | 20:57 |
| pabelanger | panda: you'd have to build it, and see if it supported file:// | 20:58 |
| pabelanger | otherwise, we'd have to see about creating one on http server | 20:58 |
| fungi | panda: the hostnames are constructed procedurally from our provider and region names in the nodepool configuration | 20:58 |
| openstackgerrit | Monty Taylor proposed openstack-infra/puppet-zuul master: Send websocket traffic to ws:// protocol https://review.openstack.org/485345 | 20:59 |
| fungi | panda: "mirror.$REGION.$PROVIDER.openstack.org" basically | 20:59 |
| panda | can I build a mirrorlist from the mirror_info.sh file ? | 20:59 |
| * EmilienM needs to go | 20:59 | |
| EmilienM | I'll follow-up on panda's work. panda: thanks for looking :-) let me know before you leave if I can review or continue something | 21:00 |
| fungi | panda: the relevant mirror hostname and urls for the node are defined in that shell snippet | 21:00 |
| fungi | so depends on what you mean by "build from" | 21:00 |
| *** jamesmcarthur has joined #openstack-infra | 21:01 | |
| *** esberglu has quit IRC | 21:02 | |
| fungi | panda: so this is an example from a live node just now: http://paste.openstack.org/show/615907 | 21:02 |
| fungi | line 1 sets the NODEPOOL_MIRROR_HOST variable for the provider and region where that node is running, and then the subsequent variables in turn use that to construct use-specific mirror urls | 21:03 |
| *** chlong_ has quit IRC | 21:04 | |
| fungi | worth noting that the shebang in that resulting script is pointless | 21:05 |
| fungi | i wonder whether we shouldn't do some sort of inline replacement instead to inject the current line 1 further down in the script, maybe after the copyright/license | 21:05 |
| *** jamesmcarthur has quit IRC | 21:06 | |
| fungi | but anyway, it works if you source it, so fine for now | 21:06 |
| panda | now I'm 100% confused. Let's recap. We have a queue of 22 hours, because there's a faulty mirror somewhere and we have to cycle through all infra mirror if something fails. We don't have the list of mirror to try anywhere, and even mirror_info.sh can't be used. we already use region specific mirrors, but apparently that's not enough. What do we have left ? | 21:08 |
| panda | pabelanger: ^ | 21:10 |
| pabelanger | I don't think the mirror is faulting, I think we are having routing issues from job to mirror. See back scroll about apache2 logs. So 2 options, setup retries=0 in yum.conf, to try forever to get the package (maybe this will give us more data on failures) and other is to create a mirrorlist to try both openstack-infra mirror and the original upstream mirror. That gives another chance to download | 21:10 |
| pabelanger | said package | 21:10 |
| fungi | if that region is suffering from network issues, we can turn it down | 21:11 |
| pabelanger | 3rd option would be to setup /etc/hosts in citycloud-sto2 as the private IP access and use tenat network over FIP, again seeing if this is cloud networking problems | 21:11 |
| SpamapS | wow.. the openstack packages in Debian have stalled out pretty hard w/o Zigo. | 21:12 |
| SpamapS | zigo: or are you working again? | 21:12 |
| zigo | SpamapS: Nobody did anything since I stopped. | 21:12 |
| SpamapS | just noticed PBR is ancient... lots of things depending on 2.0.0 + | 21:12 |
| pabelanger | I _think_ openstackci-zuul and openstackci-nodepool projects are on the same network | 21:12 |
| *** sambetts is now known as sambetts|afk | 21:13 | |
| *** shardy has quit IRC | 21:13 | |
| zigo | I really wish I could do it just on week-ends, but maintaining OpenStack isn't something you can do only on spare times. | 21:13 |
| zigo | It wouldn't be serious. | 21:13 |
| pabelanger | SpamapS: oh, that reminds me. I wanted to propose a patch to remove track-upstream in project-config for them | 21:13 |
| zigo | SpamapS: I'm still looking for a new employer, and it's getting critical for me now... :( | 21:13 |
| *** thorst has joined #openstack-infra | 21:14 | |
| *** esberglu has joined #openstack-infra | 21:14 | |
| panda | pabelanger: ok, trying to set retries on yum.conf | 21:16 |
| SpamapS | zigo: I'd guess there aren't that many packaging jobs right now. Might want to try learning Docker and Helm or something. ;-) | 21:16 |
| *** vhosakot has quit IRC | 21:17 | |
| zigo | SpamapS: As if I could do only packaging ... | 21:17 |
| zigo | SpamapS: I've been doing programming since I'm 15 y/o. | 21:17 |
| zigo | SpamapS: I contributed to about 75 OpenStack projects (most of was bug fixes). | 21:18 |
| *** ldnunes has quit IRC | 21:18 | |
| fungi | more that keeping up with the packaging would be a full-time effort (we all know you're a programmer!) | 21:19 |
| *** thorst has quit IRC | 21:19 | |
| zigo | fungi: :) | 21:20 |
| *** makowals has quit IRC | 21:22 | |
| zigo | pabelanger: Is there a way yet to have a Stretch image in infra? | 21:22 |
| pabelanger | zigo: ya, should be easier now | 21:24 |
| pabelanger | just need to make sure DIB can build it | 21:24 |
| *** thorst has joined #openstack-infra | 21:25 | |
| zigo | pabelanger: Should i try to run DIB again, and see what happens? | 21:25 |
| pabelanger | zigo: ya, the script in project-config should be better now. we remove the need for puppet now | 21:26 |
| zigo | Sounds great! | 21:26 |
| * zigo tries | 21:26 | |
| clarkb | someone was fixing bugs | 21:26 |
| *** makowals has joined #openstack-infra | 21:26 | |
| clarkb | there is at least one known issue unsure if fixerged yet | 21:26 |
| panda | pabelanger: EmilienM:https://review.openstack.org/485361 | 21:29 |
| *** ihrachys has quit IRC | 21:29 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Remove track-upstream for deb-packaging team https://review.openstack.org/485362 | 21:31 |
| inc0 | pabelanger: you were right, with gzip -9 tarballed registry is similar in size as docker save | 21:32 |
| pabelanger | zigo: fungi: ^ based on ML email from a while back (track-upstream) | 21:32 |
| pabelanger | inc0: yup, it should give you the same info, with out the need of depending on docker registry install on the local machince | 21:33 |
| inc0 | still probably keeping registry is more efficient because of multinode, but file sizes are similar | 21:33 |
| pabelanger | machine* | 21:33 |
| inc0 | well, registry is meant for multinode and that's our main concern | 21:33 |
| pabelanger | right, but if you export images to tarballs.o.o, and I wanted to use them locally, it would be more things to setup | 21:34 |
| inc0 | yup, so I guess it depends on case;) tarballs was mostly for gates | 21:34 |
| pabelanger | we can deal with that later, your testing now shouldn't affect that | 21:35 |
| *** isviridov_away has quit IRC | 21:35 | |
| *** vaidy has quit IRC | 21:35 | |
| inc0 | yeah, I already have all the code to deal with registry stuff | 21:35 |
| *** priteau has quit IRC | 21:35 | |
| inc0 | also again, from multinode perspective it's more efficient | 21:35 |
| pabelanger | inc0: right, the only use case I am thinking of, is if we want to keep an artifict of the images on tarball.o.o and upload to registery. This is what we do with pypi today | 21:35 |
| *** EricGonc_ has quit IRC | 21:36 | |
| *** bobh has quit IRC | 21:36 | |
| inc0 | I don't think we want general population to download stuff from tarballs | 21:36 |
| inc0 | I'd rather have dockerhub eat up all the traffic required | 21:36 |
| pabelanger | right, same for pypi.python.org, we want people to get the artifact from it, but it is also good policy for use to keep the original | 21:37 |
| pabelanger | again, stuff we'll work out in the spec. it shouldn't block your testing | 21:38 |
| *** rloo has quit IRC | 21:38 | |
| *** EricGonczer_ has joined #openstack-infra | 21:38 | |
| *** makowals has quit IRC | 21:40 | |
| *** boris-42___ has quit IRC | 21:41 | |
| inc0 | btw I have different apporach for easy setup for local images that works for both cases | 21:41 |
| inc0 | https://review.openstack.org/#/c/484961/1/tools/registry/Dockerfile | 21:41 |
| *** Sukhdev has quit IRC | 21:42 | |
| *** Sukhdev has joined #openstack-infra | 21:43 | |
| *** baoli has joined #openstack-infra | 21:47 | |
| *** Daisy_ has joined #openstack-infra | 21:47 | |
| *** baoli has quit IRC | 21:48 | |
| *** marst_ has joined #openstack-infra | 21:48 | |
| *** gouthamr has quit IRC | 21:49 | |
| *** baoli has joined #openstack-infra | 21:50 | |
| *** Daisy_ has quit IRC | 21:51 | |
| *** marst has quit IRC | 21:52 | |
| *** marst_ has quit IRC | 21:57 | |
| *** panda is now known as panda|off | 21:59 | |
| *** EricGonczer_ has quit IRC | 22:11 | |
| *** jcoufal_ has quit IRC | 22:13 | |
| *** felipemonteiro_ has quit IRC | 22:15 | |
| *** gordc has quit IRC | 22:16 | |
| ianw | pabelanger: hmm -> /usr/sbin/setfiles: Could not set context for /usr/lib/systemd/systemd-modules-load: Invalid argument ... that's new | 22:16 |
| ianw | and i didn't see it in my local build | 22:17 |
| *** rwsu has quit IRC | 22:17 | |
| pabelanger | ianw: oh, possible I messed something up in my patch | 22:18 |
| *** gouthamr has joined #openstack-infra | 22:18 | |
| *** rossella__ has joined #openstack-infra | 22:19 | |
| *** Apoorva has quit IRC | 22:21 | |
| *** Apoorva has joined #openstack-infra | 22:22 | |
| ianw | ahhhh ... this was on the centos build host | 22:22 |
| *** rossella_ has quit IRC | 22:22 | |
| ianw | i wonder if this is at the point that centos kernel doesn't support whatever setfiles is doing | 22:23 |
| ianw | that would be ... annoying | 22:23 |
| *** baoli has quit IRC | 22:26 | |
| openstackgerrit | Devang Doshi proposed openstack-infra/project-config master: new-project: vmware-nsx-tempest-plugin https://review.openstack.org/484990 | 22:27 |
| *** makowals has joined #openstack-infra | 22:28 | |
| *** sdague has quit IRC | 22:33 | |
| *** baoli has joined #openstack-infra | 22:36 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Fix change history cycle detection https://review.openstack.org/485368 | 22:36 |
| *** vhosakot has joined #openstack-infra | 22:37 | |
| *** dizquierdo has joined #openstack-infra | 22:37 | |
| *** dizquierdo has quit IRC | 22:39 | |
| inc0 | guys, is zuul alive? | 22:47 |
| inc0 | 2hrs in queue and none of jobs even startedf | 22:48 |
| pabelanger | zuul is running, checking nodepool | 22:50 |
| pabelanger | osic doesn't look happy | 22:51 |
| mnaser | job stats dont look fun | 22:53 |
| pabelanger | jeblair: mordred: nodepool-launcher seems to be hung | 22:53 |
| pabelanger | jeblair: mordred: how should I proceed with debug? dump threads? | 22:54 |
| fungi | i wonder if zuul is in the throes of a reconfigure event? | 22:54 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Remove export commands from tox based roles https://review.openstack.org/483936 | 22:54 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Remove subunit file size check for tox role https://review.openstack.org/484519 | 22:54 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Compress testrepository.subunit in fetch-testr-output https://review.openstack.org/484896 | 22:54 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Remove check_sudo_usage logic from tox https://review.openstack.org/484916 | 22:54 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Create tox-pep8 jobs https://review.openstack.org/484948 | 22:54 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Remove pip freeze logic https://review.openstack.org/484917 | 22:54 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Require tox_envlist for tox role https://review.openstack.org/484975 | 22:54 |
| pabelanger | I do see some exections in launcher-debug.log | 22:55 |
| jeblair | hrm | 22:56 |
| fungi | looks like it's waking back up again | 22:56 |
| pabelanger | strace seems to be stuck at pause | 22:56 |
| *** thorst has quit IRC | 22:56 | |
| jeblair | pabelanger: that's normal. | 22:56 |
| pabelanger | jeblair: cool | 22:57 |
| pabelanger | have limited knowledge of strace sadly | 22:57 |
| pabelanger | deleter is still running | 22:57 |
| jeblair | pabelanger: if you send nodepool (or zuul) sigusr2, they will stack dump. i've done that. | 22:58 |
| jeblair | though i don't see any output | 22:59 |
| fungi | hrm, yeah, basically no ready nodes and a steady count of building nodes since about 20:15 according to the graphs | 22:59 |
| *** jamesden_ has quit IRC | 23:00 | |
| pabelanger | grafana.o.o says we launched 5 nodes in infracloud about 30mins ago, but I don't see that in the logs | 23:01 |
| *** rlandy has quit IRC | 23:03 | |
| *** yamamoto has joined #openstack-infra | 23:04 | |
| jeblair | i'm not sure it's worth debugging with gdb. i vote we restart it. | 23:05 |
| fungi | yeah, if it won't cough up a thread dump... | 23:05 |
| mordred | I'm SUPER excited about the launcher stack traces | 23:05 |
| pabelanger | jeblair: ++ | 23:05 |
| fungi | not like a nodepool restart will have any noticeable negative impact on jobs | 23:06 |
| jeblair | mordred: okay for me to restart? | 23:06 |
| mordred | jeblair: yup | 23:06 |
| mordred | <class 'oslo_db.exception.DBConnectionError'> (Inner Exception: Error fetching server list on osic-cloud1:RegionOne:. (500) Server Error for url: https://cloud1.osic.xyz:8774/v2.1/7e9c4d5842b3451d94417bd0af03a0f4/servers/detail Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. | 23:06 |
| pabelanger | there we go | 23:06 |
| mordred | so - osic bombed out in some way | 23:06 |
| mordred | cloudnull: ^^ we're seeing that in our logs in case someone wants to look at it on the osic side | 23:07 |
| jeblair | #status log restarted nodepool-launcher which had frozen (did not respond to SIGUSR2) | 23:07 |
| openstackstatus | jeblair: finished logging | 23:07 |
| fungi | and that bubbled up through oslo_db? | 23:07 |
| mordred | that winds up in the error message text in the http response | 23:07 |
| fungi | ahh | 23:07 |
| mordred | when nova throws an exception | 23:07 |
| mordred | latest shade actually logs that for us - which is new | 23:08 |
| mordred | so that's kinda cool | 23:08 |
| mordred | to see it in action | 23:08 |
| fungi | better than flying blind, for sure | 23:08 |
| jeblair | 1367 building nodes | 23:09 |
| pabelanger | ya, we'll going to max our all the clouds :D | 23:09 |
| fungi | they just started to show up on the status graph too. quite a wall of server | 23:09 |
| *** rbrndt has quit IRC | 23:09 | |
| *** rbrndt has joined #openstack-infra | 23:10 | |
| *** rbrndt has quit IRC | 23:10 | |
| pabelanger | ovh didn't like that much launch errors | 23:11 |
| mordred | InvalidRequestError: This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: UPDATE statement on table 'subnode' expected to update 1 row(s); 0 were matched. | 23:12 |
| mordred | I like these new errors - they're fun :) | 23:12 |
| mordred | oh - wait - that's us! | 23:12 |
| mordred | I'd worry more about that if we weren't about to roll out a complete reworking of that system | 23:12 |
| jeblair | mordred: it's the launcher and deleter racing. it's self-healing. | 23:13 |
| mordred | looking at this - I do not like the way we're logging the server dict in the extra field - too many backslashes | 23:13 |
| pabelanger | think we are hitting ovh rabbitmq server a little bit: timed out waiting for a reply to message ID 94d2b5196f8447babd47d1a5357ea267 | 23:14 |
| mordred | but we can deal with that later | 23:14 |
| mordred | we are the world's best openstack chaosmonkey | 23:14 |
| pabelanger | nothing in osic yet, that might be expected based on the exception above | 23:15 |
| mordred | and with that - it's 8am here which means the coffee shops are open - so I'm going to go get a morning coffee | 23:15 |
| *** sbezverk has quit IRC | 23:15 | |
| pabelanger | mordred: enjoy! | 23:15 |
| mordred | please nobody pay attention to the fact that I've been online for 8 hours already | 23:15 |
| *** jaypipes has quit IRC | 23:16 | |
| jeblair | mordred: good morning! ready to start your day?! | 23:16 |
| mordred | yup! | 23:17 |
| jeblair | mordred has discovered the dark side of "it's always 5 o'clock somewhere" -- "it's always 8 am somewhere". | 23:17 |
| mordred | jeblair: well - luckily my coffe place here stays open until 1am - so when I woke up at midnight I was able to go get a cup of coffee to start my day off right | 23:17 |
| jeblair | wow | 23:18 |
| mordred | jeblair: but now that they've re-opened, I can once again go get a cup of coffee and start my day off right | 23:18 |
| *** yamamoto has quit IRC | 23:18 | |
| pabelanger | looks like everything but osic is running nodes again | 23:21 |
| *** tosky has quit IRC | 23:21 | |
| pabelanger | there we go | 23:22 |
| jeblair | i don't think osic has returned any working nodes yet | 23:22 |
| jeblair | but it has started retuning nodes which don't answer ssh | 23:22 |
| pabelanger | ya, I was looking at time to ready graph | 23:23 |
| pabelanger | paramiko exceptions it looks like | 23:24 |
| *** mika has quit IRC | 23:25 | |
| pabelanger | looks like they are coming online now | 23:25 |
| pabelanger | cool | 23:25 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Remove OOM check from tox role https://review.openstack.org/485377 | 23:28 |
| *** hongbin has quit IRC | 23:30 | |
| *** mika has joined #openstack-infra | 23:31 | |
| *** mika has joined #openstack-infra | 23:31 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Fix zuul command default function https://review.openstack.org/484279 | 23:32 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Add job's project as implicit role project https://review.openstack.org/482726 | 23:32 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Remove ansible-role from implied role names https://review.openstack.org/484962 | 23:32 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Fix json output appending https://review.openstack.org/484992 | 23:33 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Emit phase banner separate from play banner https://review.openstack.org/484993 | 23:33 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Remove noise both vertical and horizontal https://review.openstack.org/484994 | 23:33 |
| *** esberglu has quit IRC | 23:33 | |
| *** esberglu has joined #openstack-infra | 23:33 | |
| *** esberglu has quit IRC | 23:33 | |
| *** rcernin has quit IRC | 23:34 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/shade master: Initial commit of zuulv3 jobs https://review.openstack.org/484027 | 23:35 |
| *** eharney has quit IRC | 23:35 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/shade master: Initial commit of zuulv3 jobs https://review.openstack.org/484027 | 23:37 |
| *** coreyob has quit IRC | 23:42 | |
| *** hasharLunch has quit IRC | 23:43 | |
| *** coreyob has joined #openstack-infra | 23:43 | |
| *** Swami has quit IRC | 23:44 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack-infra/project-config master: Convert the OSIC to only use disk nodes https://review.openstack.org/485378 | 23:46 |
| cloudnull | pabelanger: mordred: ^ sorry I got word today that the ssd backed hardware was being powered off, we still have the spinning disk gear, so I've moved the flavor profile there | 23:48 |
| *** dangers_away has quit IRC | 23:48 | |
| cloudnull | I suspect that the dip was due to the hardware being powered down | 23:49 |
| *** rossella__ has quit IRC | 23:50 | |
| cloudnull | to get the cloud back to a happy state, I moved the s5/700 ssd flavors to the disk host aggregate and that PR should remove the old cloud profile | 23:51 |
| cloudnull | which i think is all offline at this point | 23:51 |
| *** makowals has quit IRC | 23:51 | |
| *** rossella__ has joined #openstack-infra | 23:52 | |
| *** dangers_away has joined #openstack-infra | 23:54 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!