| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Start at using CI mirrors for fedora/centos https://review.openstack.org/475607 | 00:01 |
|---|---|---|
| *** aeng has quit IRC | 00:02 | |
| *** sflanigan has quit IRC | 00:03 | |
| *** pahuang has quit IRC | 00:03 | |
| *** chlong has quit IRC | 00:03 | |
| *** jamesmcarthur has joined #openstack-infra | 00:06 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Move debian to default skip lists https://review.openstack.org/475959 | 00:06 |
| *** gmann has quit IRC | 00:06 | |
| *** gmann has joined #openstack-infra | 00:06 | |
| *** ijw has joined #openstack-infra | 00:07 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Move debian to default skip lists https://review.openstack.org/475959 | 00:08 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Use local mirror for ubuntu-minimal jobs https://review.openstack.org/475907 | 00:08 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Start at using CI mirrors for fedora/centos https://review.openstack.org/475607 | 00:08 |
| *** dizquierdo has quit IRC | 00:09 | |
| *** makowals has quit IRC | 00:10 | |
| *** jamesmcarthur has quit IRC | 00:10 | |
| *** ijw has quit IRC | 00:12 | |
| *** ijw has joined #openstack-infra | 00:12 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Move debian to default skip lists https://review.openstack.org/475959 | 00:14 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Use local mirror for ubuntu-minimal jobs https://review.openstack.org/475907 | 00:14 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Start at using CI mirrors for fedora/centos https://review.openstack.org/475607 | 00:14 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Move apt-sources to ubuntu-minimal https://review.openstack.org/475960 | 00:14 |
| *** pramodrj07 has quit IRC | 00:15 | |
| *** Swami has quit IRC | 00:15 | |
| *** sflanigan has joined #openstack-infra | 00:18 | |
| *** sflanigan has joined #openstack-infra | 00:18 | |
| *** aeng has joined #openstack-infra | 00:20 | |
| *** pahuang has joined #openstack-infra | 00:20 | |
| *** amotoki_away is now known as amotoki | 00:23 | |
| *** makowals has joined #openstack-infra | 00:23 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Move apt-sources to ubuntu-minimal / move debian to skip list https://review.openstack.org/475959 | 00:28 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Use local mirror for ubuntu-minimal jobs https://review.openstack.org/475907 | 00:29 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Start at using CI mirrors for fedora/centos https://review.openstack.org/475607 | 00:29 |
| dmsimard | ianw: you happen to know what are the specs for the nodepool-builder VMs ? | 00:31 |
| dmsimard | ianw: trying to build the upstream centos-7 image and it looks like it's complaining because the VM only has 4GB of RAM. | 00:31 |
| ianw | dmsimard: yes ... | 00:31 |
| ianw | umm, you want to turn off the tmpfs thing | 00:32 |
| ianw | nodepool builders are only 8gb | 00:32 |
| ianw | "only" | 00:33 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Move apt-sources to ubuntu-minimal / move debian to skip list https://review.openstack.org/475959 | 00:36 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Use local mirror for ubuntu-minimal jobs https://review.openstack.org/475907 | 00:36 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Start at using CI mirrors for fedora/centos https://review.openstack.org/475607 | 00:36 |
| ianw | dmsimard: DIB_NO_TMPFS might be your issue | 00:36 |
| *** Julien-zte has joined #openstack-infra | 00:37 | |
| dmsimard | ianw: yeah, running disk-image-create -x -t --no-tmpfs -o /var/lib/nodepool/dib/51 centos-minimal vm simple-init openstack-repos nodepool-base cache-devstack growroot infra-package-needs epel | 00:37 |
| * dmsimard looks up DIB_NO_TMPFS | 00:37 | |
| *** dave-mccowan has joined #openstack-infra | 00:37 | |
| ianw | oh that's no-tmpfs anyway | 00:39 |
| ianw | really shouldn't need that much to build ... can you paste log? | 00:39 |
| dmsimard | ianw: https://review.rdoproject.org/nodepool-log/upstream-centos-7.log | 00:40 |
| dmsimard | ianw: using https://github.com/rdo-infra/review.rdoproject.org-config/blob/master/nodepool/nodepool.yaml#L13-L37 with "manually" sync'd project-config nodepool elements and scripts | 00:41 |
| *** LindaWang has joined #openstack-infra | 00:43 | |
| *** gardlt has joined #openstack-infra | 00:45 | |
| *** rlandy has quit IRC | 00:45 | |
| *** makowals has quit IRC | 00:47 | |
| ianw | dmsimard: hmm, is this a bug? the cmd line argument doesn't seem to have applied | 00:47 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Fix config_drive, scheduler_hints and key_name in create_server https://review.openstack.org/475950 | 00:49 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Fix delete_ips on delete_server and add tests https://review.openstack.org/475967 | 00:49 |
| mordred | mmedvede, Shrews: ^^ that should fix the issues - thank you mmedvede for finding a giant hole in our test coverage - and sorry you had to find it | 00:49 |
| mmedvede | \o/ | 00:51 |
| mmedvede | thanks for fixing it, mordred | 00:51 |
| mordred | mmedvede: well - I've said I fixed it like 4 times today and it still wasn't fixed - but hopeefully this time is a winner | 00:52 |
| * mmedvede goes to test mordred 's patches | 00:52 | |
| ianw | dmsimard: that doesn't look like a full log ... it's missing the version stamp at the top? and importantly the argument parsing bits | 00:53 |
| *** rfolco has joined #openstack-infra | 00:53 | |
| mordred | mmedvede: I added tests in https://review.openstack.org/#/c/475967 to cover both the 404 and the config_drive issue | 00:53 |
| mmedvede | ahh, missed it | 00:54 |
| *** ijw has quit IRC | 00:54 | |
| *** ijw has joined #openstack-infra | 00:57 | |
| *** rloo has quit IRC | 00:57 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool feature/zuulv3: Add support for custom ssh port https://review.openstack.org/468752 | 00:57 |
| *** gomarivera has joined #openstack-infra | 00:59 | |
| *** cuongnv has joined #openstack-infra | 00:59 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Fix delete_ips on delete_server and add tests https://review.openstack.org/475967 | 00:59 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Fix config_drive, scheduler_hints and key_name in create_server https://review.openstack.org/475950 | 00:59 |
| mordred | meh. let's actually fix those up and make them well-formed patches | 01:00 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Fix delete_ips on delete_server and add tests https://review.openstack.org/475967 | 01:03 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Fix config_drive, scheduler_hints and key_name in create_server https://review.openstack.org/475950 | 01:03 |
| openstackgerrit | Yolande Amate proposed openstack-infra/jenkins-job-builder master: Update dynamic_scriptler_param_common to make use convert_mapping_to_xml() https://review.openstack.org/475968 | 01:04 |
| mordred | this time with release notes ... | 01:04 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Use local mirror for ubuntu-minimal jobs https://review.openstack.org/475907 | 01:05 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Start at using CI mirrors for fedora/centos https://review.openstack.org/475607 | 01:05 |
| openstackgerrit | Yolande Amate proposed openstack-infra/jenkins-job-builder master: Update dynamic_scriptler_param_common to use convert_xml() https://review.openstack.org/475968 | 01:06 |
| *** hongbin has quit IRC | 01:07 | |
| *** MatthewLi has quit IRC | 01:07 | |
| *** MatthewLi has joined #openstack-infra | 01:07 | |
| mordred | mmedvede: I'm going to try to sleep a little bit on this plane, but when I wake up I'll get those patches landed and get another release cut | 01:07 |
| mmedvede | mordred: thanks. I'll try to test it still in my env just in case. But it all looks good | 01:08 |
| *** makowals has joined #openstack-infra | 01:08 | |
| *** cshastri has joined #openstack-infra | 01:09 | |
| *** rfolco has quit IRC | 01:11 | |
| *** MatthewLi has quit IRC | 01:12 | |
| openstackgerrit | Yolande Amate proposed openstack-infra/jenkins-job-builder master: Update sidebar to make use of convert_mapping_to_xml() https://review.openstack.org/475970 | 01:14 |
| *** jascott1 has quit IRC | 01:18 | |
| *** liujiong has joined #openstack-infra | 01:18 | |
| openstackgerrit | Yolande Amate proposed openstack-infra/jenkins-job-builder master: Update priority_sorter to make use of convert_mapping_to_xml() https://review.openstack.org/475973 | 01:21 |
| *** jamesmcarthur has joined #openstack-infra | 01:21 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Fix image normalization when image has properties property https://review.openstack.org/475974 | 01:23 |
| *** MatthewLi has joined #openstack-infra | 01:23 | |
| *** jascott1 has joined #openstack-infra | 01:23 | |
| *** ijw has quit IRC | 01:25 | |
| *** JudeC has joined #openstack-infra | 01:28 | |
| *** JudeC has quit IRC | 01:36 | |
| *** Julien-zte has quit IRC | 01:36 | |
| *** gongysh has joined #openstack-infra | 01:38 | |
| *** markvoelker has quit IRC | 01:40 | |
| *** markvoelker has joined #openstack-infra | 01:41 | |
| openstackgerrit | Yolande Amate proposed openstack-infra/jenkins-job-builder master: Update batch_tasks to make use of convert_mapping_to_xml() https://review.openstack.org/475976 | 01:43 |
| *** markvoelker has quit IRC | 01:45 | |
| *** mmedvede has left #openstack-infra | 01:49 | |
| *** mmedvede has joined #openstack-infra | 01:50 | |
| *** yamahata_ has quit IRC | 01:56 | |
| *** gyee has quit IRC | 02:00 | |
| *** onovy has quit IRC | 02:02 | |
| *** esberglu has joined #openstack-infra | 02:02 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Use local mirror for ubuntu-minimal jobs https://review.openstack.org/475907 | 02:02 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Start at using CI mirrors for fedora/centos https://review.openstack.org/475607 | 02:02 |
| *** adisky__ has quit IRC | 02:03 | |
| *** onovy has joined #openstack-infra | 02:05 | |
| *** esberglu has quit IRC | 02:07 | |
| *** dhinesh has quit IRC | 02:07 | |
| openstackgerrit | Yolande Amate proposed openstack-infra/jenkins-job-builder master: Update heavyjob to make use of convert_mapping_to_xml() https://review.openstack.org/475981 | 02:08 |
| openstackgerrit | Akihiro Motoki proposed openstack-infra/project-config master: Modifies job for i18n repo to build translated doc https://review.openstack.org/471242 | 02:12 |
| *** ramishra has quit IRC | 02:15 | |
| *** aeng has quit IRC | 02:17 | |
| *** sflanigan has quit IRC | 02:17 | |
| *** mriedem has quit IRC | 02:24 | |
| *** yland has quit IRC | 02:25 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Move apt-sources to ubuntu-minimal / move debian to skip list https://review.openstack.org/475959 | 02:28 |
| *** sflanigan has joined #openstack-infra | 02:29 | |
| *** aeng has joined #openstack-infra | 02:30 | |
| *** bobh has quit IRC | 02:38 | |
| *** bobh has joined #openstack-infra | 02:38 | |
| *** baoli has joined #openstack-infra | 02:38 | |
| *** makowals has quit IRC | 02:40 | |
| *** mancdaz has quit IRC | 02:40 | |
| *** mancdaz has joined #openstack-infra | 02:43 | |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder master: Consolidate pipeline and trigger-parameterized-builds https://review.openstack.org/314108 | 02:50 |
| *** zz_dimtruck is now known as dimtruck | 02:54 | |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder master: Update ircbot plugin https://review.openstack.org/475116 | 02:55 |
| *** baoli has quit IRC | 02:56 | |
| *** yamahata_ has joined #openstack-infra | 03:01 | |
| *** MasterOfBugs has joined #openstack-infra | 03:02 | |
| *** hongbin has joined #openstack-infra | 03:05 | |
| *** dayou has quit IRC | 03:08 | |
| *** ykarel has joined #openstack-infra | 03:09 | |
| *** bobh has quit IRC | 03:11 | |
| *** dimtruck is now known as zz_dimtruck | 03:14 | |
| *** thorst has joined #openstack-infra | 03:15 | |
| *** mikal_ is now known as mikal | 03:15 | |
| *** baoli has joined #openstack-infra | 03:16 | |
| *** makowals has joined #openstack-infra | 03:18 | |
| *** tuanluong has joined #openstack-infra | 03:18 | |
| tristanC | dmsimard: ianw: indeed DIB_NO_TMPFS: '1' did the trick | 03:19 |
| *** baoli has quit IRC | 03:21 | |
| ianw | tristanC: so "--no-tmpfs" isn't working? | 03:21 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Fix image normalization when image has properties property https://review.openstack.org/475974 | 03:24 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Fix delete_ips on delete_server and add tests https://review.openstack.org/475967 | 03:24 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Fix config_drive, scheduler_hints and key_name in create_server https://review.openstack.org/475950 | 03:24 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Remove a direct mocking of _image_client https://review.openstack.org/475985 | 03:24 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Fix mismatch between port and port-id for REST call https://review.openstack.org/475986 | 03:24 |
| *** gomarivera has quit IRC | 03:25 | |
| tristanC | ianw: iiuc, no-tmpfs has no effect on the mk_build_dir/tmpfs_check common-function | 03:28 |
| *** markvoelker has joined #openstack-infra | 03:29 | |
| ianw | tristanC: --no-tmpfs just sets DIB_NO_TMPFS? --no-tmpfs) shift; export DIB_NO_TMPFS=1 ? and in nodepool (at least infra master) it's in the disk-image-create command line. just really weird? | 03:29 |
| tristanC | at least with the diskimage-builder-1.26 packaged in rdo | 03:30 |
| ianw | oh ... hmm, that is quite old. i don't remember a bug specifically related to that, but i may have forgotten | 03:31 |
| *** aeng has quit IRC | 03:34 | |
| *** sflanigan has quit IRC | 03:35 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Use local mirror for ubuntu-minimal jobs https://review.openstack.org/475907 | 03:35 |
| openstackgerrit | Merged openstack/diskimage-builder master: Start at using CI mirrors for fedora/centos https://review.openstack.org/475607 | 03:37 |
| *** gardlt has quit IRC | 03:37 | |
| *** andreww has joined #openstack-infra | 03:38 | |
| *** gardlt has joined #openstack-infra | 03:38 | |
| *** xarses_ has quit IRC | 03:40 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Move ironic-agent test to fedora-minimal https://review.openstack.org/475749 | 03:41 |
| *** Julien-zte has joined #openstack-infra | 03:43 | |
| *** nicolasbock has joined #openstack-infra | 03:46 | |
| *** aeng has joined #openstack-infra | 03:50 | |
| *** esberglu has joined #openstack-infra | 03:51 | |
| *** sflanigan has joined #openstack-infra | 03:52 | |
| *** sflanigan has joined #openstack-infra | 03:52 | |
| *** thorst has quit IRC | 03:54 | |
| *** esberglu has quit IRC | 03:55 | |
| *** yamamoto has joined #openstack-infra | 03:57 | |
| *** gardlt has quit IRC | 03:57 | |
| *** frickler_ is now known as frickler | 04:00 | |
| tristanC | ianw: dmsimard: actually it's a bit more tricky... the diskimage image type is only set when the diskimage is referenced by a provider image, which isn't the case atm | 04:01 |
| *** psachin has joined #openstack-infra | 04:02 | |
| tristanC | ianw: thus the --no-tmpfs may very well worked, its just that in this case (when the diskimage is not yet part of a provider), the command line is DIB_ARGS="-x -t --no-tmpfs" :) | 04:03 |
| *** tinwood has quit IRC | 04:05 | |
| *** tinwood has joined #openstack-infra | 04:06 | |
| *** links has joined #openstack-infra | 04:07 | |
| *** flepied has quit IRC | 04:09 | |
| *** dave-mccowan has quit IRC | 04:09 | |
| *** salv-orlando has joined #openstack-infra | 04:12 | |
| *** salv-orlando has quit IRC | 04:13 | |
| *** makowals has quit IRC | 04:14 | |
| *** Liuqing has joined #openstack-infra | 04:15 | |
| *** jamesmcarthur has quit IRC | 04:20 | |
| *** salv-orlando has joined #openstack-infra | 04:22 | |
| *** makowals has joined #openstack-infra | 04:25 | |
| *** sbezverk has quit IRC | 04:26 | |
| *** salv-orlando has quit IRC | 04:27 | |
| *** armax has joined #openstack-infra | 04:28 | |
| *** gomarivera has joined #openstack-infra | 04:29 | |
| *** gomarivera has quit IRC | 04:29 | |
| *** gomarivera has joined #openstack-infra | 04:29 | |
| *** hongbin has quit IRC | 04:30 | |
| *** MasterOfBugs has quit IRC | 04:32 | |
| *** cody-somerville has quit IRC | 04:37 | |
| *** cody-somerville has joined #openstack-infra | 04:37 | |
| *** MasterOfBugs has joined #openstack-infra | 04:38 | |
| *** adisky__ has joined #openstack-infra | 04:40 | |
| *** MasterOfBugs has quit IRC | 04:42 | |
| *** armax has quit IRC | 04:42 | |
| *** MasterOfBugs has joined #openstack-infra | 04:42 | |
| *** gouthamr has quit IRC | 04:43 | |
| pabelanger | tristanC: ianw: dmsimard: -t is missing its arguement. it should be -t qcow2 for example. It is possible you don't have clouds.yaml setup properly or missing image-type is missing in nodepool.yaml | 04:45 |
| tristanC | pabelanger: according to nodepool loadConfig method, image-type may be missing when the diskimage is not associated with a provider | 04:46 |
| *** MasterOfBugs has quit IRC | 04:47 | |
| pabelanger | right, we should default to clouds.yaml I think | 04:47 |
| pabelanger | I think some of that logic changed in feature/zuulv3 | 04:47 |
| *** bhavik1 has joined #openstack-infra | 04:48 | |
| pabelanger | for now, you could add image-type: qcow2 to your provider and try image rebuild | 04:48 |
| tristanC | pabelanger: I proposed https://review.rdoproject.org/r/7156 | 04:48 |
| ianw | ahh, nice catch | 04:49 |
| tristanC | pabelanger: though the build failed right before with "mv: cannot move '/tmp/unbound-logging.conf' to '/etc/unbound/unbound.conf.d/': Not a directory" | 04:49 |
| pabelanger | tristanC: DIB_CHECKSUM is nice to have, makes shade faster on image uploads | 04:49 |
| tristanC | pabelanger: DIB_CHECKSUM is enabled by default in sf | 04:50 |
| pabelanger | tristanC: I know clarkb did some changes to unbound recently, I haven't looked because of traveling | 04:50 |
| pabelanger | tristanC: cool | 04:50 |
| clarkb | yes I added logging | 04:51 |
| *** kzaitsev_mb has joined #openstack-infra | 04:51 | |
| clarkb | works on our images | 04:51 |
| tristanC | fwiw, full build log is https://review.rdoproject.org/nodepool-log/upstream-centos-7.log | 04:51 |
| clarkb | and nodepool-base installs the unbound paclage and itd where I add the config so should be all fine | 04:51 |
| pabelanger | is unbound.conf.d a centos thing too? | 04:53 |
| pabelanger | wow | 04:54 |
| pabelanger | nb03.o.o is out of HDD space | 04:54 |
| *** jamesmcarthur has joined #openstack-infra | 04:54 | |
| pabelanger | I guess we need more then 500GB | 04:54 |
| pabelanger | and nb04 is also full | 04:54 |
| pabelanger | image builds are down | 04:54 |
| pabelanger | going to stop nb03 and nb04 | 04:55 |
| pabelanger | I'll clean up space | 04:55 |
| clarkb | pabelanger: I wonder if that is allout of the suse build loop | 04:56 |
| clarkb | dib may not be cleaning up gracefully | 04:56 |
| pabelanger | clarkb: yes, that is possible | 04:57 |
| pabelanger | I'll pause opensuse for now | 04:57 |
| clarkb | tristanC: fwiw the unbound logging turned out to be pretty useless | 04:57 |
| clarkb | tristanC: at this point I think it would be fine to revert if we want | 04:58 |
| *** ramishra has joined #openstack-infra | 04:58 | |
| clarkb | (it doesnt log anything) | 04:58 |
| *** jamesmcarthur has quit IRC | 04:59 | |
| ianw | pabelanger: nahh, it's the looping, and dib is leaking | 04:59 |
| ianw | i cleaned it up yesterday too | 04:59 |
| MasterofJOKers | hi, what's the current (easy) way to test system-config? Previously used CONFIG_REF Parameter to project-config/tools/build-image.sh was removed, as I've seen. | 04:59 |
| ianw | i've just been fire-fighting the dib gate and haven't got back to it yet | 04:59 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Pause opensuse image builds https://review.openstack.org/476004 | 05:00 |
| pabelanger | ianw: clarkb: ^ k, lets pause | 05:00 |
| pabelanger | dib_tmp is leaking also | 05:00 |
| pabelanger | Purging that directory now | 05:00 |
| ianw | MasterofJOKers : shouldn't be required, now, since puppet left | 05:00 |
| ianw | pabelanger: yeah, in the failure case | 05:00 |
| pabelanger | ya, we have a ton of left over mount points | 05:01 |
| pabelanger | going to reboot nb04 first to clear them out | 05:01 |
| ianw | yep, i just rebooted | 05:01 |
| *** salv-orlando has joined #openstack-infra | 05:02 | |
| *** flepied has joined #openstack-infra | 05:02 | |
| MasterofJOKers | ok, then debian stable (stretch) images shouldn't be far away any more | 05:03 |
| *** makowals has quit IRC | 05:03 | |
| MasterofJOKers | http://lists.alioth.debian.org/pipermail/openstack-devel/2017-March/020999.html says, it needs that CONFIG_REF stuff figured out | 05:03 |
| ianw | MasterofJOKers: probably closer ... the puppet has been replaced with scripts in to nodepool/elements -- they may need some porting for debian. only way is to try | 05:04 |
| MasterofJOKers | ianw: I'm able to build a stretch image with a single-line patch to the 89-unbound script | 05:05 |
| MasterofJOKers | it doesn't find a rc.local, which failed initially | 05:06 |
| *** pahuang has quit IRC | 05:06 | |
| ianw | MasterofJOKers: great, if it boots, feel free to propose a change to nodepool.yaml and we can build and bring a node up | 05:06 |
| *** links has quit IRC | 05:07 | |
| MasterofJOKers | I'll give it a try later, thanks | 05:07 |
| pabelanger | ianw: if you want to +3 476004 that would help. I am in the process of deleting dib_tmp now | 05:08 |
| *** adreznec has quit IRC | 05:08 | |
| *** bnemec has quit IRC | 05:08 | |
| *** makowals has joined #openstack-infra | 05:09 | |
| ianw | pabelanger: sorry, this whole mess is because i thought i'd get the fix in quickly, but it's been a nightmare | 05:09 |
| *** dhajare has joined #openstack-infra | 05:09 | |
| ianw | the stretch release started it, as we have debian in the dib gate, and then something up with ubuntu cloud images, and these dns failures | 05:09 |
| pabelanger | Ya, I've seen the dns issues on the last few days | 05:10 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Move ironic-agent test to fedora-minimal https://review.openstack.org/475749 | 05:11 |
| *** pahuang has joined #openstack-infra | 05:12 | |
| ianw | 475051 just failed *again* on fedora mirror issues. ^ that should get rid of downloading that from the gate, fingers crossed | 05:12 |
| ianw | as soon as i can get it merged, i 'll release | 05:13 |
| pabelanger | cool | 05:13 |
| pabelanger | okay, nb03.o.o back to 30% HDD usage | 05:14 |
| pabelanger | waiting for 476004 to land, then will start it back up | 05:14 |
| *** xarses has joined #openstack-infra | 05:14 | |
| *** cuongnv has quit IRC | 05:14 | |
| openstackgerrit | Merged openstack-infra/project-config master: Pause opensuse image builds https://review.openstack.org/476004 | 05:16 |
| *** sflanigan has quit IRC | 05:17 | |
| *** cuongnv has joined #openstack-infra | 05:19 | |
| *** gongysh has quit IRC | 05:21 | |
| *** pahuang has quit IRC | 05:22 | |
| *** links has joined #openstack-infra | 05:23 | |
| clarkb | I cant chwck well on phome but we may have problem with centos too just hard to tell with other issues | 05:23 |
| clarkb | I'm fine reveting the unbound config stuff now. Not sure if d-g save_file will break though (I dont think so) | 05:24 |
| pabelanger | okay, nb03 / nb04 back online | 05:24 |
| *** armax has joined #openstack-infra | 05:24 | |
| pabelanger | building fedora-25 and debian-jessin now | 05:24 |
| pabelanger | I'll check on the status of centos-7 shortly | 05:24 |
| *** kzaitsev_mb has quit IRC | 05:25 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config master: diskimage-builder : run gate mirror repo setup https://review.openstack.org/476006 | 05:26 |
| *** aeng has quit IRC | 05:26 | |
| mordred | morning pabelanger ! | 05:27 |
| *** sree has joined #openstack-infra | 05:28 | |
| *** sflanigan has joined #openstack-infra | 05:29 | |
| *** sflanigan has joined #openstack-infra | 05:29 | |
| *** jbadiapa_ has joined #openstack-infra | 05:29 | |
| ianw | 475051 failed again! | 05:30 |
| *** dhajare has quit IRC | 05:31 | |
| ianw | * Could not resolve host: download.fedoraproject.org | 05:31 |
| ianw | dns failure this time :( | 05:31 |
| *** xarses_ has joined #openstack-infra | 05:34 | |
| *** pahuang has joined #openstack-infra | 05:35 | |
| *** andreww has quit IRC | 05:37 | |
| *** esberglu has joined #openstack-infra | 05:39 | |
| *** salv-orlando has quit IRC | 05:39 | |
| *** andreww has joined #openstack-infra | 05:39 | |
| *** andreww has quit IRC | 05:39 | |
| *** salv-orlando has joined #openstack-infra | 05:39 | |
| *** aeng has joined #openstack-infra | 05:39 | |
| *** andreww has joined #openstack-infra | 05:40 | |
| *** xarses has quit IRC | 05:40 | |
| *** xarses_ has quit IRC | 05:41 | |
| *** prateek has joined #openstack-infra | 05:42 | |
| *** xarses has joined #openstack-infra | 05:42 | |
| *** eranrom has joined #openstack-infra | 05:43 | |
| *** e0ne has joined #openstack-infra | 05:43 | |
| *** esberglu has quit IRC | 05:44 | |
| *** csomerville has joined #openstack-infra | 05:44 | |
| *** salv-orlando has quit IRC | 05:44 | |
| *** Julien-zte has quit IRC | 05:44 | |
| *** bnemec has joined #openstack-infra | 05:46 | |
| *** e0ne has quit IRC | 05:47 | |
| *** cody-somerville has quit IRC | 05:47 | |
| *** dhajare has joined #openstack-infra | 05:48 | |
| *** Julien-zte has joined #openstack-infra | 05:48 | |
| *** Julien-zte has quit IRC | 05:50 | |
| *** eranrom has quit IRC | 05:51 | |
| *** Julien-zte has joined #openstack-infra | 05:52 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: On suse the python2 dev package is python-devel https://review.openstack.org/475051 | 05:52 |
| *** xarses has quit IRC | 05:52 | |
| *** andreww has quit IRC | 05:53 | |
| *** Julien-zte has quit IRC | 05:54 | |
| *** xarses has joined #openstack-infra | 05:54 | |
| *** xarses_ has joined #openstack-infra | 05:55 | |
| *** armax has quit IRC | 05:55 | |
| *** andreas_s has joined #openstack-infra | 05:58 | |
| openstackgerrit | Merged openstack-infra/shade master: Fix config_drive, scheduler_hints and key_name in create_server https://review.openstack.org/475950 | 05:59 |
| *** adreznec has joined #openstack-infra | 06:00 | |
| *** ltomasbo|away is now known as ltomasbo | 06:03 | |
| openstackgerrit | Merged openstack-infra/project-config master: Move containers tripleo job to new format https://review.openstack.org/473159 | 06:05 |
| *** adriant has joined #openstack-infra | 06:07 | |
| adriant | Hello! I've had a project merged to openstack infra and I need to me added to the right groups | 06:08 |
| adriant | I need to be added to 'adjutant-release' and 'adjutant-core' as this has been merged: https://review.openstack.org/#/c/474867 | 06:08 |
| adriant | Thanks in advance :) | 06:08 |
| *** JudeC has joined #openstack-infra | 06:11 | |
| *** Julien-zte has joined #openstack-infra | 06:22 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Move ironic-agent test to fedora-minimal https://review.openstack.org/475749 | 06:22 |
| *** markus_z has joined #openstack-infra | 06:23 | |
| *** Liuqing has quit IRC | 06:25 | |
| *** aeng has quit IRC | 06:26 | |
| *** jtomasek_ has joined #openstack-infra | 06:28 | |
| *** bhavik1 has quit IRC | 06:28 | |
| *** pcaruana has joined #openstack-infra | 06:31 | |
| ianw | adriant: should be added | 06:32 |
| *** isaacb has joined #openstack-infra | 06:32 | |
| adriant | ianw: just relogged and I don't appear to be able to +2 my patch. :( | 06:34 |
| *** pgadiya has joined #openstack-infra | 06:35 | |
| adriant | ianw: https://review.openstack.org/#/c/475949/ | 06:35 |
| AJaeger | adriant: https://review.openstack.org/#/admin/groups/1791,members | 06:35 |
| AJaeger | https://review.openstack.org/#/admin/groups/1790,members | 06:36 |
| AJaeger | So, you should be set up | 06:36 |
| adriant | oh, cool | 06:36 |
| adriant | Hmm, is there a restriction since it's my own patch? | 06:36 |
| AJaeger | adriant: btw. I'm part of infra but I not everybody has all permissions ;) | 06:37 |
| AJaeger | adriant: you should be able to +2 your own patch | 06:37 |
| AJaeger | https://review.openstack.org/#/settings/group-memberships shows your permissions. | 06:37 |
| AJaeger | adriant: can't help further, sorry | 06:38 |
| adriant | yeah, it's odd. | 06:38 |
| ianw | adriant: make sure you're on the latest changeset? | 06:39 |
| clarkb | and hard refresh | 06:39 |
| clarkb | the web client caches your vote options | 06:40 |
| ianw | you are kidding me! | 06:40 |
| ianw | http://logs.openstack.org/51/475051/2/gate/gate-diskimage-builder-python35/6ee3cc2/console.html#_2017-06-21_06_36_20_636079 | 06:41 |
| ianw | THESE PACKAGES DO NOT MATCH THE HASHES FROM THE REQUIREMENTS FILE. If you have updated the package versions, please update the hashes. Otherwise, examine the package contents carefully; someone may have tampered with them. | 06:41 |
| ianw | i have never seen that one before | 06:41 |
| adriant | clarkb: hard refresh did it... | 06:42 |
| adriant | clarkb: it's little things like that which makes a person slowly hate gerrit bit by bit :P | 06:43 |
| *** jtomasek_ has quit IRC | 06:45 | |
| *** winggundamth has joined #openstack-infra | 06:45 | |
| pabelanger | mordred: ohai! | 06:50 |
| *** ykarel_ has joined #openstack-infra | 06:51 | |
| mordred | pabelanger: you over here already? | 06:52 |
| pabelanger | mordred: yup, hanging out in room currently. plan to head down in 40mins or so | 06:52 |
| cmurphy | whoa you guys are in the wrong timezone | 06:52 |
| pabelanger | IKR | 06:53 |
| *** ykarel has quit IRC | 06:54 | |
| *** rcernin has joined #openstack-infra | 06:59 | |
| *** markvoelker has quit IRC | 07:00 | |
| *** markvoelker has joined #openstack-infra | 07:00 | |
| *** markvoelker has quit IRC | 07:01 | |
| *** markvoelker has joined #openstack-infra | 07:01 | |
| *** markvoelker has quit IRC | 07:01 | |
| *** masuberu has quit IRC | 07:02 | |
| mordred | cmurphy: indeed - we should be in +1 not +0 ;) | 07:03 |
| *** armax has joined #openstack-infra | 07:04 | |
| cmurphy | the biergartens here in +1 agree | 07:06 |
| mtreinish | mordred: it could be weirder, I'm in +8 | 07:06 |
| *** markvoelker has joined #openstack-infra | 07:07 | |
| *** shardy has joined #openstack-infra | 07:08 | |
| *** florianf has joined #openstack-infra | 07:12 | |
| *** tesseract has joined #openstack-infra | 07:15 | |
| *** ralonsoh has joined #openstack-infra | 07:15 | |
| *** gomarivera has quit IRC | 07:18 | |
| *** aviau has quit IRC | 07:19 | |
| *** aviau has joined #openstack-infra | 07:19 | |
| *** jgrassle1 is now known as jgrassler | 07:19 | |
| therve | It looks like https://review.openstack.org/#/c/474036/ may have broken heat py3 gate | 07:20 |
| therve | clarkb, ianw, sdague any idea? | 07:20 |
| mtreinish | therve: was it running tempest? and if so how? | 07:21 |
| mtreinish | therve: do you have a link to the failure? | 07:21 |
| therve | mtreinish, http://logs.openstack.org/08/476008/1/check/gate-heat-dsvm-functional-convg-mysql-lbaasv2-py35-ubuntu-xenial/ac5ebb4/console.html#_2017-06-21_06_17_54_019167 | 07:22 |
| therve | The heat integration tests are tempest plugins | 07:22 |
| openstackgerrit | Daniel Mellado proposed openstack-infra/project-config master: Create docs gate for kuyr-tempest-plugin https://review.openstack.org/476028 | 07:23 |
| openstackgerrit | Merged openstack-infra/project-config master: Experimental py35 dsvm for ceph devstack plugin https://review.openstack.org/454890 | 07:23 |
| therve | It works fine in py2, but the tests are not loaded in py3 | 07:23 |
| *** jpich has joined #openstack-infra | 07:24 | |
| therve | I guess we should add INSTALL_TEMPEST in our config? | 07:24 |
| mtreinish | therve: or use the devstack-gate mechanisms for running tempest | 07:25 |
| mtreinish | looking at the post test hook tbh I'm not sure why it would work on python2 either after 474036 | 07:26 |
| *** hrubi has quit IRC | 07:26 | |
| mtreinish | because it's relying on something that's not installed anymore | 07:26 |
| mtreinish | so I'm not sure how it's being found at all :) | 07:26 |
| therve | Well tempest is still present | 07:26 |
| therve | We get the pypi version though | 07:26 |
| *** esberglu has joined #openstack-infra | 07:27 | |
| mtreinish | oh, it's in requirements somewhere and getting installed still? | 07:28 |
| mtreinish | therve: anyway what I would do is add: https://github.com/openstack/heat/blob/2a6cf96b845b15f1319a8f0135d713af59a96dd8/heat_integrationtests/prepare_test_env.sh to your devstack plugin in the test-config phase | 07:28 |
| *** ykarel__ has joined #openstack-infra | 07:28 | |
| mtreinish | then in the job defition set TEMPEST_PLUGINS= in the local.conf for the plugins you want to be installed | 07:29 |
| mtreinish | and also in the job definition set DEVSTACK_GATE_TEMPEST_REGEX to the regex you want to run | 07:29 |
| mtreinish | that way you're using the standard mechanisms for installing and running plugins in d-g | 07:30 |
| *** flepied has quit IRC | 07:30 | |
| *** mrunge_ has joined #openstack-infra | 07:30 | |
| *** mrunge has quit IRC | 07:30 | |
| therve | OK! | 07:31 |
| *** ykarel_ has quit IRC | 07:31 | |
| *** links has quit IRC | 07:31 | |
| *** esberglu has quit IRC | 07:31 | |
| *** alexchadin has joined #openstack-infra | 07:32 | |
| *** hrubi has joined #openstack-infra | 07:33 | |
| *** ccamacho has joined #openstack-infra | 07:34 | |
| dnikishov | hello, can I get some reviews please @ https://review.openstack.org/#/c/475323/ ? | 07:35 |
| *** amoralej|off is now known as amoralej | 07:36 | |
| *** esberglu has joined #openstack-infra | 07:37 | |
| *** jpena|off is now known as jpena | 07:37 | |
| *** jascott1 has quit IRC | 07:41 | |
| *** esberglu has quit IRC | 07:42 | |
| *** jascott1 has joined #openstack-infra | 07:42 | |
| *** links has joined #openstack-infra | 07:44 | |
| *** jascott1 has quit IRC | 07:46 | |
| *** cshen has joined #openstack-infra | 07:48 | |
| *** mhickey has joined #openstack-infra | 07:52 | |
| *** salv-orlando has joined #openstack-infra | 07:54 | |
| *** zzzeek has quit IRC | 08:00 | |
| *** zzzeek has joined #openstack-infra | 08:02 | |
| *** hjensas has quit IRC | 08:02 | |
| *** rossella__ has joined #openstack-infra | 08:09 | |
| *** e0ne has joined #openstack-infra | 08:10 | |
| *** e0ne has quit IRC | 08:13 | |
| rnoriega | AJaeger, ping | 08:14 |
| openstackgerrit | Merged openstack-infra/shade master: Fix delete_ips on delete_server and add tests https://review.openstack.org/475967 | 08:16 |
| openstackgerrit | Merged openstack-infra/shade master: Fix image normalization when image has properties property https://review.openstack.org/475974 | 08:19 |
| openstackgerrit | Merged openstack-infra/shade master: Remove a direct mocking of _image_client https://review.openstack.org/475985 | 08:19 |
| *** eranrom has joined #openstack-infra | 08:19 | |
| pabelanger | ianw: it looks like fedora-25 dibs are broken | 08:22 |
| pabelanger | http://nb03.openstack.org/dib.fedora-25.log | 08:22 |
| pabelanger | oh, this might be unbound changes clarkb did | 08:23 |
| *** esberglu has joined #openstack-infra | 08:23 | |
| *** eranrom has quit IRC | 08:25 | |
| openstackgerrit | Merged openstack-infra/shade master: Fix mismatch between port and port-id for REST call https://review.openstack.org/475986 | 08:25 |
| *** e0ne has joined #openstack-infra | 08:27 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Revert "Make unbound log errors" https://review.openstack.org/476040 | 08:27 |
| *** esberglu has quit IRC | 08:28 | |
| pabelanger | clarkb: ^ going to revert that for now, until we fix on fedora / centos | 08:28 |
| pabelanger | actually, I can fix it now | 08:29 |
| *** hashar has joined #openstack-infra | 08:31 | |
| *** kzaitsev_mb has joined #openstack-infra | 08:33 | |
| *** e0ne has quit IRC | 08:33 | |
| ianw | pabelanger: it also looks like trusty isn't working, there's an uptick in building nodes and jobs are waiting | 08:34 |
| *** rossella_ has joined #openstack-infra | 08:36 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Use conf.d for unbound on DIBs https://review.openstack.org/476043 | 08:37 |
| pabelanger | ianw: clarkb: ^ unbound fixes | 08:37 |
| *** rossella__ has quit IRC | 08:37 | |
| pabelanger | I am not sure about opensuse | 08:37 |
| jamielennox | ianw pabelanger: question i should know how to find myself - but are you running centos + ubuntu builder nodes? | 08:37 |
| jamielennox | i just found that i can't build ubuntu images on a debian box because debootstrap doesn't contain the xenial etc definitions | 08:38 |
| pabelanger | jamielennox: we run ubuntu-xenial for nb03.o.o and nb04.o.o, is that what you are sking? | 08:38 |
| jamielennox | and was wondering if you just run everything on ubuntu and it just works out ok or have to have some different base oses as well | 08:39 |
| *** gongysh has joined #openstack-infra | 08:39 | |
| pabelanger | no, sure just work. Same for centos / fedora too | 08:39 |
| pabelanger | which version of debian? | 08:40 |
| mordred | jamielennox: yah - run all the builders on xenial | 08:40 |
| jamielennox | pabelanger: jessie i think | 08:40 |
| pabelanger | I seem to remember you need backports for deboostrap I think | 08:40 |
| pabelanger | to pickup new distro support | 08:40 |
| jamielennox | i was testing it out in a container so FROM python:3 which seems to be jessie underneath | 08:41 |
| *** JudeC has quit IRC | 08:41 | |
| jamielennox | FROM python won't be sufficient, but for speed it was useful | 08:41 |
| pabelanger | Ya, we don't have much test converage on debian jessie to start with, might want to pick up something from ubuntu for better results | 08:42 |
| mordred | jamielennox: ah - that's an interesting thing to know re: python:3 | 08:42 |
| mordred | jamielennox: it's possible/not-hard to "add" support for xenial to jessie's debootstrap | 08:43 |
| *** rbergero1 is now known as rbergeron | 08:43 | |
| mordred | jamielennox: each ubuntu distro is basically just a symlink | 08:43 |
| *** rbergeron has quit IRC | 08:43 | |
| *** rbergeron has joined #openstack-infra | 08:43 | |
| mordred | jamielennox: $ ls -ltra /usr/share/debootstrap/scripts/xenial | 08:43 |
| mordred | lrwxrwxrwx 1 root root 5 Apr 21 18:41 /usr/share/debootstrap/scripts/xenial -> gutsy | 08:43 |
| StevenK | Everything since gutsy has been a symlink to it | 08:44 |
| jamielennox | mordred: yea, i noticed the symlink, but i wasn't sure that debian would be interested in dealing with ubuntu's versions | 08:44 |
| mordred | jamielennox: I think they usually have them - but xenial probably came out after the jessie freeze | 08:44 |
| jamielennox | i was more wondering if it was something that dib should be providing | 08:44 |
| mordred | jamielennox: so asa workaround, you could almost certainly add a ln -s into your dockerfile | 08:44 |
| jamielennox | StevenK: hey! | 08:44 |
| StevenK | Oh god | 08:45 |
| * StevenK dives out the window | 08:45 | |
| jamielennox | StevenK: nothing specific required, just haven't seen you around | 08:45 |
| *** rcernin has quit IRC | 08:45 | |
| StevenK | jamielennox: Er, I mean, hai :-) | 08:45 |
| *** neiljerram has joined #openstack-infra | 08:45 | |
| StevenK | jamielennox: No longer working directly upstream does that :-( | 08:45 |
| jamielennox | so everything in debian's debootstrap is a symlink to either gutsy or sid | 08:46 |
| jamielennox | oh, wait | 08:47 |
| jamielennox | scrap that | 08:47 |
| jamielennox | they are just already maintaining a number of ubuntu definitions, they just haven't got up to date with xenial | 08:47 |
| *** lucas-afk is now known as lucasagomes | 08:47 | |
| jamielennox | anyway - doesn't really matter, just interesting and i'll test it out on ubuntu tomorrow | 08:48 |
| *** tosky has joined #openstack-infra | 08:48 | |
| *** hrubi has quit IRC | 08:48 | |
| mordred | jamielennox: ++ | 08:48 |
| pabelanger | mordred: jeblair: if you are up for a review: 476043. That should fix fedora / centos image builds. | 08:49 |
| *** rcernin has joined #openstack-infra | 08:51 | |
| mordred | pabelanger: lgtm | 08:51 |
| *** hrubi has joined #openstack-infra | 08:52 | |
| *** rossella_ has quit IRC | 08:54 | |
| *** electrofelix has joined #openstack-infra | 08:55 | |
| *** Julien-zte has quit IRC | 08:56 | |
| *** dtantsur|afk is now known as dtantsur | 08:56 | |
| *** yamahata_ has quit IRC | 08:57 | |
| *** priteau has joined #openstack-infra | 08:58 | |
| *** Julien-zte has joined #openstack-infra | 08:58 | |
| *** hjensas has joined #openstack-infra | 08:58 | |
| *** rossella_ has joined #openstack-infra | 08:58 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Use conf.d for unbound on DIBs https://review.openstack.org/476043 | 08:58 |
| pabelanger | okay, updated with opensuse, since it also used /etc/unbound/conf.d | 08:59 |
| *** mhickey has quit IRC | 08:59 | |
| cuongnv | hi team, i'm working on a privsep related feature in neutron, some UTs failed with error about sudo, test report here http://logs.openstack.org/12/470912/5/check/gate-neutron-python27-ubuntu-xenial/f964314/testr_results.html.gz | 08:59 |
| cuongnv | most of the tests failed wiht this message "Stderr: sudo: no tty present and no askpass program specified" | 09:00 |
| *** aspiers has joined #openstack-infra | 09:00 | |
| cuongnv | I can run this successfully on my local env | 09:00 |
| cuongnv | i'm wondering if there is something related sudo config in jenkins os image | 09:00 |
| cuongnv | in my local env, sudo is configured to not ask password | 09:01 |
| pabelanger | cuongnv: we disable sudo by default on tox jobs, you'll need update JJB to enable it | 09:01 |
| *** hrubi has quit IRC | 09:02 | |
| *** annp has joined #openstack-infra | 09:02 | |
| cuongnv | pabelanger, what JJB is btw? | 09:02 |
| *** salv-orlando has quit IRC | 09:02 | |
| pabelanger | cuongnv: sure, jenkins job builder, that is in openstack-infra/project-config | 09:03 |
| *** salv-orlando has joined #openstack-infra | 09:03 | |
| *** salv-orlando has quit IRC | 09:03 | |
| cuongnv | pabelanger, ok, let me see and get back later, thanks | 09:03 |
| *** salv-orlando has joined #openstack-infra | 09:06 | |
| *** salv-orlando has quit IRC | 09:08 | |
| *** efoley has joined #openstack-infra | 09:08 | |
| cuongnv | pabelanger, can i just simply remove this line https://github.com/openstack-infra/project-config/blob/master/jenkins/jobs/neutron.yaml#L17? | 09:09 |
| *** salv-orlando has joined #openstack-infra | 09:09 | |
| pabelanger | cuongnv: yes, exactly right. the revoke-sudo builder removes sudo permissions | 09:10 |
| cuongnv | okay, thank you pabelanger | 09:12 |
| *** rossella_ has quit IRC | 09:16 | |
| *** stakeda has quit IRC | 09:16 | |
| openstackgerrit | zhongjun proposed openstack-infra/project-config master: Add ipv6 address in manila-tempest-minimal-dsvm-lvm-centos-7 job https://review.openstack.org/476057 | 09:18 |
| *** e0ne has joined #openstack-infra | 09:18 | |
| openstackgerrit | zhongjun proposed openstack-infra/project-config master: Add ipv6 address in manila-tempest-minimal-dsvm-lvm-centos-7 job https://review.openstack.org/476057 | 09:19 |
| strigazi | Many jobs are queued in the ci, is something wrong? | 09:19 |
| pabelanger | looking | 09:20 |
| pabelanger | Oh wow | 09:20 |
| pabelanger | ya | 09:20 |
| *** gomarivera has joined #openstack-infra | 09:20 | |
| pabelanger | ya, configure_mirror.sh is broken | 09:21 |
| pabelanger | fixing | 09:21 |
| strigazi | thanks | 09:21 |
| *** yamamoto has quit IRC | 09:22 | |
| openstackgerrit | Cuong Nguyen proposed openstack-infra/project-config master: Enable sudo privilege for neutron jobs to support privsep https://review.openstack.org/476060 | 09:23 |
| cuongnv | pabelanger can you help to take a look ^ | 09:23 |
| *** flepied has joined #openstack-infra | 09:23 | |
| *** gomarivera has quit IRC | 09:25 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Use sudo for unbound forwarding.conf in configure_mirror.sh https://review.openstack.org/476063 | 09:26 |
| pabelanger | mordred: jeblair: ^ we are failing to launch images in nodepool because of ^. do you mind reviewing | 09:26 |
| pabelanger | will revert to images from yesterday, to get some relief for the gate | 09:27 |
| *** rossella_ has joined #openstack-infra | 09:29 | |
| pabelanger | OpenStackCloudException: Tried to delete floating ip 89.42.141.57 associated with server 1640c3f3-c35c-461d-9a22-b6942b64f1f2 but there was an error finding it. Something is exceptionally broken. | 09:29 |
| mordred | pabelanger: I think you has typo | 09:30 |
| pabelanger | mordred: ^should we be concerned? that is on nodepool.o.o | 09:30 |
| pabelanger | mordred: ack | 09:30 |
| mordred | pabelanger: don't be TOO concerned - fix is in flight | 09:30 |
| mordred | pabelanger: like, be a little concerned, but not very | 09:31 |
| pabelanger | ack | 09:31 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Use sudo for unbound forwarding.conf in configure_mirror.sh https://review.openstack.org/476063 | 09:32 |
| pabelanger | ianw: mordred: ^thanks | 09:32 |
| mordred | pabelanger: lgtm | 09:33 |
| pabelanger | strigazi: thanks, ubuntu-xenial images coming online now. Working on proper fix | 09:33 |
| strigazi | pabelanger thanks | 09:33 |
| *** kzaitsev_mb has quit IRC | 09:43 | |
| *** cuongnv_ has joined #openstack-infra | 09:45 | |
| *** yamamoto has joined #openstack-infra | 09:46 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci master: Disable telemetry from fakeha-caserver job https://review.openstack.org/475736 | 09:47 |
| *** tuanluong_ has joined #openstack-infra | 09:47 | |
| *** cuongnv__ has joined #openstack-infra | 09:47 | |
| *** markus_z has quit IRC | 09:48 | |
| *** krenczewski has joined #openstack-infra | 09:48 | |
| *** cuongnv has quit IRC | 09:48 | |
| *** cuongnv__ is now known as cuongnv | 09:49 | |
| *** cuongnv_ has quit IRC | 09:50 | |
| *** tuanluong has quit IRC | 09:50 | |
| pabelanger | also rolling back ubuntu-trusty image, need that to land project-config changes | 09:50 |
| *** e0ne has quit IRC | 09:50 | |
| *** kzaitsev_mb has joined #openstack-infra | 09:50 | |
| *** hrubi has joined #openstack-infra | 09:50 | |
| rm_work | so, soon octavia can merge changes again? :P we've been stuck behind the xenial image breakage all day :( | 09:51 |
| *** e0ne has joined #openstack-infra | 09:52 | |
| pabelanger | Ya, images are rolled back now | 09:53 |
| pabelanger | http://grafana.openstack.org/dashboard/db/nodepool | 09:53 |
| pabelanger | for the launch excitment | 09:53 |
| rm_work | sweet | 09:53 |
| pabelanger | large launch happening now | 09:54 |
| pabelanger | 144 nodes | 09:54 |
| pabelanger | they sould all come online | 09:54 |
| mordred | pabelanger: large lunch? | 09:54 |
| pabelanger | food is good | 09:55 |
| *** sambetts|afk is now known as sambetts | 09:55 | |
| *** kzaitsev_mb has quit IRC | 09:55 | |
| *** Julien-zte has quit IRC | 09:55 | |
| *** hrubi has quit IRC | 09:58 | |
| *** waynr has quit IRC | 10:00 | |
| *** rossella_ has quit IRC | 10:02 | |
| *** flepied has quit IRC | 10:02 | |
| *** alexchadin has quit IRC | 10:02 | |
| *** oneswig has joined #openstack-infra | 10:04 | |
| oneswig | Hi all, is there an issue with this review for IRC infra, it seems to be stuck: https://review.openstack.org/#/c/459884/ | 10:04 |
| *** hjensas has quit IRC | 10:07 | |
| *** esberglu has joined #openstack-infra | 10:08 | |
| *** yamamoto_ has joined #openstack-infra | 10:09 | |
| *** salv-orl_ has joined #openstack-infra | 10:12 | |
| *** yamamoto has quit IRC | 10:13 | |
| *** esberglu has quit IRC | 10:13 | |
| *** salv-orlando has quit IRC | 10:14 | |
| pabelanger | oneswig: we'll have to merge it off hours, but +2 | 10:15 |
| pabelanger | let me check if there are any meetings going on atm | 10:15 |
| oneswig | OK thanks pabelanger for taking a look! | 10:15 |
| *** liujiong has quit IRC | 10:16 | |
| pabelanger | ya, it will have to be off hours | 10:16 |
| *** gongysh has quit IRC | 10:16 | |
| *** annp has quit IRC | 10:17 | |
| *** flepied has joined #openstack-infra | 10:17 | |
| *** rossella_ has joined #openstack-infra | 10:17 | |
| openstackgerrit | Merged openstack-infra/project-config master: Barbican: update tests https://review.openstack.org/469648 | 10:18 |
| *** mat128 has joined #openstack-infra | 10:19 | |
| cuongnv | pabelanger, https://review.openstack.org/#/c/476060/ mind to take a look at this ps when you have time? | 10:19 |
| *** ykarel__ is now known as ykarel | 10:19 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Add localhost explicitly to the inventory https://review.openstack.org/476081 | 10:20 |
| *** LindaWang has quit IRC | 10:21 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add non-voting jobs for ARA to test the devel branch of Ansible https://review.openstack.org/475871 | 10:23 |
| openstackgerrit | Merged openstack-infra/project-config master: Only configure jenkins cache if cache exists https://review.openstack.org/475864 | 10:24 |
| *** cshastri has quit IRC | 10:24 | |
| *** oneswig has quit IRC | 10:26 | |
| *** yamamoto_ has quit IRC | 10:30 | |
| *** yamamoto has joined #openstack-infra | 10:32 | |
| *** yamamoto has quit IRC | 10:33 | |
| *** hrubi has joined #openstack-infra | 10:37 | |
| *** cuongnv has quit IRC | 10:39 | |
| *** markus_z has joined #openstack-infra | 10:39 | |
| *** ricky_ has joined #openstack-infra | 10:43 | |
| *** ricky_ is now known as Guest85911 | 10:43 | |
| *** yamamoto has joined #openstack-infra | 10:43 | |
| *** hrubi has quit IRC | 10:44 | |
| *** wfoster has quit IRC | 10:45 | |
| *** Guest85911 is now known as ricky_ansible_fe | 10:46 | |
| *** rook has quit IRC | 10:46 | |
| *** ricky_ansible_fe is now known as rcarrillocruz_fe | 10:46 | |
| *** rcarrillocruz_fe is now known as ricky_fest | 10:46 | |
| *** hrubi has joined #openstack-infra | 10:48 | |
| *** jkilpatr has quit IRC | 10:49 | |
| openstackgerrit | Merged openstack-infra/project-config master: Use sudo for unbound forwarding.conf in configure_mirror.sh https://review.openstack.org/476063 | 10:51 |
| *** wfoster has joined #openstack-infra | 10:52 | |
| *** rook has joined #openstack-infra | 10:52 | |
| *** e0ne has quit IRC | 10:52 | |
| *** rook is now known as Guest45012 | 10:52 | |
| openstackgerrit | Ghanshyam Mann proposed openstack-infra/project-config master: Stop running puppet-* jobs on tempest unittests/doc patches https://review.openstack.org/476089 | 10:53 |
| *** yamamoto has quit IRC | 10:54 | |
| *** ericyoung has joined #openstack-infra | 10:57 | |
| *** deafult has joined #openstack-infra | 11:02 | |
| *** flepied has quit IRC | 11:06 | |
| *** ccamacho is now known as ccamacho|lunch | 11:06 | |
| *** rhallisey has joined #openstack-infra | 11:06 | |
| *** andreas_s has quit IRC | 11:07 | |
| *** rossella__ has joined #openstack-infra | 11:07 | |
| *** sree has quit IRC | 11:07 | |
| *** jkilpatr has joined #openstack-infra | 11:08 | |
| openstackgerrit | Merged openstack-infra/project-config master: Use conf.d for unbound on DIBs https://review.openstack.org/476043 | 11:08 |
| *** winggundamth has quit IRC | 11:08 | |
| *** Guest45012 is now known as rook-ugh | 11:09 | |
| *** rossella_ has quit IRC | 11:10 | |
| *** ricky_fest has quit IRC | 11:12 | |
| *** gongysh has joined #openstack-infra | 11:17 | |
| *** jbadiapa_ has quit IRC | 11:19 | |
| *** LindaWang has joined #openstack-infra | 11:21 | |
| *** gomarivera has joined #openstack-infra | 11:22 | |
| *** yamamoto has joined #openstack-infra | 11:24 | |
| *** sbezverk has joined #openstack-infra | 11:24 | |
| *** gomarivera has quit IRC | 11:27 | |
| *** yamamoto has quit IRC | 11:28 | |
| *** kzaitsev_mb has joined #openstack-infra | 11:32 | |
| *** jbadiapa_ has joined #openstack-infra | 11:33 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add voting docs jobs to kuryr-tempest-plugin https://review.openstack.org/475904 | 11:36 |
| *** hrubi has quit IRC | 11:38 | |
| *** hrubi has joined #openstack-infra | 11:41 | |
| *** dave-mccowan has joined #openstack-infra | 11:42 | |
| *** jascott1 has joined #openstack-infra | 11:43 | |
| *** ldnunes has joined #openstack-infra | 11:45 | |
| *** deafult has quit IRC | 11:46 | |
| *** ekhugen has quit IRC | 11:47 | |
| *** jascott1 has quit IRC | 11:48 | |
| *** ekhugen has joined #openstack-infra | 11:48 | |
| *** isaacb_ has joined #openstack-infra | 11:52 | |
| *** isaacb has quit IRC | 11:53 | |
| *** yamamoto has joined #openstack-infra | 11:54 | |
| *** e0ne has joined #openstack-infra | 11:56 | |
| *** zz_dimtruck is now known as dimtruck | 12:00 | |
| *** ricky_fest has joined #openstack-infra | 12:01 | |
| *** rfolco has joined #openstack-infra | 12:04 | |
| *** jpena is now known as jpena|lunch | 12:07 | |
| *** dimtruck is now known as zz_dimtruck | 12:10 | |
| *** trown|outtypewww is now known as trown | 12:12 | |
| *** tmorin has joined #openstack-infra | 12:13 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [DNM] testing glance uwsgi revert & our jobs https://review.openstack.org/476109 | 12:14 |
| *** pgadiya has quit IRC | 12:14 | |
| dmellado | pabelanger: o/ | 12:15 |
| dmellado | I've seen that some openstack gates for kuryr-kubernetes are timing out | 12:16 |
| dmellado | i.e. https://review.openstack.org/#/c/474576/ | 12:16 |
| dmellado | do you know how I could debug this? | 12:16 |
| dmellado | yolanda: AJaeger ^^ | 12:16 |
| dmellado | thanks! | 12:17 |
| dmellado | later on I'm seeing that some of them are passing | 12:18 |
| dmellado | but not all | 12:18 |
| dmellado | i.e. https://review.openstack.org/#/c/476070/2 | 12:18 |
| dmellado | is there smthing wrong happening from the infra side? | 12:18 |
| *** ccamacho|lunch is now known as ccamacho | 12:20 | |
| pabelanger | we had some bad images eariler, but that should be fixed now | 12:22 |
| *** rlandy has joined #openstack-infra | 12:24 | |
| *** danpawlik has joined #openstack-infra | 12:24 | |
| *** isaacb_ has quit IRC | 12:25 | |
| fungi | dmellado: do the ones which succeed come close to the timeout for that job, or is it more like something is getting stuck on some job runs causing the job not to complete and eventually getting caught by the timeout? | 12:26 |
| *** bauzas has quit IRC | 12:26 | |
| dmellado | fungi: I was checking on https://review.openstack.org/476070 on status.openstack.org | 12:27 |
| *** isaacb_ has joined #openstack-infra | 12:27 | |
| dmellado | and dsvm and dragonflow took quite long | 12:27 |
| dmellado | but the tempest one is a brand new gate, so I'm not sure | 12:27 |
| * dmellado brb | 12:28 | |
| fungi | dmellado: sorry, i'm not yet at a computer where i can pull up the status interface reliably (hence the questions) | 12:28 |
| dmellado | fungi: I'm triggering a 'dummy' patch | 12:28 |
| dmellado | will track it and get back to you | 12:28 |
| dmellado | thanks! | 12:29 |
| dmellado | telnet://[2001:4800:1ae1:18:f816:3eff:febb:d905]:19885 | 12:29 |
| dmellado | this is the telnet for the 'maybe-stuck' gate | 12:29 |
| *** derekh has joined #openstack-infra | 12:29 | |
| *** alexchadin has joined #openstack-infra | 12:30 | |
| pabelanger | let me check | 12:30 |
| pabelanger | 2017-06-21 10:58:44.627 | ++ /opt/stack/new/kuryr-kubernetes/devstack/plugin.sh:wait_for:349 : echo -n 'Waiting for Kubernetes API Server to respond' | 12:32 |
| pabelanger | 2017-06-21 10:58:44.630 | Waiting for Kubernetes API Server to respond++ /opt/stack/new/kuryr-kubernetes/devstack/plugin.sh:wait_for:351 : curl -o /dev/null -sIf http://10.1.185.70:8080 | 12:32 |
| pabelanger | it is looping on that | 12:32 |
| *** gordc has joined #openstack-infra | 12:32 | |
| pabelanger | job should timeout and logs get copied | 12:32 |
| *** jamesdenton has joined #openstack-infra | 12:34 | |
| *** tuanluong_ has quit IRC | 12:34 | |
| *** dprince has joined #openstack-infra | 12:35 | |
| *** derekh has quit IRC | 12:36 | |
| *** isaacb__ has joined #openstack-infra | 12:40 | |
| *** isaacb__ has quit IRC | 12:40 | |
| *** amoralej is now known as amoralej|lunch | 12:41 | |
| *** hrubi has quit IRC | 12:42 | |
| *** isaacb_ has quit IRC | 12:42 | |
| *** flepied has joined #openstack-infra | 12:43 | |
| *** derekh has joined #openstack-infra | 12:43 | |
| *** hrubi has joined #openstack-infra | 12:43 | |
| *** kzaitsev_mb has quit IRC | 12:43 | |
| *** gordc has quit IRC | 12:45 | |
| *** wolverineav has joined #openstack-infra | 12:46 | |
| *** derekh has quit IRC | 12:48 | |
| dmellado | hmmm | 12:48 |
| dmellado | then that could be an unrelated issue... | 12:49 |
| dmellado | to the gate, I mean | 12:49 |
| dmellado | apuimedo: ^^ | 12:49 |
| *** derekh has joined #openstack-infra | 12:49 | |
| *** kgiusti has joined #openstack-infra | 12:49 | |
| *** rook-ugh is now known as rook | 12:49 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci master: Fix periodic jobs release detection https://review.openstack.org/475805 | 12:49 |
| fungi | dmellado: well, if there's a race condition or similar nondeterminism which can sometimes cause it not to start, then that's a possible explanation for what you're seeing | 12:51 |
| *** eharney has joined #openstack-infra | 12:51 | |
| *** hrubi has quit IRC | 12:52 | |
| *** salv-orl_ has quit IRC | 12:54 | |
| *** mriedem has joined #openstack-infra | 12:56 | |
| *** salv-orlando has joined #openstack-infra | 12:56 | |
| *** salv-orlando has quit IRC | 12:57 | |
| dmellado | hmmm | 12:58 |
| *** salv-orlando has joined #openstack-infra | 12:58 | |
| dmellado | maybe something on that port already... | 12:58 |
| dmellado | I'm asumming swifr | 12:58 |
| *** dave-mccowan has quit IRC | 12:58 | |
| dmellado | swift-proxy shouldn't be on the gate by default... | 12:58 |
| *** jamesdenton has quit IRC | 13:00 | |
| dmellado | this is how I triggered the gate | 13:00 |
| dmellado | https://review.openstack.org/#/c/474993/10/jenkins/jobs/kuryr.yaml | 13:00 |
| *** jamesdenton has joined #openstack-infra | 13:01 | |
| dnikishov | hello, can I get some reviews please @ https://review.openstack.org/#/c/475323/ ? | 13:01 |
| *** sree has joined #openstack-infra | 13:01 | |
| *** jpena|lunch is now known as jpena | 13:01 | |
| openstackgerrit | Dmitry Tantsur proposed openstack-infra/tripleo-ci master: Add ironic-inspector to the undercloud-containers job https://review.openstack.org/476130 | 13:02 |
| *** xarses has quit IRC | 13:04 | |
| *** xarses_ has quit IRC | 13:05 | |
| *** sree has quit IRC | 13:06 | |
| *** LindaWang has quit IRC | 13:06 | |
| *** gouthamr has joined #openstack-infra | 13:06 | |
| *** salv-orlando has quit IRC | 13:08 | |
| dmellado | fungi: | 13:10 |
| *** askb has quit IRC | 13:10 | |
| *** salv-orlando has joined #openstack-infra | 13:12 | |
| *** baoli has joined #openstack-infra | 13:12 | |
| *** pradk has quit IRC | 13:12 | |
| apuimedo | dmellado: well, if it is stuck there it means the k8s container didn't start up well | 13:13 |
| *** lucasagomes is now known as lucas-brb | 13:13 | |
| *** LindaWang has joined #openstack-infra | 13:14 | |
| *** Goneri has joined #openstack-infra | 13:14 | |
| *** baoli has quit IRC | 13:16 | |
| *** thorst has joined #openstack-infra | 13:16 | |
| *** trown is now known as trown|brb | 13:16 | |
| *** jcoufal has joined #openstack-infra | 13:17 | |
| *** flepied has quit IRC | 13:17 | |
| *** tushar has quit IRC | 13:17 | |
| *** tushar has joined #openstack-infra | 13:18 | |
| *** esberglu has joined #openstack-infra | 13:19 | |
| *** trown|brb is now known as trown | 13:19 | |
| *** baoli has joined #openstack-infra | 13:20 | |
| dmellado | how would I be able to recreate that locally with dib and build-image.sh? | 13:20 |
| dmellado | pabelanger: ^^ yolanda forwarded me to you and I'll owe you a beer xD | 13:22 |
| *** thorst has quit IRC | 13:22 | |
| *** dizquierdo has joined #openstack-infra | 13:22 | |
| *** bauzas has joined #openstack-infra | 13:24 | |
| *** esberglu has quit IRC | 13:24 | |
| fungi | dmellado: i doubt it's anything to do with the image we're booting since we intentionally don't install or start anything we can reasonably leave out of it (expecting job definitions to take care of all that at runtime instead). if it's a devstack-gate based job, there should be a script included with the logs usable for rerunning the job locally with the same parameters | 13:24 |
| dmellado | fungi: I found that reproduce.sh | 13:25 |
| dmellado | will try running it locally and check | 13:25 |
| dmellado | thanks again! | 13:25 |
| fungi | yw | 13:25 |
| *** flepied has joined #openstack-infra | 13:29 | |
| *** flepied has quit IRC | 13:29 | |
| pabelanger | okay, all new DIBs should be online now | 13:29 |
| pabelanger | or coming online | 13:30 |
| *** flepied has joined #openstack-infra | 13:30 | |
| fungi | thanks for cleaning that up, pabelanger! | 13:30 |
| *** eric_young has joined #openstack-infra | 13:31 | |
| *** gomarivera has joined #openstack-infra | 13:31 | |
| *** rossella__ has quit IRC | 13:32 | |
| *** thorst has joined #openstack-infra | 13:32 | |
| *** ericyoung has quit IRC | 13:33 | |
| openstackgerrit | Thomas Herve proposed openstack-infra/project-config master: Make the py3 non voting for Heat https://review.openstack.org/476138 | 13:35 |
| *** esberglu has joined #openstack-infra | 13:35 | |
| therve | ^^ This will unblock heat gate until we can fix the tempest issue, if someone can have a look at it | 13:35 |
| *** lucas-brb is now known as lucasagomes | 13:36 | |
| *** xyang1 has joined #openstack-infra | 13:38 | |
| yamamoto | how to file a bug on storyboard? | 13:39 |
| *** gongysh has quit IRC | 13:39 | |
| *** camunoz has joined #openstack-infra | 13:40 | |
| fungi | yamamoto: create a new story (stories are an abstract structure suitable for reporting suspected defects, tracking work on new features, and so on) | 13:40 |
| pabelanger | fungi: np! | 13:41 |
| *** bobh has joined #openstack-infra | 13:41 | |
| *** amoralej|lunch is now known as amoralej | 13:41 | |
| fungi | yamamoto: you'll need to pick a git repository (project) relevant to the initial task for that story, but can add additional tasks for that repo or other repos within the story once it's created | 13:42 |
| *** dave-mccowan has joined #openstack-infra | 13:42 | |
| yamamoto | fungi: thank you. i was looking at README of devstack-gate. but it wasn't obvious to me "story" is the right one to choose. | 13:43 |
| *** jascott1 has joined #openstack-infra | 13:44 | |
| *** bobh has quit IRC | 13:45 | |
| fungi | yamamoto: i'm going to try and find time to make a second reskinned version of storyboard-webclient that's expressly for reporting new suspected defects using terminology more friendly/familiar to downstream users, i just haven't found time to start on that yet | 13:45 |
| yamamoto | https://storyboard.openstack.org/#!/story/2001076 | 13:46 |
| yamamoto | and how to reference it in commit messsages? | 13:46 |
| yamamoto | is there something like Closes-Bug: #xxx ? | 13:46 |
| *** gomarivera has quit IRC | 13:48 | |
| fungi | yamamoto: use "story" and "task" footers as described at https://docs.openstack.org/infra/manual/developers.html#working-on-bugs | 13:48 |
| *** rossella__ has joined #openstack-infra | 13:48 | |
| *** jascott1 has quit IRC | 13:48 | |
| *** dhajare has quit IRC | 13:49 | |
| fungi | yamamoto: in short, the "story" footer will leave a comment in the story with a link to the review, the "task" footer will change task state from todo to review to merged as the relevant change progresses | 13:49 |
| *** salv-orlando has quit IRC | 13:49 | |
| yamamoto | "Task: 2001076" is appropriate for a fix for the bug described in the above story? | 13:51 |
| fungi | yamamoto: that's the story number. you should see a separate (currently 4-digit) task number next to the initial task in that story | 13:52 |
| yamamoto | "Task: 4723" then? | 13:52 |
| *** xarses has joined #openstack-infra | 13:53 | |
| fungi | yamamoto: so you want "Story: 2001076" and "Task: 4723" on separate lines near the bottom of the commit message, yes | 13:53 |
| *** xarses has quit IRC | 13:53 | |
| *** xarses has joined #openstack-infra | 13:54 | |
| yamamoto | oh, i somehow thought Story: and Task: were exclusive. i can have both in a commit message? | 13:54 |
| *** zz_dimtruck is now known as dimtruck | 13:54 | |
| *** mhickey has joined #openstack-infra | 13:56 | |
| fungi | yamamoto: yes. to reiterate, the "story" footer will leave a comment in the story with a link to the review, the "task" footer will change task state from todo to review to merged as the relevant change progresses | 13:57 |
| fungi | this is because, unlike launchpad, storyboard supports creating multiple tasks for the same git repository, so you need to let it know which task to update within a story | 13:57 |
| openstackgerrit | YAMAMOTO Takashi proposed openstack-infra/devstack-gate master: reproduce.sh: Fix the dump of devstack-gate hooks https://review.openstack.org/476149 | 13:58 |
| *** rossella__ has quit IRC | 13:59 | |
| fungi | since tasks are globally unique we've talked about making the story footer optional, but i don't recall whether the sb api has added a story lookup by task number yet, and once it has we'll still need to update the its-storyboard plugin for gerrit to use that | 13:59 |
| *** adisky__ has quit IRC | 13:59 | |
| SotK | fungi: you can get the story id from the task object returned from the tasks endpoint, so we could theoretically make the its-storyboard plugin do that (it just means two API requests to get the story rather than one) | 14:00 |
| fungi | SotK: cool, and yep on the api call count... but given the relatively low frequency that this would get hit that's probably not worth the effort to optimize away by trying to add a reference-story-by-task mechanism for creating the comment | 14:02 |
| *** alexchadin has quit IRC | 14:05 | |
| *** mdrabe has joined #openstack-infra | 14:06 | |
| *** chlong has joined #openstack-infra | 14:07 | |
| yamamoto | do we plan to switch from lp to sb for a long term? | 14:08 |
| fungi | yamamoto: that's the current plan for openstack, yes: http://specs.openstack.org/openstack-infra/infra-specs/specs/task-tracker.html | 14:12 |
| *** dimtruck is now known as zz_dimtruck | 14:12 | |
| pabelanger | selinux on fedora is not happy about /etc/unbound/forwarding.conf. I wonder if we should just drop that file directly in to conf.d folder | 14:13 |
| MasterofJOKers | a question: https://git.openstack.org/cgit/openstack-infra/project-config/tree/tools/build-image.sh#n100 says, the list should match those in nodepool.yml, but I cannot find a "stackviz" entry in nodepool.yml (https://git.openstack.org/cgit/openstack-infra/project-config/tree/nodepool/nodepool.yaml) | 14:14 |
| MasterofJOKers | should that get removed? | 14:14 |
| pabelanger | MasterofJOKers: yes | 14:14 |
| pabelanger | devstack will set it up now | 14:14 |
| *** felipemonteiro has joined #openstack-infra | 14:14 | |
| MasterofJOKers | should I try to create a patch? would be my first one | 14:15 |
| *** felipemonteiro_ has joined #openstack-infra | 14:15 | |
| pabelanger | sure | 14:15 |
| *** pradk has joined #openstack-infra | 14:16 | |
| *** Julien-zte has joined #openstack-infra | 14:18 | |
| *** rbrndt has joined #openstack-infra | 14:19 | |
| *** felipemonteiro has quit IRC | 14:19 | |
| *** hongbin has joined #openstack-infra | 14:20 | |
| pabelanger | ./configure_mirror.sh: line 34: ip: command not found | 14:20 |
| pabelanger | clarkb: that is also on fedora^ | 14:21 |
| openstackgerrit | Merged openstack-dev/pbr master: docs: Don't specify pbr version in 'setup.py' https://review.openstack.org/475719 | 14:24 |
| johnsom | http://logs.openstack.org/20/475920/1/check/gate-octavia-python27-ubuntu-xenial/0ddd3fd/console.html#_2017-06-21_14_19_08_024620 | 14:25 |
| *** jamesmcarthur has joined #openstack-infra | 14:25 | |
| johnsom | Is there a known issue with zm01.openstack.org? I'm seeing stderr: 'fatal: Could not read from remote repository. | 14:26 |
| *** dizquierdo has quit IRC | 14:26 | |
| *** prateek has quit IRC | 14:26 | |
| *** zz_dimtruck is now known as dimtruck | 14:28 | |
| openstackgerrit | Johannes Kulik proposed openstack-infra/project-config master: Remove stackviz from tools/build-image.sh https://review.openstack.org/476160 | 14:30 |
| *** hrubi has joined #openstack-infra | 14:31 | |
| *** rfolco has quit IRC | 14:31 | |
| *** Julien-zte has quit IRC | 14:32 | |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci master: DONT MERGE: containers periodic test https://review.openstack.org/475747 | 14:32 |
| *** Julien-z_ has joined #openstack-infra | 14:32 | |
| *** dhajare has joined #openstack-infra | 14:33 | |
| *** Julien-z_ has quit IRC | 14:37 | |
| *** gouthamr has quit IRC | 14:37 | |
| *** rfolco has joined #openstack-infra | 14:37 | |
| *** gouthamr has joined #openstack-infra | 14:38 | |
| danpawlik | Hi, Im a begginer with Gerrit and according to Openstack gerrit document, I have a problem with "project.config: group "Project Bootstrappers" not in groups" | 14:38 |
| *** gordc has joined #openstack-infra | 14:38 | |
| danpawlik | I should add a new group via ssh | 14:38 |
| danpawlik | or it should be added in group file (All-Projects-ACLs) ? | 14:39 |
| *** ralonsoh has quit IRC | 14:40 | |
| *** markvoelker has quit IRC | 14:40 | |
| danpawlik | fungi: ^^ | 14:40 |
| *** xarses_ has joined #openstack-infra | 14:40 | |
| fungi | danpawlik: there's a bit of a chicken-and-egg situation setting up gerrit for the first time | 14:40 |
| danpawlik | sorry that im disturbing you :) | 14:40 |
| *** jamesmcarthur has quit IRC | 14:41 | |
| *** jamesmcarthur has joined #openstack-infra | 14:41 | |
| fungi | i'm trying to remember the details there since it's been a while | 14:41 |
| danpawlik | fungi: Im on this step: https://docs.openstack.org/infra/system-config/gerrit.html#access-controls | 14:42 |
| fungi | there is a "Gerrit Code Review" pseudo-account which is always guaranteed to exist that can perform some actions over the ssh api from localhost by authenticating using the gerrit ssh api host key as the ssh client identity | 14:43 |
| danpawlik | maybe I will try to add global:"some group" | 14:43 |
| fungi | though sounds like you already have a working admin account? | 14:43 |
| *** Julien-zte has joined #openstack-infra | 14:43 | |
| fungi | so yes i would just create the project bootstrappers group and the project-creator user and add the user to that group | 14:44 |
| *** hieulq_ has joined #openstack-infra | 14:44 | |
| fungi | as manual steps | 14:44 |
| danpawlik | fungi: problem disappear when I add into groups file: global:Project-Bootstrappers Project Bootstrappers | 14:45 |
| danpawlik | ack | 14:45 |
| danpawlik | global is not good, but its a test :0 | 14:45 |
| *** jamesmcarthur has quit IRC | 14:46 | |
| openstackgerrit | Andy McCrae proposed openstack-infra/project-config master: Fix zuul-clone command for keystone OSA upgrade job https://review.openstack.org/476168 | 14:50 |
| *** gomarivera has joined #openstack-infra | 14:52 | |
| *** marst has joined #openstack-infra | 14:55 | |
| *** LindaWang has quit IRC | 14:56 | |
| *** markvoelker has joined #openstack-infra | 14:57 | |
| *** dhajare has quit IRC | 15:00 | |
| *** florianf has quit IRC | 15:01 | |
| *** dizquierdo has joined #openstack-infra | 15:02 | |
| *** jaosorior has quit IRC | 15:05 | |
| *** thorst has quit IRC | 15:05 | |
| *** LindaWang has joined #openstack-infra | 15:05 | |
| *** rcernin has quit IRC | 15:07 | |
| *** ralonsoh has joined #openstack-infra | 15:07 | |
| *** yamahata_ has joined #openstack-infra | 15:09 | |
| *** ricky_fest has quit IRC | 15:10 | |
| *** jamesmcarthur has joined #openstack-infra | 15:10 | |
| *** david-lyle has joined #openstack-infra | 15:10 | |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci master: Test with bonded public interface https://review.openstack.org/385562 | 15:11 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci master: Move PublicVirtualFixedIPs to ssl-public-vip.yaml https://review.openstack.org/475419 | 15:11 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci master: Remove PublicVirtualFixedIPs from network-environment.yaml https://review.openstack.org/476179 | 15:11 |
| *** thorst has joined #openstack-infra | 15:12 | |
| *** pcaruana has quit IRC | 15:12 | |
| *** psachin has quit IRC | 15:13 | |
| *** thorst has quit IRC | 15:16 | |
| *** florianf has joined #openstack-infra | 15:16 | |
| *** sree has joined #openstack-infra | 15:16 | |
| *** flepied has quit IRC | 15:16 | |
| *** sree has quit IRC | 15:20 | |
| *** eranrom has joined #openstack-infra | 15:24 | |
| clarkb | pabelanger: wat | 15:28 |
| clarkb | pabelanger: how does fedora not have ip? | 15:28 |
| *** eranrom has quit IRC | 15:29 | |
| fungi | "minimal" image | 15:31 |
| fungi | probably needs the iproute2 package | 15:31 |
| clarkb | right but being able to have internets is pretty base functionality isn't it? | 15:31 |
| fungi | or whatever rh calls it | 15:31 |
| fungi | well, the iproute2 utils aren't necessary for setting up network connectivity | 15:32 |
| fungi | but i agree we might ought to add that to the infra package needs element if this is the case | 15:32 |
| clarkb | I'll get a patch up. It appears to be iproute on fedora, iproute2 on suse and debuntu, checking centos now | 15:34 |
| clarkb | iproute on centos too | 15:35 |
| *** thorst has joined #openstack-infra | 15:36 | |
| *** felipemonteiro_ has quit IRC | 15:38 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config master: Install `ip` everywhere so that we can manage networking https://review.openstack.org/476184 | 15:38 |
| clarkb | I think ^ will do it | 15:38 |
| clarkb | pabelanger: the suse fails are the same as last week. The unstable dib gate has meant the fix there hasn't been able to merge /me checks if the change is still outstanding | 15:39 |
| clarkb | pabelanger: ya https://review.openstack.org/#/c/475051/ basically need to get the dns and mirror things stablized then that should go through then we can release dib then suse builds should be happy | 15:40 |
| *** thorst has quit IRC | 15:40 | |
| *** Rockyg has joined #openstack-infra | 15:41 | |
| inc0 | hey guys, http://logs.openstack.org/69/475469/5/experimental/publish-kolla-dsvm-ansible-deploy-multinode-ubuntu-source-ubuntu-xenial-2-node/cae3258/console.html#_2017-06-21_04_48_05_974167 this env will work in periodic gates? | 15:42 |
| clarkb | inc0: the value will be different (will be periodic/job_name/stuff) but yes LOG_PATH should be present there as well. | 15:43 |
| *** danpawlik is now known as danpawlik_ | 15:43 | |
| inc0 | and still logs.openstack.org right? | 15:44 |
| clarkb | inc0: yes | 15:44 |
| inc0 | cool, thanks | 15:44 |
| *** dizquierdo has quit IRC | 15:45 | |
| *** jascott1 has joined #openstack-infra | 15:45 | |
| *** jamesdenton has quit IRC | 15:45 | |
| *** jamesdenton has joined #openstack-infra | 15:48 | |
| *** gyee has joined #openstack-infra | 15:49 | |
| *** tmorin has quit IRC | 15:49 | |
| *** jascott1 has quit IRC | 15:50 | |
| fungi | inc0: also if you generate subunit results from the job(s) then you can track that via openstack-health (through its api or the status page) too | 15:50 |
| *** wewe0901 has joined #openstack-infra | 15:51 | |
| clarkb | fungi: pabelanger I'll get new centos and fedora images building once 476184 merges | 15:51 |
| clarkb | fungi: any idea for how long the ready script has been in place? | 15:51 |
| inc0 | fungi: subunit results? | 15:52 |
| fungi | clarkb: what do you mean by "in place"? | 15:53 |
| clarkb | sometimes I really dislike gerrit. notice how I pushed a ps4, then WIP'd ps3... https://review.openstack.org/#/c/475608/4 | 15:54 |
| clarkb | fungi: when were images rebuilt? | 15:54 |
| clarkb | fungi: guessing somtime overnight for us as dib was unhappy and pabelanger and ianw sorted out the builders | 15:54 |
| clarkb | I guess I can just go do a listing | 15:54 |
| fungi | inc0: http://codesearch.openstack.org/?q=generate-subunit | 15:54 |
| *** salv-orlando has joined #openstack-infra | 15:55 | |
| fungi | clarkb: i believe they rolled back the failing images while we were asleep, based on channel history here | 15:55 |
| openstackgerrit | Merged openstack-infra/project-config master: Install `ip` everywhere so that we can manage networking https://review.openstack.org/476184 | 15:55 |
| clarkb | fungi: want to see if logstash reports any improvement in dns timeouts since then (though I guess only on xenial because no ip elsewhere) | 15:55 |
| inc0 | thanks fungi | 15:55 |
| *** gyee has quit IRC | 15:55 | |
| fungi | clarkb: very weird that it allowed you to leave a vote on a non-current patchset | 15:56 |
| fungi | seems likely to be a gerrit bug | 15:56 |
| clarkb | fungi: yup | 15:56 |
| *** salv-orlando has quit IRC | 15:56 | |
| clarkb | my logstash query says only one job failed dns lookups in last 12 hours and it was in sto2 \o/ I think that may mean its happier now | 15:57 |
| *** salv-orlando has joined #openstack-infra | 15:57 | |
| *** Sukhdev_ has joined #openstack-infra | 15:57 | |
| *** ralonsoh has quit IRC | 15:57 | |
| fungi | inc0: that's the format which gets indexed by the subunit2sql processor and then queried through the openstack-health api to build http://status.openstack.org/openstack-health/ | 15:57 |
| clarkb | Once fedora and centos update we should be able to get a better picture then I can revert the d-g change to collect unbound logs and abandon the other unbound related changes | 15:57 |
| *** jascott1 has joined #openstack-infra | 15:57 | |
| *** markus_z has quit IRC | 15:58 | |
| inc0 | fungi: I'll also need to think how to make people look at it;) | 15:58 |
| *** markvoelker has quit IRC | 15:58 | |
| openstackgerrit | YAMAMOTO Takashi proposed openstack-infra/project-config master: networking-midonet: Move centos jobs out of experimental https://review.openstack.org/476196 | 15:59 |
| *** markvoelker has joined #openstack-infra | 15:59 | |
| fungi | inc0: yeah, maybe people can be convinced to look at http://status.openstack.org/openstack-health/#/?searchProject=kolla each morning when they wake up or something | 15:59 |
| *** vhosakot has joined #openstack-infra | 16:00 | |
| inc0 | or somehow connect that with ircbot? | 16:00 |
| *** Apoorva has joined #openstack-infra | 16:00 | |
| fungi | inc0: or maybe http://status.openstack.org/openstack-health/#/g/project/openstack~2Fkolla and http://status.openstack.org/openstack-health/#/g/project/openstack~2Fkolla-ansible are more useful i guess | 16:00 |
| *** mdrabe has quit IRC | 16:01 | |
| fungi | inc0: if the logs are being indexed with logstash, then failures get reported by the elastic-recheck bot (currently in #openstack-qa) though i'm not sure if it has a way to handle additional channels or filter by project | 16:01 |
| fungi | probably a more useful discussion to have with the qa team | 16:02 |
| *** tesseract has quit IRC | 16:03 | |
| *** rderose has joined #openstack-infra | 16:03 | |
| *** gyee has joined #openstack-infra | 16:04 | |
| *** makowals has quit IRC | 16:05 | |
| clarkb | ya there are ways to set it up to talk elsewhere and filter it down to certain projects and only positive matches too | 16:05 |
| *** ykarel has quit IRC | 16:07 | |
| *** mdrabe has joined #openstack-infra | 16:07 | |
| *** aarefiev is now known as aarefiev_afk | 16:07 | |
| *** dtantsur is now known as dtantsur|afk | 16:09 | |
| *** felipemonteiro has joined #openstack-infra | 16:09 | |
| *** felipemonteiro_ has joined #openstack-infra | 16:10 | |
| *** yamamoto_ has joined #openstack-infra | 16:10 | |
| *** LindaWang has quit IRC | 16:10 | |
| *** hieulq_ has quit IRC | 16:12 | |
| *** salv-orl_ has joined #openstack-infra | 16:12 | |
| openstackgerrit | Kaitlin Farr proposed openstack-infra/project-config master: New project for Castellan UI plugin https://review.openstack.org/475835 | 16:13 |
| *** yamamoto has quit IRC | 16:13 | |
| *** gomarivera has quit IRC | 16:13 | |
| *** felipemonteiro has quit IRC | 16:14 | |
| *** gomarivera has joined #openstack-infra | 16:14 | |
| openstackgerrit | YAMAMOTO Takashi proposed openstack-infra/project-config master: networking-midonet: Remove mitaka jobs https://review.openstack.org/476205 | 16:14 |
| *** thorst has joined #openstack-infra | 16:14 | |
| *** salv-orlando has quit IRC | 16:15 | |
| *** gomarivera has quit IRC | 16:18 | |
| clarkb | fedora-25 and centos-7 are rebuilding now | 16:19 |
| *** thorst has quit IRC | 16:19 | |
| *** Julien-zte has quit IRC | 16:21 | |
| *** jamesmcarthur has quit IRC | 16:25 | |
| *** jamesmcarthur has joined #openstack-infra | 16:25 | |
| clarkb | I'm going to keep an eye on both to make sure they both build successfully and then that they both boot with ready scripts that don't have sads | 16:25 |
| *** jascott1 has left #openstack-infra | 16:26 | |
| clarkb | fungi: after breakfast I am going to work on getting a new gerrit branch up for 2.13.8, push the 2.13.7 additioanl changes onto it then fix the plugin builds | 16:27 |
| openstackgerrit | YAMAMOTO Takashi proposed openstack-infra/project-config master: networking-midonet: Remove mitaka jobs https://review.openstack.org/476205 | 16:27 |
| clarkb | I think the plugin builds are failing beacuse we stopped installing ant into our images when we ripped out puppet from dib | 16:27 |
| *** thorst has joined #openstack-infra | 16:27 | |
| *** thorst has quit IRC | 16:27 | |
| clarkb | so should be straightforward to have the job run the fallback bindep list if not already | 16:27 |
| fungi | clarkb: good call, or just add a bit to the job definition to install ant directly | 16:28 |
| *** sambetts is now known as sambetts|afk | 16:28 | |
| clarkb | ya I worry there are other deps like java that we don't install either :) | 16:28 |
| *** yamahata_ has quit IRC | 16:28 | |
| fungi | but yeah, then who knows what else it's missing | 16:28 |
| fungi | just adding the install-distro-packages builder to the job should do the trick | 16:29 |
| fungi | i'll propose that now | 16:29 |
| clarkb | thanks | 16:29 |
| *** jamesmcarthur has quit IRC | 16:29 | |
| fungi | clarkb: oh, hrm, it already does | 16:30 |
| fungi | maybe we removed ant from the fallback list instead? | 16:30 |
| clarkb | I guess ant is missing from that list then? | 16:30 |
| clarkb | ya | 16:30 |
| *** mhickey has quit IRC | 16:34 | |
| *** thorst has joined #openstack-infra | 16:34 | |
| *** thorst has quit IRC | 16:39 | |
| clarkb | fungi: I24eb057124c03bd856e256af4e5f9f5fda935b47 | 16:40 |
| fungi | yep. i thought what i was saying sounded familiarly like something i'd already said before | 16:41 |
| fungi | we could just tack on a shell builder to `echo ant >> /usr/local/jenkins/data/bindep-fallback.txt` before install-distro-packages runs | 16:43 |
| clarkb | I'm not finding anything that adds ant to the gerrit jobs though | 16:43 |
| fungi | right, i think it never got solved. missing followup | 16:43 |
| clarkb | also asciidoc | 16:43 |
| fungi | seems we should probably add asciidoc too | 16:43 |
| clarkb | jinx | 16:44 |
| fungi | jinx | 16:44 |
| fungi | i'm writing up the change now | 16:44 |
| clarkb | ok | 16:44 |
| clarkb | there are two places that need it gerrit.yaml and gerrit-plugin-jobs.yaml I think | 16:45 |
| fungi | do the gerrit warfile builds need it too? | 16:45 |
| fungi | right, that's what i was wondering | 16:45 |
| *** trown is now known as trown|lunch | 16:45 | |
| clarkb | I think they need at least asciidoc to build the gerrit docs? | 16:46 |
| *** thorst has joined #openstack-infra | 16:47 | |
| *** yamamoto_ has quit IRC | 16:47 | |
| *** rderose has quit IRC | 16:48 | |
| *** yamamoto has joined #openstack-infra | 16:49 | |
| *** hrubi has quit IRC | 16:50 | |
| *** jamesmcarthur has joined #openstack-infra | 16:50 | |
| *** thorst has quit IRC | 16:51 | |
| *** Sukhdev_ has quit IRC | 16:52 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config master: Install ant and asciidoc in Gerrit build jobs https://review.openstack.org/476215 | 16:52 |
| fungi | clarkb: ^ there we go | 16:53 |
| clarkb | new fedora image is uploading now. Centos is almost done building | 16:53 |
| *** hrubi has joined #openstack-infra | 16:53 | |
| *** ramishra has quit IRC | 16:54 | |
| clarkb | fungi: I wonder if you need to be root to write to that file? | 16:54 |
| fungi | clarkb: probably... oh i should sudo there | 16:54 |
| clarkb | fungi: -rw-r--r-- 1 root root 3977 Jun 21 11:10 /usr/local/jenkins/common_data/bindep-fallback.txt | 16:55 |
| clarkb | fungi: sudo won't work because it is >> | 16:55 |
| fungi | right, i'm reworking to take that into account | 16:55 |
| clarkb | you can do cat | sudo tee -a | 16:55 |
| clarkb | kk | 16:55 |
| *** ramishra has joined #openstack-infra | 16:56 | |
| *** jpich has quit IRC | 16:56 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config master: Install ant and asciidoc in Gerrit build jobs https://review.openstack.org/476215 | 16:58 |
| fungi | ahh, yeah i coulda done tee as well i guess | 16:58 |
| *** derekh has quit IRC | 16:58 | |
| clarkb | this works | 16:59 |
| *** jamesdenton has quit IRC | 16:59 | |
| *** jascott1 has joined #openstack-infra | 17:00 | |
| *** jamesdenton has joined #openstack-infra | 17:01 | |
| *** Apoorva_ has joined #openstack-infra | 17:02 | |
| *** jascott1 has quit IRC | 17:05 | |
| clarkb | fungi: I'm wondering if we have any other reviewers for ^ today | 17:05 |
| clarkb | AJaeger: are you around? | 17:05 |
| *** Apoorva has quit IRC | 17:05 | |
| *** harlowja has joined #openstack-infra | 17:07 | |
| fungi | clarkb: maybe just single-core approve in this case. it doesn't touch any critical jobs | 17:07 |
| *** hashar has quit IRC | 17:08 | |
| *** yamahata_ has joined #openstack-infra | 17:10 | |
| *** jamesmcarthur has quit IRC | 17:10 | |
| *** e0ne has quit IRC | 17:11 | |
| *** jamesmcarthur has joined #openstack-infra | 17:12 | |
| *** lucasagomes is now known as lucas-afk | 17:12 | |
| *** shardy has quit IRC | 17:13 | |
| *** links has quit IRC | 17:15 | |
| clarkb | fungi: I have approved it | 17:17 |
| *** makowals has joined #openstack-infra | 17:17 | |
| clarkb | centos images are now uploading as well | 17:17 |
| *** jamesmcarthur has quit IRC | 17:17 | |
| *** ltomasbo is now known as ltomasbo|away | 17:19 | |
| *** thorst has joined #openstack-infra | 17:19 | |
| *** netsinn has joined #openstack-infra | 17:19 | |
| fungi | after spending so much time with gerrit, i find it weird that phabricator-based workflow weird (for example, workarounds to include extra context because the diff displayed is the diff uploaded, not rendered from the backing vcs), like https://reviews.llvm.org/D34386 | 17:20 |
| fungi | er, i find phabricator-based workflows weird, i mean | 17:21 |
| *** tosky has quit IRC | 17:21 | |
| fungi | "...diffs uploaded to Phabricator should contain as much context as possible, to make it easier to see the surrounding code..." | 17:21 |
| fungi | really?!? | 17:21 |
| fungi | also, discussion placed above the diff, all on one page | 17:22 |
| *** vhosakot has quit IRC | 17:22 | |
| *** netsin has quit IRC | 17:22 | |
| clarkb | the upside to patch based systems is that you can support arbitrary vcs | 17:23 |
| *** vhosakot has joined #openstack-infra | 17:23 | |
| clarkb | darcs has a fork/reimplementation going on now too that is supposed to give you performant patch based dvcs | 17:23 |
| clarkb | but yes it definitely makes it harder to review without grabbing and applying yourself | 17:24 |
| *** thorst has quit IRC | 17:24 | |
| fungi | yeah, i guess it simplifies the review system implementation by not making it need to interact with any revision control, but you also lose functionality like detecting conflicts with other proposals, automatic parent/child relationship determination, et cetera | 17:25 |
| openstackgerrit | Merged openstack-infra/project-config master: Install ant and asciidoc in Gerrit build jobs https://review.openstack.org/476215 | 17:26 |
| fungi | i guess all of those _could_ be bolted on with an external ci system | 17:26 |
| *** thorst has joined #openstack-infra | 17:28 | |
| *** ijw has joined #openstack-infra | 17:28 | |
| *** armax has quit IRC | 17:29 | |
| clarkb | pijul is the darcs like system | 17:31 |
| *** sree has joined #openstack-infra | 17:31 | |
| *** thorst has quit IRC | 17:34 | |
| *** sree has quit IRC | 17:36 | |
| *** thorst has joined #openstack-infra | 17:38 | |
| *** makowals has quit IRC | 17:39 | |
| *** thorst has quit IRC | 17:40 | |
| *** yamamoto has quit IRC | 17:42 | |
| *** cody-somerville has joined #openstack-infra | 17:44 | |
| *** csomerville has quit IRC | 17:44 | |
| *** electrofelix has quit IRC | 17:45 | |
| *** vhosakot has quit IRC | 17:45 | |
| openstackgerrit | Andreas Florath proposed openstack/diskimage-builder master: Unify apt-sources handling https://review.openstack.org/454456 | 17:47 |
| *** jpena is now known as jpena|off | 17:48 | |
| *** yamamoto has joined #openstack-infra | 17:51 | |
| *** oanson has quit IRC | 17:52 | |
| *** oanson has joined #openstack-infra | 17:52 | |
| *** thorst has joined #openstack-infra | 17:53 | |
| clarkb | Shrews: there doesn't appear to be any way to force an image upload to a provider? An image failed to upload to both ovh regions but since we have another image that is less than 24 hours old there it won't reupload the failed uploads | 17:53 |
| clarkb | Shrews: at least that is what appears to be happening | 17:53 |
| openstackgerrit | Andreas Florath proposed openstack/diskimage-builder master: Easy DIB development: add dibdevel-run element https://review.openstack.org/444666 | 17:54 |
| clarkb | it is also not deleting the failed records | 17:54 |
| AJaeger | clarkb: now I'm around | 17:54 |
| clarkb | AJaeger: too late I already approved it :) fixes to our gerrit builds | 17:54 |
| *** tosky has joined #openstack-infra | 17:54 | |
| AJaeger | clarkb: just found in backscroll - and looks fine. | 17:55 |
| *** yamamoto has quit IRC | 17:56 | |
| *** krenczewski has quit IRC | 17:56 | |
| Shrews | clarkb: it will only upload (or keep trying to upload) the most recent image | 17:56 |
| Shrews | most recent READY image, that is | 17:56 |
| clarkb | Shrews: yes, it is not doing that | 17:56 |
| clarkb | I think because there are two images both < 24 hours old? | 17:56 |
| fungi | clarkb: did all the others already complete? | 17:56 |
| clarkb | also '2017-06-21 17:34:29,187 INFO nodepool.builder.UploadWorker.13: Image build fedora-25-0000007400 in ovh-gra1 is ready' is weird because that gets logged after it fails to upload | 17:57 |
| *** trown|lunch is now known as trown | 17:57 | |
| clarkb | fungi: yes fedora-25 appears to have reached steady state with uploads to !ovh all succeeding and ovh failing | 17:57 |
| clarkb | centos-7 is in progress but on the other builder | 17:57 |
| *** eranrom has joined #openstack-infra | 17:57 | |
| clarkb | 2017-06-21 17:20:05,884 ERROR nodepool.builder.UploadWorker.8: Failed to upload image fedora-25 to provider ovh-gra1 <- is the failure getting logged | 17:57 |
| Shrews | clarkb: something is askew then. that log message should not appear on failed uploads | 17:57 |
| *** thorst has quit IRC | 17:57 | |
| clarkb | then 14 minutes it thinks it is ready | 17:57 |
| clarkb | *14 minutes later | 17:58 |
| clarkb | oh that is two different upload workers, what is going on there | 17:58 |
| clarkb | Shrews: ok I see what happened | 17:59 |
| clarkb | Shrews: the first uploaded failed then the second succeeded. We don't remove the failed records from the db (I thought we were going to do that?) | 17:59 |
| Shrews | clarkb: i think a change to master was merged a few days back | 18:00 |
| *** jascott1 has joined #openstack-infra | 18:00 | |
| clarkb | ok so possibly just waiting on restarted services | 18:00 |
| *** thorst has joined #openstack-infra | 18:00 | |
| clarkb | https://review.openstack.org/#/c/467783/ was to zuulv3 | 18:01 |
| clarkb | maybe we didn't port to master? | 18:01 |
| *** kzaitsev_mb has joined #openstack-infra | 18:01 | |
| Shrews | clarkb: oh, https://review.openstack.org/#/c/470364/ | 18:01 |
| *** csomerville has joined #openstack-infra | 18:02 | |
| clarkb | Shrews: I think ^ is a separate problem for builds not uploads. 467783 should be ported to master to handle uploads | 18:02 |
| Shrews | those are two diff.. yeha | 18:02 |
| clarkb | I will push a cherry pick to master | 18:02 |
| clarkb | then review your thing for builds | 18:02 |
| openstackgerrit | Clark Boylan proposed openstack-infra/nodepool master: Cleanup failed upload records https://review.openstack.org/476223 | 18:04 |
| clarkb | Shrews: ^ there we go | 18:04 |
| *** hashar has joined #openstack-infra | 18:05 | |
| *** thorst has quit IRC | 18:05 | |
| *** cody-somerville has quit IRC | 18:05 | |
| openstackgerrit | Andreas Florath proposed openstack/diskimage-builder master: Remove base element functionality https://review.openstack.org/444590 | 18:06 |
| AJaeger | lbragstad: I don't understand https://review.openstack.org/#/c/476168/1/jenkins/jobs/keystone.yaml, do you have a few minutes to discuss? | 18:07 |
| *** rbrndt has quit IRC | 18:07 | |
| *** rbrndt has joined #openstack-infra | 18:07 | |
| *** rbrndt has quit IRC | 18:07 | |
| *** Apoorva_ has quit IRC | 18:09 | |
| *** Apoorva has joined #openstack-infra | 18:10 | |
| clarkb | fungi: I have created a openstack/2.13.8 branch in the gerrit repo based on v2.13.8. Will work on sorting out what needs porting there today | 18:10 |
| openstackgerrit | Merged openstack-infra/nodepool master: Consider hostname in determining local builds https://review.openstack.org/470364 | 18:11 |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config master: Build gerrit plugins against 2.13.8 https://review.openstack.org/476224 | 18:14 |
| clarkb | fungi: AJaeger ^ that makes plugins build against the new branch | 18:14 |
| *** ijw has quit IRC | 18:14 | |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci master: DONT MERGE: containers periodic test https://review.openstack.org/475747 | 18:15 |
| *** yamamoto has joined #openstack-infra | 18:15 | |
| *** JudeC has joined #openstack-infra | 18:16 | |
| *** salv-orl_ has quit IRC | 18:17 | |
| *** Rockyg has quit IRC | 18:17 | |
| fungi | clarkb: curious whether we should be branching from the tip of upstream/stable-2.13 rather than the v2.13.8 tag (the former has some additional fixes backported (8 non-merge commits several of which look like definite bug fixes) | 18:17 |
| clarkb | fungi: I think in the past zaro has cherry picked those if they were wnated | 18:18 |
| *** ijw has joined #openstack-infra | 18:18 | |
| clarkb | and added in our changes too | 18:18 |
| fungi | okay, wfm | 18:18 |
| clarkb | also looks like adding iproute fixed part of the fedora/centos problem but not all of it :/ | 18:18 |
| * clarkb digs into that more | 18:18 | |
| *** ijw has quit IRC | 18:23 | |
| *** charz has joined #openstack-infra | 18:23 | |
| *** gomarivera has joined #openstack-infra | 18:24 | |
| *** ijw has joined #openstack-infra | 18:24 | |
| *** timburke has joined #openstack-infra | 18:25 | |
| fungi | ansible question... say i want to collect stdout from specific commands run on a bunch of hosts... is there a good way to go about that without needing to feed it all into a json parser? | 18:26 |
| fungi | or am i better off using ansible to parse the inventory and then just iterate over lots of ssh commands instead? | 18:26 |
| *** marst has quit IRC | 18:27 | |
| *** marst has joined #openstack-infra | 18:28 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Do not cache stackviz https://review.openstack.org/476229 | 18:31 |
| *** jascott1 has quit IRC | 18:32 | |
| *** jascott1 has joined #openstack-infra | 18:33 | |
| *** jascott1 has quit IRC | 18:34 | |
| *** jascott1 has joined #openstack-infra | 18:34 | |
| *** hashar has quit IRC | 18:35 | |
| *** gomarivera has quit IRC | 18:36 | |
| fungi | oh, in fact the output is only even present on failure | 18:37 |
| *** makowals has joined #openstack-infra | 18:37 | |
| fungi | i guess i should use something other than a shell task for this | 18:37 |
| *** Swami has joined #openstack-infra | 18:38 | |
| openstackgerrit | Merged openstack-infra/project-config master: Build gerrit plugins against 2.13.8 https://review.openstack.org/476224 | 18:39 |
| fungi | found some options for using debug to dump stdout as a python string or list of strings, but no raw output yet | 18:41 |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config master: Restore selinux context on unbound forwarding.conf https://review.openstack.org/476233 | 18:43 |
| clarkb | fungi: what you do is register the stdout from the shell command then print the registered output | 18:43 |
| *** Sukhdev has joined #openstack-infra | 18:43 | |
| clarkb | fungi: then ya you have to pass that through a filter outisde of ansible is most likely easiest | 18:43 |
| fungi | oh, right, i can tell it to use python's print() | 18:44 |
| clarkb | fungi: I had a playbook similar to this in the past for checking this stuff | 18:44 |
| *** makowals has quit IRC | 18:44 | |
| clarkb | fungi: problem is it still wraps it in ansible's silly logging | 18:44 |
| clarkb | fungi: ianw pabelanger change above should fix centos and fedora ready scripting | 18:44 |
| *** efoley has quit IRC | 18:44 | |
| *** yamamoto has quit IRC | 18:44 | |
| clarkb | fungi: check out puppetmaster:/home/clarkb/collect_facts.yaml | 18:46 |
| clarkb | fungi: I think that likely does what you are wanting | 18:46 |
| *** hashar has joined #openstack-infra | 18:47 | |
| *** aviau has quit IRC | 18:47 | |
| fungi | oh, that's basically what i already had. issue is i would like to render multiline output from commands without parsing complex datastructures afterward | 18:47 |
| *** aviau has joined #openstack-infra | 18:47 | |
| clarkb | ya I don't think ansible makes that easy | 18:48 |
| fungi | the equivalent of `ssh somehost somecommand >> dump.txt` | 18:48 |
| fungi | i don't really care what host it ran on or other metadata | 18:48 |
| clarkb | because you have to ingest the data to an internal representation and then emit it which gets written in ansibles logging format | 18:48 |
| clarkb | which can even include cowsay :) | 18:49 |
| fungi | okay, i'll just use ansible to give me a list of hosts from the inventory and use ssh directly | 18:49 |
| fungi | now to figure out how to get ansible to list the inventory in whatever host format it would use for establishing connections | 18:50 |
| clarkb | AJaeger: fungi reviews on https://review.openstack.org/#/c/476233/ soon would be much appreciated. I will rebuild iamges with that in | 18:50 |
| mmedvede | hi infra, can anyone consider +3 this ETCD related d-g patch https://review.openstack.org/#/c/467437 ? | 18:51 |
| clarkb | mmedvede: can you just use the localrc pass through option instead? | 18:52 |
| clarkb | mmedvede: I think we are trying to avoid d-g flags that just set devstack flags (we want to reduce down to flags that require some implementation logic, like am I multinode and need to set up overlay networking) | 18:53 |
| mmedvede | clarkb: I do not think so, because d-g overrides it | 18:53 |
| clarkb | mmedvede: last match should win | 18:53 |
| clarkb | mmedvede: so your setting of it will override d-g | 18:53 |
| mmedvede | clarkb: let me recheck, I think I considered it | 18:54 |
| AJaeger | clarkb: +3 | 18:54 |
| *** ijw has quit IRC | 18:55 | |
| clarkb | AJaeger: fungi thanks much | 18:55 |
| *** ijw has joined #openstack-infra | 18:55 | |
| clarkb | the good news is that logstash shows that this should fix the vast majority of dns issues | 18:56 |
| clarkb | just need to get it working on those other platforms... | 18:56 |
| AJaeger | clarkb: if you want to speed up image builds a few seconds, consider https://review.openstack.org/476229 as well ;) | 18:57 |
| fungi | ansible --list-hosts seems to list logical names rather than what it will actually ssh into (including some systems which aren't in dns or are known by their nova uuids) | 18:57 |
| clarkb | fungi: you can likely parse the json inventory it writes to the cache directly | 18:58 |
| fungi | yup, was trying to avoid writing software, but i guess that's what it comes down to | 18:58 |
| clarkb | AJaeger: I think we cache taht tarball so that devstack-gate can install stackviz from the tarball at runtime | 18:59 |
| clarkb | AJaeger: can you check what devstack-gate does? I don't know if ^ is save | 18:59 |
| clarkb | *safe | 18:59 |
| *** ijw has quit IRC | 18:59 | |
| *** rbrndt has joined #openstack-infra | 19:02 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul feature/zuulv3: WIP: Add web-based console log streaming https://review.openstack.org/463353 | 19:02 |
| *** eranrom has quit IRC | 19:03 | |
| *** eranrom has joined #openstack-infra | 19:03 | |
| fungi | and ansible --list-hosts includes our infracloud inventory while the ansible-inventory.cache does not (i suppose because we define it statically) | 19:03 |
| *** eranrom has quit IRC | 19:03 | |
| *** eranrom has joined #openstack-infra | 19:04 | |
| *** eranrom has quit IRC | 19:04 | |
| *** eranrom has joined #openstack-infra | 19:05 | |
| *** wewe0901 has quit IRC | 19:05 | |
| *** eranrom has quit IRC | 19:05 | |
| *** ekhugen has quit IRC | 19:05 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul feature/zuulv3: WIP: Add web-based console log streaming https://review.openstack.org/463353 | 19:05 |
| *** eranrom has joined #openstack-infra | 19:05 | |
| *** eranrom has quit IRC | 19:06 | |
| *** eranrom has joined #openstack-infra | 19:06 | |
| *** eranrom has quit IRC | 19:07 | |
| *** priteau has quit IRC | 19:07 | |
| *** eranrom has joined #openstack-infra | 19:07 | |
| *** priteau has joined #openstack-infra | 19:07 | |
| *** eranrom has quit IRC | 19:07 | |
| *** nicolasbock has quit IRC | 19:07 | |
| *** eranrom has joined #openstack-infra | 19:08 | |
| openstackgerrit | Merged openstack-infra/project-config master: Restore selinux context on unbound forwarding.conf https://review.openstack.org/476233 | 19:08 |
| *** eranrom has quit IRC | 19:08 | |
| *** eranrom has joined #openstack-infra | 19:09 | |
| *** eranrom has quit IRC | 19:09 | |
| *** eranrom has joined #openstack-infra | 19:09 | |
| *** eranrom has quit IRC | 19:10 | |
| AJaeger | clarkb: you're right - will abandon. thanks | 19:10 |
| mmedvede | clarkb: I do not see how I can set the ETCD_DOWNLOAD_URL after the hardcoded value in devstack-vm-gate.sh. After it is hardcoded and between the ./devstack.sh there is nowhere to override it | 19:10 |
| *** eranrom has joined #openstack-infra | 19:10 | |
| *** eranrom has quit IRC | 19:11 | |
| clarkb | mmedvede: via DEVSTACK_LOCALCONF | 19:11 |
| *** eranrom has joined #openstack-infra | 19:11 | |
| clarkb | mmedvede: d-g appends taht to the end of the localconf section | 19:11 |
| *** eranrom has quit IRC | 19:11 | |
| *** priteau has quit IRC | 19:12 | |
| *** eranrom has joined #openstack-infra | 19:12 | |
| *** eranrom has quit IRC | 19:12 | |
| *** eranrom has joined #openstack-infra | 19:13 | |
| *** ekhugen has joined #openstack-infra | 19:13 | |
| *** eranrom has quit IRC | 19:13 | |
| *** eranrom has joined #openstack-infra | 19:13 | |
| *** eranrom has quit IRC | 19:14 | |
| *** eranrom has joined #openstack-infra | 19:14 | |
| *** eranrom has quit IRC | 19:15 | |
| mmedvede | clarkb: thanks. I'll try it and abandon the patch if it works. | 19:15 |
| *** eranrom has joined #openstack-infra | 19:15 | |
| *** eranrom has quit IRC | 19:15 | |
| *** eranrom has joined #openstack-infra | 19:16 | |
| *** eranrom has quit IRC | 19:16 | |
| *** eranrom has joined #openstack-infra | 19:17 | |
| *** eranrom has quit IRC | 19:17 | |
| *** eranrom has joined #openstack-infra | 19:17 | |
| *** eranrom has quit IRC | 19:18 | |
| *** eranrom has joined #openstack-infra | 19:18 | |
| *** eranrom has quit IRC | 19:19 | |
| *** eranrom has joined #openstack-infra | 19:19 | |
| *** thorst has joined #openstack-infra | 19:19 | |
| *** eranrom has quit IRC | 19:19 | |
| *** eranrom has joined #openstack-infra | 19:20 | |
| *** eranrom has quit IRC | 19:20 | |
| *** eranrom has joined #openstack-infra | 19:21 | |
| *** eranrom has quit IRC | 19:21 | |
| *** eranrom has joined #openstack-infra | 19:21 | |
| *** camunoz has quit IRC | 19:22 | |
| *** patriciadomin has quit IRC | 19:22 | |
| *** eranrom has quit IRC | 19:22 | |
| *** eranrom has joined #openstack-infra | 19:22 | |
| *** eranrom has quit IRC | 19:23 | |
| *** eranrom has joined #openstack-infra | 19:23 | |
| *** ijw has joined #openstack-infra | 19:23 | |
| *** eranrom has quit IRC | 19:23 | |
| *** eranrom has joined #openstack-infra | 19:24 | |
| *** eranrom has quit IRC | 19:24 | |
| *** eranrom has joined #openstack-infra | 19:25 | |
| *** eranrom has quit IRC | 19:25 | |
| *** eranrom has joined #openstack-infra | 19:26 | |
| *** eranrom has quit IRC | 19:26 | |
| *** eranrom has joined #openstack-infra | 19:26 | |
| openstackgerrit | wes hayutin proposed openstack-infra/tripleo-ci master: update toci to use --environment and --nodes options in quickstart https://review.openstack.org/475930 | 19:26 |
| *** eranrom has quit IRC | 19:27 | |
| *** makowals has joined #openstack-infra | 19:27 | |
| *** eranrom has joined #openstack-infra | 19:27 | |
| *** thorst has quit IRC | 19:27 | |
| *** eranrom has quit IRC | 19:27 | |
| *** amoralej is now known as amoralej|off | 19:28 | |
| *** eranrom has joined #openstack-infra | 19:28 | |
| *** eranrom has quit IRC | 19:28 | |
| *** eranrom has joined #openstack-infra | 19:29 | |
| *** eranrom has quit IRC | 19:29 | |
| *** eranrom has joined #openstack-infra | 19:29 | |
| *** eranrom has quit IRC | 19:30 | |
| *** eranrom has joined #openstack-infra | 19:30 | |
| *** eranrom has quit IRC | 19:31 | |
| clarkb | fedora build failed, looks like yum couldn't talk to a mirror :/ | 19:31 |
| clarkb | it is rerunning though | 19:31 |
| *** eranrom has joined #openstack-infra | 19:31 | |
| *** eranrom has quit IRC | 19:31 | |
| AJaeger | Any project-config core around to approve some older changes to clean the queue, please? https://review.openstack.org/#/c/468171/ https://review.openstack.org/#/c/467523 https://review.openstack.org/#/c/468202/ https://review.openstack.org/#/c/468337/ https://review.openstack.org/#/c/470206/ | 19:32 |
| *** eranrom has joined #openstack-infra | 19:32 | |
| *** eranrom has quit IRC | 19:32 | |
| clarkb | and second build just failed. It looks like fedora mirrors are all out of sync | 19:32 |
| clarkb | http://mirror.sjc02.svwh.net/fedora/updates/25/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for updates for example | 19:33 |
| *** eranrom has joined #openstack-infra | 19:33 | |
| clarkb | ugh | 19:33 |
| *** mat128 has quit IRC | 19:33 | |
| *** eranrom has quit IRC | 19:33 | |
| *** eranrom has joined #openstack-infra | 19:33 | |
| *** eranrom has quit IRC | 19:34 | |
| *** eranrom has joined #openstack-infra | 19:34 | |
| *** eranrom has quit IRC | 19:35 | |
| clarkb | I guess we wait for the mirrors to get up to date | 19:35 |
| *** eranrom has joined #openstack-infra | 19:35 | |
| *** eranrom has quit IRC | 19:35 | |
| *** ijw has quit IRC | 19:36 | |
| *** eranrom has joined #openstack-infra | 19:36 | |
| *** eranrom has quit IRC | 19:36 | |
| *** eranrom has joined #openstack-infra | 19:37 | |
| *** eranrom has quit IRC | 19:37 | |
| *** eranrom has joined #openstack-infra | 19:37 | |
| *** wolverineav has quit IRC | 19:38 | |
| *** eranrom has quit IRC | 19:38 | |
| clarkb | do people that use yum more than me know if my interpretation is correct ^ | 19:38 |
| *** eranrom has joined #openstack-infra | 19:38 | |
| *** wolverineav has joined #openstack-infra | 19:39 | |
| *** eranrom has quit IRC | 19:39 | |
| *** eranrom has joined #openstack-infra | 19:39 | |
| *** wolverineav has quit IRC | 19:39 | |
| *** eranrom has quit IRC | 19:39 | |
| *** eranrom has joined #openstack-infra | 19:40 | |
| *** eranrom has quit IRC | 19:40 | |
| *** wolverineav has joined #openstack-infra | 19:40 | |
| clarkb | EmilienM: ^ you may know? | 19:40 |
| *** eranrom has joined #openstack-infra | 19:41 | |
| *** eranrom has quit IRC | 19:41 | |
| *** eranrom has joined #openstack-infra | 19:41 | |
| *** eranrom has quit IRC | 19:42 | |
| *** eranrom has joined #openstack-infra | 19:42 | |
| fungi | clarkb: this tends to happen, usually some sort of mismatch on official fedora mirrors and then we're mirroring that brokenness until they correct it | 19:42 |
| *** rkukura has joined #openstack-infra | 19:42 | |
| *** eranrom has quit IRC | 19:43 | |
| clarkb | in this case it isn't our mirrors, but affecting dib's ability to make a new image to fix that selinux thing :( | 19:43 |
| clarkb | wee | 19:43 |
| *** eranrom has joined #openstack-infra | 19:43 | |
| fungi | yup | 19:43 |
| clarkb | I have started centos-7 build now too | 19:43 |
| fungi | fedora seems to have broken official mirrors semi-regularly from what we've seen | 19:43 |
| clarkb | since fedora is going nowhere fast | 19:43 |
| *** eranrom has quit IRC | 19:43 | |
| *** eranrom has joined #openstack-infra | 19:44 | |
| *** eranrom has quit IRC | 19:44 | |
| fungi | i assume they're third-party operators running those mirrors expected to use some common framework | 19:44 |
| *** flepied has joined #openstack-infra | 19:44 | |
| *** eranrom has joined #openstack-infra | 19:45 | |
| *** yamamoto has joined #openstack-infra | 19:45 | |
| *** eranrom has quit IRC | 19:45 | |
| fungi | and if there's any automated consistency checker that grooms the mirror pool, it doesn't run especially frequently (more likely they just wait for reports of broken mirrors and then remove them manually) | 19:45 |
| *** eranrom has joined #openstack-infra | 19:45 | |
| *** eranrom has quit IRC | 19:46 | |
| *** eranrom has joined #openstack-infra | 19:46 | |
| *** eranrom has quit IRC | 19:47 | |
| *** eranrom has joined #openstack-infra | 19:47 | |
| *** wolverineav has quit IRC | 19:47 | |
| *** patriciadomin has joined #openstack-infra | 19:47 | |
| *** eranrom has quit IRC | 19:48 | |
| *** eranrom has joined #openstack-infra | 19:48 | |
| *** eranrom has quit IRC | 19:48 | |
| *** eranrom has joined #openstack-infra | 19:49 | |
| *** benj_ has quit IRC | 19:49 | |
| *** eranrom has quit IRC | 19:49 | |
| *** eranrom has joined #openstack-infra | 19:49 | |
| *** eranrom has quit IRC | 19:50 | |
| *** eranrom has joined #openstack-infra | 19:50 | |
| *** eranrom has quit IRC | 19:51 | |
| *** eranrom has joined #openstack-infra | 19:51 | |
| *** benj_ has joined #openstack-infra | 19:51 | |
| *** eranrom has quit IRC | 19:52 | |
| *** eranrom has joined #openstack-infra | 19:52 | |
| *** yamamoto has quit IRC | 19:52 | |
| *** eranrom has quit IRC | 19:52 | |
| *** makowals has quit IRC | 19:53 | |
| *** eranrom has joined #openstack-infra | 19:53 | |
| clarkb | https://admin.fedoraproject.org/mirrormanager/propgation/f25_updates that does paint a picture of sadness | 19:53 |
| *** eranrom has quit IRC | 19:53 | |
| *** eranrom has joined #openstack-infra | 19:53 | |
| *** eranrom has quit IRC | 19:54 | |
| *** eranrom has joined #openstack-infra | 19:54 | |
| *** eranrom has quit IRC | 19:55 | |
| *** eranrom has joined #openstack-infra | 19:55 | |
| *** eranrom has quit IRC | 19:56 | |
| *** eranrom has joined #openstack-infra | 19:56 | |
| *** eranrom has quit IRC | 19:56 | |
| *** eranrom has joined #openstack-infra | 19:57 | |
| *** kzaitsev_mb has quit IRC | 19:57 | |
| *** eranrom has quit IRC | 19:57 | |
| *** eranrom has joined #openstack-infra | 19:57 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add openstack/valet IRC channel https://review.openstack.org/468171 | 19:58 |
| *** eranrom has quit IRC | 19:58 | |
| fungi | yeesh | 19:58 |
| *** eranrom has joined #openstack-infra | 19:58 | |
| *** eranrom has quit IRC | 19:59 | |
| *** eranrom has joined #openstack-infra | 19:59 | |
| *** eranrom has quit IRC | 20:00 | |
| *** eranrom has joined #openstack-infra | 20:00 | |
| openstackgerrit | Merged openstack-infra/project-config master: networking-odl: Fix typos (34 vs 35) https://review.openstack.org/467523 | 20:00 |
| openstackgerrit | Merged openstack-infra/project-config master: grafana: Remove python34 stuff https://review.openstack.org/467524 | 20:00 |
| openstackgerrit | Merged openstack-infra/project-config master: Config gerrit bot to Qinling https://review.openstack.org/468202 | 20:00 |
| *** eranrom has quit IRC | 20:00 | |
| *** eranrom has joined #openstack-infra | 20:01 | |
| openstackgerrit | Merged openstack-infra/project-config master: neutron-vpnaas: Make gate-neutron-dsvm-tempest-vpnaas-ubuntu-xenial voting https://review.openstack.org/468337 | 20:01 |
| *** eranrom has quit IRC | 20:01 | |
| openstackgerrit | Merged openstack-infra/project-config master: Retire gate-neutron-dsvm-tempest-vpnaas-ubuntu-trusty-nv https://review.openstack.org/470206 | 20:01 |
| *** eranrom has joined #openstack-infra | 20:02 | |
| *** kzaitsev_mb has joined #openstack-infra | 20:02 | |
| *** eranrom has quit IRC | 20:02 | |
| *** eranrom has joined #openstack-infra | 20:02 | |
| *** eranrom has quit IRC | 20:03 | |
| *** openstackgerrit has quit IRC | 20:03 | |
| *** eranrom has joined #openstack-infra | 20:03 | |
| *** eranrom has quit IRC | 20:04 | |
| *** salv-orlando has joined #openstack-infra | 20:04 | |
| *** eranrom has joined #openstack-infra | 20:04 | |
| AJaeger | fungi, thanks for reviewing! | 20:04 |
| fungi | AJaeger: thanks for weeding out the ones that weren't ready! | 20:04 |
| *** eranrom has quit IRC | 20:04 | |
| AJaeger | ;) | 20:05 |
| *** eranrom has joined #openstack-infra | 20:05 | |
| *** eranrom has quit IRC | 20:05 | |
| *** eranrom has joined #openstack-infra | 20:06 | |
| *** Sukhdev has quit IRC | 20:06 | |
| *** eranrom has quit IRC | 20:06 | |
| *** eranrom has joined #openstack-infra | 20:06 | |
| *** eranrom has quit IRC | 20:07 | |
| *** eranrom has joined #openstack-infra | 20:07 | |
| *** eranrom has quit IRC | 20:08 | |
| *** eranrom has joined #openstack-infra | 20:08 | |
| *** eranrom has quit IRC | 20:08 | |
| *** eranrom has joined #openstack-infra | 20:09 | |
| *** gouthamr has quit IRC | 20:09 | |
| *** eranrom has quit IRC | 20:09 | |
| *** eranrom has joined #openstack-infra | 20:10 | |
| *** eranrom has quit IRC | 20:10 | |
| *** eranrom has joined #openstack-infra | 20:10 | |
| *** eranrom has quit IRC | 20:11 | |
| *** eranrom has joined #openstack-infra | 20:11 | |
| *** openstackgerrit has joined #openstack-infra | 20:11 | |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder master: Update heavyjob to make use of convert_mapping_to_xml() https://review.openstack.org/475981 | 20:11 |
| *** eranrom has quit IRC | 20:12 | |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci master: DONT MERGE: containers periodic test https://review.openstack.org/475747 | 20:12 |
| *** eranrom has joined #openstack-infra | 20:12 | |
| *** eranrom has quit IRC | 20:12 | |
| *** eranrom has joined #openstack-infra | 20:13 | |
| *** eranrom has quit IRC | 20:13 | |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder master: Update batch_tasks to make use of convert_mapping_to_xml() https://review.openstack.org/475976 | 20:13 |
| *** eranrom has joined #openstack-infra | 20:13 | |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder master: Update sidebar to make use of convert_mapping_to_xml() https://review.openstack.org/475970 | 20:14 |
| *** Goneri has quit IRC | 20:14 | |
| *** eranrom has quit IRC | 20:14 | |
| *** priteau has joined #openstack-infra | 20:14 | |
| *** eranrom has joined #openstack-infra | 20:14 | |
| *** eranrom has quit IRC | 20:15 | |
| *** jcoufal has quit IRC | 20:15 | |
| *** eranrom has joined #openstack-infra | 20:15 | |
| *** eranrom has quit IRC | 20:16 | |
| *** eranrom has joined #openstack-infra | 20:16 | |
| *** eranrom has quit IRC | 20:16 | |
| *** eranrom has joined #openstack-infra | 20:17 | |
| *** eranrom has quit IRC | 20:17 | |
| *** eranrom has joined #openstack-infra | 20:17 | |
| *** eranrom has quit IRC | 20:18 | |
| *** eranrom has joined #openstack-infra | 20:18 | |
| *** eranrom has quit IRC | 20:19 | |
| *** eranrom has joined #openstack-infra | 20:19 | |
| *** eranrom has quit IRC | 20:20 | |
| *** dmsimard has quit IRC | 20:20 | |
| *** dmsimard has joined #openstack-infra | 20:21 | |
| *** hashar has quit IRC | 20:22 | |
| clarkb | looks like fedora may have finally gotten past its yum problems | 20:26 |
| clarkb | and centos is almost done converting images | 20:26 |
| *** dprince has quit IRC | 20:32 | |
| *** ijw has joined #openstack-infra | 20:32 | |
| *** shadow-x has joined #openstack-infra | 20:37 | |
| *** shadow-x has quit IRC | 20:37 | |
| *** mdrabe has quit IRC | 20:41 | |
| dhellmann | fungi, clarkb, jeblair : I'm looking into how to automate the process of generating a sitemap.xml for docs.openstack.org. Apparently the one we have now is generated using scrapy. I was thinking we could add a sitemap step to each doc build job, and then build a master file using a periodic job and "find" to find all of the subfiles. Is it possible to set up a job with read access to the AFS share for that? | 20:41 |
| *** mdrabe has joined #openstack-infra | 20:42 | |
| *** wolverineav has joined #openstack-infra | 20:43 | |
| *** gomarivera has joined #openstack-infra | 20:43 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/nodepool master: Cleanup failed upload records https://review.openstack.org/476223 | 20:43 |
| openstackgerrit | Clark Boylan proposed openstack-infra/nodepool master: Disable CleanupWorker thread for test_image_upload_fail https://review.openstack.org/476266 | 20:43 |
| clarkb | Shrews: ^ that should pass testing now | 20:43 |
| fungi | dhellmann: that ought to be possible, even with a single-use node (client setup does need a kernel module compiled and loaded but doesn't require a reboot) | 20:44 |
| *** wolverineav has quit IRC | 20:44 | |
| *** wolverineav has joined #openstack-infra | 20:45 | |
| clarkb | however keep in mind that it will be potentially very slow from europe | 20:45 |
| fungi | dhellmann: though i don't know how time consuming and bandwidth-intensive a find over the entire docs tree might be... so could make more sense to have a persistent node so it can keep the cache between runs | 20:45 |
| fungi | also as clark says, guaranteed topological proximity to the afs cell servers would probably improve performance | 20:46 |
| dhellmann | fungi : the master file would not need to be rebuilt frequently. weekly even? most of the changes are going to be in each subtree | 20:46 |
| dhellmann | ok, I'll give it some thought and put it on the future list | 20:47 |
| clarkb | and now image uploads are failing | 20:47 |
| *** flepied has quit IRC | 20:47 | |
| fungi | it's probably late where jeblair is right now, so getting suggestions from him will probably need to wait until tomorrow | 20:48 |
| *** gouthamr has joined #openstack-infra | 20:48 | |
| clarkb | IOError: Could not find a suitable TLS CA certificate bundle, invalid path: /usr/local/lib/python2.7/dist-packages/requests/cacert.pem | 20:48 |
| clarkb | something broke our requests install? | 20:48 |
| clarkb | the last requests release was on the 14th so I don't think its a new release breaking us | 20:50 |
| *** trown is now known as trown|outtypewww | 20:51 | |
| clarkb | I am going to try reinstalling requests on nb03 | 20:51 |
| clarkb | that didn't help | 20:52 |
| *** amotoki is now known as amotoki_away | 20:52 | |
| clarkb | now trying to install from source using --no-binary ':all:' | 20:53 |
| *** priteau has quit IRC | 20:56 | |
| clarkb | I do not see the cacert.pem in the sdist | 20:57 |
| clarkb | looks like new requests may hard dep on certifi and we need to restart our processes? | 20:57 |
| clarkb | I am going to restart the builder on nb04 | 20:58 |
| *** flepied has joined #openstack-infra | 20:58 | |
| *** jascott1 has quit IRC | 21:00 | |
| clarkb | except I think ianw and pabelanger restarted the builders last night so not sure why the builders would go looking in the wrong place. Maybe we did just update requests somehow? | 21:00 |
| clarkb | in any case restarting seems to make it happier | 21:00 |
| *** jascott1 has joined #openstack-infra | 21:01 | |
| clarkb | Shrews: ^ this whole mess created a few thousand failed upload records. would be good to get 476223 and 476266 in to clean that up | 21:02 |
| *** coreyob has quit IRC | 21:02 | |
| *** andreww has joined #openstack-infra | 21:03 | |
| *** coreyob has joined #openstack-infra | 21:03 | |
| *** jamesdenton has quit IRC | 21:03 | |
| *** andreww has quit IRC | 21:03 | |
| *** andreww has joined #openstack-infra | 21:05 | |
| *** xarses_ has quit IRC | 21:05 | |
| *** baoli has quit IRC | 21:07 | |
| *** jamesdenton has joined #openstack-infra | 21:07 | |
| *** jascott1 has quit IRC | 21:08 | |
| *** jamesdenton has quit IRC | 21:08 | |
| clarkb | https://github.com/requests/requests/commit/a00b7e4b25e041a62b634adc67cb02d7c3667c3e confirms my suspicions | 21:08 |
| clarkb | I'm still confused as to how this only just became a problem, BUT going to ignore that for now and make sure we get workign iamges | 21:09 |
| *** gomarivera has quit IRC | 21:09 | |
| Shrews | clarkb: +2d | 21:11 |
| clarkb | Shrews: tyty | 21:12 |
| *** kzaitsev_mb has quit IRC | 21:14 | |
| *** kgiusti has left #openstack-infra | 21:14 | |
| *** jascott1 has joined #openstack-infra | 21:15 | |
| *** mdrabe has quit IRC | 21:15 | |
| *** ldnunes has quit IRC | 21:16 | |
| clarkb | Shrews: another odd behavior, when I restart the builders to pick up the new requests cert location they were uploading images to the two infracloud regions beacuse infracloud uses a self signed cert that we trust directly and wasn't affected by the requests derp. Looks like those uploads have stuck around in zk as 'uploading' I don't think they are uploading because the processes were killed | 21:16 |
| clarkb | Shrews: do you know if we have any record curation on process start? I know nodepool does it for the instances at least, where it marks anything not in ready as delete when it starts up | 21:17 |
| clarkb | I can hack on correcting ^ if necessary | 21:17 |
| Shrews | clarkb: i *think* uploading records that aren't actually inprogress will get cleaned up | 21:19 |
| Shrews | iirc. not sure how often that occurs. lemme see | 21:19 |
| ianw | clarkb: that's not going to pass until we sort out the devstack side :( https://review.openstack.org/#/c/476108/ | 21:19 |
| clarkb | Shrews: ah ok. if you want to see them do a nodepool image-list | grep uploading | grep infracloud | grep centos | 21:19 |
| ianw | (that being the dib suse change) | 21:19 |
| clarkb | ianw: oh crud, the fail I saw had to do with yum/dnf installing python for fedora I thought | 21:20 |
| clarkb | but I gues that job just happened to fail too (I didn't look at all of them) | 21:20 |
| clarkb | ianw: mordred says that glance is doing version discovery wrong in devstack | 21:20 |
| *** eranrom has joined #openstack-infra | 21:21 | |
| Shrews | clarkb: yeah, they should get cleaned up automatically fairly shortly | 21:22 |
| clarkb | Shrews: ok good to kow, thanks | 21:22 |
| *** chlong has quit IRC | 21:22 | |
| Shrews | clarkb: that command returns nothing, so i will assume they've been removed properly | 21:24 |
| *** mdrabe has joined #openstack-infra | 21:25 | |
| *** eranrom has quit IRC | 21:25 | |
| clarkb | Shrews: | 0000007643 | 0000000001 | infracloud-chocolate | centos-7 | None | None | uploading | 00:00:48:39 | | 21:25 |
| Shrews | oh, i had a wrong grep | 21:25 |
| clarkb | you should get at least two of those one for infracloud vanilla and the other chocolate | 21:25 |
| clarkb | I think total of 4 is what you'll get now thoughas it started new uploads after the process restarted | 21:26 |
| *** dimtruck is now known as zz_dimtruck | 21:26 | |
| *** eric_young has quit IRC | 21:28 | |
| *** sree has joined #openstack-infra | 21:32 | |
| *** aeng has joined #openstack-infra | 21:34 | |
| Shrews | clarkb: oh, those are part of the latest build from what i can tell. so the records won't get cleaned up until there are two more recent builds available. | 21:36 |
| *** sree has quit IRC | 21:36 | |
| clarkb | Shrews: ok so similar to my fix above. I think we do want to clear out any uploading records on process start | 21:36 |
| clarkb | since we know they aren't uploading anymore at that point | 21:36 |
| Shrews | clarkb: yeah. the cleanup logic could be smarter there. i think we short circuit some things based on build recency | 21:37 |
| *** flepied has quit IRC | 21:37 | |
| *** esberglu has quit IRC | 21:41 | |
| *** makowals has joined #openstack-infra | 21:43 | |
| clarkb | Shrews: thanks for looking at it | 21:44 |
| *** bexelbie has quit IRC | 21:46 | |
| *** mdrabe has quit IRC | 21:48 | |
| *** zz_dimtruck is now known as dimtruck | 21:50 | |
| *** bexelbie has joined #openstack-infra | 21:50 | |
| *** esberglu has joined #openstack-infra | 21:55 | |
| EmilienM | clarkb: hey sorry I'm back here | 21:56 |
| *** jkilpatr has quit IRC | 21:56 | |
| EmilienM | clarkb: yeah indeed, it seems like the mirrors wasn't updated - so we might had some mismatch between metadatas & actual packages version | 21:57 |
| *** JudeC has quit IRC | 21:57 | |
| clarkb | EmilienM: we managed to get an image built at least | 21:57 |
| EmilienM | ok | 21:57 |
| clarkb | BUT unfortunately it apparently didn't fix our nodepool issues | 21:57 |
| clarkb | I'm booting an image now to look at why | 21:58 |
| EmilienM | :( | 21:58 |
| EmilienM | clarkb: hey quick question, do you know why Depends-On in https://review.openstack.org/#/c/472731/ doesn't work on http://logs.openstack.org/31/472731/4/check/gate-tripleo-heat-templates-pep8-ubuntu-xenial/b40abfb/console.html.gz ? tripleo-common is deployed from pip | 21:58 |
| *** esberglu has quit IRC | 22:00 | |
| *** JudeC has joined #openstack-infra | 22:00 | |
| clarkb | fungi: any idea why if I ssh into the image and run the configure mirror script both as root and jekins (nodepool uses jenkins) it works fine but when nodepool does it we get: http://paste.openstack.org/show/613337/ ? restorecon is in /usr/sbin on fedora maybe paramiko sets PATH funny via ssh or something? | 22:02 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [DNM] testing glance uwsgi revert & our jobs https://review.openstack.org/476109 | 22:03 |
| *** ijw has quit IRC | 22:03 | |
| *** armax has joined #openstack-infra | 22:03 | |
| *** ijw has joined #openstack-infra | 22:03 | |
| clarkb | EmilienM: yes, depends-on requires you install from source and configure the git repo you install from properly using the zuul refs | 22:03 |
| clarkb | EmilienM: if you install from pypi with pip depends on doesn't do anything | 22:03 |
| *** bexelbie- has joined #openstack-infra | 22:03 | |
| *** bexelbie has quit IRC | 22:04 | |
| *** bexelbie- is now known as bexelbie | 22:04 | |
| *** markvoelker has quit IRC | 22:04 | |
| EmilienM | clarkb: mhh ok | 22:05 |
| EmilienM | thx | 22:05 |
| clarkb | ya I think that must be it, PATH is being carried through ugh | 22:05 |
| fungi | clarkb: are you saying that because restorecon works but ip isn't found? at least on centos it's /usr/sbin/restorecon and /usr/sbin/ip | 22:05 |
| fungi | so they're in the same directory | 22:06 |
| clarkb | fungi: type -p restorecon is failing because it is in /usr/sbin I think | 22:06 |
| clarkb | fungi: ip failing too for that reason but we'd have at least something useable if restorecon worked | 22:06 |
| *** markvoelker_ has joined #openstack-infra | 22:06 | |
| *** rfolco has quit IRC | 22:06 | |
| ianw | yeah, hitting the old "sbin" not in path issue i think | 22:07 |
| fungi | clarkb: oh, i didn't see any actual error from type -p restorecon in your paste | 22:07 |
| fungi | just ip | 22:07 |
| clarkb | fungi: ya that is beacuse type -p restorecon is checked before running restorecon | 22:07 |
| *** pradk has quit IRC | 22:07 | |
| clarkb | since we won't have restorecon on ubuntu | 22:07 |
| fungi | oh, got it. so it's the _absence_ of restorecon invocation there you're pointing out | 22:07 |
| clarkb | fungi: ianw opinions on fixing that? I could just modify the script to append /sbin /usr/sbin /bin /usr/bin /usr/local/bin to PATH | 22:07 |
| clarkb | PATH=" | 22:08 |
| clarkb | er | 22:08 |
| ianw | i think that's pretty much what's done everywhere else (original scripted installed stuff, devstack, etc) | 22:08 |
| fungi | if that script is going to be run on multiple platforms, then yes i'd adjust the script to set its own path | 22:08 |
| clarkb | ok I'll push that up momentarily then we have to rebuild iamges again :( | 22:08 |
| ianw | $ rpm -qf /usr/sbin/restorecon | 22:09 |
| ianw | policycoreutils-2.5-20.fc25.x86_64 | 22:09 |
| clarkb | the good news is that at least on xenial dns looks much happier | 22:09 |
| ianw | 100% sure it's there? | 22:09 |
| clarkb | ianw: I'm positive it is installed | 22:09 |
| clarkb | I booted the image and cehcked it | 22:09 |
| ianw | ok, cool :) you never know, minimal can do weird tings sometimes | 22:09 |
| ianw | things even | 22:09 |
| clarkb | ya... I checked that after the other derp with ip not being in place | 22:09 |
| *** makowals has quit IRC | 22:10 | |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci master: WIP: containers periodic test https://review.openstack.org/475747 | 22:11 |
| *** markvoelker_ has quit IRC | 22:11 | |
| *** felipemonteiro_ has quit IRC | 22:12 | |
| *** salv-orl_ has joined #openstack-infra | 22:12 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config master: Set PATH when running configure mirror script https://review.openstack.org/476281 | 22:13 |
| clarkb | ianw: fungi ^ ok then with that in place I guess its try try again... | 22:14 |
| *** salv-orlando has quit IRC | 22:15 | |
| *** salv-orl_ has quit IRC | 22:15 | |
| *** Julien-zte has joined #openstack-infra | 22:15 | |
| *** salv-orlando has joined #openstack-infra | 22:16 | |
| *** markvoelker has joined #openstack-infra | 22:17 | |
| *** xyang1 has quit IRC | 22:18 | |
| *** jamesmcarthur has joined #openstack-infra | 22:18 | |
| clarkb | ianw: re secure_path is the problem that sudo in some setups will filter sbin back out again? | 22:19 |
| clarkb | ianw: iirc PATH is passed through if you don't have config filtering it | 22:19 |
| *** salv-orlando has quit IRC | 22:21 | |
| *** gordc has quit IRC | 22:21 | |
| clarkb | reading fedora's sudo setup we should be fine with setting PATH since we check for restorecon as not root and run ip as not root | 22:21 |
| *** eranrom has joined #openstack-infra | 22:21 | |
| clarkb | and then /sbin and /usr/sbin are both in secure_path | 22:21 |
| *** jamesmcarthur has quit IRC | 22:23 | |
| clarkb | ok I'm going to step away from the computer for a bit, but if I can get second review on https://review.openstack.org/476281 I can babysit that this evening to hopefully finally have working ecntos and fedora images | 22:24 |
| ianw | clarkb: oh i meant more dirs included, but it's fine | 22:25 |
| fungi | also no need to append | 22:26 |
| *** eranrom has quit IRC | 22:26 | |
| fungi | but probably fine | 22:26 |
| *** Sukhdev has joined #openstack-infra | 22:28 | |
| clarkb | ya append just to avoid missing anything | 22:28 |
| clarkb | or rather regressing by missing something | 22:28 |
| *** marst has quit IRC | 22:33 | |
| clarkb | fungi: any reason you didn't approve? should I go ahead and do so? | 22:37 |
| *** rlandy is now known as rlandy|bbl | 22:38 | |
| fungi | clarkb: sure, was waiting to confirm you didn't feel a need to act on ianw or my comments | 22:40 |
| fungi | approved now | 22:40 |
| clarkb | ah ya I don't think we need to and this feels the most conservative (avoiding regressions in anything currently in path) | 22:41 |
| clarkb | and by appending they continue to be found in the old order first | 22:41 |
| fungi | fine by me, thanks! | 22:43 |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci master: WIP: containers periodic test https://review.openstack.org/475747 | 22:46 |
| openstackgerrit | Merged openstack-infra/project-config master: Set PATH when running configure mirror script https://review.openstack.org/476281 | 22:49 |
| *** Julien-zte has quit IRC | 22:50 | |
| *** jamesmcarthur has joined #openstack-infra | 22:51 | |
| *** Julien-zte has joined #openstack-infra | 22:51 | |
| clarkb | ok image builds started again | 22:52 |
| *** armax has quit IRC | 22:52 | |
| *** armax has joined #openstack-infra | 22:53 | |
| *** eranrom has joined #openstack-infra | 23:00 | |
| *** gongysh has joined #openstack-infra | 23:01 | |
| *** Apoorva_ has joined #openstack-infra | 23:01 | |
| *** dangers is now known as dangers_away | 23:02 | |
| *** gongysh has quit IRC | 23:03 | |
| *** Apoorva has quit IRC | 23:05 | |
| *** andreww has quit IRC | 23:05 | |
| *** rbrndt has quit IRC | 23:08 | |
| *** eranrom has quit IRC | 23:09 | |
| *** tosky has quit IRC | 23:09 | |
| *** wolverineav has quit IRC | 23:10 | |
| *** armax has quit IRC | 23:10 | |
| *** sdague has quit IRC | 23:11 | |
| *** jamesmcarthur has quit IRC | 23:16 | |
| *** jamesmcarthur has joined #openstack-infra | 23:17 | |
| ianw | Total 153 kB/s | 66 MB 07:19 | 23:19 |
| ianw | so ... we can either use external mirrors, and have the unreliability, or use internal mirrors, and have this issue of slow downloads ... | 23:20 |
| ianw | i.e. it doesn't feel like i've solved anything at all :/ | 23:21 |
| *** gomarivera has joined #openstack-infra | 23:22 | |
| fungi | ianw: i'm agreeing with clarkb that we might want to consider a nearby caching proxy to a central afs-published mirror rather than keeping afs caches nearby | 23:30 |
| fungi | worth seeing whether that improves performance for more distant mirrors anyway | 23:31 |
| openstackgerrit | Adam Gandelman proposed openstack-infra/zuul feature/zuulv3: Fix parsing of github PR url https://review.openstack.org/476286 | 23:31 |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci master: WIP: containers periodic test https://review.openstack.org/475747 | 23:31 |
| *** Julien-zte has quit IRC | 23:32 | |
| *** Apoorva_ has quit IRC | 23:33 | |
| *** gomarivera has quit IRC | 23:34 | |
| *** Apoorva has joined #openstack-infra | 23:34 | |
| ianw | fungi: what about just a squid proxy to itself? or is the problem total bandwidth in/out ... with afs updates dragging in too much data so nothing gets out? | 23:41 |
| *** s-shiono has joined #openstack-infra | 23:43 | |
| openstackgerrit | Ghanshyam Mann proposed openstack-infra/project-config master: Stop running puppet-* jobs on tempest unittests/doc patches https://review.openstack.org/476089 | 23:45 |
| *** jamesmcarthur has quit IRC | 23:45 | |
| *** salv-orlando has joined #openstack-infra | 23:47 | |
| fungi | ianw: right, trying to solve for afs (udp) cache update performance over high-latency links | 23:48 |
| *** makowals has joined #openstack-infra | 23:48 | |
| fungi | though i also suspect we haven't done all we can to profile and tune afs protocol performance either | 23:48 |
| openstackgerrit | Ghanshyam Mann proposed openstack-infra/project-config master: Stop running puppet-* jobs on tempest unittests/doc patches https://review.openstack.org/476089 | 23:49 |
| clarkb | right keep central afs mirror because that gievs us snapshots, redundant copies, and anonymous ro access | 23:49 |
| clarkb | but then have every remote mirror be a caching proxy to the central mirror to get better throughput across the atlantic | 23:49 |
| clarkb | and since we manage and publish mirrors only when they are working we get that bit of reliability out of it as well | 23:50 |
| clarkb | it should be easy to add, in fact why don't I add it right now | 23:50 |
| clarkb | pabelanger: any idea why we added a third vhost instead of reusing the existing vhost on the caching proxy? | 23:52 |
| clarkb | was that intentional or can we collapse them? | 23:52 |
| *** salv-orlando has quit IRC | 23:52 | |
| ianw | so a "vos release" invalidates the cache on the region mirrors, which then has to drag the data back? | 23:53 |
| ianw | even if the file hasn't changed? that was the bit i thought afs would figure out, but maybe not | 23:54 |
| clarkb | ianw: yes sort of. Best I can tell it invalidates the metadata associated wtih the file | 23:54 |
| clarkb | ianw: and that happens even if the vos release is a noop | 23:55 |
| clarkb | one thing we will need to check though is our really basic pypi mirror serving might not set cache headers properly | 23:57 |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Add caching reverse proxy for pypi mirror https://review.openstack.org/476289 | 23:58 |
| clarkb | something like that | 23:58 |
| clarkb | but will need testing once it goes in | 23:58 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!