| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: [wip] Collect syslogs from nodes in ansible tests https://review.openstack.org/624827 | 00:00 |
|---|---|---|
| *** mriedem has quit IRC | 00:01 | |
| *** wolverineav has quit IRC | 00:01 | |
| clarkb | jonher: the parent change needs review | 00:02 |
| jonher | you're right *doh* | 00:02 |
| *** wolverineav has joined #openstack-infra | 00:04 | |
| dmsimard | completely unrelated but I found that the "ansible_bios_version" fact shows "Ubuntu-1.8.2-1ubuntu1" for a CentOS virtual machine o_O | 00:08 |
| dmsimard | Is that because of DIB or something ? | 00:09 |
| dmsimard | example: http://logs.openstack.org/17/624817/3/check/openstack-infra-multinode-integration-centos-7/9fa335d/ara-report/host/efa6e950-fc2f-471e-a581-8a739f3f0635/ | 00:09 |
| dmsimard | comes from https://github.com/ansible/ansible/blob/d6d16fa619f81315de1378d5b7a9e00c0e6ced43/lib/ansible/module_utils/facts/hardware/linux.py#L284 apparently | 00:09 |
| *** wolverineav has quit IRC | 00:10 | |
| ianw | dmsimard: hrm, is that coming from the executor? | 00:11 |
| *** jamesmcarthur has joined #openstack-infra | 00:11 | |
| ianw | no everything else looks from the node | 00:12 |
| *** witek_ has joined #openstack-infra | 00:13 | |
| dmsimard | no idea, I don't think it impacts anything, it was just weird | 00:13 |
| *** witek has quit IRC | 00:13 | |
| ianw | # cat /sys/devices/virtual/dmi/id/bios_vendor | 00:14 |
| ianw | SeaBIOS | 00:14 |
| ianw | # cat /sys/devices/virtual/dmi/id/sys_vendor | 00:14 |
| ianw | OpenStack Foundation | 00:14 |
| ianw | does this leak through from the host? | 00:14 |
| dmsimard | oh, maybe | 00:15 |
| *** hwoarang has quit IRC | 00:15 | |
| *** jamesmcarthur has quit IRC | 00:15 | |
| ianw | i think all that comes from your libvirt/kvm setup underneath | 00:16 |
| *** hwoarang has joined #openstack-infra | 00:17 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul-jobs master: Vendor the RDO repository configuration for installing OVS https://review.openstack.org/624817 | 00:18 |
| *** bobh has joined #openstack-infra | 00:19 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: [wip] Collect syslogs from nodes in ansible tests https://review.openstack.org/624827 | 00:28 |
| openstackgerrit | MarcH proposed openstack-infra/git-review master: Fix wrong and misleading "using default: None" in --verbose mode https://review.openstack.org/624840 | 00:38 |
| rm_work | is there a guide somewhere to all of the magic-vars that devstack exposes? <_< | 00:46 |
| rm_work | looking for something like ... the port that apache uses to expose the services | 00:47 |
| *** Swami has quit IRC | 00:48 | |
| *** wolverineav has joined #openstack-infra | 00:50 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: [wip] Collect syslogs from nodes in ansible tests https://review.openstack.org/624827 | 00:52 |
| *** dklyle has joined #openstack-infra | 00:53 | |
| *** wolverineav has quit IRC | 00:54 | |
| *** xarses_ has quit IRC | 00:54 | |
| *** tosky has quit IRC | 00:55 | |
| *** xarses has joined #openstack-infra | 00:56 | |
| ianw | rm_work: use the source, luke ... sorry not much of an answer but the only thing that would be accurate | 00:59 |
| rm_work | yeah, did some grepping | 01:00 |
| rm_work | i would love to assume if i didn't find it yet, it doesn't exist | 01:00 |
| rm_work | but i find that to be an unreliable metric :P | 01:00 |
| ianw | there's various SERVICE_PORT things, is that what you mean? | 01:01 |
| rm_work | hmmm maybe | 01:02 |
| rm_work | i'll dig more, thanks | 01:03 |
| *** gyee has quit IRC | 01:08 | |
| *** dave-mccowan has joined #openstack-infra | 01:17 | |
| *** diablo_rojo has quit IRC | 01:25 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: [wip] Collect syslogs from nodes in ansible tests https://review.openstack.org/624827 | 01:27 |
| *** jamesmcarthur has joined #openstack-infra | 01:30 | |
| *** dkehn has joined #openstack-infra | 01:43 | |
| *** jamesmcarthur has quit IRC | 01:57 | |
| *** mrsoul has joined #openstack-infra | 02:05 | |
| *** ssbarnea|rover has quit IRC | 02:06 | |
| *** xarses_ has joined #openstack-infra | 02:07 | |
| *** xarses has quit IRC | 02:10 | |
| openstackgerrit | Merged openstack-dev/hacking master: Fix coverage job https://review.openstack.org/624699 | 02:10 |
| *** tinwood has quit IRC | 02:10 | |
| *** tinwood has joined #openstack-infra | 02:11 | |
| ianw | clarkb: looks pretty good to start with http://logs.openstack.org/27/624827/8/check/system-config-run-base-ansible-devel/5b0bee3/hosts/ | 02:12 |
| *** dave-mccowan has quit IRC | 02:14 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Collect syslogs from nodes in ansible tests https://review.openstack.org/624827 | 02:15 |
| *** yamamoto has quit IRC | 02:21 | |
| *** rkukura has quit IRC | 02:23 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Set iptables forward drop by default https://review.openstack.org/624501 | 02:28 |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Collect syslogs from nodes in ansible tests https://review.openstack.org/624827 | 02:28 |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Import install-docker role https://review.openstack.org/605585 | 02:28 |
| *** wolverineav has joined #openstack-infra | 02:28 | |
| *** jamesmcarthur has joined #openstack-infra | 02:29 | |
| *** wolverineav has quit IRC | 02:31 | |
| *** wolverineav has joined #openstack-infra | 02:32 | |
| *** bobh has quit IRC | 02:37 | |
| *** bobh has joined #openstack-infra | 02:38 | |
| *** bobh has quit IRC | 02:42 | |
| *** jamesmcarthur has quit IRC | 02:44 | |
| *** hongbin has joined #openstack-infra | 02:45 | |
| *** wolverineav has quit IRC | 02:45 | |
| *** jamesmcarthur has joined #openstack-infra | 02:51 | |
| *** gagehugo has quit IRC | 02:55 | |
| *** jamesmcarthur has quit IRC | 02:57 | |
| *** jamesmcarthur has joined #openstack-infra | 02:57 | |
| *** yamamoto has joined #openstack-infra | 02:59 | |
| *** jamesmcarthur has quit IRC | 03:03 | |
| *** jamesmcarthur has joined #openstack-infra | 03:08 | |
| *** armax has quit IRC | 03:13 | |
| *** armax has joined #openstack-infra | 03:13 | |
| *** apetrich has quit IRC | 03:15 | |
| *** armax has quit IRC | 03:18 | |
| *** auristor has quit IRC | 03:19 | |
| *** dklyle has quit IRC | 03:22 | |
| *** jamesmcarthur has quit IRC | 03:24 | |
| *** ykarel has joined #openstack-infra | 03:26 | |
| *** jamesmcarthur has joined #openstack-infra | 03:27 | |
| *** wolverineav has joined #openstack-infra | 03:33 | |
| *** wolverineav has quit IRC | 03:37 | |
| *** bobh has joined #openstack-infra | 03:38 | |
| *** bobh has quit IRC | 03:39 | |
| *** bobh has joined #openstack-infra | 03:40 | |
| *** bobh has quit IRC | 03:40 | |
| *** wolverineav has joined #openstack-infra | 03:41 | |
| *** wolverineav has quit IRC | 03:41 | |
| *** bobh has joined #openstack-infra | 03:41 | |
| *** wolverineav has joined #openstack-infra | 03:41 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/nodepool master: Switch devstack jobs to Xenial https://review.openstack.org/624855 | 03:47 |
| *** auristor has joined #openstack-infra | 03:48 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/nodepool master: Switch devstack jobs to Xenial https://review.openstack.org/624855 | 03:48 |
| openstackgerrit | Ian Wienand proposed openstack-infra/nodepool master: Switch devstack jobs to Xenial https://review.openstack.org/624855 | 03:51 |
| ianw | clarkb: Dec 13 02:55:04 ubuntu-bionic-rax-iad-0001166870 dockerd[6991]: Error starting daemon: Error initializing network controller: Error creating default "bridge" network: could not find an available, non-overlapping IPv6 address pool among the defaults to assign to the network | 03:51 |
| ianw | http://logs.openstack.org/85/605585/18/check/system-config-run-docker/061bf5e/hosts/bionic-docker/logs/syslog | 03:51 |
| ianw | at least that's a clue | 03:51 |
| ianw | i'm guessing maybe ipv6 isn't safe on non-ipv6 hosts? | 03:52 |
| *** jamesmcarthur has quit IRC | 03:56 | |
| *** bobh has quit IRC | 03:57 | |
| *** MasterofJOKers has quit IRC | 04:01 | |
| *** jamesmcarthur has joined #openstack-infra | 04:03 | |
| *** gagehugo has joined #openstack-infra | 04:04 | |
| *** jamesmcarthur has quit IRC | 04:05 | |
| *** MasterofJOKers has joined #openstack-infra | 04:10 | |
| *** bhavikdbavishi has joined #openstack-infra | 04:16 | |
| *** hongbin has quit IRC | 04:20 | |
| *** mguiney has quit IRC | 04:23 | |
| *** armax has joined #openstack-infra | 04:26 | |
| *** armax has quit IRC | 04:30 | |
| *** armax has joined #openstack-infra | 04:32 | |
| *** yamamoto has quit IRC | 04:35 | |
| *** rkukura has joined #openstack-infra | 04:36 | |
| *** armax has quit IRC | 04:37 | |
| *** whoami-rajat has joined #openstack-infra | 04:46 | |
| *** udesale has joined #openstack-infra | 04:49 | |
| *** armax has joined #openstack-infra | 04:56 | |
| *** armax has quit IRC | 05:00 | |
| *** armax has joined #openstack-infra | 05:02 | |
| *** ykarel has quit IRC | 05:03 | |
| *** armax has quit IRC | 05:06 | |
| *** gagehugo has quit IRC | 05:12 | |
| *** ykarel has joined #openstack-infra | 05:20 | |
| *** wolverineav has quit IRC | 05:23 | |
| *** _alastor_ has joined #openstack-infra | 05:24 | |
| *** yamamoto has joined #openstack-infra | 05:32 | |
| *** haleyb has quit IRC | 05:34 | |
| *** wolverineav has joined #openstack-infra | 05:40 | |
| *** wolverineav has quit IRC | 05:51 | |
| *** armax has joined #openstack-infra | 05:56 | |
| *** armax has quit IRC | 06:00 | |
| *** bhavikdbavishi has quit IRC | 06:00 | |
| *** armax has joined #openstack-infra | 06:02 | |
| *** jamesmcarthur has joined #openstack-infra | 06:06 | |
| *** armax has quit IRC | 06:06 | |
| *** jamesmcarthur has quit IRC | 06:11 | |
| *** _alastor_ has quit IRC | 06:20 | |
| *** ccamacho has quit IRC | 06:28 | |
| *** _alastor_ has joined #openstack-infra | 06:40 | |
| *** slaweq has joined #openstack-infra | 06:57 | |
| *** quiquell|off is now known as quiquell | 06:59 | |
| *** bhavikdbavishi has joined #openstack-infra | 06:59 | |
| *** _alastor_ has quit IRC | 06:59 | |
| *** rlandy|bbl has quit IRC | 06:59 | |
| *** agopi has quit IRC | 07:07 | |
| *** rcernin has quit IRC | 07:09 | |
| *** bhavikdbavishi has quit IRC | 07:12 | |
| *** pcaruana has joined #openstack-infra | 07:12 | |
| *** yamamoto has quit IRC | 07:14 | |
| *** agopi has joined #openstack-infra | 07:15 | |
| *** armax has joined #openstack-infra | 07:23 | |
| *** apetrich has joined #openstack-infra | 07:27 | |
| *** armax has quit IRC | 07:28 | |
| *** wolverineav has joined #openstack-infra | 07:30 | |
| *** wolverineav has quit IRC | 07:34 | |
| *** _alastor_ has joined #openstack-infra | 07:35 | |
| *** _alastor_ has quit IRC | 07:40 | |
| *** aojea has joined #openstack-infra | 07:42 | |
| *** shardy has joined #openstack-infra | 07:55 | |
| *** gfidente has joined #openstack-infra | 07:55 | |
| *** gfidente is now known as gfidente|off | 07:56 | |
| *** armax has joined #openstack-infra | 07:56 | |
| *** rossella_s has joined #openstack-infra | 08:00 | |
| *** armax has quit IRC | 08:01 | |
| *** ginopc has joined #openstack-infra | 08:01 | |
| *** armax has joined #openstack-infra | 08:02 | |
| *** pgaxatte has joined #openstack-infra | 08:04 | |
| *** armax has quit IRC | 08:06 | |
| *** dklyle has joined #openstack-infra | 08:07 | |
| *** _alastor_ has joined #openstack-infra | 08:08 | |
| *** _alastor_ has quit IRC | 08:12 | |
| *** ccamacho has joined #openstack-infra | 08:22 | |
| *** imacdonn has quit IRC | 08:23 | |
| *** imacdonn has joined #openstack-infra | 08:24 | |
| *** dklyle has quit IRC | 08:27 | |
| *** yamamoto has joined #openstack-infra | 08:30 | |
| *** xek has joined #openstack-infra | 08:32 | |
| *** ykarel is now known as ykarel|lunch | 08:34 | |
| openstackgerrit | Yang Youseok proposed openstack/os-testr master: Add --failing option to run only failed testcases https://review.openstack.org/624893 | 08:46 |
| *** yamamoto has quit IRC | 08:47 | |
| *** yboaron_ has joined #openstack-infra | 08:50 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: refactor build page to use a reducer https://review.openstack.org/624894 | 08:55 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: refactor build page using a container https://review.openstack.org/624895 | 08:55 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add errors from the job-output to the build page https://review.openstack.org/624896 | 08:55 |
| *** jpena|off is now known as jpena | 09:04 | |
| *** jklare_ has joined #openstack-infra | 09:11 | |
| openstackgerrit | Yang Youseok proposed openstack/os-testr master: Add --failing option to run only failed testcases https://review.openstack.org/624893 | 09:12 |
| *** jklare has quit IRC | 09:12 | |
| *** jklare_ is now known as jklare | 09:12 | |
| *** jpich has joined #openstack-infra | 09:19 | |
| *** yamamoto has joined #openstack-infra | 09:26 | |
| frickler | corvus: just noticed that you have three held nodes 40d old, are these still needed? | 09:27 |
| *** ykarel|lunch is now known as ykarel | 09:33 | |
| *** ssbarnea|rover has joined #openstack-infra | 09:34 | |
| *** noama has joined #openstack-infra | 09:35 | |
| *** sshnaidm has quit IRC | 09:37 | |
| *** tosky has joined #openstack-infra | 09:37 | |
| *** e0ne has joined #openstack-infra | 09:41 | |
| *** markvoelker has joined #openstack-infra | 09:46 | |
| *** ssbarnea|rover has quit IRC | 09:53 | |
| *** bhavikdbavishi has joined #openstack-infra | 09:54 | |
| *** armax has joined #openstack-infra | 10:03 | |
| *** gfidente|off has quit IRC | 10:03 | |
| *** electrofelix has joined #openstack-infra | 10:04 | |
| *** gfidente has joined #openstack-infra | 10:07 | |
| *** armax has quit IRC | 10:08 | |
| *** smarcet has joined #openstack-infra | 10:16 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add errors from the job-output to the build page https://review.openstack.org/624896 | 10:16 |
| *** sshnaidm has joined #openstack-infra | 10:20 | |
| frickler | ianw: clarkb: did some googling on that docker ipv6 message and it seems that docker needs to be given a cidr to use explicitly when enabling ipv6, see https://docs.docker.com/v17.09/engine/userguide/networking/default_network/ipv6/ | 10:20 |
| frickler | as long as we are thinking about single node usage, we could probably generate an ULA prefix for that per node | 10:21 |
| frickler | doing anything that involves assigning public ipv6 addresses to containers directly would probably require us to implement support for that in openstack first | 10:24 |
| *** pbourke has quit IRC | 10:28 | |
| openstackgerrit | Jens Harbott (frickler) proposed openstack-infra/system-config master: DNM: Add fixed-cidr-v6 to docker daemon config https://review.openstack.org/624949 | 10:29 |
| frickler | ^^ that should show that the idea works, will look at auto-generating a prefix later | 10:29 |
| *** pbourke has joined #openstack-infra | 10:30 | |
| frickler | see also https://github.com/moby/moby/issues/29443 | 10:31 |
| *** bhavikdbavishi has quit IRC | 10:34 | |
| *** ssbarnea|rover has joined #openstack-infra | 10:43 | |
| *** sshnaidm has quit IRC | 10:50 | |
| *** smarcet has quit IRC | 10:50 | |
| *** dtantsur|afk is now known as dtantsur | 10:53 | |
| *** smarcet has joined #openstack-infra | 10:58 | |
| *** derekh has joined #openstack-infra | 10:59 | |
| *** wolverineav has joined #openstack-infra | 11:08 | |
| *** wolverineav has quit IRC | 11:13 | |
| *** yamamoto has quit IRC | 11:13 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/puppet-openstackid master: Updated script to support PHP7 https://review.openstack.org/624957 | 11:17 |
| *** dkehn has quit IRC | 11:17 | |
| *** _alastor_ has joined #openstack-infra | 11:26 | |
| *** _alastor_ has quit IRC | 11:30 | |
| *** armax has joined #openstack-infra | 11:34 | |
| *** armax has quit IRC | 11:39 | |
| frickler | ianw: the nodepool func tests are failing with "AttributeError: 'method' object has no attribute 'set'" during image upload, might that be related to the rackspace issues we were having? http://logs.openstack.org/55/624855/3/check/nodepool-functional-py35-src/0825cb2/controller/logs/screen-nodepool-builder.txt.gz#_Dec_13_05_24_07_006397 | 11:40 |
| *** tobias-urdin is now known as tobias-urdin_afk | 11:41 | |
| *** tobias-urdin_afk is now known as tobias-urdin | 11:42 | |
| *** tobias-urdin is now known as tobias-urdin_afk | 11:43 | |
| frickler | ah, no, that seems to be caused by recent dogpile.cache updates | 11:43 |
| *** yamamoto has joined #openstack-infra | 11:51 | |
| *** yamamoto has quit IRC | 11:51 | |
| *** yamamoto has joined #openstack-infra | 11:52 | |
| *** yamamoto has quit IRC | 11:56 | |
| *** armax has joined #openstack-infra | 11:59 | |
| openstackgerrit | Chris Dent proposed openstack-infra/project-config master: Add publish-to-pypi as os-resource-classes job https://review.openstack.org/624962 | 11:59 |
| *** yboaron_ has quit IRC | 12:04 | |
| *** dkehn has joined #openstack-infra | 12:04 | |
| *** yboaron_ has joined #openstack-infra | 12:04 | |
| *** yamamoto has joined #openstack-infra | 12:04 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/puppet-openstackid master: Updated script to support PHP7 https://review.openstack.org/624957 | 12:06 |
| *** armax has quit IRC | 12:08 | |
| *** sshnaidm has joined #openstack-infra | 12:12 | |
| *** smarcet has quit IRC | 12:19 | |
| *** bhavikdbavishi has joined #openstack-infra | 12:20 | |
| openstackgerrit | Sorin Sbarnea proposed openstack/diskimage-builder master: WIP: Avoid overriding system python packages on diskimage_builder https://review.openstack.org/624968 | 12:21 |
| *** eharney has quit IRC | 12:24 | |
| *** markvoelker has quit IRC | 12:24 | |
| *** udesale has quit IRC | 12:25 | |
| *** udesale has joined #openstack-infra | 12:26 | |
| *** armax has joined #openstack-infra | 12:35 | |
| *** armax has quit IRC | 12:39 | |
| *** lewo has joined #openstack-infra | 12:41 | |
| *** mriedem has joined #openstack-infra | 12:42 | |
| *** yamamoto has quit IRC | 12:49 | |
| openstackgerrit | Paul Bourke (pbourke) proposed openstack/diskimage-builder master: Add support for minimal Oracle Linux images https://review.openstack.org/624980 | 12:49 |
| *** e0ne has quit IRC | 12:51 | |
| *** yamamoto has joined #openstack-infra | 12:51 | |
| *** tobias-urdin_afk is now known as tobias-urdin | 12:53 | |
| *** yboaron_ has quit IRC | 13:00 | |
| *** jpena is now known as jpena|lunch | 13:01 | |
| *** rlandy has joined #openstack-infra | 13:03 | |
| *** bobh has joined #openstack-infra | 13:04 | |
| *** rh-jelabarre has quit IRC | 13:05 | |
| *** trown|outtypewww has quit IRC | 13:05 | |
| *** boden has joined #openstack-infra | 13:06 | |
| *** agopi has quit IRC | 13:09 | |
| *** markvoelker has joined #openstack-infra | 13:10 | |
| *** agopi has joined #openstack-infra | 13:11 | |
| *** agopi has quit IRC | 13:11 | |
| *** agopi has joined #openstack-infra | 13:11 | |
| *** eharney has joined #openstack-infra | 13:16 | |
| *** bobh has quit IRC | 13:18 | |
| *** yboaron_ has joined #openstack-infra | 13:19 | |
| *** bhavikdbavishi has quit IRC | 13:25 | |
| *** jamesmcarthur has joined #openstack-infra | 13:28 | |
| *** jamesmcarthur has quit IRC | 13:30 | |
| *** irclogbot_1 has quit IRC | 13:30 | |
| *** zul has quit IRC | 13:32 | |
| *** jamesmcarthur has joined #openstack-infra | 13:32 | |
| *** yboaron_ has quit IRC | 13:34 | |
| *** armax has joined #openstack-infra | 13:34 | |
| *** yboaron_ has joined #openstack-infra | 13:35 | |
| *** e0ne has joined #openstack-infra | 13:35 | |
| *** kgiusti has joined #openstack-infra | 13:38 | |
| *** armax has quit IRC | 13:39 | |
| *** jamesmcarthur has quit IRC | 13:40 | |
| *** zul has joined #openstack-infra | 13:41 | |
| *** adam_zhang has joined #openstack-infra | 13:44 | |
| *** irclogbot_1 has joined #openstack-infra | 13:45 | |
| *** e0ne has quit IRC | 13:49 | |
| *** apetrich has quit IRC | 13:49 | |
| *** apetrich has joined #openstack-infra | 13:50 | |
| *** irclogbot_1 has quit IRC | 13:52 | |
| *** tosky has quit IRC | 13:56 | |
| *** e0ne has joined #openstack-infra | 13:57 | |
| *** bobh has joined #openstack-infra | 13:58 | |
| *** yamamoto has quit IRC | 13:59 | |
| *** smarcet has joined #openstack-infra | 13:59 | |
| *** yamamoto has joined #openstack-infra | 14:00 | |
| *** tosky has joined #openstack-infra | 14:00 | |
| *** markvoelker has quit IRC | 14:03 | |
| *** yamamoto has quit IRC | 14:05 | |
| *** irclogbot_1 has joined #openstack-infra | 14:09 | |
| lennyb | Hi, I am getting strange error InternalError: (1071, u'Specified key was too long; max key length is 767 bytes') http://paste.openstack.org/show/737219/ for sqlalchemy ( working on master devstack ). Any ideas? | 14:11 |
| *** jamesmcarthur has joined #openstack-infra | 14:11 | |
| pabelanger | ssbarnea|rover: can you link a log to the most recent failure you are having with POST_FAILURE | 14:14 |
| ssbarnea|rover | pabelanger: most recent POST failure: https://zuul.openstack.org/stream/0dc38407c7c94063afc53ad126cfe932?logfile=console.log | 14:14 |
| pabelanger | ssbarnea|rover: which review is that from? | 14:15 |
| ssbarnea|rover | pabelanger: https://review.openstack.org/#/c/624381/3 -- ironic, this was an attempt to avoid that bug. | 14:16 |
| ssbarnea|rover | we had the assumption that "du" was the cause for the timeout. after this, I no longer believe that. | 14:16 |
| *** jamesmcarthur has quit IRC | 14:16 | |
| ssbarnea|rover | that is why I added the timeout with kill for du in it. | 14:16 |
| openstackgerrit | Thierry Carrez proposed openstack-infra/project-config master: Define release job for openstackclient https://review.openstack.org/625009 | 14:17 |
| pabelanger | ssbarnea|rover: right, you are getting POST_FAILURES because if this task fails, no other post-run playbooks can finish, and we have zuul report POST_FAILURE to help a user know the failure happens in post | 14:19 |
| pabelanger | ssbarnea|rover: to avoid that, you could you a block / rescue playbook and maybe ignore errors there | 14:19 |
| pabelanger | http://logs.openstack.org/81/624381/4/check/tripleo-ci-centos-7-containers-multinode-queens/0dc3840/ | 14:20 |
| pabelanger | seems to have failed in rax | 14:20 |
| pabelanger | how large is the log directory you are trying to run du on? | 14:20 |
| ssbarnea|rover | pabelanger: not sure about size but that task has a "|| true", it should not fail anyway. | 14:21 |
| ssbarnea|rover | but I have no problem on adding a block/resume. Are you trying to say that this is a bug in zuul where it remains stuck if last task in post is a failed one? | 14:22 |
| pabelanger | let me look at executor | 14:23 |
| pabelanger | but the post playbook is timing out | 14:23 |
| *** yboaron_ has quit IRC | 14:26 | |
| *** yboaron_ has joined #openstack-infra | 14:27 | |
| *** irclogbot_1 has quit IRC | 14:27 | |
| *** bobh has quit IRC | 14:27 | |
| SpamapS | I haven't ever seen it on recent OS's, but on CentOS 7 executors we occasionally saw Ansible get stuck because of SSH bugs. | 14:28 |
| *** trown|brb has joined #openstack-infra | 14:28 | |
| *** trown|brb is now known as trown | 14:29 | |
| pabelanger | ssbarnea|rover: http://paste.openstack.org/show/737224/ | 14:30 |
| pabelanger | that is what the executor sees | 14:30 |
| *** haleyb has joined #openstack-infra | 14:31 | |
| pabelanger | ansible task has hung it looks like | 14:31 |
| pabelanger | but line 21 is odd | 14:31 |
| *** jamesmcarthur has joined #openstack-infra | 14:31 | |
| pabelanger | ssbarnea|rover: http://logs.openstack.org/53/624953/1/check/tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades/607b39a/ara-report/file/25e424be-22ce-48e2-b4d5-1b02e9e06478/#line-32 | 14:32 |
| pabelanger | maybe clean up that task not to use if statements in bash | 14:32 |
| ssbarnea|rover | pabelanger: indeed. I wonder if we reached a bug in ansible and how it wrapes shell scripts when it executes it. | 14:32 |
| pabelanger | but using when | 14:32 |
| pabelanger | I'd make that task more ansible friendly | 14:32 |
| *** _alastor_ has joined #openstack-infra | 14:33 | |
| ssbarnea|rover | pabelanger: true that if by itself makes no sense to me. i am taking it out now. | 14:33 |
| ssbarnea|rover | i like bash, but in this case i see no reason to make it complicated. | 14:34 |
| pabelanger | ++ | 14:34 |
| *** irclogbot_1 has joined #openstack-infra | 14:34 | |
| ssbarnea|rover | i am curious who writes this collect_logs.sh script because is not in the same file. going to ask my folks | 14:35 |
| *** markvoelker has joined #openstack-infra | 14:36 | |
| *** _alastor_ has quit IRC | 14:38 | |
| ssbarnea|rover | pabelanger: is zuul also streaming command module or only shell? | 14:41 |
| *** priteau has joined #openstack-infra | 14:42 | |
| pabelanger | ssbarnea|rover: should be both | 14:42 |
| *** dayou has quit IRC | 14:43 | |
| *** yamamoto has joined #openstack-infra | 14:44 | |
| *** wolverineav has joined #openstack-infra | 14:45 | |
| *** albin_vass has joined #openstack-infra | 14:47 | |
| *** cdent has joined #openstack-infra | 14:47 | |
| cdent | Can I get a judgement call from some zuulish folk? gmann and I are having something of a disagreement over how and where to manage irrelevant files when it comes to templates. Discussion begins on https://review.openstack.org/#/c/625002/ | 14:48 |
| *** wolverineav has quit IRC | 14:49 | |
| cdent | my feeling is that we should have a "best but not ultimate" list on the template and just ride out the small number of misses by inheriting that | 14:49 |
| cdent | the stickler in this point is that if we want to add 'placement/tests/.*' we have to copy the whole list | 14:49 |
| cdent | I'd rather not, because the changes of a patch being solely a test change is low, and everything else is part of the "usual set" | 14:49 |
| cdent | (or should be) | 14:49 |
| *** adam_zhang has quit IRC | 14:50 | |
| ssbarnea|rover | cdent: replied. | 14:51 |
| cdent | thanks for the review ssbarnea|rover, that helps, but it doesn't get to the actual question about where the irrelevant files should be. I think they should _only_ be defined where the template is defined | 14:52 |
| ssbarnea|rover | in fact I had the impression that it was not possible to merge them with zuul anyway. | 14:53 |
| openstackgerrit | Paul Bourke (pbourke) proposed openstack/diskimage-builder master: Add support for minimal Oracle Linux images https://review.openstack.org/624980 | 14:53 |
| cdent | you have to stop using the template in order to set your own | 14:53 |
| cdent | which kind of kills the point of templates, to me | 14:53 |
| ssbarnea|rover | cdent: only overriding them was possible. which caused a lot of copy/paste issues. so unless is a very special case I would avoid adding any ignores. | 14:53 |
| cdent | thus me wanting to use a generic central thing | 14:53 |
| *** rpittau has joined #openstack-infra | 14:54 | |
| ssbarnea|rover | i see them as over-engineering, likely to cause more problems than solving. at the price of saving few resources you open a pandora box. | 14:54 |
| gmann | for me, copy/paste is all one time but not running integrated gates jobs where they are not needed obviously save infra resource | 14:55 |
| mriedem | cdent: ideally projects could override entries in irrelevant-files, but that's not the case as gmann pointed out | 14:55 |
| cdent | because it's a template | 14:55 |
| gmann | yeah, we had concluded the same in ML, let me search | 14:55 |
| mriedem | and yes the per-project irrelevant-files is not the most fun, but you do it once and re-use after that with anchors/aliases | 14:56 |
| cdent | _if_ we can figure out a way to make the template have a generic, but satisfactory, list, I think that would be the best outcome here | 14:56 |
| cdent | otherwise why have templates? | 14:56 |
| cdent | gmann is probably right that ^[^/]*/tests/.* is not safe, but we might be able to figure out something safe. might even be an ansible var that could somehow jump in there? | 14:57 |
| gmann | cdent: agree but that was something we did not find as perfect solution as far i remember from coreycb ML. | 14:57 |
| *** quiquell is now known as quiquell|lunch | 14:58 | |
| cdent | that figuring out something safe is why I've brought the issue here | 14:58 |
| mriedem | repeating from -placement: "to catch the most generic of cases - i bet there are lots of openstack projects that don't even realize they are running tempest jobs on unit test only changes, or changes to docs files outside the standard tree" | 14:58 |
| mriedem | "but b/c they push so few changes compared to the bigger projects, they just don't care" | 14:58 |
| *** panda|off is now known as panda | 14:58 | |
| cdent | mriedem: to me that is an argument for better/more flexible templates | 14:59 |
| *** gagehugo has joined #openstack-infra | 14:59 | |
| cdent | but if its not possible, then I guess them's the breaks | 15:00 |
| *** dayou has joined #openstack-infra | 15:00 | |
| mriedem | i don't disagree, | 15:01 |
| mriedem | but i also need to pick my battles to redeem some kind of sanity | 15:02 |
| mriedem | figuring out the template flexibation might be a good quest for the holidays | 15:02 |
| cdent | yeah, I wasn't aware this was going to be a big deal... | 15:02 |
| *** jamesmcarthur has quit IRC | 15:02 | |
| gmann | true but there was no best way to achieve that, it was trade-off on both side | 15:02 |
| cdent | i'll adjust my response on the current review | 15:02 |
| *** jamesmcarthur has joined #openstack-infra | 15:03 | |
| *** jamesmcarthur has quit IRC | 15:04 | |
| *** rh-jelabarre has joined #openstack-infra | 15:04 | |
| ssbarnea|rover | i seen that recently a "noop" job appeared on tripleo-ci list of jobs and I wonder why, in the past this was added by zuul only to projects without jobs defined. | 15:05 |
| mriedem | this is 2 days in a row where i'm unshowered several hours into work, i'm hoping this isn't becoming a trend | 15:05 |
| *** ykarel is now known as ykarel|away | 15:05 | |
| cdent | you still shower? congratulations | 15:06 |
| *** jamesmcarthur has joined #openstack-infra | 15:06 | |
| *** dpawlik has quit IRC | 15:06 | |
| *** dpawlik has joined #openstack-infra | 15:07 | |
| *** dpawlik has quit IRC | 15:07 | |
| gmann | cdent: http://lists.openstack.org/pipermail/openstack-dev/2018-June/131304.html | 15:09 |
| cdent | thanks. I've updated my response on the review, there are a couple of (very small) issues still | 15:09 |
| *** quiquell|lunch is now known as quiquell | 15:10 | |
| frickler | lennyb: I saw a similar error recently, but I don't remember where that was. also let's move to #openstack-qa with that | 15:12 |
| *** ykarel|away has quit IRC | 15:15 | |
| ssbarnea|rover | mriedem: can we workflow https://review.openstack.org/#/c/620950/ ? | 15:18 |
| *** jamesmcarthur has quit IRC | 15:19 | |
| mriedem | ssbarnea|rover: replied inline | 15:23 |
| mriedem | i don't see why we need the voting:1 filter on that | 15:23 |
| mriedem | what is the point? | 15:23 |
| mriedem | e-r will not process failures from non-voting jobs https://github.com/openstack-infra/elastic-recheck/blob/02d0651f29796d25af2940d557287dcc4bb58d09/elastic_recheck/elasticRecheck.py#L228 | 15:24 |
| *** graphene has joined #openstack-infra | 15:27 | |
| ssbarnea|rover | mriedem: thanks. i didn't know exact that to do. | 15:28 |
| fungi | mriedem: the benefits of working from home... the cats don't seem to care if i get too busy for a shower | 15:29 |
| fungi | ssbarnea|rover: in particular, non-voting results are skipped because they would taint the expectation that jobs are supposed to pass at least occasionally in the gate, and non-voting jobs (while we rarely have them in the gate pipeline) essentially taint the data set we're trying to analyze | 15:30 |
| *** jamesmcarthur has joined #openstack-infra | 15:32 | |
| *** armax has joined #openstack-infra | 15:34 | |
| ssbarnea|rover | fungi: that's what I told to my folks. "to avoid noise from NV jobs". | 15:34 |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck master: Identify DLRN builds fail intermittently (network errors) https://review.openstack.org/620950 | 15:35 |
| mriedem | let us never speak of this again | 15:35 |
| *** markvoelker has quit IRC | 15:38 | |
| *** andreykurilin has joined #openstack-infra | 15:39 | |
| andreykurilin | hi folks. I'm trying to customize my zuul role based on devstack_localrc.USE_PYTHON3 option, but I have an issue with that :( hope to find a help here | 15:40 |
| *** dpawlik has joined #openstack-infra | 15:40 | |
| ssbarnea|rover | mriedem: clarkb fungi : a more generic question regarding elastic-recheck: can we use it to measure/count/graph non-fatal issues? like retries, "Waiting on logger", maybe even ansible deprecation warnings? | 15:40 |
| andreykurilin | https://review.openstack.org/#/c/624409/5/tests/ci/playbooks/run-rally-task.yaml (originally, I tried "{{ devstack_localrc.get('USE_PYTHON3') or False }}", but simplified this temporary) | 15:40 |
| andreykurilin | and here is a usage of my local variable https://review.openstack.org/#/c/624409/3/tests/ci/playbooks/roles/snapshot-available-os-resources/tasks/main.yaml | 15:40 |
| andreykurilin | error: http://logs.openstack.org/09/624409/5/check/rally-task-simple-job/1971602/job-output.txt.gz#_2018-12-13_15_00_15_671511 | 15:41 |
| andreykurilin | ps: I do not need a local variable and would like to use directly the parent one from my roles, but it raised the same issue and I tried to play with accessing it | 15:41 |
| openstackgerrit | Sorin Sbarnea proposed openstack-infra/elastic-recheck master: Query: [primary] Waiting on logger https://review.openstack.org/622210 | 15:42 |
| openstackgerrit | Sorin Sbarnea proposed openstack-infra/elastic-recheck master: Query: [primary] Waiting on logger https://review.openstack.org/622210 | 15:42 |
| mriedem | ssbarnea|rover: it's not really the tool for that imo | 15:43 |
| mriedem | deprecation warnings should be rolled up elsewhere | 15:43 |
| mriedem | i know the legacy dsvm jobs used to at least capture deprecation warnings and dump them to a file | 15:43 |
| *** smarcet has quit IRC | 15:44 | |
| mriedem | http://logs.openstack.org/25/624825/1/check/legacy-grenade-dsvm-neutron-multinode-live-migration/9a0637c/logs/deprecations.txt.gz | 15:44 |
| *** dpawlik has quit IRC | 15:44 | |
| mriedem | no one looks at those however | 15:44 |
| ssbarnea|rover | mriedem: i think is a gray area because any of these warnings are things that can easily become fatal errors, each of them, including the annoying depreaction ones. | 15:44 |
| fungi | ssbarnea|rover: is that something it makes sense to provide feedback on via automated change comments? that's a big part of the reason for elastic-recheck | 15:44 |
| *** efried has joined #openstack-infra | 15:45 | |
| pabelanger | andreykurilin: you likely cannot access an ansible variable setup in a parent playbook, only if it is defined in the zuul.yaml file | 15:45 |
| fungi | basically the "we see your change failed these jobs for $reasons" | 15:45 |
| pabelanger | andreykurilin: external_bridge_mtu is the issue | 15:45 |
| andreykurilin | pabelanger: but I'm not doing anything with external_bridge_mtu . only trying to access devstack_localrc | 15:46 |
| andreykurilin | there is no place in my custom playbooks/roles where external_bridge_mtu is used | 15:47 |
| pabelanger | andreykurilin: that is not what the error is saying | 15:47 |
| mriedem | ssbarnea|rover: i think i'd rather see something like a periodic job that publishes a weekly report on deprecations | 15:47 |
| mriedem | and then teams/ptls subscribe to that and work on driving down deprecated code usage in their project | 15:48 |
| mriedem | before it's a problem | 15:48 |
| ssbarnea|rover | with warnings is quite hard because the only way to force people to fix them is to save then on succesful builds and compare them with previous succesfull execution and decide. if there is a regression, fail the build. AFAIK we have no support for this. | 15:48 |
| andreykurilin | pabelanger: ha, yes. here is a diff between patch-set #4 and #5 - https://review.openstack.org/#/c/624409/4..5/tests/ci/playbooks/run-rally-task.yaml | 15:48 |
| andreykurilin | pabelanger: http://logs.openstack.org/09/624409/4/check/rally-task-simple-job/5628544/job-output.txt.gz#_2018-12-13_13_27_26_735213 | 15:49 |
| andreykurilin | and here is another failure of playbook | 15:49 |
| andreykurilin | *of the same task | 15:49 |
| andreykurilin | the failure is expected. rally package is installed under python 2 and rally_run_under_python_3 is hardcoded to True. | 15:50 |
| andreykurilin | But this proves that the issue is in trying to access devstack_localrc | 15:50 |
| *** ccamacho has quit IRC | 15:52 | |
| *** bobh has joined #openstack-infra | 15:53 | |
| *** ykarel|away has joined #openstack-infra | 15:57 | |
| *** cdent has left #openstack-infra | 15:58 | |
| fungi | mriedem: ssbarnea|rover: yeah, integrating those into the openstack health dashboard seems like a better fit? | 16:01 |
| mriedem | i never look at that | 16:01 |
| mriedem | so idk | 16:01 |
| fungi | it can be (already has been maybe?) extended to track other things than just subunit | 16:02 |
| mriedem | you really kind of need a tool to bludgeon people over the head to care | 16:02 |
| mriedem | i seem to remember many years ago we tried making jobs fail if ERROR messages showed up in the logs | 16:02 |
| fungi | mriedem: it includes periodic jobs and has an interface to subscribe to specific sets (by project, job name, et cetera) | 16:02 |
| mriedem | and that didn't last long | 16:02 |
| *** albin_vass has quit IRC | 16:02 | |
| fungi | via rss | 16:02 |
| mriedem | fungi: then that seems like a better fit | 16:02 |
| openstackgerrit | Merged openstack-infra/elastic-recheck master: Identify DLRN builds fail intermittently (network errors) https://review.openstack.org/620950 | 16:03 |
| fungi | wouldn't be hard to send summaries of results from there to an ml | 16:03 |
| fungi | right, making services not error in their logs during job runs was "too hard" or something | 16:04 |
| fungi | (i think it had to do with separating out the negative tests?) | 16:04 |
| *** alexchadin has joined #openstack-infra | 16:07 | |
| *** jpena|lunch is now known as jpena | 16:08 | |
| *** morgan is now known as kmalloc | 16:09 | |
| *** alexchadin has quit IRC | 16:10 | |
| dmsimard | In the context of third party CI, would it be a bad idea to leverage -infra built nodepool images as static cloud-images in another nodepool ? For example, every 24h we'd download the latest image from nb01, add a layer on top to set up the right ssh keys and upload it with a "static" name to the cloud. | 16:11 |
| *** andreykurilin has quit IRC | 16:12 | |
| *** andreykurilin has joined #openstack-infra | 16:12 | |
| dmsimard | Looking to reduce some of the drift and maintenance burden of rebuilding almost the same images on RDO's end | 16:12 |
| pabelanger | dmsimard: those images will have wrong zuul key in them | 16:13 |
| pabelanger | so you wouldn't be able to ssh into as zuul user | 16:13 |
| dmsimard | pabelanger: right, hence "add a layer on top to set up the right ssh keys" | 16:13 |
| *** smarcet has joined #openstack-infra | 16:13 | |
| pabelanger | if you are doing that, you might as well keep nodepool building them from scratch? | 16:14 |
| pabelanger | another option might be to donate quota of cloud to openstack-infra and have it run the jobs, then online 1 place to manage it, over 2 | 16:14 |
| pabelanger | but, that would also mean other jobs running on it | 16:14 |
| pabelanger | However, I don't think we want our nodepool builders to become the source for 3pci systems to download images, give the way everything can be build from scratch today | 16:15 |
| fungi | dmsimard: how do you "layer" virtual machine images? | 16:16 |
| fungi | cow-mount a second block device over the rootfs? | 16:16 |
| dmsimard | fungi: libguestfs ? | 16:17 |
| fungi | i didn't realize libguestfs had a layering feature, but i'll admit i'm not super familiar with it as anything other than a mechanism for modifying the contents of filesystems on loopback-mounted block devices | 16:18 |
| dmsimard | pabelanger: I'm not convinced the idea is the best way forward, sort of trying to think outside the box :p | 16:18 |
| dmsimard | fungi: layer might not have been the right word, but you can edit images | 16:19 |
| pabelanger | dmsimard: what are the problems with the current system, technical debt? | 16:19 |
| pabelanger | fwiw: going though the same issue in ansible-network, we are wanting to spin up 3pci for vendors but want to keep the amount of things they have to do minimal. | 16:20 |
| pabelanger | looking at the idea of hosted 3pci, with executor zones | 16:21 |
| fungi | dmsimard: oh, so the question is really "does it make sense to arbitrarily edit and reuse our images downstream in third-party ci systems so they can avoid running diskimage-builder?" | 16:22 |
| *** eharney has quit IRC | 16:22 | |
| *** eharney_ has joined #openstack-infra | 16:22 | |
| *** quiquell is now known as quiquell|off | 16:22 | |
| fungi | we do make lots of assumptions in our images, and at some point the number of alterations you need to automate will result in you reimplementing the complexities of diskimage-builder just badly | 16:23 |
| dmsimard | fungi: We're essentially rebuilding the same image as upstream right now. It's sort of hacked together by pulling in project-config and putting it's path in DIB's ELEMENTS_PATH variable. | 16:23 |
| *** noama has quit IRC | 16:24 | |
| clarkb | they arent a public interface. Wedelete them if they are broken for us. We delete them when they eol, sometimes before they eol | 16:25 |
| clarkb | I think it would be more use at your own risk | 16:25 |
| clarkb | dib totally supports the take existing image and modify it workflow which means nodepool does too | 16:26 |
| clarkb | But I dont think our images are a good source for that | 16:27 |
| openstackgerrit | Paul Bourke (pbourke) proposed openstack/diskimage-builder master: Add oraclelinux support to pip-and-virtualenv-source-install https://review.openstack.org/625046 | 16:29 |
| *** e0ne has quit IRC | 16:30 | |
| *** e0ne has joined #openstack-infra | 16:30 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/puppet-openstackid master: Updated script to support PHP7 https://review.openstack.org/624957 | 16:31 |
| *** _alastor_ has joined #openstack-infra | 16:34 | |
| *** smarcet has quit IRC | 16:35 | |
| *** efried has quit IRC | 16:39 | |
| *** udesale has quit IRC | 16:41 | |
| *** _alastor_ has quit IRC | 16:43 | |
| *** tosky has quit IRC | 16:43 | |
| *** tosky has joined #openstack-infra | 16:44 | |
| ssbarnea|rover | pabelanger: fungi https://review.openstack.org/#/c/471078/ ? | 16:47 |
| fungi | ssbarnea|rover: yes, if another core reviewer approves that then it will take effect at the next gerrit restart | 16:49 |
| fungi | along with https://review.openstack.org/607699 which merged some 6 weeks ago | 16:50 |
| *** eharney_ is now known as eharney | 16:53 | |
| *** jpich has quit IRC | 16:54 | |
| *** jamesmcarthur has quit IRC | 16:59 | |
| *** graphene has quit IRC | 16:59 | |
| *** graphene has joined #openstack-infra | 17:01 | |
| *** bobh has quit IRC | 17:01 | |
| *** ginopc has quit IRC | 17:03 | |
| *** bobh has joined #openstack-infra | 17:05 | |
| *** ramishra has quit IRC | 17:06 | |
| openstackgerrit | Paul Bourke (pbourke) proposed openstack/diskimage-builder master: Add an element for oraclelinux7 https://review.openstack.org/625056 | 17:06 |
| *** bobh has quit IRC | 17:10 | |
| *** auristor has quit IRC | 17:10 | |
| *** hamzy_ has quit IRC | 17:10 | |
| openstackgerrit | Paul Bourke (pbourke) proposed openstack/diskimage-builder master: Add an element for oraclelinux7 https://review.openstack.org/625056 | 17:10 |
| *** hamzy_ has joined #openstack-infra | 17:10 | |
| clarkb | dmsimard: another thing to keep in mind is that dib should be doing pretty extensive caching so after the first image build the io cost should be fairly low | 17:11 |
| clarkb | dmsimard: mostly i don't want infra to have to make operational decisions that include figuring out if deleting an image will break half the third party CIs out there. Instead we provide the tooling to make each deployment self sufficient in that regard | 17:12 |
| *** bobh has joined #openstack-infra | 17:12 | |
| clarkb | ssbarnea|rover: mriedem I do think we (openstack) need to be better about cleaning up deprecation warnings. At the very least they very often make it hard to read through debug logs. I brought this up with the TC as a possible avenue for trying to improve quality by using that as a starting point | 17:13 |
| clarkb | Its possible we could have e-r generate a separate page of "count all the bad things that don't map to job failures" for tracking purposes? | 17:14 |
| clarkb | I agree I don't think they should go in the regular bug tracking dashboard as that should hopefully be more of a bunrdown chart of "fix these things to make the gate reliable" | 17:15 |
| mriedem | i agree deprecation warnings that people don't look at is a problem, | 17:15 |
| mriedem | i just don't know if e-r is the tool for that job | 17:15 |
| mriedem | e.g. we killed the moxstubout deprecation warnings in master and rocky in nova in the last week | 17:15 |
| mriedem | so smaller py3 console logs | 17:15 |
| fungi | exposing those via openstack-health seem reasonable, though i don't know exactly what the data mining implementation would look lik ethere | 17:16 |
| *** bobh has quit IRC | 17:16 | |
| clarkb | mriedem: \o/ | 17:16 |
| clarkb | mriedem: that particular warning makes our unittest logs huge in a variety of projects. I basically have to open log files in vim now | 17:16 |
| ssbarnea|rover | clarkb: clearly possible, and e-r would be the easiest way to measure them (minimal implementation efforth). i would make at least one attempt and see how it goes. | 17:16 |
| clarkb | mriedem: good to know nova has that claened up | 17:16 |
| fungi | tracking them in e-r when we're not going to report back to proposed changes about job failures for them seems like a poor fit | 17:16 |
| ssbarnea|rover | i am not proposing one query for each warning: more of one query for multiple/most warnings. | 17:17 |
| clarkb | ya I was thinking e-r in the sense that it knows how to query with set of queries but maybe its a new feature that is only information dashboard and not active pinging | 17:17 |
| clarkb | and e-r can render that how it renders the existing bugs, but maybe this isn't a bug? I dunno | 17:17 |
| ssbarnea|rover | having deprecation warnings is a bug by itself, IMHO | 17:18 |
| *** yboaron_ has quit IRC | 17:18 | |
| ssbarnea|rover | ... a trivial one, until the deprecation becomes an error. | 17:18 |
| clarkb | should be noted https://review.openstack.org/#/c/624817/ should fix the top bug on e-r now. waiting on https://review.openstack.org/#/c/624526/ to do extra testing that it works as expected | 17:18 |
| clarkb | mriedem: mostly fyi since thats a zuul/infra thing | 17:18 |
| fungi | having deprecation warnings appear in logs for jobs which aren't testing your deprecated features still work are probably bugs, yes | 17:18 |
| fungi | but to just say "deprecation warnings are bugs" misses a big part of what's going on | 17:19 |
| fungi | if glance deprecates a feature, for example, it probably wants to still test that the deprecated feature doesn't regress until it's eventually removed entirely. it may even want to test that exercising that deprecated feature logs a deprecation warning | 17:20 |
| clarkb | also we can track the number of warnings over tiem but if no one is fixing them its not super helpful. Unfortunately it feels like this is what the existing bug tracking on e-r is suffering from | 17:20 |
| clarkb | we basically know a bunch of stuff is unhappy, but aren't working to fix it (there has been a recent push \o/ thank you everyone for helping on that) | 17:20 |
| clarkb | fungi: yup | 17:21 |
| *** auristor has joined #openstack-infra | 17:21 | |
| clarkb | fungi: but there are whole classes of warnings like the mox related warnings that should just go away | 17:21 |
| fungi | sure, raising deprecation warnings in our dependencies is almost assuredly something we want to fix unless we need to still support older versions of those in some lts distro | 17:22 |
| clarkb | mriedem: if there is a mechanical method for cleaning up the mox stuff that might be worth a ml thread to spread that knowledge to other projects | 17:22 |
| clarkb | ? | 17:22 |
| mriedem | we ignore the warning :) | 17:22 |
| *** rpittau has quit IRC | 17:23 | |
| mriedem | https://review.openstack.org/#/c/623309/ | 17:23 |
| clarkb | bah | 17:23 |
| mriedem | dude, we have thousands of mox-based tests | 17:23 |
| mriedem | this is why we said mox removal in nova was a fools errand | 17:23 |
| clarkb | ya I know thats why I was wondering if tehre was an easy fix | 17:23 |
| mriedem | and we didn't take the community-wide goal seriously | 17:23 |
| clarkb | at least this way the logs are readable | 17:23 |
| mriedem | we tried https://review.openstack.org/#/c/579482/ | 17:23 |
| fungi | feh, goals? ;) | 17:23 |
| mriedem | but that seemed to hit some weird py3 issues | 17:24 |
| mriedem | point being we have bigger fish to fry than care about mox | 17:24 |
| fungi | at least while mox remains usable | 17:25 |
| clarkb | mriedem: do you know if we have an aggregate integrated gate fail rate published with any of the tooling? | 17:25 |
| clarkb | mriedem: I was thinking on top of watching specific bugs get fixed it would be nice to see the overall affect | 17:26 |
| clarkb | and I thought we had that somewhere, but I don't see it today. Maybe it is on the health dashboard? | 17:26 |
| fungi | oh, yeah, basically the sum of hits for all classified and unclassified failures together? | 17:26 |
| mriedem | http://status.openstack.org/elastic-recheck/data/others.html ? | 17:26 |
| clarkb | fungi: even simpler job_fails_in_gate/total_jobs | 17:26 |
| *** dtantsur is now known as dtantsur|afk | 17:27 | |
| dansmith | clarkb: mox->mock is very not mechanical and very high risk for missing stuff | 17:27 |
| clarkb | dansmith: roger | 17:27 |
| fungi | clarkb: that could be misleading since you don't know if one failure is being caused by a patch ahead of you | 17:27 |
| clarkb | fungi: good point, though if things were overall more realible we should see that trend in the right direction over time? | 17:27 |
| fungi | clarkb: i think we would want to graph the -2 votes from zuul on integrated queue repos | 17:27 |
| fungi | though even that's potentially misleading since it may vote -2 on your change because it depends on another change which got a -2 | 17:28 |
| *** derekh has quit IRC | 17:28 | |
| mriedem | clarkb: i don't really remember anything like that, we've always used grafana to track tempest job failure rates over time right? | 17:28 |
| clarkb | mriedem: ya we've done tracking of specific jobs, but I thought we had an overall gate failure percentage too (I could be misremembering) | 17:28 |
| mriedem | idk, mtrienish might remember if he were around | 17:29 |
| clarkb | prometheanfire: http://logs.openstack.org/58/624958/1/check/neutron-grenade/b62f985/logs/grenade.sh.txt#_2018-12-13_12_22_53_080 catches a legit issue, there is no 4.5.0 at https://pypi.org/simple/libvirt-python/ | 17:32 |
| clarkb | prometheanfire: is that something the requirements team can look into fixing? | 17:32 |
| prometheanfire | lol, wtf libvirt | 17:34 |
| *** arxcruz is now known as arxcruz|off|next | 17:34 | |
| *** arxcruz|off|next is now known as arxcruz|next_yea | 17:34 | |
| prometheanfire | https://libvirt.org/sources/python/ | 17:35 |
| prometheanfire | seems like they suck at publishing | 17:35 |
| prometheanfire | will bug a guy | 17:35 |
| clarkb | prometheanfire: I'm just going through e-r and wondering why pip is failing :P | 17:35 |
| prometheanfire | ya... | 17:35 |
| *** zul has quit IRC | 17:35 | |
| clarkb | (we actually catch so much legit stuff in e-r its great, if you want to find bugs to fix you can basically just pop things off the list there) | 17:35 |
| *** arxcruz|next_yea is now known as arxcruz|next_yr | 17:35 | |
| prometheanfire | wonder how it ever passed, I can only thing that they removed it | 17:36 |
| prometheanfire | or maybe we skipped it | 17:37 |
| clarkb | prometheanfire: its possible they deleted it, but by then we had already built a wheel for it or something | 17:39 |
| prometheanfire | true | 17:40 |
| *** ianychoi has quit IRC | 17:42 | |
| *** panda is now known as panda|off | 17:42 | |
| mriedem | how do i search for a specific test on http://status.openstack.org/openstack-health/#/ ? | 17:42 |
| mriedem | i can never figure out this interface | 17:42 |
| *** aojea has quit IRC | 17:42 | |
| mriedem | http://status.openstack.org/openstack-health/#/test/tempest.scenario.test_security_groups_basic_ops.TestSecurityGroupsBasicOps.test_cross_tenant_traffic ? | 17:42 |
| mriedem | i guess i just have to figure out how the url works | 17:43 |
| prometheanfire | 4.10 -> 4.8 -> 4.6 -> 4.4 -> 4.3 | 17:43 |
| clarkb | mriedem: I don't know | 17:43 |
| prometheanfire | clarkb: those are the releases we've had in the last 7 months | 17:43 |
| clarkb | prometheanfire: according to pypi? | 17:44 |
| prometheanfire | clarkb: no, what we've had in pypi, git digging | 17:44 |
| clarkb | constraints is set to 4.5.0 in that job log ? | 17:45 |
| clarkb | oh its grenade so may be on stable branch | 17:45 |
| clarkb | prometheanfire: ok I'm a derp its from https://review.openstack.org/#/c/622262/3/upper-constraints.txt | 17:46 |
| prometheanfire | depends-on https://review.openstack.org/#/c/622262/ | 17:46 |
| clarkb | which never merged | 17:46 |
| prometheanfire | :P | 17:46 |
| clarkb | fungi's earlier warning shows up to say hi | 17:47 |
| prometheanfire | that sets 4.6.0 though | 17:47 |
| clarkb | prometheanfire: patchset 3 set it to 4.5.0 which isn't valid | 17:47 |
| clarkb | then bumped to 4.6.0 | 17:47 |
| prometheanfire | ah | 17:47 |
| prometheanfire | yep | 17:47 |
| *** gfidente is now known as gfidente|afk | 17:48 | |
| clarkb | I shall restrict my looking to gate jobs to reduce chances of this in the future :) | 17:49 |
| prometheanfire | nah, was still useful in this case to bug upstream to publish all the stuff to pypi | 17:50 |
| fungi | clarkb: yep, proof that isn't an infrequent enough occurrence to simply ignore statistically | 17:50 |
| *** bobh has joined #openstack-infra | 17:51 | |
| prometheanfire | upstream has broken publishing, news at 11 | 17:52 |
| *** rkukura_ has joined #openstack-infra | 17:52 | |
| prometheanfire | not that we haven't had problems :P | 17:52 |
| *** trown is now known as trown|lunch | 17:54 | |
| dansmith | is there any known issue with the live console stuff in zuul? | 17:55 |
| *** rkukura has quit IRC | 17:55 | |
| *** rkukura_ is now known as rkukura | 17:55 | |
| dansmith | this job's latest console entry is about 10m after it started: http://zuul.openstack.org/stream/c399a0b9328844dbaecccc780e5d687a?logfile=console.log | 17:55 |
| dansmith | ah it just refreshed | 17:55 |
| fungi | dansmith: if a job reboots the node and doesn't restart the console stream process on it, that can die | 17:55 |
| fungi | other than that, it's been pretty stable once we got appropriate governors in place to stop all the oom'ing on our executors | 17:56 |
| dansmith | fungi: okay, I don't think that is what happened.. just now, tempest finished and the log jumped about 1h20m into the future | 17:56 |
| fungi | oh neat | 17:56 |
| fungi | and skipped all earlier log entries? was the job maybe hung for that long? | 17:57 |
| dansmith | no, they all seemed to come flooding in | 17:57 |
| Shrews | dansmith: if you happen to notice it again, try the finger output to see if it is the same. e.g., finger c399a0b9328844dbaecccc780e5d687a@zuul.openstack.org | 17:57 |
| dansmith | Shrews: okay | 17:57 |
| dansmith | Shrews: fwiw, that says "output truncated at 10XXX bytes" | 17:58 |
| fungi | does our websocket implementation maybe pause streaming into inactive browser tabs? | 17:58 |
| Shrews | dansmith: really? doesn't for me | 17:59 |
| fungi | that might depend on the finger client implementation | 17:59 |
| dansmith | 2018-12-13 16:26:30.945409 | controller | skip | 17:59 |
| dansmith | Output truncated at 100000 bytes... | 17:59 |
| dansmith | maybe it's a BSD finger thing | 17:59 |
| dansmith | what's the super awesome reason we're using finger for this? :) | 17:59 |
| dansmith | I expect it's a fantastic story | 17:59 |
| Shrews | dansmith: because it enables us to use finger for this :) | 17:59 |
| fungi | it's just a tcp text socket protocol that happens to support addressing | 17:59 |
| dansmith | fungi: heh okay | 18:00 |
| fungi | we needed a text socket, and a way to address specific streams | 18:00 |
| dansmith | Shrews: yeah, linux finger shows the whole thing | 18:00 |
| fungi | and that's basically finger, so why implement finger and call it by another name? | 18:00 |
| Shrews | when's the last legit reason you needed to run the finger command?? i mean, we're reviving old stuff. just wait until you see what we do with gopher | 18:00 |
| dansmith | fungi: rsh would be similar no? | 18:00 |
| *** sshnaidm is now known as sshnaidm|off | 18:01 | |
| dansmith | Shrews: not complaining :) | 18:01 |
| fungi | ignoring the authentication bits at least | 18:01 |
| fungi | yes rsh would have been fairly similar | 18:01 |
| *** wolverineav has joined #openstack-infra | 18:01 | |
| fungi | Shrews: i still want to set up an nntp gateway for our mailing lists. that would be a totally legit use of a classic protocol that doesn't see much love these days | 18:02 |
| pabelanger | ++ | 18:03 |
| Shrews | dansmith: fungi: also, finger protocol supports request forwarding which i guess we thought we might use at one time, but actually didn't | 18:05 |
| dansmith | ah yeah | 18:05 |
| fungi | well, we're considering abusing that part of the spec to be able to request streaming of other files from the node though, i thought? | 18:06 |
| clarkb | fungi: ya | 18:06 |
| *** Swami has joined #openstack-infra | 18:06 | |
| clarkb | so you can get the nova log while job is running | 18:06 |
| dansmith | Shrews: ah, you're another appstate alum eh? | 18:06 |
| Shrews | dansmith: yep | 18:07 |
| * Shrews feels googled | 18:07 | |
| * fungi went to school very near there | 18:07 | |
| * dansmith wipes his fingers clean | 18:07 | |
| fungi | i had tons of friends who went to asu too, and classmates who split their time between there and unc-a | 18:08 |
| dansmith | fungi: you went to unc-a? | 18:08 |
| fungi | yeah, grew up in that neck of the woods | 18:08 |
| dansmith | interesting | 18:09 |
| fungi | little ways south, down by the state line | 18:09 |
| *** gyee has joined #openstack-infra | 18:09 | |
| Shrews | dansmith: fun fact, i was in the first graduating class of CS grad students. we set the bar pretty low | 18:09 |
| *** rlandy is now known as rlandy|biab | 18:09 | |
| fungi | Shrews: that explains why i didn't know asu had a cs dept ;) | 18:09 |
| dansmith | heh, nice | 18:09 |
| dansmith | it seemed pretty well done by the time I got there | 18:10 |
| fungi | fwiw, back when i was in school the cs dept at unc-a was abysmal so i went for math theory | 18:10 |
| dansmith | Shrews: oh, cs grad you mean | 18:10 |
| Shrews | dansmith: yeah, grad school, not undergrad | 18:10 |
| dansmith | right right, I remember hearing that was new and small | 18:11 |
| dansmith | I went to grad at NCSU | 18:11 |
| *** e0ne has quit IRC | 18:11 | |
| fungi | my younger brother did cs at ncsu | 18:11 |
| *** smarcet has joined #openstack-infra | 18:12 | |
| fungi | i also lived in raleigh for 15 years and used to hang out or across the street from ncsu main campus all the time | 18:12 |
| fungi | hang out on or across the street | 18:12 |
| prometheanfire | clarkb: https://pypi.org/project/libvirt-python/#history uploaded | 18:12 |
| clarkb | neat | 18:14 |
| *** e0ne has joined #openstack-infra | 18:15 | |
| *** e0ne has quit IRC | 18:16 | |
| *** pgaxatte has quit IRC | 18:17 | |
| ssbarnea|rover | pabelanger: that POST_FAILURE seems to be impossible to avoid, check last results: https://review.openstack.org/#/c/624381/ -- you will still see one NV job from rdo having the same issue. | 18:19 |
| clarkb | fungi: looking at e-r (haven't checked proxy logs yet) urls like http://mirror.mtl01.inap.openstack.org/pypi/simple/pbr/ and http://mirror.mtl01.inap.openstack.org/pypi/simple/ don't appear to be cached. Not surprising given those are indexes. Any thoughts on how to convince apache that we might be ok with an hour ttl override on those artifacts anyway? | 18:19 |
| fungi | not off the top of my head, but i'm looking | 18:23 |
| fungi | i've done that with squid before but not tried it with apache | 18:23 |
| *** smarcet has quit IRC | 18:24 | |
| *** smarcet has joined #openstack-infra | 18:25 | |
| clarkb | ssbarnea|rover: is it possible that du -L is unsafe with symlink loops? | 18:25 |
| clarkb | ssbarnea|rover: just thinking out loud for reasons that it might go out to lunch and not come back | 18:25 |
| ssbarnea|rover | clarkb: yep, i read about this few minutes ago, still looking for safe alternatives. | 18:26 |
| ssbarnea|rover | i guess we can skip looking at symlinks. | 18:26 |
| clarkb | maybe start there and see if the problem goes away | 18:26 |
| clarkb | and if it does then look into ways to safely list things with symlinks that may loop | 18:27 |
| ssbarnea|rover | clarkb: is very weird because this happens randomly, probably with a 1/20 chance of happening | 18:27 |
| fungi | clarkb: CacheDefaultExpire, CacheIgnoreNoLastMod, CacheLastModifiedFactor, CacheMinExpire, CacheStoreExpired, CacheStorePrivate all look interesting | 18:28 |
| ssbarnea|rover | people are not even sure that the deadlock is at du level. | 18:28 |
| clarkb | ssbarnea|rover: ya it could happen later potentially | 18:28 |
| clarkb | fungi: let me see what headers a browser gets hitting those | 18:28 |
| *** armax has quit IRC | 18:28 | |
| fungi | clarkb: oh, and CacheIgnoreCacheControl | 18:28 |
| fungi | perhaps the most useful of all | 18:29 |
| fungi | https://httpd.apache.org/docs/2.4/mod/mod_cache.html | 18:29 |
| clarkb | fungi: Cache-Control: max-age=600 | 18:29 |
| clarkb | whereas on the actual package data its much larger | 18:29 |
| fungi | that says cache for 10 minutes after last modified i think? | 18:29 |
| fungi | is last modified set sometime in the past? | 18:30 |
| *** ykarel|away has quit IRC | 18:30 | |
| fungi | CacheStoreExpired may help if so | 18:30 |
| *** agopi_ has joined #openstack-infra | 18:30 | |
| clarkb | I don't see a last modified | 18:31 |
| fungi | in which case CacheIgnoreNoLastMod may be relevant | 18:31 |
| *** _alastor_ has joined #openstack-infra | 18:31 | |
| fungi | "Ordinarily, documents without a last-modified date are not cached." | 18:31 |
| fungi | along with CacheDefaultExpire | 18:32 |
| *** agopi has quit IRC | 18:33 | |
| fungi | we might need CacheIgnoreCacheControl in that case though since i don't know how that'll interact with the Cache-Control: max-age=600 | 18:33 |
| clarkb | reading w3 docs I think max-age may start counting from when the requset comes in? but expires is related to last modified? | 18:34 |
| clarkb | (I may have that backwards) | 18:34 |
| *** electrofelix has quit IRC | 18:34 | |
| fungi | well, regardless the apache docs claim they don't cache anything if there is no last modified time | 18:34 |
| clarkb | ya I think its from the time the request is made | 18:34 |
| clarkb | so maybe we just set CacheIgnoreNoLastMod to start | 18:34 |
| clarkb | then its a 10 minute cache refresh (it should HEAD the document to see if it has changed) | 18:35 |
| fungi | worth a shot. could set it manually on one of the mirrors with it in the emergency disable list and check it | 18:35 |
| fungi | may also want to set CacheDetailHeader on | 18:36 |
| fungi | while debugging | 18:36 |
| fungi | "an X-Cache-Detail header will be added to the response containing the detailed reason for a particular caching decision" | 18:36 |
| fungi | that sounds useful | 18:36 |
| fungi | X-Cache-Detail: "conditional cache hit: entity refreshed" from localhost | 18:37 |
| fungi | stuff like that | 18:37 |
| fungi | (taken from the docs) | 18:37 |
| *** wolverineav has quit IRC | 18:37 | |
| *** jpena is now known as jpena|off | 18:38 | |
| fungi | might even be worth only turning that on initially and repeating your tests | 18:38 |
| fungi | just so we know what its baseline reasoning for not caching is | 18:38 |
| clarkb | fungi: I think we actually log that in the server logs but not in the http responses | 18:39 |
| fungi | oh, that could be good enough then | 18:39 |
| fungi | looking | 18:39 |
| *** _alastor_ has quit IRC | 18:39 | |
| clarkb | "GET /pypi/simple/pbr/ HTTP/1.1" 200 10726 cache miss: attempting entity save "-" | 18:40 |
| fungi | cache miss: attempting entity save | 18:40 |
| fungi | yep, just found it myself | 18:41 |
| fungi | " | 18:41 |
| clarkb | "GET /pypi/simple/pbr/ HTTP/1.1" 200 10726 cache hit "-" | 18:41 |
| fungi | okay, so it is caching it sometimes? | 18:41 |
| clarkb | seems like it must be, just not very often | 18:42 |
| fungi | i wonder if sometimes it's returned differently | 18:42 |
| clarkb | and within seconds of each other it will go miss, hit, hit, miss, hit, miss | 18:43 |
| clarkb | well below the expected 10 minute cache period | 18:43 |
| *** wolverineav has joined #openstack-infra | 18:43 | |
| clarkb | fungi: pip 9.0.3 has significanlty more hits than 18.1 | 18:44 |
| *** electrofelix has joined #openstack-infra | 18:44 | |
| clarkb | possible it is the client influencing things? | 18:44 |
| fungi | could be newer pip requests no caching for indices? | 18:45 |
| clarkb | 1.6k hits with pip 9.0.3 vs 7 against 18.1 | 18:45 |
| clarkb | ya I wonder if new pip is being extra smart | 18:45 |
| *** shardy has quit IRC | 18:46 | |
| clarkb | should be able to tcpdump it against our http mirrors | 18:46 |
| fungi | yep | 18:47 |
| *** agopi_ is now known as agopi | 18:47 | |
| *** wolverineav has quit IRC | 18:48 | |
| fungi | and i agree, `wget -SO/dev/null https://pypi.org/simple/pbr/` from mirror01.mtl01.inap isn't showing any last modified time | 18:49 |
| *** electrofelix has quit IRC | 18:51 | |
| fungi | if i grab a specific package though i do see Last-Modified and Age headers in the response | 18:51 |
| clarkb | http://paste.openstack.org/show/737249/ | 18:51 |
| clarkb | 'Request header has "max_age" as 0, cache bypassed' is odd | 18:52 |
| clarkb | that says pip is told a max age of 0 but my browser gets 600 | 18:52 |
| clarkb | they could be returning different values based on the user agent | 18:53 |
| *** agopi_ has joined #openstack-infra | 18:54 | |
| *** trown|lunch is now known as trown | 18:54 | |
| *** agopi has quit IRC | 18:56 | |
| fungi | i'm finding anecdotes claiming CacheIgnoreCacheControl on will cause apache to ignore max-age even in client requests | 19:00 |
| pabelanger | dmsimard: will there be a new release of ARA to fix the deprecation warning for ara html? | 19:00 |
| openstackgerrit | Sorin Sbarnea proposed openstack-infra/elastic-recheck master: Include query results in graph https://review.openstack.org/260188 | 19:00 |
| clarkb | fungi: confirmed via tcpdump that pip sets Cache-Control: max-age=0 on its request for the index | 19:01 |
| clarkb | pip 18.1 that is | 19:01 |
| clarkb | fungi: what is annoying is we want to honor the actual server cache control value | 19:01 |
| clarkb | its just pip trying to be too smart and overriding us | 19:01 |
| fungi | right | 19:01 |
| clarkb | https://github.com/pypa/pip/blob/95f0b2a1fe721bc62a6f11197c35140e4a7450f2/src/pip/_internal/index.py#L141-L156 is the code | 19:02 |
| dmsimard | pabelanger: which one ? | 19:02 |
| clarkb | problem is when that conditional get fails because internet | 19:03 |
| clarkb | but maybe if this is the intended pip behavior we just live with it | 19:03 |
| pabelanger | dmsimard: CreateFile: https://object-storage-ca-ymq-1.vexxhost.net/v1/a0b4156a37f9453eb4ec7db5422272df/logs/4c/4c91f44a55cd37a5c80ddf7e147ba0252465195c/post/packet-ci-cloud-deploy/0aa4dbb/job-output.html#l9610 | 19:03 |
| clarkb | fungi: and maybe we start a discussion with pypi and bandersnatch about how we can effectively mirror pypi again | 19:05 |
| fungi | i'm asking in #pypa-dev if there's a way to stop setting that in requests, but going to check the source code first to make sure i'm not missing an undocumented option | 19:05 |
| clarkb | thanks | 19:06 |
| fungi | clarkb: looks like it was implemented in https://github.com/pypa/pip/pull/5791 | 19:07 |
| dmsimard | pabelanger: a fix for that actually just landed in master, I need to double check with another contributor for a patch in flight before cutting a release | 19:07 |
| pabelanger | dmsimard: ack, thanks! | 19:07 |
| ssbarnea|rover | mriedem: do you know how to run elastic-recheck locally? docs is close to zero on this. it seems that i need a config file, and there is no example to be found. should I reverse engineer it? | 19:08 |
| clarkb | fungi: fwiw a reasonable method to mirror pypi again is likely best since that is way more reliable than a proxy, but it would also be nice if tools didn't intentially break the ability to cache stuff :( | 19:09 |
| clarkb | ssbarnea|rover: there is a sample conf file in the root of the repository | 19:09 |
| *** graphene has quit IRC | 19:10 | |
| ssbarnea|rover | clarkb: oops, i missed it :D | 19:10 |
| mriedem | tox -e venv -- graph | 19:15 |
| mriedem | something like that, | 19:15 |
| mriedem | the entrypoints are in the setup.cfg | 19:15 |
| *** agopi_ is now known as agopi | 19:19 | |
| *** wolverineav has joined #openstack-infra | 19:19 | |
| clarkb | noop has snuck into some tripleo check and gate queues with other jobs running | 19:20 |
| fungi | yeah, someone mentioned that earlier and was asking about it | 19:20 |
| *** eharney has quit IRC | 19:20 | |
| *** wolverineav has quit IRC | 19:20 | |
| *** wolverineav has joined #openstack-infra | 19:20 | |
| fungi | i'm guessing one or more of their projects has added a template with a noop gate pipeline rather than no gate pipeline | 19:20 |
| *** rlandy|biab is now known as rlandy | 19:23 | |
| ssbarnea|rover | mriedem: what is the correct dataspource url to use? es_url points to a 404 url in the example/ | 19:30 |
| fungi | clarkb: okay, takeaways from #pypa-dev are that pip may not be a good tool for installing python packages in a ci system, we could propose a pr to make the cache-control header configurable (and likely have it rejected), but that perhaps maybe we ought to learn us up about some bandersnatch and run a right proper pypi mirror | 19:33 |
| mriedem | ssbarnea|rover: there are defaults in code, i don't think i ever change the actual config when generating the graph locally | 19:33 |
| AJaeger | config-core, could you go over project-config and openstack-zuul-jobs reviews, please? We have some backlog here... | 19:34 |
| clarkb | fungi: any response from them on crazy growth running mirror? | 19:35 |
| clarkb | Ii mean we know bandersnatch, its not sustainable as is | 19:35 |
| ssbarnea|rover | mriedem: that's because it has you user count path hardcoded in it. I tried to update them but it seems that it very silent and even if I put invalid values it does not complain. anyway I will find a way to get more info. once I learn hot to run it locally I will probably make some changes to make it easy to run locally without having to do anything like "make run". | 19:35 |
| clarkb | also I have zero interest in pushing PRs to pip. My most recent experience was probably the worst I've ever had with code review | 19:37 |
| fungi | clarkb: nope. the eventual answer (i'm guessing i'm talking to the lone denizen who's not afk at kubecon?) is likely to be install on a new enough platform that we can take advantage of the cleanup options in the latest version of bandersnatch | 19:37 |
| clarkb | if someone else wants to they should feel free | 19:37 |
| mriedem | ssbarnea|rover: i'm not really following you but ok. i just do: "tox -e venv -- elastic-recheck-graph ./queries" | 19:37 |
| clarkb | fungi: tge cleanup options arwnt really necessary anymore as we deleted the old afs volume :) | 19:37 |
| clarkb | I think its more about blacklisting daily uploads that would help | 19:38 |
| clarkb | but not all packages use pep440 versioning :/ | 19:38 |
| fungi | clarkb: yeah, i have a similar pr i gave up on because the maintainers wanted me to be able to tell them in absolute terms when all major distros would update their behaviors to stop doing something they've been doing for years so that a workaround i was proposing could be removed again. ideally, you know, within under a year | 19:38 |
| ssbarnea|rover | mriedem: thanks! it works. | 19:39 |
| *** diablo_rojo has joined #openstack-infra | 19:42 | |
| ssbarnea|rover | clarkb: you made me curious about pip PR stuff, because I had a similar issue myself. Initially it started with virtualenv where it went quite bad, and later with pip. And guess what it did involve the same individual from pypa team. Lucky for me the virtualenv maintenance was taken by someone new and my changes got in, more than an year later | 19:43 |
| clarkb | in my case I asked for feedback from a reviewer who didn't get back until 8 months later then got all mad that the thing I asked for more input on wasn'tfixed in the original suggested manner then suggested that there were a bunch of other problems with the PR that they wouldnt add to the PR beacuse they are secret | 19:43 |
| clarkb | so now I don't push code to pypa | 19:43 |
| *** sthussey has joined #openstack-infra | 19:43 | |
| *** pcaruana has quit IRC | 19:47 | |
| fungi | it was more of a "here are some relatively cosmetic problems with your change, fix those first and then i'll tell you why the solution is all wrong" | 19:51 |
| *** eernst has joined #openstack-infra | 19:51 | |
| clarkb | thinking about the bandersnatch option a bit more. I wonder if we want to try a rebuild on the existing host with our existing blacklist and see how big/small that ends up being | 19:53 |
| *** e0ne has joined #openstack-infra | 19:53 | |
| clarkb | or maybe I can sacrifice my local NAS to the effort | 19:54 |
| clarkb | though my network connection likely isn't as good as the mirror builders | 19:54 |
| clarkb | https://github.com/pypa/warehouse/issues/4288 seems relevant | 19:55 |
| *** tosky has quit IRC | 19:56 | |
| clarkb | specifically https://pypi.org/stats/ | 19:56 |
| *** tosky has joined #openstack-infra | 19:57 | |
| clarkb | now for some napkin math to see how big a mirror needs to be if excluding those large packages | 19:57 |
| fungi | corvus: mordred: clarkb: we've had a request from a pop-up effort around virtualization/container security across several communities, interested in maybe having us host a mailing list for them at lists.opendev.org... what's your opinion on that, and should i get working on the change to add a new mailman vhost for our shiny new domain? | 19:57 |
| fungi | they're apparently looking for some neutral territory for their cross-community discussions | 19:58 |
| *** e0ne has quit IRC | 19:58 | |
| clarkb | fungi: http://lists.openstack.org/pipermail/openstack-infra/2018-December/006246.html is I think related. My only concern doing it is that if we use OpenDev as the top level thing then people may be confused as to why these seemingly random opendev things are happening. That said its suppsoed to be a "neutral" brand to help accomodate this stuff so I think we should probably err on the side of being | 20:00 |
| clarkb | welcming and going for it | 20:00 |
| clarkb | Ok pypi total is 2.6TB according to that stats page. Napkin math says we get it down to 1TB if we exclude the top 100 packages | 20:00 |
| *** smarcet has quit IRC | 20:01 | |
| clarkb | its crazy how much bigger the tf-nightly-gpu pckage is compared to everything else | 20:02 |
| fungi | clarkb: yes, that ml thread | 20:04 |
| clarkb | skimming the list of the top one hundred packages by size we'd need some of them like numpy but the vast majority we should be able to exclude | 20:04 |
| *** wolverineav has quit IRC | 20:05 | |
| clarkb | maybe we continue to monitor pip as is through the proxy and if things get worse we can look at rebuilding bandersnatch mirror | 20:09 |
| * clarkb udpates the e-r bug | 20:10 | |
| *** wolverineav has joined #openstack-infra | 20:10 | |
| *** eernst has quit IRC | 20:11 | |
| *** wolverineav has quit IRC | 20:15 | |
| *** priteau has quit IRC | 20:15 | |
| *** kgiusti has left #openstack-infra | 20:15 | |
| clarkb | fungi: https://bugs.launchpad.net/openstack-gate/+bug/1449136/comments/6 that capture things sufficiently well? | 20:15 |
| openstack | Launchpad bug 1449136 in OpenStack-Gate "Pip fails to find distribution for package" [Undecided,New] | 20:15 |
| *** kgiusti has joined #openstack-infra | 20:16 | |
| clarkb | fungi: re the mailing list, maybe we go ahead and configure our mailman vhosting for opendev anyway (since I expect we want that in the near future for other things), then we can be ready to add the asked for list if there aren't objections? | 20:17 |
| fungi | clarkb: we could, i think, set CacheIgnoreCacheControl in the vhost as a workaround (maybe limit it to ^/simple/.*) | 20:17 |
| fungi | clarkb: yes, i was about to push up the straw man change to create the mailman domain | 20:18 |
| clarkb | fungi: ya that might be an option. is CacheIgnoreCacheControl able to be set specific to location? | 20:18 |
| fungi | clarkb: oh, i guess not. apache docs say "context: server config, virtual host" | 20:20 |
| fungi | so we could set it on all of the vhost if we wanted | 20:21 |
| clarkb | ya I think part of the problem with that is we want the really log max-age for the package data itself and shorter index age | 20:21 |
| fungi | "Warning: This directive will allow serving from the cache even if the client has requested that the document not be served from the cache. This might result in stale content being served." | 20:21 |
| clarkb | maybe we can say ~8 hours is good enough for both or something | 20:21 |
| fungi | reading the description, it's only ignoring requests which contain "Cache-Control: no-cache or Pragma: no-cache" headers | 20:22 |
| fungi | so it has nothing to do with whether those are in the responses | 20:22 |
| clarkb | ah | 20:22 |
| fungi | or "cache-control: whatever" i guess | 20:23 |
| *** wolverineav has joined #openstack-infra | 20:23 | |
| fungi | those were examples | 20:23 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Extract out common config parsing for ConfigPool https://review.openstack.org/621642 | 20:23 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Extract common config parsing for ProviderConfig https://review.openstack.org/625094 | 20:24 |
| fungi | but anyway, it's a directive to ignore client (request) supplied headers, not server (response) supplied headers | 20:24 |
| fungi | so probably safe to turn on anyway? | 20:24 |
| *** eharney has joined #openstack-infra | 20:24 | |
| clarkb | oh if that is the case it would work as we expect I think | 20:24 |
| clarkb | we want apache to honor the backend supplied cachecontrol header and ignore the pip client side header | 20:24 |
| *** wolverineav has quit IRC | 20:25 | |
| *** wolverineav has joined #openstack-infra | 20:25 | |
| clarkb | fungi: ya I think maybe we just try setting that on our mirrors. The 10 minute index cache is sufficiently short that we should have fresh data, but long enough that we should avoid making local cdn unhappy and reduce internet bound traffic | 20:25 |
| clarkb | I can push that up if you are working on the mailman vhost change | 20:28 |
| *** bobh has quit IRC | 20:28 | |
| fungi | i am | 20:30 |
| fungi | and about to start cooking dinner | 20:30 |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Set CacheIgnoreCacheControl on pypi proxy cache https://review.openstack.org/625095 | 20:32 |
| clarkb | there is the apache change | 20:32 |
| *** eernst has joined #openstack-infra | 20:33 | |
| *** diablo_rojo has quit IRC | 20:34 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config master: Add lists.opendev.org to Mailman https://review.openstack.org/625096 | 20:36 |
| fungi | and the opendev mailman change ^ | 20:36 |
| *** diablo_rojo has joined #openstack-infra | 20:36 | |
| clarkb | +2 thanks | 20:37 |
| fungi | i'll follow up to the ml thread | 20:37 |
| clarkb | between the proxy change and dmsimards fix for installing ovs for multinode bridge on centos, I think that addresses a good chunk of the infra related bugs we are tracking. The exception to that is the slow io we've seen that is more cloud specific | 20:41 |
| *** eernst has quit IRC | 20:41 | |
| clarkb | if we can get those two in and start confirming improvements I may get around to booting a new pbx tomorrow | 20:41 |
| *** diablo_rojo has quit IRC | 20:41 | |
| *** bauzas has quit IRC | 20:42 | |
| *** smarcet has joined #openstack-infra | 20:42 | |
| *** bauzas has joined #openstack-infra | 20:45 | |
| *** bobh has joined #openstack-infra | 20:48 | |
| *** bobh has quit IRC | 20:52 | |
| *** psachin has joined #openstack-infra | 20:52 | |
| *** jamesmcarthur has joined #openstack-infra | 20:52 | |
| *** lpetrut has joined #openstack-infra | 20:53 | |
| *** ianychoi has joined #openstack-infra | 20:59 | |
| *** mriedem has quit IRC | 21:02 | |
| *** wolverineav has quit IRC | 21:06 | |
| *** wolverineav has joined #openstack-infra | 21:06 | |
| *** wolverineav has quit IRC | 21:07 | |
| *** wolverineav has joined #openstack-infra | 21:07 | |
| *** markvoelker has joined #openstack-infra | 21:11 | |
| *** tpsilva has quit IRC | 21:12 | |
| *** xek has quit IRC | 21:26 | |
| *** mriedem has joined #openstack-infra | 21:26 | |
| *** psachin has quit IRC | 21:27 | |
| *** tobias-urdin has quit IRC | 21:32 | |
| clarkb | and now taking AJaeger's prompt and going through backlog of project-config and ozj | 21:32 |
| clarkb | AJaeger: for https://review.openstack.org/#/c/619235/2 how strongly do you feel about "dashboard" vs "grafana"? | 21:32 |
| clarkb | ssbarnea|rover: should https://review.openstack.org/#/c/620613/ be WIP until you are ready to move things? | 21:35 |
| *** rh-jelabarre has quit IRC | 21:35 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config master: base-test: Check that regional mirror is online https://review.openstack.org/620961 | 21:38 |
| clarkb | mordred: fungi ^ you have previously reviewed that change. I fixed the ansible syntax in it for pabelanger rather than -1 | 21:39 |
| clarkb | (I'm single core approving some of the simpler job adds and grafyaml changes) | 21:42 |
| *** kgiusti has left #openstack-infra | 21:44 | |
| ianw | mordred: are you on this dogpile thing? it's causing a pileup back through nodepool and dib :) i see some changes out there to requirements with a -1, will dig in about 1/2 hour or so | 21:47 |
| clarkb | ianw: I think shrews mentioned it | 21:48 |
| openstackgerrit | Merged openstack-infra/project-config master: kata-containers: re-enable Fedora job https://review.openstack.org/623043 | 21:49 |
| openstackgerrit | Merged openstack-infra/project-config master: import git-os-job source repo https://review.openstack.org/623023 | 21:49 |
| *** bobh has joined #openstack-infra | 21:54 | |
| openstackgerrit | Merged openstack-infra/project-config master: Separate out success/failure/timeout charts in grafana for OSA https://review.openstack.org/623341 | 21:56 |
| openstackgerrit | Merged openstack-infra/project-config master: Re-activate translation job for Trove https://review.openstack.org/623492 | 21:56 |
| clarkb | AJaeger: ^ I think I've managed to flush out a bunch of changes by approving them. Also left reviews on a few that I was less comfortable approving or needed -1s | 21:57 |
| clarkb | will look at ozj shortly | 21:57 |
| *** wolverineav has quit IRC | 21:57 | |
| openstackgerrit | Merged openstack-infra/project-config master: Neutron grafana update for co-gating section https://review.openstack.org/622418 | 21:57 |
| openstackgerrit | Merged openstack-infra/project-config master: Add new project called ku.stella https://review.openstack.org/623396 | 21:57 |
| *** wolverineav has joined #openstack-infra | 21:58 | |
| *** trown is now known as trown|outtypewww | 21:59 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add translation job for storyboard https://review.openstack.org/623508 | 21:59 |
| *** lpetrut has quit IRC | 22:00 | |
| *** bobh has quit IRC | 22:00 | |
| clarkb | prometheanfire: dhellmann: can the requirements and release teams weigh in on https://review.openstack.org/#/c/623229/2 I want to make sure there isn't something I'm missing there (otherwise I'll happily approve it) | 22:01 |
| openstackgerrit | Merged openstack-infra/project-config master: Change os-resource-classes and os-traits acl config to placement https://review.openstack.org/624387 | 22:01 |
| openstackgerrit | Merged openstack-infra/project-config master: Adding the netlify-sandbox project https://review.openstack.org/624523 | 22:01 |
| *** wolverineav has quit IRC | 22:02 | |
| prometheanfire | clarkb: reqs works with py36 | 22:03 |
| dhellmann | that change is going to touch a lot of projects, so at the very least it deserves a ML post to give teams a heads-up | 22:04 |
| dhellmann | I don't disagree that it's a good thing to do, though | 22:04 |
| clarkb | dhellmann: want to leave that comment or should I follow up with cdent? | 22:05 |
| *** dklyle has joined #openstack-infra | 22:05 | |
| *** markvoelker has quit IRC | 22:06 | |
| dhellmann | clarkb : I went ahead and left a comment | 22:06 |
| dhellmann | thanks for the heads-up | 22:06 |
| dhellmann | this is a good example of the type of change I'd like to see us pre-announce more often | 22:07 |
| dhellmann | we do a pretty good job of that, but it's always something to think about | 22:07 |
| *** eernst has joined #openstack-infra | 22:08 | |
| clarkb | dhellmann: is now a good time to approve https://review.openstack.org/#/c/614758/ too? | 22:08 |
| dhellmann | yeah, that removes a duplicate job so it shouldn't "break" anyone | 22:09 |
| dhellmann | and I think we already had a ML thread about that one | 22:09 |
| clarkb | you did | 22:09 |
| *** david-lyle has joined #openstack-infra | 22:09 | |
| clarkb | AJaeger: ok I think I got through a good chunk of the backlog there. Thank you for the reminder as I've been head down on debugging the high rate of test failure recently as well as opendev and docker and eventually these things fall off the list | 22:10 |
| *** smarcet has joined #openstack-infra | 22:11 | |
| *** dklyle has quit IRC | 22:12 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: use opensuse15 as generic name instead of opensuse150 https://review.openstack.org/619628 | 22:13 |
| clarkb | ianw: frickler: I'm catching up on ipv6 things now. Since we want to use the host network namespace anyway (and not have separate IPs for every container) maybe we can drop the ipv6 config from docker/daemon.json for simplicity now? | 22:14 |
| clarkb | we'll still be able to listen on ipv6 addrs configured on the host aiui | 22:15 |
| *** bobh has joined #openstack-infra | 22:15 | |
| clarkb | Added that comment to the docker role add change too | 22:16 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Add nodeset ubuntu-bionic-2-node for bionic multinode testing https://review.openstack.org/622613 | 22:16 |
| clarkb | fungi: are https://review.openstack.org/#/c/615658/1 and https://review.openstack.org/#/c/615659/2 safe to merge? I'm not fully caught up on the state of ansible/puppet on the wiki servers | 22:17 |
| *** boden has quit IRC | 22:18 | |
| clarkb | there are a bunch of futureparser changes behind those two that I expect are quite safe so if those two are good to go I think I can approve a large ish stack today | 22:18 |
| *** eernst has quit IRC | 22:18 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: remove the tag pipeline entry for release-notes-jobs https://review.openstack.org/614758 | 22:20 |
| fungi | clarkb: we're not currently puppeting wiki.o.o (a.k.a. wiki-upgrade-test.o.o) but are puppeting wiki-dev | 22:20 |
| fungi | so these will take effect on wiki-dev which would be great | 22:20 |
| clarkb | fungi: should I approve or would you like to? | 22:20 |
| fungi | or they'll break wiki-dev and we'll investigate, but either way no big deal | 22:21 |
| fungi | i can approve, sure | 22:21 |
| clarkb | thanks! | 22:21 |
| *** rcernin has joined #openstack-infra | 22:21 | |
| fungi | and done | 22:22 |
| clarkb | I'm still waiting on test results for the one change I want to approve today :( will have to get away with approving all the other things instead | 22:23 |
| *** david-lyle has quit IRC | 22:23 | |
| clarkb | 624526 sanity checks dmsimard's zuul-jobs ovs install for multinode bridge on centos | 22:24 |
| clarkb | we do have testing of that role as is, but figure double checking we don't berak the jobs already half broken on that is a good thing | 22:24 |
| clarkb | hrm actually found a bug in the puppet-4 stack in the logstash.o.o change | 22:27 |
| clarkb | I'll fix that once the wiki things are out of the way | 22:27 |
| corvus | fungi: lists.opendev wfm | 22:27 |
| *** wolverineav has joined #openstack-infra | 22:29 | |
| fungi | corvus: cool, i've proposed https://review.openstack.org/625096 to add a lists.opendev.org site | 22:29 |
| fungi | i can do a separate patch to zone-opendev.org to add the address records | 22:30 |
| *** smarcet has quit IRC | 22:30 | |
| clarkb | corvus: and thoughts of hosting some genericish mailing lists at that domain? That was jbryce's question on the infra mailing list | 22:30 |
| clarkb | I think they want something like rust-vmm@lists.opendev.org to be a home for the crosvm/firecracker cooperation coordination | 22:31 |
| openstackgerrit | Merged openstack-infra/project-config master: Rename neutron ryu jobs https://review.openstack.org/624814 | 22:31 |
| openstackgerrit | Merged openstack-infra/project-config master: Add publish-to-pypi as os-resource-classes job https://review.openstack.org/624962 | 22:32 |
| openstackgerrit | Merged openstack-infra/project-config master: base-test: Check that regional mirror is online https://review.openstack.org/620961 | 22:32 |
| openstackgerrit | Merged openstack-infra/project-config master: Define release job for openstackclient https://review.openstack.org/625009 | 22:32 |
| *** wolverineav has quit IRC | 22:34 | |
| *** lbragstad has quit IRC | 22:37 | |
| *** smarcet has joined #openstack-infra | 22:40 | |
| *** smarcet has quit IRC | 22:45 | |
| *** jamesmcarthur has quit IRC | 22:47 | |
| *** smarcet has joined #openstack-infra | 22:48 | |
| *** rlandy has quit IRC | 23:03 | |
| *** wolverineav has joined #openstack-infra | 23:04 | |
| *** jamesmcarthur has joined #openstack-infra | 23:04 | |
| *** jamesmcarthur has quit IRC | 23:04 | |
| *** jamesmcarthur has joined #openstack-infra | 23:05 | |
| *** jamesmcarthur has quit IRC | 23:05 | |
| *** wolverineav has quit IRC | 23:08 | |
| *** armax has joined #openstack-infra | 23:09 | |
| Shrews | ianw: re: dogpile, a change to requirements is up to limit the version, and kmalloc is looking into the dogpile.cache part of it | 23:12 |
| * kmalloc is looking into the dogpile side. | 23:12 | |
| openstackgerrit | Merged openstack-infra/system-config master: Turn on the future parser for wiki-dev.openstack.org https://review.openstack.org/615658 | 23:20 |
| openstackgerrit | Merged openstack-infra/system-config master: Turn on the future parser for wiki.openstack.org https://review.openstack.org/615659 | 23:20 |
| *** tosky has quit IRC | 23:27 | |
| *** wolverineav has joined #openstack-infra | 23:29 | |
| *** _alastor_ has joined #openstack-infra | 23:31 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Turn on the future parser for logstash.openstack.org https://review.openstack.org/615660 | 23:34 |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Turn on the future parser for subunit workers https://review.openstack.org/615661 | 23:34 |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Turn on the future parser for elasticsearch.openstack.org https://review.openstack.org/615665 | 23:34 |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Turn on future parser for one git worker https://review.openstack.org/615966 | 23:34 |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Turn on future parser for all git workers https://review.openstack.org/615967 | 23:34 |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Turn on future parser for git load balancer https://review.openstack.org/615968 | 23:34 |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Turn on future parser for mirror-update.o.o https://review.openstack.org/615991 | 23:34 |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Turn on the future parser for rax mirror https://review.openstack.org/615992 | 23:34 |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Turn on the future parser for all mirrors https://review.openstack.org/615993 | 23:34 |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Turn on the future parser for files.openstack.org https://review.openstack.org/615994 | 23:34 |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Turn on future parser for one zookeeper instance https://review.openstack.org/615995 | 23:34 |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Turn on future parser for all zookeeper instances https://review.openstack.org/615996 | 23:34 |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Turn on the future parser for one nameserver https://review.openstack.org/615998 | 23:34 |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Turn on the future parser for all nameservers https://review.openstack.org/615999 | 23:34 |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Turn on the future parser for master nameserver https://review.openstack.org/616000 | 23:34 |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Turn on the future parser for openstackid-dev https://review.openstack.org/616001 | 23:34 |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Turn on the future parser for openstackid https://review.openstack.org/616002 | 23:34 |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Turn on future parser for one nodepool launcher https://review.openstack.org/616288 | 23:34 |
| clarkb | cmurphy: fungi ^ I think I got that right | 23:34 |
| clarkb | and apologies if I didn't. Figured that fixing the minor issues and speeding up turnaroudn was probably desireable | 23:35 |
| cmurphy | thanks clarkb | 23:36 |
| clarkb | cmurphy: the give away was the zk01 -> zk[0-9]+ transition removing futureparser from the test | 23:38 |
| clarkb | it should've stayed | 23:38 |
| clarkb | so I fixed that and the other isntances of this I could find | 23:38 |
| clarkb | fungi: mordred if you are still around the bottom of that stack is basically ready other than CI testing. I'm happy to babysit it tomorrow morning we get reviews | 23:38 |
| cmurphy | oh yeah i guess i should have listened to that | 23:38 |
| *** jamesmcarthur has joined #openstack-infra | 23:39 | |
| *** slaweq has quit IRC | 23:40 | |
| *** mriedem has quit IRC | 23:41 | |
| *** eernst has joined #openstack-infra | 23:43 | |
| *** jamesmcarthur has quit IRC | 23:44 | |
| *** armax has quit IRC | 23:45 | |
| *** eernst has quit IRC | 23:48 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!