| *** xe1gyq has quit IRC | 00:00 | |
| wznoinsk | @electrofelix, hi there, a long shot but I'm using jjb 2.x and trying to create a trigger which will hold the job-name itself, so leaving something like 'recheck job_name_generated_from_templates_and_job_defs_here' as comment in gerrit would trigger only that job, can't seem to find a way to achieve that (can't find a property or xml var with job name) and don't know how to achieve it in any other way | 00:01 |
|---|---|---|
| *** iyamahat_ has quit IRC | 00:03 | |
| *** lbragstad has quit IRC | 00:03 | |
| *** jamesmcarthur has joined #openstack-infra | 00:04 | |
| *** bdodd is now known as bdodd|afk | 00:06 | |
| *** jamesmcarthur has quit IRC | 00:08 | |
| ianw | afs02 is rather unhappy and holding up release. i'm rebooting it as a first step | 00:09 |
| *** e0ne has joined #openstack-infra | 00:10 | |
| *** slaweq has joined #openstack-infra | 00:11 | |
| *** jamesmcarthur has joined #openstack-infra | 00:13 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Extend github testing using app auth https://review.openstack.org/563297 | 00:15 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Test parent-change-enqueued with github https://review.openstack.org/563242 | 00:15 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Optionally record created github clients in tests https://review.openstack.org/563402 | 00:15 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Limit search scope of getChangesDependingOn to tenant https://review.openstack.org/563196 | 00:15 |
| *** slaweq has quit IRC | 00:15 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Remove getPullBySha from FakeGithubConnection https://review.openstack.org/563416 | 00:16 |
| ianw | ok, that seems to have recovered the afs bits. the release now appears to be clearing the bad r/o copy on afs02 "Deleting extant RO_DONTUSE site on afs02.dfw.openstack.org" | 00:16 |
| *** rpioso is now known as rpioso|afk | 00:18 | |
| ianw | i wonder if the others need similar ... | 00:22 |
| *** e0ne has quit IRC | 00:25 | |
| *** diablo_rojo has quit IRC | 00:25 | |
| ianw | the answer is yes : http://paste.openstack.org/show/721847/ | 00:27 |
| ianw | i think we should run manual releases, because if things time-out during cron releases that probably makes things worse with locked volumes | 00:27 |
| *** edmondsw has quit IRC | 00:28 | |
| clarkb | ianw: I will defer to you and thank you for looking but unlikely those of us at the summit will be very helpful | 00:29 |
| *** pblaho has quit IRC | 00:29 | |
| *** arahal_ has quit IRC | 00:29 | |
| ianw | clarkb: i figured :) just echoing into here so there's some log of what's up | 00:30 |
| *** jamesmcarthur has quit IRC | 00:31 | |
| *** eumel8 has joined #openstack-infra | 00:31 | |
| *** jamesmcarthur has joined #openstack-infra | 00:32 | |
| *** pblaho has joined #openstack-infra | 00:33 | |
| *** diablo_rojo has joined #openstack-infra | 00:34 | |
| *** edmondsw has joined #openstack-infra | 00:35 | |
| *** jamesmcarthur has quit IRC | 00:36 | |
| *** gyee has quit IRC | 00:36 | |
| *** dhinesh has joined #openstack-infra | 00:37 | |
| *** edmondsw has quit IRC | 00:39 | |
| ianw | Starting ForwardMulti from 536871007 to 536871007 on afs02.dfw.openstack.org (entire volume). | 00:50 |
| ianw | Release failed: Possible communication failure | 00:50 |
| ianw | hmm, another new error | 00:50 |
| *** diablo_rojo has quit IRC | 00:51 | |
| *** hamzy has quit IRC | 00:51 | |
| *** jamesmcarthur has joined #openstack-infra | 00:56 | |
| *** baoli has joined #openstack-infra | 00:57 | |
| *** dave-mccowan has joined #openstack-infra | 00:59 | |
| *** jamesmcarthur has quit IRC | 01:01 | |
| *** rpittau has quit IRC | 01:01 | |
| *** dhinesh has quit IRC | 01:07 | |
| *** zhangfei has joined #openstack-infra | 01:07 | |
| *** markvoelker has joined #openstack-infra | 01:08 | |
| *** bdodd|afk is now known as bdodd | 01:09 | |
| *** jesslampe has quit IRC | 01:11 | |
| *** edmondsw has joined #openstack-infra | 01:11 | |
| *** slaweq has joined #openstack-infra | 01:11 | |
| *** slaweq has quit IRC | 01:15 | |
| *** edmondsw has quit IRC | 01:15 | |
| *** eumel8 has quit IRC | 01:16 | |
| ianw | ok, so for "vos listvldb" any volume that is marked as having an old r/o release on afs02 is now running a release in a screen on mirror-update with the lock for that volume held | 01:23 |
| ianw | now i guess, we wait | 01:27 |
| *** gcb has joined #openstack-infra | 01:28 | |
| *** jamesmcarthur has joined #openstack-infra | 01:35 | |
| *** baoli has quit IRC | 01:37 | |
| *** jamesmcarthur has quit IRC | 01:38 | |
| ianw | ok, not going awesome. several releases have failed, but it seems to have kicked of the salvager and /var/log/openafs/SalsrvLog has entries for the volumes. re-running the failed ones :/ | 01:39 |
| *** baoli has joined #openstack-infra | 01:40 | |
| ianw | if problems persist, i guess we zap the volume in question | 01:40 |
| corvus | yeah, could drop the bad replica and re-create | 01:46 |
| *** zhangfei has quit IRC | 02:20 | |
| *** dave-mccowan has quit IRC | 02:24 | |
| *** rlandy|rover|bbl is now known as rlandy|rover | 02:26 | |
| *** zhangfei has joined #openstack-infra | 02:32 | |
| *** bobh has joined #openstack-infra | 02:33 | |
| *** bobh has quit IRC | 02:34 | |
| *** bobh has joined #openstack-infra | 02:38 | |
| *** psachin has joined #openstack-infra | 02:44 | |
| *** ramishra has joined #openstack-infra | 02:45 | |
| *** gcb has quit IRC | 02:45 | |
| *** tpsilva has quit IRC | 03:00 | |
| *** baoli has quit IRC | 03:07 | |
| *** baoli has joined #openstack-infra | 03:09 | |
| ianw | corvus: yeah, essentially what happened : /dev/mapper/main-vicepa 4.0T 447G 3.6T 12% /vicepa | 03:13 |
| ianw | that's got to get back to about 3tb | 03:13 |
| *** baoli has quit IRC | 03:14 | |
| ianw | it's done 6gb in 2m 30s, so say 40mb/s-ish | 03:15 |
| ianw | (3.6*1024*1024 - (447 * 1024)) / 40 / 60 / 60 ~ 23h ... so yeah, tomorrow sometime is my rough guess | 03:17 |
| *** jogo has quit IRC | 03:24 | |
| *** rlandy|rover has quit IRC | 03:27 | |
| *** jogo has joined #openstack-infra | 03:31 | |
| *** eumel8 has joined #openstack-infra | 03:35 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Save and close stdout on exit https://review.openstack.org/570094 | 03:46 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Add log directory option to functional tests https://review.openstack.org/570095 | 03:46 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Rename output log files https://review.openstack.org/570096 | 03:46 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: WIP: native zuulv3 tests https://review.openstack.org/566779 | 03:46 |
| *** bobh has quit IRC | 03:53 | |
| *** hamzy has joined #openstack-infra | 03:54 | |
| *** rcernin has quit IRC | 03:58 | |
| *** hamzy has quit IRC | 03:59 | |
| *** hamzy has joined #openstack-infra | 04:01 | |
| auristor | hi ianw | 04:01 |
| *** germs has quit IRC | 04:05 | |
| ianw | auristor: hello! | 04:06 |
| auristor | did the storage backing vice partitions on afs02 disconnect and reattach in readonly mode? | 04:07 |
| auristor | anything in the VolserLog ? | 04:08 |
| *** udesale has joined #openstack-infra | 04:08 | |
| ianw | auristor: i don't believe it did; it is volume storage on rax's block device offering, and so it basically just disappeared on us in an unknown way | 04:09 |
| ianw | however, it looks like some recovery's aren't going as well as others :/ | 04:09 |
| ianw | the ubuntu mirror volume just died with -> http://paste.openstack.org/show/721850/ | 04:10 |
| auristor | I see four ForwardMulti operations taking place in parallel from afs01.dfw to afs02.dfw | 04:10 |
| auristor | what is in the VolserLog on afs02.dfw | 04:11 |
| ianw | auristor: for that ubuntu volume, grepping for it's id just gives -> http://paste.openstack.org/show/721851/ | 04:12 |
| *** zhangfei has quit IRC | 04:12 | |
| auristor | the volume id is not going to be in most of the interesting messages. Could you post anything since the reboot? | 04:14 |
| auristor | The reason I was actually seeking to find you tonight is to discuss openstack testing of kafs? | 04:16 |
| *** rcernin has joined #openstack-infra | 04:16 | |
| *** cshastri has joined #openstack-infra | 04:18 | |
| ianw | auristor: sure ... as in what we've mentioned before about load-balancing in or similar? | 04:20 |
| openstackgerrit | Merged openstack-infra/project-config master: Add system-required template for ansible-role-tripleo-keystone https://review.openstack.org/564823 | 04:20 |
| ianw | Starting ForwardMulti from 536870983 to 536870983 on afs02.dfw.openstack.org (entire volume). | 04:22 |
| ianw | Release failed: Possible communication failure | 04:22 |
| ianw | but it's still running | 04:22 |
| *** rcernin has quit IRC | 04:23 | |
| *** rcernin has joined #openstack-infra | 04:23 | |
| auristor | So wish openstack.org was running AuriStorFS :-( | 04:23 |
| *** pcaruana has joined #openstack-infra | 04:24 | |
| auristor | I'm not entirely sure about the timezones but does this log entry associate with the release failure? | 04:25 |
| auristor | Thu May 24 03:15:23 2018 1 Volser: ReadVnodes: IH_CREATE: File exists - restore aborted | 04:25 |
| ianw | i *think* so. the current time on the server is Thu May 24 04:25:31 UTC 2018 | 04:26 |
| auristor | Then about an hour later | 04:27 |
| auristor | Thu May 24 04:13:35 2018 1 Volser: Delete: volume 536870950 deleted | 04:27 |
| auristor | Thu May 24 04:13:35 2018 1 Volser: CreateVolume: volume 536870950 (mirror.ubuntu.readonly) created | 04:27 |
| openstackgerrit | Merged openstack-infra/project-config master: Import ansible-role-tripleo-modify-image https://review.openstack.org/568726 | 04:27 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: [rally] Remove old rally-cinder jobs https://review.openstack.org/562569 | 04:28 |
| auristor | which I believe coincides with this transaction | 04:28 |
| auristor | transaction: 51 created: Thu May 24 00:13:35 2018 | 04:28 |
| auristor | lastActiveTime: Thu May 24 00:13:35 2018 | 04:28 |
| auristor | volumeStatus: temporary outOfService | 04:28 |
| auristor | volume: 536870950 group: unknown partition: /vicepa procedure: Restore | 04:28 |
| auristor | calls: 1 | 04:28 |
| auristor | 0: unknown | 04:28 |
| auristor | dataRead: unknown lastReceiveTime: Thu May 24 00:25:54 2018 | 04:28 |
| auristor | dataSent: unknown lastSendTime: Thu May 24 00:25:54 2018 | 04:28 |
| auristor | error: unknown | 04:28 |
| *** pcaruana has quit IRC | 04:29 | |
| ianw | yes, i dropped the lock and cron has started another relase | 04:29 |
| auristor | which began 16 minutes ago | 04:29 |
| ianw | root 1152 1149 0 04:13 ? 00:00:00 vos release -v mirror.ubuntu | 04:29 |
| *** zhangfei has joined #openstack-infra | 04:29 | |
| ianw | 05/24/2018 03:15:23 dispatching child to salvage volume 536870950... | 04:30 |
| ianw | 05/24/2018 03:15:23 dispatching child to salvage volume 536870949... | 04:30 |
| ianw | so it looks like it noticed the corrupt volumes here, and started salvage | 04:30 |
| ianw | it's not really clear to me if that worked or not from the logs | 04:31 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Add ubuntu-bionic-arm64 nodeset https://review.openstack.org/567965 | 04:32 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Remove legacy-tempest-dsvm-multinode-live-migration job definition https://review.openstack.org/565613 | 04:32 |
| *** openstackgerrit has quit IRC | 04:34 | |
| *** d0ugal has quit IRC | 04:35 | |
| *** ykarel|afk has joined #openstack-infra | 04:35 | |
| mnaser | i wish i could help :( | 04:37 |
| ianw | so i think mirror.debian release is stuck. reating new volume 536870983 on replication site afs02.dfw.openstack.org ... Thu May 24 04:37:37 2018 trans 31 on volume 536870983 is older than 3300 seconds | 04:39 |
| ianw | transaction: 31 created: Thu May 24 01:35:28 2018 | 04:39 |
| ianw | lastActiveTime: Thu May 24 03:42:13 2018 | 04:39 |
| ianw | volumeStatus: | 04:39 |
| ianw | transactionFlags: delete | 04:39 |
| ianw | volume: 536870983 partition: /vicepa procedure: Restore | 04:39 |
| ianw | packetRead: 60408644 lastReceiveTime: Thu May 24 04:39:29 2018 | 04:39 |
| ianw | packetSend: 1 lastSendTime: Thu May 24 04:39:29 2018 | 04:39 |
| auristor | According to the SalvsrvLog output each of the volume groups that were salvaged lacked a volume link table. The salvager found 50407 vnodes on disk for the mirror.ubuntu.readonly volume. But there should be 402381 vnodes according to the RW volume. | 04:40 |
| *** openstackgerrit has joined #openstack-infra | 04:41 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Add Zuulv3 jobs for puppet functional testing https://review.openstack.org/565270 | 04:41 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Remove legacy-hacking-integration-nova https://review.openstack.org/564787 | 04:41 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Remove legacy-ubuntu-xenial-3-node https://review.openstack.org/565918 | 04:41 |
| ianw | auristor: i'm getting the feeling that is terminal for the readonly volume ... | 04:41 |
| *** janki has joined #openstack-infra | 04:42 | |
| *** dhinesh has joined #openstack-infra | 04:43 | |
| ianw | i'm going to kill this debian sync and see about delsite and re-addsite | 04:44 |
| *** aeng has quit IRC | 04:44 | |
| auristor | transaction 31 is dead. something marked it for deletion but its waiting for the associated incoming call to drop its reference count | 04:45 |
| auristor | volserver transaction reference counting and rx call linkage is one of the weaker areas of OpenAFS. try "vos endtrans afs01.dfw.openstack.org 406" and see if that terminates the call from afs01.dfw to afs02.dfw. | 04:49 |
| ianw | oh dear, i might have made things worse removing the debian r/o mirror | 04:49 |
| ianw | http://paste.openstack.org/show/721852/ | 04:49 |
| openstackgerrit | Merged openstack-infra/project-config master: Make oslo.service py35 job voting https://review.openstack.org/567078 | 04:50 |
| auristor | what command did you execute? | 04:50 |
| ianw | vos remsite -server afs02.dfw.openstack.org -partition /vicepa -id mirror.debian | 04:51 |
| auristor | vos remsite only updates the VLDB. It doesn't touch the contents of the vice partition | 04:51 |
| auristor | you can "vos addsite -server afs02.dfw.openstack.org -partition a -id mirror.debian" to restore it | 04:52 |
| auristor | If you want to remove the contents of the volume on afs02.dfw/a then use "vos zap". But first the existing transaction active on the volume must be terminated. | 04:53 |
| ianw | ok, thanks, so that's back | 04:54 |
| *** rcernin has quit IRC | 04:56 | |
| auristor | there are five "vos release" operations in flight at the moment between those two servers. | 04:56 |
| ianw | that's right | 04:57 |
| auristor | sadly, the lastSendTime and lastReceiveTime reported by vos status are not restricted to data packets but to any kind of packet which makes the values somewhat useless. | 04:57 |
| openstackgerrit | Merged openstack-infra/project-config master: Add permission for adding tags for slogging https://review.openstack.org/566760 | 04:58 |
| openstackgerrit | Merged openstack-infra/project-config master: Add missing directory for gentoo rclocal https://review.openstack.org/568957 | 04:58 |
| *** kjackal has joined #openstack-infra | 04:59 | |
| ianw | i am probably doing something else wrong, but removing or zapping the mirror.debian on afs02 doesn't seem to work -> http://paste.openstack.org/show/721853/ | 05:00 |
| ianw | i am wondering if this calls for -force | 05:01 |
| persia | If anyone happens to feel like anti-spamming channels, a current short-join noisy actor has been filling channels, and seems to be approaching the openstack channels. | 05:01 |
| auristor | the RW is not on that vice partition. remove mirror.debian.readonly | 05:01 |
| ianw | ohh, vos remove -server afs02.dfw.openstack.org -partition a -id mirror.debian.readonly | 05:02 |
| *** d0ugal has joined #openstack-infra | 05:02 | |
| ianw | VOLSER: volume is busy | 05:02 |
| auristor | but I expect that to fail because mirror.debian.readonly is in use by transaction 31 | 05:02 |
| ianw | ok, so maybe kill that transaction? | 05:03 |
| auristor | which is why I suggested trying to "vos endtrans" the matching transaction on the source server afs01.dfw | 05:03 |
| auristor | which I believe is | 05:03 |
| auristor | transaction: 96 created: Wed May 23 21:35:28 2018 | 05:03 |
| auristor | lastActiveTime: Wed May 23 21:35:28 2018 | 05:03 |
| auristor | attachFlags: busy | 05:03 |
| auristor | volume: 536870983 group: unknown partition: /vicepa procedure: ForwardMulti | 05:03 |
| auristor | calls: 0 | 05:03 |
| ianw | ahhh, right, ok, mathcing up the volume | 05:05 |
| ianw | i've ended that at it appears gone | 05:05 |
| auristor | Its not marked deleted but it didn't terminate the outgoing call | 05:06 |
| auristor | transaction: 96 created: Thu May 24 01:04:52 2018 | 05:06 |
| auristor | lastActiveTime: Thu May 24 01:04:52 2018 | 05:06 |
| auristor | attachFlags: busy | 05:06 |
| auristor | transactionFlags: delete | 05:06 |
| auristor | volume: 536870983 group: unknown partition: /vicepa procedure: ForwardMulti | 05:06 |
| auristor | calls: 0 | 05:06 |
| auristor | the incoming transaction on afs02.dfw was already marked deleted | 05:07 |
| auristor | transaction: 31 created: Wed May 23 23:42:13 2018 | 05:07 |
| auristor | lastActiveTime: Wed May 23 23:42:13 2018 | 05:07 |
| auristor | volumeStatus: temporary outOfService | 05:07 |
| auristor | transactionFlags: delete | 05:07 |
| auristor | volume: 536870983 group: unknown partition: /vicepa procedure: Restore | 05:07 |
| auristor | calls: 1 | 05:07 |
| auristor | 0: unknown | 05:07 |
| auristor | dataRead: unknown lastReceiveTime: Thu May 24 01:06:10 2018 | 05:07 |
| auristor | dataSent: unknown lastSendTime: Thu May 24 01:06:10 2018 | 05:07 |
| auristor | error: unknown | 05:07 |
| ianw | right, ok, i'm starting to see how that lines up now | 05:07 |
| ianw | so both are deleted, but stuck? | 05:08 |
| auristor | pretty much. | 05:08 |
| ianw | is it as this point we pretty much have to restart the servers? | 05:09 |
| auristor | since OpenAFS doesn't properly associate rx calls and transactions, ending a transaction does not (in many circumstances) error out the rx calls | 05:09 |
| *** d0ugal has quit IRC | 05:10 | |
| *** ccamacho has quit IRC | 05:10 | |
| auristor | another weakness of openafs: the volserver does not have a clean shutdown procedure. The bosserver simply shoots it in the head when a "bos restart" or "bos shutdown" is performed. As a result, terminating the volserver can leave the volumes for which there are active transactions in a corrupted state. | 05:11 |
| *** d0ugal has joined #openstack-infra | 05:11 | |
| ianw | at this point, i think that we can consider these r/o volumes on afs02 corrupted :/ | 05:12 |
| auristor | another weakness of openafs is poor thread safety in the volserver which can lead to data corruption if the number of open vnodes exceeds the size of the ihandle cache. | 05:12 |
| *** links has joined #openstack-infra | 05:13 | |
| auristor | what I would like to do at this point is try to terminate the calls. can you put a firewall rule in on afs02.dfw to block all traffic to/from afs01.dfw port 7005/udp ? | 05:13 |
| *** d0ugal_ has joined #openstack-infra | 05:13 | |
| auristor | then in two minutes after no ack ping packets are responded to the calls should terminate on both sides | 05:14 |
| ianw | auristor: how about i just stop all activity on the mirror updater, and restart the afs01 & afs02 servers. that will get us back to a fresh slate? | 05:15 |
| auristor | then the readonly volumes on afs02.dfw can be removed with "vos zap" and the firewall rule can be disabled | 05:15 |
| *** d0ugal has quit IRC | 05:15 | |
| auristor | restarting afs01.dfw should not be necessary | 05:16 |
| ianw | #status log mirror-update.o.o placed into emergency file, and cron is stopped on the host, pending recovery of several afs volumes | 05:17 |
| openstackstatus | ianw: finished logging | 05:17 |
| auristor | restarting afs02.dfw would be sufficient | 05:17 |
| auristor | the rx calls from afs01.dfw will terminate once afs02.dfw no longer recognizes them | 05:18 |
| ianw | mirror.debian mirror.fedora mirror.opensuse mirror.pypi mirror.ubuntu mirror.ubuntu-cloud mirror.ubuntu-ports | 05:18 |
| ianw | are the volumes out of sync right now | 05:18 |
| ianw | afs02.dfw is restarting | 05:19 |
| auristor | when afs02.dfw restarts, vos zap those volumes on afs02.dfw vicepa | 05:19 |
| auristor | afs02.dfw has restarted | 05:20 |
| ianw | auristor: would it be better to do a complete remove & add, or zap is ok? | 05:20 |
| auristor | just vos zap. "vos remove" will remove the site from the VLDB entry which you want to remain in place | 05:20 |
| *** lpetrut has joined #openstack-infra | 05:21 | |
| auristor | all transactions on afs01.dfw have ended | 05:22 |
| *** d0ugal__ has joined #openstack-infra | 05:22 | |
| ianw | alright, i just rebooted that for sanity. so we have no transactions | 05:22 |
| *** links has quit IRC | 05:24 | |
| *** d0ugal_ has quit IRC | 05:24 | |
| ianw | vos zap on the mirror.debian doesn't appear to work http://paste.openstack.org/show/721854/ | 05:24 |
| *** links has joined #openstack-infra | 05:25 | |
| ianw | oh, or i want readonly? | 05:25 |
| auristor | you want to zap the readonly volumes not the RW | 05:27 |
| ianw | Failed to start transaction on 536870983 | 05:27 |
| ianw | Volume not attached, does not exist, or not on line | 05:27 |
| ianw | i've done : vos remove -server afs02.dfw.openstack.org -partition a -id mirror.debian.readonly | 05:29 |
| auristor | vos listvol afs02.dfw doesn't show that volume as present | 05:30 |
| auristor | of the list you posted earlier only mirror.opensuse.readonly and mirror.ubuntu-cloud.readonly are present | 05:31 |
| ianw | well, i have "remove"d the mirror.debian r/o from afs02, and done a release. i will now add it back | 05:32 |
| ianw | Added replication site afs02.dfw.openstack.org /vicepa for volume mirror.debian | 05:32 |
| *** lbragstad has joined #openstack-infra | 05:32 | |
| ianw | i am now releasing that | 05:33 |
| ianw | mirror.fedora is showing itself as locked for some reason | 05:34 |
| ianw | i have unlocked it, and removed the r/o mirror on afs02, and am running a release | 05:35 |
| auristor | that isn't surprising since the fileservers were restarted and the prior "vos release" failed. | 05:35 |
| *** links has quit IRC | 05:35 | |
| *** links has joined #openstack-infra | 05:36 | |
| *** links has quit IRC | 05:36 | |
| ianw | ok, fedora now has an unreleased r/o mirror. i think i will do similar for the others, then queue up the releases to run in order | 05:36 |
| *** links has joined #openstack-infra | 05:36 | |
| *** markvoelker_ has joined #openstack-infra | 05:36 | |
| *** markvoelker has quit IRC | 05:37 | |
| *** udesale_ has joined #openstack-infra | 05:39 | |
| *** udesale has quit IRC | 05:42 | |
| *** links has quit IRC | 05:42 | |
| *** lpetrut has quit IRC | 05:43 | |
| ianw | right, unfortunately i started the debian one and don't want to kill it now | 05:45 |
| *** links has joined #openstack-infra | 05:46 | |
| *** iranzo has joined #openstack-infra | 05:47 | |
| ianw | the other mirrors that now have unpopulated r/o volumes on afs02 are : mirror.fedora mirror.pypi mirror.ubuntu mirror.ubuntu-ports | 05:48 |
| auristor | This is the rx connection from afs01.dfw. to afs02.dfw | 05:48 |
| auristor | Connection to host [23.253.73.143]:7005, Cuid b76d8ab8/a959c694 | 05:48 |
| auristor | serial 18300714, natMTU 1444, security index 2, client conn | 05:48 |
| auristor | rxkad: level clear | 05:48 |
| auristor | Received 0 bytes in 0 packets | 05:48 |
| auristor | Sent 80918784 bytes in 18256160 packets | 05:48 |
| auristor | call 0: # 1, state active, mode: sending, has_output_packets | 05:48 |
| auristor | call 1: # 0, state not initialized | 05:48 |
| auristor | call 2: # 0, state not initialized | 05:48 |
| auristor | call 3: # 0, state not initialized | 05:48 |
| ianw | is that good or bad? the only thing it should be doing right now is a "vos release" of mirror.debian to the newly created r/o mirror on afs02 | 05:49 |
| auristor | call 0 is the call from transaction 33 on afs01.dfw to transaction 31 on afs02.dfw | 05:49 |
| auristor | you can use that to monitor progress | 05:50 |
| ianw | auristor: ok ... from what i've seen, throwing 5 very large parallel vos releases at this is bad news | 05:50 |
| auristor | thread safety in the openafs volserver isn't good. try to keep the number of parallel releases to one or two. | 05:52 |
| auristor | good luck. its quite late here and I'm struggling to keep my eyes open | 05:56 |
| *** caphrim007 has quit IRC | 05:57 | |
| ianw | auristor: thank you as always. i think we have a plan ... i will write something up for infra list. i also have to go do an airport run, so will be afk for a while | 05:57 |
| *** pgadiya has joined #openstack-infra | 05:58 | |
| *** pgadiya has quit IRC | 05:58 | |
| *** dhinesh has quit IRC | 05:59 | |
| *** yamahata has joined #openstack-infra | 06:03 | |
| *** links has quit IRC | 06:03 | |
| *** links has joined #openstack-infra | 06:04 | |
| *** iyamahat has joined #openstack-infra | 06:05 | |
| *** gcb has joined #openstack-infra | 06:07 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack-infra/project-config master: Normalize projects.yaml https://review.openstack.org/570318 | 06:09 |
| *** jamesmcarthur has joined #openstack-infra | 06:09 | |
| *** dhajare has joined #openstack-infra | 06:09 | |
| *** quiquell|off is now known as quiquell | 06:15 | |
| *** links has quit IRC | 06:17 | |
| *** gcb has quit IRC | 06:20 | |
| *** links has joined #openstack-infra | 06:22 | |
| *** kjackal has quit IRC | 06:26 | |
| *** aojea has joined #openstack-infra | 06:29 | |
| *** jamielennox has joined #openstack-infra | 06:31 | |
| *** lpetrut has joined #openstack-infra | 06:33 | |
| *** udesale__ has joined #openstack-infra | 06:36 | |
| *** e0ne has joined #openstack-infra | 06:36 | |
| *** rpittau has joined #openstack-infra | 06:37 | |
| *** aojea has quit IRC | 06:38 | |
| *** lbragstad has quit IRC | 06:38 | |
| *** links has quit IRC | 06:38 | |
| *** links has joined #openstack-infra | 06:38 | |
| *** udesale_ has quit IRC | 06:38 | |
| *** eumel8 has quit IRC | 06:41 | |
| *** links has quit IRC | 06:42 | |
| *** rcernin has joined #openstack-infra | 06:43 | |
| *** e0ne has quit IRC | 06:43 | |
| *** lpetrut has quit IRC | 06:47 | |
| *** rpittau has quit IRC | 06:51 | |
| *** links has joined #openstack-infra | 06:52 | |
| *** _jbadiapa is now known as jbadiapa | 06:53 | |
| *** pcaruana has joined #openstack-infra | 06:54 | |
| *** kjackal has joined #openstack-infra | 06:57 | |
| *** tung_doan_ has joined #openstack-infra | 07:02 | |
| *** e0ne has joined #openstack-infra | 07:03 | |
| *** ccamacho has joined #openstack-infra | 07:03 | |
| *** ispp has joined #openstack-infra | 07:03 | |
| *** janki has quit IRC | 07:07 | |
| *** ccamacho has quit IRC | 07:07 | |
| *** ccamacho has joined #openstack-infra | 07:08 | |
| *** dhajare has quit IRC | 07:10 | |
| *** pcichy has joined #openstack-infra | 07:19 | |
| *** ispp has quit IRC | 07:20 | |
| *** ispp has joined #openstack-infra | 07:20 | |
| *** tesseract has joined #openstack-infra | 07:23 | |
| *** ispp has quit IRC | 07:27 | |
| *** ssbarnea_ has joined #openstack-infra | 07:28 | |
| *** dhajare has joined #openstack-infra | 07:29 | |
| *** ykarel|afk is now known as ykarel|lunch | 07:30 | |
| *** alexchadin has joined #openstack-infra | 07:32 | |
| *** jpena|off is now known as jpena | 07:37 | |
| *** aojea has joined #openstack-infra | 07:40 | |
| *** udesale__ is now known as udesale | 07:53 | |
| *** ispp has joined #openstack-infra | 07:55 | |
| *** jistr is now known as jistr|iview | 07:57 | |
| *** sshnaidm|off is now known as sshnaidm | 07:59 | |
| *** roman_g has joined #openstack-infra | 08:05 | |
| *** lpetrut has joined #openstack-infra | 08:05 | |
| *** janki has joined #openstack-infra | 08:09 | |
| *** VW has joined #openstack-infra | 08:10 | |
| *** ykarel|lunch is now known as ykarel | 08:12 | |
| *** iyamahat has quit IRC | 08:20 | |
| *** rcernin has quit IRC | 08:36 | |
| *** VW has quit IRC | 08:40 | |
| *** kjackal has quit IRC | 08:40 | |
| *** derekh has joined #openstack-infra | 08:42 | |
| *** amoralej|off is now known as amoralej | 08:55 | |
| *** roman_g has quit IRC | 08:56 | |
| *** roman_g has joined #openstack-infra | 08:57 | |
| *** zhangfei has quit IRC | 08:59 | |
| *** jistr|iview is now known as jistr | 09:07 | |
| *** ilpianista has quit IRC | 09:07 | |
| *** MasterofJOKers has quit IRC | 09:08 | |
| *** tdasilva has quit IRC | 09:08 | |
| *** toabctl has quit IRC | 09:08 | |
| *** zhangfei has joined #openstack-infra | 09:12 | |
| *** kjackal has joined #openstack-infra | 09:12 | |
| *** tung_doan_ has quit IRC | 09:14 | |
| *** tung_doan has quit IRC | 09:14 | |
| *** ilpianista has joined #openstack-infra | 09:20 | |
| *** wolverineav has joined #openstack-infra | 09:30 | |
| *** zoli is now known as zoli|lunch | 09:31 | |
| *** MasterofJOKers has joined #openstack-infra | 09:35 | |
| *** tobberydberg has quit IRC | 09:40 | |
| *** johanssone has quit IRC | 09:41 | |
| *** johanssone has joined #openstack-infra | 09:44 | |
| *** tobberydberg has joined #openstack-infra | 09:45 | |
| *** tdasilva has joined #openstack-infra | 09:47 | |
| *** jamesmcarthur has quit IRC | 09:49 | |
| *** alexchad_ has joined #openstack-infra | 09:49 | |
| *** jamesmcarthur has joined #openstack-infra | 09:50 | |
| *** alexchadin has quit IRC | 09:52 | |
| *** panda|off is now known as panda | 10:03 | |
| *** wolverineav has quit IRC | 10:07 | |
| *** wolverineav has joined #openstack-infra | 10:07 | |
| *** wolverin_ has joined #openstack-infra | 10:09 | |
| *** wolverineav has quit IRC | 10:09 | |
| *** wolverin_ has quit IRC | 10:10 | |
| *** wolverineav has joined #openstack-infra | 10:10 | |
| *** ispp has quit IRC | 10:12 | |
| *** alexchad_ has quit IRC | 10:14 | |
| ianw | well it looks like afs recovery is not going well at all :/ | 10:14 |
| *** alexchadin has joined #openstack-infra | 10:15 | |
| *** gfidente has joined #openstack-infra | 10:15 | |
| *** gfidente has joined #openstack-infra | 10:15 | |
| *** wolverineav has quit IRC | 10:15 | |
| openstackgerrit | Tung Doan proposed openstack-infra/project-config master: Add pep8, py27, and py35 test to apmec project https://review.openstack.org/570363 | 10:16 |
| openstackgerrit | Roman Gorshunov proposed openstack/diskimage-builder master: Allow to rebuild arbitrary images https://review.openstack.org/568697 | 10:20 |
| roman_g | Please, review https://review.openstack.org/#/c/568697/ | 10:26 |
| roman_g | Thank you! | 10:26 |
| roman_g | I've just did a rebase | 10:26 |
| *** shaner has quit IRC | 10:28 | |
| *** dtantsur|afk is now known as dtantsur | 10:30 | |
| *** shaner has joined #openstack-infra | 10:30 | |
| *** zoli|lunch is now known as zoli | 10:34 | |
| *** ramishra has quit IRC | 10:40 | |
| *** ssbarnea_ has quit IRC | 10:42 | |
| *** alexchadin has quit IRC | 10:53 | |
| *** alexchadin has joined #openstack-infra | 10:54 | |
| *** ispp has joined #openstack-infra | 10:54 | |
| *** alexchadin has quit IRC | 10:54 | |
| *** alexchadin has joined #openstack-infra | 10:54 | |
| *** alexchadin has quit IRC | 10:55 | |
| *** alexchadin has joined #openstack-infra | 10:55 | |
| *** alexchadin has quit IRC | 10:56 | |
| *** alexchadin has joined #openstack-infra | 10:56 | |
| *** alexchadin has quit IRC | 10:56 | |
| *** ramishra has joined #openstack-infra | 10:57 | |
| *** jpena is now known as jpena|lunch | 10:58 | |
| auristor | ianw: looks like mirror.fedora.readonly and mirror.ubuntu.readonly and mirror.ubuntu-cloud.readonly replicated. | 11:01 |
| ianw | auristor: yes fedora did seem to work. however others didn't ... i dropped a mail to openafs-devel | 11:04 |
| ianw | https://lists.openafs.org/pipermail/openafs-devel/2018-May/020491.html | 11:05 |
| ianw | mirror.ubuntu-ports has failed, and also mirror.pypi | 11:06 |
| ianw | mirror.debian is currently running | 11:06 |
| *** amoralej is now known as amoralej|off | 11:07 | |
| ianw | auristor: does a "vos remove" imply the same level of removal as "vos zap"?. i'm wondering if for ubuntu-ports/pypi if i need to zap the partially cloned volume and start again maybe? | 11:09 |
| *** jpena|lunch is now known as jpena|off | 11:10 | |
| *** rh-jelabarre has joined #openstack-infra | 11:11 | |
| auristor | the only difference between "vos remove" and "vos zap" is that "vos remove" also updates the VL entry in the location service | 11:13 |
| ianw | auristor: hmm, that's what i thought. that's why i'm not understanding how after a "vos remove" of the volume and then re-adding it, we still get errors on the clone | 11:14 |
| ianw | i mean, as far as i know, the r/w side on afs01 is fine, no errors are coming from there | 11:15 |
| auristor | please share the current logs | 11:20 |
| auristor | btw, the IPv6 address AAAA records for afsdb01.openstack.org and afsdb02.openstack.org were added back to DNS. | 11:28 |
| ianw | hmm, possibly that happened when they were upgraded to xenial? i'll take a todo note to remove them ... i don't have the access creds atm | 11:30 |
| ianw | so we've got a few more successful releases; now it's just -> mirror.pypi mirror.ubuntu mirror.ubuntu-ports | 11:31 |
| ianw | for reference the failures for ^ are in mirror-update.o.o:/root/release-failure.txt | 11:32 |
| *** udesale_ has joined #openstack-infra | 11:33 | |
| auristor | looks like mirror.ppyi.readonly wasn't properly zapped because of a missing link table. | 11:33 |
| ianw | hmm, i've just noticed that mirror.pypi is also at 100% of quota | 11:34 |
| ianw | mirror.pypi 1800000000 1799990694 100%<< 69% <<WARNING | 11:35 |
| *** udesale has quit IRC | 11:35 | |
| *** alexchadin has joined #openstack-infra | 11:36 | |
| *** zhangfei has quit IRC | 11:36 | |
| *** ldnunes has joined #openstack-infra | 11:37 | |
| ianw | #status log afs mirror.pypi quota exceeded; increased to 1.9T (2000000000) | 11:38 |
| openstackstatus | ianw: finished logging | 11:38 |
| ianw | mirror.pypi 1.9T 1.7T 90% 69% | 11:38 |
| auristor | was the server restart at 07:16:59 UTC intentional? | 11:40 |
| ianw | that was about 5 hours ago? that would have been intentional yes | 11:41 |
| *** alexchadin has quit IRC | 11:41 | |
| ianw | i think i messed up my script and started releasing everything, so started again iirc | 11:41 |
| auristor | mirror.ppypi.readonly was in the middle of a release when the restart occurred so the volume was left in a bad state. | 11:42 |
| *** udesale_ has quit IRC | 11:42 | |
| ianw | auristor: i removed and re-added the r/o mirror before trying the release on it again | 11:43 |
| ianw | that started at Thu May 24 09:32:47 UTC 2018 | 11:44 |
| *** rasca has quit IRC | 11:44 | |
| ianw | and then i guess failed straight away with | 11:45 |
| ianw | Thu May 24 09:32:47 2018 VCreateVolume: Problem creating Volume information file associated with volume header /vicepa/V0536870932.vol | 11:45 |
| ianw | Thu May 24 09:32:47 2018 VDestroyVolumeDiskHeader: Couldn't unlink disk header, error = 2 | 11:45 |
| auristor | I don't think that succeeded. | 11:46 |
| ianw | no, that release failed | 11:46 |
| ianw | Creating new volume 536870932 on replication site afs02.dfw.openstack.org: Failed to create the ro volume: : Input/output error | 11:46 |
| auristor | the vos zap failed | 11:47 |
| ianw | should i try zapping it now and see what happens? | 11:47 |
| auristor | I'm still reading | 11:48 |
| auristor | but what I think happened is that the attempt to zap failed and triggered a salvage which recreated the header and link table. | 11:48 |
| ianw | interesting. my "vos remove" commands all returned ok | 11:51 |
| auristor | there are no transactions in process at the moment. I would like you to restart afs02.dfw just in case the dafileserver's volume cache is confused. | 11:51 |
| *** gfidente has quit IRC | 11:51 | |
| ianw | alright, restarting it | 11:51 |
| *** eumel8 has joined #openstack-infra | 11:52 | |
| ianw | ok, it's back | 11:52 |
| auristor | then try to "bos salvage afs02.dfw 536870932 -forceDAFS" and share the logs again | 11:52 |
| *** d0ugal has joined #openstack-infra | 11:54 | |
| ianw | auristor: ok, bos salvage afs02.dfw.openstack.org -volume 536870932 -partition a -forceDAFS | 11:55 |
| *** gfidente has joined #openstack-infra | 11:55 | |
| ianw | http://paste.openstack.org/show/721879/ <- salvage logs | 11:56 |
| ianw | nothing in the Vol log | 11:56 |
| *** d0ugal__ has quit IRC | 11:57 | |
| auristor | the salvager agrees the volume does not exist. please try to release it again | 11:57 |
| *** jpena|off is now known as jpena | 11:58 | |
| ianw | ok, this is the huge one | 11:58 |
| ianw | well it failed | 11:59 |
| ianw | http://paste.openstack.org/show/721880/ release & server logs | 11:59 |
| ianw | there is no file /vicepa/V0536870932.vol | 12:00 |
| *** dbecker has joined #openstack-infra | 12:04 | |
| ianw | and yes, it seems that failure did trigger the salvager | 12:04 |
| ianw | 05/24/2018 11:58:58 dispatching child to salvage volume 536870932... | 12:04 |
| auristor | but the salvager isn't doing anything useful :( | 12:04 |
| ianw | http://paste.openstack.org/show/721881/ | 12:05 |
| ianw | salvage logs | 12:05 |
| auristor | please "kill -s TSTP <davolserver-pid>" four times and then try to release again. I'm not sure we will get anything helpful from OpenAFS but we might | 12:06 |
| auristor | that will increase the volserver debug log level to 125 | 12:06 |
| auristor | I will be back in ten minutes. | 12:07 |
| ianw | auristor: ok, that did increase the debug level -> http://paste.openstack.org/show/721882/ | 12:09 |
| ianw | but yeah, didn't really say anything else | 12:10 |
| *** quiquell is now known as quiquell|lunch | 12:12 | |
| *** lpetrut has quit IRC | 12:13 | |
| *** temp has joined #openstack-infra | 12:18 | |
| ianw | auristor: ok, i've removed the pypi.readonly volume, and done a release. i'll re-add it now, with the logging and see | 12:20 |
| *** lifeless_ has joined #openstack-infra | 12:24 | |
| openstackgerrit | Artem Goncharov proposed openstack-infra/zuul master: github approval type is approved https://review.openstack.org/570378 | 12:24 |
| *** lifeless has quit IRC | 12:24 | |
| *** alexchadin has joined #openstack-infra | 12:27 | |
| *** rlandy has joined #openstack-infra | 12:28 | |
| ianw | ok, one thought is i'm going to run the bandersnatch update on mirror.pypi and then release it, so it's a new revision | 12:29 |
| *** rlandy is now known as rlandy|rover | 12:30 | |
| ianw | nup, same thing | 12:31 |
| *** udesale has joined #openstack-infra | 12:31 | |
| *** temp has quit IRC | 12:32 | |
| *** eumel8 has quit IRC | 12:32 | |
| auristor | I'm back but might be pulled away. my son woke up sounding like a sea lion this morning. | 12:37 |
| ianw | :( it's getting late here too | 12:38 |
| auristor | in good spirits though. he thinks its funny and wants to go to the zoo to speak with Clyde | 12:38 |
| auristor | please strace the davolserver process and release again | 12:39 |
| auristor | I would like to find out what file is on disk that shouldn't be | 12:40 |
| *** quiquell|lunch is now known as quiquell | 12:41 | |
| *** lifeless_ has quit IRC | 12:41 | |
| ianw | auristor: people.redhat.com/~iwienand/davolserver.txt | 12:42 |
| auristor | please give me a recursive directory listing of people.redhat.com/~iwienand/davolserver.txt | 12:43 |
| auristor | sorry, the copy didn't work | 12:43 |
| *** lifeless has joined #openstack-infra | 12:44 | |
| *** trown|outtypewww is now known as trown | 12:44 | |
| ianw | /vicepa/AFSIDat/H/H+++U/special/zzzz5E=+++0 maybe ? | 12:44 |
| auristor | /vicepa/AFSIDat/H/H+++U/ | 12:44 |
| auristor | I suspect the listing is going to be huge since that is a volume with a lot of files | 12:46 |
| auristor | 3349794 files | 12:46 |
| ianw | it's already 60mb | 12:47 |
| *** szaher has quit IRC | 12:48 | |
| *** eumel8 has joined #openstack-infra | 12:48 | |
| *** rasca has joined #openstack-infra | 12:49 | |
| ianw | auristor: it's 24mb zipped ... | 12:50 |
| *** cshastri has quit IRC | 12:51 | |
| ianw | auristor: do you want the one from afs01, or the r/o mirror afs02? | 12:51 |
| auristor | just from afs02.dfw | 12:51 |
| openstackgerrit | Artem Goncharov proposed openstack-infra/zuul master: correct github review approval type is `approved` https://review.openstack.org/570378 | 12:53 |
| ianw | auristor: http://people.redhat.com/~iwienand/h+++u.txt | 12:53 |
| auristor | could you give that to me compressed? :) | 12:54 |
| ianw | that one is only like 200k | 12:55 |
| ianw | the one of afs01 is massive | 12:55 |
| ianw | 8ddd7aa468fd40f8628b05d387f19bc0 /vicepa/AFSIDat/H/H+++U/special/zzzz5E=+++0 -> afs02 | 12:56 |
| ianw | 20c455f1d83d793ceb24420340e0d810 /vicepa/AFSIDat/H/H+++U/special/zzzz5E=+++0 -> afs01 | 12:56 |
| ianw | are they supposed to be the same? | 12:56 |
| *** gfidente has quit IRC | 12:57 | |
| auristor | remove /vicepa/AFSIDat/H/H+++U/ from afs02.dfw and then restart the release | 12:57 |
| ianw | hmm, it's running, but damn it i forgot -v! | 12:59 |
| auristor | please tail VolserLog on afs02.dfw | 13:00 |
| ianw | Thu May 24 12:58:23 2018 [13] VReadVolumeDiskHeader: Couldn't open header for volume 536870932 (errno 2). | 13:01 |
| ianw | Thu May 24 12:58:23 2018 [13] attach2: forcing vol 536870932 to error state (state 0 flags 0x0 ec 103) | 13:01 |
| ianw | Thu May 24 12:58:23 2018 [13] 1 Volser: CreateVolume: volume 536870932 (mirror.pypi.readonly) created | 13:01 |
| *** ihar has quit IRC | 13:02 | |
| ianw | the /vicepa/V0536870932.vol is there now, which it complained about before | 13:02 |
| ianw | disk usage isn't really going up atm | 13:03 |
| *** myoung|off is now known as myoung | 13:03 | |
| ianw | gahh i can't believe i forgot the -v. but i'm not going to kill it | 13:05 |
| ianw | messages about Thu May 24 13:05:36 2018 [0] trans 62 on volume 536870932 is older than 420 seconds | 13:05 |
| ianw | but i guess that's expected | 13:06 |
| auristor | the directory tree /vicepa/AFSIDat/H/H+++U/ was the volume group (aka RW volume) tree. /vicepa/AFSIDat/I/I+++U/ is the RO volume tree. perhaps we should have removed that just for cleanliness. Any vnodes that are stored there will be overwritten by the incoming data. | 13:06 |
| auristor | For tracking purposes, this is the connection on afs02.dfw/7005 being used for the forward | 13:07 |
| auristor | Connection from host [104.130.138.161]:7005, Cuid 9a315399/5f9f1d4c | 13:07 |
| auristor | serial 194817, natMTU 1444, security index 2, server conn | 13:07 |
| auristor | rxkad: level clear, flags authenticated pktCksum, expires in 7179 days 14 hours 7 minutes 6 seconds | 13:07 |
| auristor | Received 454455288 bytes in 320943 packets | 13:07 |
| auristor | Sent 0 bytes in 0 packets | 13:07 |
| auristor | call 0: # 1, state active, mode: receiving | 13:07 |
| auristor | when the received bytes count approaches 1.7TB it will be close to finishing | 13:08 |
| auristor | get some sleep | 13:09 |
| ianw | ok, so far the disk usage isn't going up ... but it's doing something | 13:09 |
| *** baoli has joined #openstack-infra | 13:09 | |
| auristor | if you list the contents of /vicepa/AFSIDat/I/I+++U/ I suspect there are many files from yesterday | 13:10 |
| ianw | hmm | 13:11 |
| ianw | root@afs02:/vicepa/AFSIDat# cd I | 13:11 |
| ianw | bash: cd: I: No such file or directory | 13:11 |
| ianw | root@afs02:/vicepa/AFSIDat# ls | 13:11 |
| ianw | += =0 1= 20 4= 7= A= b= f g= H i l m= o P= Q r README S= u v= V= W x y= Z | 13:11 |
| auristor | I guess I computed that wrong | 13:12 |
| ianw | it is writing stuff out, i was just looking at it in gb which wasn't updating | 13:13 |
| *** baoli has quit IRC | 13:13 | |
| *** baoli has joined #openstack-infra | 13:13 | |
| auristor | its received ~700MB so far | 13:13 |
| ianw | what about the other two similarly corrupt volumes? | 13:14 |
| ianw | maybe we should worry about one thing at a time | 13:14 |
| auristor | which volumes are they again? | 13:14 |
| *** jistr is now known as jistr|mtg | 13:14 | |
| ianw | the two with "old releases" are now mirror.ubuntu and mirror.ubuntu-ports | 13:15 |
| ianw | with pypi now being locked and updating | 13:15 |
| *** ykarel is now known as ykarel|away | 13:17 | |
| auristor | both mirror.ubuntu.readonly and mirror.ubuntu-ports are known to afs02.dfw and are online. | 13:18 |
| auristor | mirror.ubuntu will be stored under 536870949 is Z/Z+++U | 13:18 |
| *** david-lyle has joined #openstack-infra | 13:19 | |
| *** dklyle has quit IRC | 13:19 | |
| auristor | mirror.ubuntu-ports will be stored under /vicepa/AFSIDat/=0/=0++U | 13:19 |
| *** gfidente has joined #openstack-infra | 13:20 | |
| *** zoli is now known as zoli|afk | 13:20 | |
| ianw | i will take it as a todo item to figure out how you hashed that, i'm sure it's in the docs somewhere :) | 13:21 |
| auristor | bos salvage afs02.dfw a 536870950 -forceDAFS | 13:21 |
| auristor | bos salvage afs02.dfw a 536871042 -forceDAFS | 13:21 |
| auristor | docs. you make me laugh | 13:21 |
| ianw | http://paste.openstack.org/show/721886/ | 13:22 |
| ianw | http://paste.openstack.org/show/721887/ | 13:24 |
| *** links has quit IRC | 13:24 | |
| ianw | i really don't want to release them in parallel with this huge pypi update though | 13:25 |
| auristor | agreed | 13:25 |
| auristor | vos zap should remove them if necessary | 13:25 |
| auristor | 2.2GB so far | 13:25 |
| *** wolverineav has joined #openstack-infra | 13:26 | |
| ianw | so basically AFSIDat/XX/YYYY is "isolated" for a volume? | 13:26 |
| *** david-lyle has quit IRC | 13:26 | |
| ianw | it's not like bits of, say mirror.pypi would be in /vicepa/AFSIDat/=0/=0++U | 13:27 |
| *** amoralej|off is now known as amoralej | 13:35 | |
| ianw | auristor: again thank you for helping out. i've sent an update in http://lists.openstack.org/pipermail/openstack-infra/2018-May/005951.html . for now, will let this pypi sync run, and then we can worry about the other two | 13:38 |
| ianw | now i have to head off to bed, catch everyone later! | 13:38 |
| *** ykarel|away has quit IRC | 13:46 | |
| *** esberglu has joined #openstack-infra | 13:49 | |
| *** tpsilva has joined #openstack-infra | 13:50 | |
| *** udesale has quit IRC | 13:51 | |
| *** dave-mccowan has joined #openstack-infra | 13:51 | |
| *** udesale has joined #openstack-infra | 13:51 | |
| *** dave-mcc_ has joined #openstack-infra | 13:53 | |
| *** tesseract-RH has joined #openstack-infra | 13:54 | |
| *** tesseract-RH has quit IRC | 13:55 | |
| *** shaner has quit IRC | 13:55 | |
| *** tesseract-RH has joined #openstack-infra | 13:55 | |
| *** dave-mccowan has quit IRC | 13:56 | |
| *** tesseract has quit IRC | 13:56 | |
| *** shaner has joined #openstack-infra | 13:57 | |
| *** derekh has quit IRC | 13:59 | |
| *** olivierb- has quit IRC | 13:59 | |
| *** jistr|mtg is now known as jistr | 14:02 | |
| *** slaweq has joined #openstack-infra | 14:08 | |
| *** ogalush has joined #openstack-infra | 14:10 | |
| dtantsur | hey folks! in the new zuul v3 devstack world, where do I put some logs to make them published? | 14:10 |
| *** e0ne has quit IRC | 14:12 | |
| *** ispp has quit IRC | 14:14 | |
| *** sai_ has quit IRC | 14:15 | |
| *** rook has quit IRC | 14:15 | |
| *** sai_ has joined #openstack-infra | 14:17 | |
| *** rook has joined #openstack-infra | 14:17 | |
| *** rook is now known as Guest11079 | 14:18 | |
| *** rpioso|afk is now known as rpioso | 14:18 | |
| *** dbecker has quit IRC | 14:18 | |
| *** edmondsw has joined #openstack-infra | 14:20 | |
| *** olivierbourdon38 has joined #openstack-infra | 14:20 | |
| *** derekh has joined #openstack-infra | 14:21 | |
| *** ispp has joined #openstack-infra | 14:24 | |
| *** Guest11079 has quit IRC | 14:26 | |
| *** rook_ has joined #openstack-infra | 14:29 | |
| *** pblaho has quit IRC | 14:30 | |
| *** pblaho has joined #openstack-infra | 14:32 | |
| *** ogalush has quit IRC | 14:33 | |
| *** dklyle has joined #openstack-infra | 14:36 | |
| *** netsin has joined #openstack-infra | 14:37 | |
| *** lpetrut has joined #openstack-infra | 14:39 | |
| *** ihar has joined #openstack-infra | 14:42 | |
| openstackgerrit | Gaetan Semet proposed openstack-dev/pbr master: Add leading 0 on alpha release in semver doc https://review.openstack.org/558181 | 14:42 |
| openstackgerrit | Gaetan Semet proposed openstack-dev/pbr master: cmd: 'pbr boostrap' create setup.[cfg,py] automatically https://review.openstack.org/559484 | 14:42 |
| openstackgerrit | Gaetan Semet proposed openstack-dev/pbr master: pbr git-tag command https://review.openstack.org/561731 | 14:42 |
| *** jamesmcarthur has quit IRC | 14:43 | |
| Shrews | dtantsur: i use a post playbook for the nodepool jobs. http://git.openstack.org/cgit/openstack-infra/nodepool/tree/playbooks/nodepool-functional-base/post.yaml | 14:44 |
| dtantsur | nice, I'll look into it | 14:45 |
| dtantsur | is this zuul.executor.log_root thing set early? | 14:45 |
| dtantsur | I wonder if I can put ironic BM logs there from the beginning? | 14:45 |
| frickler | dtantsur: I can also offer https://review.openstack.org/561194 as an example to add config files and systemd journals to a job based on the new v3 "devstack" job | 14:46 |
| dtantsur | cool! thanks :) | 14:47 |
| *** tesseract-RH has quit IRC | 14:47 | |
| Shrews | dtantsur: yeah, you get that var for "free": https://zuul-ci.org/docs/zuul/user/jobs.html#id1 | 14:48 |
| *** zoli|afk is now known as zoli | 14:48 | |
| dtantsur | aha, I should have read this ofc :) | 14:48 |
| *** zoli is now known as zoli|wfh | 14:48 | |
| *** zoli|wfh is now known as zoli | 14:48 | |
| *** dklyle has quit IRC | 14:48 | |
| *** dklyle has joined #openstack-infra | 14:50 | |
| *** camunoz has joined #openstack-infra | 14:50 | |
| *** tushar has joined #openstack-infra | 14:51 | |
| Shrews | dtantsur: you can't really put logs there from the beginning since it's a location on the executor, not the node where the jobs actually run | 14:53 |
| dtantsur | ah, I see | 14:54 |
| Shrews | dtantsur: you can place them in {{ ansible_user_dir }}/work/logs, then copy them over | 14:54 |
| *** jtomasek has joined #openstack-infra | 14:54 | |
| tushar | fungi : Hi | 14:56 |
| *** dbecker has joined #openstack-infra | 14:57 | |
| *** alexchadin has quit IRC | 15:00 | |
| *** slaweq has quit IRC | 15:01 | |
| *** dbecker has quit IRC | 15:03 | |
| *** tesseract has joined #openstack-infra | 15:03 | |
| *** markvoelker_ has quit IRC | 15:07 | |
| *** markvoelker has joined #openstack-infra | 15:07 | |
| *** e0ne has joined #openstack-infra | 15:09 | |
| *** iyamahat has joined #openstack-infra | 15:09 | |
| roman_g | Hello team. Please, review proposed openstack/diskimage-builder master: Allow to rebuild arbitrary images https://review.openstack.org/568697 | 15:10 |
| *** markvoelker has quit IRC | 15:11 | |
| *** yamahata has quit IRC | 15:15 | |
| *** eernst has joined #openstack-infra | 15:16 | |
| *** iyamahat has quit IRC | 15:18 | |
| *** rossella_s has quit IRC | 15:18 | |
| *** dbecker has joined #openstack-infra | 15:19 | |
| *** pcaruana has quit IRC | 15:19 | |
| *** germs has joined #openstack-infra | 15:19 | |
| *** germs has quit IRC | 15:19 | |
| *** germs has joined #openstack-infra | 15:19 | |
| *** germs has quit IRC | 15:19 | |
| *** aojea has quit IRC | 15:19 | |
| *** germs has joined #openstack-infra | 15:20 | |
| *** germs has quit IRC | 15:20 | |
| *** germs has joined #openstack-infra | 15:20 | |
| *** aojea has joined #openstack-infra | 15:20 | |
| *** rossella_s has joined #openstack-infra | 15:20 | |
| *** yamamoto has joined #openstack-infra | 15:20 | |
| *** rook_ is now known as rook | 15:20 | |
| *** yamamoto has quit IRC | 15:21 | |
| *** iyamahat has joined #openstack-infra | 15:23 | |
| *** slaweq has joined #openstack-infra | 15:25 | |
| *** jamesmcarthur has joined #openstack-infra | 15:29 | |
| *** ccamacho has quit IRC | 15:31 | |
| *** electrofelix has quit IRC | 15:32 | |
| *** edmondsw has quit IRC | 15:33 | |
| *** jamesmcarthur has quit IRC | 15:34 | |
| *** jtomasek has quit IRC | 15:35 | |
| *** ispp has quit IRC | 15:35 | |
| *** eumel8 has quit IRC | 15:38 | |
| *** janki has quit IRC | 15:40 | |
| *** slaweq has quit IRC | 15:40 | |
| *** quiquell is now known as quiquell|off | 15:42 | |
| *** e0ne has quit IRC | 15:42 | |
| *** rossella_s has quit IRC | 15:44 | |
| *** jamesmcarthur has joined #openstack-infra | 15:45 | |
| *** rossella_s has joined #openstack-infra | 15:46 | |
| *** iyamahat has quit IRC | 15:48 | |
| *** mriedem has joined #openstack-infra | 15:49 | |
| *** jamesmcarthur has quit IRC | 15:50 | |
| *** ykarel|away has joined #openstack-infra | 15:51 | |
| *** diablo_rojo has joined #openstack-infra | 15:52 | |
| *** lbragstad has joined #openstack-infra | 15:53 | |
| *** jamesmcarthur has joined #openstack-infra | 15:53 | |
| *** jamesmcarthur has quit IRC | 15:53 | |
| *** jamesmcarthur has joined #openstack-infra | 15:53 | |
| *** eumel8 has joined #openstack-infra | 15:54 | |
| *** slaweq has joined #openstack-infra | 15:54 | |
| *** sshnaidm has quit IRC | 15:56 | |
| *** edmondsw_ has joined #openstack-infra | 15:58 | |
| *** edmondsw_ has quit IRC | 15:59 | |
| *** edmondsw_ has joined #openstack-infra | 15:59 | |
| *** ykarel|away has quit IRC | 16:00 | |
| *** markvoelker has joined #openstack-infra | 16:00 | |
| *** dbecker has quit IRC | 16:02 | |
| *** eernst has quit IRC | 16:03 | |
| *** pblaho has quit IRC | 16:04 | |
| *** pblaho has joined #openstack-infra | 16:04 | |
| *** markvoelker has quit IRC | 16:05 | |
| *** ramishra has quit IRC | 16:06 | |
| *** caphrim007 has joined #openstack-infra | 16:07 | |
| *** slaweq_ has joined #openstack-infra | 16:08 | |
| *** tushar has quit IRC | 16:08 | |
| *** dmellado has quit IRC | 16:09 | |
| *** ccamacho has joined #openstack-infra | 16:09 | |
| *** gouthamr has quit IRC | 16:09 | |
| *** arahal_ has joined #openstack-infra | 16:10 | |
| *** slaweq has quit IRC | 16:11 | |
| *** markvoelker has joined #openstack-infra | 16:12 | |
| *** stevebaker has quit IRC | 16:14 | |
| *** jamesmcarthur has quit IRC | 16:17 | |
| *** jamesmcarthur has joined #openstack-infra | 16:18 | |
| *** dave-mcc_ has quit IRC | 16:19 | |
| *** mriedem1 has joined #openstack-infra | 16:21 | |
| *** markvoelker has quit IRC | 16:21 | |
| *** mriedem has quit IRC | 16:22 | |
| *** lucasagomes is now known as lucas-afk | 16:23 | |
| *** lucas-afk is now known as lucasagomes | 16:23 | |
| *** pcaruana has joined #openstack-infra | 16:23 | |
| *** eernst has joined #openstack-infra | 16:28 | |
| *** edmondsw has joined #openstack-infra | 16:29 | |
| *** trown is now known as trown|lunch | 16:29 | |
| *** aojea has quit IRC | 16:29 | |
| *** edmondsw has quit IRC | 16:30 | |
| *** edmondsw has joined #openstack-infra | 16:30 | |
| *** markvoelker has joined #openstack-infra | 16:31 | |
| *** edmondsw_ has quit IRC | 16:32 | |
| *** pcaruana has quit IRC | 16:32 | |
| *** edmondsw has quit IRC | 16:34 | |
| *** mriedem1 is now known as mriedem | 16:34 | |
| *** edmondsw has joined #openstack-infra | 16:34 | |
| *** eernst has quit IRC | 16:35 | |
| *** eernst has joined #openstack-infra | 16:36 | |
| corvus | the pypi afs volume release appears to still be in progress but progressing | 16:38 |
| *** slaweq has joined #openstack-infra | 16:38 | |
| *** diablo_rojo has quit IRC | 16:39 | |
| mnaser | https://review.openstack.org/#/q/(project:^openstack-infra/.*zuul-jobs+OR+project:openstack-infra/project-config)+AND+is:open | 16:39 |
| *** matbu has joined #openstack-infra | 16:40 | |
| *** sdatko has joined #openstack-infra | 16:40 | |
| mnaser | spotz: http://paste.openstack.org/show/721912/ | 16:40 |
| *** camunoz has quit IRC | 16:41 | |
| *** slaweq_ has quit IRC | 16:41 | |
| *** bobh has joined #openstack-infra | 16:41 | |
| *** diablo_rojo has joined #openstack-infra | 16:42 | |
| *** rossella_s has quit IRC | 16:42 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Remove getPullBySha from FakeGithubConnection https://review.openstack.org/563416 | 16:43 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Status branch protection checking for github https://review.openstack.org/535680 | 16:43 |
| *** lifeless has quit IRC | 16:43 | |
| *** udesale has quit IRC | 16:44 | |
| *** markvoelker has quit IRC | 16:44 | |
| *** camunoz has joined #openstack-infra | 16:45 | |
| *** iyamahat has joined #openstack-infra | 16:45 | |
| *** rossella_s has joined #openstack-infra | 16:46 | |
| *** netsin has quit IRC | 16:46 | |
| *** sdatko has quit IRC | 16:47 | |
| *** jamesmcarthur has quit IRC | 16:48 | |
| *** matbu has quit IRC | 16:48 | |
| *** slaweq has quit IRC | 16:48 | |
| *** jamesmcarthur has joined #openstack-infra | 16:48 | |
| *** slaweq has joined #openstack-infra | 16:49 | |
| *** sshnaidm has joined #openstack-infra | 16:49 | |
| *** sshnaidm is now known as sshnaidm|off | 16:50 | |
| *** iyamahat has quit IRC | 16:52 | |
| *** olivierbourdon38 has quit IRC | 16:56 | |
| *** eernst has quit IRC | 16:57 | |
| *** iyamahat has joined #openstack-infra | 16:57 | |
| *** eernst has joined #openstack-infra | 16:57 | |
| *** yamamoto has joined #openstack-infra | 16:59 | |
| *** olivierbourdon38 has joined #openstack-infra | 17:00 | |
| *** derekh has quit IRC | 17:00 | |
| *** edmondsw_ has joined #openstack-infra | 17:02 | |
| mtreinish | mordred: I'm wondering if I should just give up on: https://github.com/ansible/ansible/pull/23769 | 17:03 |
| *** amoralej is now known as amoralej|off | 17:03 | |
| *** dklyle has quit IRC | 17:04 | |
| *** gouthamr has joined #openstack-infra | 17:04 | |
| *** arahal_ has quit IRC | 17:04 | |
| *** matbu has joined #openstack-infra | 17:04 | |
| *** edmondsw has quit IRC | 17:06 | |
| pabelanger | mtreinish: have you asked (I'm assuming yes) on #ansible-devel recently? | 17:08 |
| pabelanger | maybe just need to get the right eyes on it | 17:08 |
| *** rossella_s has quit IRC | 17:08 | |
| *** slaweq_ has joined #openstack-infra | 17:08 | |
| pabelanger | because i do remember talking about it (mqtt callback) at ansiblefest last year (SF) | 17:09 |
| *** jamesmcarthur has quit IRC | 17:09 | |
| *** dhajare has quit IRC | 17:09 | |
| *** rossella_s has joined #openstack-infra | 17:09 | |
| mtreinish | pabelanger: well "recently" was months ago | 17:10 |
| mtreinish | I can do it again at some point | 17:11 |
| *** dtantsur is now known as dtantsur|afk | 17:11 | |
| pas-ha | hi all, just had to try to install DevStack from Kilo (sigh..) and stumbled that devstack repo does not have the 'kilo-eol' tag, it is 'kilo-2' instead. As a consequence it does not clone/install 'kilo-eol' tags but master, and expectedly fails. Is there a way to re-add a `kilo-eol` tag? | 17:11 |
| *** slaweq has quit IRC | 17:12 | |
| *** jamesmcarthur has joined #openstack-infra | 17:13 | |
| auristor | pypi afs volume replication has transfered 343GB out of 1718GB so far | 17:14 |
| *** stevebaker has joined #openstack-infra | 17:14 | |
| *** iranzo has quit IRC | 17:14 | |
| pabelanger | mtreinish: okay, not saying my efforts are going to be better then yours, but I could spend some time also pushing if you'd like. | 17:14 |
| *** jamesmcarthur has quit IRC | 17:15 | |
| *** jamesmcarthur has joined #openstack-infra | 17:15 | |
| *** slaweq_ has quit IRC | 17:15 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Don't process lock files as refs in tests https://review.openstack.org/570417 | 17:15 |
| *** yamamoto has quit IRC | 17:16 | |
| *** arahal_ has joined #openstack-infra | 17:16 | |
| *** yamamoto has joined #openstack-infra | 17:18 | |
| *** rossella_s has quit IRC | 17:18 | |
| mtreinish | pabelanger: cool, that would probably help. The main motivation for me is to get down to only maintaining 1 copy. Whether thats the infra copy or upstream in ansible | 17:18 |
| mtreinish | obviously I think having it upstream would probably be better for the overall ansible community | 17:19 |
| pabelanger | ++ | 17:19 |
| *** jamesmcarthur has quit IRC | 17:20 | |
| openstackgerrit | Merged openstack-infra/project-config master: Update executor labels https://review.openstack.org/570279 | 17:20 |
| *** rossella_s has joined #openstack-infra | 17:20 | |
| corvus | auristor: i'm worried that the release wasn't started with -localauth and the token will time out before the transaction completes | 17:22 |
| *** yamamoto has quit IRC | 17:23 | |
| *** panda is now known as panda|off | 17:23 | |
| *** tesseract has quit IRC | 17:25 | |
| corvus | auristor: it was started under k5start, but iirc, even if the local token is updated, since the single transaction will take longer than the lifetime of the token used to start it, it will eventually fail | 17:25 |
| corvus | (usually we perform initial volume releases on the fileservers so we can use -localauth) | 17:26 |
| *** eernst has quit IRC | 17:27 | |
| *** eernst has joined #openstack-infra | 17:27 | |
| *** jpena is now known as jpena|off | 17:28 | |
| *** edmondsw_ has quit IRC | 17:31 | |
| *** edmondsw has joined #openstack-infra | 17:31 | |
| *** mriedem has quit IRC | 17:32 | |
| *** lbragstad has quit IRC | 17:35 | |
| *** edmondsw has quit IRC | 17:36 | |
| *** eumel8 has quit IRC | 17:36 | |
| *** diablo_rojo has quit IRC | 17:38 | |
| *** gfidente has quit IRC | 17:39 | |
| *** arahal_ has quit IRC | 17:40 | |
| *** iyamahat has quit IRC | 17:41 | |
| *** bobh has quit IRC | 17:42 | |
| *** trown|lunch is now known as trown | 17:44 | |
| *** beagles is now known as beagles_brb | 17:46 | |
| *** dhinesh has joined #openstack-infra | 17:47 | |
| *** jamesmcarthur has joined #openstack-infra | 17:48 | |
| *** lbragstad has joined #openstack-infra | 17:49 | |
| *** arahal_ has joined #openstack-infra | 17:49 | |
| *** lpetrut has quit IRC | 17:51 | |
| *** rpittau has joined #openstack-infra | 17:53 | |
| *** edmondsw has joined #openstack-infra | 17:53 | |
| *** slaweq has joined #openstack-infra | 17:53 | |
| *** dhill has joined #openstack-infra | 17:53 | |
| *** dhill is now known as Guest96125 | 17:54 | |
| *** matbu has quit IRC | 17:54 | |
| *** wolverineav has quit IRC | 17:55 | |
| *** lbragstad has quit IRC | 17:56 | |
| *** eernst has quit IRC | 17:57 | |
| *** eernst has joined #openstack-infra | 17:57 | |
| *** edmondsw has quit IRC | 17:58 | |
| *** eumel8 has joined #openstack-infra | 18:00 | |
| *** aojea has joined #openstack-infra | 18:00 | |
| *** bobh has joined #openstack-infra | 18:01 | |
| *** pbourke has joined #openstack-infra | 18:01 | |
| *** wolverineav has joined #openstack-infra | 18:01 | |
| *** psachin has quit IRC | 18:02 | |
| *** dhajare has joined #openstack-infra | 18:03 | |
| *** efried has joined #openstack-infra | 18:04 | |
| *** wolverineav has quit IRC | 18:04 | |
| *** iyamahat has joined #openstack-infra | 18:04 | |
| *** wolverineav has joined #openstack-infra | 18:04 | |
| *** rossella_s has quit IRC | 18:04 | |
| *** jesslampe has joined #openstack-infra | 18:05 | |
| *** e0ne has joined #openstack-infra | 18:05 | |
| *** rossella_s has joined #openstack-infra | 18:06 | |
| openstackgerrit | Adam Harwell proposed openstack-infra/os-loganalyze master: Add Barbican logs to filter regex https://review.openstack.org/570212 | 18:07 |
| *** zoli is now known as zoli|gone | 18:08 | |
| *** zoli|gone is now known as zoli | 18:08 | |
| *** wolverineav has quit IRC | 18:09 | |
| *** mriedem has joined #openstack-infra | 18:09 | |
| *** arahal_ has quit IRC | 18:09 | |
| openstackgerrit | Merged openstack-infra/project-config master: Normalize projects.yaml https://review.openstack.org/570318 | 18:10 |
| *** jamesmcarthur has quit IRC | 18:11 | |
| *** jamesmcarthur has joined #openstack-infra | 18:12 | |
| *** slaweq_ has joined #openstack-infra | 18:15 | |
| *** jamesmcarthur has quit IRC | 18:16 | |
| *** slaweq has quit IRC | 18:18 | |
| *** slaweq_ has quit IRC | 18:20 | |
| *** eumel8 has quit IRC | 18:21 | |
| *** arahal_ has joined #openstack-infra | 18:23 | |
| *** larainema has quit IRC | 18:24 | |
| corvus | clarkb, fungi: where sholud we meet for lunch? | 18:25 |
| *** matbu has joined #openstack-infra | 18:26 | |
| *** e0ne has quit IRC | 18:26 | |
| *** eernst has quit IRC | 18:27 | |
| *** e0ne has joined #openstack-infra | 18:27 | |
| *** eernst has joined #openstack-infra | 18:27 | |
| *** gcb has joined #openstack-infra | 18:30 | |
| *** efried has quit IRC | 18:31 | |
| *** pbourke has quit IRC | 18:33 | |
| *** eumel8 has joined #openstack-infra | 18:34 | |
| fungi | good question--maybe near but not inside the feedback/awards session? that way anyone who wants can still duck in there if we wrap up quickly? | 18:36 |
| *** rossella_s has quit IRC | 18:36 | |
| *** jamesmcarthur has joined #openstack-infra | 18:37 | |
| *** rajinir has joined #openstack-infra | 18:37 | |
| *** rossella_s has joined #openstack-infra | 18:38 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Add multilabel support to ZooKeeper Node model https://review.openstack.org/570227 | 18:39 |
| *** beagles_brb is now known as beagles | 18:40 | |
| *** mriedem has quit IRC | 18:41 | |
| *** rpittau has quit IRC | 18:41 | |
| *** yamamoto has joined #openstack-infra | 18:41 | |
| *** yamamoto has quit IRC | 18:43 | |
| *** eumel8 has quit IRC | 18:44 | |
| fungi | corvus: clarkb: so maybe meet outside room 215? | 18:45 |
| *** rpittau has joined #openstack-infra | 18:46 | |
| clarkb | I told kata dev lounge | 18:46 |
| fungi | okay, let's do that, then | 18:46 |
| clarkb | hold on | 18:47 |
| clarkb | just asked to meet them in 304 at 1 ish | 18:47 |
| clarkb | so I guess we can do feedback session the migrate up | 18:47 |
| clarkb | fungi corvus mordred and anyone else ^ does that work? | 18:48 |
| clarkb | *I was just asked | 18:48 |
| *** yamamoto has joined #openstack-infra | 18:49 | |
| corvus | 304 at 1pm wfm | 18:49 |
| *** diablo_rojo has joined #openstack-infra | 18:49 | |
| *** yamamoto has quit IRC | 18:49 | |
| *** yamamoto has joined #openstack-infra | 18:49 | |
| clarkb | I am in 221 for image lifecycle managemebt | 18:50 |
| *** slaweq has joined #openstack-infra | 18:50 | |
| *** slaweq has quit IRC | 18:50 | |
| clarkb | then feedback session lunch thrn 304 for kata at 1 | 18:50 |
| *** rossella_s has quit IRC | 18:50 | |
| *** slaweq has joined #openstack-infra | 18:50 | |
| corvus | clarkb, fungi, jamesmcarthur: cloudflare and openstack.org are unhappy for me | 18:51 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Status branch protection checking for github https://review.openstack.org/535680 | 18:52 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Remove getRepoPermission fron FakeGithubConnection https://review.openstack.org/570428 | 18:52 |
| *** rossella_s has joined #openstack-infra | 18:52 | |
| *** arahal_ has quit IRC | 18:53 | |
| corvus | clarkb, fungi, jamesmcarthur: it's not just me. looks like cloudflaire thinks the upstream openstack.org server is offline | 18:53 |
| *** edmondsw has joined #openstack-infra | 18:53 | |
| clarkb | I pinged the foundation sysadmins | 18:54 |
| jamesmcarthur | hi corvus: on it | 18:55 |
| *** e0ne has quit IRC | 18:56 | |
| *** eernst has quit IRC | 18:57 | |
| *** wolverineav has joined #openstack-infra | 18:57 | |
| *** eernst has joined #openstack-infra | 18:57 | |
| *** edmondsw has quit IRC | 18:58 | |
| *** eernst has quit IRC | 18:58 | |
| *** eernst has joined #openstack-infra | 18:59 | |
| *** lpetrut has joined #openstack-infra | 19:01 | |
| *** slaweq has quit IRC | 19:02 | |
| *** caphrim007_ has joined #openstack-infra | 19:02 | |
| clarkb | fungi: are you in s release goals? curious to know how that goes | 19:03 |
| spotz | Clarkb he’s in here:) | 19:03 |
| *** gcb has quit IRC | 19:03 | |
| *** iyamahat has quit IRC | 19:04 | |
| *** yamamoto has quit IRC | 19:04 | |
| *** caphrim007 has quit IRC | 19:05 | |
| *** matbu has quit IRC | 19:06 | |
| *** bobh has quit IRC | 19:07 | |
| *** yamamoto has joined #openstack-infra | 19:07 | |
| *** ChanServ has quit IRC | 19:07 | |
| fungi | yep | 19:07 |
| fungi | it's a lot of meta-discussion | 19:07 |
| *** caphrim007_ has quit IRC | 19:08 | |
| *** aojea has quit IRC | 19:08 | |
| *** jesslampe has quit IRC | 19:10 | |
| jamesmcarthur | hi all - https://www.openstack.org/ is back up | 19:10 |
| jamesmcarthur | apologies for the downtime. we had an out of sync composer file | 19:10 |
| *** ChanServ has joined #openstack-infra | 19:12 | |
| *** barjavel.freenode.net sets mode: +o ChanServ | 19:12 | |
| *** e0ne has joined #openstack-infra | 19:13 | |
| *** dklyle has joined #openstack-infra | 19:13 | |
| *** tpsilva has quit IRC | 19:14 | |
| *** jamesmcarthur has quit IRC | 19:14 | |
| *** jamesmcarthur has joined #openstack-infra | 19:15 | |
| *** dbecker has joined #openstack-infra | 19:15 | |
| *** Guest96125 has quit IRC | 19:16 | |
| *** diablo_rojo has quit IRC | 19:16 | |
| *** e0ne has quit IRC | 19:16 | |
| *** bobh has joined #openstack-infra | 19:16 | |
| *** e0ne has joined #openstack-infra | 19:17 | |
| *** jamesmcarthur has quit IRC | 19:17 | |
| *** jamesmcarthur has joined #openstack-infra | 19:18 | |
| *** yamamoto has quit IRC | 19:19 | |
| *** diablo_rojo has joined #openstack-infra | 19:19 | |
| *** yamamoto has joined #openstack-infra | 19:19 | |
| *** ChanServ has quit IRC | 19:21 | |
| *** e0ne has quit IRC | 19:22 | |
| *** dirk has quit IRC | 19:24 | |
| *** antonym is now known as spy | 19:24 | |
| *** dirk has joined #openstack-infra | 19:25 | |
| *** jamesmcarthur has quit IRC | 19:25 | |
| *** eernst has quit IRC | 19:26 | |
| *** jamesmcarthur has joined #openstack-infra | 19:27 | |
| *** jamesmca_ has joined #openstack-infra | 19:29 | |
| *** ChanServ has joined #openstack-infra | 19:30 | |
| *** barjavel.freenode.net sets mode: +o ChanServ | 19:30 | |
| *** jamesmca_ has quit IRC | 19:31 | |
| *** jamesmca_ has joined #openstack-infra | 19:31 | |
| *** markvoelker has joined #openstack-infra | 19:31 | |
| *** jamesmcarthur has quit IRC | 19:32 | |
| *** markvoelker has quit IRC | 19:33 | |
| *** jlvacation is now known as Guest65555 | 19:33 | |
| *** melwitt is now known as Guest32198 | 19:33 | |
| *** puiterwijk is now known as Guest51838 | 19:33 | |
| *** beagles is now known as Guest87851 | 19:34 | |
| *** fungi is now known as Guest77695 | 19:34 | |
| *** sdake is now known as Guest14735 | 19:34 | |
| *** Hunner is now known as Guest3271 | 19:34 | |
| *** patricku is now known as Guest20314 | 19:34 | |
| *** tlbr is now known as Guest34116 | 19:34 | |
| *** v1k0d3n is now known as Guest45438 | 19:34 | |
| *** dmsimard is now known as Guest94448 | 19:34 | |
| *** kklimonda is now known as Guest19794 | 19:34 | |
| *** jpmaxman is now known as Guest95387 | 19:34 | |
| *** prometheanfire is now known as Guest62762 | 19:34 | |
| *** fkautz is now known as Guest85163 | 19:34 | |
| *** flaper87 is now known as Guest62139 | 19:34 | |
| *** honza is now known as Guest60997 | 19:34 | |
| *** kmalloc is now known as Guest70437 | 19:34 | |
| *** Tahvok is now known as Guest19737 | 19:34 | |
| *** Krenair is now known as Guest9040 | 19:34 | |
| *** tristanC is now known as Guest15248 | 19:34 | |
| *** icey is now known as Guest76648 | 19:34 | |
| *** gtmanfred is now known as Guest92760 | 19:34 | |
| *** bramwelt is now known as Guest20502 | 19:34 | |
| *** vkmc has quit IRC | 19:34 | |
| *** vkmc has joined #openstack-infra | 19:34 | |
| *** vkmc has quit IRC | 19:34 | |
| *** vkmc has joined #openstack-infra | 19:34 | |
| *** rossella_s has quit IRC | 19:34 | |
| *** Guest9040 is now known as Krenair | 19:34 | |
| *** spy is now known as antonym | 19:34 | |
| *** slaweq has joined #openstack-infra | 19:35 | |
| *** edmondsw has joined #openstack-infra | 19:35 | |
| *** yamamoto has quit IRC | 19:35 | |
| *** Guest77695 is now known as fungi | 19:35 | |
| *** jamesmca_ has quit IRC | 19:35 | |
| *** rpittau has quit IRC | 19:36 | |
| *** bandini has quit IRC | 19:36 | |
| *** Guest51838 is now known as puiterwijk | 19:37 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Add config read and node deletion notifications https://review.openstack.org/565882 | 19:37 |
| *** rossella_s has joined #openstack-infra | 19:37 | |
| *** bandini has joined #openstack-infra | 19:38 | |
| *** edmondsw has quit IRC | 19:39 | |
| *** yamamoto has joined #openstack-infra | 19:39 | |
| *** slaweq has quit IRC | 19:39 | |
| *** rlandy|rover is now known as rlandy | 19:42 | |
| *** harlowja has joined #openstack-infra | 19:42 | |
| *** rlandy is now known as rlandy|rover | 19:43 | |
| *** yamamoto has quit IRC | 19:44 | |
| *** Guest3271 is now known as Hunner | 19:48 | |
| *** Guest94448 has quit IRC | 19:51 | |
| *** jamesmcarthur has joined #openstack-infra | 19:51 | |
| *** dmsimard has joined #openstack-infra | 19:51 | |
| *** eumel8 has joined #openstack-infra | 19:52 | |
| *** markvoelker has joined #openstack-infra | 19:56 | |
| *** jamesmcarthur has quit IRC | 19:56 | |
| *** rossella_s has quit IRC | 19:58 | |
| *** rossella_s has joined #openstack-infra | 19:59 | |
| *** markvoelker has quit IRC | 20:04 | |
| *** Guest62762 is now known as prometheanfire | 20:09 | |
| openstackgerrit | megan guiney proposed openstack-infra/openstack-zuul-jobs master: docs sitemap generation automation https://review.openstack.org/524862 | 20:09 |
| *** eumel8 has quit IRC | 20:09 | |
| *** matbu has joined #openstack-infra | 20:09 | |
| *** lpetrut has quit IRC | 20:10 | |
| *** lpetrut has joined #openstack-infra | 20:11 | |
| *** dhill has joined #openstack-infra | 20:12 | |
| *** diablo_rojo has quit IRC | 20:12 | |
| *** dhill is now known as Guest64332 | 20:12 | |
| *** markvoelker has joined #openstack-infra | 20:14 | |
| *** jesslampe has joined #openstack-infra | 20:18 | |
| *** jamesmcarthur has joined #openstack-infra | 20:19 | |
| openstackgerrit | Steve Baker proposed openstack-infra/project-config master: Configure ansible-role-tripleo-modify-image zuul jobs https://review.openstack.org/569002 | 20:19 |
| *** jamesmcarthur has quit IRC | 20:20 | |
| *** rossella_s has quit IRC | 20:20 | |
| *** tpsilva has joined #openstack-infra | 20:20 | |
| *** slaweq has joined #openstack-infra | 20:22 | |
| *** jamesmcarthur has joined #openstack-infra | 20:22 | |
| *** rossella_s has joined #openstack-infra | 20:23 | |
| *** dbecker has quit IRC | 20:24 | |
| *** e0ne has joined #openstack-infra | 20:24 | |
| *** trown is now known as trown|outtypewww | 20:27 | |
| *** kjackal has quit IRC | 20:35 | |
| *** rossella_s has quit IRC | 20:38 | |
| *** rossella_s has joined #openstack-infra | 20:39 | |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/project-config master: self-healing-sig now has jobs in check+gate https://review.openstack.org/570449 | 20:41 |
| *** lifeless has joined #openstack-infra | 20:44 | |
| EmilienM | infra-core, can you please add me and stevebaker on https://review.openstack.org/#/admin/groups/1916,members ? | 20:44 |
| *** jamesmcarthur has quit IRC | 20:45 | |
| *** eumel8 has joined #openstack-infra | 20:45 | |
| *** yamamoto has joined #openstack-infra | 20:46 | |
| *** Guest64332 has quit IRC | 20:47 | |
| *** e0ne has quit IRC | 20:47 | |
| *** e0ne has joined #openstack-infra | 20:49 | |
| *** pbourke has joined #openstack-infra | 20:49 | |
| ianw | EmilienM: done | 20:50 |
| *** slaweq_ has joined #openstack-infra | 20:51 | |
| *** efried has joined #openstack-infra | 20:51 | |
| ianw | ok, mirror.pypi transaction is still slowly but surely resyncing | 20:52 |
| *** dhajare has quit IRC | 20:52 | |
| *** edmondsw has joined #openstack-infra | 20:52 | |
| auristor | ianw: 696GB so far | 20:53 |
| fungi | ianw: per corvus's earlier comment, keep an eye out for the possibility it might end up with its kerberos ticket timing out partway through | 20:53 |
| auristor | fungi: the connection authentication is as follows: | 20:54 |
| auristor | rxkad: level clear, flags authenticated pktCksum, expires in 7179 days 6 hours 21 minutes 24 seconds | 20:54 |
| *** slaweq has quit IRC | 20:55 | |
| auristor | that is the connection between the two volservers which is what matters for the volume forwarding | 20:55 |
| *** jamesmcarthur has joined #openstack-infra | 20:56 | |
| *** jesslampe has quit IRC | 20:57 | |
| *** e0ne has quit IRC | 20:57 | |
| ianw | auristor: what do you use to get that with the size count? i've been watching the "status" output, but that only shows packet count | 20:57 |
| *** Guest87851 is now known as beagles | 20:57 | |
| *** yamamoto has quit IRC | 20:58 | |
| auristor | If the vos to source volserver connection authentication expires, then the destination volume can be manually brought online "vos online afs02.dfw vicepa 536870932" and then unlock the VL entry "vos unlock 536870932". | 20:58 |
| auristor | packets are 1444 bytes of data | 20:58 |
| *** jamesmca_ has joined #openstack-infra | 20:58 | |
| EmilienM | ianw: thanks | 20:58 |
| auristor | although I'm using the rx connection output | 20:59 |
| auristor | the auristorfs vos status output includes the actual 64-bit data transfer amounts when the volservers are auristorfs. the openafs servers do not provide that level of detail. | 20:59 |
| *** jamesmc__ has joined #openstack-infra | 21:00 | |
| *** jamesmcarthur has quit IRC | 21:01 | |
| *** Guest70437 is now known as kmalloc | 21:02 | |
| *** kmalloc has quit IRC | 21:02 | |
| *** kmalloc has joined #openstack-infra | 21:02 | |
| *** kmalloc has quit IRC | 21:02 | |
| *** kmalloc has joined #openstack-infra | 21:02 | |
| *** rpittau has joined #openstack-infra | 21:02 | |
| *** jamesmca_ has quit IRC | 21:03 | |
| *** kmalloc is now known as morgan | 21:03 | |
| *** morgan is now known as kmalloc | 21:03 | |
| *** e0ne has joined #openstack-infra | 21:03 | |
| *** eumel8 has quit IRC | 21:04 | |
| *** jamesmc__ has quit IRC | 21:05 | |
| *** e0ne has quit IRC | 21:05 | |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck master: Add query for os-ansible centos ImportError bug 1773248 https://review.openstack.org/570452 | 21:06 |
| openstack | bug 1773248 in openstack-ansible "openstack-ansible-functional-centos-7 intermittently fails with "ImportError: 'module' object has no attribute 'check_specifier'"" [Undecided,New] https://launchpad.net/bugs/1773248 | 21:06 |
| *** yamamoto has joined #openstack-infra | 21:06 | |
| *** mriedem has joined #openstack-infra | 21:06 | |
| *** rossella_s has quit IRC | 21:08 | |
| *** jesslampe has joined #openstack-infra | 21:08 | |
| *** pbourke has quit IRC | 21:08 | |
| *** shardy has quit IRC | 21:08 | |
| *** shardy has joined #openstack-infra | 21:09 | |
| *** iyamahat has joined #openstack-infra | 21:10 | |
| *** rossella_s has joined #openstack-infra | 21:10 | |
| *** gothicmindfood has quit IRC | 21:11 | |
| *** gothicmindfood has joined #openstack-infra | 21:11 | |
| *** esberglu has quit IRC | 21:13 | |
| *** yamamoto has quit IRC | 21:14 | |
| *** beagles is now known as beagles_biab | 21:14 | |
| *** gothicmindfood has quit IRC | 21:14 | |
| *** gothicmindfood has joined #openstack-infra | 21:15 | |
| *** ldnunes has quit IRC | 21:15 | |
| *** rossella_s has quit IRC | 21:16 | |
| *** jamesmcarthur has joined #openstack-infra | 21:17 | |
| *** njohnston has left #openstack-infra | 21:18 | |
| *** e0ne has joined #openstack-infra | 21:18 | |
| *** rossella_s has joined #openstack-infra | 21:19 | |
| *** jamesmcarthur has quit IRC | 21:19 | |
| *** jamesmcarthur has joined #openstack-infra | 21:20 | |
| openstackgerrit | Merged openstack-infra/elastic-recheck master: Add query for os-ansible centos ImportError bug 1773248 https://review.openstack.org/570452 | 21:20 |
| openstack | bug 1773248 in openstack-ansible "openstack-ansible-functional-centos-7 intermittently fails with "ImportError: 'module' object has no attribute 'check_specifier'"" [Undecided,New] https://launchpad.net/bugs/1773248 | 21:20 |
| *** slaweq has joined #openstack-infra | 21:21 | |
| *** mriedem has quit IRC | 21:22 | |
| *** edmondsw_ has joined #openstack-infra | 21:22 | |
| *** e0ne has quit IRC | 21:23 | |
| *** matbu has quit IRC | 21:23 | |
| openstackgerrit | Merged openstack-infra/project-config master: Ensure rhel7 image could be build https://review.openstack.org/570035 | 21:23 |
| *** efried1 has joined #openstack-infra | 21:24 | |
| *** jamesmcarthur has quit IRC | 21:24 | |
| *** efried has quit IRC | 21:24 | |
| *** slaweq_ has quit IRC | 21:24 | |
| *** efried1 is now known as efried | 21:24 | |
| ianw | auristor: ahh, so your output is from auristor "rxdebug" command? | 21:24 |
| ianw | rxdebug -port 7005 -long -servers afs02.dfw.openstack.org | 21:25 |
| *** edmondsw has quit IRC | 21:26 | |
| ianw | Received 2517517768 bytes in 553814665 packets <- i'm not sure what to make of that; it doesn't seem the total amount received | 21:26 |
| *** edmondsw_ is now known as edmondsw | 21:26 | |
| *** mriedem has joined #openstack-infra | 21:26 | |
| *** jamesmcarthur has joined #openstack-infra | 21:31 | |
| auristor | the Received bytes field is 32-bit and has rolled over multiple times | 21:32 |
| *** iyamahat has quit IRC | 21:32 | |
| *** yamamoto has joined #openstack-infra | 21:32 | |
| auristor | Its the authenticated packets count that matters because those are data packets | 21:33 |
| *** iranzo has joined #openstack-infra | 21:33 | |
| *** iranzo has quit IRC | 21:33 | |
| *** iranzo has joined #openstack-infra | 21:33 | |
| *** lifeless_ has joined #openstack-infra | 21:34 | |
| *** lifeless has quit IRC | 21:34 | |
| auristor | OpenAFS rx doesn't permit the 32-bit data packet sequence number to rollover which is why the max volume transfer size is 5.63TB. Although the volume size field wraps at 2TB and once that happens the reported size of the volume will be wrong. | 21:35 |
| *** jamesmcarthur has quit IRC | 21:35 | |
| *** lpetrut has quit IRC | 21:36 | |
| *** rpittau has quit IRC | 21:36 | |
| *** Guest65555 is now known as jlvillal | 21:37 | |
| *** caphrim007 has joined #openstack-infra | 21:37 | |
| *** yamamoto has quit IRC | 21:37 | |
| *** jesslampe has quit IRC | 21:39 | |
| *** jamesmcarthur has joined #openstack-infra | 21:40 | |
| ianw | oh right, that explains why it keeps changing | 21:40 |
| *** jesslampe has joined #openstack-infra | 21:41 | |
| *** caphrim007 has quit IRC | 21:41 | |
| *** jamesmca_ has joined #openstack-infra | 21:41 | |
| *** jamesmcarthur has quit IRC | 21:42 | |
| *** rpittau has joined #openstack-infra | 21:42 | |
| edmondsw | fungi what should this look like to use a local directory path instead of url? https://review.openstack.org/#/c/555964/7/tox.ini@52 | 21:42 |
| *** bobh has quit IRC | 21:43 | |
| *** felipemonteiro has joined #openstack-infra | 21:44 | |
| felipemonteiro | this page looks like spam: https://wiki.openstack.org/wiki/LaunchPad but I can't delete it | 21:44 |
| *** pbourke has joined #openstack-infra | 21:44 | |
| *** beagles_biab is now known as beagles | 21:45 | |
| *** Guest20314 is now known as patricku | 21:46 | |
| *** wolverineav has quit IRC | 21:48 | |
| *** aeng has joined #openstack-infra | 21:49 | |
| *** wolverineav has joined #openstack-infra | 21:49 | |
| *** rossella_s has quit IRC | 21:50 | |
| *** slaweq_ has joined #openstack-infra | 21:51 | |
| *** jesslampe has quit IRC | 21:52 | |
| *** camunoz has quit IRC | 21:52 | |
| *** edmondsw_ has joined #openstack-infra | 21:52 | |
| *** rossella_s has joined #openstack-infra | 21:52 | |
| *** wolverineav has quit IRC | 21:53 | |
| *** slaweq has quit IRC | 21:54 | |
| *** bauzas is now known as PapaOurs | 21:55 | |
| *** baoli has quit IRC | 21:55 | |
| *** edmondsw has quit IRC | 21:56 | |
| *** caphrim007 has joined #openstack-infra | 21:56 | |
| *** slaweq_ has quit IRC | 21:57 | |
| *** VW has joined #openstack-infra | 22:02 | |
| *** VW has quit IRC | 22:02 | |
| *** jesslampe has joined #openstack-infra | 22:02 | |
| *** mriedem is now known as bauzabear | 22:03 | |
| *** iranzo has quit IRC | 22:03 | |
| *** bauzabear is now known as mriedem | 22:03 | |
| *** PapaOurs is now known as bauzas | 22:04 | |
| *** jesslampe has quit IRC | 22:05 | |
| *** pbourke has quit IRC | 22:05 | |
| *** yamamoto has joined #openstack-infra | 22:06 | |
| *** jesslampe has joined #openstack-infra | 22:06 | |
| *** dpawlik has joined #openstack-infra | 22:06 | |
| *** rajinir has quit IRC | 22:07 | |
| *** dpawlik_ has joined #openstack-infra | 22:08 | |
| *** edmondsw_ has quit IRC | 22:08 | |
| *** dpawlik_ has quit IRC | 22:08 | |
| *** edmondsw has joined #openstack-infra | 22:09 | |
| *** edmondsw has quit IRC | 22:10 | |
| *** edmondsw has joined #openstack-infra | 22:10 | |
| *** slaweq has joined #openstack-infra | 22:10 | |
| *** dpawlik has quit IRC | 22:11 | |
| *** rcernin has joined #openstack-infra | 22:11 | |
| *** yamamoto has quit IRC | 22:12 | |
| *** e0ne has joined #openstack-infra | 22:13 | |
| *** slaweq has quit IRC | 22:15 | |
| *** rossella_s has quit IRC | 22:16 | |
| *** rossella_s has joined #openstack-infra | 22:17 | |
| *** jesslampe has quit IRC | 22:18 | |
| *** jesslampe has joined #openstack-infra | 22:21 | |
| *** edmondsw_ has joined #openstack-infra | 22:23 | |
| *** felipemonteiro has quit IRC | 22:23 | |
| *** felipemonteiro has joined #openstack-infra | 22:24 | |
| *** edmondsw_ has quit IRC | 22:24 | |
| *** felipemonteiro has quit IRC | 22:24 | |
| *** pbourke has joined #openstack-infra | 22:26 | |
| *** rossella_s has quit IRC | 22:26 | |
| *** edmondsw_ has joined #openstack-infra | 22:26 | |
| *** edmondsw has quit IRC | 22:27 | |
| *** wolverineav has joined #openstack-infra | 22:28 | |
| *** rossella_s has joined #openstack-infra | 22:28 | |
| *** pcichy has quit IRC | 22:28 | |
| *** pcichy has joined #openstack-infra | 22:29 | |
| *** edmondsw_ has quit IRC | 22:30 | |
| *** mriedem has quit IRC | 22:32 | |
| ianw | #status log removed AAAA records for afsdb01 & afsdb02 per https://review.openstack.org/559851 | 22:33 |
| *** pbourke has quit IRC | 22:34 | |
| ianw | auristor: ^ re discussion last night. we never removed afsdb, only the afs0<1|2>.dfw & afs01.ord | 22:34 |
| *** lifeless_ has quit IRC | 22:34 | |
| *** rossella_s has quit IRC | 22:36 | |
| ianw | #status log removed AAAA records for afsdb01 & afsdb02 per https://review.openstack.org/559851 | 22:38 |
| openstackstatus | ianw: finished logging | 22:38 |
| *** rossella_s has joined #openstack-infra | 22:38 | |
| *** matbu has joined #openstack-infra | 22:42 | |
| *** yamamoto has joined #openstack-infra | 22:43 | |
| *** gyee has joined #openstack-infra | 22:43 | |
| *** mriedem has joined #openstack-infra | 22:48 | |
| *** yamamoto has quit IRC | 22:50 | |
| *** pbourke has joined #openstack-infra | 22:51 | |
| *** edmondsw has joined #openstack-infra | 22:56 | |
| *** jesslampe has quit IRC | 22:58 | |
| *** edmondsw has quit IRC | 23:00 | |
| *** jesslampe has joined #openstack-infra | 23:01 | |
| *** jamesmca_ has quit IRC | 23:01 | |
| openstackgerrit | James Page proposed openstack-infra/project-config master: Switch charm-neutron-openvswitch to py3 charm jobs https://review.openstack.org/570476 | 23:01 |
| *** jamesmcarthur has joined #openstack-infra | 23:02 | |
| *** matbu has quit IRC | 23:11 | |
| *** jamesmcarthur has quit IRC | 23:13 | |
| openstackgerrit | Merged openstack-infra/project-config master: Configure ansible-role-tripleo-modify-image zuul jobs https://review.openstack.org/569002 | 23:13 |
| *** mriedem has quit IRC | 23:14 | |
| *** jamesmcarthur has joined #openstack-infra | 23:14 | |
| *** dhill_ has joined #openstack-infra | 23:15 | |
| *** rpittau has quit IRC | 23:17 | |
| *** pbourke has quit IRC | 23:18 | |
| *** e0ne has quit IRC | 23:18 | |
| *** jamesmcarthur has quit IRC | 23:19 | |
| *** markvoelker has quit IRC | 23:19 | |
| *** markvoelker has joined #openstack-infra | 23:20 | |
| *** rossella_s has quit IRC | 23:20 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Status branch protection checking for github https://review.openstack.org/535680 | 23:20 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Remove getRepoPermission fron FakeGithubConnection https://review.openstack.org/570428 | 23:20 |
| *** mriedem has joined #openstack-infra | 23:20 | |
| *** yamamoto has joined #openstack-infra | 23:21 | |
| *** matbu has joined #openstack-infra | 23:21 | |
| *** rpittau has joined #openstack-infra | 23:23 | |
| *** rossella_s has joined #openstack-infra | 23:23 | |
| *** markvoelker has quit IRC | 23:24 | |
| *** rlandy|rover is now known as rlandy|rvr|bbl | 23:25 | |
| *** yamamoto has quit IRC | 23:27 | |
| *** jamesmcarthur has joined #openstack-infra | 23:27 | |
| *** matbu has quit IRC | 23:30 | |
| *** e0ne has joined #openstack-infra | 23:31 | |
| *** jtomasek has joined #openstack-infra | 23:31 | |
| *** jamesmcarthur has quit IRC | 23:32 | |
| *** mriedem has quit IRC | 23:33 | |
| *** lifeless has joined #openstack-infra | 23:35 | |
| *** matbu has joined #openstack-infra | 23:36 | |
| *** jamesmcarthur has joined #openstack-infra | 23:38 | |
| *** jamesmcarthur has quit IRC | 23:43 | |
| *** diablo_rojo has joined #openstack-infra | 23:45 | |
| *** jesslampe has quit IRC | 23:46 | |
| *** jamesmcarthur has joined #openstack-infra | 23:49 | |
| *** yamamoto has joined #openstack-infra | 23:53 | |
| *** jamesmcarthur has quit IRC | 23:53 | |
| *** yamamoto has quit IRC | 23:58 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!