| *** lseki has quit IRC | 00:02 | |
| *** mriedem_afk is now known as mriedem | 00:03 | |
| openstackgerrit | Merged opendev/system-config master: Add docs for deploying a new gitea server https://review.opendev.org/668030 | 00:05 |
|---|---|---|
| *** dchen has quit IRC | 00:07 | |
| *** dchen has joined #openstack-infra | 00:11 | |
| clarkb | gc'ing on gitea06 is done | 00:12 |
| clarkb | there are a few defunct git processes that we might try to clean up tomorrow if they don't go away | 00:12 |
| ianw | fungi / corvus: thanks for looking at those messages. to try and sync everyone I've sent -> http://lists.infradead.org/pipermail/linux-afs/2019-June/003078.html | 00:34 |
| ianw | basically if we know what & how to trace in a useful way, i think we can probably just write a script to capture something helpful | 00:35 |
| ianw | sorry, wrt afs offline messages | 00:35 |
| *** hamzy has joined #openstack-infra | 00:38 | |
| donnyd | clarkb I am just about ready to start testing my infra | 00:42 |
| *** diablo_rojo has quit IRC | 00:44 | |
| auristor | ianw: abort code is 106 or VOFFLINE | 00:50 |
| auristor | ianw: which is what I expect the openafs fileserver to send while the volume is releasing | 00:52 |
| diablo_rojo_phon | melwitt: I think I have a local WIP patch for the search but I'm happy to walk you through stuff. Generally it's like clarkb said. Give it a second and it will do some typeahead magic to help you tailor your search. | 00:54 |
| diablo_rojo_phon | And of course will take whatever patches you've got :) | 00:54 |
| melwitt | diablo_rojo_phon: oh, coolness, thanks for letting me know. I'm totes cool with it being already done by you :) | 00:57 |
| melwitt | for me it's taking several seconds to get anything to appear, so I thought I was doing it wrong | 00:57 |
| melwitt | but I tried it some more, waiting 30s and longer and now I get the behavior that y'all described | 00:57 |
| diablo_rojo_phon | Yeah. mordred has volunteered to help with our database indexing in hopes that it speed up the search a bit. | 01:06 |
| melwitt | noice | 01:07 |
| diablo_rojo_phon | melwitt: I'll try to get that doc patch up next week or so | 01:07 |
| diablo_rojo_phon | If you have questions in the meantime, feel free to ping me :) | 01:07 |
| melwitt | diablo_rojo_phon: awesome, thanks much | 01:07 |
| *** imacdonn has quit IRC | 01:09 | |
| *** imacdonn has joined #openstack-infra | 01:10 | |
| ianw | auristor: ok, perhaps a similar case of just turn down the logging? | 01:11 |
| ianw | or is it hard to tell the difference between "releasing and coming back" and actually offline? | 01:11 |
| auristor | do you have FileAuditLog enabled on the fileservers ? | 01:12 |
| ianw | auristor: no, i think i have dejavu of us discussing this before | 01:12 |
| auristor | if so, the requests returning "CODE 106" will be logged in the FileAuditLog | 01:13 |
| ianw | is it going to be a ridiculous amount of logging to manage on busy servers? | 01:13 |
| auristor | you can then see whether the kafs client is trying to read from each fileserver and if its getting 106 (VOFFLINE) from each | 01:14 |
| ianw | is this something openafs on the client side likely masks for us by retrying or something? | 01:16 |
| auristor | You can make the FileAuditLog be a named pipe and then have a reader process to filter the requests you want to keep such as only those that fail | 01:16 |
| auristor | The openafs client does have an algorithm for retrying another fileserver if the fileserver returns VSALVAGE, VOOFLINE, VNOVOL and VMOVED. The algorithm is somewhat broken but it works in many cases. | 01:18 |
| *** dchen has quit IRC | 01:22 | |
| ianw | interesting. we don't see this on every vos release; i guess i could correlate from apache logs and update logs if requests for AFS files actually came in for the vos release period in times we did not see any error | 01:22 |
| *** dchen has joined #openstack-infra | 01:22 | |
| auristor | But looking at the "vos release" output that was included in http://lists.infradead.org/pipermail/linux-afs/2019-June/003078.html the RO on afs02 is taken offline and upd. During that update the afs02 copy is available. Then the afs01 RO is taken offline so it can be used as the source for updating the afs02 RO which is being updated in place. | 01:23 |
| auristor | If there was a 3rd fileserver, that third copy would be available during the update of afs02 and then afs02 would be put in service while afs03 is updated. Then afs03 and afs01 would be put in service | 01:25 |
| auristor | I will talk with dhowells tomorrow to review the failover and retry algorithms in kafs | 01:26 |
| *** bhavikdbavishi has joined #openstack-infra | 01:26 | |
| ianw | great, thanks, i thought a mail thread might be better for some more prolonged debugging. | 01:28 |
| auristor | if you have an estimate for how RPCs each of the fileservers process ? | 01:28 |
| *** bhavikdbavishi1 has joined #openstack-infra | 01:29 | |
| *** bhavikdbavishi has quit IRC | 01:31 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 01:31 | |
| ianw | is this from rxdebug helpful : Free packets: 1541/9103, packet reclaims: 905, calls: 1667017653, used FDs: 63 | 01:32 |
| ianw | uptime of 108 days ... | 01:32 |
| auristor | calls is the number of rpcs but its a 32-bit integer that rolls over | 01:33 |
| ianw | ahh, in that case i don't know :) it is at 1667019421 now | 01:33 |
| ianw | so 1768 in what, 2 minutes? | 01:34 |
| ianw | say ~14/s | 01:34 |
| auristor | can you query every minute and graph the stats in a shboard? | 01:36 |
| auristor | the reason I'm asking is to estimate the amount of space required for audit logging. Assume there will be one entry per rpc with an average length of 128 characters | 01:37 |
| auristor | assuming there was no rollover the audit data log for 100 days would be about 200GB | 01:39 |
| auristor | but I think that counter has wrapped | 01:44 |
| auristor | do you have xstat_fs_test from the openafs source tree? | 01:45 |
| *** dchen has quit IRC | 01:45 | |
| *** dchen has joined #openstack-infra | 01:46 | |
| auristor | "xstat_fs_test <fileserver> 2 -once" will return per RPC statistics | 01:46 |
| auristor | Linus has pulled the afs fixes into his source tree | 01:47 |
| *** gouthamr has quit IRC | 01:49 | |
| ianw | auristor: xstat_fs_test -> http://paste.openstack.org/show/753534/ | 01:51 |
| *** dmellado has quit IRC | 01:51 | |
| auristor | Those stats are useful to dashboard | 01:51 |
| *** gouthamr has joined #openstack-infra | 01:52 | |
| *** bhavikdbavishi has quit IRC | 01:53 | |
| *** dchen has quit IRC | 01:53 | |
| *** sgw has joined #openstack-infra | 01:54 | |
| *** dchen has joined #openstack-infra | 01:54 | |
| *** dmellado has joined #openstack-infra | 01:54 | |
| *** mriedem has quit IRC | 01:54 | |
| *** rajinir has quit IRC | 01:55 | |
| *** apetrich has quit IRC | 01:58 | |
| auristor | btw, the -cb parameter on afs01.dfw is too small based upon the xstat_fs_test collection set 3 output | 01:58 |
| *** happyhemant has quit IRC | 01:59 | |
| auristor | raise it to 2000000 | 01:59 |
| ianw | so after watching the stats for a while 1577.727273/minute ? ... https://docs.google.com/a/redhat.com/spreadsheets/d/e/2PACX-1vS6x3IpFlNTo_el355S0ooNDDjk-MjHm-5TnHrci43_HarqsDiDj15WZ5OuNs3iDnrRx70sy2AxWPal/pubhtml | 02:00 |
| *** gouthamr has quit IRC | 02:02 | |
| *** dmellado has quit IRC | 02:02 | |
| *** dmellado has joined #openstack-infra | 02:05 | |
| ianw | auristor: which stat was that based on? i'll update our docs before doing it | 02:05 |
| *** gouthamr has joined #openstack-infra | 02:06 | |
| auristor | If any GSS counters are non-zero it means the fileserver is breaking callback promises before they expire due to lack of space | 02:06 |
| *** gouthamr has quit IRC | 02:07 | |
| ianw | with about twice as much data now, average calls / minute sitting at 1846 | 02:07 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Raise callbacks for AFS server https://review.opendev.org/668078 | 02:14 |
| *** dmellado has quit IRC | 02:14 | |
| *** gouthamr has joined #openstack-infra | 02:14 | |
| ianw | auristor: ^ look about right? | 02:14 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Raise callbacks for AFS server https://review.opendev.org/668078 | 02:15 |
| *** dchen has quit IRC | 02:15 | |
| *** dchen has joined #openstack-infra | 02:16 | |
| *** gouthamr has quit IRC | 02:17 | |
| *** dmellado has joined #openstack-infra | 02:17 | |
| *** jesusaur has quit IRC | 02:18 | |
| *** gouthamr has joined #openstack-infra | 02:19 | |
| *** ricolin has joined #openstack-infra | 02:20 | |
| auristor | have a +1 | 02:21 |
| auristor | I can't access the spreadsheet your posted | 02:28 |
| *** gouthamr has quit IRC | 02:29 | |
| ianw | yeah, how silly i can't seem to get it to share with anyone !redhat.com ... it's one of these new fangled google work accounts | 02:31 |
| *** gouthamr has joined #openstack-infra | 02:35 | |
| *** dmellado has quit IRC | 02:35 | |
| ianw | auristor: raw numbers -> https://ethercalc.openstack.org/1sp6l7r0xb4b | 02:35 |
| *** dmellado has joined #openstack-infra | 02:38 | |
| *** gouthamr has quit IRC | 02:42 | |
| *** dmellado has quit IRC | 02:42 | |
| *** dmellado has joined #openstack-infra | 02:45 | |
| ianw | #status log afs01/02.dfw & afs01.ord restarted with greater -cb values: see https://review.opendev.org/668078 | 02:46 |
| openstackstatus | ianw: finished logging | 02:46 |
| *** zhangfei has joined #openstack-infra | 02:49 | |
| *** gouthamr has joined #openstack-infra | 02:49 | |
| *** dmellado has quit IRC | 02:52 | |
| *** dmellado has joined #openstack-infra | 02:54 | |
| *** gouthamr has quit IRC | 02:54 | |
| *** gouthamr has joined #openstack-infra | 02:58 | |
| fungi | see, i knew we decided to run a collaborative open source spreadsheet application for a good reason ;) | 03:01 |
| *** dmellado has quit IRC | 03:02 | |
| *** dmellado has joined #openstack-infra | 03:04 | |
| *** whoami-rajat has joined #openstack-infra | 03:05 | |
| *** gouthamr has quit IRC | 03:05 | |
| *** tonyb has quit IRC | 03:05 | |
| *** gouthamr has joined #openstack-infra | 03:10 | |
| *** dmellado has quit IRC | 03:11 | |
| *** dmellado has joined #openstack-infra | 03:13 | |
| openstackgerrit | Merged opendev/system-config master: Raise callbacks for AFS server https://review.opendev.org/668078 | 03:13 |
| *** zhurong has joined #openstack-infra | 03:22 | |
| *** gouthamr has quit IRC | 03:27 | |
| *** dmellado has quit IRC | 03:29 | |
| *** gouthamr has joined #openstack-infra | 03:31 | |
| *** dmellado has joined #openstack-infra | 03:32 | |
| *** bhavikdbavishi has joined #openstack-infra | 03:32 | |
| *** gouthamr has quit IRC | 03:33 | |
| *** xinranwang has joined #openstack-infra | 03:33 | |
| *** dmellado has quit IRC | 03:34 | |
| *** tonyb has joined #openstack-infra | 03:36 | |
| *** raukadah is now known as chandankumar | 03:37 | |
| *** gouthamr has joined #openstack-infra | 03:37 | |
| *** dmellado has joined #openstack-infra | 03:39 | |
| *** psachin has joined #openstack-infra | 03:40 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [dnm] move rsync updates to new mirror host https://review.opendev.org/667782 | 03:44 |
| *** gouthamr has quit IRC | 03:49 | |
| *** dmellado has quit IRC | 03:51 | |
| *** dmellado has joined #openstack-infra | 03:55 | |
| *** gouthamr has joined #openstack-infra | 03:58 | |
| *** gouthamr has quit IRC | 04:00 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [dnm] move rsync updates to new mirror host https://review.opendev.org/667782 | 04:00 |
| *** rh-jelabarre has quit IRC | 04:00 | |
| *** ykarel|away has joined #openstack-infra | 04:02 | |
| *** gouthamr has joined #openstack-infra | 04:06 | |
| *** ykarel|away is now known as ykarel | 04:08 | |
| *** dmellado has quit IRC | 04:09 | |
| *** gouthamr has quit IRC | 04:10 | |
| *** dmellado has joined #openstack-infra | 04:13 | |
| *** dmellado has quit IRC | 04:15 | |
| *** ramishra has joined #openstack-infra | 04:15 | |
| *** gouthamr has joined #openstack-infra | 04:16 | |
| *** gouthamr has quit IRC | 04:17 | |
| *** dmellado has joined #openstack-infra | 04:20 | |
| *** gouthamr has joined #openstack-infra | 04:22 | |
| *** udesale has joined #openstack-infra | 04:24 | |
| *** gouthamr has quit IRC | 04:25 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [dnm] move rsync updates to new mirror host https://review.opendev.org/667782 | 04:29 |
| *** gouthamr has joined #openstack-infra | 04:30 | |
| *** gouthamr has quit IRC | 04:35 | |
| *** pcaruana has joined #openstack-infra | 04:36 | |
| *** dmellado has quit IRC | 04:37 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [dnm] move rsync updates to new mirror host https://review.opendev.org/667782 | 04:39 |
| *** dmellado has joined #openstack-infra | 04:41 | |
| *** gouthamr has joined #openstack-infra | 04:41 | |
| *** gouthamr has quit IRC | 04:50 | |
| *** virendra-sharma has joined #openstack-infra | 04:52 | |
| *** dmellado has quit IRC | 04:56 | |
| *** gouthamr has joined #openstack-infra | 04:57 | |
| *** dmellado has joined #openstack-infra | 05:00 | |
| *** dchen has quit IRC | 05:00 | |
| *** dchen has joined #openstack-infra | 05:05 | |
| *** gouthamr has quit IRC | 05:06 | |
| *** dmellado has quit IRC | 05:11 | |
| *** gouthamr has joined #openstack-infra | 05:12 | |
| *** dmellado has joined #openstack-infra | 05:13 | |
| *** hwoarang has quit IRC | 05:16 | |
| *** hwoarang has joined #openstack-infra | 05:19 | |
| *** gouthamr has quit IRC | 05:19 | |
| *** gouthamr has joined #openstack-infra | 05:24 | |
| *** gouthamr has quit IRC | 05:31 | |
| *** kjackal has joined #openstack-infra | 05:32 | |
| *** dmellado has quit IRC | 05:35 | |
| *** gouthamr has joined #openstack-infra | 05:37 | |
| *** dmellado has joined #openstack-infra | 05:38 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Move rsync mirror updates to new opendev.org mirror-update host https://review.opendev.org/667782 | 05:39 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Move rsync mirror updates to new opendev.org mirror-update host https://review.opendev.org/667782 | 05:41 |
| *** xinranwang has quit IRC | 05:43 | |
| *** gouthamr has quit IRC | 05:44 | |
| *** jbadiapa has quit IRC | 05:46 | |
| *** dmellado has quit IRC | 05:48 | |
| *** dmellado has joined #openstack-infra | 05:51 | |
| *** gouthamr has joined #openstack-infra | 05:51 | |
| *** gouthamr has quit IRC | 05:52 | |
| ianw | auristor / fungi : ok, calling it on the rx stats tracking, etc. notes updated in https://etherpad.openstack.org/p/opendev-mirror-afs | 05:58 |
| *** dmellado has quit IRC | 05:58 | |
| *** gouthamr has joined #openstack-infra | 05:58 | |
| *** jtomasek has joined #openstack-infra | 05:58 | |
| ianw | i'll put it on my todo to track some of those more interesting stats; i think afsmon could be expanded | 06:00 |
| *** slaweq has joined #openstack-infra | 06:00 | |
| *** dmellado has joined #openstack-infra | 06:01 | |
| auristor | ok | 06:03 |
| *** gouthamr has quit IRC | 06:03 | |
| *** yamamoto has joined #openstack-infra | 06:06 | |
| *** yamamoto_ has joined #openstack-infra | 06:06 | |
| *** dmellado has quit IRC | 06:08 | |
| *** gouthamr has joined #openstack-infra | 06:08 | |
| *** lpetrut has joined #openstack-infra | 06:09 | |
| *** lpetrut has quit IRC | 06:09 | |
| *** lpetrut has joined #openstack-infra | 06:10 | |
| *** yamamoto has quit IRC | 06:10 | |
| *** dmellado has joined #openstack-infra | 06:11 | |
| *** gouthamr has quit IRC | 06:11 | |
| *** gouthamr has joined #openstack-infra | 06:16 | |
| *** gouthamr has quit IRC | 06:16 | |
| *** dpawlik has joined #openstack-infra | 06:18 | |
| *** gouthamr has joined #openstack-infra | 06:22 | |
| *** kopecmartin|off is now known as kopecmartin | 06:24 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Enable nodepool testing for opensuse 15.1 https://review.opendev.org/667537 | 06:27 |
| *** dmellado has quit IRC | 06:27 | |
| *** gouthamr has quit IRC | 06:29 | |
| *** dmellado has joined #openstack-infra | 06:30 | |
| openstackgerrit | Merged openstack/diskimage-builder master: debootstrap: make default network interface names configurable https://review.opendev.org/664936 | 06:30 |
| *** pgaxatte has joined #openstack-infra | 06:34 | |
| *** gouthamr has joined #openstack-infra | 06:36 | |
| *** dmellado has quit IRC | 06:44 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Move rsync mirror updates to new opendev.org mirror-update host https://review.opendev.org/667782 | 06:44 |
| *** gouthamr has quit IRC | 06:46 | |
| *** dmellado has joined #openstack-infra | 06:47 | |
| *** dchen has quit IRC | 06:51 | |
| *** dchen has joined #openstack-infra | 06:52 | |
| *** ccamacho has joined #openstack-infra | 06:53 | |
| *** dchen has quit IRC | 06:57 | |
| ianw | fungi: sent a note about the oom in the logs too @ https://www.redhat.com/archives/linux-cachefs/2019-June/msg00011.html . i feel like i discussed that before. probably a second tier problem to the hard-crashes, but just wanted a reference so we can come back to it | 06:57 |
| *** dmellado has quit IRC | 06:57 | |
| *** gouthamr has joined #openstack-infra | 06:58 | |
| *** dmellado has joined #openstack-infra | 07:00 | |
| *** jbadiapa has joined #openstack-infra | 07:04 | |
| *** gouthamr has quit IRC | 07:04 | |
| *** dmellado has quit IRC | 07:05 | |
| *** dmellado has joined #openstack-infra | 07:06 | |
| *** gouthamr has joined #openstack-infra | 07:12 | |
| *** dmellado has quit IRC | 07:13 | |
| yoctozepto | > <ianw> fungi: thanks for looking in on iad ... there's also another [Thu Jun 27 15:53:34 2019] kAFS: afs_dir_check_page(32f): bad magic 1/2 is 0000 which i've never seen before | 07:13 |
| yoctozepto | oh, so finally it has revealed itself - it's bad magic behind all of this, knew it since childhood | 07:13 |
| yoctozepto | ;D | 07:14 |
| *** gouthamr has quit IRC | 07:15 | |
| *** dmellado has joined #openstack-infra | 07:16 | |
| *** rcernin has quit IRC | 07:16 | |
| *** tesseract has joined #openstack-infra | 07:17 | |
| ianw | magic is one thing the whole thing is not short of! :) | 07:18 |
| yoctozepto | ianw: agreed! | 07:19 |
| *** dmellado has quit IRC | 07:21 | |
| *** dmellado has joined #openstack-infra | 07:21 | |
| *** dmellado has quit IRC | 07:21 | |
| *** gouthamr has joined #openstack-infra | 07:23 | |
| *** dmellado has joined #openstack-infra | 07:24 | |
| *** tosky has joined #openstack-infra | 07:25 | |
| *** bhavikdbavishi has quit IRC | 07:28 | |
| *** dmellado has quit IRC | 07:30 | |
| *** tesseract has quit IRC | 07:30 | |
| *** gouthamr has quit IRC | 07:30 | |
| *** tesseract has joined #openstack-infra | 07:32 | |
| *** witek has joined #openstack-infra | 07:34 | |
| *** dmellado has joined #openstack-infra | 07:34 | |
| *** jpich has joined #openstack-infra | 07:37 | |
| *** gouthamr has joined #openstack-infra | 07:39 | |
| *** dmellado has quit IRC | 07:39 | |
| *** dmellado has joined #openstack-infra | 07:44 | |
| *** apetrich has joined #openstack-infra | 07:45 | |
| *** gouthamr has quit IRC | 07:51 | |
| *** dmellado has quit IRC | 07:51 | |
| *** dmellado has joined #openstack-infra | 07:54 | |
| *** dmellado has quit IRC | 07:54 | |
| *** dmellado has joined #openstack-infra | 07:55 | |
| *** lpetrut has quit IRC | 07:56 | |
| *** gouthamr has joined #openstack-infra | 07:57 | |
| *** gouthamr has quit IRC | 07:59 | |
| *** ralonsoh has joined #openstack-infra | 08:01 | |
| *** dmellado has quit IRC | 08:03 | |
| *** gouthamr has joined #openstack-infra | 08:04 | |
| *** Emine has joined #openstack-infra | 08:08 | |
| *** psachin has quit IRC | 08:08 | |
| *** dmellado has joined #openstack-infra | 08:08 | |
| *** gouthamr has quit IRC | 08:09 | |
| *** lucasagomes has joined #openstack-infra | 08:09 | |
| *** Lucas_Gray has joined #openstack-infra | 08:11 | |
| *** AJaeger_ has joined #openstack-infra | 08:12 | |
| *** gouthamr has joined #openstack-infra | 08:14 | |
| *** xek has joined #openstack-infra | 08:14 | |
| *** AJaeger has quit IRC | 08:15 | |
| *** gouthamr has quit IRC | 08:17 | |
| *** dmellado has quit IRC | 08:19 | |
| *** dmellado has joined #openstack-infra | 08:22 | |
| *** dmellado has quit IRC | 08:23 | |
| *** gouthamr has joined #openstack-infra | 08:23 | |
| *** dmellado has joined #openstack-infra | 08:24 | |
| *** psachin has joined #openstack-infra | 08:24 | |
| *** gouthamr has quit IRC | 08:30 | |
| *** dmellado has quit IRC | 08:33 | |
| *** whoami-rajat has quit IRC | 08:34 | |
| *** dmellado has joined #openstack-infra | 08:34 | |
| *** dmellado has quit IRC | 08:37 | |
| *** dmellado has joined #openstack-infra | 08:39 | |
| *** gouthamr has joined #openstack-infra | 08:40 | |
| *** dmellado has quit IRC | 08:46 | |
| *** gouthamr has quit IRC | 08:47 | |
| *** dmellado has joined #openstack-infra | 08:50 | |
| *** dmellado has quit IRC | 08:50 | |
| *** dmellado has joined #openstack-infra | 08:51 | |
| *** pkopec has joined #openstack-infra | 08:52 | |
| *** gouthamr has joined #openstack-infra | 08:53 | |
| *** dmellado has quit IRC | 08:54 | |
| *** gouthamr has quit IRC | 08:56 | |
| *** dmellado has joined #openstack-infra | 08:57 | |
| *** jaosorior has quit IRC | 08:57 | |
| *** jaosorior has joined #openstack-infra | 09:00 | |
| *** bhavikdbavishi has joined #openstack-infra | 09:01 | |
| *** gouthamr has joined #openstack-infra | 09:02 | |
| *** dmellado has quit IRC | 09:02 | |
| *** bstinson has quit IRC | 09:05 | |
| *** gouthamr has quit IRC | 09:05 | |
| *** dmellado has joined #openstack-infra | 09:05 | |
| *** ykarel has quit IRC | 09:05 | |
| *** ykarel has joined #openstack-infra | 09:06 | |
| *** jbadiapa has quit IRC | 09:09 | |
| *** jbadiapa has joined #openstack-infra | 09:10 | |
| *** dmellado has quit IRC | 09:12 | |
| *** dmellado has joined #openstack-infra | 09:14 | |
| *** gouthamr has joined #openstack-infra | 09:14 | |
| *** psachin has quit IRC | 09:16 | |
| *** gouthamr has quit IRC | 09:19 | |
| *** pkopec has quit IRC | 09:20 | |
| *** ricolin has quit IRC | 09:21 | |
| *** bstinson has joined #openstack-infra | 09:21 | |
| *** pkopec has joined #openstack-infra | 09:21 | |
| *** gouthamr has joined #openstack-infra | 09:23 | |
| *** happyhemant has joined #openstack-infra | 09:23 | |
| *** gouthamr has quit IRC | 09:25 | |
| *** trident has quit IRC | 09:27 | |
| *** trident has joined #openstack-infra | 09:29 | |
| *** dmellado has quit IRC | 09:29 | |
| *** gouthamr has joined #openstack-infra | 09:32 | |
| *** dmellado has joined #openstack-infra | 09:32 | |
| *** dmellado has quit IRC | 09:32 | |
| *** pkopec has quit IRC | 09:33 | |
| *** dmellado has joined #openstack-infra | 09:35 | |
| *** pkopec has joined #openstack-infra | 09:35 | |
| *** gouthamr has quit IRC | 09:37 | |
| *** ykarel is now known as ykarel|lunch | 09:40 | |
| *** gouthamr has joined #openstack-infra | 09:42 | |
| *** gouthamr has quit IRC | 09:43 | |
| *** pkopec has quit IRC | 09:44 | |
| *** dmellado has quit IRC | 09:44 | |
| *** dmellado has joined #openstack-infra | 09:47 | |
| *** jaosorior has quit IRC | 09:49 | |
| *** dmellado has quit IRC | 09:54 | |
| *** gouthamr has joined #openstack-infra | 09:54 | |
| *** pkopec has joined #openstack-infra | 09:55 | |
| *** dmellado has joined #openstack-infra | 09:56 | |
| *** emine__ has joined #openstack-infra | 09:57 | |
| *** AJaeger_ is now known as AJaeger | 09:57 | |
| *** xek has quit IRC | 09:59 | |
| *** Emine has quit IRC | 10:00 | |
| *** gouthamr has quit IRC | 10:00 | |
| *** dmellado has quit IRC | 10:02 | |
| *** ykarel|lunch is now known as ykarel|meeting | 10:03 | |
| *** dmellado has joined #openstack-infra | 10:06 | |
| *** gouthamr has joined #openstack-infra | 10:06 | |
| *** ociuhandu has joined #openstack-infra | 10:07 | |
| *** virendra-sharma has quit IRC | 10:07 | |
| *** kjackal has quit IRC | 10:09 | |
| *** kjackal has joined #openstack-infra | 10:10 | |
| *** dmellado has quit IRC | 10:11 | |
| *** gfidente has joined #openstack-infra | 10:12 | |
| *** ociuhandu has quit IRC | 10:14 | |
| *** dmellado has joined #openstack-infra | 10:15 | |
| *** ociuhandu has joined #openstack-infra | 10:17 | |
| *** gouthamr has quit IRC | 10:18 | |
| *** dmellado has quit IRC | 10:22 | |
| *** dmellado has joined #openstack-infra | 10:24 | |
| *** gouthamr has joined #openstack-infra | 10:26 | |
| *** ccamacho has quit IRC | 10:30 | |
| *** ccamacho has joined #openstack-infra | 10:31 | |
| *** gouthamr has quit IRC | 10:35 | |
| *** jpena|off is now known as jpena | 10:40 | |
| *** gouthamr has joined #openstack-infra | 10:41 | |
| *** gouthamr has quit IRC | 10:43 | |
| *** dmellado has quit IRC | 10:44 | |
| *** dmellado has joined #openstack-infra | 10:49 | |
| *** whoami-rajat has joined #openstack-infra | 10:51 | |
| *** gouthamr has joined #openstack-infra | 10:52 | |
| *** bhavikdbavishi has quit IRC | 10:56 | |
| *** gouthamr has quit IRC | 10:57 | |
| *** jpena is now known as jpena|lunch | 10:58 | |
| *** yamamoto_ has quit IRC | 11:02 | |
| *** dmellado has quit IRC | 11:04 | |
| *** ykarel|meeting has quit IRC | 11:05 | |
| *** dmellado has joined #openstack-infra | 11:05 | |
| *** ykarel|meeting has joined #openstack-infra | 11:05 | |
| *** gouthamr has joined #openstack-infra | 11:06 | |
| yoctozepto | any ask.o.o moderators lurking in here? I would like to ask how long it would take for my answer to get through https://ask.openstack.org/en/answers/122773/revisions/ thanks in advance | 11:07 |
| *** ccamacho has quit IRC | 11:11 | |
| *** yamamoto has joined #openstack-infra | 11:13 | |
| *** kjackal has quit IRC | 11:24 | |
| *** goldyfruit has quit IRC | 11:33 | |
| *** kjackal has joined #openstack-infra | 11:33 | |
| *** yikun has quit IRC | 11:33 | |
| *** Lucas_Gray has quit IRC | 11:38 | |
| *** Lucas_Gray has joined #openstack-infra | 11:41 | |
| fungi | yoctozepto: would you like to be a moderator? | 11:43 |
| *** EmilienM is now known as EvilienM | 12:02 | |
| *** rh-jelabarre has joined #openstack-infra | 12:03 | |
| openstackgerrit | Marcin Juszkiewicz proposed opendev/system-config master: do backports for Debian 'buster' https://review.opendev.org/668157 | 12:05 |
| *** ykarel|meeting is now known as ykarel | 12:05 | |
| *** ykarel is now known as ykarel|afk | 12:13 | |
| *** dpawlik has quit IRC | 12:15 | |
| *** dpawlik has joined #openstack-infra | 12:20 | |
| *** udesale has quit IRC | 12:22 | |
| *** udesale has joined #openstack-infra | 12:23 | |
| *** jpena|lunch is now known as jpena | 12:23 | |
| *** iurygregory has joined #openstack-infra | 12:24 | |
| *** electrofelix has joined #openstack-infra | 12:29 | |
| *** tdasilva has joined #openstack-infra | 12:35 | |
| *** rlandy has joined #openstack-infra | 12:37 | |
| *** jbadiapa has quit IRC | 12:37 | |
| slaweq | fungi: hi | 12:39 |
| slaweq | fungi:sorry for bothering You again but I have one more question related to tobiko | 12:39 |
| slaweq | fungi: can You also remove docs from https://docs.openstack.org/tobiko/latest/ ? | 12:39 |
| *** pcaruana has quit IRC | 12:40 | |
| *** goldyfruit has joined #openstack-infra | 12:40 | |
| *** pcaruana has joined #openstack-infra | 12:41 | |
| *** ekultails has joined #openstack-infra | 12:44 | |
| *** lseki has joined #openstack-infra | 12:48 | |
| openstackgerrit | Sean McGinnis proposed openstack/project-config master: Retire the release-schedule-generator project https://review.opendev.org/668045 | 12:49 |
| *** rfarr_ has joined #openstack-infra | 12:51 | |
| *** iurygregory has quit IRC | 12:52 | |
| *** rfarr__ has joined #openstack-infra | 12:53 | |
| *** eharney has joined #openstack-infra | 12:53 | |
| *** rfarr_ has quit IRC | 12:56 | |
| *** pkopec has quit IRC | 13:02 | |
| *** pkopec has joined #openstack-infra | 13:03 | |
| openstackgerrit | Flavio Percoco proposed zuul/nodepool master: Allow nodepool for using in-cluster configs https://review.opendev.org/668172 | 13:11 |
| fungi | #status log deleted /afs/.openstack.org/docs/tobiko at slaweq's request as a member of https://review.opendev.org/#/admin/groups/tobiko-core | 13:13 |
| openstackstatus | fungi: finished logging | 13:13 |
| slaweq | fungi: thx a lot | 13:13 |
| fungi | no problem | 13:13 |
| *** lpetrut has joined #openstack-infra | 13:23 | |
| *** mriedem has joined #openstack-infra | 13:24 | |
| *** pkopec has quit IRC | 13:24 | |
| *** sthussey has joined #openstack-infra | 13:25 | |
| *** pkopec has joined #openstack-infra | 13:25 | |
| *** yamamoto has quit IRC | 13:31 | |
| *** aaronsheffield has joined #openstack-infra | 13:31 | |
| yoctozepto | > <fungi> yoctozepto: would you like to be a moderator? | 13:36 |
| yoctozepto | are you a work-seller or something? ;D | 13:36 |
| yoctozepto | will think about it, for now I just wanted the answer to pass :D | 13:37 |
| fungi | yoctozepto: sure, and no just reminding that the way things get moderated more quickly is by having more moderators ;) | 13:39 |
| * fungi is technically a moderator by way of being an administrator of the service for purposes of basic troubleshooting, but not really an active moderator in practice | 13:39 | |
| *** jbadiapa has joined #openstack-infra | 13:42 | |
| yoctozepto | fungi: tricky | 13:42 |
| fungi | yoctozepto: the other sneaky thing about that service is that if you answer enough questions it makes you a moderator automatically... be careful! ;) | 13:45 |
| fungi | (moderator privs are automatically granted once a certain karma threshold is reached, i forget what number exactly) | 13:45 |
| yoctozepto | fungi: then we may skip some steps I guess ;D | 13:49 |
| yoctozepto | if it's going to get me anyway | 13:49 |
| yoctozepto | I thought there is a chirograph for this | 13:49 |
| openstackgerrit | Merged opendev/yaml2ical master: Add DTSTAMP and UID values to meeting instances https://review.opendev.org/667961 | 13:52 |
| *** trident has quit IRC | 13:56 | |
| *** iurygregory has joined #openstack-infra | 13:58 | |
| *** trident has joined #openstack-infra | 13:58 | |
| fungi | yoctozepto: done. if nothing else, your future answers won't go straight into the moderation queue, but the other mods probably appreciate if you check whether there's any pending items in the queue from time to time (drop-down from your username at the top when you're logged in, though it won't show the link to the queue if there's nothing waiting) | 14:03 |
| yoctozepto | fungi: k, thanks, think I will be looking out for kolla stuff, doing some good PR ;D | 14:04 |
| *** yamamoto has joined #openstack-infra | 14:05 | |
| fungi | yeah, really "moderating" just means approving anything which looks like an actual question/answer and occasionally discarding things which looks like spam | 14:07 |
| fungi | doesn't mean you have to answer anything you approve | 14:08 |
| *** ykarel|afk is now known as ykarel | 14:08 | |
| yoctozepto | fungi: I didn't even dream of it | 14:09 |
| *** dpawlik has quit IRC | 14:10 | |
| *** liuyulong has joined #openstack-infra | 14:10 | |
| yoctozepto | ok, the link's not there so we seem clean | 14:10 |
| *** lpetrut has quit IRC | 14:11 | |
| *** jbadiapa has quit IRC | 14:12 | |
| *** jbadiapa has joined #openstack-infra | 14:12 | |
| *** yamamoto has quit IRC | 14:12 | |
| dirk | infra-root: is it normal that the fedora mirroring is broken? http://grafana.openstack.org/d/ACtl1JSmz/afs?orgId=1&from=now-90d&to=now&fullscreen&panelId=23 | 14:23 |
| *** mattw4 has joined #openstack-infra | 14:30 | |
| donnyd | clarkb will be available today to do some testing with my infra? | 14:35 |
| donnyd | will *you* | 14:35 |
| corvus | dirk: i don't think those two lines are supposed to be colinear, but they do seem to be for much of the time. | 14:36 |
| dirk | corvus: well, http://grafana.openstack.org/d/ACtl1JSmz/afs?orgId=1&from=now-90d&to=now&fullscreen&panelId=15 is pretty telling | 14:36 |
| corvus | urg. i broke my local afs client setup testing the kafs stuff, and haven't rebooted back into openafs... | 14:37 |
| corvus | i guess i can use mirror-update | 14:37 |
| clarkb | donnyd: yes, my day should start in about an hour | 14:37 |
| donnyd | ok thanks | 14:37 |
| corvus | dirk: mirror.fedora.readonly 800000000 799973784 100%<< 33% <<WARNING | 14:38 |
| corvus | how big should it be? :) | 14:38 |
| corvus | (also, wow, it had a 100GB jump in 1 day) | 14:39 |
| corvus | should we make it 1tb? | 14:39 |
| clarkb | a good chunk of that is the atomic images, which I think we may not need to mirror in its entirety and instead choose the one or two images are usera need (but that is more worl than bumping volume size) | 14:40 |
| *** lpetrut has joined #openstack-infra | 14:46 | |
| *** xek has joined #openstack-infra | 14:48 | |
| *** panda has quit IRC | 14:48 | |
| *** panda has joined #openstack-infra | 14:48 | |
| fungi | just a heads up, i'm disappearing in a bit to run some pre-travel errands but should return in time for the storyboard bug scrub at 16:15z | 14:48 |
| fungi | i'll be in and out over the rest of the day so may be slow to answer | 14:49 |
| *** pgaxatte has quit IRC | 14:51 | |
| *** pkopec has quit IRC | 14:53 | |
| *** pkopec has joined #openstack-infra | 14:57 | |
| *** mattw4 has quit IRC | 15:04 | |
| *** lpetrut has quit IRC | 15:05 | |
| *** yamamoto has joined #openstack-infra | 15:08 | |
| *** bobh has joined #openstack-infra | 15:09 | |
| donnyd | clarkb what do I need to get setup first? | 15:10 |
| *** whoami-rajat has quit IRC | 15:11 | |
| *** rajinir has joined #openstack-infra | 15:12 | |
| clarkb | donnyd: if we can get an openstackci tenant/project/user to run the mirror in and a openstackzuul tenant/project/user for the test nodes that is probably step zero. Then we can work on quotas and spinning of some initial instances | 15:15 |
| donnyd | Does that name need to be specific | 15:15 |
| *** igordc has joined #openstack-infra | 15:17 | |
| clarkb | no, those names are what we typically call them but it will go in a clouds.yaml so the two names can be more arbitrary | 15:17 |
| *** whoami-rajat has joined #openstack-infra | 15:19 | |
| *** jistr is now known as jistr|off | 15:20 | |
| donnyd | ok | 15:22 |
| donnyd | Whats next | 15:23 |
| openstackgerrit | Paul Belanger proposed opendev/system-config master: Retire pabelanger as infra-root https://review.opendev.org/668192 | 15:23 |
| clarkb | usually once that is in place we spin up a mirror and a test node and do some sanity checking that stuff like nova unittests and devstack work | 15:24 |
| clarkb | that may require adding or tweaking nova flavors | 15:25 |
| clarkb | then once sanity checks look good we update the nodepool config to build images in that cloud then to launch test nodes | 15:26 |
| *** xek has quit IRC | 15:26 | |
| *** ykarel is now known as ykarel|away | 15:26 | |
| pabelanger | infra-root: I think the time as come for me to stepdown from day to day duties as infra-root^, I send a message to ML too. I still plan to be here, just that I don't have the bandwidth any more for day to day infra-root duties :( | 15:26 |
| clarkb | donnyd: if we get account details shared I can start on spinning up a mirror node | 15:28 |
| donnyd | sure can | 15:28 |
| *** jpena is now known as jpena|off | 15:31 | |
| *** jpena|off has quit IRC | 15:31 | |
| *** bhavikdbavishi has joined #openstack-infra | 15:35 | |
| *** chandankumar is now known as raukadah | 15:40 | |
| *** jistr|off is now known as jistr | 15:42 | |
| corvus | pabelanger: i, uh, have left a friendly -1 with a suggestion on your removal request :) | 15:50 |
| *** ramishra has quit IRC | 15:52 | |
| *** zhangfei has quit IRC | 16:01 | |
| *** emine__ has quit IRC | 16:04 | |
| *** mattw4 has joined #openstack-infra | 16:08 | |
| clarkb | heh I've just realized I said openstackci and openstackzuul instead of opendevci and opendevzuul. As mentioned before the names don't matter so much because they are hidden behind logical clouds.yaml config. Oh well | 16:10 |
| clarkb | also TIL openstackclient doesn't default to the one cloud you have defined in a clouds.yaml if there is only one cloud specific | 16:10 |
| *** mattw4 has quit IRC | 16:12 | |
| pabelanger | corvus: thanks! replied with some thoughts | 16:14 |
| *** mattw4 has joined #openstack-infra | 16:15 | |
| *** efried is now known as fried_rolls | 16:15 | |
| *** ociuhandu has quit IRC | 16:16 | |
| *** yamamoto has quit IRC | 16:16 | |
| *** yamamoto has joined #openstack-infra | 16:17 | |
| clarkb | infra-root I'm going to take the secrets lock and add donnyd's cloud details | 16:18 |
| *** mriedem is now known as mriedem_away | 16:18 | |
| corvus | pabelanger: ah, excellent point. i'm okay with the status quo of access to all servers, but maybe shell access to the zuul/nodepool cluster only would be appropriate if we think that's better. | 16:18 |
| *** lucasagomes has quit IRC | 16:19 | |
| *** Lucas_Gray has quit IRC | 16:19 | |
| corvus | maybe we can get a quorum of infra-root together next week and mull over it some more | 16:20 |
| corvus | (since it seems we're sort of light today) | 16:20 |
| pabelanger | Yup, not rush for removal, could even bring it up at meeting next week | 16:20 |
| *** udesale has quit IRC | 16:26 | |
| *** jpich has quit IRC | 16:28 | |
| *** jtomasek has quit IRC | 16:35 | |
| clarkb | infra-root I've updated the secrets file with fortnebula cloud and contact details | 16:35 |
| clarkb | the passwords were changed from those that were given to me | 16:35 |
| clarkb | I'll get a change up for clouds.yaml updates shortly | 16:35 |
| *** diablo_rojo has joined #openstack-infra | 16:36 | |
| *** iurygregory has quit IRC | 16:38 | |
| *** cmurphy is now known as cmorpheus | 16:41 | |
| corvus | clarkb, donnyd: \o/ thanks! | 16:41 |
| *** kmalloc is now known as needscoffee | 16:47 | |
| AJaeger | config-core, could you review https://review.opendev.org/#/c/667900/ and https://review.opendev.org/531829 , please? These are some cleanups... | 16:50 |
| *** gfidente has quit IRC | 16:52 | |
| openstackgerrit | Jeff Liu proposed zuul/zuul-operator master: Add playbooks folder and zuul job that installs k8s https://review.opendev.org/668029 | 16:54 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Add clouds.yaml entries for fortnebula cloud https://review.opendev.org/668206 | 16:57 |
| clarkb | donnyd: corvus ^ I've not yet updated hiera/host vars on bridge but maybe give that a quick look to make sure I havne't missed anything obvious | 16:58 |
| clarkb | fungi: ^ you too if around | 16:58 |
| clarkb | donnyd: I've realized that we may want to sync up with ianw on the plan for the mirror node as he is trying to transition us to bionic mirror nodes. Given that maybe I'll start sanity checking with a test node and no mirror node to run nova unittests and devstack | 16:59 |
| clarkb | in either case getting the clouds.yaml in place will drastically simplify the owrk as I'll be able to use our normal tooling | 16:59 |
| clarkb | I need to step out for a bit but I'll be back to add entries to hiera/host vars and maybe approve that chnage if it look sgood to others | 17:03 |
| donnyd | ok thanks | 17:07 |
| fungi | okay, back and catching up on highlights (actually got back just in time for the sb bug scrub but have been focusing on that until now) | 17:09 |
| *** ralonsoh has quit IRC | 17:18 | |
| *** ricolin has joined #openstack-infra | 17:19 | |
| openstackgerrit | James E. Blair proposed zuul/zuul master: WIP: Add extra tenant config https://review.opendev.org/668214 | 17:21 |
| openstackgerrit | Adam Coldrick proposed opendev/storyboard-webclient master: Render user-submitted links with nofollow attribute https://review.opendev.org/648322 | 17:23 |
| *** e0ne has joined #openstack-infra | 17:26 | |
| *** mriedem_away is now known as mriedem | 17:27 | |
| *** electrofelix has quit IRC | 17:30 | |
| openstackgerrit | Kendall Nelson proposed opendev/storyboard master: Link development.rst to contributing.rst https://review.opendev.org/645960 | 17:30 |
| donnyd | I am still hammering on getting ipv6 functioning correctly | 17:31 |
| *** ykarel|away has quit IRC | 17:34 | |
| fungi | problems with neutron config or network gear? | 17:36 |
| *** davecore has joined #openstack-infra | 17:37 | |
| openstackgerrit | Kendall Nelson proposed opendev/storyboard master: Link development.rst to contributing.rst https://review.opendev.org/645960 | 17:37 |
| *** witek has quit IRC | 17:40 | |
| openstackgerrit | Kendall Nelson proposed opendev/storyboard master: Add StoryBoard Story Tags to Docs https://review.opendev.org/667482 | 17:42 |
| *** ricolin has quit IRC | 17:45 | |
| *** ykarel|away has joined #openstack-infra | 17:49 | |
| *** whoami-rajat has quit IRC | 17:51 | |
| *** e0ne has quit IRC | 18:12 | |
| *** e0ne has joined #openstack-infra | 18:13 | |
| *** tesseract has quit IRC | 18:13 | |
| *** e0ne has quit IRC | 18:16 | |
| clarkb | taking he hiera/hostfiles lock now | 18:16 |
| clarkb | er maybe I'll fix my change's -1 first | 18:16 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Add clouds.yaml entries for fortnebula cloud https://review.opendev.org/668206 | 18:17 |
| clarkb | ok hiera/host vars lock released | 18:22 |
| clarkb | the data is up to date and 668206 should be ready for approval | 18:22 |
| *** e0ne has joined #openstack-infra | 18:26 | |
| *** e0ne has quit IRC | 18:28 | |
| mnaser | is there any issues with ovh-gra1 | 18:29 |
| mnaser | http://logs.openstack.org/03/668003/2/gate/openstack-tox-docs/bec3243/job-output.txt.gz | 18:30 |
| mnaser | ssh: connect to host 213.32.77.161 port 22: No route to host\r\nrsync: connection unexpectedly closed (0 bytes received so far) [Receiver]\nrsync error: unexplained error (code 255) at io.c(226) [Receiver=3.1.1] | 18:30 |
| clarkb | mnaser: normally that is a side effect of zuul memory issues causing zookeeper client disconnects which results in cleanup of the test nodes. But zuul scheduler memory looks fine http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=64792&rra_id=all | 18:32 |
| clarkb | possible that something else has caused zookeeper network connectivity issues. Let me see if grep shows anything in the zuul logs | 18:32 |
| mnaser | looks like the fetch-output worked fine until the rsync | 18:32 |
| mnaser | so within 3 seconds it disappeared there | 18:32 |
| clarkb | grep -i kazoo doesn't show naything in the zuul schedulre log | 18:34 |
| clarkb | corvus: ^ am I grepping for the wrong thing or does that mean we hae to consider a different failure mode here? | 18:34 |
| * clarkb pulls up nodepool logs for that node | 18:36 | |
| clarkb | 2019-06-28 18:15:04,076 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0008509452 (state: used, allocated_to: 200-0004128157) | 18:37 |
| clarkb | that seems to happen after the rsync failure so I think that implies this is a different fialure mode | 18:38 |
| clarkb | mnaser: any idea if that is a persistent issue? if not maybe it was a short network hiccup in that region or between regions? | 18:39 |
| clarkb | I guess we can keep an eye on it and if the problem continues or gets worse we can loop in ovh | 18:39 |
| mnaser | clarkb: it was the first one but i usually speak up early :P | 18:39 |
| clarkb | ya that particular failure type is often tied toa global zuul memory issue, just doesn't seem to be this time | 18:39 |
| clarkb | so hard to pin down without more data (and more failures == more data :( ) | 18:40 |
| mnaser | i'll recheck and if i see others ill loopback | 18:40 |
| openstackgerrit | Merged openstack/project-config master: Remove release-openstack-python-without-pypi https://review.opendev.org/531829 | 18:41 |
| clarkb | AJaeger: ^ theres one done | 18:41 |
| clarkb | corvus: https://review.opendev.org/#/c/668206/ zuul is happy there now and hiera/hostvars are in place if you want to reack that | 18:42 |
| fungi | logan-: network connectivity still looks pretty consistently broken, if you've missed the earlier mentions: http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=64934 | 18:44 |
| fungi | today it seems it's been unreachable >90% of the time | 18:45 |
| logan- | yep fungi, thanks. I saw, just haven't had a chance to track it down yet | 18:45 |
| fungi | no worries, just wanted to make sure you're aware! | 18:45 |
| logan- | yeah its crazy, really weird | 18:45 |
| *** pkopec has quit IRC | 18:45 | |
| *** pkopec has joined #openstack-infra | 18:48 | |
| *** ykarel|away has quit IRC | 18:52 | |
| AJaeger | thanks, clarkb | 18:53 |
| *** EvilienM is now known as EmilienM | 19:05 | |
| *** tdasilva has quit IRC | 19:06 | |
| corvus | clarkb: hah, of course we have 2 copies of that :) | 19:12 |
| *** whoami-rajat has joined #openstack-infra | 19:14 | |
| *** yamamoto has quit IRC | 19:14 | |
| *** diablo_rojo has quit IRC | 19:23 | |
| openstackgerrit | Merged opendev/system-config master: Add clouds.yaml entries for fortnebula cloud https://review.opendev.org/668206 | 19:31 |
| clarkb | corvus: thanks | 19:31 |
| clarkb | once that is in place on the nodepool nodes I think my next step is to upload a current bionic image, then boot a test node | 19:32 |
| clarkb | donnyd: ^ let me know if yo uthink that will conflict with your network debugging | 19:32 |
| clarkb | gitea06 looked a lot happier today too (load average wise) | 19:32 |
| donnyd | no, you can push forward | 19:33 |
| donnyd | I have it about figured out | 19:33 |
| *** dklyle has quit IRC | 19:36 | |
| donnyd | logan - Do you think you can give me a hand with T/S ipv6 issues | 19:40 |
| donnyd | When i setup slaac from the openstack router, the instance gets no ip address | 19:42 |
| dustinc | can anyone here help me understand how to request CI rechecks? is there a doc somewhere? (google failed me) | 19:48 |
| dustinc | specifically for https://review.opendev.org/#/c/659690/ | 19:48 |
| clarkb | dustinc: if you leave a comment of 'recheck' on the top level of the change it will rerun the tests | 19:48 |
| clarkb | I'm guessing we document that in the manual /me looks | 19:49 |
| *** fried_rolls is now known as efried | 19:49 | |
| dustinc | is there a way to get it to just rerun specific tests? | 19:49 |
| donnyd | logan NM, its working with stateless dhcpv6 | 19:50 |
| clarkb | dustinc: no, we've found in the past that people have been able to recheck changes that are broken and break testing enough times they sneak through and want to avoid being able to lock in results | 19:50 |
| dustinc | that makes sense. thanks for your helkp | 19:50 |
| dustinc | help* | 19:50 |
| *** yamamoto has joined #openstack-infra | 19:50 | |
| clarkb | dustinc: https://docs.openstack.org/infra/manual/developers.html#automated-testing list entry 6 | 19:51 |
| clarkb | that might be more hidden than it should be :/ | 19:51 |
| *** mattw4 has quit IRC | 19:51 | |
| dustinc | once I toggled CI comments on the review I noticed that there was a link in the results comment..oops.. :) | 19:51 |
| dustinc | (to that exact doc) | 19:52 |
| *** diablo_rojo has joined #openstack-infra | 19:52 | |
| logan- | donnyd just saw your message since it didn't trigger my nick highlight, but glad you got it working :) | 19:57 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Add extra tenant config https://review.opendev.org/668214 | 20:00 |
| *** yamamoto has quit IRC | 20:01 | |
| *** mattw4 has joined #openstack-infra | 20:02 | |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Add extra-config-paths tenant config option https://review.opendev.org/668214 | 20:02 |
| corvus | clarkb: you might express your concern about that being more hidden than it should be with a review of this change: https://review.opendev.org/655783 | 20:08 |
| clarkb | fungi: ^ did you muster enough js fu to make the change you talked about there? | 20:09 |
| corvus | (fwiw, any of thosse ideas would be fine with me; though i would be perfectly comfortable with renaming the button for now and if anyone later feels like contributing an improvement to only show the last, great) | 20:11 |
| clarkb | ya I think renaming the button would be sufficient | 20:13 |
| clarkb | should we just push a quick patch for that? | 20:13 |
| clarkb | though rereading this cahnge I think this will break the summary table for Zuul? | 20:15 |
| clarkb | is that a desired outcome? | 20:15 |
| *** diablo_rojo has quit IRC | 20:16 | |
| corvus | nope | 20:16 |
| clarkb | the ciRegex is used to find CI comments and that is then used to both build the summary table and hide comments I think | 20:17 |
| clarkb | (my js isn't great) | 20:17 |
| corvus | in that case, given what would be needed to fix it properly, implementing the 'show last' suggestion might not be much harder | 20:18 |
| donnyd | ok, it should all be working now | 20:18 |
| *** dklyle has joined #openstack-infra | 20:18 | |
| clarkb | donnyd: cool, we just missed the cutoff for the previous round of ansible on the host with the images so will be about 15 minutes before I can start an image upload but will do so then | 20:19 |
| donnyd | if logan- has a few minutes to review my ipv6 setup that would be great | 20:20 |
| logan- | donnyd: sure | 20:20 |
| *** slaweq has quit IRC | 20:23 | |
| clarkb | corvus: fungi: I'm deciphering js now to be more sure of that concern | 20:24 |
| *** bhavikdbavishi has quit IRC | 20:25 | |
| clarkb | corvus: fungi left a comment on the change with my findings | 20:28 |
| *** diablo_rojo has joined #openstack-infra | 20:32 | |
| fungi | clarkb: no, i haven't had time to look into how to go about trying to match on all but the last comment for a given account | 20:38 |
| openstackgerrit | James E. Blair proposed openstack/project-config master: Adjust jobs running on zuul-jobs https://review.opendev.org/668248 | 20:38 |
| clarkb | fungi: want to check my comment for the new concern to see if I've read js properly? | 20:38 |
| fungi | i think you're right, yeah | 20:38 |
| openstackgerrit | James E. Blair proposed openstack/openstack-zuul-jobs master: Remove upload-git-mirror test job https://review.opendev.org/668249 | 20:39 |
| corvus | AJaeger, clarkb: ^ those 2 changes are my agreement with AJaeger in #zuul :) | 20:40 |
| clarkb | looking | 20:41 |
| openstackgerrit | Merged zuul/zuul-jobs master: Split job definitions into multiple files https://review.opendev.org/668040 | 20:46 |
| openstackgerrit | Merged zuul/zuul-jobs master: Add a simple test of the install-nodejs role https://review.opendev.org/668043 | 20:46 |
| clarkb | current ubuntu bionic image on nb02 is being uploaded to fortnebula cloud now | 20:50 |
| clarkb | once that is in I can test booting an instance manually then running something like nova unittests on it | 20:50 |
| openstackgerrit | Merged zuul/zuul-jobs master: Move upload-git-mirror test job in-repo https://review.opendev.org/668050 | 20:55 |
| *** mattmceuen has joined #openstack-infra | 20:55 | |
| mattmceuen | Hi folks, happy Friday! Whenever someone gets a minute, I'd appreciate if I could be added as a seed member of the airship-docs-core gerrit group | 20:56 |
| corvus | mattmceuen: on it | 20:56 |
| mattmceuen | ty corvus! | 20:56 |
| corvus | mattmceuen: att.com address? | 20:57 |
| mattmceuen | yep that's fine -- mm9745@att.com | 20:57 |
| corvus | mattmceuen: er, the one gerrit has for you is different, but it's the only matt mceuen with an att.com address, so i probably got the right account. you should be gtg. :) | 20:58 |
| mattmceuen | corvus: I have all the buttons enabled I need! appreciate your help sir! | 20:58 |
| corvus | mattmceuen: np | 20:58 |
| clarkb | speaking of buttons https://arstechnica.com/science/2019/06/behind-the-scenes-at-nasas-newly-restored-historic-apollo-mission-control/ | 21:00 |
| clarkb | Now I need an excuse to go to houston | 21:00 |
| mattmceuen | oh wow! | 21:00 |
| donnyd | that is pretty awesome | 21:00 |
| donnyd | The only part is dislike about raw images is upload time... | 21:03 |
| clarkb | donnyd: indeed | 21:03 |
| fungi | i'm going to build my next workstation stand to look like a bank of those consoles | 21:03 |
| fungi | or maybe an arcade, i can't decide | 21:03 |
| openstackgerrit | Merged openstack/project-config master: Adjust jobs running on zuul-jobs https://review.opendev.org/668248 | 21:04 |
| *** kjackal has quit IRC | 21:05 | |
| clarkb | donnyd: note that if you end up fiddling with our images too, that they require config-drive on the instance and root is what ends up configured with ssh keys | 21:06 |
| clarkb | (we don't use cloud-init, we use glean instead which is far simpler but comes with a few of those restrictions) | 21:06 |
| donnyd | I wish i had time to fiddle... too many irons, not enough fire | 21:07 |
| clarkb | no worries | 21:07 |
| clarkb | upload is done. Figuring out a boot next | 21:08 |
| clarkb | oh I remember what I was forgetting now | 21:08 |
| clarkb | cloud launcher config | 21:08 |
| *** diablo_rojo has quit IRC | 21:10 | |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Add fornebula cloud to cloud launcher https://review.opendev.org/668251 | 21:11 |
| clarkb | I'll manually upload my key now too so I don't have to wait for that | 21:11 |
| *** slaweq has joined #openstack-infra | 21:11 | |
| donnyd | clarkb: thanks for sticking with this all day. Much appreciated | 21:12 |
| *** kopecmartin has quit IRC | 21:13 | |
| clarkb | clarkb-test1 is booting now | 21:15 |
| clarkb | donnyd: and thank you for the generous offer of test resources | 21:15 |
| *** slaweq has quit IRC | 21:15 | |
| *** pcaruana has quit IRC | 21:16 | |
| *** rfolco is now known as rfolco|off | 21:16 | |
| donnyd | do you think local mirrors would help speed things up a bit (.deb / .rpm) | 21:18 |
| fungi | that's basically what our static instance there will be | 21:19 |
| donnyd | Ah I C | 21:19 |
| fungi | we stick one in every provider region where we run jobs, to improve response time, even out connectivity hiccups, and reduce pressure on the external network | 21:19 |
| donnyd | yea that makes lots of sense | 21:20 |
| *** dpawlik has joined #openstack-infra | 21:20 | |
| clarkb | reduces pressure on the ipv4 nat too (if only github and docker hub could ipv6) | 21:20 |
| fungi | generally a mix of apache proxy caches with some cache-busting overrides to make the caches more effective and also some afs-backed mirrors we build up via rsync or reprepro | 21:21 |
| donnyd | there are only two tenant's (currently) in this "cloud" | 21:21 |
| fungi | since afs also does client-side caching | 21:21 |
| clarkb | fwiw I got into the instance just fine over ipv6 and am installing deps for nova unittests | 21:21 |
| clarkb | so far everything looks great | 21:21 |
| donnyd | nice | 21:22 |
| *** dpawlik has quit IRC | 21:22 | |
| donnyd | Any tuning I can do to ceph for this particular workload? | 21:22 |
| donnyd | i tuned as best i could for IOPS | 21:23 |
| fungi | mnaser may have suggestions... i know our vexxhost nodes are on ceph-baced volumes | 21:23 |
| clarkb | Not sure about ceph. For kvm you can tell it to not wait on writeback confirmation for data writes since we only run ephemeral workload there | 21:23 |
| clarkb | I don't know if kvm + ceph changes that | 21:23 |
| clarkb | https://doc.opensuse.org/documentation/leap/virtualization/html/book.virt/cha.cachemodes.html#cachemodes.descr we are happy with unsafe or writeback | 21:24 |
| clarkb | I think if both tempest and nova come back as happy The only hurdle left will be spinning up a mirror node and I want to sync with ianw on that to make sure we set that up in a way that makes sense for him as he has been working on updating those recently | 21:29 |
| clarkb | good chance we could have this in production on monday assuming things go smoothly | 21:29 |
| clarkb | I'll have to sort out what the run tempest easy mode method is now that d-g is mostly retired | 21:31 |
| clarkb | though I guess we have d-g running stable branch tests? that might be sufficient | 21:31 |
| clarkb | Ran: 16748 tests in 501.0000 sec. | 21:34 |
| donnyd | not sure if that is a good or bad number | 21:34 |
| fungi | now to figure out if that's fast or not, yeah ;) | 21:34 |
| clarkb | http://logs.openstack.org/15/651315/7/check/openstack-tox-py27/39f3565/job-output.txt.gz#_2019-06-28_03_56_17_965587 | 21:35 |
| clarkb | seems to be within the same range (note this run ran more tests than that run for some reason) | 21:35 |
| clarkb | http://logs.openstack.org/15/651315/6/check/openstack-tox-py27/9131333/job-output.txt.gz#_2019-06-15_16_02_52_110392 | 21:36 |
| clarkb | so ya I think we are in the ballpark | 21:36 |
| fungi | i like that stestr is reporting to 4 decimal places when it is clearly only summarizing to second granularity | 21:36 |
| clarkb | time to boot a second node to try a tempest run | 21:37 |
| donnyd | I am hopeful that when my disks finally finish syncing that this thing will get a little faster... | 21:38 |
| donnyd | seems like the raid sync is taking forever.... | 21:38 |
| *** happyhemant has quit IRC | 21:39 | |
| clarkb | as an interesting datapoint the mysql tests take about twice as long but the rbd etc tests seem to be about the same | 21:39 |
| clarkb | so ya disk performance could explain that | 21:39 |
| clarkb | (and the writeback thing might be a way to improve that if it remains a problem) | 21:40 |
| donnyd | its already set | 21:40 |
| donnyd | its on network=writeback | 21:40 |
| donnyd | so it should be using the rbd cache | 21:40 |
| donnyd | but horizon is painfully slow and the api's are not exactly snappy | 21:43 |
| donnyd | so I am hopeful the latency is in the sync | 21:44 |
| donnyd | if not... I am going to have to get some other disks for the controllers | 21:44 |
| clarkb | I wouldn't worry about it right now. We do have variable performance across clouds so we should do a wider check before deciding that is significantly slower | 21:45 |
| donnyd | Well I am a little bummed out, I just replaced these controllers with much newer servers and they do not seem to be any faster... if anything they are slower | 21:46 |
| clarkb | ah if you ahve your own numbers I'll definitely defer to those :) | 21:48 |
| clarkb | I have a devstack-gate reproduce.sh running now | 21:48 |
| clarkb | there is a reasonable chance I didn't edit its env vars enough to make it work without a zuul | 21:49 |
| clarkb | but we'll see | 21:49 |
| donnyd | what kinda of opstools are people running these days? | 22:00 |
| donnyd | looked at https://wiki.openstack.org/wiki/Osops | 22:01 |
| *** bobh has quit IRC | 22:01 | |
| donnyd | just not sure if its current | 22:01 |
| clarkb | around here we have a legacy set of puppet4 that is being transitioned to ansible with docker (and eventually we'll likely deploy into k8s but gitea was the plan for that and we discovered gitea can't run share everything yet) | 22:02 |
| *** pkopec has quit IRC | 22:04 | |
| donnyd | well I am going to go get some dinner. I will be back later | 22:07 |
| *** diablo_rojo has joined #openstack-infra | 22:09 | |
| *** rlandy has quit IRC | 22:10 | |
| *** slaweq has joined #openstack-infra | 22:11 | |
| *** slaweq has quit IRC | 22:16 | |
| *** raissa has joined #openstack-infra | 22:19 | |
| openstackgerrit | Kendall Nelson proposed opendev/storyboard master: Link development.rst to contributing.rst https://review.opendev.org/645960 | 22:24 |
| *** auristor has quit IRC | 22:26 | |
| *** rfarr__ has quit IRC | 22:26 | |
| *** factor has joined #openstack-infra | 22:31 | |
| clarkb | devstack spinup seemed to be about twice that of the source for my reproduce.sh. That could be related to lack of local caches and such. Will keep fiddling with it (but first to get tempest results) | 22:34 |
| *** rajinir has quit IRC | 22:42 | |
| *** goldyfruit has quit IRC | 22:47 | |
| *** diablo_rojo has quit IRC | 22:53 | |
| *** roman_g has quit IRC | 22:53 | |
| *** roman_g has joined #openstack-infra | 22:53 | |
| roman_g | fungi> i'll see if i have some notes on how we've authorized them to do that in the past | 22:59 |
| roman_g | fungi: reminding :) | 22:59 |
| roman_g | fungi: need to contact Freenode team, ask them to OP someone from openstack-infra team, and then get channel registered to openstack group | 23:00 |
| roman_g | Thank you! | 23:00 |
| clarkb | roman_g: we wanted to discuss with hogepodge when he is back from china about the correct way to set that up | 23:01 |
| clarkb | but then corvus should be able to request that from freenode for us' | 23:02 |
| roman_g | clarkb: oh, that's good. When and whom to contact later? | 23:07 |
| clarkb | roman_g: at this point maybe the best thing to do is start up an email thread? I think mostly we want to ensure that whoever has the keys for that new freenode namespace knows they have it and that they know to pass it on if necessary. And I think we were looking for hogepodge's feedback from the foundation side of things regarding that | 23:08 |
| roman_g | namespace openstack exists already | 23:10 |
| clarkb | roman_g: correct, but #airship isn't in that namespace | 23:10 |
| roman_g | yes, we just want to include it there | 23:10 |
| clarkb | meaning you want it to be part of the openstack namespace? | 23:11 |
| *** mattw4 has quit IRC | 23:11 | |
| clarkb | corvus: ^ do you know if freenode relies on the unique prefix for that to happen or do they track it some other way? | 23:11 |
| roman_g | well. I basically want channel to be registered & controlled by openstack infra and openstack IRC bots/services to work on it | 23:12 |
| clarkb | roman_g: yup thats the plan, in addition to that freenode has teh concept of groups of channels that all have a super admin basically | 23:13 |
| clarkb | and the throught was that #airship likely wants its own set of those | 23:13 |
| clarkb | the bots and services can run there either way | 23:13 |
| *** whoami-rajat has quit IRC | 23:13 | |
| *** Lucas_Gray has joined #openstack-infra | 23:14 | |
| roman_g | we are fine not having own super admin | 23:14 |
| corvus | clarkb: i believe it's by prefix; i think they can add a prefix to a group registration, or, perhaps nearly as easily create a new group registration | 23:16 |
| clarkb | ok I think my preference would be for airship to be its own group if only to simplify management of the channels (people won't have to wonder or follow some hint thread to know who to talk to) | 23:18 |
| corvus | clarkb, roman_g: i lean toward a new group registration regardless (regardless of who actually ends up as an admin) just because it will make things easier to manage in the future, and we won't have to untangle airship from openstack | 23:18 |
| clarkb | ++ | 23:18 |
| clarkb | and we can run that by hogepodge to make sure we aren't missing anything silly | 23:18 |
| roman_g | understood. thank you. | 23:19 |
| *** tosky has quit IRC | 23:26 | |
| fungi | yes, #opendev is already in its own group separate from #openstack, and the idea is that #zuul will probably ask for one of its own if it grows a need for additional channels at some point | 23:29 |
| fungi | roman_g: on other fronts, did you get a chance to try tumbleweed nodes again? we got the rsync mirror for it readded | 23:29 |
| openstackgerrit | Merged opendev/storyboard master: Add StoryBoard Story Tags to Docs https://review.opendev.org/667482 | 23:33 |
| *** aaronsheffield has quit IRC | 23:33 | |
| clarkb | hrm tempest timed out | 23:34 |
| clarkb | I'm going to restart it manually then figure out dinner and come back to this later (maybe monday) | 23:34 |
| *** sthussey has quit IRC | 23:34 | |
| roman_g | fungi: tumbleweed works perfectly. Thank you! | 23:35 |
| *** yamamoto has joined #openstack-infra | 23:36 | |
| fungi | excellent! | 23:36 |
| roman_g | I think you are one of the most responsive teams I've worked with. I appreciate your great help a lot. | 23:37 |
| fungi | we try, thanks for being patient with us when we're busy or otherwise occupied! | 23:38 |
| *** Wryhder has joined #openstack-infra | 23:39 | |
| clarkb | ok tempest is running withuot a timeout now | 23:39 |
| donnyd | clarkb: do you know why it was timing out? | 23:40 |
| *** Lucas_Gray has quit IRC | 23:40 | |
| *** Wryhder is now known as Lucas_Gray | 23:40 | |
| donnyd | i also may have to adjust my edge FW rules | 23:40 |
| clarkb | donnyd: devstack-gate has a timeout built into it | 23:41 |
| clarkb | so that jobs stop within a reasonable amount of time if things go sideawys. It hit that timeout | 23:41 |
| clarkb | that implies things are slower than we want but without full logs we probably can't debug (hence rerunning to generate better data) | 23:41 |
| donnyd | i will need to look into why... this thing has no load, 40G networking and many more fairly high performance parts.. | 23:42 |
| donnyd | I will look more into performance this weekend | 23:44 |
| clarkb | ya the networking is quite quick. Did notice that. I think it may have to do with disk as database things seemed slow | 23:45 |
| *** goldyfruit has joined #openstack-infra | 23:45 | |
| donnyd | when you say DB things seem slow.. do you mean API requests or do you mean DB(ish) things on the actual instance | 23:46 |
| clarkb | judging by the nova unittest for applying database migrations: the actual db ish things on the instance | 23:46 |
| *** ekultails has quit IRC | 23:46 | |
| donnyd | I have other storage I can use that will speed things up a lot... but will require a rebuild... and there is no redundancies | 23:47 |
| clarkb | donnyd: as a step zero you might want to double check that the instance images are served from where you expect them to be from? | 23:47 |
| clarkb | one thing I realized is I didn't boot these from volume and you had mentioned ceph so is it possible they booted on local disk? | 23:48 |
| donnyd | no | 23:48 |
| donnyd | I can see the workload coming in on Ceph | 23:48 |
| donnyd | at abysmal speeds | 23:51 |
| donnyd | just makes no sense.. all my fio testing showed between 10k and 15K IOPS at 4K block sizes | 23:53 |
| donnyd | plenty for most db operations | 23:53 |
| *** auristor has joined #openstack-infra | 23:54 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!