| *** weifan has quit IRC | 00:01 | |
| *** ociuhandu has joined #openstack-infra | 00:03 | |
| *** jamesmcarthur has quit IRC | 00:06 | |
| *** jamesmcarthur has joined #openstack-infra | 00:08 | |
| *** jamesmcarthur has quit IRC | 00:09 | |
| *** jamesmcarthur has joined #openstack-infra | 00:09 | |
| *** slaweq has joined #openstack-infra | 00:11 | |
| *** slaweq has quit IRC | 00:16 | |
| *** iurygregory has quit IRC | 00:17 | |
| *** ociuhandu has quit IRC | 00:17 | |
| *** jamesmcarthur has quit IRC | 00:33 | |
| *** jamesmcarthur has joined #openstack-infra | 00:33 | |
| *** jamesmcarthur has quit IRC | 00:39 | |
| *** gyee has quit IRC | 00:46 | |
| *** goldyfruit has quit IRC | 00:50 | |
| *** goldyfruit has joined #openstack-infra | 00:53 | |
| *** betherly has joined #openstack-infra | 00:55 | |
| *** betherly has quit IRC | 01:00 | |
| *** jamesmcarthur has joined #openstack-infra | 01:09 | |
| *** apetrich has quit IRC | 01:12 | |
| *** ociuhandu has joined #openstack-infra | 01:15 | |
| *** betherly has joined #openstack-infra | 01:15 | |
| *** jamesmcarthur has quit IRC | 01:15 | |
| *** imacdonn has quit IRC | 01:16 | |
| *** imacdonn has joined #openstack-infra | 01:17 | |
| *** ramishra has joined #openstack-infra | 01:20 | |
| *** betherly has quit IRC | 01:20 | |
| *** ruffian_sheep has joined #openstack-infra | 01:23 | |
| *** ociuhandu has quit IRC | 01:28 | |
| *** whoami-rajat has joined #openstack-infra | 01:31 | |
| *** jamesmcarthur has joined #openstack-infra | 01:40 | |
| *** apetrich has joined #openstack-infra | 01:44 | |
| *** betherly has joined #openstack-infra | 01:57 | |
| *** apetrich has quit IRC | 01:58 | |
| *** betherly has quit IRC | 02:02 | |
| *** goldyfruit has quit IRC | 02:07 | |
| *** goldyfruit has joined #openstack-infra | 02:08 | |
| *** mattw4 has quit IRC | 02:09 | |
| openstackgerrit | Masayuki Igawa proposed opendev/subunit2sql master: Fix gate error "mysqladmin: command not found" https://review.opendev.org/671379 | 02:11 |
|---|---|---|
| *** slaweq has joined #openstack-infra | 02:11 | |
| *** slaweq has quit IRC | 02:15 | |
| *** ociuhandu has joined #openstack-infra | 02:26 | |
| *** yamamoto has joined #openstack-infra | 02:27 | |
| openstackgerrit | Masayuki Igawa proposed opendev/subunit2sql master: Fix gate error "mysqladmin: command not found" https://review.opendev.org/671379 | 02:27 |
| fungi | pabelanger: here's a question which might be right up your alley... http://lists.openstack.org/pipermail/openstack-discuss/2019-July/007865.html | 02:33 |
| fungi | (someone has mistakenly e-mailed an openstack mailing list looking for asterisk help) | 02:33 |
| openstackgerrit | Masayuki Igawa proposed opendev/subunit2sql master: Fix gate error "mysqladmin: command not found" https://review.opendev.org/671379 | 02:37 |
| *** ociuhandu has quit IRC | 02:38 | |
| *** bhavikdbavishi has joined #openstack-infra | 02:44 | |
| *** bhavikdbavishi has quit IRC | 02:48 | |
| *** jamesmcarthur has quit IRC | 02:54 | |
| *** jamesmcarthur has joined #openstack-infra | 02:55 | |
| *** jamesmcarthur has quit IRC | 02:59 | |
| *** yamamoto has quit IRC | 03:02 | |
| openstackgerrit | Masayuki Igawa proposed opendev/subunit2sql master: Fix gate error "mysqladmin: command not found" and etc https://review.opendev.org/671379 | 03:02 |
| *** betherly has joined #openstack-infra | 03:09 | |
| *** betherly has quit IRC | 03:14 | |
| *** yamamoto has joined #openstack-infra | 03:16 | |
| *** yamamoto has quit IRC | 03:16 | |
| *** yamamoto has joined #openstack-infra | 03:17 | |
| openstackgerrit | Masayuki Igawa proposed opendev/subunit2sql master: Fix gate error "mysqladmin: command not found" and etc https://review.opendev.org/671379 | 03:18 |
| *** yamamoto has quit IRC | 03:21 | |
| *** goldyfruit has quit IRC | 03:30 | |
| *** redrobot has quit IRC | 03:32 | |
| *** rh-jelabarre has quit IRC | 03:34 | |
| *** jamesmcarthur has joined #openstack-infra | 03:35 | |
| *** psachin has joined #openstack-infra | 03:36 | |
| *** ociuhandu has joined #openstack-infra | 03:36 | |
| *** Guest99405 has joined #openstack-infra | 03:39 | |
| *** betherly has joined #openstack-infra | 03:40 | |
| *** bhavikdbavishi has joined #openstack-infra | 03:44 | |
| *** betherly has quit IRC | 03:45 | |
| *** ociuhandu has quit IRC | 03:50 | |
| openstackgerrit | Masayuki Igawa proposed opendev/subunit2sql master: Add .zuul.yaml and python 3 Train unit tests https://review.opendev.org/671389 | 03:51 |
| *** slaweq has joined #openstack-infra | 04:11 | |
| *** yamamoto has joined #openstack-infra | 04:13 | |
| *** apetrich has joined #openstack-infra | 04:16 | |
| *** slaweq has quit IRC | 04:16 | |
| lennyb | fungi, thanks. Outdated https://github.com/openstack/networking-mlnx repo is NOT longer needed for us. What is a procedure for infra team to delete/archived it? We prefer not having this repo at all to prevent users using outdated code. | 04:19 |
| *** yamamoto_ has joined #openstack-infra | 04:20 | |
| *** yamamoto has quit IRC | 04:20 | |
| *** aaronsheffield has joined #openstack-infra | 04:21 | |
| *** yamamoto has joined #openstack-infra | 04:21 | |
| *** yamamoto_ has quit IRC | 04:24 | |
| *** sgw has joined #openstack-infra | 04:26 | |
| *** ykarel|away has joined #openstack-infra | 04:37 | |
| openstackgerrit | Merged opendev/subunit2sql master: Fix gate error "mysqladmin: command not found" and etc https://review.opendev.org/671379 | 04:38 |
| *** udesale has joined #openstack-infra | 04:39 | |
| *** ociuhandu has joined #openstack-infra | 04:48 | |
| *** yamamoto has quit IRC | 04:52 | |
| openstackgerrit | Simon Westphahl proposed zuul/zuul master: Record and report time for trigger events https://review.opendev.org/671204 | 04:56 |
| *** yamamoto has joined #openstack-infra | 04:57 | |
| *** auristor has quit IRC | 04:59 | |
| *** ociuhandu has quit IRC | 05:00 | |
| *** auristor has joined #openstack-infra | 05:03 | |
| *** ykarel|away has quit IRC | 05:04 | |
| *** jamesmcarthur has quit IRC | 05:10 | |
| *** slaweq has joined #openstack-infra | 05:11 | |
| *** slaweq has quit IRC | 05:15 | |
| *** yamamoto has quit IRC | 05:18 | |
| *** ykarel|away has joined #openstack-infra | 05:20 | |
| *** ykarel|away is now known as ykarel | 05:24 | |
| *** yamamoto has joined #openstack-infra | 05:31 | |
| *** ccamacho has quit IRC | 05:33 | |
| *** pcaruana has joined #openstack-infra | 05:44 | |
| *** yamamoto has quit IRC | 05:45 | |
| *** yamamoto has joined #openstack-infra | 05:46 | |
| *** yamamoto has quit IRC | 05:46 | |
| *** yamamoto has joined #openstack-infra | 05:49 | |
| *** yamamoto has quit IRC | 05:49 | |
| openstackgerrit | Merged opendev/subunit2sql master: Add attachments flag to get_test_runs_by_status https://review.opendev.org/552152 | 05:52 |
| *** ociuhandu has joined #openstack-infra | 05:57 | |
| *** logan- has quit IRC | 06:00 | |
| *** logan_ has joined #openstack-infra | 06:01 | |
| *** logan_ is now known as logan- | 06:01 | |
| AJaeger | lennyb: https://docs.openstack.org/infra/manual/drivers.html#retiring-a-project | 06:01 |
| *** ociuhandu has quit IRC | 06:02 | |
| *** dpawlik has joined #openstack-infra | 06:04 | |
| *** ruffian_sheep14 has joined #openstack-infra | 06:12 | |
| *** igordc has quit IRC | 06:14 | |
| *** ruffian_sheep has quit IRC | 06:14 | |
| *** ruffian_sheep14 is now known as ruffian_sheep | 06:18 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/project-config master: Normalize projects.yaml https://review.opendev.org/671407 | 06:23 |
| *** yamamoto has joined #openstack-infra | 06:26 | |
| *** slaweq has joined #openstack-infra | 06:28 | |
| *** strigazi has quit IRC | 06:31 | |
| *** raukadah is now known as chandankumar | 06:35 | |
| *** yamamoto has quit IRC | 06:35 | |
| fungi | AJaeger: in this case lennyb is asking about just deleting the stale mirror of it on github | 06:43 |
| fungi | not retiring the project (it will continue to be maintained in opendev) | 06:44 |
| *** rcernin has quit IRC | 06:44 | |
| fungi | lennyb: i'm about to go to sleep (nearly 3am here) but can do it when i wake up | 06:46 |
| *** joeguo has quit IRC | 06:47 | |
| fungi | lennyb: if one of the people listed here https://review.opendev.org/#/admin/groups/networking-mlnx-release can e-mail the openstack-infra mailing list requesting deletion of the stale copy on github, that should be sufficient authority (optionally, get one of them to add you to that group and then i'm happy to do it on your authority) | 06:48 |
| *** odicha has joined #openstack-infra | 06:49 | |
| * fungi disappears for some low-power standby time | 06:50 | |
| *** yamamoto has joined #openstack-infra | 06:51 | |
| openstackgerrit | Masayuki Igawa proposed openstack/project-config master: Cleanup in-tree removed jobs https://review.opendev.org/671412 | 06:55 |
| *** kopecmartin|off is now known as kopecmartin | 06:59 | |
| *** ociuhandu has joined #openstack-infra | 07:00 | |
| openstackgerrit | Merged openstack/project-config master: Normalize projects.yaml https://review.opendev.org/671407 | 07:03 |
| *** jbadiapa has joined #openstack-infra | 07:04 | |
| *** xek has joined #openstack-infra | 07:06 | |
| *** ginopc has joined #openstack-infra | 07:07 | |
| *** ociuhandu has quit IRC | 07:08 | |
| *** pgaxatte has joined #openstack-infra | 07:11 | |
| *** rpittau|afk is now known as rpittau | 07:11 | |
| *** aaronsheffield has quit IRC | 07:16 | |
| AJaeger | fungi: good night - ah, missed the context. Thakns for explaining. | 07:27 |
| *** ruffian_sheep16 has joined #openstack-infra | 07:36 | |
| *** ccamacho has joined #openstack-infra | 07:36 | |
| *** ruffian_sheep has quit IRC | 07:36 | |
| *** ykarel is now known as ykarel|lunch | 07:37 | |
| *** ruffian_sheep16 is now known as ruffian_sheep | 07:41 | |
| *** ruffian_sheep has quit IRC | 07:52 | |
| *** dchen has quit IRC | 07:55 | |
| *** ralonsoh has joined #openstack-infra | 07:56 | |
| *** dchen has joined #openstack-infra | 07:56 | |
| *** lucasagomes has joined #openstack-infra | 07:56 | |
| *** ociuhandu has joined #openstack-infra | 08:07 | |
| *** pkopec has joined #openstack-infra | 08:07 | |
| *** dchen has quit IRC | 08:07 | |
| *** dtantsur|afk is now known as dtantsur | 08:09 | |
| *** yolanda has joined #openstack-infra | 08:09 | |
| *** yolanda has quit IRC | 08:10 | |
| *** ruffian_sheep has joined #openstack-infra | 08:10 | |
| *** ociuhandu has quit IRC | 08:11 | |
| *** ruffian_sheep has quit IRC | 08:16 | |
| *** ruffian_sheep has joined #openstack-infra | 08:19 | |
| *** derekh has joined #openstack-infra | 08:20 | |
| *** rascasoft has quit IRC | 08:25 | |
| *** rascasoft has joined #openstack-infra | 08:25 | |
| *** sshnaidm|afk is now known as sshnaidm | 08:27 | |
| *** tosky has joined #openstack-infra | 08:31 | |
| *** tkajinam has quit IRC | 08:40 | |
| *** xek has quit IRC | 08:42 | |
| *** xek has joined #openstack-infra | 08:43 | |
| *** iurygregory has joined #openstack-infra | 08:46 | |
| *** xek has quit IRC | 08:47 | |
| *** ruffian_sheep has quit IRC | 08:48 | |
| openstackgerrit | Joshua Hesketh proposed opendev/glean master: Sync when writing the file https://review.opendev.org/652238 | 08:48 |
| *** yamamoto has quit IRC | 08:51 | |
| *** xek has joined #openstack-infra | 08:51 | |
| *** ruffian_sheep has joined #openstack-infra | 08:52 | |
| *** xek has quit IRC | 08:52 | |
| *** yamamoto has joined #openstack-infra | 08:57 | |
| *** ykarel|lunch is now known as ykarel | 09:00 | |
| *** arxcruz|ruck is now known as arxcruz | 09:01 | |
| *** apetrich has quit IRC | 09:06 | |
| *** apetrich has joined #openstack-infra | 09:07 | |
| openstackgerrit | Joshua Hesketh proposed opendev/system-config master: Update zuul status URL's to opendev https://review.opendev.org/671435 | 09:16 |
| openstackgerrit | Joshua Hesketh proposed opendev/system-config master: Toggle CI should also hide old zuul comments https://review.opendev.org/671436 | 09:16 |
| *** yamamoto has quit IRC | 09:17 | |
| *** apetrich has quit IRC | 09:19 | |
| openstackgerrit | Natal Ngétal proposed openstack/diskimage-builder master: [Configuration] Switch to stestr. https://review.opendev.org/629414 | 09:19 |
| *** tdasilva_ has quit IRC | 09:25 | |
| *** tdasilva_ has joined #openstack-infra | 09:26 | |
| *** xek has joined #openstack-infra | 09:26 | |
| *** hrw has joined #openstack-infra | 09:26 | |
| hrw | morning | 09:26 |
| hrw | who I should speak to when it comes to instances in linaro-london? | 09:27 |
| *** electrofelix has joined #openstack-infra | 09:28 | |
| hrw | we created os.extra.large.infra flavour there to give 16vcpu/16GB ram instances for infra as current 8vcpu ones do not cope with kolla ci requirements | 09:28 |
| *** apetrich has joined #openstack-infra | 09:31 | |
| *** tdasilva_ has quit IRC | 09:47 | |
| *** tdasilva_ has joined #openstack-infra | 09:47 | |
| mordred | hrw: you want to make a patch to update the flavor name here: https://opendev.org/openstack/project-config/src/branch/master/nodepool/nl03.openstack.org.yaml#L401-L424 | 09:49 |
| *** yamamoto has joined #openstack-infra | 09:50 | |
| *** gfidente|afk is now known as gfidente | 09:52 | |
| *** yamamoto has quit IRC | 09:53 | |
| hrw | mordred: thanks. will do | 09:54 |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: Zuul CLI: allow access via REST https://review.opendev.org/636315 | 09:55 |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: Use a requests session to simplify auth'd calls https://review.opendev.org/670511 | 09:56 |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/project-config master: Linaro London: use new bigger flavour https://review.opendev.org/671445 | 10:03 |
| openstackgerrit | Kien Nguyen proposed opendev/system-config master: Allow to rsync Centos Software Collections repo https://review.opendev.org/671449 | 10:07 |
| hrw | patch created, sent, can move to other stuff | 10:09 |
| mordred | hrw: sweet! | 10:12 |
| hrw | mordred: time to add some reviewers ;D | 10:13 |
| *** ruffian_sheep has quit IRC | 10:14 | |
| *** jamesmcarthur has joined #openstack-infra | 10:15 | |
| *** thiago__ has joined #openstack-infra | 10:15 | |
| *** tdasilva_ has quit IRC | 10:17 | |
| *** jamesmcarthur has quit IRC | 10:19 | |
| *** pkopec has quit IRC | 10:20 | |
| *** pkopec has joined #openstack-infra | 10:20 | |
| *** joeguo has joined #openstack-infra | 10:22 | |
| *** siqbal has joined #openstack-infra | 10:23 | |
| *** whoami-rajat has quit IRC | 10:30 | |
| *** yamamoto has joined #openstack-infra | 10:32 | |
| *** takamatsu has joined #openstack-infra | 10:36 | |
| *** yamamoto has quit IRC | 10:39 | |
| siqbal | Hi, I'm working on devstack and trying to locate CommsPowerManagement (https://github.com/intel/CommsPowerManagement/) within devstack, which file on devstack I can place this which would be suitable for SST. Any help is appreciated | 10:52 |
| AJaeger | siqbal: for devstack, best use #openstack-qa | 10:54 |
| *** hrw has left #openstack-infra | 11:01 | |
| openstackgerrit | M V P Nitesh proposed openstack/diskimage-builder master: efibootmgr is need to create boot entries https://review.opendev.org/671456 | 11:01 |
| siqbal | AJarger Thanks I'll try that channel | 11:04 |
| *** yamamoto has joined #openstack-infra | 11:05 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Build a docker image of the existing gerrit 2.13 https://review.opendev.org/671457 | 11:09 |
| openstackgerrit | Merged openstack/pbr master: Fix parsing on egg names with dashes from git URLs https://review.opendev.org/648727 | 11:16 |
| *** goldyfruit has joined #openstack-infra | 11:20 | |
| mordred | knikolla: I just tried to use an application credential on MOC with the following clouds.yaml: http://paste.openstack.org/show/754541/ and I got Attempted to authenticate with an unsupported method. (HTTP 401) (Request-ID: req-ccced950-9dfb-4a44-81ed-c61af2951287) | 11:29 |
| mordred | knikolla: this is using python-openstackclient that I just installed via pip into a venv | 11:30 |
| mordred | knikolla: (creating teh application credential worked great!) | 11:31 |
| *** rh-jelabarre has joined #openstack-infra | 11:33 | |
| mordred | cmurphy: ^^ also just keeping you in the loop, since you typically know everything | 11:34 |
| *** tesseract has joined #openstack-infra | 11:38 | |
| *** eharney has quit IRC | 11:40 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Add clouds.yaml entry for MOC control plane project https://review.opendev.org/671463 | 11:51 |
| mordred | infra-root: ^^ that doesn't work yet (see issue raised with knikolla) but I've put the secret into host_vars and have put information into root secrets | 11:51 |
| *** rfolco is now known as rfolco|rover | 12:12 | |
| *** dpawlik has quit IRC | 12:16 | |
| mnaser | mordred, knikolla: i think the issue here is the keystone is not configurd with application credentials enabled | 12:18 |
| mnaser | if i remember right, the api for app creds works regardless if its enabled or not | 12:18 |
| *** joeguo has quit IRC | 12:19 | |
| mnaser | 'application_credential' has to be inside [auth]/methods | 12:21 |
| mordred | mnaser: awesome - thanks! | 12:24 |
| mnaser | http://lists.openstack.org/pipermail/openstack-infra/2019-July/006422.html | 12:25 |
| mnaser | scary typo in the title | 12:25 |
| *** eharney has joined #openstack-infra | 12:26 | |
| *** udesale has quit IRC | 12:26 | |
| *** thiago__ has quit IRC | 12:26 | |
| *** thiago__ has joined #openstack-infra | 12:27 | |
| *** udesale has joined #openstack-infra | 12:27 | |
| mordred | mnaser: hah | 12:28 |
| *** thiago__ has quit IRC | 12:33 | |
| *** thiago__ has joined #openstack-infra | 12:33 | |
| *** tdasilva_ has joined #openstack-infra | 12:35 | |
| *** thiago__ has quit IRC | 12:38 | |
| openstackgerrit | Joshua Hesketh proposed opendev/system-config master: Add some pointers on the OpenDev PPA https://review.opendev.org/670952 | 12:40 |
| openstackgerrit | Merged zuul/zuul-jobs master: Add add-launchpad-credentaials test job https://review.opendev.org/670207 | 12:42 |
| openstackgerrit | Merged zuul/zuul-jobs master: Add add-sshkey test job https://review.opendev.org/670208 | 12:44 |
| *** piotrowskim has quit IRC | 12:47 | |
| *** ekultails has joined #openstack-infra | 12:50 | |
| pabelanger | fungi: ha | 12:53 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Evaluate CODEOWNERS settings during canMerge check https://review.opendev.org/644557 | 12:54 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Evaluate CODEOWNERS settings during canMerge check https://review.opendev.org/644557 | 12:56 |
| *** gtarnaras has joined #openstack-infra | 12:58 | |
| *** aaronsheffield has joined #openstack-infra | 12:59 | |
| openstackgerrit | Merged opendev/system-config master: Add letsencrypt documentation https://review.opendev.org/669602 | 12:59 |
| *** siqbal has quit IRC | 13:03 | |
| *** siqbal has joined #openstack-infra | 13:03 | |
| openstackgerrit | Merged zuul/zuul master: Record and report time for trigger events https://review.opendev.org/671204 | 13:05 |
| stephenfin | harlowja: You might be interested in http://lists.openstack.org/pipermail/openstack-discuss/2019-July/007878.html | 13:11 |
| stephenfin | harlowja: Assuming you agree, any chance I could get added as core on x/doc8 so I can close out the few open patches there? | 13:11 |
| * stephenfin is clearly on a crusade to be core of allllll the projects | 13:12 | |
| *** Guest99405 is now known as redrobot | 13:13 | |
| *** dpawlik has joined #openstack-infra | 13:15 | |
| pabelanger | fungi: replied | 13:17 |
| openstackgerrit | Stephen Finucane proposed openstack/project-config master: Ends project gating for Bandit https://review.opendev.org/671483 | 13:25 |
| *** rfarr has joined #openstack-infra | 13:26 | |
| *** rfarr_ has joined #openstack-infra | 13:26 | |
| *** rfarr_ has quit IRC | 13:26 | |
| openstackgerrit | Merged zuul/zuul master: Parse nodeset.group.nodes as a list https://review.opendev.org/671372 | 13:28 |
| mordred | stephenfin: watch out - I'll make you core on sdk and then I'll make you be ptl and then I'll ghost | 13:32 |
| *** siqbal90 has joined #openstack-infra | 13:32 | |
| openstackgerrit | Merged opendev/system-config master: Streamline documented bup setup process https://review.opendev.org/666194 | 13:33 |
| stephenfin | 😨😨😨 | 13:33 |
| *** siqbal has quit IRC | 13:35 | |
| openstackgerrit | Francis Deslauriers proposed opendev/git-review master: Add labels on change submission https://review.opendev.org/666301 | 13:36 |
| *** beekneemech has joined #openstack-infra | 13:36 | |
| *** bnemec has quit IRC | 13:37 | |
| *** tdasilva_ has quit IRC | 13:41 | |
| knikolla | mordred: mnaser: oh right! I’ll make a note to enable app creds as an authentication method asap. | 13:41 |
| *** tdasilva_ has joined #openstack-infra | 13:42 | |
| openstackgerrit | Merged opendev/system-config master: Remove apport package from ubuntu servers https://review.opendev.org/652143 | 13:43 |
| mordred | knikolla: woot! | 13:45 |
| mordred | knikolla: with app creds, if I want to use the cred by name, I need to provide username too, yes? | 13:46 |
| *** sgw has quit IRC | 13:47 | |
| *** bhavikdbavishi has quit IRC | 13:50 | |
| *** rfarr has quit IRC | 13:53 | |
| openstackgerrit | Merged zuul/zuul master: Fix GitEventFilter.__init__ https://review.opendev.org/611989 | 13:54 |
| *** whoami-rajat has joined #openstack-infra | 13:55 | |
| mnaser | i'd like some help, tripleo+openstack-ansible are looking to work together on sharing ansible code | 13:59 |
| mnaser | i'd like to create a group in gerrit called openstack-ansible-collab | 13:59 |
| mnaser | seeded with tripleo-ansible-core and openstack-ansible-core | 13:59 |
| mnaser | but.. how | 13:59 |
| mnaser | i cant find a 'list of groups' inside gerrit/acls/openstack/*.config .. i just see things assigned to a group though | 14:00 |
| *** mattmceuen has left #openstack-infra | 14:02 | |
| *** bhavikdbavishi has joined #openstack-infra | 14:03 | |
| fungi | mnaser: they're created automatically when they're referenced | 14:04 |
| mnaser | oh.. okay cool | 14:04 |
| fungi | so just update an acl to use that group for some permission, and then one of us can seed the membership for you to get you going | 14:04 |
| mnaser | ok great, thanks fungi | 14:05 |
| fungi | any time! | 14:05 |
| *** sgw has joined #openstack-infra | 14:05 | |
| *** tdasilva_ has quit IRC | 14:09 | |
| *** yolanda has joined #openstack-infra | 14:09 | |
| *** tdasilva_ has joined #openstack-infra | 14:09 | |
| *** ykarel is now known as ykarel|away | 14:12 | |
| *** tdasilva_ is now known as tdasilva | 14:15 | |
| *** ccamacho has quit IRC | 14:15 | |
| *** ykarel|away has quit IRC | 14:17 | |
| *** jamesmcarthur has joined #openstack-infra | 14:19 | |
| *** ccamacho has joined #openstack-infra | 14:20 | |
| *** Lucas_Gray has joined #openstack-infra | 14:22 | |
| *** iurygregory has quit IRC | 14:23 | |
| *** jamesmcarthur has quit IRC | 14:23 | |
| *** jamesmcarthur has joined #openstack-infra | 14:24 | |
| *** Wryhder has joined #openstack-infra | 14:25 | |
| *** Lucas_Gray has quit IRC | 14:27 | |
| *** Wryhder is now known as Lucas_Gray | 14:27 | |
| AJaeger | mnaser: you can also just add one group to another ACL, so no need to create a new ACL... | 14:27 |
| fungi | yes, one permission can be delegated to multiple groups | 14:29 |
| fungi | or a group can contain a mix of other groups and individuals | 14:29 |
| fungi | the latter is a bit more flexible if you want to avoid making more changes to the acl over time | 14:30 |
| fungi | hard-coding a list of groups into the acl means more acl patches to change that | 14:30 |
| *** ricolin_ is now known as ricolin | 14:34 | |
| *** iurygregory has joined #openstack-infra | 14:40 | |
| *** jamesmcarthur has quit IRC | 14:43 | |
| *** priteau has joined #openstack-infra | 14:48 | |
| clarkb | gitea06 has not OOM'd since the swapfile was added | 14:51 |
| clarkb | gitea01 however has OOM'd since we added haproxy logging | 14:52 |
| mordred | clarkb: both of those things are good | 14:52 |
| mordred | clarkb: I mean, ooming isn't good - but ooming with logging in place to find the source of the oom is better than the opposite | 14:53 |
| *** kaiokmo has joined #openstack-infra | 14:58 | |
| clarkb | there is an IP in vexxhost that does a bunch of requests around when it happens | 14:59 |
| clarkb | does not appears to be one of ours (grepped sytem-cofnig which has our inventory and the nodepool launcher logs) | 14:59 |
| roman_g | Hello team. This https://review.opendev.org/#/c/668665/ change has 2xCR+2 and WF+1, and also has `Depends-on:` another change which has been merged already. How to I push Gerrit to merge it? | 15:02 |
| *** kopecmartin is now known as kopecmartin|off | 15:03 | |
| clarkb | roman_g: if the projects share a gate queue (thats a bit of config you can set) then zuul with automatically enqueue the child when the parent merges. Otherwise you can either recheck it which will rerun check then gate then merge, or reapprove it which should just run gate jobs then merge it | 15:04 |
| fungi | recheck's working now? | 15:04 |
| fungi | (for this case) | 15:04 |
| roman_g | clarkb: got it. Thank you! | 15:04 |
| clarkb | fungi: it always worked? | 15:04 |
| roman_g | fungi: trying | 15:04 |
| fungi | i thought it needed a second workflow +1 or the workflow +1 removed and readded | 15:04 |
| fungi | because of gerrit not emitting an event when zuul replaces the old verify +1 with a new one | 15:05 |
| clarkb | fungi: oh that was fixed ages ago by zaro by having gerrit always emit vote details and not just the delta in the events | 15:05 |
| fungi | ahh | 15:05 |
| *** tdasilva has quit IRC | 15:05 | |
| clarkb | fungi: there is actually a change proposed to zuul to undo that behavior change whcih I have -1'd | 15:05 |
| fungi | heh | 15:05 |
| *** tdasilva has joined #openstack-infra | 15:05 | |
| mordred | clarkb: can you tell what sort of requests? like, is it doing a full clone of nova or something like that? and/or I wonder if putting in a rate limit would help | 15:06 |
| mordred | (if it's just someone ooming us by doing a shotgun of requests all at once, that sounds like something we'd want to be resilient against) | 15:06 |
| clarkb | mordred: no I haven't tracked it back that far yet. First trying to get some general stats on requestors | 15:09 |
| mordred | ++ | 15:09 |
| *** eharney_ has joined #openstack-infra | 15:12 | |
| clarkb | The IP making the most requests over that hour block of time is an OVH IP that we do end up using during that block of time (so that implies there is maybe a job inefficiency we can cleanup but that happens after 07:30 when oom was at 07:00 so probably unrelated to the OOM) | 15:13 |
| *** eharney has quit IRC | 15:15 | |
| clarkb | http://logs.openstack.org/78/671378/2/check/openstack-tox-docs/ba3dec1/job-output.txt.gz#_2019-07-18_07_48_18_626049 is the potential job inefficiency | 15:15 |
| openstackgerrit | Tobias Rydberg proposed opendev/irc-meetings master: After transforming publiccloud wg to sig, meeting info changes as well. https://review.opendev.org/671509 | 15:15 |
| clarkb | I've also just now realized I wasn't grepping in the correct place for the vexxhost ip in nodepool (due to rollover) | 15:15 |
| clarkb | so need to double check on that | 15:15 |
| clarkb | though it is an ipv4 IP and we should have requests from ipv6 from our test nodes in vexxhost so that implies it isn't one of ours | 15:16 |
| *** eharney_ is now known as eharney | 15:17 | |
| *** Lucas_Gray has quit IRC | 15:17 | |
| clarkb | ok confirmed does not show up in any of the nodepool logs on nl03 created since july 1 | 15:20 |
| clarkb | The top requestor is that job I linked above, then the vexxhost IP that does a lot around the OOM then an addr from NTT | 15:21 |
| clarkb | so I think that means the vexxhost IP is likely the one to look at since the job doesn't overlap with the OOM | 15:21 |
| *** ykarel|away has joined #openstack-infra | 15:23 | |
| *** odicha has quit IRC | 15:25 | |
| cmurphy | i'm cleaning out my gerrit queue, could i get someone to review https://review.opendev.org/628189 ? i promise the log in the commit message used to prove my point | 15:25 |
| *** tdasilva_ has joined #openstack-infra | 15:26 | |
| mordred | cmurphy: and the puppet output presumably already doesn't show hiera values, so we're asserting it's safe to not no_log? | 15:26 |
| corvus | (i assumed we no_logged it just because it was verbose) | 15:27 |
| clarkb | ya the reason we don't log that iirc is beacuse it is very verbose | 15:27 |
| clarkb | not because it was secrets | 15:27 |
| cmurphy | mordred: it's just a file copy so i can't imagine there are secrets | 15:27 |
| clarkb | should be safe to approve but may increase log verbosity | 15:27 |
| corvus | clarkb: what's the actual request in that log you pasted? | 15:27 |
| clarkb | corvus: well there are thousands of requests from that IP over that hour according to the haproxy log | 15:28 |
| *** tdasilva has quit IRC | 15:28 | |
| clarkb | corvus: it isn't just one, but also that happens well after the OOM so I think it is unrelated | 15:28 |
| mordred | clarkb: yeah - I was going to ask that as well - looking through the log I see all the urls, but I haven't found what it's doing with the urls et | 15:29 |
| clarkb | I'm currently trying to work out how to correlate the haproxy logs to the macaron logs from gitea | 15:30 |
| clarkb | it would be nice if they both logged the ports they were using oh well | 15:30 |
| corvus | oooh i see it. | 15:31 |
| corvus | clarkb, mordred: https://opendev.org/openstack/tempest/src/branch/master/tools/generate-tempest-plugins-list.py | 15:32 |
| corvus | that stuff is the output from this script | 15:32 |
| corvus | which apparently takes 14m to run | 15:32 |
| clarkb | I've got what I think is a very suspicious connection on the haproxy side (based on how long it takes to process and nearness to the OOM) I'll work on getting a paste up with some of these log details | 15:33 |
| corvus | that's only fetching a raw file (eg https://opendev.org/openstack/tempest/raw/branch/master/setup.cfg) so, honestly, it's about as nice as it could be. i'm not sure we should frown on it. | 15:33 |
| clarkb | ya was mostly just wanting to sanity check that the job shouldn't be relying on zuul's git repos instead | 15:34 |
| mordred | corvus: yeah - it seems like an operation we should be able to handle | 15:34 |
| clarkb | if it is sanely requesting git repo data without bypassing zuul then that should be fine | 15:34 |
| *** bhavikdbavishi has quit IRC | 15:37 | |
| clarkb | http://paste.openstack.org/show/754556/ | 15:45 |
| clarkb | I've tried to collect the relevant logs I've found so far there and sanitized IP addrs | 15:45 |
| clarkb | (the IP addr I didn't sanitize is for the load balancer) | 15:45 |
| *** mattw4 has joined #openstack-infra | 15:46 | |
| clarkb | I guess we can check other instances of OOMs for similar git cat file errors just prior? | 15:46 |
| *** pgaxatte has quit IRC | 15:47 | |
| clarkb | the git cat file error does not show up on the last OOM on gitea06 so probably unrelated | 15:48 |
| mordred | clarkb: anything in indexer? | 15:49 |
| *** bhavikdbavishi has joined #openstack-infra | 15:49 | |
| clarkb | mordred: not sure I understand the question | 15:49 |
| clarkb | the only error near gitea06's last oom was a user lookup that failed | 15:49 |
| mordred | clarkb: a file-based indexer could easily have concurrency problems - the git cat-file is related to the call "processRepoIndexerOperationQueue" | 15:49 |
| mordred | ah - nod | 15:49 |
| mordred | well - there goes that theory :) | 15:50 |
| clarkb | haproxy log format is configurable and we can add the source port of the backend connection to the logs (but we'd need to log that data on the gitea side too for it to be useful and I don't know if we can do that) | 15:51 |
| clarkb | as a sanity check the clocks on gitea-lb and gitea01 seem to be in sync | 15:54 |
| clarkb | I think I've managed to line up some of the requests at least | 15:55 |
| clarkb | but the timestamps seem just off? hard to tell | 15:56 |
| openstackgerrit | James E. Blair proposed opendev/system-config master: WIP: Add gerrit to gitea job https://review.opendev.org/671162 | 15:57 |
| *** beekneemech has quit IRC | 15:57 | |
| *** bnemec has joined #openstack-infra | 15:58 | |
| clarkb | we can set ACCESS_LOG_TEMPLATE to update the macaron logs | 15:58 |
| *** mattw4 has quit IRC | 15:58 | |
| clarkb | which needs to be enabled with ENABLE_ACCESS_LOG | 15:59 |
| clarkb | however the macaron context object doesn't seem to expose the remote port (just the remote addr) | 16:00 |
| *** iurygregory has quit IRC | 16:01 | |
| *** gtarnaras has quit IRC | 16:01 | |
| *** eharney has quit IRC | 16:02 | |
| *** ginopc has quit IRC | 16:02 | |
| *** lucasagomes has quit IRC | 16:05 | |
| *** rpittau is now known as rpittau|afk | 16:05 | |
| clarkb | I'm having a hard time matching haproxy logs up with the gitea logs so looking at the OOM logs a bit more closely. There are git processes with 700MB or resident memory, interesting | 16:06 |
| mordred | clarkb: that's a lot of memory for a git operation | 16:07 |
| corvus | we build that image from debian-testing; i wonder if any git updates have happened since | 16:07 |
| corvus | (maybe we got a bad version of git?) | 16:07 |
| mordred | corvus: I was just thinking the same thing | 16:08 |
| clarkb | `/usr/lib/git-core/git pack-objects --revs --thin --stdout --progress --delta-base-offset` is currently using > 1GB on gitea01 and is running in nova's repo | 16:08 |
| corvus | git version 2.20.1 | 16:08 |
| mordred | maybe it's just that git takes a lot of ram to process nova? | 16:08 |
| clarkb | This may also explain why it is so slow if looking up nova causes it to OOM then try again | 16:09 |
| corvus | more specifically debian version 1:2.20.1-2 | 16:09 |
| corvus | which is what's current | 16:09 |
| mordred | corvus: yeah - or if looking up nova causes it to read all of nova into ram from disk | 16:10 |
| clarkb | We may be able to try reproducing this with direct nova renders like https://gitea01.opendev.org/openstack/nova/$stuff | 16:10 |
| clarkb | any idea what is packing objects? | 16:10 |
| clarkb | does not appear to be our gc cron (based on ps having no gc lines and our cron starting 4 hours ago) | 16:11 |
| clarkb | oh wait packing objects happens when you clone nova right? | 16:11 |
| clarkb | or when you fetch things and it neesd a new pack | 16:11 |
| mordred | does it? so random clones/fetches could cause nova to decide to repack? | 16:11 |
| clarkb | its possible this was always in OOM territory and swap on old rax setup caused it to mostly work | 16:12 |
| mordred | yeah | 16:12 |
| corvus | i think it's more like the smart http protocol generates dynamic object packs depending on what the remote side is missing | 16:12 |
| clarkb | mordred: ya I think if say my repo locally is out of sync with upstream but has half the objects in upstreams packfile then upstream will make a new packfile of the delta and send that to me | 16:12 |
| clarkb | corvus: ya | 16:12 |
| mordred | what both of you said | 16:12 |
| corvus | makes it twice as true | 16:13 |
| mordred | does that mean it's twice the price? | 16:13 |
| *** siqbal90 has quit IRC | 16:13 | |
| corvus | mordred: yes, special price for you. | 16:13 |
| clarkb | clearly if we just delete the openwrt image in nova everything will be better | 16:14 |
| clarkb | if only we could do that without writing all of novas history | 16:14 |
| clarkb | (I also don't know that will help much) | 16:14 |
| mordred | corvus: we so should have done that when we did the bzr->git transform - would have been a good time to rewrite the history | 16:14 |
| mordred | now that I think about it | 16:14 |
| mordred | clarkb: I choose to believe it would at least help something | 16:15 |
| corvus | yeah. i don't think i knew there was an openwrt image in there then; in fact, i think i only learned about that like a year or two ago. | 16:15 |
| openstackgerrit | Merged zuul/nodepool master: Don't pause static pool on single label quota https://review.opendev.org/667371 | 16:16 |
| mordred | would turning on git lfs support help with the openwrt image? like, maybe push nova into a gitea with lfs enabled and maybe it'll cause the openwrt image to go into LFS with a pointer in the real repo and nobody will need to derefernce the LFS unless they actually want those commits? /me has no idea how git lfs works | 16:16 |
| clarkb | I need to go make tea and find some breakfast but I'm beginning to think that nova is the thread we should pull on and it may lead us to better gitea performance for nova too | 16:16 |
| mordred | clarkb: yeah. I was having a similar thought | 16:17 |
| clarkb | mordred: I think you have to add objects with lfs at commit time | 16:17 |
| clarkb | or something, its not automagic | 16:17 |
| mordred | ugh. oh well :) | 16:17 |
| clarkb | other thoughts really quickly for short term help: rebuild all the servers with more disk and swapfiles (https://review.opendev.org/#/c/671102/) and/or scale the servers up on memory and maybe reduce server count | 16:21 |
| *** siqbal90 has joined #openstack-infra | 16:24 | |
| *** gyee has joined #openstack-infra | 16:27 | |
| openstackgerrit | Merged opendev/ansible-role-puppet master: Revert "Don't log puppot module sync" https://review.opendev.org/628189 | 16:37 |
| *** ccamacho has quit IRC | 16:37 | |
| *** sshnaidm is now known as sshnaidm|off | 16:42 | |
| *** ramishra has quit IRC | 16:49 | |
| *** panda is now known as panda|off | 16:51 | |
| *** igordc has joined #openstack-infra | 16:54 | |
| *** chandankumar is now known as raukadah | 16:56 | |
| *** derekh has quit IRC | 17:00 | |
| *** udesale has quit IRC | 17:00 | |
| *** xek has quit IRC | 17:06 | |
| *** mattw4 has joined #openstack-infra | 17:08 | |
| *** eernst has joined #openstack-infra | 17:11 | |
| *** eharney has joined #openstack-infra | 17:13 | |
| *** mattw4 has quit IRC | 17:14 | |
| *** mattw4 has joined #openstack-infra | 17:14 | |
| *** eernst has quit IRC | 17:15 | |
| *** igordc has quit IRC | 17:15 | |
| *** igordc has joined #openstack-infra | 17:16 | |
| *** tosky has quit IRC | 17:17 | |
| *** panda|off has quit IRC | 17:19 | |
| *** panda has joined #openstack-infra | 17:21 | |
| *** armax has quit IRC | 17:28 | |
| openstackgerrit | Matthew Thode proposed openstack/diskimage-builder master: support alternate portage directories https://review.opendev.org/671530 | 17:33 |
| prometheanfire | infra may need this for builds ^ | 17:34 |
| prometheanfire | I'm currently running a basic systemd build to test | 17:34 |
| *** tdasilva_ is now known as tdasilva | 17:45 | |
| *** aprice has quit IRC | 17:45 | |
| *** hogepodge has quit IRC | 17:45 | |
| *** petevg has quit IRC | 17:45 | |
| *** clayg has quit IRC | 17:45 | |
| *** seyeongkim has quit IRC | 17:45 | |
| *** guilhermesp has quit IRC | 17:46 | |
| *** aprice has joined #openstack-infra | 17:47 | |
| *** hogepodge has joined #openstack-infra | 17:47 | |
| clarkb | I have confirmed via direct clone against gitea06 (using that one since it has swap) that cloning nova seems to take 1.3GB memory or so resident for about 2.5 minutes | 17:47 |
| clarkb | though that time value is likely related to teh bw between my home and vexxhost | 17:47 |
| *** petevg has joined #openstack-infra | 17:48 | |
| *** clayg has joined #openstack-infra | 17:48 | |
| *** guilhermesp has joined #openstack-infra | 17:48 | |
| *** seyeongkim has joined #openstack-infra | 17:48 | |
| corvus | clarkb, mordred: on a related but different tack, i just did a little bit of poking at gitea source code; i suspect (but have not proven) that the lag in rendering nova is due to looking up the last commit info for each item in the directory, with directories themselves having an especially high cost (the only time it gets down to 1-2 seconds is in a directory with no subdirs). gitea uses go-git | 17:51 |
| corvus | to traverse the git tree (i don't think it shells out for this). the only thing it caches is the commit count for the repo. so opportunities here are: 1) dig further into the go-git tree lookups to see if there's room for improvement; 2) look into caching more stuff (but i'm not sure we'd be able to have a large enough cache to keep the initial render time for nova reasonable) | 17:51 |
| *** ralonsoh has quit IRC | 17:51 | |
| clarkb | corvus: and maybe 3) don't lookup that data? or only do it on files? | 17:52 |
| clarkb | for memory usage it appears there are git settings we could tweak like core.packedGitWindowSize which limits how much of a pack file is mmap'd into memory at once | 17:52 |
| clarkb | by default it is mmaping the whole thing which seems to be where the memory use is coming from | 17:53 |
| clarkb | that said if the issue is due to mmaping files we may not suffer a huge penalty to lean on swap (since its going to disk anyway right?) | 17:53 |
| clarkb | so maybe leaning on swap isn't the end of the world here | 17:53 |
| corvus | clarkb: also mmap pages may not necessarily be resident | 17:55 |
| *** bhavikdbavishi has quit IRC | 17:56 | |
| clarkb | https://git-scm.com/docs/git-config/1.9.2#git-config-corebigFileThreshold may also help though I don't think nova has any files that are bigger than that in it | 17:56 |
| clarkb | so we would be tuning that value down | 17:56 |
| clarkb | https://git-scm.com/docs/git-config/1.9.2#git-config-corepackedGitWindowSize defaults to 1GB in our 64bit systems which lines up closely to the 1.3GB resident sizes we see | 17:57 |
| * clarkb checks how big nova's pack files are | 17:57 | |
| mordred | clarkb: although those settings would only impact things where gitea is shelling out to git, not necessarily places where it's using go-git | 17:58 |
| fungi | #status log deleted stale networking-mlnx repository mirror from the openstack organization on github at lennyb's request | 17:58 |
| openstackstatus | fungi: finished logging | 17:58 |
| clarkb | nova's largest packfile is 548MB | 17:58 |
| clarkb | mordred: unless go git also respects git config? | 17:58 |
| mordred | clarkb: yeah. it's a good question | 17:59 |
| clarkb | mordred: in this case the memory use we see is from shelled out git processes | 17:59 |
| mordred | that is an excellent point | 17:59 |
| clarkb | at least the large git processes I've been able to identify are | 17:59 |
| corvus | yeah, the "nova web page is slow" and "gitea is ooming" probs are tangentially related, but not identical | 18:00 |
| clarkb | https://git-scm.com/docs/git-config/1.9.2#git-config-packdeltaCacheSize is another option that may play into the OOMing | 18:00 |
| clarkb | worht noting that the cloning doesn't seem particularly slow though and seems more limited by bw than by memory on the gitea node | 18:03 |
| clarkb | which does make me wonder if dipping into swap a bit really would be fine | 18:03 |
| *** psachin has quit IRC | 18:04 | |
| fungi | we had swap partitions on the old cgit servers, right? | 18:04 |
| clarkb | yes | 18:04 |
| *** armax has joined #openstack-infra | 18:05 | |
| *** weifan has joined #openstack-infra | 18:06 | |
| *** irclogbot_1 has quit IRC | 18:07 | |
| clarkb | I'm wary of changing defaults in git's config just to accomodate memory usage beacuse it is using the memory to avoid unnecessary disk writes and aid performance. If we start tweaking that we open ourselves to much poorer performance potentially (and we have to track and set up that config everytime we configure a new repo or maybe we just do the big through special? manuals neutron nova) | 18:07 |
| *** altlogbot_2 has quit IRC | 18:07 | |
| *** irclogbot_3 has joined #openstack-infra | 18:08 | |
| *** altlogbot_3 has joined #openstack-infra | 18:08 | |
| *** electrofelix has quit IRC | 18:10 | |
| fungi | #status log recovered corrupt https://etherpad.openstack.org/p/security-sig-newsletter content and restored it at the old name after moving the corrupt pad to "broken-security-sig-newsletter" | 18:20 |
| openstackstatus | fungi: finished logging | 18:20 |
| fungi | gagehugo: ^ lmk if you need anything else | 18:20 |
| gagehugo | fungi: ty! | 18:21 |
| openstackgerrit | Matthew Thode proposed openstack/diskimage-builder master: support alternate portage directories https://review.opendev.org/671530 | 18:23 |
| fungi | gonna go grab some groceries around the corner real quick before the openstack release management meeting starts | 18:23 |
| fungi | brb | 18:23 |
| *** gfidente has quit IRC | 18:34 | |
| openstackgerrit | Matthew Thode proposed openstack/diskimage-builder master: support alternate portage directories https://review.opendev.org/671530 | 18:40 |
| *** weifan has quit IRC | 18:46 | |
| openstackgerrit | Matt McEuen proposed openstack/project-config master: Remove airship/utils from infrastructure https://review.opendev.org/671547 | 18:49 |
| openstackgerrit | Matt McEuen proposed openstack/project-config master: Remove airship/berth from infra https://review.opendev.org/671550 | 18:58 |
| *** weifan has joined #openstack-infra | 19:00 | |
| *** ricolin_ has joined #openstack-infra | 19:03 | |
| *** weifan has quit IRC | 19:04 | |
| *** weifan has joined #openstack-infra | 19:05 | |
| *** ricolin has quit IRC | 19:05 | |
| *** weifan has quit IRC | 19:05 | |
| *** weifan has joined #openstack-infra | 19:12 | |
| *** weifan has quit IRC | 19:15 | |
| *** siqbal90 has quit IRC | 19:19 | |
| *** _erlon_ has joined #openstack-infra | 19:23 | |
| *** weifan has joined #openstack-infra | 19:25 | |
| *** weifan has quit IRC | 19:30 | |
| *** weifan has joined #openstack-infra | 19:32 | |
| *** e0ne has joined #openstack-infra | 19:38 | |
| *** weifan has quit IRC | 19:39 | |
| *** weifan has joined #openstack-infra | 19:39 | |
| *** weifan has quit IRC | 19:44 | |
| *** whoami-rajat has quit IRC | 19:44 | |
| openstackgerrit | Matthew Thode proposed openstack/diskimage-builder master: support alternate portage directories https://review.opendev.org/671530 | 19:48 |
| *** joeguo has joined #openstack-infra | 19:54 | |
| corvus | clarkb: yeah, i think we should subscribe to the "be efficient by getting the process done faster" school of thought on this. | 20:01 |
| *** pcaruana has quit IRC | 20:01 | |
| openstackgerrit | James E. Blair proposed opendev/system-config master: WIP: Add gerrit to gitea job https://review.opendev.org/671162 | 20:02 |
| *** weifan has joined #openstack-infra | 20:04 | |
| *** tesseract has quit IRC | 20:05 | |
| openstackgerrit | James E. Blair proposed opendev/system-config master: Build a docker image of the existing gerrit 2.13 https://review.opendev.org/671457 | 20:09 |
| corvus | mordred: ^ i made a trivial fix to that | 20:09 |
| mordred | corvus: I agree with your trivial fix | 20:10 |
| mordred | corvus: I'm hoping that image helps us move forward and isn't just a waste of time, but it makes sense in my headhole | 20:12 |
| fungi | Shrews: i'm starting to think there may be some sort of image leak... noticing image 5675b0f7-510f-422b-bbda-325e5d7d4c27 (centos-7-1562751318) is hanging around in vexxhost-sjc1 but nodepool doesn't seem to think it exists any longer | 20:12 |
| corvus | mordred: totally | 20:12 |
| fungi | Shrews: that was uploaded 8 days ago | 20:12 |
| corvus | mordred: in https://review.opendev.org/671463 i think you need to add some fake entries | 20:14 |
| mordred | corvus: oh! so I do | 20:15 |
| mordred | corvus: I'm going to mark that WIP for now - MOC needs an update before we can move forward | 20:15 |
| corvus | kk | 20:16 |
| Shrews | fungi: http://paste.openstack.org/show/754568/ | 20:16 |
| mordred | corvus: I left the WIP with a note about what's IP | 20:16 |
| fungi | openstack.exceptions.ConflictException: ConflictException: 409: Client Error for url: https://image-sjc1.vexxhost.us/v2/images/f670e6be-953b-4d4b-a931-6cbb5b568410, Conflict | 20:16 |
| fungi | um, neat! | 20:16 |
| Shrews | fungi: it's been repeatedly trying to delete it. why do you think nodepool doesn't think it exists? | 20:17 |
| fungi | oh, i grepped image-list for it and didn't see it | 20:17 |
| fungi | maybe i fat-fingered thaht | 20:17 |
| Shrews | fungi: possible it doesn't show up in image-list given the state it is in. i forget offhand | 20:18 |
| Shrews | real question is... wth does that error mean? | 20:19 |
| fungi | yeah, that's mostly what i was wondering | 20:19 |
| clarkb | I think that means two images have the same id? | 20:20 |
| clarkb | usually you get that when using nes and there is a collision | 20:20 |
| fungi | https://opendev.org/openstack/openstacksdk/src/branch/master/openstack/exceptions.py#L119-L121 | 20:21 |
| fungi | not very helpful | 20:21 |
| clarkb | corvus I need to look at 06's swap usagebutmaybe we can make a smaller swapfile on the older servers with less disk then they will get updated when replaced | 20:21 |
| *** joeguo has quit IRC | 20:21 | |
| corvus | clarkb: ++ | 20:21 |
| *** joeguo has joined #openstack-infra | 20:22 | |
| Shrews | fungi: oh actually, my grep skills failed me too. i only see a single attempt to delete that image | 20:22 |
| clarkb | http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=66750&rra_id=all is current 06 swap usage. Based on that even 1GB swapfile would be ~3x max usage there | 20:23 |
| clarkb | http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=66636&rra_id=all 01 should be able to support a 1 or 2 GB swap file | 20:23 |
| fungi | Shrews: okay, so maybe nodepool didn't put that image back in the delete list when it encountered a 409 coming back from the api? | 20:23 |
| clarkb | infra-root so how about that as the immediate bandaid, I manually add a 1GB swapfile to 01-05,07-08 | 20:23 |
| corvus | clarkb: ++ | 20:24 |
| fungi | clarkb: sounds safe to me, and a reasonable precaution, yes | 20:24 |
| Shrews | fungi: maybe. the znode for in zookeeper is gone | 20:24 |
| Shrews | for it | 20:24 |
| clarkb | and anyone else want to review https://review.opendev.org/#/c/671102/ before it gets approved which will help ensure replacement gitea servers get properly sized swapfiles? | 20:24 |
| mordred | clarkb: ++ | 20:25 |
| mordred | clarkb: lgtm | 20:25 |
| clarkb | oh wow so my tumblweed updates from yesterday include warnings when you try to paste shell code beacuse ti might get executed (I'm pasting into a vi buffer) | 20:26 |
| corvus | clarkb: what process is producing those warnings? | 20:26 |
| mordred | corvus: probably systemd | 20:27 |
| corvus | via dbus | 20:28 |
| clarkb | looks like xfce4-terminal is doing it on input to it | 20:28 |
| mordred | corvus: running inside of a hidden browser tab | 20:28 |
| clarkb | according to xprop | 20:28 |
| corvus | clarkb: oh neat, that's not as terrible as i feared :) | 20:29 |
| *** eharney has quit IRC | 20:32 | |
| Shrews | fungi: well, it's possible we've identified the one cloud operation we do not retry. i can't recall if this is intentional or not. i'm guessing not | 20:32 |
| clarkb | ok all of the other gitea servers now have 1GB swapfiles | 20:34 |
| clarkb | 05 is the one with the least amount of disk after that with over 6GB | 20:34 |
| clarkb | the others are just under 10GB | 20:34 |
| clarkb | I'm going to trigger replication now across all of the giteas to ensure there are no lost refs on any of them | 20:34 |
| Shrews | fungi: i've copied the log file off to my home directory so i can look more closely at this tomorrow. i have a PT session coming up now | 20:35 |
| fungi | Shrews: no worries, just wanted to make sure i wasn't the only one aware of a possible image leak | 20:35 |
| clarkb | 01 is replicating now | 20:36 |
| openstackgerrit | James E. Blair proposed opendev/system-config master: WIP: run a gerrit container https://review.opendev.org/630406 | 20:38 |
| fungi | Shrews: okay if i delete that image manually, or do you want to keep it around for further inspection? | 20:38 |
| corvus | mordred: ^ that had bitrotted alot, i'm dragging it into the present | 20:38 |
| Shrews | fungi: go ahead | 20:39 |
| fungi | thanks! done, it deleted cleanly, no errors | 20:39 |
| fungi | so whatever it's problem, it was transient | 20:40 |
| fungi | er, its problem | 20:40 |
| openstackgerrit | Slawek Kaplonski proposed openstack/project-config master: Update Neutron grafana dashboard https://review.opendev.org/671568 | 20:40 |
| corvus | mordred: i'm confused about the error in http://logs.openstack.org/57/671457/2/check/system-config-build-image-gerrit/8c6fd8d/job-output.txt.gz#_2019-07-18_20_26_09_088293 | 20:43 |
| corvus | mordred: it looks like even the current build shouldn't work because there's no "FROM foo as gerrit" line in the dockefile | 20:44 |
| petevg | fungi: Hi! Is there a cap of half an hour on build time in the gate? And if so, is there any way to get that raised for a project? (I ask because x/microstack times out before the snap finishes building ... https://review.opendev.org/#/c/671358/) | 20:46 |
| fungi | petevg: that's just a default. you can override it with whatever timeout you need | 20:46 |
| openstackgerrit | Merged opendev/system-config master: Disable cloud launcher cron job during CI https://review.opendev.org/670946 | 20:47 |
| openstackgerrit | Merged opendev/system-config master: Publish .log files as text/plain https://review.opendev.org/670934 | 20:47 |
| openstackgerrit | Merged opendev/system-config master: Add mirror-update to run_all.sh https://review.opendev.org/670927 | 20:47 |
| fungi | petevg: an example: https://opendev.org/openstack/nova/src/branch/master/.zuul.yaml#L9 | 20:47 |
| fungi | (value is in seconds) | 20:47 |
| petevg | fungi: I was just looking for an example. Thank you :-) | 20:47 |
| fungi | you're welcome! | 20:48 |
| openstackgerrit | James E. Blair proposed opendev/system-config master: Remove targets from docker build https://review.opendev.org/671570 | 20:48 |
| clarkb | fungi: petevg there is a cap of 3 hours though iirc | 20:49 |
| corvus | mordred: ^ if that works, my best guess is that we just tightnened up the image build jobs after writing the original version of that. | 20:49 |
| fungi | petevg: just be wary that the longer a job runs, the longer your test results for a change will be delayed. so if you have a job that takes 2-3 hours to complete and it runs for most changes, then developers wait at least that long for aggregate job results | 20:49 |
| *** dpawlik has quit IRC | 20:53 | |
| openstackgerrit | Merged opendev/system-config master: mirror-update: update docs for mirror-update.opendev.org https://review.opendev.org/668833 | 20:54 |
| openstackgerrit | Merged opendev/system-config master: launch-node.py : add option to skip ipv6 address checks https://review.opendev.org/667548 | 20:54 |
| openstackgerrit | Merged opendev/system-config master: Use swapfile if no extra device is present https://review.opendev.org/671102 | 20:55 |
| *** priteau has quit IRC | 21:05 | |
| *** raissa has joined #openstack-infra | 21:06 | |
| *** raissa has quit IRC | 21:06 | |
| *** raissa has joined #openstack-infra | 21:07 | |
| *** raissa has joined #openstack-infra | 21:07 | |
| *** raissa has quit IRC | 21:08 | |
| *** eharney has joined #openstack-infra | 21:08 | |
| *** pkopec has quit IRC | 21:10 | |
| *** e0ne has quit IRC | 21:17 | |
| *** ekultails has quit IRC | 21:25 | |
| *** igordc has quit IRC | 21:27 | |
| *** igordc has joined #openstack-infra | 21:28 | |
| openstackgerrit | James E. Blair proposed opendev/system-config master: WIP: run a gerrit container https://review.opendev.org/630406 | 21:28 |
| openstackgerrit | Merged zuul/zuul master: Add "supercedes" pipeline option https://review.opendev.org/670670 | 21:33 |
| openstackgerrit | James E. Blair proposed opendev/system-config master: WIP: Add gerrit to gitea job https://review.opendev.org/671162 | 21:37 |
| *** beekneemech has joined #openstack-infra | 21:38 | |
| *** beekneemech has quit IRC | 21:38 | |
| openstackgerrit | James E. Blair proposed opendev/system-config master: Build a docker image of the existing gerrit 2.13 https://review.opendev.org/671457 | 21:39 |
| openstackgerrit | Matthew Thode proposed openstack/diskimage-builder master: support alternate portage directories https://review.opendev.org/671530 | 21:41 |
| mordred | corvus: huh. yeah - that was some of our earliest dockerfileing | 21:55 |
| openstackgerrit | Matt McEuen proposed openstack/project-config master: Remove airship/berth from infra https://review.opendev.org/671550 | 22:00 |
| openstackgerrit | Matt McEuen proposed openstack/project-config master: Remove airship/utils from infrastructure https://review.opendev.org/671547 | 22:01 |
| *** weifan has quit IRC | 22:01 | |
| *** weifan has joined #openstack-infra | 22:02 | |
| *** bnemec has quit IRC | 22:03 | |
| *** weifan has quit IRC | 22:06 | |
| *** weifan has joined #openstack-infra | 22:06 | |
| *** slaweq has quit IRC | 22:09 | |
| mnaser | was there an infra issue at some point inside gitea today? | 22:09 |
| mnaser | http://logs.openstack.org/05/671105/2/check/openstack-ansible-upgrade-aio_metal-ubuntu-bionic/44da912/job-output.txt.gz#_2019-07-18_20_40_35_367422 | 22:09 |
| mnaser | but the commit does exist https://opendev.org/openstack/ansible-hardening/commit/ae01cbc824c7cf8aa7eac054fde481dacc441b68 | 22:09 |
| mnaser | (this is the first time we do upgrade jobs though, so we might be broken, but i fail to see how it woudl do that) | 22:10 |
| clarkb | mnaser: thats the issue I've been working through the last little while | 22:10 |
| *** weifan has quit IRC | 22:10 | |
| clarkb | (or at least is likely that) | 22:10 |
| mnaser | clarkb: gah, sorry, your name is all white in my irc client so i kinda get blind to it | 22:10 |
| mnaser | i see the text about replication and what not :( | 22:10 |
| *** weifan has joined #openstack-infra | 22:10 | |
| clarkb | basically what happens is if the gitea server OOMs while gerrit si replicating there is a non zero chance that oomkiller kills the git process that is handling the gerrit replication | 22:12 |
| *** weifan has quit IRC | 22:12 | |
| clarkb | and when that happens the ref doesn't exist on that backend until some other event causes it to rereplicate | 22:12 |
| fungi | for some arbitrary commit | 22:12 |
| clarkb | The ooming is happening because git can be quite large to handle nova (and probably openstack manuals and neutron) | 22:12 |
| clarkb | so what I've done is added swapfiles to all of the nodes which seems to have fixed it for gitea06 whihc was my original test node | 22:12 |
| clarkb | unfortunately all of the nodes but 06 are somewhat disk constrainted so started with only 1GB swapfiles | 22:13 |
| clarkb | And now I'm rereplicating everything to ensure they are caught up | 22:13 |
| clarkb | but this should hopefully prevent oomkiller from running again | 22:13 |
| fungi | do we think there's a chance we have jobs recently starting to clone things from gitea instead of using local copies? | 22:13 |
| clarkb | and we'll be stable going forward | 22:13 |
| fungi | (or maybe even not so recently starting) | 22:13 |
| *** bnemec has joined #openstack-infra | 22:13 | |
| clarkb | fungi: I identified at least one job that was hitting opendev.org instaed of zuul repos but it wasn't doing full clones just fetching a file at a ref | 22:14 |
| fungi | ahh | 22:14 |
| clarkb | that info is all in haproxy logs now though if we want to uniq -c and then grep the results out of nodepool/zuul | 22:14 |
| clarkb | the next step which I'm somewhat less enthusiastic about but we've been meaning to do it anyway is to rebuild all of the gitea nodes with more disk | 22:15 |
| fungi | i just remember that time we accidentally uploaded images with incomplete git mirrors (back in the old days) and our git farm melted down to slag in moments | 22:15 |
| clarkb | maybe I can get help with that and get checked on my gitea backend rebuild docs | 22:15 |
| *** weifan has joined #openstack-infra | 22:15 | |
| fungi | yeah, i ought to be able to find time to try out your docs on at least one | 22:16 |
| fungi | (but not tonight) | 22:16 |
| *** eharney has quit IRC | 22:16 | |
| clarkb | ya I think I've got a good enough bandaid for the next little while | 22:16 |
| clarkb | separately corvus thinks he has identifed the cause of some of the slowness browsing nova in gitea | 22:18 |
| clarkb | corvus: ^ that might be worth a response to sean's ml thread? | 22:18 |
| *** weifan has quit IRC | 22:20 | |
| corvus | clarkb: i'd rather not give anyone the incorrect impression that i'm working on that | 22:21 |
| corvus | if there's someone to collaborate with, sure, but i don't think anyone has respended yet? | 22:22 |
| clarkb | ya I haven't seen anyone volunteer yet | 22:22 |
| clarkb | the last gitea backend is replicating now (06) | 22:36 |
| clarkb | in theory we'll avoid these ref not found issues now | 22:36 |
| *** bnemec has quit IRC | 22:42 | |
| *** bnemec has joined #openstack-infra | 22:43 | |
| *** hwoarang has quit IRC | 22:44 | |
| corvus | fingerprint = base64.b64encode(hashlib.sha256(base64.b64decode(public_key.encode('ascii'))).digest()).decode('ascii').replace('=','') | 22:44 |
| corvus | just, you know, in case you were wondering how to do that in python | 22:45 |
| *** armax has quit IRC | 22:45 | |
| *** hwoarang has joined #openstack-infra | 22:45 | |
| corvus | actually, it's better than that if you don't already start with the b64 public key | 22:46 |
| corvus | fingerprint = base64.b64encode(hashlib.sha256(base64.b64decode(paramiko.RSAKey.from_private_key_file(ssh_key_file).ssh_key.get_base64().encode('ascii'))).digest()).decode('ascii').replace('=','') | 22:46 |
| mordred | corvus: that makes perfect sense | 22:57 |
| *** tkajinam has joined #openstack-infra | 22:58 | |
| *** mattw4 has quit IRC | 23:07 | |
| *** weifan has joined #openstack-infra | 23:08 | |
| donnyd | clarkb: Soooo my equipment was supposed to be here today, and instead of a machine with 7 pcie slots.. i got a box full of someone elses parts... So I have to return those and wait.. again for the right stuff | 23:09 |
| donnyd | should be here the 24th | 23:09 |
| clarkb | whoops | 23:09 |
| donnyd | So I lowered the quota down to 50 to see if it helps with the timeouts till i get it fixed | 23:10 |
| donnyd | so frustating | 23:10 |
| *** rcernin has joined #openstack-infra | 23:15 | |
| *** aaronsheffield has quit IRC | 23:17 | |
| *** tdasilva has quit IRC | 23:19 | |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Allow projects to borrow ssh keys https://review.opendev.org/671637 | 23:34 |
| corvus | mordred: i'm going to hand https://review.opendev.org/671457 back over to you now -- the builds are running, but they are both failing (probably something you could debug with a local docker build) | 23:40 |
| openstackgerrit | James E. Blair proposed opendev/system-config master: WIP: Add gerrit to gitea job https://review.opendev.org/671162 | 23:45 |
| *** armax has joined #openstack-infra | 23:50 | |
| openstackgerrit | James E. Blair proposed opendev/system-config master: WIP: run a gerrit container https://review.opendev.org/630406 | 23:50 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!