| rdogerrit | rdo-trunk created nfvinfo master: Promote CBS tags update for nfvsig-2.11-release https://review.rdoproject.org/r/c/nfvinfo/+/34624 | 00:10 |
|---|---|---|
| rdogerrit | rdo-trunk created nfvinfo master: Promote CBS tags update for nfvsig-2.13-release https://review.rdoproject.org/r/c/nfvinfo/+/34625 | 00:11 |
| rdogerrit | rdo-trunk created nfvinfo master: Promote CBS tags update for nfvsig-2.15-release https://review.rdoproject.org/r/c/nfvinfo/+/34626 | 00:11 |
| rdogerrit | rdo-trunk created rdoinfo master: Bump rdoinfo xena-uc tags to latest upper-constraints https://review.rdoproject.org/r/c/rdoinfo/+/34627 | 00:13 |
| rlandy|ruck|bbl | stevebaker: great - uploaded image! | 01:30 |
| *** rlandy|ruck|bbl is now known as rlandy|ruck | 01:31 | |
| rlandy|ruck | '/home/zuul/overcloud-full.qcow2': Could not open '/home/zuul/overcloud-full.qcow2': No such file or directory | 01:31 |
| rlandy|ruck | hmm ok - we have some work to do there | 01:31 |
| stevebaker | rlandy|ruck: where is that? | 01:37 |
| rlandy|ruck | stevebaker: in the failing fs001 job that ran afterwards | 01:40 |
| rlandy|ruck | from the testproject | 01:41 |
| rlandy|ruck | https://logserver.rdoproject.org/82/16882/35/check/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-master/964a292/job-output.txt | 01:41 |
| stevebaker | rlandy|ruck: ok, I'll look into that, and the fs001 failure on the quickstart change https://review.rdoproject.org/zuul/build/28d7b1c331db48d3b45cfc955276e948/log/job-output.txt | 01:48 |
| stevebaker | rlandy|ruck: but the image is built and uploaded, so we could go ahead with landing this? https://review.rdoproject.org/r/c/rdo-jobs/+/34251 | 01:49 |
| * rlandy|ruck looks | 01:54 | |
| rlandy|ruck | stevebaker: yep - I'll merge that | 01:54 |
| stevebaker | cool | 01:57 |
| rdogerrit | Merged rdo-jobs master: Upload overcloud-hardened-uefi-full in main integration https://review.rdoproject.org/r/c/rdo-jobs/+/34251 | 02:04 |
| rlandy|ruck | stevebaker: I can look into the modify image failure more tomorrow | 02:09 |
| rlandy|ruck | well, my tomorrow | 02:09 |
| stevebaker | rlandy|ruck: ok, I'll let you know how I get on | 02:10 |
| rdogerrit | rdo-trunk created openstack/glance-distgit rpm-master: openstack-glance: failed to build cdd1f372 https://review.rdoproject.org/r/c/openstack/glance-distgit/+/34628 | 03:47 |
| rdogerrit | Yatin Karel proposed nfvinfo master: Promote CBS tags update for nfvsig-2.13-release https://review.rdoproject.org/r/c/nfvinfo/+/34625 | 04:53 |
| rdogerrit | Yatin Karel proposed nfvinfo master: Promote CBS tags update for nfvsig-2.15-release https://review.rdoproject.org/r/c/nfvinfo/+/34626 | 04:54 |
| *** marios is now known as marios|ruck | 05:51 | |
| *** ykarel|away is now known as ykarel | 06:12 | |
| *** amoralej|off is now known as amoralej | 07:00 | |
| ykarel | amoralej, would first need https://review.rdoproject.org/r/c/nfvinfo/+/34596 | 07:02 |
| ykarel | others rebased over it | 07:02 |
| amoralej | ah, right | 07:12 |
| amoralej | merging | 07:12 |
| amoralej | ykarel, any finding with the devstack issue? | 07:12 |
| rdogerrit | Merged nfvinfo master: Don't trigger master/wallaby jobs in ovs/ovn 2.13 updates https://review.rdoproject.org/r/c/nfvinfo/+/34596 | 07:16 |
| ykarel | amoralej, looks performance related, on the held node running the failed test manually passes | 07:16 |
| ykarel | and running group of tests it fails randomly | 07:17 |
| amoralej | we may try the uwsgi options reported in stackoverflow | 07:17 |
| amoralej | just to test | 07:17 |
| amoralej | makes sense to be perf | 07:17 |
| ykarel | when tempest runs with high concurrency, i noticed high load in system | 07:17 |
| ykarel | and when i tried with concurrency <=2 it worked fine | 07:18 |
| amoralej | but it may be an issue, i think those servers have the same capacity | 07:18 |
| ykarel | so i updated test patch to use concurrency =2 | 07:18 |
| amoralej | ah | 07:18 |
| amoralej | that's the last recheck | 07:18 |
| amoralej | right? | 07:18 |
| ykarel | yes | 07:18 |
| amoralej | ok, good | 07:18 |
| ykarel | it's running currently | 07:18 |
| ykarel | amoralej, wrt uwsgi option i am testing in held node | 07:18 |
| ykarel | listen = 1024 | 07:19 |
| ykarel | u mean ^ setting or something else? | 07:19 |
| amoralej | ykarel, https://stackoverflow.com/questions/36156887/uwsgi-raises-oserror-write-error-during-large-request | 07:23 |
| amoralej | buffer-size=8192 | 07:24 |
| amoralej | ignore-sigpipe=true | 07:24 |
| amoralej | ignore-write-errors=true | 07:24 |
| rdogerrit | Merged nfvinfo master: Promote CBS tags update for nfvsig-2.13-release https://review.rdoproject.org/r/c/nfvinfo/+/34625 | 07:24 |
| ykarel | amoralej, ohkk, let me try those | 07:25 |
| rdogerrit | rdo-trunk created openstack/diskimage-builder-distgit victoria-rdo: diskimage-builder: failed to build 585f34e0 https://review.rdoproject.org/r/c/openstack/diskimage-builder-distgit/+/34629 | 07:47 |
| *** rpittau|afk is now known as rpittau | 07:49 | |
| *** bhagyashris_ is now known as bhagyashris | 08:05 | |
| ykarel | amoralej, tempest tests passed with concurrency 2 | 08:08 |
| ykarel | although tempest itself failed | 08:08 |
| amoralej | how it failed? | 08:08 |
| ykarel | https://logserver.rdoproject.org/49/34549/21/check/devstack-platform-centos-9-stream/82f1c32/job-output.txt | 08:08 |
| ykarel | Worker 1 (574 tests) => N/A | 08:09 |
| ykarel | ERROR: InvocationError for command /opt/stack/tempest/.tox/tempest/bin/tempest run --regex '(?!.*\[.*\bslow\b.*\])(^tempest\.api)' --concurrency=2 (exited with code 1) | 08:09 |
| amoralej | weird | 08:10 |
| amoralej | no error message, right? | 08:10 |
| ykarel | yes | 08:12 |
| tosky | {1} tempest.api.volume.test_volumes_backup.VolumesBackupsTest.test_backup_create_attached_volume [] ... inprogress <-- never seen this | 08:16 |
| amoralej | ah, good catch | 08:17 |
| amoralej | i was looking for failures | 08:17 |
| amoralej | but right, that's in progress | 08:17 |
| amoralej | and note Worker 1 has no time | 08:18 |
| amoralej | suspictious | 08:18 |
| ykarel | i think i have seen such issue in past | 08:22 |
| ykarel | related to subunit | 08:22 |
| ykarel | i mean https://bugzilla.redhat.com/show_bug.cgi?id=1728646 | 08:27 |
| ykarel | https://bugs.launchpad.net/cinder/+bug/1728640 | 08:27 |
| rdogerrit | rdo-trunk created openstack/glance-distgit rpm-master: openstack-glance: failed to build e89c8652 https://review.rdoproject.org/r/c/openstack/glance-distgit/+/34630 | 08:45 |
| jcapitao | ykarel: Caused by: com.sonymobile.tools.gerrit.gerritevents.ssh.SshException: error: fatal: Applying label "Verified": 1 is restricted | 08:57 |
| jcapitao | from Jenkins logs when gerrit plugin tries to comment the patchset | 08:57 |
| ykarel | ahh | 08:59 |
| ykarel | jcapitao, u getting it in new jenkins, right? | 08:59 |
| jcapitao | the plugin uses the gerrit cli, I can reproduce the issue locally with "ssh rdo-ci gerrit review 801456,1 --label Verified=+1 --code-review 0" | 09:00 |
| jcapitao | ykarel: yes | 09:00 |
| jcapitao | the issue we have in ci.c.o should be the same | 09:03 |
| ykarel | yes | 09:05 |
| ykarel | seems some permission issue with rdo-ci user | 09:05 |
| jcapitao | yeah I suspect Gerrit permissions have changed for rdo-ci user | 09:06 |
| jcapitao | do we have the webui password for rdo-ci user ? | 09:06 |
| ykarel | no idea about it's creds | 09:07 |
| ykarel | jcapitao, https://docs.opendev.org/opendev/system-config/latest/third_party.html#permissions-on-your-third-party-system | 09:12 |
| ykarel | amoralej, disable-write-exception=true is the parameter which suppress OSError | 09:24 |
| amoralej | and, with that | 09:24 |
| ykarel | but tempest tests still fail | 09:24 |
| amoralej | with similar issues? | 09:24 |
| ykarel | amoralej, yes similar errors on tempest side | 09:26 |
| amoralej | ok | 09:26 |
| *** bhagyashris_ is now known as bhagyashris | 09:35 | |
| jcapitao | ykarel: thanks for pointing me to the URL | 09:58 |
| jcapitao | https://review.opendev.org/c/opendev/ci-sandbox/+/801748 | 09:58 |
| jcapitao | so we can confirm it's issue permission | 09:59 |
| rdogerrit | User pojadhav created rdo-jobs master: Refactor CentOS-7 FS010 job name to match with upstream https://review.rdoproject.org/r/c/rdo-jobs/+/34632 | 10:21 |
| ykarel | jcapitao, so is it working? i see some messages from CI on ^? | 10:22 |
| ykarel | okk comments are there but issue is with vote | 10:23 |
| *** ykarel is now known as ykarel|lunch | 10:25 | |
| jcapitao | ykarel|lunch: there is the +1 vote from RedHat RDO CI in https://review.opendev.org/c/opendev/ci-sandbox/+/801748 | 10:26 |
| rdogerrit | User pojadhav created rdo-infra/ci-config master: Refactor CentOS-7 FS010 job to match with upstream https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/34633 | 10:32 |
| *** jcapitao is now known as jcapitao_lunch | 10:40 | |
| rdogerrit | rdo-trunk created openstack/os-net-config-distgit victoria-rdo: os-net-config-13.2.1-1 https://review.rdoproject.org/r/c/openstack/os-net-config-distgit/+/34634 | 10:50 |
| rdogerrit | rdo-trunk created puppet/puppet-tripleo-distgit victoria-rdo: puppet-tripleo-13.6.3-1 https://review.rdoproject.org/r/c/puppet/puppet-tripleo-distgit/+/34635 | 10:55 |
| rdogerrit | rdo-trunk created openstack/tripleoclient-distgit victoria-rdo: python-tripleoclient-14.2.2-1 https://review.rdoproject.org/r/c/openstack/tripleoclient-distgit/+/34636 | 10:55 |
| rdogerrit | rdo-trunk created openstack/tripleo-ansible-distgit victoria-rdo: tripleo-ansible-2.4.0-1 https://review.rdoproject.org/r/c/openstack/tripleo-ansible-distgit/+/34637 | 10:56 |
| rdogerrit | rdo-trunk created openstack/tripleo-common-distgit victoria-rdo: openstack-tripleo-common-13.2.2-1 https://review.rdoproject.org/r/c/openstack/tripleo-common-distgit/+/34638 | 10:56 |
| rdogerrit | rdo-trunk created openstack/tripleo-heat-templates-distgit victoria-rdo: openstack-tripleo-heat-templates-13.4.0-1 https://review.rdoproject.org/r/c/openstack/tripleo-heat-templates-distgit/+/34639 | 10:56 |
| rdogerrit | rdo-trunk created openstack/tripleo-image-elements-distgit victoria-rdo: openstack-tripleo-image-elements-12.2.2-1 https://review.rdoproject.org/r/c/openstack/tripleo-image-elements-distgit/+/34640 | 10:56 |
| rdogerrit | rdo-trunk created openstack/tripleo-validations-distgit victoria-rdo: openstack-tripleo-validations-13.4.0-1 https://review.rdoproject.org/r/c/openstack/tripleo-validations-distgit/+/34641 | 10:56 |
| *** ykarel|lunch is now known as ykarel | 11:52 | |
| ykarel | jcapitao_lunch, okk good | 11:52 |
| ykarel | so need to follow with opendev why permission on account missing/removed | 11:54 |
| ykarel | amoralej, so last run green on https://review.rdoproject.org/r/c/testproject/+/34549 | 12:03 |
| ykarel | that tempest cli failure too seems random | 12:03 |
| amoralej | so, the only change is concurrency:2 | 12:04 |
| amoralej | right? | 12:04 |
| ykarel | yes right | 12:05 |
| ykarel | last two runs were with concurrency 2 | 12:05 |
| ykarel | and all tempest tests passed in both | 12:05 |
| ykarel | just the first run failed with that inprogress one | 12:06 |
| amoralej | tosky, geguileo ^ | 12:06 |
| tosky | amoralej, ykarel: what was the concurrency before this change? 4? | 12:07 |
| ykarel | tosky, yes, nproc/2 | 12:08 |
| ykarel | and nproc=8 | 12:08 |
| tosky | so resources | 12:08 |
| ykarel | yes, i noticed there was high load when running with concurrency 3+ | 12:15 |
| *** jcapitao_lunch is now known as jcapitao | 12:17 | |
| *** amoralej is now known as amoralej|lunch | 12:28 | |
| *** ykarel_ is now known as ykarel | 12:38 | |
| *** amoralej|lunch is now known as amoralej | 13:29 | |
| spotz_ | Amoralej you see the email about the nova gates being blocked from block storage? Sounds like your issue from yesterday | 14:55 |
| amoralej | spotz_, lemme check, anyway it seems performance related, ykarel_ made it work by decreasing concurrency in tempest | 14:56 |
| *** ykarel_ is now known as ykarel | 14:56 | |
| spotz_ | Interesting but glad some progress was made | 14:58 |
| amoralej | spotz_, yes, seems unrelated | 14:59 |
| amoralej | thanks for the heads up anyway | 14:59 |
| sshnaidm | amoralej, hey, do you know if we can speed it up a little? https://src.fedoraproject.org/rpms/ansible-collection-ansible-netcommon/pull-request/1# | 15:06 |
| sshnaidm | maybe to ping some people..? | 15:07 |
| amoralej | yeah, in fedora-devel | 15:07 |
| amoralej | let's try | 15:07 |
| sshnaidm | in which network it is now? | 15:07 |
| *** ykarel is now known as ykarel|away | 15:08 | |
| sshnaidm | seems libera.. | 15:08 |
| *** rlandy is now known as rlandy|ruck | 15:31 | |
| *** pojadhav is now known as pojadhav|afk | 15:56 | |
| *** rpittau is now known as rpittau|afk | 16:02 | |
| *** marios|ruck is now known as marios|out | 16:24 | |
| rdogerrit | Joel Capitao created rdo-infra/ci-config master: Add TripleO quickstart gate jobs to new Jenkins https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/34643 | 16:25 |
| rdogerrit | Merged openstack/tripleo-image-elements-distgit victoria-rdo: openstack-tripleo-image-elements-12.2.2-1 https://review.rdoproject.org/r/c/openstack/tripleo-image-elements-distgit/+/34640 | 16:29 |
| rdogerrit | Merged openstack/os-net-config-distgit victoria-rdo: os-net-config-13.2.1-1 https://review.rdoproject.org/r/c/openstack/os-net-config-distgit/+/34634 | 16:30 |
| rdogerrit | Merged openstack/tripleo-validations-distgit victoria-rdo: openstack-tripleo-validations-13.4.0-1 https://review.rdoproject.org/r/c/openstack/tripleo-validations-distgit/+/34641 | 16:30 |
| rdogerrit | Merged puppet/puppet-tripleo-distgit victoria-rdo: puppet-tripleo-13.6.3-1 https://review.rdoproject.org/r/c/puppet/puppet-tripleo-distgit/+/34635 | 16:31 |
| rdogerrit | Merged openstack/tripleo-heat-templates-distgit victoria-rdo: openstack-tripleo-heat-templates-13.4.0-1 https://review.rdoproject.org/r/c/openstack/tripleo-heat-templates-distgit/+/34639 | 16:31 |
| rdogerrit | Merged openstack/tripleo-common-distgit victoria-rdo: openstack-tripleo-common-13.2.2-1 https://review.rdoproject.org/r/c/openstack/tripleo-common-distgit/+/34638 | 16:31 |
| *** amoralej is now known as amoralej|off | 16:32 | |
| rdogerrit | Merged openstack/tripleo-ansible-distgit victoria-rdo: tripleo-ansible-2.4.0-1 https://review.rdoproject.org/r/c/openstack/tripleo-ansible-distgit/+/34637 | 16:32 |
| rdogerrit | Merged openstack/tripleoclient-distgit victoria-rdo: python-tripleoclient-14.2.2-1 https://review.rdoproject.org/r/c/openstack/tripleoclient-distgit/+/34636 | 16:34 |
| rdogerrit | Nicolas HICHER created config master: zuul: update psi_public_nodepool_tripleo secret https://review.rdoproject.org/r/c/config/+/34644 | 19:52 |
| rdogerrit | Merged config master: zuul: update psi_public_nodepool_tripleo secret https://review.rdoproject.org/r/c/config/+/34644 | 20:01 |
| *** sshnaidm is now known as sshnaidm|afk | 20:10 | |
| *** rlandy|ruck is now known as rlandy|ruck|biab | 23:17 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!