| *** rlandy has quit IRC | 00:28 | |
| *** rfolco|rover has quit IRC | 01:49 | |
| *** jmasud has joined #oooq | 02:05 | |
| *** chem has quit IRC | 02:07 | |
| *** chem has joined #oooq | 02:09 | |
| *** jmasud has quit IRC | 02:30 | |
| *** skramaja has joined #oooq | 03:56 | |
| *** dmellado has quit IRC | 04:25 | |
| *** ykarel|away is now known as ykarel | 04:42 | |
| *** jmasud has joined #oooq | 04:44 | |
| *** jmasud has quit IRC | 04:55 | |
| *** jtomasek has joined #oooq | 04:56 | |
| *** marios has joined #oooq | 04:57 | |
| *** jmasud has joined #oooq | 05:03 | |
| *** ratailor has joined #oooq | 05:15 | |
| *** ysandeep|away is now known as ysandeep | 05:22 | |
| *** saneax_AFK has joined #oooq | 05:29 | |
| pojadhav|ruck | ykarel, https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-master&job_name=periodic-tripleo-centos-8-master-containers-build-push still failing :( issue is like something different. | 05:45 |
|---|---|---|
| pojadhav|ruck | ykarel, do you have any idea what could be the other things we need to check to avoid this "urllib3.exceptions.LocationParseError: Failed to parse: https://trunk.rdoproject.org/api-centos8-master-uc/api/report_result" | 05:46 |
| ykarel | pojadhav|ruck, somehow updated urllib3 is not installed in those jobs | 05:46 |
| ykarel | need to find that | 05:46 |
| ykarel | pojadhav|ruck, were you able to find root cause for it? | 05:47 |
| pojadhav|ruck | ykarel, okay | 05:47 |
| ykarel | why it started happening suddenly | 05:47 |
| pojadhav|ruck | ykarel, nope :( | 05:47 |
| ykarel | ack /me will try to find later once we get why old urllib3 is being installed | 05:48 |
| pojadhav|ruck | ykarel, ok :) | 05:48 |
| ykarel | or an alternative to fix it for time being | 05:49 |
| ykarel | pojadhav|ruck, also u checked what all releases affected? | 05:58 |
| pojadhav|ruck | ykarel, okay | 05:58 |
| ykarel | i see bug is marked fix released | 05:58 |
| ykarel | so good to update it in fix all release whichever affected | 05:59 |
| *** jmasud has quit IRC | 05:59 | |
| pojadhav|ruck | ykarel, hmm right | 06:00 |
| *** jmasud has joined #oooq | 06:00 | |
| *** jmasud has quit IRC | 06:27 | |
| *** udesale has joined #oooq | 06:28 | |
| *** jmasud has joined #oooq | 06:39 | |
| *** jbadiapa has joined #oooq | 06:55 | |
| *** saneax_AFK is now known as saneax_ | 07:00 | |
| *** jmasud has quit IRC | 07:01 | |
| *** ccamacho has joined #oooq | 07:08 | |
| *** amoralej|off is now known as amoralej | 07:23 | |
| *** jmasud has joined #oooq | 07:30 | |
| *** tosky has joined #oooq | 07:32 | |
| *** jmasud has quit IRC | 07:51 | |
| *** jpena|off is now known as jpena | 07:59 | |
| *** jmasud has joined #oooq | 08:02 | |
| *** ysandeep is now known as ysandeep|afk | 08:04 | |
| *** kopecmartin|pto is now known as kopecmartin | 08:13 | |
| marios | sshnaidm|afk: o/ i vaguely remember you mentioning that before but can't find more info from commits "'collections' is not a valid attribute? https://23fd78c0f6b01b9c4209-59c294b8f46eb75cda2bd004e2a1c97f.ssl.cf1.rackcdn.com/733818/1/check/tripleo-buildimage-overcloud-full-centos-8/d4674cc/job-output.txt | 08:16 |
| *** jmasud has quit IRC | 08:21 | |
| *** jmasud has joined #oooq | 08:22 | |
| sshnaidm|afk | marios, looking | 08:22 |
| *** sshnaidm|afk is now known as sshnaidm | 08:22 | |
| *** dtantsur|afk is now known as dtantsur | 08:23 | |
| marios | sshnaidm: thanks | 08:23 |
| sshnaidm | marios, is this collection installed? | 08:26 |
| marios | sshnaidm: no i haven't done anything more that that review ... to i need to add it to the project requirements probably? | 08:27 |
| marios | s/to/do | 08:27 |
| sshnaidm | marios, it's not installed in a job, the error message is very misleading.. | 08:27 |
| marios | sshnaidm: sthing like quickstart-extras-requirements.txt:9:git+https://opendev.org/openstack/tripleo-operator-ansible/#egg=tripleo-operator-ansible | 08:27 |
| marios | ? | 08:27 |
| sshnaidm | marios, we install it in jobs that uses quickstart by this, yes ^ | 08:28 |
| marios | sshnaidm: ok thanks for the check, i will try and add it to the ci repo then | 08:28 |
| sshnaidm | marios, in build job we don't use quickstart-extras maybe? need to check | 08:28 |
| marios | sshnaidm: this one is kind of a special case cos the role lives in tripleo-ci | 08:28 |
| marios | sshnaidm: well the build image role in t-q-e actually uses the role in tripleo-ci | 08:28 |
| sshnaidm | marios, well, we have it in https://github.com/openstack/tripleo-quickstart/blob/master/quickstart-extras-requirements.txt#L9 | 08:30 |
| sshnaidm | marios, so need to figure out how we install it in build job that don't use it | 08:30 |
| *** jfrancoa has joined #oooq | 08:31 | |
| marios | sshnaidm: yeah that's the one i pointed to above... also looking at tripleo-ci i see a few hits for that like https://opendev.org/openstack/tripleo-ci/src/branch/master/roles/run-test/templates/toci_gate_test.sh.j2#L141 | 08:31 |
| marios | sshnaidm: but ok at least i have somethign to investigate now thanks | 08:31 |
| *** ykarel is now known as ykarel|lunch | 08:31 | |
| marios | brb | 08:32 |
| *** derekh has joined #oooq | 08:48 | |
| *** pojadhav|ruck is now known as pojadhav|lunch | 08:59 | |
| zbr | panda: can you have a look at https://logserver.rdoproject.org/04/27504/10/check/molecule-container-push-delegated-centos-7/841fed8/tox/reports.html ? | 09:12 |
| panda | zbr: relative path is wrong. | 09:13 |
| zbr | yep, but the question is why. what is the cwd of zuul jobs? | 09:14 |
| *** ykarel|lunch is now known as ykarel | 09:15 | |
| zbr | as part of making these playbooks work also with remote hosts, we need to re-think that part. | 09:15 |
| arxcruz | akahat: try to add environment in the include_task | 09:15 |
| zbr | even if in that particular case remote is the same as local, but using ssh connection | 09:16 |
| zbr | which is a good way to find bugs | 09:16 |
| akahat | arxcruz, ack | 09:16 |
| panda | zbr: the only way I see it working without a relative path is if we build and install a python package. Right now it may be quicker to just adjust the relative path to make it work | 09:17 |
| arxcruz | sshnaidm: https://fea0582e4c3b9beb3429-889b82e09682580bca3a9c4b43adfe87.ssl.cf5.rackcdn.com/718639/5/check/tripleo-ci-centos-8-standalone/473c5fd/job-output.txt | 09:17 |
| arxcruz | fails with the ansible collections, this is https://review.opendev.org/718639 | 09:17 |
| sshnaidm | arxcruz, yeah, I saw, the collection wasn't installed there, will look into it today | 09:18 |
| zbr | what kind of contoction is that? looks little bit as python module, but only from far away | 09:21 |
| zbr | do we have an expected deployment location relative to user home? i can just copy the files for the moment | 09:21 |
| arxcruz | kopecmartin: https://tree.taiga.io/project/tripleo-ci-board/task/1787?kanban-status=1447275 I create since you worked on this, please update when you have time | 09:26 |
| kopecmartin | arxcruz: sure | 09:27 |
| kopecmartin | arxcruz: our work within the container nightmare is done, do we wanna keep there a task to update the doc (just as a reminder as the docs update are out of our hands) ? | 09:36 |
| arxcruz | kopecmartin: I don't know, I'll ask in the meeting | 09:37 |
| kopecmartin | yeah, let's make it a small bullet in the agenda | 09:37 |
| *** ysandeep|afk is now known as ysandeep | 09:39 | |
| *** jschlueter has quit IRC | 09:43 | |
| arxcruz | amoralej: regarding https://bugzilla.redhat.com/show_bug.cgi?id=1839468 what do we need to do? I saw your comment there | 09:48 |
| openstack | bugzilla.redhat.com bug 1839468 in openstack-tempest "[DDF] This binary is missing in the container" [High,Assigned] - Assigned to chkumar | 09:48 |
| arxcruz | kopecmartin: ^ | 09:48 |
| amoralej | arxcruz, chandankumar is taking care afaik | 09:48 |
| *** jmasud has quit IRC | 09:53 | |
| chandankumar | arxcruz, https://tree.taiga.io/project/tripleo-ci-board/task/1788 had taken care here | 09:57 |
| *** ykarel is now known as ykarel|afk | 10:00 | |
| chandankumar | amoralej, Hello | 10:00 |
| chandankumar | amoralej, can you take a look at this issue https://logserver.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-8-buildimage-overcloud-full-master/419668a/build.log | 10:00 |
| chandankumar | amoralej, error: unpacking of archive failed on file /usr/share/ansible/plugins/modules/pacemaker_cluster.py;5eded785: cpio: open failed - Inappropriate ioctl for device | 10:01 |
| amoralej | cheecking | 10:01 |
| chandankumar | it seems to be some how rpm is corrupted | 10:01 |
| amoralej | mmm | 10:01 |
| amoralej | that's ususally because of conflicting file vs symlink | 10:01 |
| chandankumar | anything we can do to prevent that? | 10:02 |
| amoralej | it's weird | 10:03 |
| amoralej | may it be installing ansible-pacemaker from source before installing the rpm? | 10:04 |
| amoralej | let me try to reproduce out of the job | 10:06 |
| chandankumar | ah let me find that | 10:06 |
| chandankumar | pojadhav|lunch, ^^ | 10:06 |
| *** pojadhav|lunch is now known as pojadhav|ruck | 10:17 | |
| pojadhav|ruck | chandankumar, okay | 10:17 |
| pojadhav|ruck | amoralej, chandankumar : bug reported - https://bugs.launchpad.net/tripleo/+bug/1882664 | 10:24 |
| openstack | Launchpad bug 1882664 in tripleo "error: unpacking of archive failed on file /usr/share/ansible/plugins/modules/pacemaker_cluster.py;5eded785: cpio: open failed - Inappropriate ioctl for device" [High,Triaged] | 10:24 |
| chandankumar | pojadhav|ruck, thanks! | 10:24 |
| *** dmellado has joined #oooq | 10:28 | |
| *** ykarel|afk is now known as ykarel | 10:35 | |
| amoralej | pojadhav|ruck, chandankumar did you see it more that once? | 10:36 |
| amoralej | i could not reproduce it with same versions | 10:36 |
| pojadhav|ruck | amoralej, https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-master&job_name=periodic-tripleo-centos-8-buildimage-overcloud-full-master | 10:37 |
| pojadhav|ruck | last two failure was the same for pacemaker | 10:37 |
| pojadhav|ruck | oops, only latest last failure coming with issue pacemaker | 10:39 |
| amoralej | pojadhav|ruck, last one is related to less memory that needed iiurc | 10:39 |
| pojadhav|ruck | amoralej, okay | 10:40 |
| amoralej | mmm | 10:40 |
| amoralej | wait | 10:40 |
| amoralej | no, pojadhav|ruck you are right | 10:40 |
| pojadhav|ruck | in build.log its failing for pacemaker | 10:41 |
| pojadhav|ruck | https://logserver.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-8-buildimage-overcloud-full-master/419668a/build.log | 10:41 |
| amoralej | yep | 10:41 |
| amoralej | previous one image was built | 10:41 |
| *** ykarel is now known as ykarel|mtg | 10:43 | |
| chandankumar | amoralej, pojadhav|ruck https://bugs.launchpad.net/tripleo/+bug/1882664 in last 5 days, 4 times seen this issue | 10:51 |
| openstack | Launchpad bug 1882664 in tripleo "error: unpacking of archive failed on file /usr/share/ansible/plugins/modules/pacemaker_cluster.py;5eded785: cpio: open failed - Inappropriate ioctl for device" [High,Triaged] | 10:51 |
| chandankumar | in post_failure also we have seen this | 10:52 |
| amoralej | chandankumar, but it's random, right? | 10:52 |
| chandankumar | yes, | 10:53 |
| amoralej | chandankumar, but always complains of the same package and file, right? | 10:54 |
| chandankumar | amoralej, yes it is ansible-pacemaker | 10:56 |
| *** udesale_ has joined #oooq | 10:57 | |
| *** udesale has quit IRC | 10:59 | |
| *** ykarel|mtg is now known as ykarel | 11:06 | |
| *** jschlueter has joined #oooq | 11:07 | |
| marios | pojadhav|ruck: o/ have you seen that one yet/before? tripleo-ci-centos-7-containers-multinode-rocky fails with TASK [undercloud-deploy : Run podman --version -> fatal: [undercloud]: FAILED! => https://ae0e522512acc42c22ec-66f615728c2509811faa1d1728f27489.ssl.cf1.rackcdn.com/730763/15/check/tripleo-ci-centos-7-containers-multinode-rocky/207d06e/job-output.txt | 11:23 |
| pojadhav|ruck | marios, \o..nope | 11:24 |
| marios | pojadhav|ruck: well, seems new... going to recheck https://zuul.openstack.org/builds?job_name=tripleo-ci-centos-7-containers-multinode-rocky | 11:24 |
| pojadhav|ruck | marios, okay | 11:25 |
| marios | pojadhav|ruck: actually that error i pointed to above is not fatal... the job is a timeout | 11:25 |
| pojadhav|ruck | marios, ohh okay | 11:25 |
| marios | pojadhav|ruck: looks like tempest timeout fyi | 11:26 |
| pojadhav|ruck | marios, okay.. let me know if it is consistent timeouts.. | 11:26 |
| marios | pojadhav|ruck: ack | 11:27 |
| marios | pojadhav|ruck: the patch is cursed been trynig to merge it for at least a week now https://review.opendev.org/#/c/730763/15 | 11:27 |
| *** jpena is now known as jpena|lunch | 11:32 | |
| pojadhav|ruck | marios, you posted a recheck, lets see how it goes.. because that job having previous success.. If it fails again. will look into it. :) | 11:34 |
| marios | pojadhav|ruck: ack | 11:37 |
| *** amoralej is now known as amoralej|lunch | 11:54 | |
| soniya29 | arxcruz: I have updated the patch, please have a look https://review.opendev.org/#/c/720434/ | 11:56 |
| *** rfolco has joined #oooq | 11:57 | |
| soniya29 | weshay|ruck, meeting? | 12:00 |
| weshay|ruck | soniya29, aye | 12:01 |
| arxcruz | soniya29: ack | 12:05 |
| rfolco | chandankumar, o/ | 12:11 |
| *** rfolco is now known as rfolco|rover | 12:11 | |
| rfolco|rover | chandankumar, looking for ideas on how to fix the root cause of missing files in overcloud image... | 12:12 |
| chandankumar | rfolco|rover, \o/ | 12:16 |
| chandankumar | rfolco|rover, I have left with just one thing | 12:17 |
| chandankumar | rfolco|rover, may be confirming whether openssl-lib package has private dir on centos-8 image http://images.rdoproject.org/CentOS-8-x86_64-GenericCloud.qcow2 | 12:17 |
| chandankumar | I tried to reproduce locally in centos-8 image everything is fine there | 12:18 |
| rfolco|rover | chandankumar, what did you reproduce exactly ? | 12:18 |
| chandankumar | rfolco|rover, I pulled centos-8.1 container there | 12:19 |
| chandankumar | there I tried to look into openssl-lib package it has directory | 12:19 |
| rfolco|rover | chandankumar, container ? | 12:20 |
| weshay|ruck | chandankumar, rfolco|rover pojadhav|ruck sync for 10 min? | 12:21 |
| weshay|ruck | https://meet.google.com/pbj-bcin-uss?authuser=1 | 12:21 |
| rfolco|rover | weshay|ruck, yep | 12:21 |
| pojadhav|ruck | weshay|ruck, yup | 12:21 |
| *** rlandy has joined #oooq | 12:22 | |
| *** Goneri has joined #oooq | 12:25 | |
| *** Goneri has quit IRC | 12:30 | |
| *** derekh has quit IRC | 12:30 | |
| *** Goneri has joined #oooq | 12:33 | |
| *** ratailor has quit IRC | 12:38 | |
| chandankumar | rfolco|rover, against this patch https://review.opendev.org/#/c/734112/ image sanity not running in check | 12:40 |
| *** jpena|lunch is now known as jpena | 12:40 | |
| chandankumar | we need that also | 12:40 |
| rfolco|rover | chandankumar, ok just leave your -1, I'll switch image_sanity: false to true | 12:42 |
| rfolco|rover | chandankumar, ok done | 12:45 |
| rfolco|rover | https://review.opendev.org/734112 Enable and fix image_sanity check | 12:45 |
| chandankumar | will take a look soon thansk! | 12:46 |
| rlandy | ysandeep: hi - I'm going to ask infra to take back the nodes we had held | 12:54 |
| *** derekh has joined #oooq | 12:54 | |
| ysandeep | rlandy, go ahead.. I took yatin and sshnaidm but we were unable to figure out the issue. | 12:55 |
| ykarel | ysandeep, u tried the updates we discussed yesterday? | 12:56 |
| ysandeep | ykarel, yes.. i pinged you this morning about them in pm.. not sure if my messages reached you. | 12:57 |
| ykarel | ysandeep, nope it didn't reached | 12:57 |
| ykarel | but let me cross check :) | 12:57 |
| ykarel | ysandeep, hmm reached, reading | 12:57 |
| *** amoralej|lunch is now known as amoralej | 12:57 | |
| ysandeep | ykarel, :) | 12:57 |
| *** saneax_ is now known as saneax_AFK | 12:59 | |
| weshay|ruck | rlandy, chatting w/ sagi in https://meet.google.com/diq-fpug-jxe?authuser=1 | 13:00 |
| *** skramaja has quit IRC | 13:01 | |
| *** derekh has quit IRC | 13:05 | |
| rlandy | weshay|ruck: sshnaidm: https://bugzilla.redhat.com/show_bug.cgi?id=1845266 | 13:09 |
| openstack | bugzilla.redhat.com bug 1845266 in releng "Significant slowdown in running jobs in PSI upshift - internal zuul" [Unspecified,New] - Assigned to apevec | 13:09 |
| *** derekh has joined #oooq | 13:09 | |
| chandankumar | brb | 13:12 |
| weshay|ruck | sshnaidm, rlandy https://docs.google.com/document/d/1wA9JVR2cv2Hd00FHVpvJjiySN-QsWnTvpxqXdP11VAk/edit | 13:19 |
| *** derekh has quit IRC | 13:28 | |
| rfolco|rover | arxcruz, rfolco, zbr, panda, sshnaidm, rlandy, marios, ysandeep, bhagyashris, soniya29, pojadhav, akahat, weshay, chandankumar | 13:30 |
| rfolco|rover | community mtg | 13:30 |
| rfolco|rover | https://meet.google.com/bqx-xwht-wky?authuser=1 | 13:30 |
| *** TrevorV has joined #oooq | 13:31 | |
| rfolco|rover | I don't see any topics https://hackmd.io/IhMCTNMBSF6xtqiEd9Z0Kw#TripleO-CI-Notes | 13:31 |
| zbr | panda: chandankumar: no-brainer: https://review.rdoproject.org/r/#/c/27640/ | 13:42 |
| zbr | panda: brief look at https://review.rdoproject.org/r/#/c/28011/ ? if ok, i remove wip and we can merge. | 13:45 |
| zbr | there is a lot of cleanup to do. | 13:45 |
| chandankumar | rfolco|rover, marios https://trello.com/c/VX93jXg8/1391-cixlp1867602tripleociproa-overcloud-deploy-failed-due-overcloud-image-missing-some-folders-under-var | 13:48 |
| zbr | i wonder if anyone know what is the cause of [WARNING]: log file at /home/zuul/zuul-output/logs/ansible-execution.log is not writeable and we cannot create it, aborting | 13:50 |
| chandankumar | qemu-img | 13:53 |
| rfolco|rover | marios, who let you cut your hair ? | 13:59 |
| marios | rfolco|rover: you like? :D | 13:59 |
| rfolco|rover | no too scary yeah | 14:01 |
| chandankumar | rfolco|rover, may be compairing the rpm list with fs02 | 14:03 |
| chandankumar | may be it will give some clue? | 14:03 |
| weshay|ruck | pojadhav|ruck, rfolco|rover FYI.. promoting master w/ these results https://trunk.rdoproject.org/api-centos8-master-uc/api/civotes_agg_detail.html?ref_hash=e7bd3f89aa388fa5ad8bc19b00124c11 | 14:14 |
| rfolco|rover | weshay|ruck, looks normal except overcloud image | 14:16 |
| weshay|ruck | rfolco|rover, ya.. since fs002 worked.. we should be ok | 14:17 |
| rfolco|rover | weshay|ruck, cool | 14:17 |
| *** jtomasek has quit IRC | 14:24 | |
| *** jtomasek has joined #oooq | 14:28 | |
| pojadhav|ruck | weshay|ruck, ack | 14:37 |
| *** jtomasek has quit IRC | 14:46 | |
| ykarel | weshay|ruck, is the image published good one with the above hash? | 14:49 |
| ykarel | no images are absent, so no good | 14:50 |
| ykarel | https://images.rdoproject.org/centos8/master/rdo_trunk/e7bd3f89aa388fa5ad8bc19b00124c11/ | 14:50 |
| ykarel | i think if we promote master, upstream ovb will be down and like tox molecule too | 14:50 |
| ykarel | s/like/likely/ | 14:51 |
| weshay|ruck | ykarel, the overcloud image hash is linked to tripleo-ci-testing | 14:55 |
| weshay|ruck | ykarel, it's fine | 14:55 |
| ykarel | weshay|ruck, i see missing there too https://images.rdoproject.org/centos8/master/rdo_trunk/tripleo-ci-testing/ | 14:55 |
| weshay|ruck | lrwxrwxrwx. 1 uploader uploader 32 Jun 9 08:31 tripleo-ci-testing -> e7bd3f89aa388fa5ad8bc19b00124c11 | 14:56 |
| ykarel | https://trunk.rdoproject.org/centos8-master/tripleo-ci-testing/delorean.repo.md5 | 14:56 |
| ykarel | overcloud-full image not there in ^^ | 14:56 |
| weshay|ruck | k.. now it is | 14:57 |
| ykarel | hmm now visible, u copied working one there? | 14:58 |
| weshay|ruck | ya | 14:58 |
| ykarel | okk then good | 14:58 |
| weshay|ruck | :) | 14:58 |
| rlandy | sshnaidm: ted.upshift.rdu2.redhat.com/logs/36/200436/13/check/tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001-internal/fa90ade/job-output.txt shows the log where stack delete errors out | 15:14 |
| rlandy | https://sf.hosted.upshift.rdu2.redhat.com/logs/36/200436/13/check/tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001-internal/fa90ade/job-output.txt | 15:14 |
| rlandy | 2020-06-09 00:54:40.271293 | TASK [ovb-manage : Find out all servers] | 15:14 |
| rlandy | 2020-06-09 00:54:45.312998 | Timeout exception waiting for the logger. Please check connectivity to [10.0.109.19:19885] | 15:14 |
| rlandy | 2020-06-09 00:54:45.313907 | primary | ERROR | 15:14 |
| rlandy | http://git.app.eng.bos.redhat.com/git/openstack/tripleo-ci-internal-config.git/tree/playbooks/tripleo-rdo-base/post-ovb-delete.yaml#n22 | 15:15 |
| rlandy | should juts call ovb-manage | 15:15 |
| rlandy | just | 15:15 |
| sshnaidm | hmm, weird error | 15:24 |
| sshnaidm | rlandy, Failed to establish a new connection: [Errno -2] Name or service not known | 15:24 |
| rlandy | https://github.com/rdo-infra/review.rdoproject.org-config/blob/master/roles/ovb-manage/tasks/ovb-delete-stack.yml#L16 | 15:24 |
| sshnaidm | rlandy, seems like dns problem, internal server maybe requires an internal dns to use | 15:25 |
| rlandy | ^^ not right for downstream | 15:25 |
| sshnaidm | rlandy, exactly | 15:25 |
| rlandy | sshnaidm: yep - will put up a patch | 15:25 |
| marios | this all hands is stressing me out :/ i can't shake the feeling that the train is leaving and we're not on it :D | 15:27 |
| rfolco|rover | chandankumar, pojadhav|ruck pls notify if any progress on images | 15:28 |
| marios | can i get some votes please there https://review.opendev.org/#/c/730763/ if the current run is green i want to hit +A asap | 15:30 |
| marios | chandankumar: weshay|ruck: sshnaidm: rlandy: rfolco|rover: ^^ | 15:31 |
| marios | please think positive thoughts when voting so we can overcome the curse! | 15:31 |
| rfolco|rover | can somebody please explain me the to_build var here ? https://opendev.org/openstack/tripleo-ci/src/branch/master/roles/oooci-build-images/tasks/main.yaml#L36-L43 | 15:35 |
| rfolco|rover | marios, you know ? | 15:35 |
| rfolco|rover | 2020-06-09 03:09:25.908849 | TASK [oooci-build-images : Install python-tripleoclient for Image build] | 15:35 |
| rfolco|rover | 2020-06-09 03:09:26.033491 | primary | skipping: Conditional result was False | 15:35 |
| marios | rfolco|rover: i think chandankumar added that one ... its to be used for ovb jobs whether to build or download images? | 15:35 |
| rfolco|rover | this is my testproject, so let me check a real periodic one if it skips | 15:36 |
| chandankumar | rfolco|rover, yes it is related to build images flag | 15:36 |
| marios | rfolco|rover: e.g. like that https://review.rdoproject.org/r/#/c/26127/ | 15:36 |
| panda | rlandy: do we have a good candidate for internal rhel8/rhos-17 ? I'd like to try som internal promoter runs. | 15:37 |
| rlandy | panda: now is not a good time | 15:38 |
| rlandy | we can't get the container build-push job to complete so no good candidate | 15:38 |
| weshay|ruck | arxcruz, https://65e1a25c51492e951164-19af9f1afe439ab8b4c1c622e371839a.ssl.cf2.rackcdn.com/731357/2/gate/puppet-horizon-tripleo-standalone/0a052ee/logs/undercloud/var/log/tempest/stestr_results.html | 15:39 |
| weshay|ruck | arxcruz, horizon service :/ | 15:39 |
| arxcruz | weshay|ruck: checking | 15:39 |
| panda | rlandy: ok, just keep in mind, the internal promoter is ready for some spin. | 15:39 |
| rlandy | panda: the downstream cloud is not in a good place for experimentation atm | 15:40 |
| rlandy | will be in touch when it is | 15:40 |
| *** ykarel is now known as ykarel|away | 15:46 | |
| *** marios is now known as marios|out | 16:04 | |
| *** dtantsur is now known as dtantsur|afk | 16:05 | |
| rlandy | sshnaidm: pls review https://review.rdoproject.org/r/#/c/28025/ | 16:07 |
| *** jmasud has joined #oooq | 16:08 | |
| arxcruz | weshay|ruck: lol you won't imagine the problem... | 16:09 |
| weshay|ruck | arxcruz, this is going to be silly isn't | 16:11 |
| *** jtomasek has joined #oooq | 16:13 | |
| arxcruz | weshay|ruck: https://review.opendev.org/734638 | 16:13 |
| rlandy | AttributeError: 'AnsibleUnicode' object has no attribute 'get'? | 16:17 |
| rlandy | linters failure | 16:17 |
| rlandy | weshay|ruck: sshnaidm: want to meet again to conclude upshift discussion before tomorrow's prod chain meeting? | 16:31 |
| weshay|ruck | I'm avail | 16:31 |
| sshnaidm | rlandy, ack | 16:32 |
| sshnaidm | rlandy, where? | 16:33 |
| rlandy | https://meet.google.com/bma-gdtb-rmt | 16:33 |
| rlandy | sshnaidm: weshay|ruck: ^^ | 16:33 |
| *** amoralej is now known as amoralej|off | 16:35 | |
| weshay|ruck | sshnaidm, rlandy https://docs.google.com/document/d/1wA9JVR2cv2Hd00FHVpvJjiySN-QsWnTvpxqXdP11VAk/edit | 17:04 |
| *** jpena is now known as jpena|off | 17:06 | |
| sshnaidm | rlandy, weshay|ruck 2020-06-07 12:39:40.059349 | primary | Total 48 kB/s | 504 kB 00:10 | 17:08 |
| rlandy | sshnaidm: https://review.rdoproject.org/r/#/c/28025/ | 17:10 |
| sshnaidm | rlandy, weshay|ruck just search "| Total" in job log, it's fun | 17:10 |
| rlandy | oh dear me | 17:13 |
| *** udesale_ has quit IRC | 17:13 | |
| ysandeep | rlandy, hey o/ okay if we merge https://code.engineering.redhat.com/gerrit/#/c/201168/ ? | 17:14 |
| ysandeep | Trying to santize things for BM jobs, Also trying https://review.opendev.org/#/c/734654/ to make things cleaner for rhel8/centos8 | 17:18 |
| *** marios|out has quit IRC | 17:21 | |
| *** saneax_AFK has quit IRC | 17:26 | |
| sshnaidm | rlandy, I changed dns in private subnet to something that I got from shared3 network | 17:39 |
| sshnaidm | rlandy, there was some old dns, not sure where from.. | 17:39 |
| ysandeep | chandankumar, Looking for pointer on how can we get rid of net.ifnames=0 parameter from overcloud-full image in downstream. | 17:40 |
| ysandeep | chandankumar, sry, i didn't look at time.. you are probably sleeping by now :) will discuss tomorrow | 17:49 |
| *** ysandeep is now known as ysandeep|away | 17:49 | |
| *** jmasud has quit IRC | 17:57 | |
| rlandy | sshnaidm: where did you make the dns change? | 18:03 |
| sshnaidm | rlandy, private subnet on psi | 18:03 |
| sshnaidm | rlandy, dns that should be allocated to nodes by dhcp | 18:03 |
| sshnaidm | bbl | 18:07 |
| *** sshnaidm is now known as sshnaidm|afk | 18:07 | |
| *** chem has quit IRC | 18:21 | |
| *** chem has joined #oooq | 18:23 | |
| weshay|ruck | rlandy, https://docs.engineering.redhat.com/display/PRODCHAIN/2020-06-10+Meeting+Notes | 18:30 |
| weshay|ruck | under discussion items | 18:30 |
| weshay|ruck | rlandy, /me adds for oyu | 18:30 |
| weshay|ruck | you | 18:30 |
| rlandy | lol - not added yet | 18:31 |
| * rlandy was looking for the item | 18:31 | |
| rlandy | thanks | 18:31 |
| *** jmasud has joined #oooq | 18:34 | |
| rlandy | weshay|ruck: comparison data added to the doc | 18:44 |
| rlandy | No RDO vs Upshift - just Upshift before and after | 18:45 |
| weshay|ruck | rlandy, k.. I'll get something in the cockpit for it | 18:45 |
| weshay|ruck | rfolco|rover, stein's been trying to promote for days.. but the promoter is boosted | 18:48 |
| rfolco|rover | weshay|ruck, boosted how | 18:48 |
| weshay|ruck | rfolco|rover, overcloud image promotion http://38.102.83.109/centos7_stein.log | 18:49 |
| *** jmasud has quit IRC | 18:49 | |
| weshay|ruck | rfolco|rover, I think the error is accurate though | 18:50 |
| weshay|ruck | 43c1a3f01d1f595575fe8984dce3779a57937006_8d9f1861 is not there | 18:50 |
| *** jmasud has joined #oooq | 18:50 | |
| weshay|ruck | rfolco|rover, fyi.. hacking it w/ latest know good overcloud image | 18:51 |
| rfolco|rover | weshay|ruck, I may have a patch for missing files issue. It might be the cache=none (unsafe) default causing this on qemu-img convert. I am going to test w/ cache=writetrough... hacking dib qemu-img convert command | 18:52 |
| weshay|ruck | k | 18:52 |
| weshay|ruck | thanks | 18:52 |
| *** jtomasek has quit IRC | 18:52 | |
| *** jmasud has quit IRC | 19:04 | |
| rfolco|rover | weshay|ruck, we got a lucky run on https://review.opendev.org/#/c/734112/, image_sanity is enabled on check and it passes (and no missing files found) .... so I'd say https://review.opendev.org/#/c/734112/ is ready to merge since the check does what we want | 19:18 |
| weshay|ruck | looking | 19:19 |
| weshay|ruck | rfolco|rover, I'm not ready to fail gates on the overcloud-image build for missing files yet | 19:22 |
| weshay|ruck | we only need to fail in periodic | 19:22 |
| rfolco|rover | weshay|ruck, it won't fail. It will check and generate the rpm_va missing files logs | 19:23 |
| rfolco|rover | weshay|ruck, it will fail on periodic only | 19:23 |
| weshay|ruck | oh right.. ur other patch.. roger that | 19:23 |
| rfolco|rover | weshay|ruck, pls double check if I am missing anything | 19:23 |
| rfolco|rover | weshay|ruck, we can also trigger more couple times before merging if you like | 19:24 |
| rfolco|rover | to catch the failures and not fail the job | 19:24 |
| rfolco|rover | also, testing a dib hack https://review.opendev.org/#/c/734677/ on testproject https://review.rdoproject.org/r/#/c/27986/ | 19:25 |
| weshay|ruck | we have a test run in rdo showing it catching a failing image I think right? | 19:25 |
| rfolco|rover | weshay|ruck, we still did not have a failure in check for missing files... we had other cpio failures before | 19:26 |
| weshay|ruck | rfolco|rover, ya.. false negative here https://logserver.rdoproject.org/12/28012/1/check/periodic-tripleo-centos-8-buildimage-overcloud-full-master/d7c8773/rpm_va.txt | 19:27 |
| rfolco|rover | weshay|ruck, so we can retrigger my testproject a couple times to catch a "check" failure on missing files and confirm the job PASSES | 19:28 |
| *** jfrancoa has quit IRC | 19:37 | |
| weshay|ruck | rfolco|rover, great | 20:04 |
| *** jmasud has joined #oooq | 20:10 | |
| weshay|ruck | rlandy, https://snapshot.raintank.io/dashboard/snapshot/GwSyBms9Ed8y7zyzyZdeyKfZ7T40SZHo?orgId=2&fullscreen&panelId=16 | 20:11 |
| weshay|ruck | rlandy, query is SELECT max("duration") FROM "build" WHERE ("cloud" = 'tripleo-ci' and "job_name" =~ /.*standalone.*/) GROUP BY time(30m) | 20:12 |
| rlandy | very fancy | 20:12 |
| *** jmasud has quit IRC | 20:42 | |
| *** jbadiapa has quit IRC | 20:48 | |
| *** jmasud has joined #oooq | 20:57 | |
| *** jmasud has quit IRC | 21:22 | |
| *** jmasud has joined #oooq | 21:24 | |
| *** TrevorV has quit IRC | 21:26 | |
| *** jmasud has quit IRC | 21:26 | |
| *** jmasud has joined #oooq | 21:27 | |
| *** jmasud has quit IRC | 21:41 | |
| *** jmasud has joined #oooq | 21:50 | |
| *** jmasud has quit IRC | 21:52 | |
| *** jmasud has joined #oooq | 22:01 | |
| *** rfolco|rover has quit IRC | 22:05 | |
| *** jmasud has quit IRC | 22:10 | |
| *** jmasud has joined #oooq | 22:14 | |
| *** jmasud has quit IRC | 22:15 | |
| *** jmasud has joined #oooq | 22:38 | |
| *** jmasud has quit IRC | 22:41 | |
| *** tosky has quit IRC | 22:59 | |
| *** ccamacho has quit IRC | 23:42 | |
| *** ysandeep|away is now known as ysandeep | 23:42 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!