| hubbot | FAILING CHECK JOBS: gate-tripleo-ci-centos-7-container-to-container-upgrades-master-nv, tripleo-quickstart-extras-gate-newton-delorean-full-minimal | check logs @ https://review.openstack.org/472607 and fix them ASAP. | 00:12 |
|---|---|---|
| *** strattao has quit IRC | 00:17 | |
| *** strattao has joined #oooq | 00:24 | |
| *** jbadiapa has quit IRC | 00:51 | |
| *** jbadiapa has joined #oooq | 00:53 | |
| *** Goneri has joined #oooq | 01:01 | |
| *** jbadiapa has quit IRC | 01:01 | |
| *** jbadiapa has joined #oooq | 01:02 | |
| *** tcw has quit IRC | 01:10 | |
| *** tcw has joined #oooq | 01:13 | |
| *** jbadiapa has quit IRC | 01:36 | |
| *** jbadiapa has joined #oooq | 01:38 | |
| *** Goneri has quit IRC | 01:41 | |
| *** myoung_ has joined #oooq | 02:01 | |
| *** _atoth has joined #oooq | 02:01 | |
| *** rnoriega_ has joined #oooq | 02:01 | |
| *** weshay_ has joined #oooq | 02:01 | |
| *** lhinds|out has joined #oooq | 02:02 | |
| *** hubbot_ has joined #oooq | 02:02 | |
| *** rasca_ has joined #oooq | 02:02 | |
| *** rasca has quit IRC | 02:02 | |
| *** hubbot has quit IRC | 02:02 | |
| *** lhinds- has joined #oooq | 02:03 | |
| *** hubbot has joined #oooq | 02:03 | |
| *** weshay has quit IRC | 02:03 | |
| *** faceman- has joined #oooq | 02:03 | |
| *** rasca has joined #oooq | 02:03 | |
| *** weshay has joined #oooq | 02:03 | |
| *** rnoriega- has joined #oooq | 02:04 | |
| *** pliu has quit IRC | 02:04 | |
| *** pliu has joined #oooq | 02:04 | |
| *** faceman has quit IRC | 02:05 | |
| *** jschlueter has quit IRC | 02:05 | |
| *** jjoyce has quit IRC | 02:05 | |
| *** lhinds has quit IRC | 02:05 | |
| *** myoung has quit IRC | 02:05 | |
| *** jschlueter has joined #oooq | 02:05 | |
| *** rnoriega has quit IRC | 02:06 | |
| *** weshay_ has quit IRC | 02:06 | |
| *** rnoriega_ has quit IRC | 02:06 | |
| *** _atoth has quit IRC | 02:06 | |
| *** hubbot_ has quit IRC | 02:06 | |
| *** lhinds|out has quit IRC | 02:06 | |
| *** jjoyce has joined #oooq | 02:07 | |
| *** rasca_ has quit IRC | 02:07 | |
| *** myoung_ has quit IRC | 02:07 | |
| *** myoung has joined #oooq | 02:08 | |
| hubbot | FAILING CHECK JOBS: gate-tripleo-ci-centos-7-container-to-container-upgrades-master-nv, tripleo-quickstart-extras-gate-newton-delorean-full-minimal | check logs @ https://review.openstack.org/472607 and fix them ASAP. | 02:12 |
| *** trown|outtypewww has quit IRC | 02:18 | |
| *** jbadiapa has quit IRC | 02:46 | |
| *** jbadiapa has joined #oooq | 02:49 | |
| *** ykarel|away has joined #oooq | 03:09 | |
| *** d0ugal_ has joined #oooq | 03:15 | |
| *** d0ugal has quit IRC | 03:16 | |
| *** ykarel_ has joined #oooq | 03:39 | |
| *** jbadiapa has quit IRC | 03:41 | |
| *** ykarel|away has quit IRC | 03:42 | |
| *** jbadiapa has joined #oooq | 03:42 | |
| *** ykarel__ has joined #oooq | 03:48 | |
| *** ykarel_ has quit IRC | 03:51 | |
| *** d0ugal_ has quit IRC | 04:03 | |
| *** d0ugal__ has joined #oooq | 04:03 | |
| *** udesale has joined #oooq | 04:11 | |
| hubbot | FAILING CHECK JOBS: gate-tripleo-ci-centos-7-container-to-container-upgrades-master-nv, tripleo-quickstart-extras-gate-newton-delorean-full-minimal | check logs @ https://review.openstack.org/472607 and fix them ASAP. | 04:12 |
| *** ykarel__ is now known as ykarel | 04:34 | |
| *** ratailor has joined #oooq | 04:48 | |
| *** pgadiya has joined #oooq | 04:58 | |
| *** pgadiya has quit IRC | 04:58 | |
| *** marios has joined #oooq | 05:23 | |
| *** links has joined #oooq | 05:23 | |
| *** ruck-rover-bot has joined #oooq | 05:39 | |
| ruck-rover-bot | !gatestatus | 05:39 |
| openstack | ruck-rover-bot: Error: "gatestatus" is not a valid command. | 05:39 |
| hubbot | FAILING CHECK JOBS: gate-tripleo-ci-centos-7-container-to-container-upgrades-master-nv, tripleo-quickstart-extras-gate-newton-delorean-full-minimal | check logs @ https://review.openstack.org/472607 and fix them ASAP. | 05:39 |
| *** ruck-rover-bot has quit IRC | 05:39 | |
| *** quiquell|off is now known as quiquell|ruck | 05:39 | |
| *** ccamacho has quit IRC | 05:41 | |
| quiquell|ruck | Any cores around to review this https://review.openstack.org/#/c/561767/ | 06:04 |
| quiquell|ruck | Master will fail without this | 06:05 |
| *** pgadiya has joined #oooq | 06:05 | |
| *** pgadiya has quit IRC | 06:05 | |
| hubbot | FAILING CHECK JOBS: gate-tripleo-ci-centos-7-container-to-container-upgrades-master-nv, tripleo-quickstart-extras-gate-newton-delorean-full-minimal | check logs @ https://review.openstack.org/472607 and fix them ASAP. | 06:12 |
| *** anande has joined #oooq | 06:14 | |
| *** anande has quit IRC | 06:18 | |
| *** anande has joined #oooq | 06:18 | |
| *** jfrancoa has joined #oooq | 06:25 | |
| *** holser__ has joined #oooq | 06:33 | |
| ykarel | quiquell|ruck, required patch is merged | 06:47 |
| *** florianf has joined #oooq | 06:48 | |
| quiquell|ruck | ykarel: The timeout metadat or the auth_url problem ? | 06:48 |
| ykarel | for master, this is the required one: https://review.openstack.org/#/c/561768 | 06:48 |
| ykarel | auth url one | 06:48 |
| quiquell|ruck | I was looking at t he wrong one | 06:49 |
| quiquell|ruck | ykarel: Cool thanks | 06:49 |
| quiquell|ruck | In the other one jobs were failing | 06:49 |
| ykarel | other? | 06:50 |
| ykarel | queens? | 06:50 |
| quiquell|ruck | https://review.openstack.org/#/c/561767/ | 06:51 |
| ykarel | Okk, i haven't checked this, puppet-heat change should fix the master | 06:51 |
| ykarel | quiquell|ruck, also in queens two job failed: fs017 and fs020 | 06:52 |
| ykarel | for fs20 one we hit the tempest failure again: https://bugs.launchpad.net/tripleo/+bug/1744907 | 06:52 |
| openstack | Launchpad bug 1744907 in tripleo "Tempest test: "test_create_second_image_when_first_image_is_being_saved" failing in featureset020 periodic job" [High,Triaged] | 06:52 |
| ykarel | i commented on it | 06:52 |
| quiquell|ruck | ykarel: With 561768 merged we still need 561767 ? | 06:52 |
| quiquell|ruck | Ok going back in a few | 06:53 |
| *** quiquell|ruck is now known as quique|ruck|afk | 06:53 | |
| ykarel | quiquell|ruck, atleast for unblocking master should not be needed as undercloud install is passed in current master run | 06:54 |
| ykarel | 561767 not needed ^^ | 06:55 |
| *** pgadiya has joined #oooq | 06:56 | |
| *** pgadiya has quit IRC | 06:56 | |
| *** ccamacho has joined #oooq | 06:58 | |
| *** jbadiapa has quit IRC | 07:01 | |
| *** jbadiapa has joined #oooq | 07:01 | |
| *** ccamacho has quit IRC | 07:04 | |
| *** ccamacho has joined #oooq | 07:05 | |
| *** jbadiapa has quit IRC | 07:05 | |
| *** jbadiapa has joined #oooq | 07:06 | |
| *** tesseract has joined #oooq | 07:14 | |
| *** quique|ruck|afk is now known as quiquell|ruck | 07:17 | |
| quiquell|ruck | ykarel: Cool, that was fast | 07:18 |
| quiquell|ruck | ykarel: Let's seck out fs017 | 07:25 |
| quiquell|ruck | failing test_telemetry_integration.TestTelemetryIntegration | 07:27 |
| quiquell|ruck | This is tempest but not the metadata | 07:27 |
| ykarel | quiquell|ruck, yes it's different issue | 07:29 |
| quiquell|ruck | ok | 07:29 |
| ykarel | i remember you mentioned this issue yesterday | 07:30 |
| quiquell|ruck | chandankumar, arxcruz: TestTelemetryIntegration.test_autoscaling [459.908993s] ... FAILED at queens | 07:30 |
| quiquell|ruck | https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset017-queens/e6cb365/undercloud/home/jenkins/tempest_output.log.txt.gz | 07:30 |
| quiquell|ruck | Show it also yesterday | 07:30 |
| ykarel | it looks some issue is gnocchi ^^ | 07:31 |
| *** skramaja has joined #oooq | 07:33 | |
| quiquell|ruck | ykarel: Also fs020 is different from metadata timeout I think | 07:34 |
| quiquell|ruck | https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-queens/165b64e/undercloud/home/jenkins/tempest_output.log.txt.gz | 07:34 |
| ykarel | quiquell|ruck, yes that's different | 07:34 |
| quiquell|ruck | MismatchError: <bound method type.create_image_from_server of <class 'tempest.api.compute.images.test_images_oneserver_negative.ImagesOneServerNegativeTestJSON'>> returned {} | 07:35 |
| ykarel | metadata is one fixed | 07:35 |
| quiquell|ruck | Ok, going to investigate | 07:35 |
| ykarel | quiquell|ruck, i commented on the bug about this failure | 07:35 |
| ykarel | https://bugs.launchpad.net/tripleo/+bug/1744907 | 07:35 |
| openstack | Launchpad bug 1744907 in tripleo "Tempest test: "test_create_second_image_when_first_image_is_being_saved" failing in featureset020 periodic job" [High,Triaged] | 07:35 |
| quiquell|ruck | ykarel: Ok, going to check the other, thanks !! | 07:37 |
| jaosorior | quiquell|ruck: is that an error in tempest in the undercloud? | 07:40 |
| quiquell|ruck | jaosorior: I think so | 07:40 |
| jaosorior | against the undercloud? | 07:40 |
| jaosorior | why are we testing autoscaling for the undercloud? O_o | 07:40 |
| quiquell|ruck | It's run in the undercloud, but I don't think it's testing undercloud | 07:41 |
| *** bogdando has joined #oooq | 07:41 | |
| * quiquell|ruck checking the test | 07:41 | |
| ykarel | correct, it's running in undercloud but testing overcloud | 07:42 |
| jaosorior | quiquell|ruck: I stumbled upon that same issue when trying to enable TLS for the overcloud https://bugs.launchpad.net/ceilometer/+bug/1764451 | 07:42 |
| openstack | Launchpad bug 1764451 in Ceilometer "Ceilometer agent polling isn't working in TripleO with TLS enabled" [Critical,New] | 07:42 |
| jaosorior | not entirely sure if it's TLS-related... but it seemed to appear that way | 07:43 |
| *** ykarel is now known as ykarel|lunch | 07:43 | |
| quiquell|ruck | Now that you mention it, I show some reviews abouit CAs | 07:43 |
| quiquell|ruck | And I think this test was failing | 07:43 |
| jaosorior | lets see | 07:46 |
| *** kopecmartin has joined #oooq | 07:51 | |
| quiquell|ruck | https://bugs.launchpad.net/tripleo/+bug/1764660 | 07:52 |
| openstack | Launchpad bug 1764660 in tripleo "Failing tempest test_telemetry_integration.TestTelemetryIntegration with UPDATE_FAILED" [Undecided,New] | 07:52 |
| jaosorior | quiquell|ruck: the issue looks almost the same, except that it's reproduced without TLS | 07:53 |
| *** pgadiya has joined #oooq | 07:56 | |
| *** pgadiya has quit IRC | 07:57 | |
| *** lucas-brb is now known as lucasagomes | 08:04 | |
| *** d0ugal__ has quit IRC | 08:05 | |
| *** d0ugal has joined #oooq | 08:05 | |
| *** d0ugal has quit IRC | 08:05 | |
| *** d0ugal has joined #oooq | 08:05 | |
| quiquell|ruck | jaosorior: Looking for the duplicate log, I cannot find the ceilometer logs | 08:05 |
| jaosorior | quiquell|ruck: I don't find the ceilo logs, but the tempest error logs look similar | 08:06 |
| jaosorior | fails in the same teardown, in the same spot | 08:07 |
| *** amoralej|off is now known as amoralej | 08:08 | |
| *** jtomasek has quit IRC | 08:08 | |
| *** agopi has quit IRC | 08:12 | |
| hubbot | FAILING CHECK JOBS: gate-tripleo-ci-centos-7-container-to-container-upgrades-master-nv, tripleo-quickstart-extras-gate-newton-delorean-full-minimal | check logs @ https://review.openstack.org/472607 and fix them ASAP. | 08:12 |
| *** tosky has joined #oooq | 08:14 | |
| *** rnoriega- is now known as rnoriega | 08:23 | |
| *** zoli is now known as zoli|wfh | 08:23 | |
| *** ykarel|lunch is now known as ykarel | 08:24 | |
| *** zoli|wfh is now known as zoli | 08:24 | |
| quiquell|ruck | jaosorior: Confirmed that it's the same | 08:29 |
| jaosorior | funky | 08:30 |
| *** udesale_ has joined #oooq | 08:32 | |
| *** udesale has quit IRC | 08:34 | |
| *** ruck-rover-bot has joined #oooq | 08:55 | |
| ruck-rover-bot | !gatestatus | 08:55 |
| openstack | ruck-rover-bot: Error: "gatestatus" is not a valid command. | 08:55 |
| hubbot | FAILING CHECK JOBS: gate-tripleo-ci-centos-7-container-to-container-upgrades-master-nv, tripleo-quickstart-extras-gate-newton-delorean-full-minimal | check logs @ https://review.openstack.org/472607 and fix them ASAP. | 08:55 |
| *** ruck-rover-bot has quit IRC | 08:55 | |
| *** jtomasek has joined #oooq | 08:57 | |
| *** udesale__ has joined #oooq | 09:32 | |
| *** adarazs is now known as adarazs_lunch | 09:32 | |
| *** ratailor_ has joined #oooq | 09:33 | |
| *** udesale_ has quit IRC | 09:35 | |
| *** ratailor has quit IRC | 09:35 | |
| *** ratailor__ has joined #oooq | 10:00 | |
| *** ratailor_ has quit IRC | 10:02 | |
| *** ratailor__ has quit IRC | 10:04 | |
| *** ratailor has joined #oooq | 10:08 | |
| hubbot | FAILING CHECK JOBS: gate-tripleo-ci-centos-7-container-to-container-upgrades-master-nv, tripleo-quickstart-extras-gate-newton-delorean-full-minimal | check logs @ https://review.openstack.org/472607 and fix them ASAP. | 10:12 |
| *** tcw has quit IRC | 10:44 | |
| *** tcw has joined #oooq | 10:45 | |
| *** zoli is now known as zoli|wfh | 10:49 | |
| *** zoli|wfh is now known as zoli | 10:49 | |
| *** moguimar has quit IRC | 10:51 | |
| *** jtomasek has quit IRC | 10:59 | |
| *** lucasagomes is now known as lucas-hungry | 11:14 | |
| *** ruck-rover-bot has joined #oooq | 11:14 | |
| ruck-rover-bot | !gatestatus | 11:14 |
| openstack | ruck-rover-bot: Error: "gatestatus" is not a valid command. | 11:14 |
| hubbot | FAILING CHECK JOBS: gate-tripleo-ci-centos-7-container-to-container-upgrades-master-nv, tripleo-quickstart-extras-gate-newton-delorean-full-minimal | check logs @ https://review.openstack.org/472607 and fix them ASAP. | 11:14 |
| *** ruck-rover-bot has quit IRC | 11:14 | |
| *** tosky has quit IRC | 11:15 | |
| *** tosky has joined #oooq | 11:15 | |
| *** jaosorior has quit IRC | 11:24 | |
| *** jaosorior has joined #oooq | 11:27 | |
| *** atoth has joined #oooq | 11:27 | |
| ykarel | quiquell|ruck, looks like some issue is going in promotion of queens, last 3 attempts failed: http://38.145.34.55/queens.log | 11:29 |
| quiquell|ruck | This are the logs of the promotion script ? | 11:32 |
| *** udesale__ has quit IRC | 11:34 | |
| panda|rover|off | quiquell|ruck: yes | 11:35 |
| *** panda|rover|off is now known as panda | 11:35 | |
| *** panda is now known as panda|rover | 11:35 | |
| panda|rover | quiquell|ruck: the promotion process is stuck | 11:36 |
| quiquell|ruck | It sayis that there are other instances of the promoter ? | 11:37 |
| panda|rover | quiquell|ruck: we have a virtual lock in place to not run two promotion at the same time for the same release | 11:37 |
| panda|rover | quiquell|ruck: that log line means that another promotion process is running and the promotion script aborted | 11:38 |
| quiquell|ruck | Looks like the lock has not ben cleaned up | 11:38 |
| quiquell|ruck | In the last promotion | 11:38 |
| quiquell|ruck | Or maybe it failed without clean this up | 11:38 |
| panda|rover | quiquell|ruck: it may mean: 1) there is a promotion in progress and it's taking some time to upload the images | 11:38 |
| panda|rover | quiquell|ruck: 2) the process is stuck | 11:38 |
| *** jtomasek has joined #oooq | 11:39 | |
| quiquell|ruck | panda|rover: Going for lunch now | 11:39 |
| panda|rover | quiquell|ruck: it's not a file lock, is a virtual socket lock, so when the process exits it's automatically cleared | 11:39 |
| quiquell|ruck | It's always cleand up | 11:39 |
| quiquell|ruck | Oook | 11:39 |
| panda|rover | quiquell|ruck: yeah, ok, if it doesn't change when you're back we'll discuss what to do | 11:39 |
| quiquell|ruck | panda|rover: Thanks man I will be quick | 11:40 |
| *** quiquell|ruck is now known as quique|ruck|food | 11:40 | |
| panda|rover | take your time | 11:40 |
| *** adarazs_lunch is now known as adarazs | 11:44 | |
| *** moguimar has joined #oooq | 11:50 | |
| *** amoralej is now known as amoralej|lunch | 11:52 | |
| *** trown has joined #oooq | 12:00 | |
| *** panda|rover is now known as panda|rover|lnc | 12:02 | |
| sshnaidm | trown, fs37 fails for me in libvirt too in overcloud deploy :/ seems like network related: http://paste.openstack.org/show/719378/ | 12:04 |
| *** skramaja_ has joined #oooq | 12:04 | |
| *** skramaja has quit IRC | 12:05 | |
| trown | sshnaidm: using kickstart method? | 12:06 |
| sshnaidm | trown, yeah | 12:06 |
| trown | sshnaidm: k... my attempt to try it got messed up by power outage here yesterday | 12:08 |
| hubbot | FAILING CHECK JOBS: gate-tripleo-ci-centos-7-container-to-container-upgrades-master-nv, tripleo-quickstart-extras-gate-newton-delorean-full-minimal | check logs @ https://review.openstack.org/472607 and fix them ASAP. | 12:12 |
| *** quique|ruck|food is now known as quiquell|ruck | 12:13 | |
| *** lucas-hungry is now known as lucasagomes | 12:19 | |
| *** ratailor has quit IRC | 12:19 | |
| *** rfolco|off is now known as rfolco | 12:20 | |
| *** amoralej|lunch is now known as amoralej | 12:26 | |
| amoralej | could we merge https://review.openstack.org/#/c/561482/ ? | 12:26 |
| weshay | you get a promotion, you get a promotion.. WE ALL GET PROMOTIONS... in the prod chain that is :) | 12:29 |
| weshay | that is all | 12:29 |
| trown | you watch alot of oprah? | 12:29 |
| *** panda|rover|lnc is now known as panda|rover | 12:30 | |
| panda|rover | quiquell|ruck: still stuck ? | 12:30 |
| weshay | I just watch her meme's it's a good enough summary | 12:30 |
| quiquell|ruck | panda|rover: Just arrived from lunch | 12:31 |
| quiquell|ruck | panda|rover: promotion script still here ERROR promoter Another promoter process is running | 12:32 |
| panda|rover | quiquell|ruck: ok, when you have time I'll drive you through the promoter | 12:32 |
| *** ykarel is now known as ykarel|away | 12:34 | |
| *** skramaja_ is now known as skramaja | 12:36 | |
| *** rlandy has joined #oooq | 12:36 | |
| ykarel|away | weshay, readd fs20 to promotion criteria: https://review.rdoproject.org/r/#/c/13400/, https://review.rdoproject.org/r/#/c/13352/ | 12:38 |
| weshay | ykarel|away, thanks | 12:39 |
| weshay | quiquell|ruck, panda|rover you guys get the promotion server resolved? | 12:43 |
| panda|rover | weshay: in a minute | 12:44 |
| weshay | panda|rover, we have promotions for master, queens, and pike | 12:44 |
| *** ykarel|away has quit IRC | 12:45 | |
| weshay | panda|rover, we need to follow up today before you go re: the design for release config in upgrades | 12:47 |
| *** atoth has quit IRC | 12:47 | |
| panda|rover | weshay: ok | 12:47 |
| weshay | trown, rlandy https://review.rdoproject.org/r/#/c/13401/ | 12:48 |
| weshay | amoralej, ^ | 12:48 |
| amoralej | we are about to get two promotions | 12:52 |
| amoralej | wow | 12:52 |
| amoralej | in a shot | 12:52 |
| weshay | amoralej, three | 12:52 |
| weshay | master, queens, pike | 12:53 |
| weshay | ocata failed on one job | 12:53 |
| amoralej | i'm checking current openstack-periodic | 12:53 |
| amoralej | didn't check the 24h | 12:53 |
| trown | rlandy: did you ever have success with my etherpad yesterday? | 12:53 |
| amoralej | nice work! | 12:53 |
| weshay | amoralej, well it's ci + rdo packaging nice work + the time of the cycle | 12:54 |
| weshay | and yatin | 12:54 |
| amoralej | yeah | 12:54 |
| rlandy | trown: somewhat - I left notes at the bottom of the etherpad | 12:55 |
| rlandy | I am stuck on nodepool setup | 12:55 |
| weshay | sshnaidm, nice catch | 12:55 |
| rlandy | see the last comment | 12:55 |
| trown | rlandy: ah just saw that... probably DNS issue | 12:55 |
| trown | rlandy: is your virthost on redhat VPN? | 12:55 |
| rlandy | trown: yep | 12:56 |
| rlandy | trown: I can try that role again but the path looks wrong | 12:56 |
| rlandy | like I am missing the first piece of it | 12:56 |
| rlandy | busy debugging that now | 12:56 |
| trown | rlandy: ok, I think I hit the same thing... the DNS servers that the VPN config puts in resolv.conf dont seem to resolve centos repos... | 12:56 |
| amoralej | btw, https://review.openstack.org/#/c/561484 is also good to go, i tested it in https://review.openstack.org/#/c/561808 | 12:57 |
| rlandy | trown: ok - what did you do about it? and another dns server to resolv.conf? | 12:57 |
| trown | rlandy: I just manually added "nameserver 8.8.8.8" to the resolv.conf on my virthost | 12:57 |
| trown | a bit annoying ... and there is probably a better way | 12:57 |
| rlandy | trown: k- let me retry with that - will add to notes | 12:57 |
| rlandy | trown: really - this is the whole point of the 'try out' phase | 12:58 |
| rlandy | so we get these points out in the open, doc them | 12:58 |
| rlandy | so that we know what to answer devs when they ping us for support | 12:58 |
| trown | rlandy: ya I am happy to work on it with you today since it might be faster to uncover all my hacks :P | 12:58 |
| rlandy | trown: k, cool - I'll rekick with that change - and ping you with the next issue | 13:00 |
| trown | nice | 13:00 |
| trown | im trying out fs010 with queens to see if I get anything different wrt the deploy issue | 13:00 |
| *** atoth has joined #oooq | 13:03 | |
| panda|rover | weshay: amoralej we are cleaning up dangling docker images in the promoter server, then unblock the processes | 13:03 |
| weshay | thanks | 13:05 |
| weshay | !gatestatus | 13:06 |
| openstack | weshay: Error: "gatestatus" is not a valid command. | 13:06 |
| weshay | ! help | 13:06 |
| openstack | weshay: (help [<plugin>] [<command>]) -- This command gives a useful description of what <command> does. <plugin> is only necessary if the command is in more than one plugin. | 13:06 |
| hubbot | FAILING CHECK JOBS: gate-tripleo-ci-centos-7-container-to-container-upgrades-master-nv, tripleo-quickstart-extras-gate-newton-delorean-full-minimal | check logs @ https://review.openstack.org/472607 and fix them ASAP. | 13:06 |
| hubbot | weshay: (help [<plugin>] [<command>]) -- This command gives a useful description of what <command> does. <plugin> is only necessary if the command is in more than one plugin. You may also want to use the 'list' command to list all available plugins and commands. | 13:06 |
| panda|rover | uh-oh | 13:07 |
| panda|rover | the bots are fighting again | 13:07 |
| weshay | adarazs, fyi ^ | 13:07 |
| quiquell|ruck | panda|rover: Rise of the Robots :-) | 13:08 |
| adarazs | umm, I can change hubbot's prefix to be not "!" but why do we have the openstack bot here? | 13:08 |
| adarazs | weshay, panda|rover ^ | 13:08 |
| weshay | adarazs, heh.. I thought you did that :) | 13:08 |
| panda|rover | adarazs: so we can get this | 13:08 |
| adarazs | nope. also why is there a ruck and rover bot that calls this bot's command? :) | 13:09 |
| panda|rover | https://bugs.launchpad.net/tripleo/+bug/1762419 | 13:09 |
| openstack | Launchpad bug 1762419 in tripleo "Collect logs task are not collecting the virt-customize logs from periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset002-$release-upload " [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami) | 13:09 |
| panda|rover | adarazs: ^ | 13:09 |
| adarazs | panda|rover: ah, I think I can add that for hubbot, if we need it. | 13:09 |
| weshay | arxcruz, chandankumar fyi.. https://ci.centos.org/artifacts/rdo/jenkins-tripleo-quickstart-extras-gate-newton-delorean-full-minimal-5363/undercloud/home/stack/tempest_output.log.gz | 13:09 |
| panda|rover | adarazs: I don't know, I didn't add it. | 13:10 |
| weshay | arxcruz, chandankumar that is newton | 13:10 |
| chandankumar | weshay: https://review.openstack.org/#/c/561580/ | 13:10 |
| arxcruz | weshay: https://review.openstack.org/#/c/561580/ | 13:10 |
| arxcruz | lol | 13:10 |
| chandankumar | hehe | 13:10 |
| chandankumar | what a timing :-) | 13:10 |
| arxcruz | yup | 13:10 |
| weshay | nice | 13:10 |
| weshay | chandankumar++ | 13:10 |
| hubbot | weshay: chandankumar's karma is now 11 | 13:10 |
| weshay | quiquell|ruck, the ci.centos job failures are covered :) ^ | 13:11 |
| adarazs | so FWIW I can change hubbot's prefix, not sure if that will break the ruck-and-rover bot that is calling "!gatestatus" :P :) | 13:11 |
| weshay | adarazs, there is a bot called ruck-and-rover? | 13:11 |
| quiquell|ruck | weshay: Cool | 13:11 |
| quiquell|ruck | weshay: That my thing | 13:12 |
| adarazs | weshay: I see it from time to time, all it does it say "!gatestatus" from time to time,. | 13:12 |
| quiquell|ruck | It connects to IRC and get the gatestatus | 13:12 |
| weshay | k | 13:12 |
| weshay | quiquell|ruck, ok.. you should do a show and tell maybe for me and attila | 13:12 |
| rlandy | trown: we're moving now :) | 13:13 |
| weshay | sooner rather than later... | 13:13 |
| weshay | quiquell|ruck, and don't worry if it's very much WIP | 13:13 |
| weshay | quiquell|ruck, I didn't realize you were working on a bot | 13:13 |
| adarazs | quiquell|ruck: but what's the point? I mean there's some issue with hubbot's scheduling, but it can actually schedule the gatestatus command itself. | 13:13 |
| quiquell|ruck | weshay: It still in a kind of embarrassing stage | 13:13 |
| *** ykarel|away has joined #oooq | 13:13 | |
| weshay | quiquell|ruck, doesn't really matter | 13:13 |
| weshay | quiquell|ruck, we're looking at the high level design and your thoughts.. | 13:14 |
| weshay | not at the code | 13:14 |
| weshay | quiquell|ruck, please schedule 1/2 this week w/ myself and adarazs | 13:14 |
| weshay | adarazs, I should probably check in w/ you re: hubbot | 13:15 |
| *** Goneri has joined #oooq | 13:15 | |
| weshay | adarazs, I'll go look at the card first I suppose | 13:15 |
| adarazs | weshay: I'm updating the card from time to time. | 13:15 |
| rlandy | trown: export NODEPOOL_CENTOS_MIRROR=$NODEPOOL_CENTOS_MIRROR? also did you define a docker proxy on virthost? | 13:15 |
| quiquell|ruck | weshay, adarazs: I will do | 13:15 |
| adarazs | weshay: I have the multi-change part done, now I'm doing the filtering and then it's good. also added unittests for the whole thing meanwhile \o/ :) | 13:15 |
| trown | rlandy: no on docker proxy, and ya that export just resolves to "" for me... but seems to work | 13:16 |
| rlandy | weirdness but ok | 13:17 |
| rlandy | trown: 'Wait for provisioned hosts to become reachable' also failed for me - but carrying on to see if we can complete here | 13:19 |
| rlandy | trown: and we're running toci_gate_test | 13:23 |
| trown | rlandy: nice | 13:23 |
| rlandy | this is kind of slick - I like it | 13:23 |
| trown | rlandy: will be interesting to see if you hit the same deploy issue I hit on fs10 | 13:23 |
| rlandy | time will tell | 13:23 |
| rlandy | trown: where did your install bail out? | 13:24 |
| rlandy | oh - I see | 13:24 |
| trown | rlandy: during deploy | 13:24 |
| rlandy | the notes - sorry | 13:24 |
| trown | ya overcloud deploy | 13:25 |
| *** links has quit IRC | 13:38 | |
| rlandy | trown: well, my deploy is already further along that the ks one - 'Install the undercloud' task | 13:42 |
| rlandy | which could have been my fault but anyways | 13:43 |
| panda|rover | quiquell|ruck: ok, cleanup finished, cron reactivated | 13:44 |
| weshay | rlandy, nice | 13:44 |
| weshay | thanks panda|rover | 13:45 |
| quiquell|ruck | panda|rover: Cool, we are back on promotion track then | 13:45 |
| panda|rover | quiquell|ruck: I'd wait for the next execution to be sure :) | 13:46 |
| rlandy | weshay: yep trown's done some good work here | 13:46 |
| panda|rover | where is the "provision" card in the board ? can't find it anymore | 13:50 |
| trown | panda|rover: that is the experiment card | 13:50 |
| *** udesale has joined #oooq | 13:51 | |
| *** adarazs is now known as adarazs_afk | 13:51 | |
| panda|rover | trown: there was a card with the word "provision" written on it, that I told it was too big the last meeting. It wasn't the experiment card .... | 13:54 |
| trown | panda|rover: yes in said meeting we deleted the provision card, because until we know how we will provision via the experiment card... it is difficult to break down what needs to be done | 13:56 |
| panda|rover | trown: ok, can we try together in 1 hour ? | 13:56 |
| trown | panda|rover: i could try... things are not so different than they were yesterday though | 13:58 |
| panda|rover | trown: ok, let's see how far we can get in 15 minutes. | 13:58 |
| panda|rover | 15 minutes meeting 1 hour from now | 13:59 |
| panda|rover | ish | 13:59 |
| trown | sure | 14:00 |
| rlandy | preping the containers i staking a while | 14:01 |
| *** udesale has quit IRC | 14:02 | |
| quiquell|ruck | myoung, panda|rover, weshay: Can we move to friday's bug triage one hour up ? | 14:04 |
| *** panda|rover is now known as panda|rover|mtg | 14:05 | |
| panda|rover|mtg | quiquell|ruck: myoung is PTO for the next week | 14:06 |
| weshay | quiquell|ruck, /me is going to cancel that | 14:07 |
| weshay | should be every two weeks | 14:07 |
| weshay | we just did it | 14:07 |
| weshay | quiquell|ruck, panda|rover|mtg just ignore the invite as I wont be able to remove it | 14:08 |
| rlandy | trown: prep-containers is taking a while so I tried to ssh to the undercloud (from a separate window) to see the log. -> port 22: No route to host | 14:08 |
| rlandy | seen that? | 14:08 |
| quiquell|ruck | Ohhh | 14:08 |
| quiquell|ruck | ok | 14:08 |
| trown | rlandy: hmm nope... but i always start a screen before doing the deploy | 14:10 |
| quiquell|ruck | weshay, adarazs_afk: I am going to remove the IRC part from the script, and show you the stuff the next week | 14:10 |
| quiquell|ruck | It's ok ? | 14:10 |
| weshay | sounds good | 14:10 |
| rlandy | ugh - didn't do that - adding to notes | 14:10 |
| weshay | quiquell|ruck, if you want to use irc, just beta it in a private irc channel man | 14:11 |
| * weshay has nothing against it.. but we'll want to also work w/ what we have | 14:11 | |
| quiquell|ruck | weshay: Hubbot is not present in a private channel | 14:12 |
| hubbot | FAILING CHECK JOBS: gate-tripleo-ci-centos-7-container-to-container-upgrades-master-nv, tripleo-quickstart-extras-gate-newton-delorean-full-minimal | check logs @ https://review.openstack.org/472607 and fix them ASAP. | 14:13 |
| rlandy | undercloud is unreachable :( | 14:13 |
| quiquell|ruck | The script is just recollecting info, and !gatestatus is one of some | 14:13 |
| weshay | quiquell|ruck, k.. anyway.. it's important to telegraph your work with others a bit more | 14:13 |
| quiquell|ruck | Agree | 14:14 |
| weshay | so let's chat w/ adarazs_afk this week.. quiquell|ruck and then you really should send something out to openstack-dev[tripeo][ci] | 14:14 |
| *** brault_ has quit IRC | 14:18 | |
| weshay | quiquell|ruck, probably a good idea to become very familiar w/ v | 14:20 |
| weshay | https://review.rdoproject.org/r/#/c/13356/ | 14:20 |
| rlandy | weshay: still have a 1-on-1 in my calendar for today - and a new calendar meeting for friday. delete today's meeting? | 14:22 |
| quiquell|ruck | weshay: Ok, will take a look, also a json interface for hubbot maybe is not a bad idea | 14:24 |
| rlandy | trown: hmmm ... looks like the container prep is stalling. going to try rerun with screen session | 14:26 |
| weshay | ok.. folks who want to join.. community meeting https://bluejeans.com/7050859455 | 14:32 |
| rlandy | trown: the problem was with the images stored, my minidell ran out out space | 14:34 |
| rlandy | and the vm stalled | 14:34 |
| trown | oh | 14:35 |
| rlandy | when I deleted the images, the vm could resume | 14:35 |
| rlandy | will try rerun toci-gate-test | 14:35 |
| *** adarazs_afk is now known as adarazs | 14:38 | |
| *** skramaja has quit IRC | 14:38 | |
| *** quiquell|ruck is now known as quiquell|off | 14:46 | |
| sshnaidm | trown, rlandy myoung rfolco if you try kickstart, please use the latest patchset: https://review.openstack.org/#/c/543429/ | 14:49 |
| rlandy | sshnaidm: k - thanks | 14:50 |
| rlandy | trown: just for comparison. how large is your /dev/root partition | 14:50 |
| rlandy | pretty sure I did a standard install | 14:50 |
| * rlandy will have to reinstall minidell with f27 anyways | 14:51 | |
| sshnaidm | rlandy, default centos installation sets a little / partition | 14:59 |
| *** agopi has joined #oooq | 15:02 | |
| rlandy | fedora - but yeah - same default small /root partition | 15:10 |
| rlandy | resizing | 15:10 |
| *** ccamacho has quit IRC | 15:11 | |
| *** panda|rover|mtg is now known as panda|rover | 15:11 | |
| trown | rlandy: hmm maybe you just need different path for images... but ya i have 266G available on my root partition | 15:15 |
| trown | rlandy: what partition is large in your setup? just /home? | 15:17 |
| trown | I hate that default... | 15:17 |
| rlandy | trown: yep - my large partition is home | 15:26 |
| rlandy | so I try modify the images path | 15:26 |
| rlandy | I want to reinstall my minidell nayways | 15:27 |
| rlandy | but not really today :( | 15:27 |
| rlandy | would like to finish this investigation | 15:27 |
| trown | k... you can try with a /home path instead of /opt/vm_images | 15:27 |
| trown | I think you might run into permission stuff though | 15:27 |
| rlandy | yep - redoing with that option | 15:27 |
| trown | maybe you can make a world readable directory in /home | 15:27 |
| rlandy | should be o if I set /home/my_images to be a 777 | 15:28 |
| trown | ya that is a bit overkill :P, but should work | 15:28 |
| trown | rlandy: as long as you are hacking in there... probably want to set subnode-1 to some other flavor (compute works), and then set "control_memory: 16384" and "control_vcpu: 4" | 15:30 |
| trown | rlandy: I think that is likely the cause of my deploy failures, running with that now | 15:30 |
| *** ykarel|away has quit IRC | 15:35 | |
| *** marios has quit IRC | 15:35 | |
| rlandy | trown: k | 15:42 |
| *** moguimar has quit IRC | 15:42 | |
| dpeacock | Is it possible to snapshot a domain? For example when I am trying to snapshot I get unexpected error. | 15:44 |
| dpeacock | [stack@gandalf ~]$ virsh snapshot-create-as --domain undercloud --name "blank-undercloud" | 15:44 |
| dpeacock | error: internal error: Child process (/bin/qemu-img snapshot -c blank-undercloud) unexpected exit status 1 | 15:44 |
| *** moguimar has joined #oooq | 15:49 | |
| rlandy | trown: at what point are you making the ^^ modification? | 15:50 |
| rlandy | - name: control_0 | 15:51 |
| rlandy | flavor: control | 15:51 |
| rlandy | topology: >- | 15:51 |
| rlandy | --compute-scale 0 | 15:51 |
| rlandy | in the nodes file | 15:51 |
| trown | rlandy: I just did it in the playbook with vars | 15:55 |
| rlandy | trown: sorry for being stupid here but if I read the nodes fiel correctly subnode-1 is a controller node | 15:56 |
| rlandy | so I can add the "control_memory: 16384" and "control_vcpu: 4" params there | 15:56 |
| rlandy | but now you want to set subnode-1 to be compute? | 15:56 |
| trown | neither node is anything at that point | 15:57 |
| rlandy | trown: fine - so I can leave things as they are until I get to the toci script | 15:57 |
| trown | ya toci doesnt use any of that, it is just what we are passing to libvirt setup | 15:58 |
| rlandy | okie dokie | 16:02 |
| rlandy | I see the override | 16:02 |
| rlandy | sorry - changes in a few doff places | 16:02 |
| trown | and the flavors there are arbitrary too... can make them anything in the flavor list | 16:02 |
| rlandy | got it | 16:02 |
| trown | i didnt use undercloud, because I think there is some stuff we have turned on/off based on flavor==undercloud | 16:03 |
| rlandy | here we go again | 16:05 |
| *** ykarel|away has joined #oooq | 16:09 | |
| hubbot | FAILING CHECK JOBS: gate-tripleo-ci-centos-7-container-to-container-upgrades-master-nv, tripleo-quickstart-extras-gate-newton-delorean-full-minimal | check logs @ https://review.openstack.org/472607 and fix them ASAP. | 16:13 |
| *** jfrancoa has quit IRC | 16:14 | |
| *** links has joined #oooq | 16:15 | |
| panda|rover | weshay: you wanted to sync on the upgrades ? | 16:16 |
| weshay | panda|rover, yes sir.. now? | 16:16 |
| panda|rover | weshay: I'm available, will stay for another hour at least, so no rush | 16:17 |
| weshay | panda|rover, I'm avail | 16:17 |
| weshay | in channel now | 16:17 |
| *** bogdando has quit IRC | 16:23 | |
| *** strattao has quit IRC | 16:29 | |
| *** trown is now known as trown|lunch | 16:35 | |
| chandankumar | panda|rover: weshay http://rhos.op.redhat.com/roster/rhos-dfg-tripleo now it is updated | 16:36 |
| *** strattao has joined #oooq | 16:55 | |
| *** lucasagomes is now known as lucas-afk | 16:57 | |
| *** holser__ has quit IRC | 16:58 | |
| *** ykarel|away has quit IRC | 17:12 | |
| weshay | chandankumar++ | 17:16 |
| hubbot | weshay: chandankumar's karma is now 12 | 17:16 |
| *** zoli is now known as zoli|gone | 17:22 | |
| *** zoli|gone is now known as zoli | 17:22 | |
| *** panda|rover is now known as panda|rover|off | 17:24 | |
| *** links has quit IRC | 17:35 | |
| *** links has joined #oooq | 17:47 | |
| *** sshnaidm is now known as sshnaidm|off | 17:50 | |
| *** dsneddon has quit IRC | 17:51 | |
| *** dsneddon has joined #oooq | 17:52 | |
| *** trown|lunch is now known as trown | 17:53 | |
| *** dsneddon has quit IRC | 17:54 | |
| *** dsneddon has joined #oooq | 17:54 | |
| *** sshnaidm|off has quit IRC | 17:54 | |
| rlandy | trown: still going - needed to add a nameserver to the subnode-0 | 17:55 |
| trown | rlandy: why to subnode-0? shouldnt it get that via dhcp from the virthost? | 17:56 |
| rlandy | trown: could not reach https://pypi.python.org/simple/ | 17:57 |
| trown | rlandy: hmm... i was having issues with pypi yesterday, but didnt change anything and they went away | 17:58 |
| trown | i was getting endless redirects though | 17:58 |
| rlandy | I love deterministic fixes like that :) | 17:58 |
| rlandy | maybe I hit the same thing | 17:58 |
| rlandy | installing the undercloud now | 17:58 |
| rlandy | space on /root looks good now | 17:59 |
| *** kopecmartin has quit IRC | 18:04 | |
| rlandy | trown; for better or worse, I hope we're hitting the big issues now rather than later | 18:05 |
| *** tesseract has quit IRC | 18:06 | |
| hrybacki | rlandy: when the toci script runs on the undercloud node for an OVB deployment on RDO Cloud -- do you know where exactly it's pulling roles from? I keep trying to make changes locally in /opt/stack/new and /opt/stack/tripleo-quickstart-extras but I'm not seeing my changes in the logs. I'm trying to avoid submitting a billion patchsets and clogging upstream CI | 18:07 |
| rlandy | hrybacki: where is the change you are looking for? | 18:09 |
| rlandy | if you are using the reproducer, you will notice that there are two ZUUL_CHANGES defined | 18:10 |
| hrybacki | rlandy: (this time) I'm trying to modify baremetal-undercloud-full.yml playbook to run the tripleo-inventory role prior to the rest of the stuff | 18:10 |
| hrybacki | rlandy: okay my setup is complicated | 18:10 |
| hrybacki | I'm pulling in changes against both oooq and oooq-e (this works and is fine) | 18:11 |
| rlandy | so you're modifying the playbook you're using? | 18:11 |
| hrybacki | however, after doing the intial ovb-manage-stack and nodepool roles I want to modify the playbook /on the undercloud/ before I execute the toci script | 18:11 |
| rlandy | ok - that should be right | 18:11 |
| hrybacki | right. However altering the playbook on the undercloud, then running it, doesn't actually consume my change and I cannot figure out why | 18:12 |
| hrybacki | then running* the toci script | 18:12 |
| hrybacki | I don't think the toci script is wiping out /opt/stack or anything so I'm confused | 18:13 |
| rlandy | looking on new | 18:13 |
| rlandy | are there various new dirs there? | 18:13 |
| hubbot | FAILING CHECK JOBS: gate-tripleo-ci-centos-7-container-to-container-upgrades-master-nv, tripleo-quickstart-extras-gate-newton-delorean-full-minimal | check logs @ https://review.openstack.org/472607 and fix them ASAP. | 18:13 |
| rlandy | /opt/stack/new | 18:13 |
| hrybacki | rlandy: there is one | 18:13 |
| rlandy | and in that one? | 18:13 |
| hrybacki | I've tried modifying the playbook in both /opt/stack/new as well as /opt/stack/tripleo-quickstart-extras and neither worked | 18:13 |
| * rlandy checks | 18:14 | |
| hrybacki | is /opt/stack/new just a link to /opt/stack ? | 18:14 |
| *** agopi has quit IRC | 18:14 | |
| hrybacki | yes | 18:14 |
| *** agopi has joined #oooq | 18:15 | |
| rlandy | I would think changing /opt/stack/tripleo-quickstart-extras/playbooks would work | 18:17 |
| hrybacki | same however: https://paste.fedoraproject.org/paste/ukNXXarDwa2GlY8z9jnJBQ | 18:17 |
| amoralej | please, take a while to review https://review.openstack.org/#/c/561484/ so that i can get it merged for tomorrow | 18:22 |
| amoralej | thanks in advance | 18:22 |
| *** amoralej is now known as amoralej|off | 18:22 | |
| rlandy | hrybacki: I would think it is pulling playbooks from /opt/stack/tripleo-quickstart-extras | 18:23 |
| rlandy | and you start from the toci-gate-test run? | 18:23 |
| rlandy | with the change in place | 18:24 |
| hrybacki | rlandy: yes. toci_gate_test-oooq.sh to be precise | 18:24 |
| rlandy | hrybacki: check /tmp dir | 18:24 |
| hrybacki | /after/ making my changes | 18:24 |
| hrybacki | on the undercloud? | 18:24 |
| rlandy | LOCAL_WORKING_DIR="$WORKSPACE/.quickstart" | 18:26 |
| hrybacki | I'm not really sure what that means in context of this part of the deployment. Figuring which things are where during point X on a reproducer build is really hard. Esp. since the logs are all quieted | 18:27 |
| rlandy | $LOCAL_WORKING_DIR/playbooks/$playbook "${PLAYBOOKS_ARGS[$playbook]:-}" | 18:27 |
| rlandy | trown: ^^ can you confirm this? | 18:28 |
| rlandy | when you are on the undercloud, | 18:28 |
| rlandy | you edit /opt/stack/tripleo-quickstart-extras | 18:28 |
| rlandy | then you kick toci-gate-test | 18:28 |
| * hrybacki nods | 18:29 | |
| rlandy | and which in turn runs quickstart | 18:29 |
| rlandy | that the playbooks in workspace should have the change? | 18:30 |
| rlandy | and I'm ... "overcloud_deploy_result": "failed" | 18:30 |
| rlandy | hrybacki: in the mean time, check a tmp.xxx directory/.quickstart/playbooks file | 18:35 |
| hrybacki | rlandy: on the undercloud? | 18:35 |
| rlandy | yes | 18:36 |
| hrybacki | okay, I'm re-running the reproducer atm. Should ^^ exist prior the invocation of the toci script? | 18:36 |
| *** apetrich_ has joined #oooq | 18:38 | |
| *** agopi has quit IRC | 18:42 | |
| hrybacki | rlandy: okay. Making local changes to /opt/stack does not work | 18:47 |
| hrybacki | confirmed that /tmp/.quickstart/playbooks/baremetall-full-undercloud.yml is not what is in /opt/stack/tripleo-quickstart-extras/playbook/baremetal-full-undercloud.yml | 18:48 |
| hrybacki | which makes me wonder why we have /opt/stack/tripleo-quickstart* on the undercloud in the first place. Do we /expect/ it to use what is in /opt/stack/ or do we expect it to re-pull from upstream based on zuul_changes? | 18:49 |
| hrybacki | if the latter, how can I inject changes? | 18:49 |
| rlandy | it is my understanding that ZUUL_CHANGES are incorporated into the /opt/ dirs and those are run with toci-gate_test | 18:50 |
| hrybacki | rlandy: okay -- well that is unfortunately not what is happening. /opt/stack is a helluva red herring :( | 18:51 |
| rlandy | hrybacki: do you have ZUUL_CHANGES defined anywhere? | 18:53 |
| hrybacki | rlandy: it's in the env_vars_to_src.sh that is created and pushed from localhost to the undercloud | 18:53 |
| rlandy | do you source that at all? | 18:54 |
| hrybacki | I do | 18:54 |
| rlandy | do you see those changes? | 18:54 |
| rlandy | in your run? | 18:54 |
| hrybacki | yes, but they are also in /opt/stack to begin with | 18:54 |
| hrybacki | which is why I thought /opt/stack was the source of truth | 18:54 |
| rlandy | if they are in ZUUL_CHANGES and that is source and /opt/stack and they are in not in your run, the the reproducer would be broken for everyone | 18:56 |
| hrybacki | maybe there is a confusion here rlandy | 18:56 |
| hrybacki | I want to add a change /on top/ of what is being sucked in by ZUUL_CHANGES | 18:57 |
| hrybacki | to avoid submitting a new patchset and then bumping my ZUUL_CHANGES | 18:57 |
| hrybacki | to avoid killing CI | 18:57 |
| hrybacki | this is not what reproducer was meant to do but I don't know of a better way ot test out CI using the CI workflow on CI | 18:58 |
| trown | hrybacki: changes should get pulled from /opt/stack | 19:01 |
| hrybacki | rlandy: trown https://paste.fedoraproject.org/paste/C18QtGEix-K9Wrk-I9hguA sums up my issue. I can totally start throwing up patchsets | 19:01 |
| hrybacki | trown: I thought so too | 19:01 |
| rlandy | hrybacki: you can make changes in a tmp dir and reuse that temp dir - last shot ... can you look at the extras req file in the quickstart you have in opt | 19:02 |
| hrybacki | rlandy: looking | 19:02 |
| rlandy | is it pulling from source? | 19:02 |
| hrybacki | this makes no sense: git+file:///opt/stack/tripleo-quickstart-extras/#egg=tripleo-quickstart-extras | 19:03 |
| hrybacki | so it should be | 19:03 |
| rlandy | that should be right | 19:03 |
| hrybacki | the only thing I can think of is that somewhere the toci script is pulling changes from gerrit instead | 19:03 |
| trown | hrybacki: hmm... maybe has to do with this section of quickstart.sh https://github.com/openstack/tripleo-quickstart/blob/master/quickstart.sh#L132-L152 | 19:04 |
| hrybacki | rlandy: I'll show you so you can know I'm not insane. Lemme re-deploy | 19:04 |
| trown | hrybacki: you could try removing that section in /opt/stack/tripleo-quickstart/quickstart.sh and then see if your change is included | 19:04 |
| *** apetrich_ has quit IRC | 19:04 | |
| hrybacki | I can do that | 19:04 |
| trown | hrybacki: if so, that is a bug... and an annoying one because we just fixed a bug there | 19:05 |
| rlandy | oh - that change we tried to revert | 19:05 |
| trown | or so I thought anyways | 19:05 |
| rlandy | it is coming back to haunt us | 19:05 |
| rlandy | this should work | 19:05 |
| hrybacki | ack. I'll confirm or not that soon trown rlandy | 19:05 |
| rlandy | trown: side point ... hit an overcloud deploy error | 19:06 |
| trown | rlandy: what step? or at what point? | 19:06 |
| trown | rlandy: I have yet to get a successful deploy... trying with both nodes 16G ram and 6 vcpus on fs016... in deploy now | 19:07 |
| rlandy | trown: just started looking at it ... Write the puppet step_config manifest ... /var/lib/mistral/7a1f549e-8023-43b5-8d04-60a6f7dd12c3/common_deploy_steps_tasks.yam | 19:07 |
| rlandy | <class 'ansible.errors.AnsibleUndefinedVariable'>\nexception: 'role_data_step_config' is undefined" | 19:07 |
| rlandy | looks fixable | 19:07 |
| trown | ok well that is at least the same thing I hit the first time I tried fs010 too, it is what I put in the etherpad | 19:08 |
| trown | my theory is that error is just some bad error handling and the actual error is some timeout earlier... but I couldnt confirm that | 19:08 |
| trown | that error makes no sense in relation to deploy on libvirt vs cloud though | 19:09 |
| rlandy | trown: did you just rerun? | 19:09 |
| rlandy | you seem to be further now | 19:09 |
| trown | rlandy: I have been trying other featuresets, beefier vms | 19:09 |
| trown | rlandy: just trying to get success with something, then work backwards | 19:09 |
| trown | rlandy: but no, I have yet to get much farther than that | 19:10 |
| *** florianf has quit IRC | 19:10 | |
| rlandy | k - I'll keep going | 19:11 |
| trown | my current deploy with fs016 and the giant vms is looking promising | 19:12 |
| trown | I have seen multiple services come up in the journal on subnode-1 | 19:12 |
| rlandy | ugh - and it's April - why is it still feezing outside?? | 19:17 |
| rlandy | freezing | 19:17 |
| hrybacki | rlandy: this is how you know global warming isn't real | 19:18 |
| *** apetrich has quit IRC | 19:18 | |
| *** apetrich has joined #oooq | 19:19 | |
| rlandy | bring on the global warming!! | 19:19 |
| rlandy | hrybacki: any luck with that change? | 19:21 |
| hrybacki | rlandy: still waiting on initial stack deploy to finish | 19:21 |
| hrybacki | re-running the toci script has never worked well for me | 19:22 |
| *** agopi has joined #oooq | 19:27 | |
| hrybacki | rlandy: trown there is one other (not always re-ocurring) bug. ansible isn't getting installed always | 19:30 |
| hrybacki | https://www.irccloud.com/pastebin/iyLcSDoD/ | 19:30 |
| hrybacki | but that seems to come and go at its own will | 19:33 |
| trown | my favorite kind of bugs | 19:42 |
| rlandy | yeah - it's so boring when the same situation happens repeatedly ?! | 19:43 |
| hrybacki | you two have been working in CI too long :P | 19:46 |
| hrybacki | y'all are sick | 19:46 |
| trown | i was being sarcastic :P | 19:47 |
| hrybacki | apparently my deadpan is also effective over text \_0_/ | 19:48 |
| *** apetrich has quit IRC | 19:54 | |
| *** apetrich has joined #oooq | 19:55 | |
| hrybacki | trown: well that bootstrapping part is needed. /tmp/.quickstart isn't being populated w/o it | 19:56 |
| hrybacki | =/ | 19:57 |
| hrybacki | running to the gym but I'll be online again later. trown rlandy, here is output from the toci script: https://paste.fedoraproject.org/paste/KMIyEKt5JSrYYhd4j8Xe1w if it helps. | 19:59 |
| hrybacki | note I did remove the lines advised by trown from quickstart.sh prior to invocation. | 19:59 |
| hrybacki | bbiab | 19:59 |
| rlandy | k - thanks | 20:00 |
| rlandy | trown: agrree - that common deploy error looks more fs related than env install | 20:04 |
| rlandy | have you tried fs001? | 20:04 |
| trown | rlandy: nope... fs016 seems stuck just further in deploy... | 20:06 |
| trown | i am starting to get a bit pessimistic this will work | 20:06 |
| rlandy | stuck on what? | 20:06 |
| trown | stuck in the deploy somewhere... step 2 | 20:07 |
| *** holser__ has joined #oooq | 20:07 | |
| trown | and that is with 2x 16G ram vms... | 20:08 |
| rlandy | when we deploy ovb | 20:08 |
| rlandy | we don't use anything larger | 20:08 |
| rlandy | networking? | 20:08 |
| trown | fs01 is ovb btw | 20:09 |
| rlandy | hmmm - just looking for anything we know works | 20:09 |
| * rlandy starts to feel warmly towards rdocloud | 20:10 | |
| trown | im going to try fs07 on pike, since that is not containers... just for some other data point | 20:12 |
| hubbot | FAILING CHECK JOBS: gate-tripleo-ci-centos-7-container-to-container-upgrades-master-nv, tripleo-quickstart-extras-gate-newton-delorean-full-minimal | check logs @ https://review.openstack.org/472607 and fix them ASAP. | 20:13 |
| *** agopi has quit IRC | 20:17 | |
| *** agopi has joined #oooq | 20:25 | |
| rlandy | looking for fs to try next | 20:33 |
| rlandy | trown: looking at hrybacki's error - maybe he' still picking up https://github.com/openstack/tripleo-quickstart/commit/ffa34a686d70843cd8eddf59d673e62c6b29291a#diff-8846dd18c9ee9c09dadeee541156c2b8? | 20:39 |
| *** sshnaidm|off has joined #oooq | 20:41 | |
| *** links has quit IRC | 20:42 | |
| rlandy | hmmm ... 8d175c5c-4d3a-4a7a-9da4-b769f5ae3b86 | overcloud | 4392b3daac8742b7bbbd0131ab3dade6 | CREATE_COMPLETE | 2018-04-17T18:21:36Z | None | 20:44 |
| rlandy | but the server list is empty | 20:44 |
| rlandy | something wrong there | 20:44 |
| *** jtomasek has quit IRC | 20:49 | |
| trown | rlandy: multinode... no server list | 20:49 |
| trown | no nova | 20:49 |
| trown | what fs was that? | 20:50 |
| rlandy | watching the deploy again | 20:50 |
| rlandy | the overcloud registered as completed before | 20:50 |
| *** holser__ has quit IRC | 20:51 | |
| rlandy | trown: so stack completes - in config_download | 20:53 |
| trown | k.. my deploy with fs07 on pike just started | 20:53 |
| trown | oh ok, so it didnt actually finish | 20:53 |
| rlandy | ansible still running | 20:53 |
| rlandy | but the stack gets marked as create_complete | 20:54 |
| rlandy | still going - before it failed quickly | 20:54 |
| trown | k, will check on this fs07 deploy later tonight... though even if it works, I am not really sure what our next steps should be | 21:00 |
| trown | if we cant actually fit the container jobs on a minidell... the whole thing seems a bit low value | 21:01 |
| *** trown is now known as trown|outtypewww | 21:01 | |
| rlandy | Render deployment file for InstanceIdDeployment is hanging | 21:05 |
| *** rfolco is now known as rfolco|off | 21:10 | |
| *** tcw has quit IRC | 21:16 | |
| *** tcw has joined #oooq | 21:17 | |
| *** Goneri has quit IRC | 21:23 | |
| *** agopi has quit IRC | 21:25 | |
| *** agopi has joined #oooq | 21:26 | |
| *** agopi has quit IRC | 21:30 | |
| *** agopi has joined #oooq | 21:33 | |
| *** agopi has quit IRC | 21:44 | |
| hubbot | FAILING CHECK JOBS: gate-tripleo-ci-centos-7-container-to-container-upgrades-master-nv, tripleo-quickstart-extras-gate-newton-delorean-full-minimal | check logs @ https://review.openstack.org/472607 and fix them ASAP. | 22:13 |
| hrybacki | rlandy: I'll rebase my patches. I may not have that | 22:30 |
| rlandy | just a thought | 22:36 |
| hrybacki | rlandy: last shot -- tomorrow I just bombard CI :P | 22:38 |
| rlandy | please do | 22:38 |
| rlandy | we need to address this | 22:38 |
| hrybacki | if we haven't hit capacity limits. That seems to happen to me every Wednesday | 22:38 |
| hrybacki | makes sense queues would start to build up then | 22:38 |
| rlandy | I had a similar problem getting changes into dwonstream jobs | 22:38 |
| rlandy | downstream | 22:39 |
| hrybacki | =/ | 22:39 |
| *** agopi has joined #oooq | 22:56 | |
| *** atoth has quit IRC | 23:25 | |
| *** rlandy is now known as rlandy|bbl | 23:27 | |
| *** tosky has quit IRC | 23:41 | |
| *** Goneri has joined #oooq | 23:52 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!