| hubbot | FAILING CHECK JOBS: gate-tripleo-ci-centos-7-container-to-container-upgrades-master-nv, tripleo-quickstart-extras-gate-newton-delorean-full-minimal | check logs @ https://review.openstack.org/472607 and fix them ASAP. | 00:14 |
|---|---|---|
| hubbot | FAILING CHECK JOBS: gate-tripleo-ci-centos-7-container-to-container-upgrades-master-nv, tripleo-quickstart-extras-gate-newton-delorean-full-minimal | check logs @ https://review.openstack.org/472607 and fix them ASAP. | 02:14 |
| *** rfolco|off has quit IRC | 02:55 | |
| *** rfolco|off has joined #oooq | 02:55 | |
| *** udesale has joined #oooq | 03:08 | |
| *** ykarel has joined #oooq | 03:10 | |
| ykarel | Looks like promotion server is down, other servers on rdo cloud are also down, due to compute node shutdown few hours back. | 03:47 |
| ykarel | Can someone check | 03:47 |
| *** ykarel has quit IRC | 04:11 | |
| hubbot | FAILING CHECK JOBS: tripleo-quickstart-extras-gate-newton-delorean-full-minimal | check logs @ https://review.openstack.org/472607 and fix them ASAP. | 04:15 |
| *** jaganathan has joined #oooq | 04:18 | |
| *** ykarel has joined #oooq | 04:34 | |
| *** holser__ has joined #oooq | 04:44 | |
| ykarel | quiquell|off, FYI: https://bugs.launchpad.net/tripleo/+bug/1765008 | 04:48 |
| openstack | Launchpad bug 1765008 in neutron "Tempest API tests failing for stable/queens branch" [Critical,In progress] - Assigned to Hongbin Lu (hongbin.lu) | 04:48 |
| *** saneax has quit IRC | 04:54 | |
| *** ratailor has joined #oooq | 04:54 | |
| *** holser__ has quit IRC | 05:00 | |
| *** sanjayu is now known as saneax | 05:02 | |
| *** saneax is now known as Guest17057 | 05:03 | |
| *** Guest17057 is now known as saneax | 05:08 | |
| *** links has joined #oooq | 05:18 | |
| *** udesale_ has joined #oooq | 05:19 | |
| *** udesale has quit IRC | 05:19 | |
| *** pgadiya has joined #oooq | 05:21 | |
| *** pgadiya has quit IRC | 05:21 | |
| ykarel | quiquell|off, This as well: https://bugs.launchpad.net/tripleo/+bug/1765123 | 05:26 |
| openstack | Launchpad bug 1765123 in tripleo "dib-run-parts fails with /tmp/tmpkGJoNl/pre-install.d/05-rpm-epel-release: line 9: DISTRO_NAME: unbound variable" [Critical,Fix released] | 05:26 |
| *** marios has joined #oooq | 05:31 | |
| *** ccamacho has quit IRC | 05:38 | |
| *** quiquell|off is now known as quiquell|ruck | 05:43 | |
| quiquell|ruck | ykarel: Good morning | 05:43 |
| ykarel | quiquell|ruck, Good morning | 05:44 |
| quiquell|ruck | there is a RDO cloud problem | 05:45 |
| ykarel | quiquell|ruck, yes | 05:49 |
| ykarel | VMs were shutdown, so they need to be booted again | 05:50 |
| *** yolanda has joined #oooq | 05:52 | |
| quiquell|ruck | ykarel: They just need to restart the thing, they didn't lose the computes | 05:57 |
| quiquell|ruck | Humm promoter machine is down | 05:58 |
| quiquell|ruck | Ahh it's running in a RDO instance... | 06:00 |
| quiquell|ruck | Silly me | 06:00 |
| ykarel | yes they run on RDO Cloud | 06:00 |
| ykarel | so all those needs to be started again to get the promotion and jobs running | 06:01 |
| hubbot | FAILING CHECK JOBS: tripleo-quickstart-extras-gate-newton-delorean-full-minimal | check logs @ https://review.openstack.org/472607 and fix them ASAP. | 06:15 |
| quiquell|ruck | ykarel: periodic-tripleo-centos-7-master-containers is still failing with the fixes ? | 06:22 |
| *** ratailor has quit IRC | 06:23 | |
| ykarel | quiquell|ruck, containers-build? | 06:23 |
| quiquell|ruck | yep | 06:23 |
| ykarel | and which fixes? | 06:23 |
| *** ratailor has joined #oooq | 06:24 | |
| ykarel | instack-undercloud? | 06:24 |
| quiquell|ruck | we have the diskimage-buildr and the epel | 06:24 |
| ykarel | yes, instack-undercloud fix was not included in repo | 06:24 |
| *** kopecmartin has joined #oooq | 06:24 | |
| *** ratailor has quit IRC | 06:24 | |
| *** jfrancoa has joined #oooq | 06:24 | |
| quiquell|ruck | ykarel: with the instack-undercloud "brctl: command not found" | 06:25 |
| ykarel | where? | 06:25 |
| quiquell|ruck | gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens | 06:25 |
| *** ratailor has joined #oooq | 06:26 | |
| ykarel | quiquell|ruck, can you share the log link? | 06:27 |
| quiquell|ruck | https://logs.rdoproject.org/68/562468/1/openstack-check/gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens/Z791a984d19a74607912704442db7813c | 06:28 |
| ykarel | no that's brctl not found is not an issue | 06:28 |
| ykarel | Acutall issue is instance can't be spawned: 2018-04-19 06:08:13.050 | +(/opt/stack/new/tripleo-ci/toci_gate_test.sh:252): ./testenv-client -b 192.168.103.254:4730 -t 17400 --envsize 4 --ucinstance 2e2ff0cb-af1c-4dc8-8cd6-755539de08c4 --net-iso multi-nic -- ./toci_quickstart.sh | 06:29 |
| ykarel | 2018-04-19 06:08:13.083 | +(/opt/stack/new/tripleo-ci/toci_gate_test.sh:246): sleep 1200 | 06:29 |
| ykarel | quiquell|ruck, did you power on the VMs on rdocloud | 06:29 |
| ykarel | If those runs this issue should be fixed | 06:29 |
| quiquell|ruck | ykarel: You mean jenkins ones ? | 06:35 |
| ykarel | quiquell|ruck, like promote server, te-broker | 06:36 |
| quiquell|ruck | ykarel: Don't even know howto, let me find the procedure | 06:36 |
| ykarel | quiquell|ruck, Ok | 06:37 |
| ykarel | quiquell|ruck, do you have access to tripleo-infra and openstack-nodepool tenant | 06:40 |
| *** florianf has joined #oooq | 06:40 | |
| *** holser__ has joined #oooq | 06:41 | |
| quiquell|ruck | ykarel: Do you have the RCs around ? | 06:42 |
| ykarel | i don't have access | 06:42 |
| quiquell|ruck | ykarel: Ok got the document, let me find someone that have access to it already | 06:44 |
| ykarel | Ok | 06:45 |
| *** tcw has quit IRC | 06:53 | |
| quiquell|ruck | ykarel: Have to leave my kid in the kindergarden, let see if there are some news about VMs when I am back | 06:54 |
| *** quiquell|ruck is now known as quiquell|ruck|af | 06:54 | |
| ykarel | quiquell|ruck, Okk | 06:54 |
| *** quiquell|ruck|af is now known as quique|ruck|afk | 06:54 | |
| ykarel | EmilienM, do you have access | 06:56 |
| ykarel | ^^ | 06:56 |
| EmilienM | I don't | 06:56 |
| EmilienM | sorry | 06:56 |
| ykarel | Okk | 06:56 |
| * EmilienM goes to bed now | 06:56 | |
| ykarel | EmilienM, good night | 06:56 |
| *** bogdando has joined #oooq | 06:56 | |
| quique|ruck|afk | ykarel: gn | 06:58 |
| quique|ruck|afk | EmilienM: I mean | 06:58 |
| ykarel | :) | 06:58 |
| *** ccamacho has joined #oooq | 06:58 | |
| quique|ruck|afk | ykarel: rdo ticke open, maybe they can do it for us | 07:03 |
| ykarel | quique|ruck|afk, may be other things also need to be taken care, so one who is familiar with this stuff should do it | 07:05 |
| ykarel | we can wait for guys to come | 07:05 |
| quique|ruck|afk | ykarel: Let me go back and continue with the bugs | 07:06 |
| ykarel | Ok | 07:06 |
| *** tesseract has joined #oooq | 07:10 | |
| *** skramaja has joined #oooq | 07:17 | |
| *** skramaja has quit IRC | 07:23 | |
| *** skramaja has joined #oooq | 07:24 | |
| *** amoralej|off is now known as amoralej | 07:32 | |
| *** dmellado has joined #oooq | 07:32 | |
| *** quique|ruck|afk is now known as quiquell|ruck | 07:34 | |
| quiquell|ruck | I am back | 07:34 |
| *** zoli|gone is now known as zoli|wfh | 07:35 | |
| *** zoli|wfh is now known as zoliXXL | 07:36 | |
| quiquell|ruck | ykarel: tripleo-infra and openstack-nodepool start ing | 07:37 |
| ykarel | u mean vms? who started? | 07:38 |
| quiquell|ruck | dmanchado | 07:41 |
| quiquell|ruck | I ask for it | 07:41 |
| quiquell|ruck | Promoter is up now | 07:42 |
| quiquell|ruck | sova too | 07:43 |
| quiquell|ruck | How do we check the openstack-nodepool ?it's huge | 07:44 |
| ykarel | quiquell|ruck, also check te-broker | 07:49 |
| ykarel | me also don't know much about these servers | 07:49 |
| quiquell|ruck | arxcruz: Do you have access to te-broker too ? | 07:49 |
| quiquell|ruck | ykarel: We still need to check the instack-undercloud change ? | 07:51 |
| ykarel | quiquell|ruck, u mean https://review.openstack.org/#/c/562468/ | 07:52 |
| ykarel | if yes, once te-broker is up, we can try, check-rdo on ^^ | 07:53 |
| *** ykarel is now known as ykarel|lunch | 07:53 | |
| *** tosky has joined #oooq | 08:02 | |
| quiquell|ruck | ykarel|lunch: nodepool is up going to recheck master and queens of instack change | 08:08 |
| *** lucas-afk is now known as lucasagomes | 08:11 | |
| dmellado | quiquell|ruck: why are you trying to check-rdo on an already merged patch, dude? | 08:14 |
| dmellado | https://review.openstack.org/#/c/561479/ | 08:14 |
| dmellado | doesn't make any sense | 08:14 |
| quiquell|ruck | dmellado: Wrong chrome tab :-/ totally my fault | 08:14 |
| dmellado | quiquell|ruck: heh, no worries! | 08:15 |
| hubbot | FAILING CHECK JOBS: gate-tripleo-ci-centos-7-container-to-container-upgrades-master-nv, tripleo-quickstart-extras-gate-newton-delorean-full-minimal, gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master | check logs @ https://review.openstack.org/472607 and fix them ASAP. | 08:15 |
| quiquell|ruck | dmellado: Can we stop it ? | 08:15 |
| dmellado | quiquell|ruck: not that I'm aware of | 08:15 |
| *** gkadam has joined #oooq | 08:16 | |
| quiquell|ruck | It will waste precious resource then :-/ and maybe a forest tree fells down | 08:16 |
| dmellado | quiquell|ruck: we'll make you do a donation to greenpeace out of your pocket | 08:26 |
| quiquell|ruck | There is no way to force periodic jobs ? | 08:32 |
| arxcruz | dmellado: hola | 08:33 |
| dmellado | arxcruz: hey | 08:33 |
| arxcruz | i mean, hallo | 08:33 |
| dmellado | quiquell|ruck: yeah, for sure | 08:33 |
| dmellado | did you get to read all zuulv3 documentation? | 08:33 |
| arxcruz | wie geht es dir ? | 08:33 |
| dmellado | Ich bin Heute ein bißchen müde | 08:34 |
| arxcruz | ich auch | 08:34 |
| arxcruz | immer müde :D | 08:34 |
| quiquell|ruck | dmellado: Will recheck | 08:35 |
| arxcruz | dmellado: meine Torther ist Deutsch sprechen besser als mich | 08:35 |
| dmellado | quiquell|ruck: https://docs.openstack.org/infra/system-config/zuulv3.html | 08:35 |
| dmellado | arxcruz: no wonder xD | 08:35 |
| arxcruz | dmellado: yeah :) next semester i'll join german classes | 08:36 |
| arxcruz | it's kind of impossible to live here without know german | 08:36 |
| quiquell|ruck | dmellado: It's a jenkins job from RDO | 08:37 |
| *** ykarel|lunch is now known as ykarel | 08:38 | |
| ykarel | arxcruz, quiquell|ruck looks like dlrn-promoter is not running | 08:46 |
| quiquell|ruck | It's up | 08:47 |
| ykarel | i can't see master.log, queens.log etc at http://38.145.34.55/ | 08:47 |
| * quiquell|ruck checking now that he knows how it works | 08:48 | |
| quiquell|ruck | Wait they are commented out | 08:49 |
| quiquell|ruck | I am going to test panda's global lock change | 08:50 |
| ykarel | hmm, good to try as many promotions would be running now | 08:52 |
| quiquell|ruck | ykarel: Runnign for queens | 08:54 |
| quiquell|ruck | s/runnign/running/ | 08:54 |
| quiquell|ruck | btw if we use a global lock maybe if doesn't make sense to have more than one process running | 08:55 |
| quiquell|ruck | We can just make the process iterate on releases | 08:55 |
| panda|rover|off | mmhh | 09:04 |
| quiquell|ruck | panda|rover|off: Good Morning | 09:04 |
| panda|rover|off | good.. | 09:05 |
| quiquell|ruck | Yeah no so good, we have to wait a little to see the state of things | 09:05 |
| quiquell|ruck | Some good changes have been merged | 09:05 |
| quiquell|ruck | Take a look https://review.rdoproject.org/etherpad/p/ruckrover-sprint12 | 09:05 |
| panda|rover|off | ok do we have probl ms with the te-broker ? | 09:17 |
| ykarel | quiquell|ruck, did a little correction on etherpad | 09:18 |
| ykarel | containers-build for master passed | 09:20 |
| quiquell|ruck | ykarel: Thanks, I am still learning the terminology | 09:20 |
| quiquell|ruck | master and queens ? | 09:21 |
| ykarel | queens containers-build failed as instack-undercloud patch is not there | 09:21 |
| ykarel | quiquell|ruck, for terminology if not understood, can ask me | 09:21 |
| quiquell|ruck | We have to wait for queens periodic job yep | 09:21 |
| quiquell|ruck | I will | 09:22 |
| quiquell|ruck | ykarel: Pike is not affected by the instack-undercloud problem ? | 09:22 |
| ykarel | quiquell|ruck, no | 09:22 |
| ykarel | only queens and master | 09:22 |
| quiquell|ruck | ok | 09:22 |
| quiquell|ruck | Let's see if we see a master promotion soon | 09:22 |
| ykarel | hoping to see it | 09:23 |
| ykarel | panda|rover|off, re. te-broker we were affected earlier: https://review.rdoproject.org/jenkins/job/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset002-pike-upload/153/consoleText | 09:23 |
| ykarel | for now don't know if it's fixed or not | 09:23 |
| ykarel | can you check when available | 09:23 |
| panda|rover|off | ykarel: the te-broker timed out, usually means that it wasn't able to get the nodes in time, this is usually caused by high load on rdocloud, checking the logs on the server | 09:32 |
| *** panda|rover|off is now known as panda|rover | 09:32 | |
| ykarel | panda|rover, Ok | 09:32 |
| *** agopi has quit IRC | 09:40 | |
| ykarel | logs from 2 hours back: https://review.rdoproject.org/jenkins/job/gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/11348/consoleText | 09:46 |
| ykarel | for te-broker | 09:46 |
| ykarel | 30 minutes ago: https://review.rdoproject.org/jenkins/job/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/173/consoleText | 09:47 |
| ykarel | so some issue is there | 09:47 |
| quiquell|ruck | panda|rover: https://review.rdoproject.org/r/13437 | 09:55 |
| panda|rover | quiquell|ruck: yep, I was looking at it | 09:55 |
| quiquell|ruck | Changed the comment we don't need a daemon yet | 09:56 |
| quiquell|ruck | Hummm.. yest cron will run more than one of the promoters | 09:56 |
| quiquell|ruck | Let's daemoniza this thing and add it to systemd | 09:56 |
| quiquell|ruck | panda|rover: Don't know if there is something parallelizable that we miss if we iterate one by one | 09:57 |
| panda|rover | quiquell|ruck: let's discuss first | 09:58 |
| quiquell|ruck | panda|rover: sure, bj ? | 09:58 |
| panda|rover | quiquell|ruck: in a minute, I'm looking at the te-broker | 09:59 |
| quiquell|ruck | Sure, just let me know | 09:59 |
| panda|rover | quiquell|ruck: we have 400 instances in the nodepool-tenant, time to use the big cannon | 10:02 |
| quiquell|ruck | panda|rover: big cannon ? | 10:03 |
| panda|rover | quiquell|ruck: ronelle's cleanup script | 10:03 |
| panda|rover | that's whay it's timing out | 10:04 |
| panda|rover | we are out of resources | 10:04 |
| panda|rover | quiquell|ruck: ok, bj | 10:05 |
| ykarel | queens phase 1 promoted, promoter running for pike phase1 | 10:06 |
| quiquell|ruck | panda|rover: Make sense, veryjob have to be running now | 10:06 |
| ykarel | amoralej, ^^ | 10:07 |
| quiquell|ruck | Ups sorry ykarel | 10:07 |
| amoralej | cool | 10:07 |
| quiquell|ruck | panda|rover: https://bluejeans.com/7891065232 | 10:07 |
| *** adarazs is now known as adarazs_lunch | 10:12 | |
| hubbot | FAILING CHECK JOBS: gate-tripleo-ci-centos-7-container-to-container-upgrades-master-nv, tripleo-quickstart-extras-gate-newton-delorean-full-minimal, gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master | check logs @ https://review.openstack.org/472607 and fix them ASAP. | 10:15 |
| quiquell|ruck | ykarel: We have already clean up some of the openstack-nodepool instances | 10:45 |
| quiquell|ruck | Let's check it out that there are no timeouts now | 10:45 |
| ykarel | quiquell|ruck, in current master periodic run all jobs failed, so can check in next run. | 10:47 |
| ykarel | can check in rdo jobs on any master patch | 10:47 |
| *** zoliXXL is now known as zoli|lunch | 10:47 | |
| ykarel | quiquell|ruck, let's check here: https://review.rdoproject.org/jenkins/job/gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/11379/console | 10:48 |
| ykarel | running for around 5 minutes | 10:49 |
| quiquell|ruck | ykarel: Have do a check-rdo at the instack queens change | 10:50 |
| ykarel | Ok can check that as well to see if it fixes undercloud install | 10:51 |
| ykarel | but those jobs not started yet, will take some time to start | 10:52 |
| ykarel | queens also started: https://review.rdoproject.org/jenkins/job/gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens/397/console | 10:57 |
| amoralej | https://review.openstack.org/#/c/561482/ is merged, may i get reviews on https://review.openstack.org/#/c/561484 please | 11:01 |
| panda|rover | amoralej: will this work with the older releases ? | 11:04 |
| amoralej | it should | 11:04 |
| amoralej | panda|rover, it was tested in https://review.openstack.org/#/c/561673 | 11:05 |
| amoralej | we need the new repo only in master | 11:05 |
| amoralej | we create the new repo file only in master, in the repos where the delorean-<release>-build-deps.repo doesn't exist, it will do nothing | 11:06 |
| amoralej | https://review.openstack.org/#/c/561808/ is also testing this | 11:06 |
| panda|rover | amoralej: ok, yeah, empty results from the find | 11:06 |
| amoralej | yeah | 11:06 |
| ykarel | quiquell|ruck, panda|rover ./testenv-client still failed: https://review.rdoproject.org/jenkins/job/gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/11379/console | 11:08 |
| panda|rover | ykarel: maybe this started before the clean up started | 11:09 |
| panda|rover | quiquell|ruck: when did we start the cleanup ? | 11:09 |
| quiquell|ruck | 5537 2018-04-19 12:25:33 bash -x ovb-tenant-cleanup.sh | 11:11 |
| quiquell|ruck | from my time zone | 11:11 |
| quiquell|ruck | That's near 45 minutes ago | 11:11 |
| quiquell|ruck | It has finish already though | 11:11 |
| ykarel | panda|rover, how much time it generally takes, https://review.rdoproject.org/jenkins/job/gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens/397/console is running for about 13 minutes | 11:12 |
| panda|rover | ykarel: it's clearly not working | 11:19 |
| panda|rover | the te-broker did not reboot properly | 11:19 |
| panda|rover | I can't even access | 11:19 |
| panda|rover | I see only multinode nodes spawning in the tenant | 11:20 |
| amoralej | panda|rover, thanks for your review! | 11:23 |
| ykarel | panda|rover, so it's down or just not reachable | 11:24 |
| panda|rover | ykarel: I'm in, the services are not running | 11:25 |
| ykarel | Okk, please start those :) | 11:25 |
| panda|rover | it's very slow :( | 11:29 |
| panda|rover | keystoneauth1.exceptions.catalog.EndpointNotFound: Could not find requested endpoint in Service Catalog. | 11:31 |
| panda|rover | mmmhh | 11:31 |
| panda|rover | this was happening before too | 11:34 |
| panda|rover | ykarel: ok, services started | 11:34 |
| *** lucasagomes is now known as lucas-hungry | 11:35 | |
| panda|rover | ykarel: it should work now | 11:35 |
| ykarel | panda|rover, ok doing recheck on one job to check | 11:36 |
| ykarel | panda|rover, so looks like it's working, environment created in https://review.rdoproject.org/jenkins/job/gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/11382/console few minutes back | 11:42 |
| panda|rover | ykarel: yep. I also proposed a change to prevent this to happen again | 11:46 |
| panda|rover | ykarel: https://review.rdoproject.org/r/13441 | 11:46 |
| ykarel | nice | 11:46 |
| ykarel | panda|rover, quiquell|ruck if we want, we can rerun master promotion now(currently scheduled to run after 10 hours) as issues are fixed | 11:47 |
| quiquell|ruck | ykarel: You mean force the periodics ? | 11:48 |
| ykarel | quiquell|ruck, yes | 11:48 |
| quiquell|ruck | Don't know how | 11:48 |
| panda|rover | last time we forced the promotion we got bad results | 11:49 |
| panda|rover | but we were runing promotions every 4 hours | 11:49 |
| panda|rover | ykarel: you're sure about the 10 hours ? | 11:49 |
| ykarel | panda|rover, next run is at 22:00 UTC | 11:50 |
| panda|rover | ykarel: if another pipeline is started, it will change the tripleo-ci-testing link and nothing will test the correct link anymore | 11:50 |
| panda|rover | ykarel: even for master ?? | 11:50 |
| *** atoth has joined #oooq | 11:50 | |
| ykarel | panda|rover, 4 hour periodic was changed some day ago due to large zuul queue | 11:50 |
| ykarel | yes for master and queens | 11:50 |
| panda|rover | ok then | 11:51 |
| weshay | panda|rover, quiquell|ruck I think we can skip a sync today as long as we're monitoring https://rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/all%20top%20level%20multijobs/ | 11:55 |
| weshay | criteria for rdo2 again.. is one virt, one bm pass, --> update criteria config and roll out | 11:56 |
| chandankumar | weshay: Good morning sir | 11:56 |
| *** zoli|lunch is now known as zoli | 11:56 | |
| *** zoli is now known as zoli|wfh | 11:56 | |
| weshay | chandankumar, howdy | 11:56 |
| chandankumar | weshay: I need some help on this https://review.openstack.org/#/q/topic:only_containerized_uc+(status:open+OR+status:merged) | 11:56 |
| *** zoli|wfh is now known as zoli | 11:56 | |
| chandankumar | weshay: I wanted to run this featureset | 11:56 |
| chandankumar | but i am not sure where i am doing wrong | 11:56 |
| weshay | hrm.. I don't think zuul runs unmerged jobs | 11:57 |
| weshay | chandankumar, my suggestion would be to temp.. use a production fs number in a DNM patch and put in the content you want | 11:57 |
| chandankumar | weshay: you mean to say to use fs027 itself? | 11:58 |
| weshay | chandankumar, ya. .but in a DNM patch | 11:59 |
| chandankumar | weshay: yup sir let me do that | 11:59 |
| quiquell|ruck | weshay: For me is ok the skip | 12:00 |
| weshay | yes.. yes it is .. trust me | 12:00 |
| weshay | any folks have time to help review some zuul changes? | 12:00 |
| weshay | https://review.openstack.org/#/c/562347/ | 12:00 |
| weshay | https://review.openstack.org/#/c/562353/ | 12:01 |
| weshay | this is for upgrades | 12:01 |
| weshay | I have a minus 1 from zuul w/ no feedback | 12:01 |
| weshay | maybe zuul went down last night? | 12:01 |
| weshay | oh I see it | 12:02 |
| weshay | nvrmind | 12:02 |
| *** adarazs_lunch is now known as adarazs | 12:02 | |
| panda|rover | weshay: zuul specifies the error it's hitting, you have two vars: section on the same job description | 12:04 |
| *** amoralej is now known as amoralej|lunch | 12:06 | |
| *** panda|rover is now known as panda|rover|lnc | 12:10 | |
| *** quiquell|ruck is now known as quique|ruck|food | 12:11 | |
| hubbot | FAILING CHECK JOBS: gate-tripleo-ci-centos-7-container-to-container-upgrades-master-nv, gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens-branch, gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-pike-branch, gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master, gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-ocata-branch, (1 more message) | 12:15 |
| quique|ruck|food | tripleo-doc typo | 12:25 |
| quique|ruck|food | https://review.openstack.org/#/c/562560/ | 12:25 |
| *** rfolco|off is now known as rfolco | 12:26 | |
| *** agopi has joined #oooq | 12:28 | |
| *** rlandy has joined #oooq | 12:30 | |
| *** quique|ruck|food is now known as quiquell|ruck | 12:31 | |
| *** trown|outtypewww is now known as trown | 12:32 | |
| *** lucas-hungry is now known as lucasagomes | 12:33 | |
| *** agopi has quit IRC | 12:38 | |
| *** agopi has joined #oooq | 12:39 | |
| quiquell|ruck | weshay, rlandy: Trying to rebuild IPA here, but need the rhel-server image | 12:40 |
| *** agopi_ has joined #oooq | 12:40 | |
| quiquell|ruck | where can I get the rhel-server-7.4-x86_64-kvm.qcow2 ? | 12:41 |
| rlandy | quiquell|ruck: you're rebuilding on rhel 7.4 as opposed to 7.5? | 12:43 |
| quiquell|ruck | rlandy: I mean the 7.5 version | 12:44 |
| quiquell|ruck | Ok I can get it from the tenat | 12:44 |
| quiquell|ruck | tenant | 12:44 |
| rlandy | ci-rhos should have one | 12:44 |
| rlandy | yeah | 12:44 |
| rlandy | has all internal builds | 12:44 |
| rlandy | also see the internal release file? | 12:44 |
| rlandy | see line 29? | 12:45 |
| rlandy | that download dir has a bunch of internal stuff if you ever need it | 12:45 |
| quiquell|ruck | Getting rhel-7.5-server-x86_64-latest | 12:46 |
| quiquell|ruck | rlandy: What's the internal release file ? | 12:47 |
| rlandy | release files in tripleo-environments | 12:47 |
| quiquell|ruck | Ahh ok | 12:47 |
| quiquell|ruck | Thanks! | 12:47 |
| rlandy | rhos-12.yml is te one that jobs uses | 12:47 |
| rlandy | rfolco: forwarded you the ticket for tracking | 12:48 |
| rfolco | rlandy, thank you very much | 12:48 |
| *** agopi__ has joined #oooq | 12:53 | |
| *** agopi has quit IRC | 12:55 | |
| *** agopi_ has quit IRC | 12:55 | |
| *** agopi__ has quit IRC | 12:55 | |
| *** agopi has joined #oooq | 12:56 | |
| *** agopi has quit IRC | 12:56 | |
| *** ratailor has quit IRC | 12:59 | |
| *** agopi has joined #oooq | 13:02 | |
| panda|rover|lnc | trown: looking at the POC change, didn't you want to use a different baremetal template for this workflow ? | 13:05 |
| trown | panda|rover|lnc: I updated that | 13:05 |
| panda|rover|lnc | GRRR, gertty | 13:06 |
| panda|rover|lnc | trown: ok, thanks. | 13:06 |
| trown | panda|rover|lnc: line 45-52 in https://review.openstack.org/#/c/561630/8/roles/libvirt/setup/overcloud/tasks/main.yml | 13:06 |
| panda|rover|lnc | trown: yes yes, I'm too quick for gertty, I check the review before it has time to sync the latest patchset | 13:07 |
| panda|rover|lnc | trown: I see it now | 13:07 |
| *** panda|rover|lnc is now known as panda|rover | 13:07 | |
| *** Goneri has joined #oooq | 13:07 | |
| *** ykarel is now known as ykarel|away | 13:12 | |
| *** ykarel|away has quit IRC | 13:23 | |
| quiquell|ruck | rlandy: What's the image used as the base for the IPA in this stack ? | 13:24 |
| quiquell|ruck | I am trying to rebuild based on rhel 7.5 but it fails a openstack-selinux | 13:24 |
| quiquell|ruck | rhel-7.5-server-x86_64-latest should be good ? | 13:25 |
| rlandy | whatever rel eng built it on - it was not built was the tenant - you can check with jjoyce on #rhos-delivery for the exact image of the build | 13:27 |
| jjoyce | quiquell|ruck: That should work | 13:30 |
| quiquell|ruck | ok | 13:31 |
| jjoyce | As long as you are rhel-7.5 based and up to date. | 13:31 |
| chandankumar | weshay: arxcruz kopecmartin I am skipping today's scrum, I have updated the card, I need to leave early, sorry for that :-) | 13:36 |
| arxcruz | is myoung available ? | 13:36 |
| chandankumar | arxcruz: I think he is attending conference | 13:38 |
| arxcruz | yup | 13:38 |
| kopecmartin | arxcruz, should we reschedule? | 13:38 |
| arxcruz | up to you guys, i also update my card yesterday, didn't make any updates, still waiting patches pass | 13:39 |
| kopecmartin | I don't have any updates neither | 13:39 |
| kopecmartin | we can move it for tomorrow, or just skip it, on Monday we have another meeting | 13:40 |
| *** amoralej|lunch is now known as amoralej | 13:45 | |
| amoralej | one last push to https://review.openstack.org/#/c/561484/ ? | 13:46 |
| rlandy | trown: hello - the current error I am hitting is with repo-setup on the second node - no repos are enabled | 13:54 |
| trown | rlandy: hmm... I just had a full run deploy with fs10 ... maybe after scrum we could do a tmate? | 13:55 |
| rlandy | idk - seems setup unrelated | 13:55 |
| rlandy | going to try this one more time | 13:55 |
| trown | rlandy: ok... i am also starting from scratch running only commands in the etherpad | 13:59 |
| amoralej | trown++ | 14:00 |
| * rlandy follows etherpad - just changing location of images | 14:00 | |
| hubbot | amoralej: trown's karma is now 42 | 14:00 |
| panda|rover | weshay: joining scrum ? | 14:02 |
| trown | rlandy: hmm one thing to check wrt repos... if your vpn connection on the virthost resets, it will overwrite resolv.conf there | 14:02 |
| rlandy | trown: I edited resolv.conf | 14:03 |
| rlandy | didn't help | 14:03 |
| rlandy | but retrying | 14:03 |
| trown | k | 14:03 |
| rlandy | on both nodes | 14:03 |
| quiquell|ruck | amoralej: What's this hubbot karma thing ? | 14:03 |
| adarazs | quiquell|ruck: just a fun thing to play with. :) | 14:07 |
| quiquell|ruck | adarazs: That the most interesting feature | 14:07 |
| amoralej | quiquell|ruck, don't be evil and will get karma :) | 14:07 |
| adarazs | hubbot: coin | 14:08 |
| hubbot | adarazs: tails | 14:08 |
| adarazs | hubbot: dice | 14:08 |
| hubbot | adarazs: (dice <dice>d<sides>) -- Rolls a die with <sides> number of sides <dice> times. For example, 2d6 will roll 2 six-sided dice; 10d10 will roll 10 ten-sided dice. | 14:08 |
| adarazs | hubbot: dice 2d10 | 14:08 |
| hubbot | adarazs: 3 and 5 | 14:08 |
| adarazs | !eightball Will it rain today? | 14:08 |
| hubbot | adarazs: No chance. | 14:08 |
| openstack | adarazs: Error: "eightball" is not a valid command. | 14:08 |
| adarazs | hubbot: roulette | 14:09 |
| hubbot | adarazs: *click* | 14:09 |
| adarazs | phew. (it's russian roulette :) | 14:09 |
| quiquell|ruck | Rise of the Robots again !!! | 14:12 |
| hubbot | FAILING CHECK JOBS: gate-tripleo-ci-centos-7-container-to-container-upgrades-master-nv, gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-pike-branch, gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master, gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-ocata-branch, tripleo-quickstart-extras-gate-newton-delorean-full-minimal, (1 more message) | 14:15 |
| *** skramaja has quit IRC | 14:16 | |
| *** ykarel|away has joined #oooq | 14:22 | |
| *** ykarel|away is now known as ykarel | 14:23 | |
| *** quiquell|ruck is now known as quiquell|off | 14:25 | |
| rlandy | trown: side question ... you removed the ZUUL_CHANGES from the etherpad in env_to_src? | 14:28 |
| trown | rlandy: ya that is not the place to inject it, and the patch I needed merged now | 14:28 |
| rlandy | not needed - right - see the merge - ok | 14:29 |
| trown | rlandy: for quickstart/extras/tripleo-ci changes have to be exported before running nodepool-setup | 14:29 |
| weshay | arxcruz, you here? | 14:46 |
| arxcruz | weshay: yes | 14:47 |
| *** bogdando has quit IRC | 14:51 | |
| *** Goneri has quit IRC | 14:54 | |
| *** bogdando has joined #oooq | 14:56 | |
| *** Goneri has joined #oooq | 14:57 | |
| kopecmartin | arxcruz, how is it with the meeting? the link seems to be occupied :) | 15:03 |
| *** bogdando has quit IRC | 15:03 | |
| arxcruz | kopecmartin: i think we can skip for today, since chandankumar isn't here and we already update the cards :) | 15:09 |
| kopecmartin | kopecmartin, agree | 15:09 |
| weshay | quiquell|off, panda|rover looks like queens can promote in rdo2 | 15:13 |
| weshay | quiquell|off, panda|rover config looks good, so it should promote when the promoter frees up | 15:15 |
| weshay | :) | 15:15 |
| weshay | rdo2 aint so bad :) | 15:16 |
| *** ykarel is now known as ykarel|away | 15:19 | |
| *** tcw has joined #oooq | 15:21 | |
| *** ccamacho has quit IRC | 15:23 | |
| *** ykarel|away has quit IRC | 15:24 | |
| *** marios has quit IRC | 15:30 | |
| *** kopecmartin has quit IRC | 15:35 | |
| *** ccamacho has joined #oooq | 15:42 | |
| *** ccamacho has quit IRC | 15:43 | |
| *** ccamacho has joined #oooq | 15:43 | |
| panda|rover | trown: rlandy, all gates on POC passed, approved, moving card to complete | 15:50 |
| rlandy | ack- thanks | 15:50 |
| trown | nice thanks | 15:50 |
| *** links has quit IRC | 16:01 | |
| *** jfrancoa has quit IRC | 16:03 | |
| *** holser__ has quit IRC | 16:03 | |
| *** ccamacho has quit IRC | 16:06 | |
| hubbot | FAILING CHECK JOBS: gate-tripleo-ci-centos-7-container-to-container-upgrades-master-nv, gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-pike-branch, gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master, gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-ocata-branch, tripleo-quickstart-extras-gate-newton-delorean-full-minimal, (1 more message) | 16:15 |
| *** florianf has quit IRC | 16:35 | |
| *** lucasagomes is now known as lucas-afk | 16:40 | |
| *** trown is now known as trown|lunch | 17:05 | |
| *** zoli is now known as zoli|gone | 17:08 | |
| *** zoli|gone is now known as zoli | 17:08 | |
| *** gkadam has quit IRC | 17:09 | |
| *** tcw has quit IRC | 17:15 | |
| *** udesale_ has quit IRC | 17:24 | |
| *** tcw has joined #oooq | 17:32 | |
| rlandy | trown|lunch: "overcloud_deploy_result": "passed" !!!! | 17:38 |
| rlandy | +(/opt/stack/tripleo-ci/toci_gate_test-oooq.sh:312): echo 'Run completed' | 17:44 |
| rlandy | Run completed | 17:44 |
| rlandy | panda|rover: ^^ fs10 - verified | 17:44 |
| rlandy | some dns and probably keys tweaks needed | 17:45 |
| panda|rover | rlandy: you verified the basic POC ? | 17:46 |
| rlandy | panda|rover: ack | 17:46 |
| rlandy | with trown|lunch's help | 17:46 |
| panda|rover | rlandy: good! | 17:47 |
| panda|rover | ready for the real work now :P | 17:47 |
| *** panda|rover is now known as panda|rover|off | 17:47 | |
| rlandy | yep | 17:47 |
| *** tcw has quit IRC | 18:00 | |
| *** amoralej is now known as amoralej|off | 18:03 | |
| *** tcw has joined #oooq | 18:04 | |
| *** tcw has quit IRC | 18:05 | |
| *** tcw has joined #oooq | 18:05 | |
| hubbot | FAILING CHECK JOBS: gate-tripleo-ci-centos-7-container-to-container-upgrades-master-nv, gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-pike-branch, gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master, gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-ocata-branch, tripleo-quickstart-extras-gate-newton-delorean-full-minimal, (1 more message) | 18:15 |
| weshay | bah.. /me hates EURO RUCK ROVERS | 18:20 |
| *** trown|lunch is now known as trown | 18:21 | |
| trown | rlandy: nice one | 18:21 |
| rlandy | trown: I think the deployment behaves better when you are watching it :) | 18:21 |
| trown | if only I could just watch every CI job all the time | 18:22 |
| trown | although that does not seem to be the case on my minidell :P | 18:23 |
| rlandy | trown: I think this task ''TASK [nodepool-setup : Wait for provisioned hosts to become reachable]' fails because it just doesn't wait long enough | 18:46 |
| rlandy | the nodes are ssh available about a minute after the task bails | 18:46 |
| rlandy | actually scratch that - subnode-0 is - subnode-1 hit the same issue as a while back unreachable | 18:49 |
| weshay | trown, we I delegate_to: localhost it's not using the default ansible_user I'm running with | 18:52 |
| weshay | it's using the user specified in the play | 18:52 |
| weshay | trying to override w/ become_user: foo | 18:53 |
| weshay | is also not working | 18:53 |
| trown | hmm | 18:53 |
| weshay | unless | 18:53 |
| weshay | I need become: true become_user: foo | 18:53 |
| trown | ya you do | 18:53 |
| weshay | hrm.. k | 18:53 |
| * weshay tries that | 18:53 | |
| weshay | doesn't work :((((( | 18:54 |
| *** tesseract has quit IRC | 18:55 | |
| weshay | trown, check this out http://pastebin.test.redhat.com/579379 | 18:56 |
| weshay | - name: Create virthost access key | 18:57 |
| weshay | delegate_to: localhost | 18:57 |
| weshay | become: true | 18:57 |
| weshay | become_user: "{{ ansible_user }}" | 18:57 |
| weshay | command: > | 18:57 |
| weshay | ssh-keygen -f {{ virt_host_key }} -N '' | 18:57 |
| weshay | -C 'ansible_generated_virt_host' | 18:57 |
| weshay | -t rsa -b 4096 | 18:57 |
| weshay | args: | 18:57 |
| weshay | creates: "{{ virt_host_key }}" | 18:57 |
| weshay | ah I see why | 18:58 |
| weshay | something w/ "{{ ansible_user }}" | 18:58 |
| weshay | become: true | 19:05 |
| weshay | become_user: "{{ hostvars['localhost']['ansible_user'] }}" | 19:05 |
| weshay | is the way | 19:05 |
| weshay | trown, fyi ^ | 19:05 |
| trown | ah makes sense | 19:06 |
| *** tosky has quit IRC | 20:15 | |
| hubbot | FAILING CHECK JOBS: gate-tripleo-ci-centos-7-container-to-container-upgrades-master-nv, gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-pike-branch, gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master, gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-ocata-branch, tripleo-quickstart-extras-gate-newton-delorean-full-minimal, (1 more message) | 20:15 |
| *** tosky has joined #oooq | 20:16 | |
| *** atoth has quit IRC | 20:41 | |
| rlandy | trown: looking at sshnaidm|off's patch, it looks like updating packages requires the reboot. | 20:41 |
| rlandy | is there any point in updating packages if we skip the reboot? | 20:41 |
| rlandy | if not, the solution is not a great idea | 20:41 |
| rlandy | to run w/o updated packages | 20:42 |
| rlandy | I would rather edit the resolv.conf again that not update pckages | 20:42 |
| rlandy | packages | 20:42 |
| trown | oh right for kernel updates... | 20:43 |
| rlandy | yeah - so skipping the reboot is bad | 20:43 |
| trown | unless we move the update to virt-customize before the vms ever boot | 20:44 |
| trown | then we only do it once as well | 20:44 |
| trown | rather than once per subnode | 20:44 |
| rlandy | that is a better idea | 20:45 |
| rlandy | looking into assigning external dns server | 20:46 |
| rlandy | trown: btw ... added 'https://review.openstack.org/562825 WIP: Add external DNS server to the /etc/resolv.conf' so we don't forget that change while working on another | 20:47 |
| trown | nice | 20:48 |
| *** trown is now known as trown|outtypewww | 20:49 | |
| *** rfolco is now known as rfolco|off | 21:06 | |
| *** Goneri has quit IRC | 21:17 | |
| *** jtomasek has quit IRC | 21:28 | |
| *** jtomasek has joined #oooq | 21:29 | |
| *** agopi has quit IRC | 21:41 | |
| hubbot | FAILING CHECK JOBS: gate-tripleo-ci-centos-7-container-to-container-upgrades-master-nv, tripleo-quickstart-extras-gate-newton-delorean-full-minimal | check logs @ https://review.openstack.org/472607 and fix them ASAP. | 22:15 |
| *** tosky has quit IRC | 23:10 | |
| *** Goneri has joined #oooq | 23:40 | |
| *** agopi has joined #oooq | 23:41 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!