| *** Goneri has quit IRC | 00:12 | |
| *** Goneri has joined #oooq | 00:22 | |
| *** Goneri has quit IRC | 00:27 | |
| hubbot | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-3nodes-multinode, tripleo-ci-centos-7-scenario001-multinode-oooq-container, tripleo-ci-centos-7-scenario007-multinode-oooq-container, tripleo-ci-centos-7-scenario002-multinode-oooq-container, tripleo-ci-centos-7-containers-multinode, tripleo-ci-centos-7-scenario003-multinode-oooq-container, (1 more message) | 00:48 |
|---|---|---|
| *** lhinds has quit IRC | 01:59 | |
| *** rnoriega has quit IRC | 01:59 | |
| *** faceman has quit IRC | 02:00 | |
| *** pliu has quit IRC | 02:00 | |
| *** sanjayu_ has joined #oooq | 02:02 | |
| *** rnoriega has joined #oooq | 02:14 | |
| *** pliu has joined #oooq | 02:15 | |
| *** lhinds has joined #oooq | 02:15 | |
| *** faceman has joined #oooq | 02:17 | |
| *** sanjayu_ has quit IRC | 02:18 | |
| *** vinaykns has joined #oooq | 02:25 | |
| *** vvaldez has quit IRC | 02:47 | |
| *** vvaldez has joined #oooq | 02:47 | |
| hubbot | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-3nodes-multinode, tripleo-ci-centos-7-scenario001-multinode-oooq-container, tripleo-ci-centos-7-scenario007-multinode-oooq-container, tripleo-ci-centos-7-scenario002-multinode-oooq-container, tripleo-ci-centos-7-containers-multinode, tripleo-ci-centos-7-scenario003-multinode-oooq-container, (1 more message) | 02:48 |
| *** links has joined #oooq | 03:23 | |
| *** vvaldez has quit IRC | 03:32 | |
| *** vinaykns has quit IRC | 03:33 | |
| *** hrybacki has quit IRC | 03:40 | |
| *** udesale has joined #oooq | 03:40 | |
| *** hrybacki has joined #oooq | 03:42 | |
| *** ykarel has joined #oooq | 03:46 | |
| *** sanjayu_ has joined #oooq | 03:46 | |
| *** sdoran has quit IRC | 03:48 | |
| *** ajo has quit IRC | 03:49 | |
| *** hrybacki has quit IRC | 03:50 | |
| *** sanjayu__ has joined #oooq | 03:50 | |
| *** sanjayu_ has quit IRC | 03:53 | |
| *** dtrainor has quit IRC | 04:29 | |
| *** dtrainor has joined #oooq | 04:29 | |
| *** openstack has joined #oooq | 04:29 | |
| hubbot | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224 | 04:48 |
| *** skramaja has joined #oooq | 04:58 | |
| *** pgadiya has joined #oooq | 05:09 | |
| *** pgadiya has quit IRC | 05:09 | |
| *** ratailor has joined #oooq | 05:16 | |
| *** ajo has joined #oooq | 05:24 | |
| *** sdoran has joined #oooq | 05:28 | |
| *** bogdando has joined #oooq | 05:28 | |
| *** hrybacki has joined #oooq | 05:30 | |
| *** udesale_ has joined #oooq | 05:33 | |
| *** quiquell|off is now known as quiquell|rover | 05:34 | |
| *** udesale has quit IRC | 05:36 | |
| *** hrybacki has quit IRC | 05:55 | |
| *** sdoran has quit IRC | 05:57 | |
| quiquell|rover | sshnaidm|afk: https://review.rdoproject.org/r/14344 | 05:57 |
| quiquell|rover | sshnaidm|afk: To hash the DLRN hash in the skipped | 05:57 |
| *** ajo has quit IRC | 05:58 | |
| *** jtomasek has joined #oooq | 06:11 | |
| *** hrybacki has joined #oooq | 06:17 | |
| bogdando | o/ PTAL https://review.openstack.org/#/q/topic:localcon+(status:open+OR+status:merged) | 06:18 |
| quiquell|rover | bogdando: What means PTAL ? | 06:22 |
| bogdando | quiquell|rover: Please Take A(Another) Look | 06:24 |
| *** ajo has joined #oooq | 06:24 | |
| *** bandini has quit IRC | 06:24 | |
| quiquell|rover | bogdando: ack | 06:26 |
| *** bandini has joined #oooq | 06:26 | |
| *** sdoran has joined #oooq | 06:26 | |
| *** ykarel_ has joined #oooq | 06:33 | |
| *** ykarel has quit IRC | 06:36 | |
| *** ykarel__ has joined #oooq | 06:42 | |
| *** ykarel_ has quit IRC | 06:42 | |
| *** ykarel__ is now known as ykarel | 06:43 | |
| *** kopecmartin has joined #oooq | 06:44 | |
| hubbot | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224 | 06:48 |
| quiquell|rover | nick quique|rover|afk | 07:00 |
| quiquell|rover | ups | 07:00 |
| quiquell|rover | :-) | 07:00 |
| *** quiquell|rover is now known as quique|rover|afk | 07:00 | |
| *** anande has joined #oooq | 07:00 | |
| *** tesseract has joined #oooq | 07:04 | |
| *** udesale__ has joined #oooq | 07:06 | |
| *** udesale_ has quit IRC | 07:09 | |
| *** amoralej|off is now known as amoralej | 07:21 | |
| *** dtantsur|afk is now known as dtantsur | 07:21 | |
| *** tosky has joined #oooq | 07:30 | |
| *** quique|rover|afk is now known as quiquell|rover | 07:36 | |
| *** ccamacho has joined #oooq | 07:43 | |
| bogdando | one more thing discovered while exploring the wonderful world of libvirt reproducer https://review.openstack.org/576772 , PTAL | 07:44 |
| bogdando | so now it almost works for me, except that it puts '127.0.0.1 subnode-1' into my localhost virthost | 07:45 |
| bogdando | tried with https://review.openstack.org/#/c/576455/ as well... | 07:45 |
| bogdando | no luck with rdo cloud, neither with libvirt reproducing so far ;( | 07:46 |
| *** holser_ has joined #oooq | 07:50 | |
| quiquell|rover | arxcruz: Do you know the issue here ? http://logs.openstack.org/85/564285/20/check/tripleo-ci-centos-7-scenario001-multinode-oooq-container/ef7b33c/logs/undercloud/home/zuul/tempest/tempest.html.gz | 07:51 |
| ykarel | quiquell|rover, ^^ is same as alerted bug for pike | 07:55 |
| arxcruz | quiquell|rover: it seems to me random failure, the volume was in error state | 07:55 |
| arxcruz | but i can be wrong | 07:55 |
| ykarel | arxcruz, we are seeing same in stable/pike promotion job | 07:55 |
| ykarel | so it's not random | 07:55 |
| arxcruz | okay | 07:56 |
| quiquell|rover | ykarel: I know, was just debugging it | 07:56 |
| *** gkadam has joined #oooq | 07:56 | |
| quiquell|rover | arxcruz: Is persistent and a promotion blocker for pike | 07:56 |
| ykarel | quiquell|rover, Okk | 07:56 |
| ykarel | quiquell|rover, do you have reproducer? | 07:56 |
| arxcruz | quiquell|rover: can i dig into your reproduced env ? | 07:56 |
| quiquell|rover | ykarel, arxcruz: http://logs.openstack.org/85/564285/20/check/tripleo-ci-centos-7-scenario001-multinode-oooq-container/ef7b33c/logs/reproducer-quickstart.sh | 07:57 |
| *** anande has quit IRC | 07:57 | |
| ykarel | bogdando, which kernel u have | 07:57 |
| bogdando | 4.10 something, ykarel, basically this part now worksfor me | 07:57 |
| bogdando | just stack on the 2nd step of the repro script | 07:58 |
| bogdando | stuck | 07:58 |
| ykarel | bogdando, i think you just need LIBGUESTFS_BACKEND_SETTINGS: force_tcg | 07:58 |
| ykarel | other vars are not required, can you try with just ^^ | 07:59 |
| bogdando | ykarel: https://pastebin.com/f97PVQkW | 07:59 |
| bogdando | ykarel: I could just add this param to the set of env vars supported as well | 07:59 |
| bogdando | it will be set to empty by default so no worries | 08:00 |
| bogdando | ykarel: um, I think I said it wrong, I have 4.13, but it only works with 4.10 | 08:00 |
| bogdando | but never-mind, I'm mostly concerned with the latter blocker | 08:01 |
| ykarel | bogdando, have never tried libvirt reproducer, | 08:02 |
| bogdando | ykarel: me too :) | 08:02 |
| ykarel | so it would be good to know if someone already faced this, or it's specific to ur environment | 08:03 |
| bogdando | now it seemed to be the right time, given that I cannot provision a repro env on rdo lcoud | 08:03 |
| bogdando | but... sigh | 08:03 |
| ykarel | bogdando, what issue u face when trying with rdocloud | 08:03 |
| bogdando | the FIP related prolly | 08:03 |
| bogdando | node never comes back and it hangs on waiting for it | 08:04 |
| bogdando | tied yday, will try today again | 08:04 |
| bogdando | tried | 08:04 |
| ykarel | quiquell|rover faced something similar yesterday ^^ | 08:04 |
| ykarel | quiquell|rover, u got it fixed? | 08:04 |
| bogdando | some time ago what helped is recreating all my routers :S | 08:04 |
| bogdando | don't want to come through that again | 08:05 |
| bogdando | but I might have to... | 08:05 |
| quiquell|rover | ykarel, bogdando: I miss some RPMs | 08:05 |
| quiquell|rover | ykarel, bogdando: libvirt-python, python-lxml, libguest-tools. | 08:06 |
| bogdando | quiquell|rover: I lost the context for that, sorry | 08:07 |
| bogdando | what is that for? | 08:07 |
| arxcruz | quiquell|rover: i'm bringing up an env on my tenant, it will take a while :) | 08:07 |
| arxcruz | ykarel: ^ | 08:07 |
| tosky | ykarel: hi, talking with arxcruz we realized that https://review.openstack.org/#/c/576356/ and https://review.openstack.org/#/c/568869/ are mutually exclusive | 08:07 |
| bogdando | quiquell|rover: I have installed those but libvirt-python | 08:07 |
| ykarel | arxcruz, ack | 08:08 |
| tosky | ykarel: and we would prefer the latter, so don't be scared about the upcoming -W on your patch | 08:08 |
| ykarel | tosky, no issues, anypatch we can get early is fine | 08:08 |
| ykarel | we are out of promotion for few days | 08:08 |
| ykarel | and this seems to the last issue | 08:08 |
| bogdando | quiquell|rover: never-mind I have them all installed | 08:08 |
| quiquell|rover | ykarel: do you have the link of the "Member vs member" bug ? | 08:11 |
| ykarel | quiquell|rover, https://bugs.launchpad.net/tripleo/+bug/1777451 | 08:12 |
| openstack | Launchpad bug 1777451 in tripleo "Error: /Stage[main]/Ceph::Rgw::Keystone::Auth/Keystone_role Duplicate entry found with name Member" [Critical,Fix released] - Assigned to Quique Llorente (quiquell) | 08:12 |
| quiquell|rover | ykarel: got it | 08:12 |
| quiquell|rover | thanks | 08:12 |
| bogdando | folks, those of you who had a chance to try the libvirt reproduce script, do you know something of https://pastebin.com/f97PVQkW ?.. the generated inventory and ssh config look weird to me | 08:12 |
| bogdando | why it point my localhost virthost to the subnode VM ip?.. | 08:13 |
| quiquell|rover | ykarel: This bug is pending on https://review.openstack.org/#/c/568869/ to be merged ? | 08:16 |
| *** ykarel_ has joined #oooq | 08:17 | |
| *** ykarel has quit IRC | 08:20 | |
| *** anande has joined #oooq | 08:44 | |
| hubbot | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224 | 08:48 |
| *** ykarel_ is now known as ykarel|lunch | 08:48 | |
| *** sshnaidm|afk is now known as sshnaidm | 08:57 | |
| *** ykarel_ has joined #oooq | 09:11 | |
| *** ykarel|lunch has quit IRC | 09:13 | |
| *** panda|off is now known as panda | 09:33 | |
| *** ykarel_ has quit IRC | 09:35 | |
| *** ykarel_ has joined #oooq | 09:35 | |
| quiquell|rover | sshnaidm: Don't know why /var/log/ceph/ is not collected here http://logs.openstack.org/85/564285/20/check/tripleo-ci-centos-7-scenario004-multinode-oooq-container/5894ea4/logs/subnode-2/ | 09:38 |
| quiquell|rover | panda: ^ | 09:39 |
| *** matbu has quit IRC | 09:41 | |
| sshnaidm | quiquell|rover, http://logs.openstack.org/85/564285/20/check/tripleo-ci-centos-7-scenario004-multinode-oooq-container/5894ea4/logs/subnode-2/var/log/extra/docker/containers/ | 09:43 |
| quiquell|rover | arxcruz, chandankumar: https://review.openstack.org/#/c/568869/ | 09:43 |
| quiquell|rover | failing at http://logs.openstack.org/69/568869/37/gate/python-tempestconf-tempest-packstack-demo/01f6331/ | 09:43 |
| quiquell|rover | sshnaidm: ack, thanks ! | 09:44 |
| quiquell|rover | sshnaidm: And why, log file = /var/log/ceph/ceph-rgw-centos-7-ovh-bhs1-0000241067.log doesn't appear ? | 09:45 |
| *** matbu has joined #oooq | 09:46 | |
| arxcruz | quiquell|rover: it's a random failure, for sure this time, need to recheck :( | 09:50 |
| quiquell|rover | arxcruz: ok | 09:51 |
| *** ykarel__ has joined #oooq | 09:59 | |
| *** ykarel_ has quit IRC | 10:02 | |
| *** zoli is now known as zoli|lunch | 10:03 | |
| *** ykarel__ is now known as ykarel | 10:04 | |
| ykarel | arxcruz, tosky -2 from zuul | 10:05 |
| tosky | ykarel: old | 10:05 |
| tosky | both patches were hit by a random failure | 10:05 |
| tosky | rechecked the one that should land | 10:05 |
| ykarel | arxcruz, tosky should we consider temporary patch in RDO | 10:06 |
| ykarel | as next periodic run is in 2 hours | 10:06 |
| *** anande has quit IRC | 10:06 | |
| tosky | which temporary patch? | 10:06 |
| ykarel | tosky, https://review.openstack.org/#/c/568869/ as a patch in rpm package | 10:06 |
| ykarel | once upstream merge we can revert RDO revie | 10:07 |
| tosky | I can't make that decision; it's the kind of things I'm not a big fan of | 10:07 |
| ykarel | weshay|ruck, what u say? | 10:08 |
| quiquell|rover | ykarel: It's not the first time we do that, we can try | 10:10 |
| tosky | "next periodic run" is the promotion run? | 10:10 |
| quiquell|rover | ykarel: but having sporadic erros is not a good sign | 10:10 |
| tosky | quiquell|rover: it's unrelated | 10:10 |
| quiquell|rover | tosky: jobs checked at promotions | 10:10 |
| tosky | unrelated | 10:11 |
| tosky | infra issue, which we can't pinpoint | 10:11 |
| quiquell|rover | tosky: a recheck will take 2,5 hours or so | 10:11 |
| tosky | once daily? | 10:11 |
| tosky | anyway, I won't block the effort | 10:12 |
| tosky | but I can only say that I don't see why a promotion should happen now with the keystone issues still around | 10:13 |
| tosky | "luckily" not catched by some jobs, but still - sahara is broken right now | 10:13 |
| tosky | mpf, but the show... sorry, the pipeline must go on | 10:13 |
| tosky | so please propose this RPM patch if you think it may do something good | 10:14 |
| *** anande has joined #oooq | 10:14 | |
| quiquell|rover | ykarel: If we miss this window, when is the next one ? | 10:15 |
| ykarel | quiquell|rover, after 5 hours this one | 10:15 |
| quiquell|rover | ykarel: Let's wait for recheck and for Wes I think | 10:15 |
| ykarel | but it's good if we try the patch and move forward | 10:16 |
| ykarel | tosky, weshay|ruck quiquell|rover https://review.rdoproject.org/r/#/c/14348/ | 10:19 |
| tosky | seen | 10:19 |
| quiquell|rover | ykarel: ILet's do it, if this is not going to inroduce any regression | 10:21 |
| ykarel | quiquell|rover, delorean build on tempestconf patches which conflicts with the patch will fail, but that's for very less time | 10:25 |
| quiquell|rover | ykarel: You have to rebase your change ? | 10:30 |
| ykarel | checking | 10:31 |
| ykarel | quiquell|rover, rebase on what? | 10:31 |
| quiquell|rover | ykarel: nevermind | 10:32 |
| ykarel | ack | 10:33 |
| hubbot | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224 | 10:48 |
| *** jaganathan has quit IRC | 10:49 | |
| *** jaosorior has quit IRC | 10:56 | |
| quiquell|rover | sshnaidm, panda: https://review.rdoproject.org/r/#/c/14344/ change on a promoter trace | 10:59 |
| sshnaidm | quiquell|rover, commented | 11:02 |
| quiquell|rover | have to leave for a fiew minutes | 11:03 |
| *** quiquell|rover is now known as quiquell|rover|b | 11:03 | |
| *** quiquell|rover|b is now known as quique|rover|bbl | 11:03 | |
| *** quique|rover|bbl has quit IRC | 11:09 | |
| *** ratailor has quit IRC | 11:15 | |
| *** amoralej is now known as amoralej|out | 11:20 | |
| *** udesale__ has quit IRC | 11:20 | |
| *** zoli|lunch is now known as zoli | 11:21 | |
| *** holser_ has quit IRC | 11:35 | |
| *** jaosorior has joined #oooq | 11:36 | |
| *** atoth has joined #oooq | 11:42 | |
| marios | panda: defined here https://github.com/openstack-infra/tripleo-ci/blob/cf6b217b2e4f15edbf08dc60f60845e3eb500abc/scripts/tripleo.sh#L1249 | 11:49 |
| marios | used here | 11:49 |
| marios | this is what i was referring to | 11:49 |
| marios | ^ | 11:49 |
| marios | https://github.com/openstack-infra/tripleo-ci/blob/cf6b217b2e4f15edbf08dc60f60845e3eb500abc/scripts/tripleo.sh#L1383 | 11:49 |
| marios | *=* 12:30:57 *=*=*= "" 1249 function ovs_vxlan_bridge | 11:49 |
| marios | panda https://github.com/openstack-infra/zuul-jobs/blob/master/playbooks/multinode/pre.yaml | 11:51 |
| *** quiquell has joined #oooq | 11:54 | |
| *** quiquell is now known as quiquell|rover | 11:54 | |
| *** ajo has quit IRC | 12:00 | |
| *** ajo has joined #oooq | 12:01 | |
| *** chem has joined #oooq | 12:03 | |
| *** ykarel_ has joined #oooq | 12:17 | |
| *** ykarel has quit IRC | 12:19 | |
| *** rlandy has joined #oooq | 12:22 | |
| weshay|ruck | quiquell|rover, howdy | 12:24 |
| quiquell|rover | weshay|ruck: Hi wes, trying to find the issue with pike | 12:26 |
| weshay|ruck | quiquell|rover, see my bugs? | 12:27 |
| quiquell|rover | weshay|ruck: What bugs ? | 12:28 |
| quiquell|rover | weshay|ruck: I am checking why RBD is returning 0GB of ceph disk space on pike | 12:29 |
| arxcruz | weshay|ruck: before you ask, still didn't merge the path, gates are complaining with random failures :( | 12:30 |
| weshay|ruck | quiquell|rover, lp. two promotion blockers | 12:30 |
| weshay|ruck | alerts | 12:30 |
| quiquell|rover | weshay|ruck: That are the ones I am looking at | 12:30 |
| quiquell|rover | weshay|ruck: I just looking at the one regarding nova, I think they are related | 12:31 |
| quiquell|rover | weshay|ruck: The nova one has some issues with ceph | 12:31 |
| quiquell|rover | weshay|ruck: I have reproduce it, I am digging | 12:31 |
| weshay|ruck | quiquell|rover, k.. can I borrow you for a sec about the promoter? | 12:31 |
| quiquell|rover | weshay|ruck: sure | 12:32 |
| weshay|ruck | bluejeans and tmux a on promoter | 12:32 |
| *** bogdando has quit IRC | 12:43 | |
| *** tcw has joined #oooq | 12:43 | |
| *** myoung|off is now known as myoung | 12:48 | |
| hubbot | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224 | 12:48 |
| *** ykarel_ is now known as ykarel | 12:50 | |
| *** skramaja has quit IRC | 12:52 | |
| *** trown|outtypewww is now known as trown | 12:53 | |
| rlandy | weshay|ruck; quiquell|rover; just fyi - the periodic reproducer for multinode was broken ... https://review.openstack.org/#/c/576632/ | 12:53 |
| quiquell|rover | rlandy: Yep, noted, want to open a bug | 12:53 |
| quiquell|rover | rlandy: It's not generating the par of seting up things | 12:54 |
| rlandy | quiquell|rover: fix is above | 12:54 |
| rlandy | quiquell|rover: you can just edit the commit message to close your bug | 12:54 |
| weshay|ruck | rlandy heh | 12:55 |
| weshay|ruck | k.. on it | 12:55 |
| weshay|ruck | quickstart.sh is also busted | 12:55 |
| rlandy | weshay|ruck; quiquell|rover: secondly, the bm nodes patch merged last night. I am rekicking https://rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/tripleo-quickstart-master-rdo_trunk-baremetal-hp_dl360_envE-single_nic_vlans/144/console to test master | 12:55 |
| rlandy | if that works, we need to ff the stale branches | 12:55 |
| rlandy | stable | 12:56 |
| weshay|ruck | rlandy, I think I ff queens not sure if I got it though | 12:57 |
| weshay|ruck | will check | 12:57 |
| weshay|ruck | :q | 12:57 |
| *** anande has quit IRC | 12:59 | |
| *** vvaldez has joined #oooq | 13:00 | |
| trown | marios: panda: rlandy: interest in group hacking on networking stuff? I made quite a bit of progress yesterday | 13:01 |
| panda | trown: yes | 13:03 |
| marios | trown: o/. yes | 13:03 |
| rlandy | trown; sure | 13:03 |
| trown | https://bluejeans.com/9281082121 | 13:04 |
| rlandy | weshay|ruck: I guess master needs a ff as well? | 13:04 |
| *** vvaldez_ has joined #oooq | 13:06 | |
| ajo | yikes | 13:06 |
| rlandy | weshay|ruck: I guess that makes sense - since I rebuilt an old version | 13:06 |
| ajo | I'm getting this all the time | 13:06 |
| ajo | task path: /home/tripleo/.quickstart/usr/local/share/ansible/roles/undercloud-deploy/tasks/create-scripts.yml:96 | 13:06 |
| ajo | Wednesday 20 June 2018 08:46:45 -0400 (0:00:04.570) 0:30:00.388 ******** | 13:06 |
| ajo | fatal: [undercloud]: FAILED! => {"changed": false, "failed": true, "msg": "AnsibleUndefinedVariable: 'container_build_id' is undefined"} | 13:06 |
| ajo | any idea of what could that be? | 13:06 |
| *** vvaldez has quit IRC | 13:07 | |
| *** vvaldez_ is now known as vvaldez | 13:07 | |
| *** amoralej|out is now known as amoralej | 13:07 | |
| *** zoli is now known as zoli|afk | 13:09 | |
| *** zoli|afk is now known as zoli | 13:10 | |
| rlandy | myoung: I want to rekick a master bm job - but I want it to use the latest head of master on tripleo-quickstart | 13:12 |
| rlandy | is that possible? | 13:12 |
| myoung | rlandy: sure...can rebase the stable branch, or change the job config to point to upstream master | 13:13 |
| myoung | of TQ | 13:13 |
| myoung | rlandy: is this just for a one-off test and you don't want to rebase the stable/master branch? | 13:13 |
| myoung | rlandy: IMHO for {TQ, TQE}::stable/master we should make a job that fires frequently to rebase it, or modify the sbtest jobs to be a little more permissive and just do a smoke test (ovb) to ensure patches are basicallly ok. We end up needing to rebase stable/master so often | 13:15 |
| myoung | rlandy: HTH, let me know | 13:16 |
| ykarel | ajo, have you tried with clean workdir also? | 13:16 |
| ajo | hmm, ykarel no, but I can try , I didn't know that influenced it | 13:18 |
| ykarel | ajo, i have seen issues related to ansible cache when using same workdir, so good to try with clean workdir | 13:19 |
| ykarel | in anycase it's a bug in tq | 13:19 |
| *** holser_ has joined #oooq | 13:20 | |
| ajo | ack ykarel there I go | 13:20 |
| ajo | I'll tell you now it goes | 13:20 |
| ykarel | ack | 13:20 |
| ajo | ykarel: and thanks a lot | 13:20 |
| *** holser_ has quit IRC | 13:21 | |
| *** holser_ has joined #oooq | 13:22 | |
| weshay|ruck | quiquell|rover, panda join internal #sf-dfg | 13:22 |
| quiquell|rover | weshay|ruck: ack | 13:30 |
| *** jaosorior has quit IRC | 13:42 | |
| *** ykarel has quit IRC | 13:44 | |
| *** ykarel has joined #oooq | 13:44 | |
| *** Goneri has joined #oooq | 13:47 | |
| *** udesale has joined #oooq | 13:47 | |
| weshay|ruck | rfolco, ping | 13:52 |
| rfolco | weshay|ruck, hi | 13:52 |
| rfolco | weshay|ruck, just saw your ping on #tripleo, sorry. Want me to bj now? | 13:53 |
| *** udesale_ has joined #oooq | 13:55 | |
| *** vinaykns has joined #oooq | 13:55 | |
| *** udesale has quit IRC | 13:56 | |
| *** ykarel is now known as ykarel|afk | 13:56 | |
| rfolco | weshay|ruck, just ping when you are ready | 13:57 |
| rlandy | myoung; sorry was in meeting - to get back to you ... I was wrong stable/master and master are not at the same level ... https://code.engineering.redhat.com/gerrit/gitweb?p=tripleo-quickstart.git;a=shortlog | 14:02 |
| *** moguimar has quit IRC | 14:02 | |
| rlandy | myoung: ok - so we have to ff stable/master | 14:03 |
| rlandy | which iiuc, is what the bm jobs for master run on? | 14:03 |
| rlandy | tq/tqe related | 14:03 |
| ajo | ykarel|afk: still the same | 14:04 |
| ajo | _quickstart.log:fatal: [undercloud]: FAILED! => {"changed": false, "failed": true, "msg": "AnsibleUndefinedVariable: 'container_build_id' is undefined"} | 14:04 |
| *** Goneri has quit IRC | 14:04 | |
| ajo | bash ./quickstart.sh --clean --teardown all --release master-tripleo-ci --nodes ../3ctlr_1comp.yml --config ../pacemaker.yml 127.0.0.2 | 14:04 |
| ajo | I have also tried the stock pacemaker.yml and 3ctrl_1comp.yml | 14:05 |
| myoung | rlandy: double checking, but if they haven't been changed yes, all jobs on rhos-jenkins are running off stable branches | 14:06 |
| myoung | rhos-dev-jenkins* | 14:06 |
| myoung | rlandy: confirmed: http://git.app.eng.bos.redhat.com/git/tripleo-environments.git/tree/jenkins/jobs/tripleo-quickstart/tripleo-quickstart-baremetal.yml#n20 | 14:07 |
| rlandy | myoung: thanks | 14:08 |
| myoung | rlandy: IMHO https://code.engineering.redhat.com/gerrit/#/c/141919 should be merged as well, it's a time bomb that could impact BM RDO jobs | 14:08 |
| weshay|ruck | myoung, fyi.. tempest squad has 0 deliverables for shift on stack.. confirmed we'll have to have chandan standdown on his thoughts there | 14:08 |
| quiquell|rover | weshay|ruck: Do I have to assist to CI scalation ? | 14:09 |
| myoung | weshay|ruck: ack, was about to respond. also ack those questions were specifically to push back and/or have PM drive priority. chandankumar had identified work there and didn't seem to mesh with Plans / goals | 14:09 |
| rlandy | myoung: ok - I'll confirm with weshay|ruck and quiquell|rover when they are less busy - otherwise I just just ff master | 14:09 |
| rlandy | it's only two days back | 14:09 |
| myoung | rlandy: {nod} rebase away, you should have push rights there | 14:09 |
| weshay|ruck | quiquell|rover, ? | 14:10 |
| weshay|ruck | myoung, I'm getting input from the stakeholders now | 14:11 |
| myoung | weshay|ruck: cool cool | 14:11 |
| myoung | weshay|ruck: to be clear - I'm not advocating that we do work there :) | 14:11 |
| quiquell|rover | weshay|ruck: Didn't remember if I assisted as ruck or rover last time | 14:11 |
| weshay|ruck | myoung, aye cool | 14:11 |
| weshay|ruck | quiquell|rover, ugh.. too many $things.. what is the context | 14:12 |
| *** moguimar has joined #oooq | 14:12 | |
| weshay|ruck | quiquell|rover, for rdo sf migration? | 14:12 |
| quiquell|rover | weshay|ruck: ci scalation meeting | 14:12 |
| weshay|ruck | oh | 14:12 |
| quiquell|rover | who go there | 14:12 |
| weshay|ruck | quiquell|rover, https://specs.openstack.org/openstack/tripleo-specs/specs/policy/ci-team-structure.html | 14:13 |
| weshay|ruck | ruck - Attends the meetings where the team needs to be represented | 14:13 |
| * myoung needs a hash map lately to keep track of register load/spill. lambda on the hash bucket depth is thankfully not > 2 | 14:13 | |
| quiquell|rover | weshay|ruck: ack | 14:13 |
| * myoung goes back to sprint things and is dropping off IRC unless pinged. | 14:13 | |
| *** ykarel|afk is now known as ykarel | 14:13 | |
| quiquell|rover | rlandy: Hi, ykarel forced the changes needed for master in the spec files | 14:19 |
| *** Goneri has joined #oooq | 14:20 | |
| *** jfrancoa has joined #oooq | 14:20 | |
| quiquell|rover | ykarel: Do you if we have already the changes for master at DLRN ? | 14:22 |
| ykarel | quiquell|rover, yes jobs are running with that package, updated the card | 14:23 |
| *** marios has quit IRC | 14:24 | |
| *** zoli is now known as zoli|wfh | 14:24 | |
| *** zoli|wfh is now known as zoli | 14:24 | |
| quiquell|rover | myoung, rlandy, panda: unit tests for promoter https://review.rdoproject.org/r/#/c/14084/ | 14:28 |
| * myoung will look at ^^ later on today after being nose down on sprint stuff :) | 14:29 | |
| myoung | but quiquell|rover - cool! more UT == \o/ | 14:29 |
| quiquell|rover | myoung: :-), let's get this merged, I have the jobs for it too | 14:29 |
| quiquell|rover | leaving now | 14:30 |
| *** quiquell|rover is now known as quiquell|off | 14:30 | |
| weshay|ruck | hrm.. we lost dlrn votes on queens rdo phase 2 | 14:44 |
| hubbot | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224 | 14:48 |
| weshay|ruck | rasca, https://ci.centos.org/view/rdo/view/promotion-pipeline/job/rdo_trunk-promote-master-current-tripleo/ | 14:54 |
| *** marios has joined #oooq | 15:05 | |
| rlandy | panda: https://trello.com/c/aSKFUa9z/826-turn-off-the-vxlan-networking-creation-in-our-jobs- pls review DoD and description. we can move to in progress if suitable | 15:21 |
| rlandy | panda: also, are you working on https://trello.com/c/1gb8UuYM/825-create-the-inventory? or just commenting the card as UA? | 15:23 |
| rlandy | if not, I will pick it up | 15:23 |
| *** jfrancoa has quit IRC | 15:26 | |
| panda | rlandy: I've put a patch to discover the inventory format | 15:26 |
| panda | rlandy: and I had the result | 15:27 |
| rlandy | panda: yeah - ok, I see that, your name was just not on the card - but you have done the work so I'll pick up something else | 15:28 |
| *** gkadam has quit IRC | 15:30 | |
| *** hamzy has quit IRC | 15:31 | |
| *** jtomasek has quit IRC | 15:36 | |
| *** jtomasek has joined #oooq | 15:36 | |
| *** hamzy has joined #oooq | 15:38 | |
| *** links has quit IRC | 15:41 | |
| *** sanjayu__ has quit IRC | 15:42 | |
| *** Goneri has quit IRC | 15:43 | |
| *** Goneri has joined #oooq | 15:43 | |
| *** rlandy is now known as rlandy|mtg | 15:48 | |
| trown | panda: rlandy|mtg woot successful fs016 deploy using zuul-jobs multinode playbook for network setup | 16:17 |
| rlandy|mtg | very nice | 16:18 |
| rlandy|mtg | trown++ | 16:18 |
| hubbot | rlandy|mtg: trown's karma is now 1 | 16:18 |
| trown | I will work on etherpad instructions this afternoon | 16:18 |
| weshay|ruck | NICEEEEE | 16:24 |
| *** ykarel is now known as ykarel|away | 16:24 | |
| panda | trown: did you disable vxlan networking on undercloud-setup ? | 16:24 |
| panda | trown: or it's idempotent ? | 16:25 |
| trown | panda: no I disabled it via extra-vars in toci scripts | 16:25 |
| trown | panda: I updated https://trello.com/c/foDsucAu/827-create-instruction-to-use-the-libvirt-reproducer-to-test-zuulv3-migration-end-to-end as well with description and DoD | 16:26 |
| *** tesseract has quit IRC | 16:34 | |
| weshay|ruck | trown, maybe a distracting question.. but how far does that role goal to support more than n+1 nodes? | 16:44 |
| trown | weshay|ruck: doesnt move that forward at all | 16:44 |
| weshay|ruck | k | 16:44 |
| weshay|ruck | because we hardcode that in the repro | 16:45 |
| trown | it doesnt add any extra debt there though | 16:45 |
| trown | ya it is in nodepool-setup that we hardcode what nodes to provision | 16:45 |
| trown | this work is entirely after that | 16:45 |
| *** rlandy|mtg is now known as rlandy | 16:46 | |
| rlandy | arxcruz++ | 16:46 |
| hubbot | rlandy: arxcruz's karma is now 4 | 16:46 |
| rlandy | thanks for the tempest tour | 16:46 |
| *** kopecmartin has quit IRC | 16:47 | |
| hubbot | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224 | 16:48 |
| *** sshnaidm is now known as sshnaidm|off | 16:52 | |
| *** udesale_ has quit IRC | 16:55 | |
| *** trown is now known as trown|lunch | 16:58 | |
| *** atoth has quit IRC | 17:01 | |
| tosky | ykarel|away, arxcruz, quiquell|off is it possible that, applying that patch downstream to tempestconf, now we can't merge the patch anymore because the tripleo job fails trying during the DLRN rebuild to apply it? | 17:02 |
| tosky | weshay|ruck: ^^ | 17:03 |
| tosky | namely: http://logs.openstack.org/69/568869/37/check/tripleo-ci-centos-7-containers-multinode/639fe67/job-output.txt.gz#_2018-06-20_16_26_28_713292 | 17:03 |
| weshay|ruck | tosky, it means you have a spec file problem http://logs.openstack.org/69/568869/37/check/tripleo-ci-centos-7-containers-multinode/639fe67/logs/delorean_logs/9e/25/9e25fe1e3db769f31a7661a4c6bd73ed221fd74e_dev/build.log.txt.gz | 17:05 |
| weshay|ruck | tosky, is this the change that yatin put in to patch tempestconf? | 17:05 |
| tosky | weshay|ruck: it means that the patch is already applied and DLRN fails applying it | 17:06 |
| *** dtantsur is now known as dtantsur|afk | 17:06 | |
| tosky | at least that's how I read that error | 17:06 |
| weshay|ruck | tosky, does it conflict w/ https://github.com/rdo-packages/tempestconf-distgit/commit/44ad698b312101630482f31cbf10b8e7aa722a23 | 17:08 |
| tosky | weshay|ruck: it's the same patch! | 17:08 |
| tosky | we are trying to merge it | 17:08 |
| tosky | of course if DLRN tries to apply it again, it will fail | 17:08 |
| weshay|ruck | then you have to update the spec :) | 17:08 |
| tosky | I have to revert the patch | 17:08 |
| tosky | so if some promotion was happening, it won't happen | 17:09 |
| ykarel|away | tosky, https://review.rdoproject.org/r/#/c/14351/ | 17:09 |
| tosky | ykarel|away: yes, but I can't remove the -W | 17:09 |
| ykarel|away | tosky, done | 17:09 |
| tosky | I can add my +w | 17:09 |
| tosky | thanks, merging | 17:09 |
| weshay|ruck | nice | 17:09 |
| tosky | did this temporary workaround at least help any promotion? | 17:10 |
| rlandy | trown|lunch: panda: pls review comment in https://trello.com/c/QZmCayai/829-change-the-name-of-the-bridge-name-that-is-created-in-the-multinode-playbook-or-change-our-name-in-all-the-place-where-we-hardoc and let me know your thoughts | 17:10 |
| ykarel|away | tosky, yes some jobs passed with that | 17:10 |
| ykarel|away | so we are good | 17:10 |
| tosky | but now they will fail again | 17:10 |
| ykarel|away | tosky, yup but we can wait until upstream merges | 17:10 |
| tosky | can I immediately recheck the tempestconf patch, or should I wait a bit until the package is rebuilt? | 17:11 |
| tosky | should I check the queue on trunk.rdoproject.org? | 17:11 |
| ykarel|away | or if weshay|ruck says we can take the last hash and run failing jobs again, only one job failed | 17:12 |
| ykarel|away | tosky, you need to wait until the spec merges | 17:12 |
| weshay|ruck | it's pointless today due to zuulv2 -> v3 migration | 17:12 |
| ykarel|away | weshay|ruck, ack, so if that finishes that can be tried if tempestconf patch sees infra issues again | 17:12 |
| tosky | uh, what is pointless today? Check trunk.rdoproject.org? Take the last hash? | 17:13 |
| weshay|ruck | tosky, rdo sf is going down | 17:13 |
| arxcruz | quiquell|off: have the env up and running, i'll debug the tempest problem now | 17:13 |
| arxcruz | weshay|ruck: ^ the boot volume pattern on pike | 17:13 |
| tosky | weshay|ruck: so no more packages rebuilt today? | 17:13 |
| weshay|ruck | arxcruz, I'm testing our fix on that | 17:13 |
| weshay|ruck | tosky, don't think so | 17:13 |
| arxcruz | weshay|ruck: oh... (╯°□°）╯︵ ┻━┻ | 17:14 |
| weshay|ruck | arxcruz, https://review.openstack.org/#/c/576865/ | 17:14 |
| weshay|ruck | arxcruz, check those changes | 17:14 |
| *** atoth has joined #oooq | 17:14 | |
| arxcruz | ack | 17:23 |
| *** ykarel|away has quit IRC | 17:25 | |
| *** amoralej is now known as amoralej|off | 17:27 | |
| *** links has joined #oooq | 17:37 | |
| *** links has quit IRC | 17:39 | |
| *** atoth has quit IRC | 17:39 | |
| *** atoth has joined #oooq | 17:39 | |
| *** links has joined #oooq | 17:40 | |
| *** zoli is now known as zli|gone | 17:43 | |
| *** zli|gone is now known as zoli|gone | 17:43 | |
| *** zoli|gone is now known as zoli | 17:44 | |
| *** sanjayu__ has joined #oooq | 17:45 | |
| rlandy | weshay|ruck: green again on bm master ... https://rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/tripleo-quickstart-master-rdo_trunk-baremetal-hp_dl360_envE-single_nic_vlans/ | 17:48 |
| weshay|ruck | BOO YA | 17:49 |
| weshay|ruck | hating on jenkins atm | 17:52 |
| weshay|ruck | myoung, ping.. any idea why all of a sudden the dlrn-api-report-full is not running on queens? | 17:53 |
| myoung | weshay|ruck: link? | 17:53 |
| weshay|ruck | myoung, for example https://rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/periodic-queens-rdo_trunk-featureset020-1ctlr_1comp_64gb/93/console | 17:54 |
| myoung | which jobs? | 17:54 |
| * myoung looks | 17:54 | |
| myoung | holy slow jenkins batman | 17:54 |
| weshay|ruck | sucks | 17:54 |
| weshay|ruck | ya | 17:54 |
| myoung | weshay|ruck: huh...the post build task is still there... | 17:56 |
| * myoung looks more | 17:56 | |
| myoung | (thank heavens for the log server) | 17:58 |
| *** tcw has quit IRC | 17:58 | |
| myoung | weshay|ruck: wierd...the last thing to run is collect logs --> https://thirdparty.logs.rdoproject.org/jenkins-periodic-queens-rdo_trunk-featureset020-1ctlr_1comp_64gb-93/console.txt.gz#_2018-06-20_14_42_58_974 | 17:59 |
| weshay|ruck | ya | 17:59 |
| myoung | oh wait...it runs after collect logs | 18:00 |
| * myoung rolls his eyes and goes back to jenkins | 18:00 | |
| myoung | is it just queens? | 18:01 |
| weshay|ruck | myoung, not sure | 18:01 |
| myoung | weshay|ruck: ok...so...wierd. | 18:02 |
| myoung | last time it ran (the post build task) was | 18:02 |
| myoung | https://rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/periodic-queens-rdo_trunk-featureset020-1ctlr_1comp_64gb/88/console | 18:02 |
| myoung | #89, no dice --> https://rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/periodic-queens-rdo_trunk-featureset020-1ctlr_1comp_64gb/89/console | 18:02 |
| weshay|ruck | reviews on https://review.openstack.org/#/c/576851/ | 18:03 |
| weshay|ruck | https://review.openstack.org/#/c/576858 | 18:03 |
| weshay|ruck | to fix pike check/gate jobs | 18:04 |
| myoung | weshay|ruck: i don't see anything in JJB changes that would cause this...there is a config change there but not sure where is coming from | 18:04 |
| * myoung digs some more | 18:04 | |
| myoung | weshay|ruck: back in 1 hr, will look more. something started not working between #88 and #89 yesterday, from job config history there's a delta there that doesn't match up with JJB push job. not sure yet what's going on there | 18:10 |
| *** myoung is now known as myoung|lunch | 18:11 | |
| *** links has quit IRC | 18:11 | |
| *** links has joined #oooq | 18:12 | |
| *** holser_ has quit IRC | 18:14 | |
| *** links has quit IRC | 18:14 | |
| weshay|ruck | trown|lunch, rlandy https://review.openstack.org/#/c/576858 | 18:31 |
| *** yolanda_ has joined #oooq | 18:35 | |
| *** yolanda__ has joined #oooq | 18:39 | |
| *** yolanda has quit IRC | 18:39 | |
| *** yolanda_ has quit IRC | 18:40 | |
| *** atoth has quit IRC | 18:40 | |
| hubbot | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224 | 18:48 |
| *** myoung|lunch is now known as myoung | 19:06 | |
| *** yolanda_ has joined #oooq | 19:08 | |
| *** trown|lunch is now known as trown | 19:09 | |
| *** yolanda_ has quit IRC | 19:09 | |
| *** yolanda_ has joined #oooq | 19:10 | |
| *** yolanda__ has quit IRC | 19:12 | |
| *** holser_ has joined #oooq | 19:12 | |
| trown | rlandy: looks good, added a comment with a few other vars we need | 19:12 |
| *** sanjayu__ has quit IRC | 19:13 | |
| *** yolanda_ has quit IRC | 19:17 | |
| *** yolanda has joined #oooq | 19:19 | |
| myoung | weshay|ruck: looking at this more, both pike and master have not ran since yesterday, kicking one of them in isolation to debug this and see if it's more than just queens | 19:21 |
| myoung | weshay|ruck: double checked the configs, the post build task is simply not firing, it should | 19:21 |
| weshay|ruck | myoung, k.. not critical | 19:23 |
| weshay|ruck | 2018-06-20 19:15:57,173 10439 INFO promoter Promoting the container images for dlrn hash c98ac7866bebe26b5de35b2b2fe2a3d69a6a1d92 on queens to current-tripleo-rdo-internal | 19:23 |
| myoung | weshay|ruck: so it's not server wide, just that job. We have seen in the past that jenkins get's a little wierd/sticky when hand edits are made to job configs...hypothesis: the "apply/save" operation get's a little borked when multiple post build tasks are in play. rlandy and I have seen this before but never fully root caused it. Usually just deleting them and re-pushing the job configs resolves the issue | 19:24 |
| myoung | also confirmed that stuff like https://rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/tripleo-quickstart-master-rdo_trunk-baremetal-hp_dl360_envE-single_nic_vlans/145/consoleText (that ran ealier today) is working | 19:24 |
| myoung | weshay|ruck: the other wierd thing is a hand edit was made to the job (https://rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/periodic-queens-rdo_trunk-featureset020-1ctlr_1comp_64gb/jobConfigHistory/showDiffFiles?timestamp1=2018-06-18_21-37-08×tamp2=2018-06-19_14-34-26) that doesn't match up with the push job. going to just re-push and see what happens. | 19:25 |
| myoung | also ack not urgent ;) | 19:25 |
| weshay|ruck | myoung, I updated the timeout | 19:35 |
| myoung | ahhh ok that's consistent with what we've seen in the past | 19:38 |
| myoung | weshay|ruck: i repushed https://rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/tripleo-environments-jjb-push/85/console after nuking them by hand...might need to update that timeout in jjb | 19:39 |
| weshay|ruck | myoung, ya.. the odd thing was it was 360 in jjb | 19:44 |
| weshay|ruck | when I looked | 19:44 |
| weshay|ruck | afaict | 19:44 |
| weshay|ruck | %gatestatus | 19:45 |
| hubbot | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224 | 19:45 |
| weshay|ruck | hrm.. why does that still run /me looks | 19:45 |
| myoung | weshay|ruck: wierd. {shrug} https://rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/periodic-queens-rdo_trunk-featureset020-1ctlr_1comp_64gb/94 is running again for good measure on https://trunk.rdoproject.org/centos7-queens/c9/8a/c98ac7866bebe26b5de35b2b2fe2a3d69a6a1d92_1462a256 | 19:51 |
| weshay|ruck | myoung, that job got real slow really quickly | 19:52 |
| rlandy | myoung: weshay|ruck: I'm going to ff stable/queens | 19:58 |
| rlandy | since master is ok | 19:59 |
| *** Goneri has quit IRC | 19:59 | |
| weshay|ruck | rlandy, ah dam.. I meant to double check that | 20:00 |
| weshay|ruck | sorry | 20:00 |
| weshay|ruck | rlandy, thanks | 20:00 |
| rlandy | weshay|ruck: no worries - there is still pike to fix :) - testing queens now | 20:01 |
| * myoung thinks something is screwey with our jenkins master...again (overprovisioned?) | 20:04 | |
| myoung | i'm positive (like before) when I'm done logging an IT ticket the search or a bigger known prime number (or whatever else is going on in the lab...rendering the next pixar movie?) will be complete and we'll be down to sub-30ms response times lol | 20:08 |
| myoung | HA! and we're back to fast perf. | 20:08 |
| myoung | :) | 20:08 |
| myoung | (aside) fun fact about primes! This January M77232917 was discovered, the largest prime number known (unless we've found a new one). It's 2^277,232,917 - 1, and has 23,249,425 digits. https://www.mersenne.org/primes/press/M77232917.html | 20:12 |
| * myoung ducks | 20:12 | |
| weshay|ruck | trown, good idea / bad idea https://review.openstack.org/#/c/576858/ ? | 20:15 |
| rlandy | panda: trown: when we refer to upstream, does that include rdo cloud? I assume so with the zuul v3 migration | 20:16 |
| rlandy | ^^ wrt networking tasks | 20:16 |
| panda | trown: rlandy I got carried on after gathering inventory, and reparented the job to multinode. https://review.openstack.org/#/c/576879/ is running correctly (will maybe block at vxlan networking) | 20:17 |
| panda | rlandy: yes, but the priority is on upstream openstack, then we'll spread the changes everywhere | 20:17 |
| rlandy | https://review.openstack.org/#/c/576645 include settings for multinode and multinode rdocloud | 20:17 |
| *** holser_ has quit IRC | 20:18 | |
| rlandy | panda: so maybe the multinode-rdocloud change should be in a different review? | 20:18 |
| rlandy | can discuss tomorrow | 20:18 |
| panda | rlandy: it depends if we'll be able to test rdocloud zuulv3 changes this sprint | 20:18 |
| rlandy | panda: lastly, I updated the DoD and description in https://trello.com/c/1gb8UuYM/825-create-the-inventory-so-that-the-multinode-playbook-can-work-properly | 20:18 |
| rlandy | pls review | 20:19 |
| weshay|ruck | rlandy, trown https://review.openstack.org/#/c/576916/ | 20:19 |
| weshay|ruck | ah crud how is that on master | 20:19 |
| weshay|ruck | why does that say master branch??? | 20:21 |
| weshay|ruck | my local copy is on the queens branch | 20:21 |
| rlandy | : ${ZUUL_CHANGES:="openstack/tripleo-upgrade:master:refs/changes/08/576908/3^openstack/tripleo-upgrade:master:refs/changes/16/576916/1"} | 20:22 |
| panda | rlandy: trown great descriptions and DOD on all the cards, removed maturity tags except design | 20:26 |
| rlandy | panda: cool - thanks | 20:26 |
| weshay|ruck | ok.. that looks better | 20:27 |
| weshay|ruck | https://review.openstack.org/#/c/576970/ | 20:27 |
| weshay|ruck | not ready for review | 20:27 |
| weshay|ruck | should work though | 20:27 |
| rlandy | weshay|ruck: https://review.openstack.org/#/c/573819/? ready to merge this change? can w+1 | 20:28 |
| weshay|ruck | rlandy, done | 20:29 |
| * rlandy hopes that does not break anything | 20:29 | |
| rlandy | has potential to mess with multinode | 20:29 |
| rlandy | panda: sorry to ask this question again ... but I guess I didn't grok the answer the first time around ... are you working on the inventory - otherwise I;ll pick it up now | 20:33 |
| *** jtomasek has quit IRC | 20:46 | |
| panda | rlandy: you can pick it, I've really been playing with upstream CI and not the reproducer. | 20:47 |
| rlandy | panda: k - thanks | 20:48 |
| *** panda is now known as gozer | 20:48 | |
| gozer | tomorrow I'll be one with zuul | 20:48 |
| *** gozer is now known as panda|off | 20:48 | |
| myoung | in looking at what can be removed from toci for first iteration, unclear what we're doing with tebroker, or even how it works. do we have a doc/readme/anything on it's role moving forward? looking at what we need to preserve vs. obliterate in the new order of things | 20:48 |
| hubbot | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224 | 20:48 |
| rlandy | myoung: te-broker is something of tribal knowledge | 20:49 |
| rlandy | I can explain some stuff if that helps | 20:49 |
| rlandy | basically we can't expose tenant passwords | 20:49 |
| myoung | rlandy: aye in https://review.openstack.org/#/c/576834 sshnaidm|off said "it's mostly an oral tradition" | 20:49 |
| myoung | lol | 20:49 |
| rlandy | myoung: that actually very funny - I am sure weshay|ruck will enjoy that analogy as well | 20:50 |
| myoung | also came across this: https://review.rdoproject.org/r/#/c/12666 from sprint 10 - https://trello.com/c/I9P9CNih/21-restore-te-broker) don't know if I'm just looking in the wrong place | 20:50 |
| panda|off | myoung: it's also about ovb, so you can leave it for now, until we're sure rdo sf is migrated | 20:50 |
| myoung | but was abandoned | 20:51 |
| rlandy | myoung: as panda|off says, it's used to create the stacks needed for ovb jobs | 20:51 |
| rlandy | since we can't use ovb-stack-create in a public cloud | 20:51 |
| myoung | rlandy: ack, thx | 20:51 |
| myoung | i see | 20:51 |
| rlandy | ben nemec wrote it to enable ovb upstream | 20:51 |
| weshay|ruck | rlandy, the talmud of ci | 20:51 |
| myoung | does zuul v3 have some metaphor / analogy for ansible vault? | 20:51 |
| * myoung reads some more zuul docs | 20:52 | |
| panda|off | zuul vult | 20:52 |
| panda|off | yep, time to shut up and go. | 20:52 |
| weshay|ruck | oy zuul ga vult | 20:53 |
| myoung | holy moley. oldest talmud is 1342!!! http://daten.digitale-sammlungen.de/~db/bsb00003409/images/index.html | 20:53 |
| * rlandy is rolling !!! | 20:53 | |
| myoung | hrm...is there any reason we couldn't use https://docs.openstack.org/infra/manual/zuulv3.html#secret-variables (https://docs.openstack.org/infra/zuul/user/encryption.html) directly? | 20:55 |
| * myoung wonders if te-brokere can be added to the list of things that are going away | 20:56 | |
| rlandy | myoung; it's a whole discussion - probably not for this sprint | 20:56 |
| myoung | don't have the history to understand if this was already explored | 20:56 |
| myoung | k | 20:56 |
| myoung | gotcha | 20:56 |
| rlandy | I can take you through it some time | 20:56 |
| rlandy | ping me and I can fill you in | 20:57 |
| * myoung nods and removes te-broker from the list for s15 | 20:57 | |
| myoung | rlandy: ack will do, in 16+ | 20:57 |
| rlandy | but ... deciding what is out of sprint is not really my job - so whatever TC and UA decide | 20:58 |
| panda|off | I would leave it added, with low priority | 20:59 |
| panda|off | It may be time to discuss this even if we don't remove it now. | 21:00 |
| myoung | rlandy: let's chat tomorrow in scrum. I'm just going thru these scripts. I have a "Gleaming Hammer of Deletion" (w/ +20 vs bash) and everything is potentially a nail until proven otherwise. | 21:00 |
| rlandy | myoung: ack | 21:00 |
| *** trown is now known as trown|outtypewww | 21:01 | |
| weshay|ruck | rlandy, ping.. have you ever seen this amqp error? https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset002-master-upload/9c3c747/undercloud/home/jenkins/overcloud_prep_images.log.txt.gz | 21:02 |
| weshay|ruck | openstack overcloud node import instackenv.json | 21:02 |
| weshay|ruck | 2018-06-20 15:23:09 | MessageDeliveryFailure: Unable to connect to AMQP server on undercloud.internalapi.localdomain:5672 after None tries: (0, 0): (403) ACCESS_REFUSED - Login was refused using authentication mechanism AMQPLAIN. For details see the broker logfile. | 21:02 |
| rlandy | looking | 21:02 |
| rlandy | openstack overcloud node import instackenv.json is at the start of introspection work | 21:03 |
| rlandy | weshay|ruck: no - have not seen that one before ... | 21:04 |
| weshay|ruck | k.. thanks | 21:04 |
| rlandy | the instackenv.json is on the undercloud at that point | 21:04 |
| rlandy | weshay|ruck: do you see that error in multiple jobs? | 21:05 |
| rlandy | it basically creates the ironic nodes | 21:06 |
| rlandy | from the instackenv.json | 21:06 |
| *** myoung is now known as myoung|off | 21:07 | |
| weshay|ruck | rlandy, ovb | 21:08 |
| weshay|ruck | and only the upload job | 21:08 |
| rlandy | weird guess it's logging those nodes | 21:09 |
| vinaykns | I am working on installing the undercloud...it is taking long time to get done.! | 21:15 |
| vinaykns | it is getting stuck at /usr/local/share/ansible/roles/undercloud-deploy/tasks/install-undercloud.yml | 21:16 |
| *** Goneri has joined #oooq | 21:19 | |
| *** Goneri has quit IRC | 21:31 | |
| *** yolanda has quit IRC | 21:35 | |
| weshay|ruck | vinaykns, it should take about 30min | 21:38 |
| weshay|ruck | for the undercloud | 21:38 |
| weshay|ruck | vinaykns, although I think mine failed there too | 21:38 |
| * weshay|ruck looks | 21:38 | |
| weshay|ruck | ah.. no in prep-images | 21:39 |
| weshay|ruck | ah.. tripleo blesses us w/ so many ways to fail | 21:39 |
| *** yolanda has joined #oooq | 21:40 | |
| vinaykns | Oh..in that case i have to wait for some more time...i didn't know it to be normal. | 21:40 |
| hubbot | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224 | 22:48 |
| *** vvaldez has quit IRC | 22:51 | |
| *** vvaldez has joined #oooq | 22:51 | |
| *** jbadiapa has quit IRC | 22:55 | |
| rlandy | weshay|ruck: for bm, master and queens look ok. ff stable/pike will include a lot of changes (since 05/14). can try that tomorrow | 23:04 |
| *** vinaykns has quit IRC | 23:19 | |
| weshay|ruck | rlandy, k.. really I should be doing that stuff | 23:24 |
| *** rlandy has quit IRC | 23:26 | |
| *** tosky has quit IRC | 23:42 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!