| hubbot | adarazs: An error has occurred and has been logged. Please contact this bot's administrator for more information. | 00:11 |
|---|---|---|
| weshay | panda, you have creds to the te-broker/ | 00:23 |
| weshay | ? | 00:23 |
| hubbot | adarazs: An error has occurred and has been logged. Please contact this bot's administrator for more information. | 02:12 |
| *** dougbtv_ has quit IRC | 02:56 | |
| *** udesale has joined #oooq | 03:10 | |
| *** udesale has quit IRC | 03:19 | |
| *** udesale has joined #oooq | 03:31 | |
| *** links has joined #oooq | 03:37 | |
| hubbot | adarazs: An error has occurred and has been logged. Please contact this bot's administrator for more information. | 04:12 |
| *** udesale has quit IRC | 04:28 | |
| *** udesale has joined #oooq | 04:39 | |
| *** udesale has quit IRC | 04:49 | |
| *** udesale has joined #oooq | 04:50 | |
| *** marios has joined #oooq | 05:08 | |
| *** saneax-_-|AFK is now known as saneax | 05:10 | |
| *** ykarel has joined #oooq | 05:10 | |
| ykarel | OVB jobs failing with: +(/opt/stack/new/tripleo-ci/toci_gate_test.sh:252): sleep 1200 | 05:12 |
| ykarel | 2018-05-15 04:07:03,339 - testenv-client - INFO - Received job : Couldn't retrieve env | 05:12 |
| *** pgadiya has joined #oooq | 05:34 | |
| *** pgadiya has quit IRC | 05:34 | |
| *** udesale_ has joined #oooq | 05:36 | |
| *** udesale has quit IRC | 05:39 | |
| *** udesale_ has quit IRC | 05:40 | |
| *** udesale has joined #oooq | 05:40 | |
| *** jbadiapa has quit IRC | 06:00 | |
| hubbot | adarazs: An error has occurred and has been logged. Please contact this bot's administrator for more information. | 06:12 |
| *** udesale has quit IRC | 06:38 | |
| *** bogdando has joined #oooq | 06:40 | |
| *** udesale has joined #oooq | 06:40 | |
| *** brault has quit IRC | 07:01 | |
| *** brault has joined #oooq | 07:06 | |
| *** sshnaidm has joined #oooq | 07:09 | |
| *** tesseract has joined #oooq | 07:13 | |
| *** florianf has joined #oooq | 07:18 | |
| *** holser__ has joined #oooq | 07:20 | |
| *** sshnaidm is now known as sshnaidm|rover | 07:21 | |
| *** skramaja has joined #oooq | 07:23 | |
| *** kopecmartin has joined #oooq | 07:24 | |
| *** tosky has joined #oooq | 07:49 | |
| *** lucas-afk is now known as lucasagomes | 08:04 | |
| sshnaidm|rover | panda, trown|outtypewww please review reproducer patch change: https://review.openstack.org/#/c/565839/ | 08:09 |
| hubbot | FAILING CHECK JOBS on stable/ocata: gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-ocata, tripleo-ci-centos-7-undercloud-upgrades, tripleo-ci-centos-7-undercloud-oooq @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-3nodes-multinode, gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master, gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-ocata-branch, gate-tripleo-ci- (2 more messages) | 08:12 |
| *** holser__ has quit IRC | 08:13 | |
| *** holser__ has joined #oooq | 08:13 | |
| *** ykarel is now known as ykarel|lunch | 08:15 | |
| *** florianf has quit IRC | 08:40 | |
| *** florianf has joined #oooq | 08:44 | |
| *** gkadam has joined #oooq | 08:51 | |
| *** ykarel|lunch is now known as ykarel | 08:51 | |
| *** trown|outtypewww has quit IRC | 08:52 | |
| *** trown has joined #oooq | 08:53 | |
| *** jbadiapa has joined #oooq | 09:15 | |
| *** udesale_ has joined #oooq | 09:33 | |
| *** udesale has quit IRC | 09:34 | |
| *** jaosorior has quit IRC | 10:00 | |
| *** jaosorior has joined #oooq | 10:01 | |
| *** udesale_ has quit IRC | 10:01 | |
| *** udesale has joined #oooq | 10:03 | |
| *** zoli is now known as zoli|lunch | 10:05 | |
| *** tosky has quit IRC | 10:11 | |
| *** tosky has joined #oooq | 10:12 | |
| hubbot | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario002-multinode-oooq-container, tripleo-ci-centos-7-scenario003-multinode-oooq-container, tripleo-ci-centos-7-3nodes-multinode, tripleo-ci-centos-7-scenario001-multinode-oooq-container, gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens, gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-queens, tripleo-ci-centos-7-scenario004 (2 more messages) | 10:12 |
| *** zoli|lunch is now known as zoli | 10:24 | |
| *** tesseract is now known as info | 10:41 | |
| *** info is now known as tesseract | 10:41 | |
| *** zoli is now known as zoli|afk-tpb | 11:13 | |
| *** dtantsur|afk is now known as dtantsur | 11:14 | |
| *** atoth has joined #oooq | 11:20 | |
| *** holser__ has quit IRC | 11:32 | |
| *** holser___ has joined #oooq | 11:32 | |
| *** agopi has quit IRC | 11:46 | |
| *** zoli|afk-tpb is now known as zoli | 11:59 | |
| weshay | sshnaidm|rover, ykarel you guys cool w/ me promoting queens? | 12:01 |
| sshnaidm|rover | weshay, no | 12:01 |
| *** skramaja has quit IRC | 12:01 | |
| *** skramaja_ has joined #oooq | 12:01 | |
| weshay | why? | 12:01 |
| weshay | sshnaidm|rover, it's 100% down upstream | 12:01 |
| weshay | thanks for fixing the dns issue btw | 12:02 |
| sshnaidm|rover | weshay, a lot of issues actually, the latest was rdo cloud networking problems, and seems like it's resolved now | 12:02 |
| sshnaidm|rover | weshay, promotion jobs should start right now | 12:02 |
| weshay | sshnaidm|rover, rdo cloud 3rd party jobs are failing for all kinds of reasons | 12:03 |
| weshay | sshnaidm|rover, what do you mean? | 12:03 |
| sshnaidm|rover | weshay, rdo cloud network issues was last problem | 12:03 |
| weshay | sshnaidm|rover, k.. and it's fixed? | 12:03 |
| weshay | so it's worth just to wait | 12:04 |
| sshnaidm|rover | weshay, promotion jobs will start in a minute, they should be ok afaik | 12:05 |
| ykarel | yup we can wait for next run | 12:05 |
| weshay | sshnaidm|rover, ok.. ur the boss, will wait | 12:05 |
| ykarel | weshay, btw which hash u were promoting | 12:05 |
| sshnaidm|rover | weshay, fixed all known issues for me at least.. | 12:05 |
| weshay | sshnaidm|rover, where or how did you fix my dns f up? | 12:06 |
| sshnaidm|rover | weshay, I wrote in mail - removed it from network config | 12:07 |
| sshnaidm|rover | weshay, and then set up right dns in dns server | 12:07 |
| weshay | sshnaidm|rover, ok. why did you have to change the dns server? | 12:08 |
| sshnaidm|rover | weshay, if you want to advertise dns to all it's ok, but you need to hack dhclient configuration of dns server not to receive DNS from dhcp server | 12:08 |
| sshnaidm|rover | weshay, bluejeans please | 12:09 |
| weshay | sshnaidm|rover, ya.. we can change the sysconfig to be static | 12:09 |
| weshay | and chattr resolv.conf | 12:09 |
| hubbot | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario002-multinode-oooq-container, tripleo-ci-centos-7-scenario003-multinode-oooq-container, tripleo-ci-centos-7-3nodes-multinode, tripleo-ci-centos-7-scenario001-multinode-oooq-container, gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens, gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-queens, tripleo-ci-centos-7-scenario004 (2 more messages) | 12:12 |
| *** udesale_ has joined #oooq | 12:18 | |
| weshay | https://review.rdoproject.org/jenkins/job/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset010-queens/204/ | 12:18 |
| *** udesale has quit IRC | 12:19 | |
| weshay | ykarel, was master rdo1 impacted by rdo-cloud? | 12:20 |
| weshay | networking | 12:20 |
| ykarel | weshay, https://trello.com/c/1oq5yHGU | 12:28 |
| ykarel | 3 tempest tests failed in last two runs, | 12:28 |
| ykarel | i was not able to reproduce it locally, so rechecked again. I am not sure if failures were impacted by rdo-cloud | 12:29 |
| weshay | ykarel, k.. does weirdo have the notion of a skip list? | 12:31 |
| ykarel | weshay, kind of | 12:32 |
| ykarel | weshay, https://github.com/openstack/puppet-openstack-integration/blob/master/run_tests.sh#L322 | 12:32 |
| ykarel | but weirdo don't have, we need to do that in poi or packstack | 12:32 |
| *** dougbtv_ has joined #oooq | 12:32 | |
| *** rlandy has joined #oooq | 12:33 | |
| weshay | crud I think my networking is going down at my house lolz | 12:33 |
| weshay | can't get to https://ci.centos.org/job/weirdo-master-promote-packstack-scenario001/1244/console | 12:34 |
| ykarel | weshay, me to can't get to ^^ | 12:35 |
| weshay | ykarel, heh | 12:35 |
| weshay | k | 12:35 |
| weshay | ykarel, we'll be requiring a master promotion.. so if needed we need your card off the tripleo-ci board and onto the escalation board | 12:37 |
| weshay | cix board | 12:37 |
| ykarel | weshay, Ok | 12:38 |
| weshay | sshnaidm|rover, I thought we had fixed the introspection issue w/ an image rebuild https://review.rdoproject.org/jenkins/job/gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/13864/console | 12:38 |
| *** dougbtv_ has quit IRC | 12:40 | |
| weshay | sshnaidm|rover, fyi.. rdo2 master was and probably still is blocked on https://review.openstack.org/#/c/566129/ | 12:41 |
| weshay | can somebody please double check the te-broker log http://38.145.34.41/testenv-worker.log as it's not fully rendering for me | 12:45 |
| weshay | it does look like we have log rotate on the box | 12:45 |
| sshnaidm|rover | weshay, not sure about last image.. I'm looking at random job and it's stuck on image prepare: https://review.rdoproject.org/jenkins/job/gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/13868/console | 12:46 |
| weshay | sshnaidm|rover, ya.. I've seen that multiple times now | 12:46 |
| sshnaidm|rover | weshay, I configured logrotate today, but you have keys there and can log in | 12:46 |
| weshay | sshnaidm|rover, yup.. thank you sir.. I am in | 12:46 |
| ykarel | weshay, running phase1 packstack job again, last run failed due to Caused: java.io.IOException: Backing channel 'JNLP4-connect connection from n4-112.cloud.ci.centos.org/172.19.4.112:59406' is disconnected. | 12:50 |
| weshay | ykarel, k.. thanks | 12:53 |
| *** Goneri has joined #oooq | 13:06 | |
| myoung|ruck | o/ morning! | 13:09 |
| myoung|ruck | weshay, sshnaidm|rover yall still on bj? | 13:13 |
| rlandy | trown: hi - I removed the pylint stuff from https://review.openstack.org/#/c/568287/ - it's just pep8 now. If we want pylint later, the commit history is there to add it back. I removed the WIP as this shoudl pass now | 13:28 |
| *** dougbtv_ has joined #oooq | 13:28 | |
| weshay | I'm in a 1-1 | 13:30 |
| weshay | rlandy, https://review.rdoproject.org/r/#/c/13777/ | 13:30 |
| rlandy | weshay: is there something wrong with my comment there? | 13:31 |
| trown | rlandy: cool, fixing up the comments on the base patch from quiquell|off now, but I think it should be good to go after that | 13:31 |
| rlandy | trown: yep - working on testing those patches together today | 13:32 |
| rlandy | the tests run in the tox jobs | 13:32 |
| myoung|ruck | sshnaidm|rover: do you have a few to sync from last night? when I left for the evening we were about to get a queens periodic. don't want to duplicate efforts... | 13:34 |
| sshnaidm|rover | myoung|ruck, we didn't get promotions for queens because of dns issues, rdo cloud network issues, running promotion jobs atm | 13:35 |
| myoung|ruck | ssh ya saw they just kicked a little while ago | 13:36 |
| myoung|ruck | sshnaidm|rover: * | 13:36 |
| myoung|ruck | sshnaidm|rover: just saw your mail from last night | 13:36 |
| trown | panda: wrt https://review.openstack.org/567320 I was going to wire that in to the actual release dictionary in a later patch... would you rather me do it in the same patch? | 13:36 |
| trown | panda: no real reason not to... when I was working on that, base patch was not close to being done though | 13:37 |
| sshnaidm|rover | myoung|ruck, and we have problems with introspection again | 13:37 |
| myoung|ruck | arxcruz: were the dns changes you made yesterday to tempestmail instance doc'd anywhere? | 13:37 |
| arxcruz | myoung|ruck: i'm still working on the ansible playbook for that | 13:38 |
| arxcruz | myoung|ruck: give me some time :D working on nova stuff on tempestconf right now | 13:39 |
| myoung|ruck | arxcruz: cool, is that tracked by a card? just had a note to check back from yesterday...no rush | 13:39 |
| arxcruz | myoung|ruck: i can create a card, it's something out of sprint though | 13:39 |
| sshnaidm|rover | rlandy, did you see my comment in https://review.openstack.org/#/c/567060/ ? | 13:40 |
| myoung|ruck | arxcruz: stay on sprint...just a ping. | 13:40 |
| rlandy | sshnaidm|rover: about your parallel effort? yes | 13:41 |
| myoung|ruck | sshnaidm|rover: https://bugs.launchpad.net/tripleo/+bug/1771318 is concerning...that job totally needs to fail in that case | 13:41 |
| openstack | Launchpad bug 1771318 in tripleo "No nova-compute image in docker after promotion" [High,Triaged] - Assigned to Sagi (Sergey) Shnaidman (sshnaidm) | 13:41 |
| rlandy | really that is waiting on the panda's patch | 13:42 |
| rlandy | when the release file out is confirmed, will return to that | 13:42 |
| rlandy | but I will look at your work | 13:42 |
| panda | trown: just really wanted to know where this was going to be used, this or another patch is fine. | 13:43 |
| panda | trown: but I think you can ask requests directly to handle retries | 13:43 |
| trown | panda: hmm I looked and didnt see it in requests doc... suppose I could google it :P | 13:44 |
| rlandy | sshnaidm|rover: afaict, 567060 includes the basic elements of dumping env vars and the playbook commands that 565740 adds | 13:44 |
| rlandy | if you think functionality is missing, pls comment on 567060 and we will add it there | 13:44 |
| panda | trown: found this today http://www.coglib.com/~icordasc/blog/2014/12/retries-in-requests.html https://www.peterbe.com/plog/best-practice-with-retries-with-requests | 13:46 |
| rlandy | panda: re: https://review.openstack.org/#/c/566565/ | 13:47 |
| rlandy | can you complete that patch now? | 13:47 |
| trown | panda: there is no built in option for retries... looking at those blogs... http://docs.python-requests.org/en/master/api/ | 13:48 |
| trown | panda: hmm that involves totally rewriting what I did to use a persistent session | 13:50 |
| trown | panda: that seems all a bit more complicated than we need to download a 5 line file | 13:53 |
| ykarel | weshay, (packstack scenario 001)failed again, so good to move it to CIX | 13:53 |
| *** agopi has joined #oooq | 13:54 | |
| panda | trown: ... no control: difficulty X. just a bit more control: difficulty 10^X | 13:57 |
| panda | trown: leave the for then. | 13:58 |
| myoung|ruck | all: gentle reminder, #tripleo meeting starts now, CI community meeting starts immediatly after @https://bluejeans.com/7050859455, if anyone's chatty will have an open line in that room during #tripleo | 14:00 |
| panda | rlandy: the patch is missing and argument I can change. Then we need to bargain on what we want to do if the script fails | 14:00 |
| rlandy | panda: bargain? what are you offering??? | 14:01 |
| *** skramaja_ is now known as skramaja | 14:02 | |
| panda | my eternal and unlimited gratitude. And a pat in the back for free | 14:02 |
| rlandy | oh - I thought there was a possibility of a large donation to my swiss bank account in the deal but I'll settle eternal and unlimited gratitude for your | 14:03 |
| trown | panda: those blogs have a cool url i had not seen though ... that httpbin url | 14:03 |
| trown | panda: trying out my for loop with some of those | 14:03 |
| rlandy | panda: let's settle that so I can finish the work on 567060 | 14:03 |
| rlandy | let me know when you can chat | 14:03 |
| hubbot | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario002-multinode-oooq-container, tripleo-ci-centos-7-scenario003-multinode-oooq-container, tripleo-ci-centos-7-3nodes-multinode, tripleo-ci-centos-7-scenario001-multinode-oooq-container, gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens, gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-queens, tripleo-ci-centos-7-scenario004 (2 more messages) | 14:12 |
| *** gkadam has quit IRC | 14:12 | |
| *** gkadam has joined #oooq | 14:14 | |
| rlandy | panda: if the script fails, shouldn't we just bail out of the whole run? | 14:16 |
| rlandy | the script should be well tested itself | 14:17 |
| rlandy | so if it fails, the failure should be genuine | 14:17 |
| panda | rlandy: my concerd about this is that we are making all the jobs. ALL. OF. THEM. pass through a script that we implemented and tested in 10 days. Who's next ruck and rover ? | 14:19 |
| rlandy | idk - probably I am | 14:27 |
| panda | OK, if we inject the script and make it work for everything and something brakes, we will probably know for sure the week after this gets merged. At that point our decision affect ruck and rover time so they need to be informed. Or team should use some of their time to fix it. | 14:32 |
| myoung|ruck | all: community meeting starts shortly, folks popping in. | 14:32 |
| panda | hope for the best, prepare for the worst | 14:36 |
| *** apetrich has quit IRC | 14:36 | |
| rlandy | panda: what is your alternative suggestion? | 14:37 |
| *** trown is now known as trown|brb | 14:37 | |
| panda | rlandy: run the script only for the jobs in scope | 14:38 |
| rlandy | panda: hmmm - but we nee release output now for all jobs | 14:38 |
| rlandy | need | 14:39 |
| panda | rlandy: it's already defined as empty if the script output doesn't exist | 14:40 |
| *** tosky has quit IRC | 14:40 | |
| rlandy | panda: the biggest thing failing right now is the undefined script | 14:40 |
| panda | rlandy: and if the key is not found,then we use the default | 14:40 |
| rlandy | and path to output | 14:40 |
| rlandy | so ok to kick the script only with a few jobs | 14:40 |
| rlandy | but we need to lock down name of script and output | 14:41 |
| rlandy | so we can finish integration work | 14:41 |
| *** trown|brb is now known as trown | 14:41 | |
| panda | rlandy: I'm ok to put it for all the jobs too, if we prepare a contingency plan IF something goes wrong. | 14:41 |
| rlandy | fyi ... I am out on monday | 14:41 |
| rlandy | panda: I thought our original plan was all the jobs but I agree to a switch to remove it if things go wrong | 14:42 |
| *** tosky has joined #oooq | 14:44 | |
| *** apetrich has joined #oooq | 14:48 | |
| sshnaidm|rover | myoung|ruck, weshay master containers build failed: http://paste.openstack.org/show/721017/ | 14:58 |
| ykarel | :( | 15:00 |
| *** ykarel is now known as ykarel|afk | 15:02 | |
| *** apetrich has quit IRC | 15:06 | |
| panda | rlandy: sorry I had 4 meeetings in a row. I'm available to talk if you want | 15:12 |
| panda | meeeeting | 15:12 |
| * panda gets some more eeee | 15:13 | |
| rlandy | panda: k - in a few minutes - just fixing a review | 15:14 |
| *** apetrich has joined #oooq | 15:15 | |
| sshnaidm|rover | myoung|ruck, seems like dashboard doesn't work: https://dashboards.rdoproject.org/master | 15:15 |
| myoung|ruck | sshnaidm|rover: guessing the most recent rdocloud hiccups might have caused a reboot...looks like the feeder script isn't running | 15:16 |
| *** skramaja has quit IRC | 15:18 | |
| *** links has quit IRC | 15:21 | |
| *** udesale_ has quit IRC | 15:25 | |
| weshay | trown, panda rlandy https://review.rdoproject.org/r/#/c/13790/1 | 15:29 |
| weshay | can we get some eyes on that.. worked for me on cli | 15:29 |
| weshay | thanks sshnaidm|rover | 15:30 |
| sshnaidm|rover | weshay, myoung|ruck, bbl, gonna pray for queens | 15:30 |
| rlandy | panda: ready to meet when you are | 15:31 |
| myoung|ruck | sshnaidm|rover: looks great, in meeting and haven't had a chance to try / verify it yet | 15:31 |
| *** sshnaidm|rover is now known as sshnaidm|bbl | 15:31 | |
| bogdando | PTAL folks https://review.openstack.org/#/c/568326 | 15:31 |
| weshay | baruch beashvile ha malca | 15:32 |
| bogdando | not sure why my dependency graph is broken | 15:32 |
| bogdando | ;( | 15:32 |
| weshay | bogdando, was about to ping you | 15:32 |
| weshay | bogdando, do you have 5 min? | 15:32 |
| bogdando | weshay: sure thing! | 15:32 |
| *** saneax is now known as saneax-_-|AFK | 15:33 | |
| bogdando | btw, an update http://lists.openstack.org/pipermail/openstack-dev/2018-May/130513.html | 15:33 |
| bogdando | just sent | 15:33 |
| weshay | https://bluejeans.com/whayutin | 15:33 |
| weshay | k | 15:33 |
| weshay | I think I get it and like it.. just want to be sure | 15:33 |
| panda | rlandy: bj/gcerami ? | 15:36 |
| rlandy | panda: joining | 15:38 |
| *** ykarel|afk is now known as ykarel|away | 15:39 | |
| panda | weshay: myoung|ruck in what occasion we upload the container but they are not there ? | 15:40 |
| weshay | panda, last promotion | 15:44 |
| *** ykarel|away has quit IRC | 15:44 | |
| myoung|ruck | panda: last night | 15:45 |
| EmilienM | weshay texted me, he had power outage | 15:52 |
| EmilienM | he's relocating | 15:52 |
| EmilienM | time to take the day off folks!!! wes is gone !! | 15:52 |
| *** sanjay__u has quit IRC | 15:59 | |
| *** bogdando has quit IRC | 16:03 | |
| panda | myoung|ruck: and what was the reason ? | 16:03 |
| panda | oh, wes is not here /nick panda|party | 16:04 |
| myoung|ruck | panda: https://bugs.launchpad.net/tripleo/+bug/1771318 is tracking, I don't know that we've got root cause, but overnight sshnaidm|bbl confirmed it and manually pushed the nova container for last promotion. | 16:05 |
| openstack | Launchpad bug 1771318 in tripleo "No nova-compute image in docker after promotion" [High,Triaged] - Assigned to Sagi (Sergey) Shnaidman (sshnaidm) | 16:05 |
| *** trown is now known as trown|lunch | 16:08 | |
| *** lucasagomes is now known as lucas-afk | 16:09 | |
| panda | myoung|ruck: ok thanks | 16:12 |
| hubbot | FAILING CHECK JOBS on stable/ocata: gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-ocata, tripleo-ci-centos-7-undercloud-upgrades @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-3nodes-multinode, gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master, gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-ocata-branch, gate-tripleo-ci-centos-7-ovb-3ctlr_1comp- (2 more messages) | 16:12 |
| weshay | panda, back had a power outtage | 16:14 |
| weshay | panda, back had a power outtage | 16:14 |
| panda | weshay: oh, no /nick panda|noparty | 16:18 |
| panda | rlandy: https://review.openstack.org/566565 latest patchset should be final | 16:18 |
| weshay | panda, what's up am I missing something? | 16:18 |
| panda | trown|lunch: we can talk about this, since weshay is on PTO for the next two weeks, and having a plan in place means anyway we'll have to wait for fixes to merge, I would prefer to confine the use of the script to the 2 jobs in scope + 2 other jobs to test the backwards compatibility of the script. So we can make small but useful steps at a time | 16:20 |
| weshay | panda, starting this coming wednesday | 16:20 |
| weshay | sorry next wednesday | 16:20 |
| panda | weshay: nothing, just following up on EmilienM joke | 16:20 |
| weshay | end of sprint | 16:21 |
| weshay | EmilienM, is not funny EVER | 16:21 |
| *** tesseract has quit IRC | 16:21 | |
| rlandy | panda: k - thanks - will take a look in a bit | 16:21 |
| EmilienM | weshay: go pay your electricity bills dude | 16:23 |
| weshay | I'm broke as a joke | 16:23 |
| weshay | baller on a budget | 16:24 |
| *** zoli is now known as zoli|gone | 16:26 | |
| *** zoli|gone is now known as zoli | 16:26 | |
| *** panda is now known as panda|bbl | 16:29 | |
| *** jaosorior has quit IRC | 16:31 | |
| *** dtantsur is now known as dtantsur|afk | 16:33 | |
| weshay | sshnaidm|bbl, myoung|ruck probably going to promote queens.. fs10 timed out.. waiting on 20 and 37 neither is critical | 16:36 |
| *** marios has quit IRC | 16:41 | |
| *** gkadam has quit IRC | 17:07 | |
| *** yolanda_ has quit IRC | 17:09 | |
| rlandy | ha - kicking tox on select file - finally!! | 17:16 |
| *** kopecmartin has quit IRC | 17:17 | |
| *** yolanda_ has joined #oooq | 17:20 | |
| *** trown|lunch is now known as trown | 17:22 | |
| *** tosky has quit IRC | 17:33 | |
| *** tosky has joined #oooq | 17:38 | |
| myoung|ruck | weshay: (back) and ack | 17:39 |
| myoung|ruck | sshnaidm|bbl: when do you want to trade |ruck for |rover? | 17:40 |
| * myoung|ruck wants to work on some bugz | 17:40 | |
| *** holser__ has joined #oooq | 17:51 | |
| *** ssbarnea has joined #oooq | 17:51 | |
| *** holser___ has quit IRC | 17:51 | |
| *** holser__ has quit IRC | 17:56 | |
| *** links has joined #oooq | 18:01 | |
| myoung|ruck | weshay, sshnaidm|bbl, hrm...(from queens.log) | 18:11 |
| myoung|ruck | 2018-05-15 09:27:32,098 25359 INFO promoter FINISHED promotion process | 18:11 |
| myoung|ruck | 2018-05-15 18:00:14,808 27234 DEBUG promoter No other promoters running. Acquired lock and continuing with promot | 18:11 |
| myoung|ruck | was down for a bit | 18:11 |
| *** dougbtv_ has quit IRC | 18:11 | |
| *** ykarel|away has joined #oooq | 18:12 | |
| hubbot | FAILING CHECK JOBS on stable/ocata: gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-ocata, tripleo-ci-centos-7-undercloud-upgrades @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-3nodes-multinode, gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master, gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-ocata-branch, gate-tripleo-ci-centos-7-ovb-3ctlr_1comp- (2 more messages) | 18:13 |
| weshay | sigh | 18:13 |
| myoung|ruck | in the wee hours | 18:14 |
| *** florianf has quit IRC | 18:16 | |
| rlandy | weshay: finally - this works https://review.openstack.org/#/c/568287/10/zuul.d/layout.yaml | 18:17 |
| rlandy | selected kicking of zuul tests | 18:17 |
| weshay | nice rlandy | 18:19 |
| weshay | rlandy, it's a little confusing to me why that's in layout.yml | 18:20 |
| weshay | when I look at other config... | 18:20 |
| weshay | but who said that I had to understand it | 18:20 |
| weshay | rlandy, trown can you guys see if you can get fs50 to work w/ the release config.. keep fielding questions from E | 18:21 |
| weshay | https://bugs.launchpad.net/tripleo/+bug/1768997 | 18:21 |
| openstack | Launchpad bug 1768997 in tripleo "containerized-undercloud-upgrades: dlrn tasks don't run" [Critical,Triaged] | 18:21 |
| weshay | not sure where you guys are at w/ the flow | 18:21 |
| *** ykarel|away has quit IRC | 18:22 | |
| weshay | so maybe we can get some updates to the bug at the min.. and look at getting it fixed in the longer run | 18:22 |
| rlandy | same as https://review.openstack.org/#/c/563526/6/zuul.d/layout.yaml | 18:23 |
| weshay | myoung|ruck, make sure that queens promotion goes through please :) | 18:23 |
| weshay | rlandy, ah k | 18:23 |
| rlandy | weshay: yep - just updating the dry-run review with panad's updated | 18:23 |
| * weshay relocates, power should be back on | 18:23 | |
| weshay | rlandy, thanks | 18:23 |
| rlandy | then will try it | 18:23 |
| myoung|ruck | weshay: aye looking at it now | 18:25 |
| myoung|ruck | weshay: /me is plowing thru http://38.145.34.55/queens.log as i type this | 18:26 |
| *** holser__ has joined #oooq | 18:28 | |
| *** holser__ has quit IRC | 18:32 | |
| rlandy | panda: ok - rebased my patch on 566565 latest - updating to only copy release file if it exists | 18:33 |
| *** agopi has quit IRC | 18:41 | |
| myoung|ruck | weshay: something funky going on...promoter isn't seeing your patch (https://github.com/rdo-infra/ci-config/commit/fe555e06c88911e810948540aa09b7b21ff7efe2) | 18:49 |
| * myoung|ruck looks deeper, captured https://review.rdoproject.org/etherpad/p/ruckrover-sprint13 @ L85 | 18:49 | |
| weshay | myoung|ruck, ya.. I saw that it still had fs10 in criteria | 18:50 |
| * myoung|ruck is on the box | 18:50 | |
| myoung|ruck | weshay: ci-config wasn't up to date...was still @ 11 may | 18:53 |
| weshay | k | 18:53 |
| myoung|ruck | next time around should pick up new criteria. exploring why...should be updated all the times | 18:53 |
| weshay | rlandy, trown reminder to pick up this bug and comment https://bugs.launchpad.net/tripleo/+bug/1768997 :) | 18:54 |
| openstack | Launchpad bug 1768997 in tripleo "containerized-undercloud-upgrades: dlrn tasks don't run" [Critical,Triaged] | 18:54 |
| myoung|ruck | weshay: also still had our edits from friday...stashed them | 18:54 |
| weshay | ah.. that's probably why | 18:54 |
| myoung|ruck | yeah...guessing is silently failing to pull | 18:55 |
| myoung|ruck | I'm writing a RFE (in my spare time HAH) to track improvements. when I flip to |rover I want to address it | 18:55 |
| myoung|ruck | (this and other stuff) | 18:55 |
| myoung|ruck | sshnaidm|bbl: ^^ | 18:55 |
| myoung|ruck | weshay killing the current promotoer script and rekicking. why wait eh? | 18:56 |
| weshay | +1 | 18:56 |
| rlandy | weshay: what we are currently working on is the releases script | 18:57 |
| rlandy | whether it fixes picking up the right release per zuul change, not sure | 18:58 |
| rlandy | we can try it | 18:58 |
| weshay | rlandy, anything that would drive that forward.. all that needs to happen really.. is that the right release is passed to the update playbook | 18:58 |
| rlandy | what happened to the fs naming :( | 18:58 |
| weshay | and the upgrade role uses the passed release.. and not a calculated one | 18:58 |
| weshay | rlandy, with regards to? fs | 18:58 |
| rlandy | tripleo-ci-centos-7-containerized-undercloud-upgrades | 18:59 |
| weshay | rlandy, it's an upstream job | 19:00 |
| weshay | rlandy, the upstream jobs still do not use fs$$$ | 19:00 |
| rlandy | ok - let's see what kicks with current changes | 19:01 |
| weshay | rlandy, note the upgrade role uses a var != release | 19:01 |
| weshay | I think | 19:02 |
| rlandy | weshay: k - just in the middle of commit a fix - will look in a few | 19:02 |
| weshay | %gatestatus | 19:09 |
| hubbot | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario002-multinode-oooq-container, tripleo-ci-centos-7-scenario003-multinode-oooq-container, tripleo-ci-centos-7-containers-multinode, tripleo-ci-centos-7-3nodes-multinode, tripleo-ci-centos-7-scenario001-multinode-oooq-container, tripleo-ci-centos-7-scenario004-multinode-oooq-container, gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-queens, (2 more messages) | 19:09 |
| *** agopi has joined #oooq | 19:10 | |
| *** holser__ has joined #oooq | 19:11 | |
| weshay | myoung|ruck, fyi https://bugs.launchpad.net/tripleo/+bug/1771414 | 19:16 |
| openstack | Launchpad bug 1771414 in tripleo "RFE: install ovb-tenant-cleanup.sh script in tripleo infra tenant and execute via cron" [High,Triaged] | 19:16 |
| weshay | sshnaidm|bbl, ^ | 19:16 |
| *** holser__ has quit IRC | 19:16 | |
| myoung|ruck | weshay: ahh thanks, crossing that one off my list | 19:16 |
| myoung|ruck | (to log) | 19:16 |
| myoung|ruck | weshay: so far container download seems to be moving right along | 19:17 |
| myoung|ruck | We've pulled *** 42 *** so far in just a few mins | 19:18 |
| weshay | myoung|ruck, please update https://bugs.launchpad.net/tripleo/+bug/1770972 | 19:19 |
| openstack | Launchpad bug 1770972 in tripleo "CI: Images introspection fails in OVB jobs" [Critical,Triaged] - Assigned to Derek Higgins (derekh) | 19:19 |
| weshay | https://logs.rdoproject.org/65/566565/7/openstack-check/gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/Z14cd35a437b546d0a25ba0721c29b5d0/undercloud/home/jenkins/overcloud_prep_images.log.txt.gz | 19:19 |
| *** links has quit IRC | 19:20 | |
| *** jaosorior has joined #oooq | 19:21 | |
| rlandy | weshay: ok - looking into that bug now with our current sprint stuff | 19:24 |
| weshay | rlandy, thanks | 19:25 |
| weshay | EmilienM, ^ | 19:25 |
| EmilienM | hi | 19:28 |
| EmilienM | context? -sorry- | 19:28 |
| weshay | EmilienM, your fs50 dlrn patch bug | 19:38 |
| EmilienM | ah nice ! | 19:47 |
| *** jtomasek has quit IRC | 19:51 | |
| hubbot | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario002-multinode-oooq-container, tripleo-ci-centos-7-scenario003-multinode-oooq-container, tripleo-ci-centos-7-containers-multinode, tripleo-ci-centos-7-3nodes-multinode, tripleo-ci-centos-7-scenario001-multinode-oooq-container, tripleo-ci-centos-7-scenario004-multinode-oooq-container, gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-queens, (2 more messages) | 20:13 |
| *** agopi has quit IRC | 20:25 | |
| *** holser__ has joined #oooq | 20:27 | |
| *** sshnaidm|bbl is now known as sshnaidm|rover | 20:42 | |
| weshay | myoung|ruck, k | 21:00 |
| weshay | do you have bugs on scen001/002 | 21:00 |
| weshay | myoung|ruck, for master | 21:01 |
| weshay | looks like the same issue | 21:01 |
| myoung|ruck | introspection ovb...recreate is running now | 21:02 |
| weshay | myoung|ruck, cool.. how about scen001/002? | 21:04 |
| myoung|ruck | walking thru sova | 21:04 |
| myoung|ruck | now | 21:04 |
| myoung|ruck | scen001 has had 4 fails today, 2 are the ovb issue, looking at the other 2 | 21:05 |
| *** trown is now known as trown|outtypewww | 21:05 | |
| myoung|ruck | scen002 is on deck | 21:05 |
| weshay | scen001/002 are both failing on the same issue | 21:06 |
| myoung|ruck | gah...never mind. details in etherpad... | 21:06 |
| weshay | myoung|ruck, also failing in the gate.. so + alert | 21:06 |
| sshnaidm|rover | weshay, myoung|ruck so what is with queens? | 21:06 |
| myoung|ruck | it's about to promote | 21:07 |
| weshay | sshnaidm|rover, almost done promoting | 21:07 |
| weshay | myoung|ruck, what line | 21:07 |
| myoung|ruck | it's pushed everything to docker | 21:07 |
| myoung|ruck | sshnaidm|rover: cleaning up and about to push symlinks via dlrn | 21:07 |
| weshay | myoung|ruck, what line in the etherpad | 21:07 |
| myoung|ruck | weshay: you are generally about 20 seconds ahead of me. moment. | 21:08 |
| myoung|ruck | or hours. i dunno | 21:08 |
| myoung|ruck | 31 | 21:08 |
| weshay | line 31? | 21:10 |
| weshay | third party? | 21:10 |
| weshay | weird.. myoung|ruck I see http://logs.openstack.org/45/560445/29/check/tripleo-ci-centos-7-scenario002-multinode-oooq-container/6c9f8c0/logs/undercloud/home/zuul/overcloud_prep_containers.log.txt.gz#_2018-05-15_17_11_35 | 21:16 |
| myoung|ruck | weshay: yes...that's the missing nova container i had thought we u/l last night | 21:17 |
| myoung|ruck | worming thru logs now here https://review.rdoproject.org/etherpad/p/ruckrover-sprint13 | 21:17 |
| myoung|ruck | @31 | 21:17 |
| myoung|ruck | i need a few mins | 21:19 |
| *** holser__ has quit IRC | 21:21 | |
| myoung|ruck | weshay: ugh | 21:49 |
| myoung|ruck | queens promotion just failed | 21:49 |
| myoung|ruck | failed: [localhost] (item=[u'nova-placement-api', u'current-tripleo']) => {"changed": false, "item": ["nova-placement-api", "current-tripleo"], "msg": "Error removing image docker.io/tripleoqueens/centos-binary-nova-placement-api:current-tripleo - UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)"} | 21:49 |
| myoung|ruck | it actually failed while cleaning up | 21:49 |
| myoung|ruck | which borked out the calling script...which failed the promoter and it gave up, instead of continuing with promotion workflow | 21:50 |
| myoung|ruck | 2018-05-15 21:41:28,797 25051 ERROR promoter Command '['ansible-playbook', '/home/centos/ci-config/ci-scripts/container-push/container-push.yml']' returned non-zero exit status 2 | 21:50 |
| myoung|ruck | Traceback (most recent call last): | 21:50 |
| myoung|ruck | File "/home/centos/ci-config/ci-scripts/dlrnapi_promoter/dlrnapi_promoter.py", line 142, in tag_containers | 21:50 |
| myoung|ruck | env=env, stderr=subprocess.STDOUT).split("\n") | 21:50 |
| myoung|ruck | File "/usr/lib64/python2.7/subprocess.py", line 575, in check_output | 21:50 |
| myoung|ruck | raise CalledProcessError(retcode, cmd, output=output) | 21:50 |
| myoung|ruck | it should loop around and try again. I don't have root cause for why it failed to remove...that's a new one | 21:51 |
| weshay | fak | 21:51 |
| myoung|ruck | I can look later too...i'm beyond late at this point for "adulting have-to" | 21:51 |
| myoung|ruck | one of the RFE's im thinking of for the promoter is to leave a few promotions worth of images in the local registry, so in this specific case (if...err..WHEN...it happens again) we don't pay the tax for pulling / pushing all these image layers again | 21:52 |
| myoung|ruck | it'll just loop around and try again for the delta | 21:52 |
| myoung|ruck | panda|bbl: ^^ have you seen this before (random failure to remove an image from promoter's local registry? | 21:53 |
| * myoung|ruck transmogrifies himself into a taxi and will return later | 21:53 | |
| *** myoung|ruck is now known as myoung|ruck|afk | 21:53 | |
| *** Goneri has quit IRC | 21:53 | |
| * myoung|ruck|afk guesses this could have been an RDO cloud hiccup on i/o access to underlying FS? | 21:54 | |
| sshnaidm|rover | second time it should pass | 21:56 |
| myoung|ruck|afk | weshay, panda|bbl the really wierd thing is that the image is actually removed from the local registry...it's gonzo | 21:56 |
| myoung|ruck|afk | unless someone else just reached in and nuked it | 21:56 |
| myoung|ruck|afk | nope...i'm the only one in there. | 21:56 |
| myoung|ruck|afk | oh wait... sshnaidm|rover is entos pts/0 2018-05-15 09:29 (bzq-79-181-125-206.red.bezeqint.net) you? | 21:57 |
| myoung|ruck|afk | sshnaidm|rover: did you remove? | 21:58 |
| sshnaidm|rover | myoung|ruck|afk, yes | 21:58 |
| sshnaidm|rover | myoung|ruck|afk, remove what? | 21:58 |
| myoung|ruck|afk | nova-placement-api image (see above). will be back later to check on queens take 2 | 21:58 |
| * sshnaidm|rover did nothing | 21:59 | |
| myoung|ruck|afk | sshnaidm|rover: ack, just checking...strange phantom image :) | 22:00 |
| hubbot | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario002-multinode-oooq-container, tripleo-ci-centos-7-scenario003-multinode-oooq-container, tripleo-ci-centos-7-containers-multinode, tripleo-ci-centos-7-3nodes-multinode, tripleo-ci-centos-7-scenario001-multinode-oooq-container, tripleo-ci-centos-7-scenario004-multinode-oooq-container, gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-queens, (2 more messages) | 22:13 |
| *** saneax-_-|AFK is now known as saneax | 23:01 | |
| rlandy | weshay: still around? | 23:22 |
| rlandy | weshay: so you can see the args passed ... http://logs.openstack.org/60/567060/17/check/tripleo-ci-centos-7-undercloud-upgrades/ec045e7/logs/playbook_executions.txt.gz | 23:23 |
| rlandy | we are passing the same release file to all | 23:23 |
| rlandy | for fs047 we should be running the release script | 23:23 |
| rlandy | would have to try a review with a couple other combined | 23:25 |
| *** tosky has quit IRC | 23:26 | |
| rlandy | added a test review - will watch jobs | 23:45 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!