| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-containerized-undercloud-upgrades, tripleo-ci-centos-7-standalone-upgrade @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens-branch, tripleo-ci-centos-7-scenario007-multinode-oooq-container, tripleo-ci-centos-7-standalone-upgrade @ https://review.openstack.org/560445, stable/ocata: tripleo-ci-centos-7-ovb- (1 more message) | 00:14 |
|---|---|---|
| *** rfolco has joined #oooq | 01:31 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-containerized-undercloud-upgrades, tripleo-ci-centos-7-standalone-upgrade @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens-branch, tripleo-ci-centos-7-scenario007-multinode-oooq-container, tripleo-ci-centos-7-standalone-upgrade @ https://review.openstack.org/560445, stable/ocata: tripleo-ci-centos-7-ovb- (1 more message) | 02:14 |
| *** rfolco has quit IRC | 02:59 | |
| *** ykarel has joined #oooq | 03:36 | |
| *** udesale has joined #oooq | 04:08 | |
| *** saneax has joined #oooq | 04:13 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-containerized-undercloud-upgrades, tripleo-ci-centos-7-scenario007-multinode-oooq-container, tripleo-ci-centos-7-standalone-upgrade @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-scenario004-multinode-oooq-container @ https://review.openstack.org/602248, master: tripleo-ci-centos-7-standalone-upgrade @ (1 more message) | 04:14 |
| *** skramaja has joined #oooq | 04:14 | |
| *** ykarel has quit IRC | 04:33 | |
| *** ykarel has joined #oooq | 04:49 | |
| *** ykarel has quit IRC | 04:53 | |
| *** ratailor has joined #oooq | 06:11 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-containerized-undercloud-upgrades, tripleo-ci-centos-7-scenario007-multinode-oooq-container, tripleo-ci-centos-7-standalone-upgrade @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-scenario004-multinode-oooq-container @ https://review.openstack.org/602248, master: tripleo-ci-centos-7-standalone-upgrade @ (1 more message) | 06:14 |
| *** ykarel has joined #oooq | 06:28 | |
| *** jbadiapa has joined #oooq | 06:57 | |
| *** quiquell|off is now known as quiquell | 07:06 | |
| *** jfrancoa has joined #oooq | 07:09 | |
| *** ykarel_ has joined #oooq | 07:15 | |
| *** ykarel has quit IRC | 07:17 | |
| *** ykarel__ has joined #oooq | 07:20 | |
| *** ykarel_ has quit IRC | 07:23 | |
| *** ykarel_ has joined #oooq | 07:24 | |
| *** ykarel__ has quit IRC | 07:27 | |
| *** ykarel_ is now known as ykarel|lunch | 07:31 | |
| *** ykarel_ has joined #oooq | 07:35 | |
| *** ykarel|lunch has quit IRC | 07:38 | |
| *** brault has joined #oooq | 07:41 | |
| *** brault has quit IRC | 07:43 | |
| *** arxcruz|next_yr is now known as arxcruz | 07:47 | |
| arxcruz | happy new year people! :D | 07:48 |
| *** matbu has joined #oooq | 07:58 | |
| quiquell | arxcruz: o/ happy new year mate | 08:00 |
| arxcruz | quiquell: feliz cumpleaños | 08:00 |
| arxcruz | hmm doesn't seems correct | 08:00 |
| arxcruz | is it navidad? | 08:01 |
| arxcruz | cumpleaños is birthday right ? | 08:01 |
| quiquell | arxcruz: yep, navidad is the word | 08:01 |
| *** quiquell is now known as quiquell|brb | 08:04 | |
| *** jtomasek has joined #oooq | 08:06 | |
| *** ccamacho has joined #oooq | 08:11 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-containerized-undercloud-upgrades, tripleo-ci-centos-7-scenario007-multinode-oooq-container, tripleo-ci-centos-7-standalone-upgrade @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-scenario004-multinode-oooq-container @ https://review.openstack.org/602248, master: tripleo-ci-centos-7-standalone-upgrade @ (1 more message) | 08:14 |
| *** tosky has joined #oooq | 08:14 | |
| *** ykarel_ is now known as ykarel | 08:33 | |
| *** rascasoft has joined #oooq | 08:39 | |
| chandankumar | arxcruz: Happy New year :-) | 08:45 |
| *** bogdando has joined #oooq | 08:48 | |
| arxcruz | chandankumar: happy new year | 08:51 |
| chandankumar | arxcruz: :-) | 08:51 |
| *** quiquell|brb is now known as quiquell | 08:52 | |
| ssbarnea | happy new year full of green builds to everyone! | 09:05 |
| *** bogdando has quit IRC | 09:10 | |
| *** bogdando has joined #oooq | 09:10 | |
| *** holser_ has joined #oooq | 09:11 | |
| *** saneax has quit IRC | 09:18 | |
| quiquell | ssbarnea: hoppy new year | 09:20 |
| *** holser_ has quit IRC | 09:21 | |
| *** holser_ has joined #oooq | 09:22 | |
| panda | still trying to understand who am I and what I'm doing here | 09:44 |
| panda | I hate coming back from holidays. | 09:44 |
| marios | panda: happy new year! | 09:48 |
| marios | woo | 09:48 |
| panda | marios: do I know you ? | 09:49 |
| panda | probably I do | 09:49 |
| quiquell | It's time to feed panda fake information to gain access to his brain | 09:51 |
| *** dtantsur has joined #oooq | 09:52 | |
| panda | that's a good idea. I'd too like to access my brain some time, especially if we can find a way to not puke in disgust like the other times I was able to do it. | 09:54 |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-containerized-undercloud-upgrades, tripleo-ci-centos-7-scenario007-multinode-oooq-container, tripleo-ci-centos-7-standalone-upgrade @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-scenario004-multinode-oooq-container @ https://review.openstack.org/602248, master: tripleo-ci-centos-7-standalone-upgrade @ (1 more message) | 10:14 |
| arxcruz | chandankumar: I hope this works https://review.openstack.org/#/c/627500/ :) | 10:24 |
| *** skramaja has quit IRC | 10:24 | |
| arxcruz | chandankumar: but later, need to change the post_run | 10:24 |
| chandankumar | arxcruz: yes, I am still experimenting with it | 10:32 |
| chandankumar | arxcruz: https://review.openstack.org/#/c/627482/ another experiment | 10:32 |
| arxcruz | cool | 10:33 |
| *** bogdando has quit IRC | 10:33 | |
| chandankumar | panda: marios ssbarnea sshnaidm Happy New Year :-) | 10:34 |
| chandankumar | panda: please do not say you donot know us :-) | 10:34 |
| marios | same to you chandankumar o/ | 10:36 |
| chandankumar | marios: :-) | 10:42 |
| ssbarnea | marios: quiquell : can we workflow https://review.openstack.org/#/c/627620/1 ? | 11:02 |
| quiquell | +2 | 11:04 |
| rascasoft | happy new year folks ;) | 11:05 |
| marios | o/ rascasoft :D | 11:05 |
| quiquell | rascasoft: hey dude | 11:07 |
| rascasoft | yo quiquell | 11:08 |
| *** quiquell is now known as quiquell|brb | 11:11 | |
| marios | ssbarnea: i will have a look but not right now | 11:13 |
| *** udesale has quit IRC | 11:18 | |
| *** holser_ is now known as holser|lunch | 11:37 | |
| *** sshnaidm is now known as sshnaidm|afk | 11:48 | |
| ssbarnea | marios: i seen that you got some sprint related CRs in limbo, i will try to ping those that can workflow them to +W them. | 12:05 |
| ssbarnea | marios: https://review.openstack.org/#/c/627520/ for example, right? | 12:06 |
| marios | ssbarnea: sure thanks | 12:09 |
| ssbarnea | marios: there are lots of them so it will require time to make them happen. also I am happy to assist anyone needing help. | 12:10 |
| *** quiquell|brb is now known as quiquell | 12:11 | |
| *** saneax has joined #oooq | 12:12 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-containerized-undercloud-upgrades, tripleo-ci-centos-7-scenario007-multinode-oooq-container, tripleo-ci-centos-7-standalone-upgrade @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-containers-multinode @ https://review.openstack.org/602248, master: tripleo-ci-centos-7-standalone-upgrade @ https://review.openstack.org/560445, stable/ocata: (1 more message) | 12:14 |
| *** ratailor has quit IRC | 12:22 | |
| marios | ssbarnea: err which else? | 12:22 |
| marios | ssbarnea: maybe this one for extra_tags https://review.openstack.org/604768 which i created a sprint issues card for https://tree.taiga.io/project/tripleo-ci-board/issue/542?kanban-status=2027733 | 12:23 |
| marios | ssbarnea: otherwise i'm looking at two main things scen1 and scen4 like https://review.openstack.org/#/q/topic:replace-scen1 https://review.openstack.org/#/q/topic:replace-scen4 | 12:23 |
| *** udesale has joined #oooq | 12:26 | |
| ssbarnea | marios: please use "oooq/" prefix for topics, is makes much easier to focus on sprint ones if we do. | 12:30 |
| *** sanjayu_ has joined #oooq | 12:36 | |
| *** saneax has quit IRC | 12:39 | |
| *** rlandy has joined #oooq | 12:39 | |
| *** rlandy is now known as rlandy|rover | 12:42 | |
| quiquell | rlandy|rover: o/ Happy new year | 12:42 |
| rlandy|rover | quiquell: hey - happy new year to you!! | 12:42 |
| *** rfolco has joined #oooq | 12:43 | |
| *** sshnaidm|afk is now known as sshnaidm | 12:43 | |
| rlandy|rover | wow - queues are already up to 4 hours | 12:44 |
| rlandy|rover | people go going quickly | 12:44 |
| marios | rlandy|rover: happy new year ! | 12:44 |
| marios | at least there was master promotion on new years day | 12:44 |
| rlandy|rover | marios: hey - that's the best news yet | 12:45 |
| rlandy|rover | happy new year to you!! | 12:45 |
| rlandy|rover | hope you got some time off | 12:45 |
| marios | rlandy|rover: yup was great thanks and u? | 12:45 |
| rlandy|rover | marios: too good :) ... feels like I have been away forever | 12:46 |
| *** holser|lunch is now known as holser_ | 12:46 | |
| quiquell | new years promotion :-) | 12:46 |
| marios | ssbarnea: ack i mean i'm up for using some standard topic if we are all doing that, however please dont edit the scen1/4 ones unless you can also go through and change it in taiga stories/tasks too please | 12:46 |
| *** bogdando has joined #oooq | 12:46 | |
| marios | ssbarnea: (so i'd rather not for the existing things, but am happy to conform :) for anything new? | 12:46 |
| *** quiquell is now known as quiquell|lunch | 12:48 | |
| ssbarnea | marios: i only asked, didn't change them. i find very useful to have the custom gerrit menu that filters by our topic prefix, saved me a lot of time. | 12:49 |
| ssbarnea | rlandy|rover: welcome back! 2x coffee dose needed. | 12:50 |
| rlandy|rover | ssbarnea: totally - happy new year to you! | 12:50 |
| marios | ssbarnea: ack it has come up before but maybe you need to sell it more like bring it up again as item for improvement and make it clear so everyone tries to use it so we can see the benefit? | 12:51 |
| marios | ssbarnea: otherwise, yeah i'm manually going through taiga each morning lookinga nd opening the ready for reviews cards | 12:51 |
| marios | ssbarnea: would be nice if there was some query, however by doing that, we lose one vital piece of information, the 'ready for review bit' i guess we can signal it on the review with a comment | 12:52 |
| ssbarnea | marios: query 1: https://review.openstack.org/#/q/intopic:%5Eoooq/.*+status:open | 12:52 |
| ssbarnea | marios: query 2 (where you didn't vote): https://review.openstack.org/#/q/intopic:%5Eoooq/.*+status:open+-label:Workflow=-1+label:Verified%3E0++label:Code-Review=0%252Cuser=self | 12:53 |
| ssbarnea | marios: the "ready for review" is very easily translated to a gerrit query. | 12:54 |
| marios | ssbarnea: thanks i'll try them tomorrow morning (noted). will it just give me your reviews though ;) i.e. are you the only one using it currently | 12:54 |
| rlandy|rover | quiquell|lunch: can we close https://bugs.launchpad.net/tripleo/+bug/1808118? | 12:54 |
| openstack | Launchpad bug 1808118 in tripleo "f28 standalon fails at container prep" [Critical,In progress] - Assigned to Quique Llorente (quiquell) | 12:54 |
| rfolco | panda, rlandy|rover: happy 2019! can you make a new pass on https://review.rdoproject.org/r/#/q/topic:f28_promotion | 13:04 |
| rlandy|rover | rfolco: yep - just checking out phase 1 failure | 13:08 |
| rlandy|rover | marios: do you think https://trello.com/c/ftaxprqH/833-cixlp1804812tripleociproa-masternoop-multinode-container-update-job-failed-with-runtimeerror-update-failed-with-ustatus-urunning is still valid - looks to be passing quite well lately | 13:12 |
| rlandy|rover | no update on the bug since your comment | 13:13 |
| rlandy|rover | 12/04 | 13:13 |
| ykarel | rlandy|rover, i filed to bugs: https://bugs.launchpad.net/tripleo/+bug/1810269 and https://bugs.launchpad.net/tripleo/+bug/1810136 | 13:14 |
| openstack | Launchpad bug 1810269 in tripleo "tempest.api.compute.admin.test_agents.AgentsAdminTestJSON tests failing with Internal Server Error" [Undecided,New] | 13:14 |
| openstack | Launchpad bug 1810136 in tripleo "standalone-upgrade job failing while upgrading mariadb" [Undecided,New] | 13:14 |
| ykarel | these affected master promotion jobs | 13:14 |
| rlandy|rover | ykarel; thanks - looking | 13:15 |
| panda | rfolco: ok. Pass. | 13:16 |
| ssbarnea | rlandy|rover: can you help with https://review.openstack.org/#/c/611887/2 ? --- sshnaidm: you didn't ack my reply on it. | 13:16 |
| marios | quiquell|lunch: rfolco o/ reviews please when you have some time we were missing some stuff indeed as per yatin comments. https://review.openstack.org/#/c/627965/ and added some notes on the sanity checking in https://tree.taiga.io/project/tripleo-ci-board/task/448 just now (under Test Req) cc rfolco panda thanks | 13:16 |
| marios | ykarel: ^^ | 13:16 |
| marios | quiquell|lunch: rfolco is worth looking into your scenarios or whoever will be working on those | 13:17 |
| rfolco | panda, thanks. | 13:17 |
| ykarel | ack | 13:17 |
| *** trown|outtypewww is now known as trown | 13:17 | |
| rfolco | marios, will see asap | 13:17 |
| marios | rfolco: thanks in your own time | 13:18 |
| sshnaidm | ssbarnea, I saw your reply, it still doesn't make any sense for me | 13:19 |
| *** dougbtv has joined #oooq | 13:20 | |
| marios | sshnaidm: hey man can you please check https://github.com/sshnaidm/sova/pull/52 thanks | 13:20 |
| sshnaidm | marios, thanks, merged | 13:22 |
| sshnaidm | marios, can you please do the same for "gates" branch? | 13:22 |
| sshnaidm | marios, it will add it to gates jobs monitoring | 13:23 |
| rlandy|rover | ykarel: wrt https://bugs.launchpad.net/tripleo/+bug/1810136 - yeah sporadic - last run is a pass ... makes things complicated | 13:23 |
| openstack | Launchpad bug 1810136 in tripleo "standalone-upgrade job failing while upgrading mariadb" [Undecided,New] | 13:23 |
| marios | sshnaidm: ack in a bit | 13:23 |
| marios | sshnaidm: thanks | 13:23 |
| rfolco | marios, I thought I heard from weshay|ruck in the planning meeting he asking you to take care of all scenarios. Am I wrong ? | 13:23 |
| ykarel | rlandy|rover, yes this seems an intermittent issue, good to keep an eye, checkling logstash again | 13:23 |
| marios | rfolco: no | 13:23 |
| marios | rfolco: in fact the opposite | 13:23 |
| marios | rfolco: NOT to | 13:24 |
| marios | rfolco: but i decided to just do 1/4 because sprint in trouble | 13:24 |
| marios | and it still leaves 2/3 | 13:24 |
| marios | the rationale was for other folks to pick those up | 13:24 |
| marios | not us | 13:24 |
| marios | rfolco: ^ | 13:24 |
| ykarel | rlandy|rover, so from logstash query i can see recent failures | 13:24 |
| ykarel | so issue is live | 13:24 |
| ykarel | checking http://logstash.openstack.org/#/dashboard/file/logstash.json?query=message:%5C%22Can't%20connect%20to%20local%20MySQL%20server%20through%20socket%5C%22 | 13:24 |
| rfolco | ok I can take a look at 2/3 today | 13:25 |
| rfolco | marios, thx | 13:25 |
| marios | rfolco: i mean is a special sprint where availability is really mixed bag | 13:25 |
| rlandy|rover | ykarel: yes - will take that bug to track - not sure about promotion blocker status | 13:25 |
| ykarel | it affected promotion job atleast twice iirc | 13:26 |
| ykarel | not twice, it affected 6 runs | 13:27 |
| ykarel | https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-7-singlenode-featureset050-upgrades-master | 13:27 |
| ykarel | last run passed, u can keep an eye for next runs though, but good to involve mariadb folks on it | 13:27 |
| rlandy|rover | ykarel: not in promotion criteria though: https://github.com/rdo-infra/ci-config/blob/master/ci-scripts/dlrnapi_promoter/config/master.ini | 13:28 |
| ykarel | rlandy|rover, it's there periodic-singlenode-featureset050 | 13:28 |
| rlandy|rover | singlenode-featureset050-upgrades | 13:29 |
| ykarel | it reports periodic-singlenode-featureset050 to dlrnapi | 13:29 |
| rlandy|rover | it does report bit not sure it considers it | 13:29 |
| ykarel | rlandy|rover, https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-singlenode-featureset050-upgrades-master/2d79e5d/job-output.txt.gz#_2019-01-01_13_32_56_853109 | 13:29 |
| ssbarnea | panda: do you want me to fix https://review.openstack.org/#/c/607288/ ? i can easily do it. | 13:29 |
| rlandy|rover | anyways - I'll check it | 13:30 |
| ykarel | --job-id periodic-singlenode-featureset050 | 13:30 |
| ykarel | ack | 13:30 |
| weshay|ruck | morning folks | 13:30 |
| rlandy|rover | weshay|ruck: hello - happy new year | 13:31 |
| weshay|ruck | :) | 13:31 |
| weshay|ruck | happy new year all | 13:31 |
| chandankumar | weshay|ruck: Happy new year :-) | 13:32 |
| ykarel | weshay|ruck, Happy new year!!! | 13:32 |
| weshay|ruck | wow.. 2019 is awesome.. master promoted, ykarel is on patrol with us.. we got this! | 13:32 |
| marios | o/ weshay|ruck | 13:32 |
| weshay|ruck | howdy | 13:33 |
| marios | weshay|ruck: yeah new years day present | 13:33 |
| marios | master promotion | 13:33 |
| marios | also christmas day present | 13:33 |
| weshay|ruck | ya... I saw :) | 13:33 |
| marios | 2019 bodes well | 13:33 |
| * weshay|ruck catches up and reads | 13:33 | |
| weshay|ruck | quiquell|lunch, we have like 3 meetings today :) lolz | 13:33 |
| weshay|ruck | er.. that's friday nevermind | 13:35 |
| weshay|ruck | rlandy|rover, wow.. check out how long the tripleo gate queue is, but still at only about 3.5 hours | 13:37 |
| weshay|ruck | I don't think I've ever seen that | 13:37 |
| rlandy|rover | weshay|ruck: yeah - that was my first comment today ... | 13:37 |
| rlandy|rover | <rlandy|rover> wow - queues are already up to 4 hours | 13:38 |
| rlandy|rover | <rlandy|rover> people go going quickly | 13:38 |
| rlandy|rover | weshay|ruck: the rdo queue is longer | 13:38 |
| rlandy|rover | weshay|ruck: there are ovb jobs queued | 13:38 |
| rlandy|rover | but we don't have a lot of stacks | 13:38 |
| rlandy|rover | 5 hr 34 min | 13:38 |
| weshay|ruck | 25 stacks | 13:39 |
| weshay|ruck | hrm | 13:39 |
| rlandy|rover | weird, right? | 13:39 |
| ykarel | weshay|ruck, rlandy|rover is this caused by moving away from te-broker? i am noticing this queue from some time | 13:39 |
| rlandy|rover | ykarel: we had 100 stacks going before the break | 13:40 |
| weshay|ruck | ykarel, no.. javier and crew must have it limited in nodepool | 13:40 |
| rlandy|rover | I am not sure | 13:40 |
| weshay|ruck | has nothing to do w/ not using the te-broker | 13:40 |
| ykarel | weshay|ruck, ack, good to cross check the quotas | 13:41 |
| ykarel | as queue started growing recently | 13:41 |
| weshay|ruck | ykarel, aye.. rdo cloud went down over the break.. so maybe they brought it up slowly and havent moved to full throttle yet | 13:42 |
| rlandy|rover | checking quotas | 13:43 |
| panda | ssbarnea: you just rebased ? | 13:43 |
| ssbarnea | panda: yep. | 13:43 |
| ykarel | weshay|ruck, ack, | 13:43 |
| * ykarel need to leave, will check later | 13:44 | |
| weshay|ruck | ykarel++ | 13:44 |
| hubbot1 | weshay|ruck: ykarel's karma is now 8 | 13:44 |
| weshay|ruck | thanks | 13:44 |
| rlandy|rover | ykarel: thanks for checking all that earlier | 13:44 |
| weshay|ruck | rlandy|rover, anything we need to do immediately? I was going to catch up a bit | 13:45 |
| rlandy|rover | weshay|ruck: no limit on the tenant itself - checking providers | 13:45 |
| weshay|ruck | rlandy|rover, I think it's nodepool | 13:45 |
| rlandy|rover | weshay|ruck: there are two bugs ykarel logged | 13:45 |
| weshay|ruck | rlandy|rover, ya.. the selinux one is still happening | 13:45 |
| rlandy|rover | looking into those ... one is a rdo phase 1 blocker | 13:45 |
| weshay|ruck | sshnaidm, ^ | 13:45 |
| weshay|ruck | k | 13:45 |
| rlandy|rover | https://bugs.launchpad.net/tripleo/+bug/1810269 | 13:45 |
| openstack | Launchpad bug 1810269 in tripleo "tempest.api.compute.admin.test_agents.AgentsAdminTestJSON tests failing with Internal Server Error" [Critical,Triaged] - Assigned to Ronelle Landy (rlandy) | 13:45 |
| weshay|ruck | rlandy|rover, same issue w/ turned off podman for | 13:46 |
| rlandy|rover | weshay|ruck: the second is sporadic ... https://bugs.launchpad.net/tripleo/+bug/1810136 | 13:46 |
| openstack | Launchpad bug 1810136 in tripleo "standalone-upgrade job failing while upgrading mariadb" [Undecided,New] | 13:46 |
| weshay|ruck | rlandy|rover, /me wonders if we'll see it reset the gate this morning | 13:46 |
| weshay|ruck | oh | 13:46 |
| rlandy|rover | we shoudl talk about those before the escallation meeting | 13:46 |
| weshay|ruck | those are both new to me | 13:46 |
| rlandy|rover | yeah - logged today | 13:46 |
| rlandy|rover | weshay|ruck: go catch up - will chat with you when you have a moment | 13:46 |
| weshay|ruck | rlandy|rover, ack.. /me reads through them | 13:47 |
| sshnaidm | weshay|ruck, hi | 13:47 |
| weshay|ruck | howdy sir | 13:47 |
| rlandy|rover | the second I am not sure is a promotion blocker | 13:47 |
| sshnaidm | weshay|ruck, so what is the question..? | 13:47 |
| weshay|ruck | sshnaidm, the issue w/ podman/docker that we found but couldn't recreate and we approved turning podman back on.. ( which was fine ) https://trello.com/c/oQ6SYXAk | 13:48 |
| weshay|ruck | still happening | 13:48 |
| *** quiquell|lunch is now known as quiquell | 13:48 | |
| weshay|ruck | sshnaidm, leading me to think more about how to put a patch or dep through the gauntlet of promotion jobs | 13:48 |
| quiquell | weshay|ruck: o/ happy new year | 13:49 |
| weshay|ruck | sshnaidm, just following up on our conversation that the issue came back and our rechecks upstream never failed | 13:49 |
| weshay|ruck | quiquell, HAPPY HAPPY JOY JOY | 13:49 |
| ssbarnea | panda: only one more conflict, but I am not sure how to fix this one, better to allow you to rebase it https://review.openstack.org/#/c/617617/18 | 13:51 |
| rlandy|rover | weshay|ruck: stack numbers are coming up now | 14:08 |
| rlandy|rover | 28 | 14:08 |
| rlandy|rover | so maybe queue will clear | 14:08 |
| weshay|ruck | rlandy|rover, it should go up to 100 I think | 14:11 |
| rlandy|rover | not sure why we queued so many jobs | 14:11 |
| rlandy|rover | 6 hr waits | 14:11 |
| *** ykarel is now known as ykarel|away | 14:12 | |
| rlandy|rover | possibly resources are short | 14:13 |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-containerized-undercloud-upgrades, tripleo-ci-centos-7-scenario007-multinode-oooq-container, tripleo-ci-centos-7-standalone-upgrade @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-containers-multinode @ https://review.openstack.org/602248, master: tripleo-ci-centos-7-standalone-upgrade @ https://review.openstack.org/560445, stable/ocata: (1 more message) | 14:14 |
| *** ykarel|away has quit IRC | 14:19 | |
| arxcruz | weshay|ruck: hi boss, happy new year :) | 14:25 |
| weshay|ruck | :)) | 14:26 |
| arxcruz | weshay|ruck: can we delete this job https://ci.centos.org/view/rdo/view/promotion-pipeline/job/tripleo-quickstart-promote-newton-delorean-minimal/ ? | 14:26 |
| weshay|ruck | yup | 14:26 |
| arxcruz | weshay|ruck: and also mark https://bugs.launchpad.net/tripleo/+bug/1773445 as invalid/wont fix | 14:26 |
| openstack | Launchpad bug 1773445 in tripleo "tripleo-quickstart-extras-gate-newton-delorean-full-minimal fails to install undercloud - Access denied for user 'heat'@'192.168.24.1" [High,Triaged] - Assigned to Arx Cruz (arxcruz) | 14:26 |
| rlandy|rover | weshay|ruck: would https://bugs.launchpad.net/tripleo/+bug/1810136 be assigned to upgrades team or would we keep it? | 14:29 |
| openstack | Launchpad bug 1810136 in tripleo "standalone-upgrade job failing while upgrading mariadb" [Undecided,New] | 14:29 |
| *** ykarel|away has joined #oooq | 14:34 | |
| rlandy|rover | ykarel|away: https://trello.com/c/5aO6agvo/853-cixlp1810269tripleociproa-tempestapicomputeadmintestagentsagentsadmintestjson-tests-failing-with-internal-server-error - you have the fix - do you want to take ownership? otherwise I can to track it | 14:36 |
| weshay|ruck | rlandy|rover, /me looks | 14:38 |
| rlandy|rover | weshay|ruck: also pls check me on ^^ ... it's been a while | 14:38 |
| weshay|ruck | rlandy|rover, that is for the upgrades team | 14:38 |
| rlandy|rover | ok - will send over | 14:39 |
| rlandy|rover | jfrancoa: hello ... happy new year to you ... can you review https://bugs.launchpad.net/tripleo/+bug/1810136 - sporadic but pormotion blocker | 14:42 |
| openstack | Launchpad bug 1810136 in tripleo "standalone-upgrade job failing while upgrading mariadb" [Critical,Triaged] | 14:42 |
| jfrancoa | rlandy|rover: hey, happy new year! | 14:42 |
| jfrancoa | rlandy|rover: sure, I'll have a look at it right away | 14:43 |
| rlandy|rover | thank you | 14:43 |
| weshay|ruck | rlandy|rover, fyi... be on the look out for https://bugs.launchpad.net/tripleo/+bug/1810054 | 14:44 |
| openstack | Launchpad bug 1810054 in tripleo "mulitnode jobs failing on gathering facts from subnode-2" [Critical,Triaged] | 14:44 |
| weshay|ruck | rlandy|rover, re: stacks http://dashboard-ci.tripleo.org/d/cEEjGFFmz/cockpit?orgId=1&panelId=179&fullscreen&from=1541079923041&to=now | 14:45 |
| weshay|ruck | ykarel|away, fyi ^ | 14:46 |
| weshay|ruck | rlandy|rover, anything you wanted to chat about prior to the cix call? | 14:46 |
| weshay|ruck | I've reviewd the bugs | 14:46 |
| rlandy|rover | weshay|ruck: I dpn;t think so - if the bugs are in order | 14:47 |
| weshay|ruck | rfolco, panda happy new year guys... can we sync up for 15min today? | 14:47 |
| rlandy|rover | the podman failure is back | 14:47 |
| rfolco | weshay|ruck, happy 2019! Sure, let me know when. | 14:47 |
| weshay|ruck | I'll let panda pick | 14:48 |
| *** ykarel|away is now known as ykarel | 14:51 | |
| weshay|ruck | arxcruz, have you seen this? http://logs.openstack.org/47/627047/1/gate/tripleo-ci-centos-7-standalone/94b845c/job-output.txt.gz#_2019-01-02_14_16_37_139166 | 14:51 |
| ykarel | weshay|ruck, ack | 14:51 |
| arxcruz | weshay|ruck: checking | 14:51 |
| arxcruz | weshay|ruck: fixed, vm reseted, the service didn't start | 14:52 |
| weshay|ruck | arxcruz, rock on | 14:53 |
| arxcruz | i need to add it to some openshift to self health | 14:53 |
| weshay|ruck | arxcruz, see if you can get the health of that service onto the cockpit | 14:53 |
| weshay|ruck | would be a good exercise for you and quiquell | 14:53 |
| arxcruz | quiquell: can we sync this later? | 14:54 |
| weshay|ruck | arxcruz, some sort of health check on http://dashboard-ci.tripleo.org/d/cEEjGFFmz/cockpit?orgId=1 | 14:54 |
| quiquell | arxcruz: sure | 14:54 |
| quiquell | we have need something like cockÃpit-ci.tripleo.org that points to this , this url is toooo long | 14:55 |
| quiquell | sshnaidm: ^ do you know wher to change that ? | 14:55 |
| weshay|ruck | panda, you available? | 14:56 |
| sshnaidm | quiquell, well.. you can't set just url in dns afaik, need to create dns for cockpit.. and then create server for it and set redirect in nginx | 14:58 |
| ykarel | weshay|ruck, rlandy|rover also fs020 master is failing for long(TIMING out at overcloud-prep-images), and is out of promotion criteria, the RDO phase1 issue(bumping rfc3896) must have hit in fs020 | 14:59 |
| weshay|ruck | arxcruz, chandankumar we have four workers on the standalone jobs for tempest.. no immediate action required, but does that seem right to you? | 15:00 |
| weshay|ruck | http://logs.openstack.org/47/627047/1/gate/tripleo-ci-centos-7-standalone/94b845c/logs/stackviz/#/testrepository.subunit/timeline | 15:00 |
| weshay|ruck | see this in a job, so askig | 15:01 |
| weshay|ruck | tempest.exceptions.BuildErrorException: Server 03d7c3a3-28c9-4c3c-9938-44fd9f8fb180 failed to build and is in ERROR status | 15:01 |
| weshay|ruck | Details: {u'message': u'Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 03d7c3a3-28c9-4c3c-9938-44fd9f8fb180.', u'code': 500, u'created': u'2019-01-02T13:50:00Z'} | 15:01 |
| rlandy|rover | ykarel: the overcloud-prep-images issue I remember from before break - been going for a while - clean node failure | 15:01 |
| rlandy|rover | ykarel: wrt RDO phase1 issue - what happens are your change is merged in Fedora? | 15:02 |
| ykarel | rlandy|rover, it's merged in Fedora, and now in queue | 15:02 |
| ykarel | in RDO | 15:02 |
| arxcruz | weshay|ruck: you mean the failure, or how the tests are distributed through workers? if is the second, it looks ok | 15:02 |
| ykarel | rlandy|rover, https://review.rdoproject.org/r/#/c/18044/ waiting for nodes in CI, same issue as tripleo job | 15:02 |
| ykarel | once this patch merges we need one more patch in rdoinfo to promote to -testing | 15:03 |
| arxcruz | unfortunately this is how it works, tests under same "namespace" runs in the same worker, like all TestNetworkBasicOps | 15:03 |
| weshay|ruck | arxcruz, well.. would reducing the workers to 3 free up more resources? | 15:03 |
| quiquell | sshnaidm: ack, will take a look later on | 15:03 |
| rlandy|rover | ykarel: ok - great - thanks for taking care of it | 15:03 |
| weshay|ruck | arxcruz, to prevent the out of resources error | 15:03 |
| arxcruz | weshay|ruck: probably yes, but i can't tell you for sure without tests | 15:04 |
| weshay|ruck | arxcruz, we had multinode jobs at 3 workers I thought | 15:04 |
| ykarel | rlandy|rover, and about fs020 i am not aware, is there a bug? good to make it high priority | 15:05 |
| ykarel | as it's failing for long | 15:05 |
| weshay|ruck | rlandy|rover, oh ya.. ykarel just reminded me.. we need to reset the criteria for master | 15:05 |
| * weshay|ruck looks | 15:05 | |
| rlandy|rover | weshay|ruck: yeah - #periodic-ovb-1ctlr_1comp-featureset020 | 15:06 |
| rlandy|rover | ykarel is correct as always :) | 15:06 |
| arxcruz | weshay|ruck: however, this is not a resource error | 15:06 |
| arxcruz | weshay|ruck: http://logs.openstack.org/47/627047/1/gate/tripleo-ci-centos-7-standalone/94b845c/logs/undercloud/var/log/containers/nova/nova-compute.log.txt.gz#_2019-01-02_13_49_54_600 | 15:06 |
| rlandy|rover | we hit the introspection error on fs020 | 15:06 |
| rlandy|rover | hence no tempest | 15:07 |
| weshay|ruck | rlandy|rover, k | 15:07 |
| rlandy|rover | and promotion went through | 15:07 |
| weshay|ruck | not too worried about it | 15:07 |
| rlandy|rover | weshay|ruck: want to put fs020 back in criteria? | 15:08 |
| quiquell | sshnaidm: To create the server for the cockpit-ci sub domain we need a new IP ? | 15:08 |
| rlandy|rover | we were supposed to update that whole file | 15:08 |
| quiquell | sshnaidm: virtualhost is not enough ? | 15:08 |
| rlandy|rover | for the new system | 15:08 |
| quiquell | sshnaidm: Or is nonseense what I am saying ? | 15:08 |
| weshay|ruck | rlandy|rover, https://review.rdoproject.org/r/18053 | 15:08 |
| sshnaidm | quiquell, we need to register a subdomain firstly | 15:09 |
| sshnaidm | quiquell, to the same IP | 15:09 |
| sshnaidm | quiquell, I'll ask dprince, he's tripleo domain maintainer | 15:09 |
| rlandy|rover | weshay|ruck: +2 - as soon as we have ci on that , we can merge it | 15:10 |
| quiquell | sshnaidm: ack, then virtualhost will do the rest with a url rewrite rule I suppose | 15:10 |
| rlandy|rover | 7 hour queues?! | 15:10 |
| sshnaidm | quiquell, no need rewrite, just redirect in nginx | 15:11 |
| quiquell | sshnaidm: ack, just remember my old times with apache httpd | 15:13 |
| quiquell | cool cool | 15:13 |
| quiquell | thanks | 15:13 |
| sshnaidm | quiquell, yeah, it was a nightmare :) | 15:13 |
| sshnaidm | quiquell, np, nginx rulez ) | 15:13 |
| quiquell | sshnaidm: and it's faster | 15:14 |
| sshnaidm | quiquell, yeah, now it's much more than http server | 15:14 |
| quiquell | ÂHoly sh no timeouts at "Upstreawm CI stats" | 15:14 |
| marios | sshnaidm: o/ thanks posted to gates branch here https://github.com/sshnaidm/sova/pull/53 as requested | 15:14 |
| sshnaidm | marios, great, thanks! | 15:15 |
| quiquell | arxcruz: Have to drop, do you want to sync tomorrow, we are in the same time zone it will be easy | 15:15 |
| weshay|ruck | quiquell, lolz | 15:16 |
| arxcruz | quiquell: sure, in the morning is better, since it's more calm :) | 15:16 |
| weshay|ruck | ya.. looks good | 15:16 |
| weshay|ruck | atm | 15:16 |
| quiquell | weshay|ruck: that's why I drop I don't want to see timeouts at 2019 for now :-) | 15:16 |
| weshay|ruck | quiquell, smart | 15:16 |
| weshay|ruck | lolz | 15:16 |
| *** jrist has joined #oooq | 15:17 | |
| *** quiquell is now known as quiquell|off | 15:17 | |
| *** rfolco is now known as rfolco|lunch | 15:18 | |
| weshay|ruck | panda, rfolco 4pm utc.. let's jump on blue | 15:18 |
| marios | panda: weshay|ruck rlandy|rover rfolco|lunch sshnaidm review please if you have time lets merge this https://review.rdoproject.org/r/#/c/17962/ is blocking https://review.openstack.org/#/c/604768/ | 15:22 |
| *** jrist has quit IRC | 15:22 | |
| rlandy|rover | done | 15:25 |
| rlandy|rover | need that merged? | 15:25 |
| rlandy|rover | marios: ^^ | 15:25 |
| rlandy|rover | marios: merging | 15:26 |
| marios | yeah thanks rlandy|rover | 15:26 |
| *** jrist has joined #oooq | 15:26 | |
| *** jfrancoa has quit IRC | 15:39 | |
| *** ykarel is now known as ykarel|awat | 15:48 | |
| *** ykarel|awat is now known as ykarel|away | 15:48 | |
| rlandy|rover | weshay|ruck: hmmm ... create_in_prgress stacks for 21 minutes - seems wrong | 15:49 |
| rlandy|rover | fde0aad4-896d-4303-a8e8-78145709a0e8 | baremetal_4177 | CREATE_IN_PROGRESS | 2019-01-02T15:21:47Z | None | 15:49 |
| rlandy|rover | need to watch those | 15:49 |
| *** udesale has quit IRC | 15:53 | |
| *** rfolco|lunch is now known as rfolco | 15:55 | |
| *** ykarel|away has quit IRC | 15:55 | |
| *** bogdando has quit IRC | 15:59 | |
| *** kopecmartin is now known as kopecmartin|off | 16:04 | |
| rlandy|rover | weshay|ruck: hello ... I'm doing a diff of fs020 and fs001 to compare why we are hitting the stall/failure on prep-images ... | 16:12 |
| rlandy|rover | we never see this on fs001 | 16:12 |
| rlandy|rover | we have two clear diff | 16:13 |
| rlandy|rover | 40undercloud_clean_nodes: >- | 16:13 |
| rlandy|rover | 41 {% if release not in ['newton','ocata','pike'] -%} | 16:13 |
| rlandy|rover | 42 true | 16:13 |
| rlandy|rover | 43 {%- else -%} | 16:13 |
| rlandy|rover | 44 false | 16:13 |
| rlandy|rover | 45 {%- endif -%} | 16:13 |
| rlandy|rover | 46undercloud_inspection_extras: false | 16:13 |
| rlandy|rover | ^^ fs001 | 16:13 |
| rlandy|rover | that are not in fs020 | 16:14 |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-containerized-undercloud-upgrades, tripleo-ci-centos-7-scenario007-multinode-oooq-container, tripleo-ci-centos-7-standalone-upgrade @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-containers-multinode @ https://review.openstack.org/602248, master: tripleo-ci-centos-7-standalone-upgrade @ https://review.openstack.org/560445, stable/ocata: (1 more message) | 16:14 |
| weshay|ruck | rlandy|rover, hrm. interesting.. sec on the line w/ panda and rfolco | 16:22 |
| rlandy|rover | weshay|ruck: there are other diff - when you have time, I'd like to go through the diffs and see which ones are intentional and which are just missed updates | 16:27 |
| *** ykarel|away has joined #oooq | 16:29 | |
| marios | have a good day folks bai | 16:33 |
| *** sanjayu_ has quit IRC | 16:34 | |
| rlandy|rover | panda: sshnaidm: do you know if it's possible to exclude certain providers for upstream check jobs? | 16:53 |
| rlandy|rover | rfolco: ^^? | 16:56 |
| *** brault has joined #oooq | 16:58 | |
| *** tosky has quit IRC | 17:23 | |
| *** saneax has joined #oooq | 17:41 | |
| sshnaidm | rlandy|rover, I don't know | 17:43 |
| sshnaidm | rlandy|rover, better ask in #openstack-infra | 17:43 |
| rlandy|rover | sshnaidm: no worries - working on the issue with dmsimard | 17:43 |
| weshay|ruck | thanks! | 17:48 |
| weshay|ruck | reviews on https://review.openstack.org/#/c/627870/ | 17:55 |
| *** ykarel|away has quit IRC | 18:01 | |
| rlandy|rover | weshay|ruck: have a minute to review the fs020 diffs? | 18:04 |
| weshay|ruck | rlandy|rover, yes | 18:04 |
| rlandy|rover | weshay|ruck: bj may be quickest | 18:04 |
| *** sshnaidm is now known as sshnaidm|afk | 18:05 | |
| weshay|ruck | rlandy|rover, yup.. /me going | 18:05 |
| rlandy|rover | weshay|ruck: your bj? | 18:05 |
| *** saneax has quit IRC | 18:05 | |
| weshay|ruck | sure | 18:06 |
| weshay|ruck | rlandy|rover, https://www.traditionalmedicinals.com/products/breathe-easy/ | 18:07 |
| *** trown is now known as trown|lunch | 18:10 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-containerized-undercloud-upgrades, tripleo-ci-centos-7-scenario007-multinode-oooq-container, tripleo-ci-centos-7-standalone-upgrade @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-containers-multinode @ https://review.openstack.org/602248, master: tripleo-ci-centos-7-standalone-upgrade @ https://review.openstack.org/560445, stable/ocata: (1 more message) | 18:14 |
| *** panda is now known as panda|off | 18:15 | |
| ssbarnea | weshay|ruck: rlandy|rover : a comment on https://review.openstack.org/#/c/627633/2/roles/create-reproducer-script/templates/reproducer-quickstart.sh.j2@35 -- low prio | 18:27 |
| *** holser_ has quit IRC | 18:47 | |
| *** saneax has joined #oooq | 19:02 | |
| *** dtantsur is now known as dtantsur|afk | 19:09 | |
| *** saneax has quit IRC | 19:12 | |
| *** trown|lunch is now known as trown | 19:25 | |
| sshnaidm|afk | ssbarnea, why do you do it? https://review.openstack.org/#/c/627881/3/quickstart.sh | 19:58 |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-containerized-undercloud-upgrades, tripleo-ci-centos-7-standalone-upgrade @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-containerized-undercloud-upgrades @ https://review.openstack.org/560445, stable/ocata: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001 @ https://review.openstack.o | 20:14 |
| rlandy|rover | arxcruz: I'm editing fs020 to include current changes to fs001 - does ssl_overcloud have to be false for any tempest reason? | 20:14 |
| *** panda|off has quit IRC | 20:30 | |
| *** panda has joined #oooq | 20:33 | |
| *** trown is now known as trown|outtypewww | 22:00 | |
| *** holser_ has joined #oooq | 22:10 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-containerized-undercloud-upgrades, tripleo-ci-centos-7-standalone-upgrade @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-containerized-undercloud-upgrades @ https://review.openstack.org/560445, stable/ocata: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001 @ https://review.openstack.o | 22:14 |
| weshay|ruck | rlandy|rover, when you have a sec tomorrow can you add me to testproject? | 22:15 |
| rlandy|rover | weshay|ruck: you have no access to testproject? | 22:16 |
| weshay|ruck | I can clone it, but I can't run git-review -s | 22:16 |
| weshay|ruck | hrm.. I see you are not a member either | 22:16 |
| weshay|ruck | sec.. let me try something | 22:16 |
| rlandy|rover | hmmmm ... nhicher added me | 22:17 |
| weshay|ruck | how do you check it out? | 22:17 |
| weshay|ruck | git clone https://review.rdoproject.org/r/testproject | 22:17 |
| weshay|ruck | ? | 22:17 |
| weshay|ruck | maybe rdo is going down again | 22:18 |
| rlandy|rover | [remote "origin"] | 22:19 |
| rlandy|rover | url = https://review.rdoproject.org/r/testproject | 22:19 |
| rlandy|rover | yeah - let me try again | 22:19 |
| rlandy|rover | git clone https://review.rdoproject.org/r/testproject worked for me | 22:20 |
| weshay|ruck | it's a bare repo though | 22:20 |
| weshay|ruck | nothing in it | 22:20 |
| rlandy|rover | ls -la | 22:20 |
| rlandy|rover | add a .zuul.yaml | 22:21 |
| rlandy|rover | copy mine | 22:21 |
| weshay|ruck | OH | 22:21 |
| rlandy|rover | there is nothing in there by design | 22:21 |
| weshay|ruck | OH | 22:21 |
| weshay|ruck | thanks | 22:21 |
| rlandy|rover | make it depends-on the review you want to test | 22:21 |
| rlandy|rover | if it's periodic, use my hack | 22:21 |
| rlandy|rover | weshay|ruck: lastly - I'm updating the ruck-rover etherpad ... did I miss anything from today - see 01/Jan section https://review.rdoproject.org/etherpad/p/ruckrover-sprint24 | 22:22 |
| weshay|ruck | rlandy|rover, looks good, latest updates go at the top though :) | 22:24 |
| weshay|ruck | and push the older stuff down | 22:25 |
| weshay|ruck | thank you | 22:25 |
| rlandy|rover | ok - makes sense | 22:25 |
| *** tosky has joined #oooq | 22:30 | |
| *** jtomasek has quit IRC | 22:36 | |
| *** holser_ has quit IRC | 22:39 | |
| rlandy|rover | ntpdate[21226]: no server suitable for synchronization found ... hmmm | 23:05 |
| *** rascasoft has quit IRC | 23:10 | |
| *** holser_ has joined #oooq | 23:16 | |
| rlandy|rover | weshay|ruck: going out ... will be back in about three hours - if anything explodes between now and then | 23:22 |
| *** rlandy|rover is now known as rlandy|rover|bbl | 23:23 | |
| weshay|ruck | rlandy|rover|bbl, ur fine.. we're not on pager duty :) | 23:38 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!