| *** agopi has joined #oooq | 00:42 | |
| hubbot | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224 | 00:57 |
|---|---|---|
| *** gkadam_ has quit IRC | 01:07 | |
| *** gkadam has joined #oooq | 01:10 | |
| *** agopi has quit IRC | 01:49 | |
| *** agopi has joined #oooq | 01:58 | |
| *** holser_ has joined #oooq | 02:06 | |
| *** pliu_ has joined #oooq | 02:29 | |
| *** Amro-Egyptian has joined #oooq | 02:40 | |
| Amro-Egyptian | Pm me and tell me how can I become a successful person because I am a big loser | 02:41 |
| *** Amro-Egyptian has quit IRC | 02:44 | |
| hubbot | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224 | 02:57 |
| *** jaganathan has joined #oooq | 03:11 | |
| *** skramaja has joined #oooq | 03:39 | |
| *** udesale has joined #oooq | 03:42 | |
| *** holser_ has quit IRC | 03:44 | |
| *** ykarel has joined #oooq | 04:20 | |
| *** ratailor has joined #oooq | 04:50 | |
| hubbot | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224 | 04:57 |
| *** gkadam has quit IRC | 05:04 | |
| *** gkadam has joined #oooq | 05:08 | |
| *** jfrancoa has joined #oooq | 05:24 | |
| *** gkadam has quit IRC | 05:31 | |
| *** pliu_ has quit IRC | 05:44 | |
| quiquell|off | ykarel: Good morning | 05:48 |
| *** quiquell|off is now known as quiquell|rover | 05:49 | |
| quiquell|rover | ykarel: Looks like the 404 error of sqlite is gone ? | 05:49 |
| ykarel | hmm not seeing again | 05:49 |
| ykarel | quiquell|rover, Good Morning | 05:49 |
| quiquell|rover | ykarel: Closing the bug then | 05:49 |
| ykarel | quiquell|rover, but we have to used infra mirror | 05:50 |
| ykarel | so it should be tracked and closed | 05:50 |
| ykarel | so you can comment that not seeing now, and plan to fix it | 05:50 |
| quiquell|rover | ykarel: You mean that the mirror used is wrong ? | 05:50 |
| ykarel | sshnaidm, also mentioned that yesterdsy | 05:50 |
| ykarel | mirror.centos.org should not be used | 05:51 |
| quiquell|rover | ykarel: Yes... will try to find it. | 05:51 |
| quiquell|rover | ykarel: What should be the correct one ? | 05:51 |
| ykarel | quiquell|rover, infra one, they are added in the job, look for NODEPOOL_CENTOS_MIRROR | 05:53 |
| ykarel | like rax,dfw etc | 05:54 |
| quiquell|rover | ykarel: I think that's in the release file | 05:54 |
| ykarel | may be now | 05:54 |
| ykarel | not | 05:54 |
| *** sanjayu__ has joined #oooq | 06:33 | |
| quiquell|rover | ykarel: We are missing centos-qemu-ev | 06:33 |
| quiquell|rover | I don't see it here http://logs.openstack.org/85/577785/4/gate/tripleo-ci-centos-7-containerized-undercloud-upgrades/838d24d/logs/undercloud/etc/yum.repos.d/ | 06:33 |
| quiquell|rover | So it doesn't get overwritten | 06:33 |
| ykarel | so is it coming from container image? | 06:36 |
| quiquell|rover | ykarel: Yep, but their repor get overwitten with the hosts repos | 06:36 |
| quiquell|rover | but this one is not present | 06:36 |
| quiquell|rover | I have also foound this | 06:37 |
| quiquell|rover | https://bugs.centos.org/view.php?id=14825 | 06:37 |
| quiquell|rover | Going to check the prep repos | 06:37 |
| quiquell|rover | Could be missing something | 06:37 |
| quiquell|rover | https://bugs.centos.org/view.php?id=14825 | 06:37 |
| ykarel | quiquell|rover, ack | 06:37 |
| ykarel | quiquell|rover, we need https://review.openstack.org/#/c/579426/ for pike, queens | 06:39 |
| quiquell|rover | ykarel: Did not land... pufff | 06:40 |
| quiquell|rover | ykarel: Let's move it a little | 06:41 |
| quiquell|rover | sshnaidm: To fix fs020 tempest problem https://review.openstack.org/#/c/579426/ | 06:41 |
| quiquell|rover | just need the +1w | 06:41 |
| quiquell|rover | Any core can help ^ | 06:41 |
| ykarel | quiquell|rover, also proposed inclusion of fs020 master to promotion criteria:- https://review.rdoproject.org/r/#/c/14572/ | 06:49 |
| quiquell|rover | ykarel: Ok, let's merge the other one first | 06:50 |
| quiquell|rover | ykarel: We just do a PUT from repos.d but we have to also remove the ones that are not in the host | 06:53 |
| marios | quiquell|rover: ack | 06:53 |
| quiquell|rover | marios: Try to add a +1w pleasee | 06:53 |
| marios | quiquell|rover: did | 06:53 |
| quiquell|rover | marios: thanks | 06:53 |
| marios | np | 06:54 |
| quiquell|rover | ykarel: ^ | 06:54 |
| ykarel | quiquell|rover, marios Thanks | 06:54 |
| ykarel | but zuul queue is large so we need to wait | 06:55 |
| marios | grateful for your reviews please if you have some time thanks minor fixes for repro issues i ran into https://review.openstack.org/#/c/578081/ https://review.openstack.org/#/c/578768/ https://review.openstack.org/#/c/579587/ | 06:56 |
| ykarel | quiquell|rover, okk good to update ur finding in the bug | 06:56 |
| hubbot | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224 | 06:57 |
| quiquell|rover | marios: Now that we are in review mode, this other one needs +1w https://review.openstack.org/#/c/576277/ | 06:58 |
| quiquell|rover | marios: It will help close the previous sprint tasks | 06:58 |
| quiquell|rover | marios: Humm panda|off has a | 06:58 |
| quiquell|rover | -1 | 06:58 |
| *** quiquell|rover is now known as quique|rover|bbl | 06:59 | |
| *** tesseract has joined #oooq | 07:06 | |
| *** ykarel is now known as ykarel|lunch | 07:19 | |
| *** amoralej|off is now known as amoralej | 07:19 | |
| *** bogdando has joined #oooq | 07:25 | |
| *** quique|rover|bbl is now known as quiquell|rover | 07:34 | |
| *** florianf has joined #oooq | 07:37 | |
| *** Goneri has joined #oooq | 07:53 | |
| sshnaidm | quiquell|rover, I merge https://review.openstack.org/#/c/578447 , so various problems could be expected | 08:00 |
| quiquell|rover | sshnaidm: My last day as a rover, hate you !!! :-) | 08:01 |
| sshnaidm | quiquell|rover, hehe :) tell me please if you see something | 08:01 |
| quiquell|rover | sshnaidm: Sure, just joking | 08:01 |
| *** matbu has joined #oooq | 08:02 | |
| *** ccamacho has joined #oooq | 08:08 | |
| *** dtrainor_ has quit IRC | 08:11 | |
| *** dtrainor_ has joined #oooq | 08:12 | |
| *** ykarel|lunch is now known as ykarel | 08:13 | |
| *** kopecmartin has joined #oooq | 08:15 | |
| quiquell|rover | sshnaidm: I have cleanup the rdo tenant (I think) but the rdo performance graph still show bad stacks | 08:19 |
| sshnaidm | quiquell|rover, looking.. | 08:20 |
| *** tosky has joined #oooq | 08:20 | |
| quiquell|rover | sshnaidm: I have use rhos-dev-ci sourcerc and ./ci-scripts/infra-cleanup/ovb-tenant-cleanup.sh | 08:20 |
| quiquell|rover | This is it ? | 08:20 |
| sshnaidm | quiquell|rover, mmm.. not sure, what is rhos-dev-ci? | 08:21 |
| sshnaidm | quiquell|rover, last stack I see is from 2018-06-30 | 08:22 |
| sshnaidm | quiquell|rover, script runs once in 15 min, so maybe just need to wait a little | 08:22 |
| quiquell|rover | sshnaidm: What do you do to cleanup tenant ? | 08:28 |
| sshnaidm | quiquell|rover, I didn't :) I think this script should run automatically somewhere | 08:29 |
| sshnaidm | quiquell|rover, and it should source credentials from openstack-nodepool tenant | 08:30 |
| quiquell|rover | sshnaidm: The problem is alwasy the credentials that's why is not at any crontab | 08:30 |
| *** gkadam has joined #oooq | 08:33 | |
| sshnaidm | quiquell|rover, it can run on te-broker | 08:34 |
| sshnaidm | quiquell|rover, I think you just use wrong credentials, what is tenant in your rc file? | 08:36 |
| * sshnaidm is running now the script | 08:37 | |
| quiquell|rover | sshnaidm: >rhos-dev-ci | 08:37 |
| sshnaidm | quiquell|rover, yeah, it's wrong tenant.. not sure what is this tenant at all | 08:37 |
| sshnaidm | quiquell|rover, will send you by mail | 08:37 |
| quiquell|rover | sshnaidm: thanks ! | 08:37 |
| *** holser_ has joined #oooq | 08:38 | |
| quiquell|rover | sshnaidm, ykarel: Do this make sense ? https://review.openstack.org/#/c/579803/ | 08:38 |
| quiquell|rover | is a fix for the centos mirror | 08:38 |
| ykarel | ack will check | 08:41 |
| *** holser_ has quit IRC | 08:46 | |
| sshnaidm | quiquell|rover, does COPY merge? | 08:46 |
| quiquell|rover | sshnaidm: Yep, what it does is really change the change files | 08:47 |
| quiquell|rover | sshnaidm: But it merges | 08:47 |
| *** holser_ has joined #oooq | 08:47 | |
| *** ykarel is now known as ykarel|mtg | 08:47 | |
| *** holser_ has quit IRC | 08:52 | |
| *** matbu has quit IRC | 08:56 | |
| hubbot | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224 | 08:57 |
| quiquell|rover | sshnaidm: Have you send me the creds ? | 09:03 |
| sshnaidm | quiquell|rover, sent now | 09:06 |
| quiquell|rover | sshnaidm: Ok, cleaning now | 09:10 |
| sshnaidm | panda|off, can you re-review please? https://review.openstack.org/#/c/576277 replied to your comment | 09:10 |
| sshnaidm | quiquell|rover, I ran this script already | 09:10 |
| quiquell|rover | aahh ok | 09:10 |
| quiquell|rover | stoping | 09:10 |
| sshnaidm | quiquell|rover, nevermind, you can run it again | 09:10 |
| sshnaidm | quiquell|rover, to see if everything works | 09:11 |
| sshnaidm | quiquell|rover, there is one stack in delete_complete, try to delete it with -s option | 09:11 |
| quiquell|rover | sshnaidm: RDO cloud performance is good now | 09:11 |
| quiquell|rover | sshnaidm: I am going to add an alarm to it | 09:15 |
| *** ratailor has quit IRC | 09:24 | |
| *** ratailor has joined #oooq | 09:25 | |
| *** zoli is now known as zoli|lunch | 09:33 | |
| *** jfrancoa has quit IRC | 09:36 | |
| quiquell|rover | sshnaidm: RDO nodepool alerts https://review.rdoproject.org/r/#/c/14577/ | 09:36 |
| *** jfrancoa has joined #oooq | 09:36 | |
| sshnaidm | quiquell|rover, for 20 min? | 09:38 |
| quiquell|rover | sshnaidm: Last in 20m | 09:39 |
| quiquell|rover | I hve set 5 min but it alerts no data | 09:39 |
| quiquell|rover | s/set/test/ | 09:39 |
| sshnaidm | quiquell|rover, yeah, the script runs every 15 min | 09:39 |
| quiquell|rover | sshnaidm: 20m is good | 09:39 |
| sshnaidm | quiquell|rover, ok, we'll need to think about auto-cleaning though.. | 09:39 |
| quiquell|rover | sshnaidm: It take the last | 09:39 |
| quiquell|rover | sshnaidm: crontab a te-broker sounds like a plan | 09:40 |
| sshnaidm | ya | 09:40 |
| quiquell|rover | sshnaidm: But let's keep the alarm | 09:40 |
| amoralej | quiquell|rover, we've merged https://review.rdoproject.org/r/#/c/14522/ let us know if you see anything abnormal | 09:45 |
| quiquell|rover | amoralej: ack | 09:47 |
| chandankumar | kopecmartin: arxcruz please have a look at sprint 16 ideas https://etherpad.openstack.org/p/tripleo-tempest-sprint16 | 09:55 |
| arxcruz | chandankumar: so, i'm only seeing tasks, but not an epic targeting a specific goal is that correct ? | 09:57 |
| arxcruz | also, under new itens, mostly are not exactly 'new' | 09:57 |
| chandankumar | arxcruz: the goal is closing items out related to python-tempestconf | 09:57 |
| *** ratailor has quit IRC | 09:57 | |
| arxcruz | chandankumar: so they are not sprint 16 ideas ;) | 09:58 |
| chandankumar | arxcruz: yup | 09:58 |
| *** ratailor has joined #oooq | 09:58 | |
| *** matbu has joined #oooq | 10:01 | |
| kopecmartin | chandankumar, looks fine | 10:02 |
| quiquell|rover | arxcruz: Those OVB jobs were failing ? | 10:04 |
| *** panda|off is now known as panda | 10:06 | |
| *** matbu has quit IRC | 10:06 | |
| *** zoli|lunch is now known as zoli | 10:07 | |
| quiquell|rover | sshnaidm, panda, marios, arxcruz: Correct release file for periodic jobs https://review.openstack.org/#/c/578793/ | 10:13 |
| *** dtantsur|afk is now known as dtantsur | 10:14 | |
| sshnaidm | quiquell|rover, did you test it locally? | 10:14 |
| quiquell|rover | sshnaidm: Just did the unit test, will test with a periodic reproducer | 10:15 |
| quiquell|rover | sshnaidm: thanks | 10:15 |
| quiquell|rover | sshnaidm: maybe a dry-run it's enough | 10:15 |
| sshnaidm | quiquell|rover, you can run with dry mode and just to see | 10:15 |
| sshnaidm | quiquell|rover, yep | 10:16 |
| quiquell|rover | sshnaidm: Read my mind :-) | 10:16 |
| *** ykarel|mtg is now known as ykarel|afk | 10:16 | |
| quiquell|rover | sshnaidm: btw RR Cockpit alert system filter for IRC https://review.rdoproject.org/r/14578 | 10:16 |
| sshnaidm | quiquell|rover, ack | 10:16 |
| quiquell|rover | marios: About reproducer changes, we were talking about creating libvirt reproducer job | 10:17 |
| quiquell|rover | marios: So we can test them | 10:17 |
| quiquell|rover | marios: Since we cannot access RDO credentials within RDO | 10:18 |
| sshnaidm | quiquell|rover, actually testing reproducer doesn't require to run the whole job, I think it's enough to run noop.yml from the job and that's it | 10:20 |
| sshnaidm | quiquell|rover, it will reveal 99% of problems we have with it | 10:20 |
| sshnaidm | quiquell|rover, or maybe just a dry run or kind of | 10:20 |
| quiquell|rover | sshnaidm: Maybe we can add a matching system for the output | 10:20 |
| quiquell|rover | sshnaidm: at least ansible playbook calls | 10:21 |
| arxcruz | quiquell|rover: which ovb jobs ? | 10:24 |
| quiquell|rover | arxcruz: You talk about fs035 OVB jobs in the bug ? | 10:25 |
| marios | quiquell|rover: cool, yeah i tested them 'manually' by applying and running on my virthost yesterday too fwiw | 10:25 |
| marios | quiquell|rover: which one were you thinking of in particular /me checks for comments https://review.openstack.org/#/c/578081/8/roles/create-reproducer-script/templates/reproducer-quickstart.sh.j2 is kinda funky sshnaidm wdyt :) bash hash | 10:27 |
| marios | quiquell|rover: but generally pretty simple changes just dependency checking (and the one for the missing directory at https://review.openstack.org/#/c/579587/2/roles/create-reproducer-script/templates/reproducer-quickstart.sh.j2 | 10:27 |
| arxcruz | quiquell|rover: yes... | 10:27 |
| sshnaidm | marios, and why hash, and not just array of packages? | 10:31 |
| sshnaidm | marios, seems like kind of overcomplicated | 10:31 |
| sshnaidm | quiquell|rover++ | 10:40 |
| hubbot | sshnaidm: quiquell|rover's karma is now 2 | 10:40 |
| quiquell|rover | Yei !!! := | 10:40 |
| quiquell|rover | :-) | 10:40 |
| sshnaidm | quiquell|rover, do you need to update https://review.rdoproject.org/r/#/c/14578/ ? | 10:40 |
| marios | sshnaidm: well because its 'virtualenv' the command /usr/bin but python-virtualenv the package | 10:40 |
| marios | sshnaidm: is the main reason | 10:41 |
| marios | sshnaidm: please add comments i am very happy to revise | 10:41 |
| quiquell|rover | sshnaidm: yep, will notify you when it's ready, want to refactor stuff too | 10:41 |
| sshnaidm | quiquell|rover, ack | 10:41 |
| marios | sshnaidm: it started as array of packages, we can make it with that, and instead of command -v do rpm -qa | grep $package | 10:42 |
| marios | sshnaidm: would also work fine | 10:42 |
| marios | sshnaidm: thanks for checking please comment when you have time | 10:42 |
| hubbot | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224, stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291 | 10:57 |
| *** holser_ has joined #oooq | 10:57 | |
| *** ykarel|afk is now known as ykarel | 11:10 | |
| amoralej | is barbican at some of the FS we run in periodic jobs? | 11:35 |
| amoralej | i'm getting a problem with it in pike and i'd like to check how (if) we fixed it for master/queens | 11:35 |
| amoralej | it seems we are missing creator role in pike, is that familiar to someone? | 11:36 |
| *** udesale has quit IRC | 11:36 | |
| quiquell|rover | sshnaidm: alert bot hanges ready for review | 11:40 |
| ykarel | amoralej, fs017 | 11:44 |
| *** amoralej is now known as amoralej|lunch | 11:46 | |
| quiquell|rover | sshnaidm: --dry-run was not merge in the reproducer ? | 11:46 |
| quiquell|rover | sshnaidm: Ahh forget about hat | 11:46 |
| sshnaidm | quiquell|rover, it's feature of tripleo-ci | 11:46 |
| quiquell|rover | sshnaidm: Yep... ok | 11:47 |
| sshnaidm | quiquell|rover, just export dry_run=1 in it | 11:49 |
| quiquell|rover | sshnaidm: ok | 11:51 |
| *** quiquell|rover is now known as quique|rover|lch | 11:55 | |
| *** ratailor has quit IRC | 11:56 | |
| *** trown|outtypewww is now known as trown | 12:01 | |
| *** zoli is now known as zoli|afk | 12:04 | |
| *** holser_ has quit IRC | 12:12 | |
| *** rlandy has joined #oooq | 12:30 | |
| *** ykarel has quit IRC | 12:31 | |
| weshay|ruck | sshnaidm, so we need to revisit sova and the promotion jobs | 12:38 |
| sshnaidm | weshay|ruck, let's do it after Paul finishes with moving all | 12:38 |
| weshay|ruck | aye | 12:38 |
| sshnaidm | weshay|ruck, and renaming | 12:39 |
| weshay|ruck | FYI https://trello.com/b/0VFswmht/rdo-infra-retrospective?menu=filter&filter=label:Sprint%2015 | 12:54 |
| weshay|ruck | quique|rover|lch, sshnaidm, arxcruz, chandankumar kopecmartin, rlandy, trown panda rfolco ^ | 12:55 |
| weshay|ruck | retrospective board | 12:55 |
| weshay|ruck | add cards now.. earn twice the interest | 12:55 |
| weshay|ruck | double point tuesday | 12:56 |
| weshay|ruck | these deals can't be beat | 12:56 |
| panda | mmhh,, usually twice the interests, twice the risk | 12:56 |
| weshay|ruck | push, pull or drag in your comments.. we'll take'em | 12:56 |
| weshay|ruck | the risk | 12:56 |
| hubbot | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224, stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291 | 12:57 |
| *** quique|rover|lch is now known as quiquell|rover | 12:57 | |
| *** zoli|afk is now known as zoli | 13:02 | |
| weshay|ruck | rlandy, ping | 13:03 |
| rlandy | weshay|ruck: hello | 13:03 |
| weshay|ruck | rlandy, retro | 13:04 |
| *** kopecmartin has quit IRC | 13:06 | |
| *** kopecmartin has joined #oooq | 13:06 | |
| *** kopecmartin has quit IRC | 13:10 | |
| *** kopecmartin has joined #oooq | 13:10 | |
| *** amoralej|lunch is now known as amoralej | 13:25 | |
| *** tcw has quit IRC | 13:28 | |
| *** tcw has joined #oooq | 13:31 | |
| *** skramaja has quit IRC | 13:40 | |
| *** agopi has quit IRC | 13:42 | |
| weshay|ruck | sshnaidm, there is a dci/sova sync | 13:51 |
| weshay|ruck | in 10min | 13:51 |
| sshnaidm | weshay|ruck, I asked Goneri to postpone it because of retrospective | 13:51 |
| sshnaidm | Goneri, did you see my mail? ^^ | 13:51 |
| Goneri | We can off-course postpone | 13:51 |
| Goneri | no sorry, can we do it in 40 minutes? or tomorrow? | 13:52 |
| sshnaidm | Goneri, if we don't have US folks in meeting, we can do tomorrow | 13:52 |
| sshnaidm | Goneri, it's 4 July holiday | 13:53 |
| Goneri | ah! I forgot that. Well, I postpone same time next week | 13:53 |
| sshnaidm | Goneri, cool, thanks | 13:53 |
| *** ykarel has joined #oooq | 14:06 | |
| *** agopi has joined #oooq | 14:09 | |
| *** agopi_ has joined #oooq | 14:11 | |
| *** atoth has joined #oooq | 14:11 | |
| *** agopi has quit IRC | 14:14 | |
| *** agopi_ is now known as agopi | 14:14 | |
| *** jaganathan has quit IRC | 14:19 | |
| *** jtomasek has joined #oooq | 14:24 | |
| *** gkadam_ has joined #oooq | 14:28 | |
| *** gkadam has quit IRC | 14:30 | |
| *** jtomasek has quit IRC | 14:44 | |
| *** bogdando has quit IRC | 14:50 | |
| *** zoli is now known as zoli|PTO | 14:51 | |
| *** zoli|PTO is now known as zoli | 14:52 | |
| hubbot | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224, stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291 | 14:57 |
| rlandy | trown: panda: https://trello.com/c/tO42ubRD/824-create-new-base-jobs - what still needs to be tested - where are you leaving off? | 15:04 |
| quiquell|rover | + | 15:05 |
| *** quiquell|rover is now known as quiquell|off | 15:05 | |
| rlandy | I have to set up a job for the performance guys today - can pick that stuff up afterwards | 15:05 |
| panda | rlandy: mainly the scenario007 failure | 15:05 |
| panda | rlandy: becasue we can't migrate jobs otherwise | 15:06 |
| rlandy | panda: I thought trown was reproducing that now? | 15:06 |
| trown | rlandy: yep, scenario007-008 failures. I am trying now with libvirt and old undercloud-setup networking | 15:06 |
| trown | rlandy: but assuming that passes, we then need to investigate why it fails with new zuul-jobs networking | 15:07 |
| rlandy | ok - following | 15:08 |
| trown | rlandy: I dont have much yet on actually figuring out what might be wrong, but that is what I am working on today | 15:09 |
| rlandy | trown: yeah, I understand you are working on that - that is why I was asking where I should pick up - so waiting on your results | 15:11 |
| rlandy | can try a multinode reproducer with 008 in the mean time | 15:11 |
| ykarel | is there some issue with promoter? queens not promoted: https://trunk-primary.rdoproject.org/api-centos-queens/api/civotes_detail.html?commit_hash=097712f2d4ed4a9ed3734ee61e7bd15cf0353bdd&distro_hash=16f2eba62e9f8648425eedc7769f7b57b6cafe82, fs020 is out of promotion criteria | 15:11 |
| ykarel | weshay|ruck, quiquell|off ^^ | 15:11 |
| weshay|ruck | chandankumar, | 15:16 |
| weshay|ruck | %gatestatus | 15:16 |
| hubbot | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224, stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291 | 15:16 |
| weshay|ruck | tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades known issue chandankumar | 15:17 |
| weshay|ruck | chandankumar, legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata need to look at | 15:17 |
| weshay|ruck | chandankumar, https://review.openstack.org/564291 | 15:17 |
| weshay|ruck | chandankumar, #oooq-test | 15:20 |
| *** kopecmartin has quit IRC | 15:20 | |
| weshay|ruck | chandankumar, #tripleo-ci | 15:21 |
| *** sanjayu__ has quit IRC | 15:28 | |
| *** weshay|ruck is now known as weshay | 15:29 | |
| *** sshnaidm is now known as sshnaidm|rover | 15:29 | |
| *** chandankumar is now known as chkumar|ruck | 15:29 | |
| panda | rlandy: trown is the main different between scen4 and scen7 the use of OVN ? | 15:34 |
| trown | panda: hmm looks that way | 15:36 |
| panda | rlandy: trown since network basic ops is succeding in the other scenarios, it's either OVN failure, or we are making OVN fail | 15:36 |
| panda | rlandy: trown I'll ask someone from OVN if it can see something there | 15:37 |
| rlandy | also scenario008 | 15:38 |
| panda | lucasagomes: hey, do you see anything evidently problematic with OVN in this job ? http://logs.openstack.org/53/579653/2/check/tripleo-ci-centos-7-scenario007-multinode-oooq-container/d89bd62/logs/undercloud/home/zuul | 15:39 |
| panda | lucasagomes: we are changing a bit the way we connect undercloud to overcloud in this job, compared to the previous | 15:39 |
| panda | rlandy: trown: mmmhhh I think the dummy patch I created was branched incorrectly | 15:42 |
| panda | rlandy: trown I'm going to rebase it | 15:42 |
| panda | ... it was missing 437 commits | 15:43 |
| * panda facepalms | 15:44 | |
| rlandy | reproducer failed | 15:44 |
| rlandy | might explain things | 15:44 |
| rlandy | panda: scenario008 not OVN? | 15:49 |
| panda | rlandy: scenario8 is not OVN, but in my non rebased patch scenario008 wasn't even present, not sure how we could get so far in the test. | 15:51 |
| panda | rlandy: scenario008 uses opendaylight | 15:52 |
| rlandy | idk - but 008 also has the tempest failure | 15:53 |
| *** holser_ has joined #oooq | 15:53 | |
| *** ccamacho has quit IRC | 15:54 | |
| lucasagomes | panda, checking... sorry I was in a meeting | 16:01 |
| panda | lucasagomes: hold on on that, sorry, maybe it's just a rebasing problem | 16:03 |
| panda | lucasagomes: thanks for your availability, I'll ping you again tomorrow in case | 16:03 |
| lucasagomes | panda, ack... Cause I was thinking maybe it's the datapath problem with Centos 7.5. But lemme know if the rebase solves, if not I can dig into it | 16:04 |
| lucasagomes | and check it for ya | 16:04 |
| panda | lucasagomes: thanks! | 16:04 |
| *** panda is now known as panda|off | 16:12 | |
| *** agopi is now known as agopi|brb | 16:19 | |
| *** tesseract has quit IRC | 16:20 | |
| *** yolanda has joined #oooq | 16:24 | |
| *** gkadam__ has joined #oooq | 16:30 | |
| *** gkadam_ has quit IRC | 16:32 | |
| *** holser_ has quit IRC | 16:36 | |
| *** holser_ has joined #oooq | 16:37 | |
| *** trown is now known as trown|lunch | 16:44 | |
| *** amoralej is now known as amoralej|off | 16:45 | |
| *** ykarel is now known as ykarel|away | 16:46 | |
| *** ykarel|away has quit IRC | 16:56 | |
| hubbot | FAILING CHECK JOBS on stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224 | 16:57 |
| chkumar|ruck | weshay: you were saying pike promotion is blocked, http://38.145.34.131:3000/d/pgdr_WVmk/cockpit?orgId=1 -> Tripleo CI promotion alerts tab -> It is Red, How to find which one is failing for pike? | 17:02 |
| weshay | chkumar|ruck, promotions panel --> skipped pike promotions | 17:03 |
| weshay | 48330519caa17e40dd12d0fc9cdf3270e43d8a1b_342b0af8 | 17:03 |
| weshay | tripleo-ci-testing | 17:03 |
| weshay | current-tripleo | 17:03 |
| weshay | periodic-ovb-1ctlr_1comp-featureset020 | 17:03 |
| weshay | chkumar|ruck, https://review.rdoproject.org/jenkins/job/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-pike/809/ | 17:04 |
| weshay | although jenkins is going away tomorrow | 17:04 |
| weshay | and we'll need sshnaidm|rover's help in using cistatus | 17:04 |
| weshay | or the zuulv3 build page | 17:04 |
| weshay | https://review.rdoproject.org/zuul3/builds.html | 17:05 |
| chkumar|ruck | weshay: https://review.openstack.org/#/c/579888/ | 17:05 |
| chkumar|ruck | already fixed by amoralej|off | 17:06 |
| weshay | chkumar|ruck, how would a change to fs006 fix fs020? | 17:08 |
| chkumar|ruck | weshay: https://logs.rdoproject.org/openstack-periodic-24hr/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-pike/635da6c/tempest.html.gz | 17:08 |
| weshay | chkumar|ruck, if that is the fix.. fs20 also needs to be updated | 17:08 |
| chkumar|ruck | volume encrytion related tests are failing here | 17:09 |
| weshay | and any fs running tempest I think | 17:09 |
| *** dtantsur is now known as dtantsur|afk | 17:09 | |
| chkumar|ruck | weshay: yup sending apatch | 17:09 |
| weshay | thanks | 17:13 |
| *** agopi|brb is now known as agopi | 17:24 | |
| *** holser_ has quit IRC | 17:25 | |
| *** trown|lunch is now known as trown | 17:43 | |
| chkumar|ruck | weshay: sshnaidm|rover arxcruz https://review.openstack.org/#/c/579937/ | 17:49 |
| chkumar|ruck | it will fix pike failures | 17:49 |
| chkumar|ruck | it is a side effect of refactoring | 17:49 |
| weshay | chkumar|ruck, thanks | 17:52 |
| rlandy | chkumar|ruck: left a comment on https://review.openstack.org/#/c/579937/1 | 17:56 |
| chkumar|ruck | rlandy: thanks :-) | 17:58 |
| chkumar|ruck | weshay: rlandy feel fre to take a look at this series https://review.openstack.org/#/q/topic:refstack-support+(status:open+OR+status:merged) | 18:01 |
| *** ykarel|away has joined #oooq | 18:11 | |
| *** tosky has quit IRC | 18:14 | |
| rlandy | weshay: ping re:browbeat job | 18:15 |
| weshay | rlandy, hey | 18:15 |
| rlandy | weshay: just confirming ... we are adding a job to zuul.d within the browbeat repo | 18:17 |
| rlandy | based on current ovb tests | 18:17 |
| rlandy | to run in rdocloud | 18:17 |
| rlandy | on experimental for the moment? | 18:17 |
| *** yolanda_ has joined #oooq | 18:18 | |
| rlandy | ie: currently there are browbeat pep tests running - but no zuul.d | 18:20 |
| weshay | rlandy, let's run it w/ check, but just on one file | 18:20 |
| rlandy | so we leave that | 18:20 |
| weshay | rlandy, if you run w/ experimental it will never run | 18:20 |
| rlandy | and just work on ovb job | 18:20 |
| *** yolanda has quit IRC | 18:20 | |
| *** yolanda__ has joined #oooq | 18:27 | |
| *** yolanda_ has quit IRC | 18:30 | |
| *** jtomasek has joined #oooq | 18:41 | |
| *** jtomasek has quit IRC | 18:51 | |
| rlandy | agopi: hi - where is baremetal-virt-undercloud-yoda-browbeat.yml playbook? | 18:51 |
| rlandy | oh nvm | 18:52 |
| rlandy | https://github.com/openstack/browbeat/blob/master/ansible/oooq/baremetal-virt-undercloud-yoda-browbeat.yml | 18:52 |
| agopi | https://github.com/openstack/browbeat/tree/master/ansible/oooq | 18:52 |
| agopi | yes | 18:52 |
| rlandy | agopi: is that the playbook you want run? | 18:53 |
| rlandy | for the check job? | 18:53 |
| rlandy | baremetal-virt-undercloud-tripleo-browbeat.yml? | 18:53 |
| agopi | this one rlandy | 18:54 |
| *** tcw has quit IRC | 18:54 | |
| agopi | https://github.com/openstack/browbeat/blob/master/ansible/oooq/baremetal-virt-undercloud-int-browbeat.yml | 18:54 |
| agopi | but itll take around 3-4 hours to run, so we'll probably compress it more once i get my hands on the tenant | 18:54 |
| *** tcw has joined #oooq | 18:55 | |
| rlandy | agopi: maybe we need a new playbook | 18:55 |
| rlandy | for minimal run | 18:55 |
| rlandy | that we can kick in vi | 18:55 |
| rlandy | ci | 18:55 |
| agopi | yes rlandy | 18:56 |
| agopi | well need to compress the benchmark | 18:57 |
| agopi | https://github.com/openstack/browbeat/blob/4052a93f50c422b5f320b1f4fa7ac6a793eaa9d2/ansible/oooq/roles/template-configs/templates/browbeat-basic.yaml.j2 | 18:57 |
| agopi | this is what CI runs for now | 18:57 |
| rlandy | panda|off: trown: I have the tempest failure on 008 in my reproducer | 18:57 |
| hubbot | FAILING CHECK JOBS on stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: legacy-tripleo-ci-centos-7-container-to-container-upgrades-master @ https://review.openstack.org/560445, stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224 | 18:57 |
| *** florianf has quit IRC | 18:58 | |
| rlandy | trown: panda|off: and latest ci run has the failure http://logs.openstack.org/53/579653/3/check/tripleo-ci-centos-7-scenario008-multinode-oooq-container/ad33190/logs/tempest.html.gz | 19:00 |
| agopi | correction the benchmarking takes only 10 mins | 19:00 |
| agopi | the jobs take 3-4 hours for setting up | 19:00 |
| rlandy | agopi: any idea off the top of your head what we could cut | 19:02 |
| rlandy | I am adding the example job | 19:02 |
| rlandy | but it will only kick on one file for now | 19:02 |
| rlandy | while we test it | 19:02 |
| agopi | okay rlandy. | 19:06 |
| agopi | thanks rlandy | 19:06 |
| *** matbu has joined #oooq | 19:09 | |
| trown | rlandy: ya I am trying to look at what is different, because my undercloud-setup run passed | 19:15 |
| rlandy | trown: I have an idea ... | 19:18 |
| rlandy | checkout the mtus on http://logs.openstack.org/92/578892/1/check/tripleo-ci-centos-7-scenario008-multinode-oooq-container/c09d260/logs/undercloud/var/log/extra/network.txt.gz | 19:18 |
| rlandy | see br-ex | 19:18 |
| trown | rlandy: yep just saw that | 19:18 |
| rlandy | passing 008 job | 19:18 |
| rlandy | on ours it's 1400 | 19:18 |
| rlandy | I can drop that on my reproducer and see if the tempest test runs | 19:19 |
| trown | hmm here it is 1300 even... weird it would be different http://logs.openstack.org/02/568602/1/check/tripleo-ci-centos-7-scenario007-multinode-oooq-container/04b57c0/logs/subnode-2/var/log/extra/network.txt.gz | 19:19 |
| trown | oh right | 19:19 |
| rlandy | probably doesn;t matter | 19:19 |
| rlandy | as long as it's low enough | 19:19 |
| rlandy | 1350 is what we usually use on ovb | 19:20 |
| rlandy | trying that | 19:20 |
| trown | rlandy: odd that it is 1350 on undercloud, but 1450 on subnode http://logs.openstack.org/53/579653/2/check/tripleo-ci-centos-7-scenario007-multinode-oooq-container/d89bd62/logs/undercloud/var/log/extra/network.txt.gz | 19:21 |
| rlandy | because we reset it on the undercloud | 19:22 |
| rlandy | the bridge itself is set up with 1450 | 19:22 |
| rlandy | afterwards we reset the undercloud to be lower | 19:22 |
| rlandy | http://logs.openstack.org/92/578892/1/check/tripleo-ci-centos-7-scenario008-multinode-oooq-container/c09d260/logs/undercloud/home/zuul/vxlan_networking.sh.log.txt.gz#_2018-07-03_11_59_12 | 19:22 |
| rlandy | mtu changed - rekicking tempest | 19:26 |
| trown | rlandy: ah right... it looks like we can just add a var for bridge_mtu... the default is pretty convoluted, but if we set it explicitly it will override all of this https://github.com/openstack-infra/zuul-jobs/blob/master/roles/multi-node-bridge/tasks/common.yaml#L56-L88 | 19:26 |
| rlandy | yeah - let's set that to 1350 | 19:27 |
| rlandy | testing that now | 19:27 |
| rlandy | the other test has ctrlplane set to 1350 as well | 19:27 |
| trown | k, I will go ahead and edit that patch too, so if it works we could merge | 19:28 |
| rlandy | perfect | 19:28 |
| *** ykarel|away has quit IRC | 19:30 | |
| rlandy | hmm ... my local test failed but that maybe because the first test failed - clean up | 19:30 |
| rlandy | diff failure | 19:30 |
| trown | ok I have a local run starting too | 19:31 |
| rlandy | trown: k - kicking mine again | 19:37 |
| rlandy | here's hoping | 19:37 |
| rlandy | but I've had enough mtu issues before to be fairly hopeful this will work | 19:38 |
| trown | ya seems promising | 19:38 |
| trown | only so many things could be different | 19:38 |
| trown | I checked rpm versions already | 19:38 |
| rlandy | reproducer will need an update as well | 19:41 |
| *** sanjay__u has joined #oooq | 19:41 | |
| rlandy | but I can do that afterwards | 19:41 |
| rlandy | we need to watch the ctrlplane as well | 19:43 |
| rlandy | not sure what will happen there | 19:43 |
| rlandy | may adjust with the br-ex | 19:43 |
| *** jfrancoa has quit IRC | 19:56 | |
| *** yolanda_ has joined #oooq | 20:14 | |
| *** yolanda has joined #oooq | 20:16 | |
| *** yolanda__ has quit IRC | 20:18 | |
| *** yolanda_ has quit IRC | 20:18 | |
| *** yolanda_ has joined #oooq | 20:19 | |
| rlandy | trown: zuul just reset? | 20:20 |
| trown | rlandy: hmm dont see 579653 in zuul... | 20:22 |
| rlandy | yeah - all recent jobs | 20:22 |
| *** yolanda has quit IRC | 20:22 | |
| rlandy | Patch Set 3: Verified-1 | 20:23 |
| rlandy | This change depends on a change that failed to merge. | 20:23 |
| *** yolanda__ has joined #oooq | 20:24 | |
| *** yolanda_ has quit IRC | 20:24 | |
| trown | well my local reproducer is on overcloud deploy... so should get results on that anyways | 20:31 |
| rlandy | mine is also running | 20:31 |
| rlandy | our job is back :) | 20:52 |
| rlandy | but queued | 20:52 |
| trown | my reproducer failed... | 20:55 |
| trown | br-ex on undercloud is 1400 and subnode is 1350 | 20:55 |
| hubbot | FAILING CHECK JOBS on stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: legacy-tripleo-ci-centos-7-container-to-container-upgrades-master @ https://review.openstack.org/560445, stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224 | 20:57 |
| rlandy | weird | 20:59 |
| rlandy | must be reset somewhere | 21:00 |
| * rlandy checks my reproducer | 21:00 | |
| rlandy | yep | 21:01 |
| rlandy | mine ass well | 21:01 |
| rlandy | checking br-ctrlplane | 21:01 |
| rlandy | trown: check your undercloud.conf | 21:02 |
| rlandy | local_mtu is set at 1400 | 21:03 |
| trown | yep | 21:03 |
| rlandy | that would reset ctlplane | 21:03 |
| rlandy | we could try reset that | 21:04 |
| trown | I have to go get the screamin demons ... I think we might want someone familiar with those scenarios to take a look if the CI still fails | 21:04 |
| rlandy | trown: sure ... happy july 4th | 21:05 |
| trown | ya have a good day off | 21:05 |
| *** trown is now known as trown|outtypewww | 21:05 | |
| *** Goneri has quit IRC | 21:28 | |
| *** gkadam__ has quit IRC | 21:35 | |
| *** agopi is now known as agopi|off | 22:46 | |
| *** agopi|off has quit IRC | 22:54 | |
| hubbot | FAILING CHECK JOBS on stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: legacy-tripleo-ci-centos-7-container-to-container-upgrades-master @ https://review.openstack.org/560445, stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224 | 22:57 |
| *** agopi has joined #oooq | 23:44 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!