Monday, 2019-01-14

*** tosky has quit IRC00:22
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario003-standalone @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario007-multinode-oooq-container,  (2 more messages)00:23
*** dougbtv has quit IRC01:02
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario003-standalone @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario007-multinode-oooq-container,  (2 more messages)02:24
*** saneax has joined #oooq02:34
*** brault has joined #oooq03:04
*** brault has quit IRC03:09
*** udesale has joined #oooq04:23
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario003-standalone, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053, tripleo-ci-centos-7-containerized-undercloud-upgrades @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario007-multinode-oooq- (2 more messages)04:24
*** ratailor has joined #oooq05:10
*** saneax has quit IRC05:27
*** skramaja has joined #oooq05:49
*** saneax has joined #oooq05:53
*** brault has joined #oooq06:00
*** brault has quit IRC06:05
*** udesale has quit IRC06:11
*** udesale has joined #oooq06:12
*** ykarel has joined #oooq06:19
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario003-standalone, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053, tripleo-ci-centos-7-containerized-undercloud-upgrades @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario007-multinode-oooq- (2 more messages)06:24
*** jbadiapa has joined #oooq06:46
*** udesale has quit IRC06:58
*** udesale has joined #oooq06:58
*** jfrancoa has joined #oooq07:06
*** jtomasek has joined #oooq07:35
*** rascasoft has joined #oooq07:37
*** brault has joined #oooq07:42
*** ccamacho has joined #oooq07:42
*** kopecmartin has joined #oooq07:52
*** gkadam has joined #oooq08:00
*** ratailor_ has joined #oooq08:18
*** ratailor has quit IRC08:21
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario003-standalone, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053, tripleo-ci-centos-7-containerized-undercloud-upgrades @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario007-multinode-oooq- (2 more messages)08:24
*** tosky has joined #oooq08:37
quiquellsshnaidm|off: ping me08:52
*** ssbarnea|rover has joined #oooq08:54
*** jpena|off is now known as jpena08:58
mariospanda|off: o/ let me know when you're in we can have that chat about centralise zuul layout standalone scenarios story  https://tree.taiga.io/project/tripleo-ci-board/us/585?milestone=215191 task https://tree.taiga.io/project/tripleo-ci-board/task/60309:01
*** skramaja has quit IRC09:03
chandan_kumararxcruz|ruck: kopecmartin Please have a look at these user stories https://tree.taiga.io/project/tripleo-ci-board/us/554 and https://tree.taiga.io/project/tripleo-ci-board/us/234 when free, Thanks :-)09:10
*** panda|off is now known as panda09:12
ssbarnea|roverarxcruz|ruck: morning! how was you weekend?09:14
ssbarnea|roverbtw, who has some experience with kube? as in already deployed his first cluster.09:15
pandamarios: haing a small problem with headphones, I'll ping you as soon as I solve09:20
*** chandan_kumar is now known as chandankumar09:21
*** chem has joined #oooq09:23
*** apetrich has joined #oooq09:25
mariospanda: k09:26
ssbarnea|roverssbarnea|rover: good weekend, i avoided irc and sped few hours trying to make a working kube cluster with kubespray.09:27
*** holser_ has joined #oooq09:29
*** derekh has joined #oooq09:32
*** bogdando has joined #oooq09:34
*** skramaja has joined #oooq09:35
*** ratailor__ has joined #oooq09:56
*** ratailor_ has quit IRC09:58
*** ratailor__ has quit IRC10:05
toskyarxcruz|ruck: hi, where can I find the definition of periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset021-master and periodic-tripleo-ci-centos-7-multinode-1ctlr_1comp-featureset018-master , where sahara tests are failing right now? I suspect the settings of those tests was changed10:07
mariostosky: here's a trick chandankumar showed me https://codesearch.rdoproject.org/?q=periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset021-master&i=nope&files=&repos=10:11
marios:)10:11
toskyoh, on RDO jobs!10:13
toskyI was checking on codesearch.openstack.org10:13
toskygrrrr10:13
mariostosky: yeah i didn't know of rdo hound either10:14
toskythanks10:15
mariostosky: passing it on ;)10:15
pandamarios: ready10:16
mariospanda: set10:16
marioshttps://redhat.bluejeans.com/7661925373/ ?10:16
mariospanda: ^10:16
pandago10:16
* marios fires blue10:16
* marios switch off death metal10:16
ssbarnea|roverdo we have any reasons for not merging the molecule job? is already 3+ mo old. https://review.openstack.org/#/c/613672/10:17
ssbarnea|rover... or maybe we love testing code manually and this would work against it.10:18
*** ratailor has joined #oooq10:19
chandankumarmarios: :-)10:21
arxcruz|ruckssbarnea|rover: tosky hey, sorry, i'm still with my daughter sick, so i'm a little bit off in the morning, my wife need to go to german class10:22
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario003-standalone, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053, tripleo-ci-centos-7-containerized-undercloud-upgrades @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario007-multinode-oooq- (2 more messages)10:24
toskyokidoki10:24
ssbarnea|rovermarios: skipping conversion of roles in few cases was on purpose, see my comments on https://review.openstack.org/#/c/620146/610:26
*** sshnaidm|off is now known as sshnaidm10:27
quiquell sshnaidm o/ do you have a minute or two ?10:28
*** d0ugal has quit IRC10:29
mariosssbarnea|rover: ack noted for later thanks10:29
sshnaidmquiquell, sure10:32
quiquellsshnaidm: now we can run reproducer zuul job within repro https://review.rdoproject.org/r/#/c/18204/10:33
quiquellsshnaidm: It can help you with the OVB stuff, it encrypts secrets at startup10:33
sshnaidmquiquell, cool, will look10:35
sshnaidmquiquell, although I'd prefer to review it *before* merge..10:35
quiquellsshnaidm: let me know of any issue with it, you just need to put the gerrit ssh key at your ~/.ssh10:35
quiquellsshnaidm: yep... Didn't want to rebase the libvirt thing...10:35
quiquellsshnaidm: I have to stop merging stuff there10:36
sshnaidmquiquell, agree, I think we really need to stop merging w/o review10:36
quiquellsshnaidm: ack10:37
quiquellsshnaidm: btw, fixes the centos7 run, have added sone patchsets to your review10:37
quiquellsshnaidm: It was related to selinux activated at ~/.ssh directory when we mount it at docker10:37
quiquellsshnaidm: Maybe we have to do different and not mount stuff10:38
*** dtantsur|afk is now known as dtantsur10:39
*** d0ugal has joined #oooq10:48
sshnaidmssbarnea|rover, commented on https://review.openstack.org/#/c/613672/10:52
pandachandankumar: do you have any clue what's happening here https://review.openstack.org/627500 ? mind if I ask infra ? I've very curious what's missing10:53
sshnaidmssbarnea|rover, also I'm not sure we need to check quickstart.sh and build-test role in the same job, seems like completely different things10:54
sshnaidmssbarnea|rover, I'd test each role in its separate job10:54
sshnaidmssbarnea|rover, and quickstart.sh stuff in separate quickstart.sh job10:55
ssbarnea|roversshnaidm: this approach does not scale, att all. count how many roles we have, we cannot add this number of jobs.10:55
ssbarnea|roversshnaidm: my plan is to implement an option to allow running a subset of molecule tests10:56
ssbarnea|rovermainly this was the purpose of tags.10:56
sshnaidmssbarnea|rover, firstly not every role we are testing, secondly we will run roles job only if this role is changed, like we do with featuresets10:56
ssbarnea|roverbut really I cannot see how we could split per role.10:56
sshnaidmssbarnea|rover, what is the problem to split per role? molecule is tailored to test a role actually10:57
ssbarnea|roversshnaidm: what is changed is not really obvious as there are many common areas and more important: a role could break even without any change in any tripleo repo. just some new rpms/os minor update could break it.10:57
sshnaidmssbarnea|rover, great, and you'll detect it in role job10:58
sshnaidmssbarnea|rover, mainly roles break when people change there something, all the rest we can ignore for now10:58
*** ratailor_ has joined #oooq10:58
*** skramaja has quit IRC10:59
chandankumarpanda: regarding above review, things are working now, this time role was not found11:00
chandankumarpanda: I think https://review.openstack.org/#/c/630422 adding ansible.cfg to os_temepst will fix it11:00
ssbarnea|rovercounting roles alone: tq:11 tqe:42 tqci:5 tqcommon: 5+5 -- and these are not all repos. can we add 60 jobs?11:00
sshnaidmssbarnea|rover, and testing quickstart.sh for us is very low priority atm, much more important to test roles that aren't tested in CI with changes - like build-test role for example11:01
quiquellsshnaidm: build-test-package and install-built-repo11:01
sshnaidmssbarnea|rover, start with a few critical for now.. most of our roles are tested pretty well in CI11:01
sshnaidmquiquell, yeah, exactly11:02
*** ratailor has quit IRC11:02
quiquellsshnaidm: they are both in the same page of importance11:02
ssbarnea|roversshnaidm: i am not against splits, but I think is too soon to do it from start.11:02
quiquellssbarnea|rover: I would reahter to use molecule for it's purpose with is test roles not to us it as a docker starte and test qucikstart.sh11:02
sshnaidmssbarnea|rover, I think this is only important for us, nobody is interested in "install-deps" of quickstart tbh11:02
ssbarnea|roverwe need to let it grow a little bit in order to see what is the best approach for splitting testing via molecule.11:02
ssbarnea|rovernow the job runs in 12min, much shorter than any other jobs.11:02
ssbarnea|roversshnaidm: i am interested as I found bugs around it and lack of testing prevented correct testing.11:03
sshnaidmssbarnea|rover, I don't know what are arguing with again, but we can definitely discuss it in our near scrum or design meeting11:03
sshnaidmssbarnea|rover, till then I think you need to hold it11:04
ssbarnea|roveras quiquell said, qs.sh may be not important for you. but may be for others.11:04
sshnaidmssbarnea|rover, please hold on this patch, we'll discuss it widely11:04
quiquellssbarnea|rover: Didn't say that :-) sorry about my english11:05
sshnaidmssbarnea|rover, but my issues in the patch aren't related, you can still address them11:06
ssbarnea|roversshnaidm: sure, looking now to fix them. and reg the split, needs a meeting.11:08
*** udesale has quit IRC11:13
quiquellssbarnea|rover, sshnaidm: Does it make sense for roles like build-test-package and install-build-repo to live in their own repo ?11:13
quiquellssbarnea|rover, sshnaidm: So we put molecule testss there11:14
sshnaidmquiquell, we can have tests in the same repo too, it doesn't affect anything11:20
sshnaidmquiquell, afaiu it's not welcomed from infra to create a repo for each role..11:20
quiquellsshnaidm: Yep I know, but since roles ar kind of self contained, make sense to keep them the importnat ones at least in their own repo11:21
quiquellsshnaidm: I see some example upstream like the modify-images11:22
sshnaidmquiquell, well, these ones are "product" roles, and we have like "test" roles.. but all this is questionable of course11:24
quiquellsshnaidm: ack11:25
sshnaidmquiquell, we have even a separate role for setting up docker registry in tripleo, hmm..11:26
sshnaidmin its own repo I mean11:26
quiquellsshnaidm: build-test-package is bigger than that for example11:26
sshnaidmI think panda wanted to move them all to tripleo-ci at all11:26
quiquellholy sh11:27
sshnaidmor to move tripleo-ci to roles, I don't remember11:27
quiquelldon't know maybe I am biases by the repro stuff11:27
ssbarnea|roverideally tests should be stored in the same repo as tested code. but the tq/tqe is more complex because it cannot practically tested in isolation. (one of the reasons why i supported merge of the two repos).11:28
quiquellssbarnea|rover: some roles cannot but others can, I see build-test-package very isoleted on its own11:28
ssbarnea|roveranother issue is that while we should aim to test roles in isolation, we may not be able to do this (primary due to the extra time needeD)11:28
quiquellssbarnea|rover: So you propose kind of molecule for integration testing more than role testing ?11:29
ssbarnea|roverquiquell: sure, build-test-package is probably a good example of something that we should be able to test isolated.11:29
quiquellssbarnea|rover: yep... I really want to put some molecule at just build-test-package and put a complext test suite with testinfra thingy11:30
sshnaidmideally all roles could be used isolated, it's not the case currently, but should be done in some point11:30
ssbarnea|roverquiquell: yep, i am more towards integration, mostly for practical reasons: testing being more close to how is used. also faster to add testing for a big piece of code.11:30
ssbarnea|roverisolated testing take considerable more time, not against it (at all)11:30
quiquellbreaking build-test-package is a pain in the ass11:30
sshnaidmand Alex merged some changed to build-test role to make it capable to run isolated11:30
quiquellssbarnea|rover: You need both integration and unit test11:31
ssbarnea|roverprobably the optimal way is somewhere in between: partially isolated pieces, but not pure mode.11:31
quiquellssbarnea|rover: you have to cover not all the code but at least all the algorithmic complexity of stuff11:31
ssbarnea|roverquiquell: nobody argues that. still we have limited amount of time and we cannot ever reach a perfect testing coverage. we need to find best trade-offs.11:32
quiquellssbarnea|rover: agree, even if we are going to deprecate quickstart.sh we don't know when, and we get experience with molecule11:33
quiquellssbarnea|rover: thanks for the effort btw11:33
ssbarnea|roverthere is one aspect of molecule that I didn't yet narrow dowsn: scenarios. this is the feature supposed to be used to slice testing. still, need to dig on it. i avoided it so far because it required to duplicate a lot of config.11:38
*** rfolco has joined #oooq11:38
quiquellssbarnea|rover: scenarios like build-test-package for old zuul and new zuul for example11:40
quiquellssbarnea|rover: so you can run the set test suite with different setups11:40
quiquellssbarnea|rover: something like that ?11:40
ssbarnea|roverquiquell: maybe. i was thinking more about a scenario for testing package building, one for testing qs.sh,...11:43
arxcruz|ruckssbarnea|rover: https://review.rdoproject.org/zuul/status so far so good, fingers crossed :D11:46
sshnaidmssbarnea|rover, do you know what is going on with https://review.openstack.org/#/c/620613/ ?11:51
sshnaidmssbarnea|rover, is bnemec ready to move?11:52
sshnaidmquiquell, why did you move base job to untrusted? https://review.rdoproject.org/r/#/c/18204/7/files/projects/zuul-config/zuul.d/jobs.yaml11:53
quiquellsshnaidm: This is not untrusted zuul-config is trusted project11:54
quiquellsshnaidm: This is just a duplication of the RDO 'config' project11:55
sshnaidmquiquell, I think I'm completely lost in this patch..11:55
quiquellsshnaidm: Is a little inception11:56
quiquellsshnaidm: The idea is to put at zuul-config (the repro trusted project) the needed base job to run the repro zuul jobs inside the repro11:57
*** ratailor__ has joined #oooq11:57
sshnaidmquiquell, is zuul-config copy of rdo config job?11:58
*** ratailor_ has quit IRC11:59
quiquellsshnaidm: kind of, we cannot import jobs fron 'config'11:59
*** ykarel is now known as ykarel|away12:00
sshnaidmquiquell, why not to clone config then?12:02
*** ratailor__ has quit IRC12:02
sshnaidmquiquell, we can use these jobs if they don't have secrets I think12:02
quiquellsshnaidm: playbooks are different12:02
quiquellsshnaidm: They have a lot of stuff upstream that we don't care12:03
quiquellsshnaidm: we already talked about it12:03
sshnaidmquiquell, I don't remember though, otherwise we'll need to maintain all in two places12:04
sshnaidmquiquell, for example we'll need to redefine all ovb stuff too12:04
*** holser_ is now known as holser|lunch12:04
quiquellsshnaidm: We can take a look at OVB base jobs stuff and improve it12:05
quiquellsshnaidm: but using directly base was not easy12:05
quiquellsshnaidm: I mean 'config' project12:05
sshnaidmquiquell, I thought about cloning it, removing all secrets from ovb jobs and pushing to local config/zuul-config or whatever12:06
quiquellsshnaidm: you mean at startup time12:07
quiquellsshnaidm: we can do that12:07
quiquellsshnaidm: adapt the base and push it12:07
sshnaidmquiquell, yeah, when starting12:07
quiquellsshnaidm: to push from ansible secrets is a good example12:07
*** ykarel|away has quit IRC12:07
quiquellsshnaidm: like it12:07
quiquellI have an ansible issue12:08
quiquellMaybe someone can help me12:08
quiquellThe filter "ipaddr" depends on python-netaddr12:08
quiquellBut if you install it from the same role that calls the filter it does not work12:08
quiquellif have to install it before12:09
quiquellto load ansible with python-netaddr12:09
sshnaidmquiquell, seems like need to install it before invoking ansible?12:11
quiquellsshnaidm: yep12:11
quiquellmake it fix the thing12:11
quiquellsshnaidm: it's related to http://git.openstack.org/cgit/openstack/tripleo-quickstart/tree/roles/libvirt/setup/overcloud/tasks/main.yml#n2212:11
quiquellsshnaidm: role approach does not work12:12
quiquellsshnaidm: installing it in the same approach12:12
sshnaidmquiquell, well, better to install reqs: http://git.openstack.org/cgit/openstack/tripleo-quickstart/tree/requirements.txt#n512:13
quiquellsshnaidm: This is bigger than just the libvirt role12:14
quiquellsshnaidm: we can comment it in the review, let me make it work with just pre install the package12:16
quiquellsshnaidm: hiting virt-resize issue at zuul job :-(12:21
sshnaidmquiquell, yeah, it's very annoying issue..12:23
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario007-multinode-oooq-container, tripleo-ci-centos-7-scenario003-standalone @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario003-standalone @ https://review.openstack.org/560445, stable/queens:  (1 more message)12:24
quiquellsshnaidm: humm I was running the expanded template with "command" let me run it with shell12:24
sshnaidmquiquell, maybe try to create script from template and run it..12:27
quiquellsshnaidm: already doing it12:27
quiquellsshnaidm: that's the issue12:27
quiquellsshnaidm: even with that12:27
*** jpena is now known as jpena|lunch12:27
arxcruz|ruckquiquell: sshnaidm have you guys seen this:12:28
arxcruz|ruck2019-01-14 11:54:42.068843 | primary | cp: cannot stat '/home/zuul/workspace/hash_info_fedora.sh': No such file or directory12:28
arxcruz|ruckhttp://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-fedora-28-centos-7-containers-standalone-master/9e2d22e/job-output.txt.gz12:28
arxcruz|ruckwondering if is a known issue or shall i open a bug12:28
arxcruz|ruckmarios: ^12:29
quiquellarxcruz|ruck: related to panda stuff about fedora and hashes I think12:29
quiquellarxcruz|ruck: let me find the review12:29
arxcruz|ruckquiquell: ok12:30
marios arxcruz|ruck ack though not seen that12:31
*** quiquell is now known as quiquell|lunch12:48
*** rlandy has joined #oooq12:58
*** quiquell|lunch is now known as quiquell13:05
*** ykarel has joined #oooq13:05
pandaarxcruz|ruck: does it fail ?13:12
pandaarxcruz|ruck: looking13:12
quiquellsshnaidm: calling the virt-resize.sh generated script from a created playbook working fine...13:18
quiquellsshnaidm: Something is broken by the role that breaks virt-resize :-(13:18
*** holser|lunch is now known as holser_13:20
ssbarnea|roverquiquell: virt-resize does not need much external help for breaking things ;)13:27
quiquellssbarnea|rover: yep man... I am geting crazy13:28
quiquellssbarnea|rover: call it manually out of the ansible role works fine, but within the role fails13:28
quiquellssbarnea|rover: calling withing the role after calling it manually works fine :-/13:28
quiquellssbarnea|rover: and also calling it from a custom playbook works fine too13:28
quiquellwhat a pain13:28
ssbarnea|roverquiquell: re resize, i asked last week but got no reply. I was wondering why not caching the already resized image instead of resizing it each time. we could save a good number of minutes.13:29
arxcruz|ruckpanda: yup, it fails13:29
quiquellssbarnea|rover: what is the already resized image ?13:31
*** jpena|lunch is now known as jpena13:31
rlandyquiquell: I am reading through https://review.rdoproject.org/r/#/c/18121/13:32
quiquellrlandy: thanks... I see force_tcg at virt-resize error13:32
quiquellrlandy: http://logs.rdoproject.org/21/18121/49/check/tripleo-ci-reproducer-fedora-28-libvirt/feef926/job-output.txt.gz13:33
ssbarnea|roverquiquell: just one exampe (there are more), https://github.com/openstack/tripleo-quickstart/blob/364c4ecce906f34fcc3eed7e16d2b79b25497509/roles/libvirt/setup/overcloud/tasks/libvirt_nodepool.yml#L20-L42 -- where is the idempotency on this?13:33
rlandyquiquell: well - talking about design here - you're allowing for both nodepool nodes and libvirt nodes13:33
quiquellrlandy: What do you mean ?13:34
quiquellrlandy: the idea is to use nodepool nodes backed up by libvirt with static nodepool provider13:35
quiquellrlandy: same you did but set up at start up13:35
quiquellrlandy: also test libvirt at zuul in the role13:35
rlandyquiquell: ok - so ... https://review.rdoproject.org/r/#/c/18121/49/README.md - nodepool_provider13:35
rlandywhen are the three options used?13:35
rlandynodepool - ovb and multinode/singlenode etc.13:36
rlandylibvirt - this case13:36
rlandyhost??13:36
quiquellthere is no nodepool option13:36
quiquellyou mean openstack ?13:36
rlandyyes - openstack13:36
rlandybut I know what that is13:36
rlandythe host vs. libvirt13:37
* rlandy comments on review13:37
rlandywe can discuss there13:37
quiquellso host is used for CI the reproducer role mostly (use openstack need some work and libvirt was not there yet)13:37
quiquellrlandy: when libvirt is working fine we can remove "host" option13:38
quiquellrlandy: host was put in place to ahve some CI in the role before libvirt implementation13:38
quiquellrlandy: so we test reproducer startup and the like13:38
pandaarxcruz|ruck: https://review.rdoproject.org/r/1825013:40
pandaarxcruz|ruck: uhm, not ready yet13:41
pandaarxcruz|ruck: nevermind, ready for revie13:41
pandaw13:41
pandaasd13:41
rfolcopanda, I see you created an etherpad and recorded steps to build f28 containers w/ openstack command. You don't mention kolla hacks for supporting fedora, does it work out-of-the-box ?13:42
pandarfolco: I don't know, I haven't tested. I've gathered the informations alex gave me13:43
rfolcopanda, ah, so yes, he might be assuming you applied his patches first13:44
pandarfolco: he pointed me to some patches, but he mentioned only (patches to make python3 work)13:45
panda""13:45
rfolcopanda, given that user story, I have a task in the f28 to try that method... should I move that task to #559 reproducer f28 container build ( by doc ) ?13:45
pandarfolco: wait.13:45
rfolcopanda, https://tree.taiga.io/project/tripleo-ci-board/task/573?kanban-status=144727813:46
rfolcopanda, this is about changes in the tripleo-buildcontainers playbook to support f28/py313:47
rfolcopanda, do not want to dup work if user story #559 is supposed to cove rit13:48
rfolcocover-it*13:48
rlandyquiquell: btw ... added this task ... https://tree.taiga.io/project/tripleo-ci-board/task/607?kanban-status=144727413:48
rlandywill start work on that this afternoon13:49
pandarfolco: I think we established this is out of scope for this sprint13:50
quiquellrlandy: nice!13:50
quiquellrlandy: this libvirt CI thing is killing me13:50
rfolcopanda, k will unpin thx13:53
sshnaidmarxcruz|ruck, ssbarnea|rover limit for ovb stacks was increased to 150 in CI today, so please keep an eye on it..13:54
arxcruz|rucksshnaidm: ack13:54
ssbarnea|roverok13:54
arxcruz|ruckssbarnea|rover: so far we have 2 failures on rocky release13:55
arxcruz|ruckthe other ovb jobs are either success or still running13:55
quiquellsshnaidm: the virt-resize issue is related to environment variables and kernel version ansible is doing something there13:55
sshnaidmquiquell, ack13:56
sshnaidmquiquell, does it work consistently with the script?13:56
quiquellsshnaidm: calling manually yes13:57
sshnaidmquiquell, but not from ansible?13:57
quiquellsshnaidm: nope13:57
quiquellsshnaidm: is like supermin trying to get the vmlinuz13:58
quiquellsshnaidm: in ansible it thinks the info is at environment variables and fails13:58
quiquell:-/13:58
sshnaidmhmm13:58
quiquellsshnaidm: comments13:59
quiquellhttps://tree.taiga.io/project/tripleo-ci-board/task/470?kanban-status=144727513:59
quiquellI have paste working and not working output from virt-size regarding supermin: kernel13:59
arxcruz|ruckssbarnea|rover: http://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-rocky/4cee386/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz14:02
arxcruz|rucknew failure14:02
arxcruz|rucktwo actually, one on swift docker image with the missing library libibverbs.so.114:02
arxcruz|ruckand on keystone db sync command with 2019-01-14 13:31:27 |         "sqlalchemy.exc.ArgumentError: Could not parse rfc1738 URL from string ''",14:03
quiquellsshnaidm: found Âit !!!14:06
quiquellthis is the issue http://git.openstack.org/cgit/openstack/tripleo-quickstart/tree/playbooks/libvirt-nodepool.yml#n2214:06
sshnaidmquiquell, is it it? https://github.com/openstack/tripleo-quickstart/commit/b469bc0923824303cf234a2f62e39abcbffdf3aa14:06
quiquellsshnaidm: We are seting the SUPERMIN varas to empty but they are set14:08
quiquellsshnaidm: this is wrong14:08
sshnaidmquiquell, seems like need to revert the commit above ^14:09
quiquellsshnaidm: well maybe default(omit) is enough14:12
quiquellLet me test that so we keep the functionality14:12
quiquellsshnaidm: well I am using the roles14:13
ssbarnea|roverquiquell: test about omit, not sure if is supported in this use-case.14:14
quiquellsshnaidm: not the playbook so I can just remove it14:14
ssbarnea|roverlets hope we do not have to endup with extra tasks to build the list of variables14:14
quiquellssbarnea|rover: humm ok14:14
*** vinaykns has joined #oooq14:18
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario007-multinode-oooq-container, tripleo-ci-centos-7-scenario003-standalone @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario003-standalone @ https://review.openstack.org/560445, stable/queens:  (1 more message)14:24
sshnaidmquiquell, but this creates problems with our old reproducer too, I'd remove it at all..14:24
sshnaidmquiquell, either "omit" if it works14:25
quiquellsshnaidm: Jut removing it, but I don't see the playbook just the roles14:25
quiquellsshnaidm: removing it from my review14:25
quiquellI mean I don't "use" the playbook just the roles14:25
quiquellI want to depend only on roles14:26
quiquellsshnaidm: removed the vars and it goes beyond the issue14:41
*** jfrancoa has quit IRC14:45
*** jfrancoa has joined #oooq14:58
sshnaidmthere is a zuul session in techtalks now, if someone is interested: https://bluejeans.com/191945600215:06
*** ykarel is now known as ykarel|away15:06
quiquellsshnaidm: thanks15:08
mariossshnaidm: thanks15:08
*** derekh has quit IRC15:11
*** derekh has joined #oooq15:21
*** apetrich has quit IRC15:25
*** apetrich has joined #oooq15:39
*** weshay has joined #oooq15:51
weshayWOW.. .that sucked15:52
weshayboston lab reset knocked out my bouncer15:52
weshay3+ years of uptime15:52
weshaylolz15:52
marioshow rude15:52
quiquellnow is the heaven of uptimes15:53
chandankumarDoes someone uses a permanent tunneling from RH openvpn to local network?15:53
quiquellweshay: what do we do with those dashboard-ci reviews you have with -1 ?15:53
*** apetrich has quit IRC15:54
weshayquiquell, ya.. I need to completely reset them... because they got into a merge conflict and w/ all that json.. it just is too hard15:55
weshaytried to get to that last week and failed15:55
quiquellweshay: ack15:55
weshaywill try again today.. since it's already off to a "start"15:55
chandankumarweshay: sshnaidm I donot know it is the right change for role discovery https://review.openstack.org/#/c/628421/5/ansible.cfg ?15:56
chandankumarneed help here15:56
weshaychandankumar, why do you need to specify roles/os_tempest15:57
weshaywhy not just drop it in roles15:57
weshaychandankumar, we need to chat about "how to add a role"15:57
weshaywe have doc on it, just not sure if it's up 2 date15:57
chandankumarweshay: what about tomorrow?15:59
chandankumarweshay: I will setup a call tomorrow by 6:00 P.M. ist if you are free?15:59
pandaquiquell: where is the session ? your BJ ?16:00
chandankumarweshay: https://review.openstack.org/#/c/629127/ with this change it dumps the role in venv path/share/ansible/roles16:01
chandankumarweshay: but the role is not getting detected so i putted that change http://logs.openstack.org/00/627500/39/check/tripleo-ci-centos-7-standalone-os-tempest/53d1b64/job-output.txt.gz#_2019-01-14_11_30_20_71292716:03
chandankumarweshay: invite sent, feel free to modify it16:05
*** apetrich has joined #oooq16:11
*** ykarel|away has quit IRC16:20
pandarfolco: updated etherpad to apply kolla patches16:23
pandarfolco: line 1216:23
rfolcopanda, nice16:23
rfolcopanda, that is... you Majesty16:24
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario007-multinode-oooq-container, tripleo-ci-centos-7-scenario003-standalone @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario003-standalone @ https://review.openstack.org/560445, stable/queens:  (1 more message)16:24
*** chandankumar is now known as codemonster16:31
*** gkadam has quit IRC16:33
*** kopecmartin is now known as kopecmartin|off16:34
*** rfolco is now known as rfolco|brb16:37
mariosweshay: o/ i want to reach out for other folks to check https://bugs.launchpad.net/tripleo/+bug/1811004 - I just added comment #9 rabbit credentials issue? & attachment false-positive errors so maybe deployment dfg. question is should i file a card in prod chain trello ?16:49
openstackLaunchpad bug 1811004 in tripleo "tripleo-ci-centos-7-scenario003-standalone fails tempest.scenario.test_minimum_basic.TestMinimumBasicScenario" [Critical,Triaged]16:49
*** codemonster is now known as chkumar|out16:49
mariosweshay: or perhaps pidone16:50
weshaysec16:58
pandarfolco|brb: so is the US done now ?16:59
*** ykarel|away has joined #oooq17:00
*** ykarel|away has quit IRC17:05
*** vinaykns has quit IRC17:13
*** dsneddon has joined #oooq17:18
mariosweshay: emailed. will follow up tomorrow17:19
*** dsneddon has quit IRC17:19
*** dsneddon has joined #oooq17:19
*** jfrancoa has quit IRC17:22
*** dtantsur is now known as dtantsur|afk17:23
*** bogdando has quit IRC17:28
rlandyweshay: btw: got the bm slave up and running again17:30
weshayah nice17:31
weshaythank you17:31
weshayrfolco|brb, panda is user story going to merge and close prior to rfolco|brb going on pto?17:32
rlandydepends on master promotion, no?17:35
rlandyI am ready to w+ the patches when you tell me17:35
*** ccamacho has quit IRC17:36
pandaweshay: if we want to wait for a master promotion, probably not17:40
weshaypanda, ah crap that's right.. thanks for reminding me17:41
weshayall start looking into those jobs17:41
weshayodd day17:41
pandaweshay: rlandy fedora mixed job passed but reported failure for a bug fixed here https://review.rdoproject.org/r/1825017:43
*** lyarwood has joined #oooq17:44
lyarwoodweshay: Hey, mandre suggested that the tripleoci team might be able to upload the new placement-api image to tripleomaster ahead of the next promotion to unblock some extraction tasks that I'm behind on. Would you know anyone I could talk to in the team about that?17:46
weshaylyarwood, I think anyone can push images17:47
weshaylyarwood, can you email me some details so we can try and help?17:47
lyarwoodweshay: yeah sure I can do that now, thanks!17:48
pandalyarwood: weshay you need credentials for the dockerhub account17:48
lyarwoodweshay: btw should I cc any particular list on this email?17:51
weshaylyarwood, panda is a good choice :)17:52
weshaygcerami@redhat.com17:53
* panda waves17:53
lyarwood^_^17:53
lyarwoodack thanks, panda, I'll send the details over now17:53
lyarwoodor we could just talk here17:53
lyarwoodhttps://review.openstack.org/#/c/626891/ landed 10 days ago in tripleo-common but without a promotion the required image hasn't been pulled into the reg as yet17:54
lyarwoodpanda: ^ that contains the image name etc is that enough for you to go on? Everything should be ready to go in Kolla etc.17:55
weshaya good reason to help make sure all the promotion-blockers are fixed :)17:55
lyarwoodweshay: I'm likely looking at the wrong dashboard but I don't see any issues for TripleO master CI at the moment ;)17:59
pandalol17:59
weshaylyarwood, ya.. cix board is probably the best view but which dashboard are you looking at?17:59
lyarwoodthe internal rhos-release one18:00
lyarwoodit's red, but has 0 issues listed18:00
pandaweshay: lyarwood uploading an image is easy if we already have it. If it was never built we may want to wait for a promotion18:00
pandaeven though I don't know if any job ibn the promotion is currently testing this container18:00
weshaylyarwood, ah ya.. we need to update that to read: stein18:00
lyarwoodnothing is going to test this container at the moment18:00
*** derekh has quit IRC18:00
weshaylyarwood, https://trello.com/b/j4IcIomh/production-chain-escalation18:00
lyarwoodweshay: ack thanks18:01
rlandypanda; looking18:02
pandalyarwood: the workflow for a new item is usually: make sure it's tested in some promotion job, then put it in check and eventually gate. right weshay ? not sure if we are applying this for new containers too18:03
pandabut if we upload without testing it in promotion, we may have eventually in our check/gates soemthing that is not fully tested18:04
weshaypanda, you are conflating new jobs with a new feature18:04
pandawith the rest18:04
weshayya.. the promotion pipeline would just guarantee any current job or tests do not break18:04
weshaywe are not indicating the $new feature is tested18:04
weshayor $new container18:04
pandaweshay: so is it ok to just buiild and upload the image now ?18:05
weshaypanda, I don't think so18:05
weshaynot unless we know for sure it does not break anything18:05
pandaok, that's what I was trying to express.18:06
lyarwoodright the fact that this image exists isn't going to break anything18:06
weshayk k18:06
pandalyarwood: so, yes, a stap would be to make one of the job in the promotion pipeline to tes the new container18:06
lyarwoodI can't see how that could work18:06
lyarwoodthe awkward thing here is that this is an existing service18:07
lyarwoodcurrently within Nova18:07
lyarwoodthat we are extracting into a seperate project for this cycle18:07
weshayya.. lyarwood it's best to just wait like everyone else18:07
weshaynot sure if you want to take the chance in breaking upstream jobs18:07
weshayI doubt it18:07
lyarwoodthere's zero chance of breaking upstream jobs if nothing is touching this new image18:08
lyarwoodas is the case at the moment18:08
lyarwoodbut sure, any idea when the next promotion would be?18:08
weshayok.. lyarwood raise it in the tripleo mtg tomorrow and if you get approval from the ptl and folks.. that's fine by me18:09
lyarwoodweshay: ah good point thanks18:09
pandalyarwood: in the meantime, if you have an image already built, once and if we get the green light I can pull it are reupload it to the repo very quickly18:10
pandas/are/and18:11
lyarwoodpanda: I don't to hand but could have one ready by the meeting tomorrow18:11
pandalyarwood: ok18:11
*** vinaykns has joined #oooq18:11
*** panda is now known as panda|off18:12
lyarwoodweshay / panda ; thanks for your help guys :)18:12
weshay:)18:12
*** jpena is now known as jpena|off18:12
weshayarxcruz|ruck, you still around?18:13
*** bogdando has joined #oooq18:13
*** bogdando has quit IRC18:13
*** rfolco|brb is now known as rfolco18:14
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario007-multinode-oooq-container, tripleo-ci-centos-7-scenario003-standalone @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario003-standalone @ https://review.openstack.org/560445, stable/queens:  (1 more message)18:24
*** chem has quit IRC18:38
*** saneax has quit IRC18:52
weshaypanda|off, ssbarnea|rover have you guys seen this? 2019-01-14 11:54:42.068843 | primary | cp: cannot stat '/home/zuul/workspace/hash_info_fedora.sh': No such file or directory18:55
weshayhttps://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-fedora-28-centos-7-containers-standalone-master/9e2d22e/job-output.txt.gz#_2019-01-14_11_54_42_06884318:55
* weshay checks bugs18:55
ssbarnea|rovernope. give me few mins to finish raising current patch and i will have a look18:56
weshayfyi https://bugs.launchpad.net/tripleo/+bug/181171518:58
openstackLaunchpad bug 1811715 in tripleo "cp: cannot stat '/home/zuul/workspace/hash_info_fedora.sh': No such file or directory" [Critical,Triaged]18:58
rlandyweshay: https://review.rdoproject.org/r/#/c/18250/19:10
* weshay looks19:11
rlandyfor above bug19:11
weshaynice19:11
weshayrlandy, is that right? https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-fedora-28-centos-7-containers-standalone-master/9e2d22e/logs/19:13
weshayre:cp {{ workspace }}/hash_info_fedora.sh {{ workspace }}/hash_info.sh19:14
rlandyweshay:sorry - right in waht aspect?19:14
weshayline 3219:14
rlandyweshay: looks like the two match to me19:15
weshay2019-01-14 11:54:42.068843 | primary | cp: cannot stat '/home/zuul/workspace/hash_info_fedora.sh': No such file or directory19:15
* rlandy gets comments19:15
rlandy<panda> weshay: rlandy fedora mixed job passed but reported failure for a bug fixed here https://review.rdoproject.org/r/1825019:15
weshayI guess this takes care of it19:15
weshay          cp {{ workspace }}/hash_info.sh {{ workspace  }}/hash_info_fedora.sh19:15
weshay+119:16
ssbarnea|roverweshay: do we need diff filename for fedora and centos? what is correct for the mixed job? centora? ;)19:17
weshaycentora19:17
weshaylolz19:17
weshayit's going away as soon as f28 containers are working19:17
weshayrfolco, sprint update to openstack-discuss?19:28
rfolcoweshay, sent on Friday...19:31
rfolcoweshay, oh are you asking me to send an update with ovb's correction ?19:33
weshayrfolco, got it http://lists.openstack.org/pipermail/openstack-discuss/2019-January/001679.html19:34
weshayrfolco, I think it missed my filter because openstack-dev@ is deprecated to openstack-discuss19:34
weshaythanks19:34
rfolcoweshay, ah ok cool19:35
weshayrfolco, aye.. thanks man :)19:35
*** holser_ is now known as holser|eod19:55
*** holser|eod has quit IRC19:55
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario003-standalone @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-scenario001-multinode-oooq-container @ https://review.openstack.org/602248, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario003-standalone @  (1 more message)20:24
rlandyweshay: any idea why this change was only made to minimal? https://review.openstack.org/#/c/583042 ( reading: https://bugs.launchpad.net/tripleo/+bug/1782165 )20:26
openstackLaunchpad bug 1782165 in tripleo "ntp servers blocked. Can't deploy undercloud with oooq due to wrong NTP configuration" [Critical,Incomplete] - Assigned to wes hayutin (weshayutin)20:26
rlandyI am testing this change on fs001 ... https://review.openstack.org/#/c/630759/20:27
arxcruz|ruckweshay: still around?20:27
arxcruz|ruckweshay: some of the bugs i open are not generating cards on cix, or updating the dashboard, is there something i am missing?20:28
rlandyarxcruz|ruck: did you add promotion-blocker?20:30
arxcruz|ruckrlandy++20:30
hubbot1arxcruz|ruck: rlandy's karma is now 4220:30
arxcruz|ruckno, i haven't20:30
arxcruz|ruckor i didn't20:30
arxcruz|rucki'll do that :)20:31
arxcruz|ruckrlandy: oh, wait, i did20:31
rlandyarxcruz|ruck: pls paste bug - I'll check20:31
rlandyalso  it take a while20:31
arxcruz|ruckrlandy: nevermind, i just saw on new/triage list one of them20:32
rlandyk20:32
arxcruz|ruckrlandy: shall i move it to critical pchain?20:32
* weshay looks20:32
arxcruz|ruckbecause the rhos release have master with 0 issues, but 13 days20:32
weshayarxcruz|ruck, master has a shit ton of issues20:34
weshayit's just using the wrong label20:34
arxcruz|ruckweshay: (╯°□°)╯︵ ┻━┻20:34
weshayrlandy, sorry.. slow today.. maybe I need more info on blue20:34
weshayarxcruz|ruck, go look at the cix board20:34
weshay:)20:34
* weshay will udpate the the rhos board label20:35
arxcruz|ruckweshay: yeah, i saw the issues today in the meeting, that's why i was wondering why the board was 13 days and 0 issues20:35
weshayarxcruz|ruck, the dashboard is master.. we moved cix from master to $release because the tracking was getting jacked20:36
arxcruz|ruckweshay: sorry, i didn't get it...20:37
* arxcruz|ruck come from german classes, takes a while to switch context properly :P20:37
weshayarxcruz|ruck, I'll show you20:37
weshayblue20:37
weshayarxcruz|ruck, https://code.engineering.redhat.com/gerrit/gitweb?p=rhosp-dashboards.git;a=tree20:40
rlandyweshay: np - I'll get back to you once the test completes20:40
arxcruz|ruckweshay: https://bugs.launchpad.net/tripleo/+bug/181168220:47
openstackLaunchpad bug 1811682 in tripleo "centos-binary-swift-object failed with cannot load glue library: libibverbs.so.1: cannot open shared object file: No such file or directory"" [Critical,Triaged]20:47
weshayrlandy, thanks for looking into that20:53
weshayhttp://codesearch.openstack.org/?q=net-config-simple-bridge.yaml&i=nope&files=&repos=20:53
weshayvs.20:53
weshayhttp://codesearch.openstack.org/?q=net-config-undercloud.yaml&i=nope&files=&repos=20:53
weshayrlandy, I'll ask in #tripleo20:54
*** hamzy_ has joined #oooq21:03
*** hamzy has quit IRC21:11
rlandyit may not help - I'm testing21:17
weshayrlandy, it's a little troubling to see that it's really split 50/50 on the use of both21:39
weshayI think I was just TOLD to use the one in minimal21:40
rlandyweshay: it may not make a difference21:40
rlandyI dont have result yet to prove one way or another21:40
rlandyI am just going off the comment in the bug that it helped21:40
rlandyugh - stupid check job fails introspection :(22:01
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario003-standalone @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-scenario001-multinode-oooq-container @ https://review.openstack.org/602248, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario003-standalone @  (1 more message)22:25
*** vinaykns has quit IRC22:26
*** rascasoft has quit IRC23:10
*** rascasoft has joined #oooq23:21
*** rlandy has quit IRC23:25
*** rascasoft has quit IRC23:26
*** rfolco has quit IRC23:45
*** rfolco has joined #oooq23:45

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!