| *** tosky has quit IRC | 00:02 | |
| *** dsneddon has joined #oooq | 00:03 | |
| *** dsneddon has quit IRC | 00:13 | |
| *** ssbarnea|rover has quit IRC | 00:13 | |
| *** ssbarnea|rover has joined #oooq | 00:17 | |
| *** saneax has quit IRC | 00:26 | |
| *** matbu has quit IRC | 00:39 | |
| *** dsneddon has joined #oooq | 00:40 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-containerized-undercloud-upgrades, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset022, tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph- (2 more messages) | 00:48 |
|---|---|---|
| *** dsneddon has quit IRC | 00:51 | |
| *** rlandy has quit IRC | 00:51 | |
| *** dsneddon has joined #oooq | 01:00 | |
| *** hamzy has joined #oooq | 01:04 | |
| *** dsneddon has quit IRC | 01:04 | |
| *** dsneddon has joined #oooq | 01:37 | |
| *** dsneddon has quit IRC | 01:57 | |
| *** dsneddon has joined #oooq | 01:59 | |
| *** dsneddon has quit IRC | 02:12 | |
| *** dsneddon has joined #oooq | 02:12 | |
| *** dsneddon has quit IRC | 02:26 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-containerized-undercloud-upgrades, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024, tripleo-ci-centos-7-ovb-1ctlr_1comp- (2 more messages) | 02:48 |
| *** dsneddon has joined #oooq | 02:55 | |
| *** dsneddon has quit IRC | 03:03 | |
| *** dsneddon has joined #oooq | 03:14 | |
| *** apetrich has quit IRC | 03:15 | |
| *** dsneddon has quit IRC | 03:23 | |
| *** dsneddon has joined #oooq | 03:30 | |
| *** skramaja has joined #oooq | 03:32 | |
| *** dsneddon has quit IRC | 03:36 | |
| *** ykarel|away has joined #oooq | 03:43 | |
| *** dsneddon has joined #oooq | 03:44 | |
| *** dsneddon has quit IRC | 03:51 | |
| *** dsneddon has joined #oooq | 04:28 | |
| *** dsneddon has quit IRC | 04:33 | |
| *** chkumar|out is now known as chandankumar | 04:38 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053, tripleo-ci-centos-7-containerized-undercloud-upgrades, tripleo-ci-centos-7-undercloud-containers, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ (3 more messages) | 04:48 |
| *** agopi has quit IRC | 04:50 | |
| *** dsneddon has joined #oooq | 05:05 | |
| *** dsneddon has quit IRC | 05:11 | |
| *** ykarel|away has quit IRC | 05:20 | |
| *** ykarel|away has joined #oooq | 05:32 | |
| *** ykarel|away is now known as ykarel | 05:33 | |
| *** udesale has joined #oooq | 05:34 | |
| chandankumar | quiquell|off: marios \o/ | 05:35 |
| chandankumar | quiquell|off: marios https://review.openstack.org/#/c/633185/ keystone specific workaround no longer needed as os_tempest gates are fixed | 05:36 |
| chandankumar | quiquell|off: marios we will get the necessary patches landed there | 05:36 |
| *** dsneddon has joined #oooq | 05:41 | |
| *** agopi has joined #oooq | 05:45 | |
| *** udesale has quit IRC | 05:46 | |
| *** udesale has joined #oooq | 05:48 | |
| *** udesale has quit IRC | 05:53 | |
| *** udesale has joined #oooq | 05:54 | |
| *** quiquell|off is now known as quiquell | 06:39 | |
| quiquell | chandankumar: o/ | 06:39 |
| quiquell | chandankumar: cool, let's abandon that | 06:39 |
| marios | o/ folks | 06:39 |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053, tripleo-ci-centos-7-containerized-undercloud-upgrades, tripleo-ci-centos-7-undercloud-containers, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ (3 more messages) | 06:48 |
| *** udesale has quit IRC | 07:02 | |
| *** matbu has joined #oooq | 07:03 | |
| *** udesale has joined #oooq | 07:05 | |
| *** udesale has quit IRC | 07:06 | |
| *** udesale has joined #oooq | 07:08 | |
| *** udesale has quit IRC | 07:09 | |
| *** udesale has joined #oooq | 07:09 | |
| *** jfrancoa has joined #oooq | 07:15 | |
| quiquell | sshnaidm: o/ | 07:17 |
| quiquell | sshnaidm: Do you know what's the difference between "rdo-cloud" and "rdo-cloud-tripleo" ? | 07:17 |
| quiquell | sshnaidm: Looks like we have two kind of cloud names for RDO | 07:17 |
| *** saneax has joined #oooq | 07:18 | |
| *** kopecmartin|off is now known as kopecmartin | 07:26 | |
| marios | quiquell: can't spot any nits in https://review.rdoproject.org/r/#/c/18569/9 still having problems there (added some thoughts but can't spot any issues yet) | 07:27 |
| quiquell | marios: we are still hit by cloudflare issue, this is the fix https://review.rdoproject.org/r/18621 | 07:29 |
| quiquell | marios: Forgot we have two cloud names for RDO | 07:29 |
| marios | quiquell: didn't know that (do you know why/what we have 2 for?) | 07:30 |
| *** ykarel is now known as ykarel|lunch | 07:30 | |
| quiquell | marios: I think it's related to OVB | 07:30 |
| quiquell | marios: always forget | 07:30 |
| quiquell | :-/ | 07:30 |
| marios | k | 07:30 |
| quiquell | marios: can you review/merge the stuff, and we focus on dryrun I think we can even run it at host | 07:31 |
| marios | ack | 07:31 |
| marios | quiquell: what needs merging | 07:31 |
| quiquell | marios: https://review.rdoproject.org/r/18621 | 07:31 |
| quiquell | marios: to use RDO docker registry for rdo-cloud-tripleo too | 07:32 |
| marios | qk | 07:32 |
| sshnaidm | quiquell, yeah, we have and there is no difference | 07:34 |
| quiquell | sshnaidm: Don't we have it there for OVB ? | 07:34 |
| quiquell | sshnaidm: or something with credentials ? | 07:34 |
| sshnaidm | quiquell, nope | 07:34 |
| quiquell | sshnaidm: now that we are reworking OVB maybe is no longer needed | 07:35 |
| sshnaidm | quiquell, it's not related to ovb | 07:35 |
| quiquell | sshnaidm: ack, will ask at RDO maybe they can remove it | 07:35 |
| sshnaidm | quiquell, did you see this beauty? https://logs.rdoproject.org/58/18558/23/check/tripleo-ci-reproducer-fedora-28-libvirt/f8aeab5/tripleo-ci-reproducer/virt-resize.log | 07:37 |
| quiquell | sshnaidm: so we have to use raw ? | 07:38 |
| quiquell | sshnaidm: so we don't get hit by bug ? | 07:38 |
| sshnaidm | quiquell, why raw? | 07:38 |
| quiquell | sshnaidm: It's reading a qcow2 as raw | 07:39 |
| quiquell | sshnaidm: that's the failure | 07:39 |
| sshnaidm | quiquell, it seems like kvm bug, related to nested | 07:39 |
| sshnaidm | quiquell, no, it's a message not related | 07:39 |
| quiquell | sshnaidm: '/tmp/libguestfsm1bvyN/overlay1.qcow2' and probing guessed raw. | 07:39 |
| quiquell | sshnaidm: this does not look good | 07:39 |
| sshnaidm | quiquell, it's just not related warning | 07:40 |
| quiquell | sshnaidm: ack | 07:40 |
| sshnaidm | quiquell, I think we already had problems with nested there, ykarel|lunch jpena|off - is it right? | 07:40 |
| quiquell | sshnaidm: so we can detect the issue and do fallback to force_foobar | 07:40 |
| sshnaidm | quiquell, that's why we still run libvirt jobs on ci.centos | 07:40 |
| quiquell | nested is not production wise I think, it's like use at your own responsability | 07:41 |
| sshnaidm | quiquell, maybe we can move there.. | 07:41 |
| quiquell | sshnaidm: I am testing the force flag now, for fedora28 it takes same time, for centos it timesout | 07:41 |
| sshnaidm | quiquell, I think it's worth to try next sprint to define a job on ci.centos, to see how it goes there | 07:42 |
| quiquell | why ci.centos ? | 07:42 |
| sshnaidm | quiquell, there are other problems, but libvirt part works fine there | 07:42 |
| quiquell | It depends on hypervisor | 07:42 |
| quiquell | So maybe there are right in that front | 07:42 |
| sshnaidm | quiquell, because we have there some space and libvirt works fine | 07:42 |
| quiquell | ci.centos means jenkins ? | 07:42 |
| sshnaidm | quiquell, yeah | 07:43 |
| quiquell | argg | 07:43 |
| sshnaidm | yeah | 07:43 |
| quiquell | sshnaidm: don't we have a RDO replacement or vexxhost ? | 07:43 |
| quiquell | maybe at vexxhost it works fine | 07:43 |
| sshnaidm | quiquell, we have vexxhost a little bit, but not sure what is with nested there. | 07:43 |
| quiquell | sshnaidm: how do we test at vexxhost or we cannot use it directly ? | 07:44 |
| sshnaidm | quiquell, we don't use nested in our jobs | 07:45 |
| sshnaidm | quiquell, only in ci.centos | 07:45 |
| quiquell | Don't want to play with jenkins | 07:45 |
| quiquell | I am in love with zuul now | 07:45 |
| sshnaidm | quiquell, I think weshay wanted to set up libvirt jobs on rdo cloud but it didn't work because of these problems, iirc | 07:45 |
| quiquell | sshnaidm: also do we have openshift ? | 07:45 |
| sshnaidm | quiquell, let's discuss on meeting | 07:45 |
| quiquell | sshnaidm: or something like that ? | 07:46 |
| sshnaidm | quiquell, no, we have some tenant in internal cloud | 07:46 |
| quiquell | sshnaidm: or maybe just don't test it in centos7 | 07:46 |
| sshnaidm | upshift | 07:46 |
| quiquell | since we are moving to fedora28 | 07:46 |
| quiquell | upshift | 07:46 |
| quiquell | what about upshift ? | 07:46 |
| sshnaidm | quiquell, it's in internal rh network and rdo can't work with it still | 07:46 |
| sshnaidm | quiquell, apevev promised to look into it how to connect internal clouds, but didn't hear from him about it | 07:47 |
| quiquell | sshnaidm: so let's comment | 07:47 |
| sshnaidm | quiquell, and that's a pity.. we could run there ovb reproducer jobs | 07:47 |
| quiquell | sshnaidm: It's importnat we tesst libvirt reproducer at centos7 ? | 07:47 |
| quiquell | sshnaidm: :-( | 07:48 |
| sshnaidm | quiquell, I don't know, but better to test it anywhere than not to test it | 07:48 |
| quiquell | sshnaidm: I mean test it just at fedora28 | 07:48 |
| quiquell | sshnaidm: since we are moving to something like it | 07:48 |
| quiquell | sshnaidm: so we have solid CI at least | 07:48 |
| sshnaidm | quiquell, if something works like f28 - that's fine, it will check our workflow, for centos7 we can try to set up something in ci.centos later | 07:48 |
| quiquell | sshnaidm: yep, let's comment it in the meeting | 07:49 |
| sshnaidm | quiquell, I would test on centos still because of different versions of packages like libvirt, etc, just to ensure it still works | 07:50 |
| sshnaidm | quiquell, and python2 | 07:50 |
| sshnaidm | quiquell, I don't believe we can break something else centos specific | 07:50 |
| *** gkadam has joined #oooq | 07:51 | |
| *** ccamacho has joined #oooq | 07:52 | |
| quiquell | marios, sshnaidm, chandankumar: Can you help me spot the issue here https://review.rdoproject.org/r/#/c/18569 ? zuul is not helping me and locally it works | 07:54 |
| sshnaidm | quiquell, ansible_user_dir ? | 07:55 |
| marios | quiquell: yeah i was thikning same yesterday sshnaidm i.e. maybe the user_dir /me looks again | 07:59 |
| marios | quiquell: is a bit confusing now with the keys | 07:59 |
| quiquell | what's the issue with that ? | 07:59 |
| marios | user_key, gerrit_key, rdo_key | 07:59 |
| quiquell | marios: I have just answered | 07:59 |
| quiquell | marios: well gerrit ones default to user_key | 07:59 |
| sshnaidm | quiquell, this is the error: http://paste.openstack.org/show/744232/ | 07:59 |
| quiquell | marios: people will usually need to create new one without password and with -m PEM and set user_key | 07:59 |
| sshnaidm | quiquell, https://review.rdoproject.org/zuul/stream/5c9bb6469f2844a39869e2483694f77e?logfile=console.log | 08:00 |
| quiquell | marios: gerrit ones will default to it | 08:00 |
| *** jtomasek has joined #oooq | 08:00 | |
| marios | quiquell: sure man just saying i mean lets get it into working state and that part is mostly about documenting what you need to set vs optional/defaults | 08:00 |
| marios | quiquell: user_key is undefined ? did you include it in vars :) | 08:00 |
| quiquell | marios: fuck... this is out of the role :-/ | 08:01 |
| quiquell | marios: that's why it was working for me locally :-/ | 08:01 |
| quiquell | sshnaidm: thanks so much | 08:01 |
| sshnaidm | quiquell, seems like user_key clashes with some zuul roles | 08:02 |
| quiquell | I have do too much of find and replace | 08:02 |
| marios | sshnaidm: yeah i was just thinkign that i wonder if user_key is reserved | 08:02 |
| quiquell | sshnaidm: is just that I have use user_key only at role stuff note zuul stuff | 08:02 |
| quiquell | sshnaidm: at zuul we know where they keys are | 08:02 |
| quiquell | Humm | 08:04 |
| quiquell | nope | 08:04 |
| sshnaidm | interesting if we can set up a gerrit user with read only permissions maybe, or just a gerrit user, and use it in all this stuff | 08:05 |
| sshnaidm | we don't really need our user to be used | 08:05 |
| sshnaidm | maybe could be a problem 64 connection.. | 08:06 |
| quiquell | sshnaidm: yep, also the 64 connections issue can be a problem with CI | 08:06 |
| quiquell | :-/ | 08:06 |
| quiquell | sshnaidm: would be nice to be able to use zuul without stream permissions | 08:06 |
| quiquell | sshnaidm: zuul with anonymous user | 08:06 |
| sshnaidm | quiquell, not sure will be accepted anon users, it's a whole | 08:07 |
| sshnaidm | a hole :) | 08:07 |
| quiquell | sshnaidm: What I test first was using git connection instead of gerrit | 08:07 |
| sshnaidm | quiquell, but all registered should be able to listen | 08:07 |
| quiquell | sshnaidm: but then we don't have Depends-On and all the normal workflow | 08:07 |
| quiquell | marios, sshnaidm: Also this is good https://review.rdoproject.org/r/#/c/18613/ | 08:12 |
| quiquell | It prints the "finger uuid@localhost" comand to stream stuff from command line | 08:12 |
| quiquell | Also waits for UUID to appear | 08:12 |
| marios | quiquell: i was about to vote and you updated | 08:14 |
| marios | quiquell: but lgtm | 08:14 |
| marios | checking v3 | 08:14 |
| quiquell | sshnaidm: fedora28 timeout at virt-resize <- http://logs.rdoproject.org/39/18539/5/check/tripleo-ci-reproducer-fedora-28-libvirt/982570c/job-output.txt.gz | 08:14 |
| quiquell | sshnaidm: nested is a pain in the ass | 08:14 |
| quiquell | marios: yep jus found the poll thing | 08:14 |
| quiquell | marios: that was part of an experiment about calling the command with async | 08:14 |
| quiquell | marios: but no good for MVP | 08:15 |
| quiquell | marios: sorry about the quick fix | 08:15 |
| marios | quiquell: ack, why did you have poll i thought it is only with 'async' | 08:16 |
| marios | quiquell: anyway is gone now | 08:16 |
| sshnaidm | quiquell, maybe we don't need to resize in ci | 08:16 |
| quiquell | marios: forgot to remove it :-/ | 08:16 |
| quiquell | sshnaidm: but then it will fail at virt-customize I think | 08:17 |
| sshnaidm | quiquell, yeah.. | 08:17 |
| quiquell | sshnaidm: nested is not for production, I think I read it somwhere | 08:17 |
| quiquell | sshnaidm: what baremetal options do we have ? | 08:17 |
| quiquell | sshnaidm: it's going to timeout here too with f28 https://softwarefactory-project.io/zuul/t/rdoproject.org/stream/4d5ff28d681a4af2a876a03548fde121?logfile=console.log | 08:19 |
| quiquell | sshnaidm: it has the force_fcg stuff | 08:19 |
| quiquell | marios: dryrun working at centos7 host http://logs.rdoproject.org/39/18539/5/check/tripleo-ci-reproducer-centos-7-host/fa53229/tripleo-ci-reproducer/logs/01/1001/1/check/tripleo-ci-centos-7-standalone-reprozuul-dryrun/1b2469f/job-output.txt.gz | 08:20 |
| *** ykarel|lunch is now known as ykarel | 08:22 | |
| marios | quiquell: nice | 08:22 |
| ykarel | sshnaidm, atleast in Fedora we have issues running in RDO Cloud, we are running Fedora jobs without enabling nested_virtualization, In CentOS also we see issues some times, | 08:23 |
| quiquell | marios: hehe I think we have found a f28 standalone issue at RDO :-/ | 08:23 |
| quiquell | marios: we need to create /etc/nodepool dir the the images they use does not have it :-) | 08:24 |
| ykarel | sshnaidm, i have not raised the issue for sometime, i think need to raise it again with kvm info, so kvm guys can help in sorting this out | 08:24 |
| ykarel | last time kvm info was missing, so bug was closed | 08:24 |
| *** apetrich has joined #oooq | 08:25 | |
| quiquell | sshnaidm: so maybe we have to add "openstack" nodepool_driver to this tests and put libvirt elsewhere ? | 08:28 |
| sshnaidm | quiquell, what do you mean? to run tests on some tenant? | 08:29 |
| sshnaidm | quiquell, we have this test tenant actually | 08:29 |
| quiquell | sshnaidm: Didn't have time to create CI for openstack nodepool_provider | 08:29 |
| quiquell | sshnaidm: It needs secrets and the like | 08:29 |
| quiquell | sshnaidm: And we have to see what happends with multipleo nodepool towards same tenant | 08:30 |
| sshnaidm | quiquell, yeah | 08:30 |
| quiquell | Next sprint | 08:30 |
| sshnaidm | quiquell, if they has the same metadata, will remove vms of each other | 08:30 |
| quiquell | sshnaidm: Yep we have to mark them somehow | 08:31 |
| quiquell | sshnaidm: related to job execution or change id or the like | 08:32 |
| quiquell | puff f28 libvirt is timing out now like always | 08:35 |
| quiquell | :-/ | 08:35 |
| quiquell | sshnaidm: we can try to bypass virt-resize for CI ? | 08:35 |
| sshnaidm | quiquell, yes, but you say still need to use virt-customize? | 08:40 |
| sshnaidm | quiquell, it's the same libvirt guest stuff | 08:41 |
| quiquell | sshnaidm: yep | 08:41 |
| quiquell | sshnaidm: damn... | 08:41 |
| sshnaidm | ci.centos :D | 08:41 |
| quiquell | sshnaidm: argg don't want oo | 08:41 |
| sshnaidm | the old good jenkins | 08:41 |
| sshnaidm | 1.0 | 08:41 |
| sshnaidm | nice ui and not overloaded interface at all | 08:42 |
| *** gkadam has quit IRC | 08:42 | |
| quiquell | sshnaidm: Have an idea with openstck nodepool_driver | 08:44 |
| quiquell | nodepoo_provider | 08:44 |
| quiquell | sshnaidm: what about usinz zuul queue for those jobs ? | 08:45 |
| quiquell | sshnaidm: so they run one after another ? | 08:45 |
| marios | ykarel: woo looks like standalone periodics started running again https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic&job_name=periodic-tripleo-ci-centos-7-scenario001-standalone-master | 08:46 |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053, tripleo-ci-centos-7-containerized-undercloud-upgrades, tripleo-ci-centos-7-undercloud-containers, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ (3 more messages) | 08:48 |
| quiquell | panda: o/ | 08:50 |
| quiquell | panda: Aren't we calling "zuul autohold" in this sprint ? | 08:50 |
| quiquell | panda: https://tree.taiga.io/project/tripleo-ci-board/task/569?kanban-status=1447276 | 08:50 |
| ykarel | marios, yes parent container-build job is fine now | 08:54 |
| ykarel | quiquell, sshnaidm so just checked, LIBGUESTFS_BACKEND_SETTING setting is not at all working | 08:54 |
| ykarel | because a new line is inserted | 08:55 |
| quiquell | ykarel: damn... | 08:55 |
| ykarel | quiquell, sshnaidm force_tcg\n vs force_tcg | 08:55 |
| ykarel | so set it correctly and it should work | 08:55 |
| *** tosky has joined #oooq | 08:55 | |
| *** sanjayu_ has joined #oooq | 08:56 | |
| *** sanjayu_ has quit IRC | 08:57 | |
| *** jpena|off is now known as jpena | 08:57 | |
| quiquell | ykarel: thanks again | 08:57 |
| sshnaidm | ykarel, hmm.. didn't think newline would break environment variable | 08:59 |
| ykarel | sshnaidm, it's breaking :( | 08:59 |
| ykarel | may be libguestfs string compares | 09:00 |
| ykarel | force_tcg == force_tcg\n | 09:00 |
| ykarel | i tried locally force_tcg\n doesn't work | 09:00 |
| quiquell | ykarel: also I miss some virt-customize there | 09:01 |
| quiquell | ykarel: let's fix all | 09:01 |
| ykarel | quiquell, ack | 09:02 |
| quiquell | sshnaidm: maybe we are still good and we don't need ci.centos | 09:03 |
| quiquell | ykarel, sshnaidm: https://review.openstack.org/633444 | 09:03 |
| ykarel | quiquell, also redirect stderr | 09:04 |
| ykarel | otherwise timeout logs will not be there in virt-resize.log | 09:04 |
| ykarel | i added last night | 09:04 |
| quiquell | all of them you mean | 09:05 |
| quiquell | ack | 09:05 |
| quiquell | will create virt-customize.log | 09:05 |
| quiquell | ykarel: recheck https://review.openstack.org/633444 | 09:07 |
| ykarel | quiquell, i meant https://review.openstack.org/#/c/633444/6..8/roles/libvirt/setup/overcloud/tasks/libvirt_nodepool.yml line 36 | 09:08 |
| sshnaidm | cool, let's see | 09:08 |
| ykarel | u changed &> --> > | 09:08 |
| ykarel | &> is required to put stderr to file | 09:09 |
| ykarel | to catch logs if those tasks timesout | 09:09 |
| sshnaidm | ykarel, 2>&1 > will work too | 09:09 |
| quiquell | Yep | 09:09 |
| ykarel | sshnaidm, not working | 09:09 |
| ykarel | you can check previous logs | 09:10 |
| sshnaidm | ykarel, how is that? | 09:10 |
| ykarel | or can try locally | 09:10 |
| quiquell | sh vs bash ? | 09:10 |
| sshnaidm | always worked for me | 09:10 |
| sshnaidm | damn.. didn't work now | 09:12 |
| ykarel | sshnaidm, can u try now | 09:12 |
| ykarel | yes it's not working | 09:12 |
| sshnaidm | ykarel, can you try if " > file 2>&1" works? | 09:13 |
| ykarel | sshnaidm, yes it works | 09:15 |
| ykarel | sshnaidm, why not just &> | 09:15 |
| sshnaidm | ok, so it should be after | 09:15 |
| ykarel | ^^ also works | 09:15 |
| sshnaidm | ykarel, yeah, agree, just curious | 09:15 |
| quiquell | ssbarnea|rover: have a question here https://review.rdoproject.org/r/#/c/18517 | 09:20 |
| *** bogdando has joined #oooq | 09:22 | |
| quiquell | So I move it after redirection | 09:23 |
| ssbarnea|rover | quiquell: replied/ | 09:25 |
| *** saneax has quit IRC | 09:25 | |
| ssbarnea|rover | ykarel: sshnaidm please try to avoid the &> construct as is not supported on all platforms shells (or even older bash). Plain old recipe is fully portable: 2>&1 >file | 09:26 |
| quiquell | ssbarnea|rover: the cleanup stuff is well tested ? | 09:26 |
| quiquell | ykarel, sshnaidm: is ok now ? https://review.openstack.org/633444 | 09:27 |
| quiquell | ssbarnea|rover: ^ | 09:27 |
| *** saneax has joined #oooq | 09:27 | |
| panda | quiquell: we are ? | 09:28 |
| ssbarnea|rover | quiquell: yep, manually run the deployment playbook on production, login to te-broker and run: journalctl -t ovb-tenant-cleanup | 09:29 |
| quiquell | panda: btw CI fails with the post thing | 09:29 |
| quiquell | ssbarnea|rover: ack, thanks | 09:29 |
| quiquell | panda: we have to figure it how to check that since job now fails at post :-/ | 09:29 |
| ssbarnea|rover | you will see a nice log of hourly cleanups, and it seems that during business hours we always have somethign to clean. only during night it did not find lefovers. | 09:29 |
| ykarel | quiquell, using redirect with command module in some task | 09:29 |
| panda | quiquell: calling autohold is missing ? Isn't that something that the user does ? | 09:29 |
| ykarel | quiquell, redirection is fine now | 09:29 |
| quiquell | ykarel: just found :-/ | 09:29 |
| quiquell | ykarel: let me fix | 09:29 |
| panda | quiquell: also there were not jobs last time I tried | 09:30 |
| ssbarnea|rover | panda: thanks for the hint with journalctl, systemd-cat is a very useful command! | 09:30 |
| panda | quiquell: checking | 09:30 |
| quiquell | panda: just rebased the stuff | 09:30 |
| panda | ssbarnea|rover: yes, sorry I wanted to talk with you about systemd-cat | 09:30 |
| panda | ssbarnea|rover: but yesterday I was completely gone | 09:30 |
| quiquell | panda: now the CI launch job so they fail at post, we should "parse" the failure if it's not related to post hack | 09:30 |
| panda | quiquell: I'll take care of it | 09:31 |
| quiquell | panda: anyway lgtm I mean is working as expected just CI is fixing and we can document the call autohold thing | 09:32 |
| quiquell | s/is/need/ | 09:33 |
| panda | quiquell: ok, I can put it anywhere, is it ok if I choose a random point in the README file ? | 09:34 |
| sshnaidm | quiquell, should be fine | 09:34 |
| ssbarnea|rover | sshnaidm: login to te-broker and run the script from root home directory as that is the deployed one. i wanted to paste the DRY mode but it contains credentials :D | 09:34 |
| sshnaidm | ssbarnea|rover, which bash versions don't support &> ? | 09:34 |
| sshnaidm | ssbarnea|rover, than source script with set +x | 09:35 |
| sshnaidm | ssbarnea|rover, no logs should contain credentials! | 09:35 |
| quiquell | panda: Maybe the README but the role the generates the scipt ? | 09:35 |
| ssbarnea|rover | sshnaidm: sure, I will do this to avoid the leak. the official logs do not have this problem, only the dry one but i will fix it for both. | 09:36 |
| sshnaidm | ssbarnea|rover, and don't forget to address review issues please | 09:36 |
| sshnaidm | ssbarnea|rover, *before* you upload the new patchset | 09:37 |
| sshnaidm | ssbarnea|rover, *with* comments | 09:37 |
| ssbarnea|rover | sshnaidm: bash < 4 does not support &> if i remember well, or it was zsh / not sure. | 09:37 |
| sshnaidm | ssbarnea|rover, no, it was supported in bash 3 | 09:37 |
| panda | quiquell: where are we putting the user related stuff ? in the generator or the role ? | 09:38 |
| quiquell | panda: I think in the generator | 09:38 |
| quiquell | panda: https://review.openstack.org/#/c/631067/38/roles/create-zuul-based-reproducer/templates/README-reproducer-zuul-based-quickstart.html.j2 | 09:39 |
| panda | ssbarnea|rover: bash 4 was released more than 7 years ago | 09:39 |
| ssbarnea|rover | sshnaidm: yep works with both old bash and zsh, i don't rememeber where I got the error as I am sure I encountered it from time to time during last year. | 09:40 |
| sshnaidm | ssbarnea|rover, what's about kernel 2.4? | 09:40 |
| sshnaidm | quiquell, f28 resized during 2 minutes.. | 09:43 |
| ssbarnea|rover | sshnaidm: regarding the unrelated fixes, i will remove them, Still the reason why they were part of this patch was that i seen the warnings while testing the te-broker role, thus I fixes them at the same time. | 09:43 |
| sshnaidm | ssbarnea|rover, in your time: https://www.freshconsulting.com/atomic-commits/ | 09:44 |
| quiquell | sshnaidm: that's good I suppose | 09:44 |
| quiquell | sshnaidm: let's see virt-customize stuff :-/ I have also increase the timeout 2 hours | 09:45 |
| sshnaidm | quiquell, yeah, either tcg is not set again | 09:45 |
| quiquell | for libvirt | 09:45 |
| quiquell | sshnaidm: we will see in the logs | 09:45 |
| quiquell | what a pain... | 09:45 |
| ssbarnea|rover | sshnaidm: no nee to click the link, i know what you mean. lets hope i will not enough doing subatomic commits like the spellchecks fixes we get from china based bots ;) | 09:45 |
| quiquell | damn.. don't know what's the issue here really https://review.rdoproject.org/r/#/c/18569/ | 09:46 |
| quiquell | zuul fails at totally unrelated stuff | 09:46 |
| sshnaidm | quiquell, hmm.. https://logs.rdoproject.org/58/18558/26/check/tripleo-ci-reproducer-fedora-28-libvirt/f2d4af5/job-output.txt.gz#_2019-01-30_09_44_17_703752 | 09:46 |
| quiquell | https://paste.fedoraproject.org/paste/aPKvEdOwCHRT-r3mHt35~Q | 09:46 |
| sshnaidm | quiquell, you need to rename user_key I think | 09:47 |
| quiquell | sshnaidm: maybe is used by zuul or something ? | 09:47 |
| ykarel | but LIBGUESTFS_BACKEND_SETTING is working now as epected | 09:47 |
| sshnaidm | quiquell, yes | 09:47 |
| ykarel | -machine accel=tcg | 09:47 |
| quiquell | how lucky I am | 09:47 |
| sshnaidm | virt-customize: error: libguestfs error: open: ~/.ssh/id_rsa.pub: No such | 09:48 |
| sshnaidm | file or directory | 09:48 |
| sshnaidm | virt-customize really doesn't like ~ | 09:49 |
| sshnaidm | iirc it requires either full path or $HOME | 09:49 |
| quiquell | sshnaidm: but why now ? | 09:50 |
| quiquell | sshnaidm: maybe the force thing ? | 09:50 |
| panda | because it's the end of the sprint | 09:50 |
| sshnaidm | ¯\_(ツ)_/¯ | 09:50 |
| sshnaidm | difficult to believe libvirt is done in the same company.. | 09:51 |
| sshnaidm | quiquell, {{ pub_key }} - maybe it has a new value? | 09:52 |
| quiquell | sshnaidm: where at libvirt review ? | 09:53 |
| quiquell | panda: as expected :-) | 09:54 |
| sshnaidm | quiquell, line 124 https://review.openstack.org/#/c/633444/6..8/roles/libvirt/setup/overcloud/tasks/libvirt_nodepool.yml | 09:54 |
| sshnaidm | quiquell, if you set it to something like "~/.. " it fails | 09:54 |
| quiquell | tasks/vars/libvirt_nodepool_vars.yml | 09:56 |
| quiquell | Is there | 09:56 |
| quiquell | we don't reset that | 09:56 |
| ykarel | quiquell, looks like changing command to shell is related no LIBGUESTFS thing | 09:56 |
| quiquell | arggg!!! | 09:57 |
| sshnaidm | quiquell, https://github.com/openstack/tripleo-quickstart/blob/master/roles/libvirt/setup/overcloud/tasks/vars/libvirt_nodepool_vars.yml#L12 | 09:57 |
| sshnaidm | quiquell, yeah, right | 09:57 |
| quiquell | sshnaidm: it's all good but we will have to override that with the other review "user_pub_key" | 09:58 |
| sshnaidm | ykarel, what do you mean? | 09:58 |
| *** derekh has joined #oooq | 09:58 | |
| ykarel | sshnaidm, i replied to <quiquell> sshnaidm: maybe the force thing | 09:59 |
| ykarel | in last PS command module is changed to shell module, and that can do something to ~ path | 09:59 |
| quiquell | ykarel: maybe we cannot upload stuff with force ? | 09:59 |
| quiquell | ykarel: but shell understand ~ isn't it ? | 10:00 |
| quiquell | ykarel: ahh wait | 10:00 |
| quiquell | ykarel: single quotation | 10:00 |
| quiquell | ykarel: maybe not expanding | 10:00 |
| ykarel | possibly u can do local test to confirm | 10:00 |
| quiquell | ls -la '~/.ssh' | 10:00 |
| quiquell | have to be that | 10:00 |
| quiquell | removed quotation let's check | 10:02 |
| quiquell | libguestfs: trace: get_backend_settings = ["force_tcg"] | 10:03 |
| quiquell | At least is working | 10:03 |
| ykarel | yup that what i mentioned, but LIBGUESTFS_BACKEND_SETTING is working now as epected | 10:04 |
| quiquell | getenforce not found is becasuse it's at /usr/sbin/ | 10:04 |
| ykarel | yup and it's not in PATH | 10:04 |
| quiquell | virt-resize has work like a charm :-) | 10:05 |
| quiquell | ykarel: the getenforce Is kind of warning I think | 10:05 |
| ykarel | yes | 10:05 |
| ykarel | u can ignore it | 10:05 |
| *** holser_ has joined #oooq | 10:09 | |
| quiquell | panda: I think f28 standalone job does not work at RDO | 10:11 |
| panda | quiquell: you said "the autohold feature is working as expected" how did you test it ? | 10:11 |
| quiquell | panda: have we run it ? | 10:11 |
| quiquell | panda: docker-compose exec scheduler zuul autohold ... | 10:12 |
| quiquell | panda: before job ends | 10:12 |
| panda | quiquell: f28 works at rdo, it's in the promotion pipeline | 10:12 |
| quiquell | panda: I think you can even set it before starts didn't test that | 10:12 |
| panda | quiquell: it has some random tempest failure, but generaly works | 10:13 |
| quiquell | panda: don't find the jobs at RDO zuul they are all skiped | 10:13 |
| quiquell | panda: https://softwarefactory-project.io/zuul/t/rdoproject.org/builds?job_name=periodic-tripleo-ci-fedora-28-standalone-master | 10:14 |
| quiquell | sshnaidm: looks like providers name is what is filter at tenants to do the cleanup | 10:15 |
| sshnaidm | quiquell, yeah, need to test | 10:15 |
| quiquell | ykarel: Inject ssh public key into the image working | 10:16 |
| quiquell | ykarel: was the quoting | 10:16 |
| ykarel | panda, rocky and master both promoting together? | 10:16 |
| ykarel | quiquell, ack | 10:16 |
| ykarel | panda, http://38.145.34.55/rocky.log and http://38.145.34.55/master.log | 10:16 |
| quiquell | force thing is going to save us | 10:16 |
| *** sshnaidm has quit IRC | 10:19 | |
| quiquell | marios: You know what was the issue ? the commit comment with {{ user_key }} on it after changing git commit zuul works :-/ | 10:21 |
| quiquell | Looks like somehow zuul try to expand the git commit | 10:22 |
| marios | quiquell: how do you mean? like it was failing lint or someting? | 10:22 |
| quiquell | Can it be possible ? | 10:22 |
| marios | wut | 10:22 |
| quiquell | going to test that | 10:22 |
| quiquell | marios: my last PS is just the commit message and now it works | 10:23 |
| quiquell | :-/ | 10:23 |
| quiquell | panda: tripleo-ci-fedora-28-standalone does not work out of the box at RDO http://logs.rdoproject.org/22/18622/2/check/tripleo-ci-fedora-28-standalone/0f39f69/ | 10:24 |
| quiquell | panda: do we have to set something in the job ? | 10:24 |
| panda | ykarel: I see only master running | 10:24 |
| *** dtantsur|afk is now known as dtantsur | 10:25 | |
| ykarel | panda, http://38.145.34.55/rocky.log is showing false message? | 10:25 |
| ykarel | promoter Promoting the container images for dlrn hash 2172030885b2f117cd96cee52b4a850d3c072a13 on rocky to current-tripleo | 10:25 |
| panda | quiquell: something is wrong with the image ... iptables is missing | 10:26 |
| *** sshnaidm has joined #oooq | 10:26 | |
| quiquell | panda: this will break f28 periodics ? | 10:27 |
| panda | ykarel: it probably timed out, and we don't log timeouts there, only in the journal. I'm looking | 10:27 |
| ykarel | ack | 10:28 |
| panda | quiquell: probably already has, I'll take a look | 10:28 |
| quiquell | marios, sshnaidm: Confirmed using "{{ user_key }}" at git commit is no good | 10:28 |
| quiquell | marios, sshnaidm: test review https://review.rdoproject.org/r/#/c/18623/ | 10:28 |
| panda | quiquell: periodic working fine | 10:29 |
| panda | quiquell: https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-fedora-28-centos-7-containers-standalone-master/906726f/job-output.txt.gz | 10:29 |
| quiquell | panda: they use different image or something ? | 10:29 |
| *** derekh has quit IRC | 10:29 | |
| panda | quiquell: different than what ? I don't know, there is an upstream-fedora28 node in the nodeset | 10:30 |
| quiquell | panda: found it, looks like tripleo-ci-fedora-28-standalone use different image at RDO https://softwarefactory-project.io/zuul/t/rdoproject.org/job/tripleo-ci-fedora-28-standalone | 10:31 |
| quiquell | rdo-fedora-28 | 10:31 |
| quiquell | ykarel, sshnaidm: force_fcg working fine and not much latency | 10:36 |
| quiquell | f28 failed for different reason | 10:36 |
| ykarel | ack | 10:37 |
| arxcruz|ruck | u guys talking about this? | 10:37 |
| arxcruz|ruck | 2019-01-30 06:40:53 | requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url: https://trunk.registry.rdoproject.org/v2/tripleomasterF28/centos-binary-cron/manifests/tripleo-ci-testing | 10:37 |
| panda | arxcruz|ruck: is that the pur fedora pipelinee ? | 10:40 |
| panda | purrr purrr purrr | 10:40 |
| panda | meow | 10:41 |
| arxcruz|ruck | panda: http://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-fedora-28-master-containers-build/55bf94b/logs/undercloud/home/zuul/undercloud_install.log.txt.gz | 10:41 |
| panda | arxcruz|ruck: yep, that's the master fedora28 containers build, which is in experimental state, and destined to fail until at least the end of next sprint | 10:41 |
| arxcruz|ruck | panda: ack, removing the critical from the bug | 10:42 |
| panda | arxcruz|ruck: cat you paste the bug ? | 10:42 |
| panda | arxcruz|ruck: I'll add a not to it | 10:43 |
| panda | comment #4: NOT | 10:43 |
| arxcruz|ruck | https://bugs.launchpad.net/tripleo/+bug/1813913 | 10:43 |
| openstack | Launchpad bug 1813913 in tripleo "periodic-tripleo-fedora-28-master-containers-build failing with unauthorized access to rdo registry" [Medium,Triaged] - Assigned to Gabriele Cerami (gcerami) | 10:43 |
| panda | I added a not | 10:43 |
| panda | arxcruz|ruck: thanks | 10:43 |
| arxcruz|ruck | jpena: is delorean api offline ? | 10:44 |
| jpena | arxcruz|ruck: https://trunk.rdoproject.org/api-centos-master-uc/api/civotes.html works for me | 10:45 |
| arxcruz|ruck | jpena: http://paste.openstack.org/show/744243/ | 10:45 |
| jpena | arxcruz|ruck: let me check the logs | 10:45 |
| quiquell | panda: Going to add some repro fixed at "issues" in the sprint is that ok or better to put at user stories ? | 10:46 |
| quiquell | fixes | 10:46 |
| panda | quiquell: is it out of band work ? | 10:46 |
| quiquell | panda: well is related about usability and libvirt | 10:47 |
| panda | quiquell: put it in us | 10:47 |
| quiquell | panda: https://review.rdoproject.org/r/#/c/18621/, https://review.rdoproject.org/r/#/c/18621/ and https://review.rdoproject.org/r/#/c/18558/ | 10:47 |
| quiquell | is usability and fixing CI | 10:47 |
| quiquell | panda: ack | 10:47 |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053, tripleo-ci-centos-7-containerized-undercloud-upgrades, tripleo-ci-centos-7-undercloud-containers, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ (3 more messages) | 10:48 |
| panda | ykarel: yep it timed out, we are not logging it even in journald, but times match, every run has two hours to complete, and rocky took from 7:21 to 9:15. I wonder what took so long if container push just started at 9:05. | 10:50 |
| jpena | arxcruz|ruck: oh, I know what happened | 10:50 |
| arxcruz|ruck | jpena: more important, do you know who to blame? | 10:51 |
| jpena | yep | 10:51 |
| arxcruz|ruck | even more important, the person to blame is me? | 10:51 |
| arxcruz|ruck | :P | 10:51 |
| panda | ykarel: ah eyah because it has been push all other phases too | 10:51 |
| *** derekh has joined #oooq | 10:51 | |
| jpena | so, there's a cron job that tries to renew the SSL certificate, and then reloads httpd | 10:51 |
| arxcruz|ruck | jpena: do you require me to open a bug/request to rdo team ? | 10:51 |
| panda | ykarel: each run tries to promote both current-tripleo and current-tripleo-rdo | 10:51 |
| jpena | the promotion call happened right during that reload, so it failed | 10:52 |
| jpena | I'll need to check that to reduce the likelyhood of it happening | 10:52 |
| arxcruz|ruck | jpena: (╯°□°）╯︵ ┻━┻ | 10:52 |
| ykarel | panda, ack | 10:52 |
| arxcruz|ruck | jpena: want me to open a bug, or not necessary? | 10:53 |
| panda | arxcruz|ruck: so when something goes wrong and you smile, it's because you found someone to blame :) | 10:53 |
| arxcruz|ruck | panda: ;) | 10:53 |
| jpena | arxcruz|ruck: not needed, I'll take care of it | 10:53 |
| arxcruz|ruck | panda: only when the person to blame is not me hehehe | 10:53 |
| ykarel | panda, master promoter failed too with: remote error: tls: handshake failure | 10:53 |
| ssbarnea|rover | sshnaidm: just remembered, "&|" is the newer construct that is not supported by many bash versions. | 10:55 |
| panda | ykarel: I think this is what arxcruz|ruck and jpena are talking about | 10:59 |
| ykarel | panda, possibly, jpena is that happen also on docker registry? | 11:00 |
| ssbarnea|rover | sshnaidm: and the requested logs: https://seashells.io/v/Fm6jeMHe (dry and xtrace) and https://seashells.io/v/xkkkvaXh (non dry) | 11:00 |
| jpena | ykarel: no, the registry SSL certificates are renewed differently | 11:02 |
| ykarel | jpena, ack so panda this is different, i think it happens from time to time | 11:02 |
| jpena | ykarel: did that tls handshake failure happen when connecting to a cloudflare server? | 11:02 |
| panda | ykarel: also, it's not on rdo registry | 11:02 |
| panda | ykarel: it's during dockerhub upload | 11:02 |
| ykarel | panda, yeah right, it's docker hub | 11:02 |
| jpena | ykarel: quiquell was looking into something similar a few days ago | 11:03 |
| quiquell | ssbarnea|rover: this was just an empty review https://review.rdoproject.org/r/#/c/18623/ | 11:05 |
| quiquell | ssbarnea|rover: just confirm that zuul has problems with jinja2 stuff at git commetns | 11:05 |
| quiquell | ssbarnea|rover: This is the real one https://review.rdoproject.org/r/#/c/18569/ | 11:05 |
| ssbarnea|rover | quiquell: can we talk about the `ssh-add -L` thing? | 11:06 |
| quiquell | ssbarnea|rover: Yep, I want better solution | 11:06 |
| quiquell | ssbarnea|rover: your blue ? | 11:06 |
| ssbarnea|rover | quiquell: https://bluejeans.com/2655417928 yep | 11:06 |
| *** holser_ is now known as holser|lunch | 11:23 | |
| *** udesale has quit IRC | 11:26 | |
| *** dsneddon has quit IRC | 11:27 | |
| chandankumar | quiquell: marios panda ssbarnea|rover https://review.openstack.org/#/c/631708/ is merged on os_tempest side | 11:31 |
| chandankumar | quiquell: marios panda sshnaidm https://review.openstack.org/#/c/627500/ is good to go to finish os_Tempest integration | 11:31 |
| chandankumar | and here is the latest result http://logs.openstack.org/00/627500/79/check/tripleo-ci-centos-7-standalone-os-tempest/dc59561/logs/stestr_results.html | 11:32 |
| ssbarnea|rover | chandankumar: is anything wrong with your rdo account? i am unable to add you as reviewer to https://review.rdoproject.org/r/#/c/18626/ - what username you have? maybe you full name is missing from user profile? | 11:34 |
| chandankumar | ssbarnea|rover: chkumar246 | 11:34 |
| ssbarnea|rover | chandankumar:again the buggy user lookup on rdo, found you (failed to find when typinh only ch, but later it started). If you could put full name in profile it would make it easier to do lookups. | 11:35 |
| marios | chandankumar: ack looking (so stays no vote for now) | 11:40 |
| quiquell | marios, panda: To close old bug https://review.openstack.org/#/c/602025/ | 11:41 |
| ssbarnea|rover | marios: chandankumar quiquell : please merge https://review.rdoproject.org/r/#/c/18626/ to fix the currently broken pep8 job on ci-config | 11:41 |
| quiquell | ssbarnea|rover: done | 11:41 |
| ssbarnea|rover | tx! | 11:41 |
| quiquell | chandankumar: I see a success here http://logs.openstack.org/00/627500/79/check/tripleo-ci-centos-7-standalone-os-tempest/dc59561/ | 11:43 |
| quiquell | chandankumar: Can we workflow ? | 11:43 |
| chandankumar | marios: yes, as per discussion with team, we have to keep the job to nv till we integrate existing skip list from validate tempest | 11:43 |
| chandankumar | quiquell: yes sir | 11:43 |
| quiquell | chandankumar: where are the tempest output ? | 11:43 |
| marios | ssbarnea|rover: quiquell beat me to it | 11:44 |
| quiquell | I don't find the html and the like | 11:44 |
| marios | ssbarnea|rover: plus i am not core there anyway :D | 11:44 |
| quiquell | marios: Do the honors with chandankumar's one | 11:44 |
| chandankumar | quiquell: under logs folder stestr_html | 11:44 |
| sshnaidm | chandankumar, +2 | 11:44 |
| quiquell | wait | 11:44 |
| sshnaidm | chandankumar++ | 11:44 |
| hubbot1 | sshnaidm: chandankumar's karma is now 7 | 11:44 |
| quiquell | chandankumar: why do we have .html and .html.gz ? | 11:44 |
| chandankumar | quiquell: http://logs.openstack.org/00/627500/79/check/tripleo-ci-centos-7-standalone-os-tempest/dc59561/logs/stestr_results.html | 11:45 |
| sshnaidm | quiquell, I think infra scripts are gzipping everything | 11:45 |
| sshnaidm | not sure why html was skipped | 11:46 |
| quiquell | sshnaidm: but there is also non gziped version of the html | 11:46 |
| quiquell | sshnaidm, chandankumar: Let's just merge is like minor minor minor issue | 11:46 |
| quiquell | marios: You merge ? | 11:46 |
| sshnaidm | let's merge together | 11:46 |
| chandankumar | quiquell: might be this https://github.com/openstack/tripleo-quickstart-extras/commit/df2401d7668a97ae41d565b6d61284c38bb2c731#diff-6d36e7f77753585e9b236544cf148893 | 11:46 |
| marios | quiquell: waiting for zuul to report? | 11:47 |
| marios | quiquell: ah v79 was just commit message k | 11:47 |
| quiquell | marios: ack | 11:47 |
| marios | quiquell: ssbarnea|rover chandankumar is there problem with rdo jobs? | 11:48 |
| quiquell | what's the issue | 11:48 |
| chandankumar | marios: yes I think I need to fix the rdo jobs | 11:48 |
| marios | quiquell: https://review.openstack.org/#/c/627500/79 Patch Set 79: Verified-1 | 11:48 |
| marios | chandankumar: Unknown project git.openstack.org/openstack/openstack-ansible-os_tempest | 11:48 |
| quiquell | Then we cannot merge this | 11:49 |
| chandankumar | quiquell: I know the fix | 11:49 |
| quiquell | Or one will not work without the other ? | 11:49 |
| quiquell | chandankumar: zuul config is missing this project like in reproducer | 11:50 |
| quiquell | chandankumar: software factory zuul I suppose | 11:50 |
| chandankumar | quiquell: you mean software factory config? | 11:51 |
| chandankumar | quiquell: repo link? | 11:51 |
| sshnaidm | arxcruz|ruck, sova will be down for a while, migrating | 11:52 |
| chandankumar | quiquell: where is name: tripleo-ci-base is defined? | 11:52 |
| arxcruz|ruck | sshnaidm: ack | 11:53 |
| quiquell | chandankumar: It's more complicated, let me find it | 11:53 |
| sshnaidm | chandankumar, sorry, need to add something to SF for that to work | 11:57 |
| * sshnaidm will prepare | 11:57 | |
| chandankumar | sshnaidm: ack, thanks :-) | 11:57 |
| *** dsneddon has joined #oooq | 12:00 | |
| sshnaidm | chandankumar, https://softwarefactory-project.io/r/#/c/14909/ | 12:02 |
| chandankumar | sshnaidm: thanks! | 12:03 |
| *** jpena is now known as jpena|lunch | 12:04 | |
| *** dsneddon has quit IRC | 12:07 | |
| *** gkadam has joined #oooq | 12:08 | |
| quiquell | sshnaidm: foce_tcg is working | 12:15 |
| *** gkadam is now known as gkadam-bmgr | 12:19 | |
| sshnaidm | quiquell, cool | 12:22 |
| sshnaidm | quiquell, so we're good with libvirt jobs? | 12:23 |
| sshnaidm | chandankumar, did you think to run our jobs on these 3 roles? | 12:24 |
| quiquell | sshnaidm: https://tree.taiga.io/project/tripleo-ci-board/task/675?kanban-status=1447276 | 12:24 |
| sshnaidm | chandankumar, do we have job on os_tempest? | 12:25 |
| quiquell | sshnaidm: we have to merge changes at libvirt roles :-/ | 12:25 |
| sshnaidm | quiquell, if we use tcg by default we will slow down all reproducers for all people | 12:26 |
| sshnaidm | quiquell, nested has problems on CI mostly | 12:26 |
| sshnaidm | quiquell, can we make it an option and then force it in CI? | 12:26 |
| quiquell | sshnaidm: we have to do the opposite that is done now | 12:27 |
| quiquell | sshnaidm: by default not doing it and doing it if env var is present | 12:27 |
| quiquell | sshnaidm: but that will be next sprint I think | 12:28 |
| quiquell | What we really have to merge is the RDO mirror for rdo-cloud-tripleo | 12:30 |
| quiquell | It's like impossible to merge with the libvirt issue | 12:31 |
| chandankumar | sshnaidm: nope | 12:31 |
| chandankumar | sshnaidm: once it is available I will add it there | 12:31 |
| sshnaidm | quiquell, seems like it's impossible to "omit" env var in ansible | 12:34 |
| quiquell | sshnaidm: yep, maybe a flag to the role | 12:34 |
| sshnaidm | quiquell, we can duplicate tasks.. | 12:34 |
| quiquell | sshnaidm: nooo | 12:34 |
| quiquell | sshnaidm: we can put the settings to empty by default | 12:34 |
| quiquell | and put the env in the CI | 12:35 |
| quiquell | going to test that | 12:35 |
| sshnaidm | quiquell, maybe to extract to separate file.. but not sure environment will work so | 12:35 |
| sshnaidm | quiquell, empty is not equal to not defined | 12:35 |
| chandankumar | sshnaidm: https://review.openstack.org/633931 added here for os_tempest gating to tripleo jonb | 12:35 |
| chandankumar | sshnaidm: yes I am thinking of running it on all three projects | 12:36 |
| sshnaidm | quiquell, I'm afraid it's like with this issues of bogdando patch | 12:36 |
| sshnaidm | chandankumar, cool! | 12:36 |
| bogdando | which patch? | 12:36 |
| marios | quiquell: what do you think about https://review.rdoproject.org/r/#/c/18539/ we need something more there? do you want to merge it (with the hard-coded mirror?) or leave as WIP | 12:36 |
| marios | quiquell: wondering if i should tidy up remove WIP or what (for sprint close) | 12:37 |
| marios | quiquell: i always thought this was just a test reveiw but you gave me the impression you want to merge something here hence the question | 12:38 |
| *** dsneddon has joined #oooq | 12:39 | |
| quiquell | marios: After the RDO docker mirror we define mirror_fqdn at job definition zuul_site_mirror_fqdn | 12:40 |
| chandankumar | no way timedout at http://logs.openstack.org/00/627500/79/check/tripleo-ci-centos-7-undercloud-containers/1b0acb6/ | 12:40 |
| chandankumar | sshnaidm: quiquell I am rebasing the whole stuff | 12:41 |
| sshnaidm | bogdando, https://github.com/openstack/tripleo-quickstart/commit/b469bc0923824303cf234a2f62e39abcbffdf3aa | 12:42 |
| quiquell | marios: we ara doing this inse the zuul playbook now https://github.com/rdo-infra/ansible-role-tripleo-ci-reproducer/blob/master/playbooks/tripleo-ci-reproducer/run.yaml#L17 | 12:42 |
| quiquell | marios: maybe we can do a set_fact and use that value for the inner job | 12:42 |
| sshnaidm | bogdando, it breaks virt-resize sometimes, because if it's not set - it's set to empty | 12:42 |
| marios | quiquell: ok so i don't need to pass it in the job you mean | 12:42 |
| quiquell | marios: what do you think ? | 12:42 |
| quiquell | marios: Also the mirror you set only works if job is runing at RDO cloud | 12:43 |
| sshnaidm | bogdando, and empty var breaks things.. but I don't find a good ansible way to omit env vars.. | 12:43 |
| marios | quiquell: sure (but same i mean you have when rdo-cloud in https://github.com/rdo-infra/ansible-role-tripleo-ci-reproducer/blob/49414460b1c40794a869828d51c4e44db0741534/playbooks/tripleo-ci-reproducer/run.yaml#L17 ) | 12:43 |
| bogdando | sigh | 12:43 |
| quiquell | marios: this var is local to the task | 12:44 |
| marios | quiquell: i see | 12:44 |
| quiquell | marios: maybe you can convert it into a fact using "set_fact" | 12:44 |
| quiquell | marios: so you can use it | 12:44 |
| quiquell | in case is not set you set somwhere upstream not RDO | 12:44 |
| marios | quiquell: ok but still, we want to merge some dryrun job here or just for testing https://review.rdoproject.org/r/#/c/18539/5 | 12:44 |
| quiquell | marios: just for testing at the moment | 12:44 |
| quiquell | marios: also dryrun at host is not working a f28 | 12:44 |
| marios | quiquell: right this is my understanding but previously i though you wanted to merge something | 12:45 |
| quiquell | marios: just run dryrun for -libvirt and hello-reproducer for -host | 12:45 |
| quiquell | marios: I though that we will be able, but it's not the case, sorry for confusion | 12:45 |
| quiquell | Going to lunch something | 12:45 |
| *** quiquell is now known as quiquell|lunch | 12:45 | |
| marios | quiquell: np man was just not sure | 12:45 |
| *** dsneddon has quit IRC | 12:46 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053, tripleo-ci-centos-7-containerized-undercloud-upgrades, tripleo-ci-centos-7-undercloud-containers, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ (3 more messages) | 12:48 |
| *** trown|outtypewww is now known as trown | 12:48 | |
| *** dsneddon has joined #oooq | 12:48 | |
| *** rlandy has joined #oooq | 12:54 | |
| *** holser|lunch is now known as holser_ | 12:54 | |
| *** dsneddon has quit IRC | 12:55 | |
| quiquell|lunch | rlandy: o/ | 12:56 |
| quiquell|lunch | rlandy: commented https://review.openstack.org/#/c/631067/ | 12:56 |
| *** quiquell|lunch is now known as quiquell | 12:57 | |
| rlandy | thanks- fixing | 12:57 |
| marios | wdyt folks moving ci into new section https://review.openstack.org/#/c/633771 http://logs.openstack.org/71/633771/1/check/openstack-tox-docs/73b789b/html/ | 12:57 |
| quiquell | rlandy: I have some conecerns with newgrp | 12:57 |
| quiquell | rlandy: can break people's system | 12:57 |
| rlandy | hmmm ... yeah - it's what we've always told people to do :) | 12:58 |
| panda | break their systems | 13:00 |
| rfolco | ping call marios, quiquell, sshnaidm, weshay, panda, rlandy, arxcruz, mwhahaha, rfolco, chkumar, ssbarnea, kopecmartin | 13:00 |
| weshay | arxcruz|ruck, so.. can you add the lp #'s for the issues you are referring to | 13:01 |
| arxcruz|ruck | weshay: sure | 13:01 |
| weshay | arxcruz|ruck, k.. cool.. as soon as you give status, drop | 13:01 |
| weshay | and join our retro :) | 13:01 |
| weshay | k? | 13:01 |
| *** jpena|lunch is now known as jpena | 13:02 | |
| weshay | ssbarnea|rover, can you join the retro | 13:03 |
| *** dsneddon has joined #oooq | 13:06 | |
| marios | https://trello.com/b/0VFswmht/rdo-infra-retrospective?menu=filter&filter=label:UniSprint4 | 13:07 |
| marios | quiquell: ^ | 13:07 |
| marios | kopecmartin: created on for you https://trello.com/c/EFW8B1uq/785-sucks-being-forced-to-fill-in-cards | 13:12 |
| kopecmartin | marios, :D :D amazing, I remember I've created a very similar card few sprints back | 13:12 |
| *** dsneddon has quit IRC | 13:14 | |
| arxcruz|ruck | weshay: ack, sorry, was editing the status :) | 13:16 |
| *** ykarel is now known as ykarel|away | 13:17 | |
| marios | quiquell: nobel peace prize | 13:19 |
| quiquell | hehe | 13:20 |
| apetrich | hey arxcruz|ruck this was merged https://review.openstack.org/#/c/633214/1/roles/nodepool-setup/defaults/main.yml but I'm having issues with rdo-cloud and reproducer for me it fails and says the image md5 is d56d54f110654dfd29b0e8ed56e6cda8 | 13:26 |
| apetrich | arxcruz|ruck, is that something you are seeing or is just something wrong on my setup | 13:26 |
| arxcruz|ruck | apetrich: check if what you're using already have cirros downloaded | 13:26 |
| arxcruz|ruck | or maybe the md5 is wrong? | 13:26 |
| arxcruz|ruck | apetrich: not seeing that on my side | 13:27 |
| apetrich | arxcruz|ruck, I just cleared my deployments on rdo-cloud and called a gate-checker reproducer. it is using the 3.6 image | 13:28 |
| apetrich | arxcruz|ruck, "The checksum for /opt/cache/files/cirros-0.3.6-x86_64-disk.img did not match a30f156a83471adf89ebb70dc96d0c82; it was d56d54f110654dfd29b0e8ed56e6cda8." | 13:28 |
| apetrich | arxcruz|ruck, if I change by hand the to the d5... the deployment works | 13:29 |
| ska | Can someone show me what the KVM/virt networks look like in a deployed OOOq environment? | 13:29 |
| arxcruz|ruck | apetrich: let me check | 13:29 |
| apetrich | arxcruz|ruck, cheers | 13:29 |
| arxcruz|ruck | apetrich: yup, the md5 is wrong | 13:30 |
| arxcruz|ruck | the md5 in the patch is for the source a30f156a83471adf89ebb70dc96d0c82 cirros-0.3.6-source.tar.gz | 13:30 |
| chandankumar | arxcruz|ruck: I have taken the md5 from cirros download website | 13:30 |
| arxcruz|ruck | chandankumar: you did, but it was the wrong one | 13:31 |
| arxcruz|ruck | chandankumar: https://download.cirros-cloud.net/0.3.6/MD5SUMS | 13:31 |
| apetrich | arxcruz|ruck, cool. I will submit a patch | 13:31 |
| arxcruz|ruck | apetrich: ok :) | 13:31 |
| apetrich | arxcruz|ruck, https://review.openstack.org/633941 cheers, mate | 13:35 |
| *** dsneddon has joined #oooq | 13:41 | |
| ska | I'm installing TripleO manually on KVM in order to learn the internal mechanics of it, and I'm wondering if I can make my Nics on the Non-Provisioning network as kvm-bridged via br0. | 13:48 |
| *** dsneddon has quit IRC | 13:50 | |
| *** dsneddon has joined #oooq | 13:51 | |
| *** ykarel|away has quit IRC | 13:56 | |
| *** dsneddon has quit IRC | 13:57 | |
| *** dsneddon has joined #oooq | 14:26 | |
| *** ykarel|away has joined #oooq | 14:29 | |
| arxcruz|ruck | apetrich: danke schön :D | 14:30 |
| *** dsneddon has quit IRC | 14:33 | |
| *** ykarel|away is now known as ykarel | 14:36 | |
| *** skramaja has quit IRC | 14:42 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053, tripleo-ci-centos-7-containerized-undercloud-upgrades, tripleo-ci-centos-7-undercloud-containers, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ (3 more messages) | 14:48 |
| *** quiquell is now known as quiquell|off | 14:48 | |
| chandankumar | sshnaidm: weshay need to leave see ya tomorrow! | 14:49 |
| rlandy | ssbarnea|rover: if you have time, I'd like your opinion/help on how we can test the user-based reproducer work | 14:57 |
| rlandy | bash linting? | 14:57 |
| ssbarnea|rover | rlandy: sure, it will be my pleasure to give it a try | 15:04 |
| *** dsneddon has joined #oooq | 15:05 | |
| rlandy | ssbarnea|rover: let's add a task to next sprint | 15:06 |
| weshay | sshnaidm, rlandy https://review.openstack.org/#/c/631067/39/roles/create-zuul-based-reproducer/templates/launcher-playbook.yaml.j2 | 15:07 |
| weshay | line 65 | 15:07 |
| *** udesale has joined #oooq | 15:08 | |
| *** dsneddon has quit IRC | 15:11 | |
| *** quiquell|off has quit IRC | 15:19 | |
| *** quiquell has joined #oooq | 15:33 | |
| *** quiquell is now known as quiquell|off | 15:33 | |
| sshnaidm | ssbarnea|rover, can you mute please? | 15:34 |
| ssbarnea|rover | sshnaidm: done | 15:35 |
| weshay | arxcruz|ruck, what's the issue w/ periodic-tripleo-ci-fedora-28-ovb-1ctlr_1comp-featureset002-master-upload | 15:38 |
| weshay | arxcruz|ruck, fyi https://trello.com/c/vbRp5cl9/857-cixlp1810706tripleociproapromotion-jobmasterfs030ovn-tempest-test-failing-with-no-ipv4-addresses-found-in-port-dict | 15:39 |
| weshay | is marked done, but the bug is still open | 15:39 |
| arxcruz|ruck | weshay: checking | 15:39 |
| weshay | https://bugs.launchpad.net/tripleo/+bug/1810706 | 15:39 |
| openstack | Launchpad bug 1810706 in tripleo "[promotion job][master][FS030][OVN] Tempest test failing with No IPv4 addresses found in: <port dict> " [Critical,Triaged] | 15:39 |
| arxcruz|ruck | weshay: i'll close the bug | 15:39 |
| weshay | chandankumar, arxcruz|ruck also https://trello.com/c/OpWmOkAZ/742-cixlp1792862tripleociproa-master-telemetry-tempest-integration-tests-failed-giving-unable-to-complete-operation-on-subnet-b13c76 | 15:40 |
| weshay | is marked done.. | 15:40 |
| weshay | arxcruz|ruck, chandankumar /me compares bug #'s from http://dashboard-ci.tripleo.org/d/cEEjGFFmz/cockpit?orgId=1&panelId=155&fullscreen | 15:40 |
| weshay | w/ the cix board | 15:40 |
| weshay | arxcruz|ruck, so afaict.. we have one promotion blocker atm | 15:41 |
| *** dsneddon has joined #oooq | 15:41 | |
| arxcruz|ruck | weshay: so, there weren't promotion, the job fails because delorean ssl issue | 15:41 |
| arxcruz|ruck | weshay: the telemetry one i wasn't aware | 15:41 |
| arxcruz|ruck | the ipv4 is passing now | 15:41 |
| arxcruz|ruck | marked as fixed release | 15:42 |
| *** dsneddon has quit IRC | 15:47 | |
| weshay | arxcruz|ruck, thanks! | 15:47 |
| arxcruz|ruck | weshay: i did nothing :( | 15:47 |
| marios | wdyt folks moving ci into new section https://review.openstack.org/#/c/633771 http://logs.openstack.org/71/633771/2/check/openstack-tox-docs/266ee8b/html/ | 15:48 |
| rlandy | panda: do you want to bj to go through the reproducer user stuff? | 15:56 |
| rlandy | sshnaidm: have you tried a multinode job with reproducer? have an issue with bridge setup? | 15:56 |
| panda | rlandy: ok in ~1h ? | 15:57 |
| *** gkadam-bmgr has quit IRC | 15:57 | |
| sshnaidm | rlandy, tried. but not recently | 15:57 |
| sshnaidm | rlandy, on rdo cloud | 15:58 |
| sshnaidm | rlandy, I didn't have issues, but it was a while ago | 15:58 |
| arxcruz|ruck | after the whole evening on meetings, time for german classes :) | 15:58 |
| *** arxcruz|ruck is now known as arxcruz | 15:58 | |
| arxcruz | chandankumar: quiquell|off the ruck and rover are all yours | 15:59 |
| rlandy | sshnaidm: k - think there is an issue with libvirt set up then - will fix | 15:59 |
| arxcruz | ssbarnea|rover: change your nick :P | 15:59 |
| rlandy | panda: sure - ping me when you are ready to chat | 15:59 |
| *** saneax has quit IRC | 16:01 | |
| *** udesale has quit IRC | 16:02 | |
| *** agopi is now known as agopi|food | 16:11 | |
| *** dsneddon has joined #oooq | 16:14 | |
| *** ccamacho has quit IRC | 16:16 | |
| *** kopecmartin is now known as kopecmartin|off | 16:20 | |
| *** ssbarnea|rover has quit IRC | 16:20 | |
| *** ssbarnea has joined #oooq | 16:21 | |
| *** dtantsur is now known as dtantsur|afk | 16:32 | |
| ssbarnea | sshnaidm: https://review.rdoproject.org/r/#/c/18517/ -- ovb-cleanup ok now? | 16:38 |
| sshnaidm | ssbarnea, commented | 16:42 |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053, tripleo-ci-centos-7-containerized-undercloud-upgrades, tripleo-ci-centos-7-undercloud-containers, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ (3 more messages) | 16:48 |
| *** jfrancoa has quit IRC | 16:54 | |
| *** ykarel is now known as ykarel|away | 17:03 | |
| rlandy | weshay: not sure why the bridge creation errors - works when I execute it manually | 17:11 |
| rlandy | output is far from verbose | 17:12 |
| rlandy | install requirements are all there | 17:12 |
| weshay | k.. need a few | 17:13 |
| rlandy | no need to look - just fyi - it wasn't what I thought yesterday | 17:14 |
| rlandy | will look more | 17:14 |
| *** agopi|food is now known as agopi | 17:18 | |
| *** derekh has quit IRC | 17:33 | |
| weshay | panda, http://logs.openstack.org/67/631067/39/check/tripleo-ci-centos-7-standalone/fd7d6ca/logs/ | 17:35 |
| weshay | panda, http://logs.openstack.org/67/631067/39/check/tripleo-ci-centos-7-standalone/fd7d6ca/logs/reproducer-zuul-based-quickstart.tar | 17:37 |
| weshay | panda, https://tree.taiga.io/project/tripleo-ci-board/epic/5 | 17:40 |
| weshay | rlandy, python-libselinux and a few other reqs | 17:44 |
| weshay | the quickstart.sh --install-deps | 17:44 |
| weshay | what happened? | 17:44 |
| weshay | missing | 17:44 |
| rlandy | weshay: what happened? | 17:45 |
| weshay | those things are missing | 17:46 |
| rlandy | https://review.openstack.org/#/c/631067/39/roles/create-zuul-based-reproducer/templates/launcher-env-setup-playbook.yaml.j2 | 17:47 |
| rlandy | see line 42 | 17:47 |
| weshay | ah.. maybe I got an old version, sorry if that's the case | 17:47 |
| weshay | ah ya.. dang | 17:47 |
| weshay | 39 | 17:47 |
| weshay | rlandy, the ci is far behind your changes | 17:48 |
| *** bogdando has quit IRC | 17:49 | |
| ssbarnea | can someone point me to a role that installs docker *service* on centos? the upstream install-docker role does not work with centos and I want to fix this. | 17:50 |
| weshay | https://review.openstack.org/#/c/631067/39/roles/create-zuul-based-reproducer/templates/reproducer-zuul-based-quickstart.sh.j2 | 17:50 |
| weshay | line 42 rlandy ? | 17:50 |
| rlandy | launcher-env-setup-playbook.yaml.j2 | 17:51 |
| weshay | lolz | 17:51 |
| weshay | sorry | 17:51 |
| rlandy | np | 17:51 |
| weshay | rlandy, we still need libselinux in the script | 17:52 |
| rlandy | weshay: also quiquell|off is opposed to using newgrp | 17:52 |
| weshay | rlandy, oh ya I know | 17:52 |
| rlandy | weshay: quickstart.sh --install-deps should install libselinux | 17:52 |
| weshay | I may have to outrank him on that :) | 17:52 |
| weshay | rlandy, but it's from ansible | 17:52 |
| rlandy | at least it tried to when I tried it | 17:52 |
| weshay | it wont work | 17:53 |
| weshay | just failed | 17:53 |
| weshay | ASK [Copy roles and libraries for tripleo-quickstart] ************************************************************************************************************************************************************** | 17:53 |
| weshay | failed: [localhost] (item=roles) => {"changed": false, "checksum": "e4c9cbb66dcf4e06b3d73d359aa7726d0cfe16f7", "item": "roles", "msg": "Aborting, target uses selinux but python bindings (libselinux-python) aren't installed!"} | 17:53 |
| weshay | failed: [localhost] (item=library) => {"changed": false, "checksum": "dd26984f7b07572630362c3a8c9d877bad031aca", "item": "library", "msg": "Aborting, target uses selinux but python bindings (libselinux-python) aren't installed!"} | 17:53 |
| rlandy | weshay: so all of quickstart.sh --install-deps should move to the .sh or just libselinux-python install? | 17:54 |
| weshay | rlandy, hrm | 17:54 |
| weshay | chicken and egg | 17:55 |
| weshay | can't clone w/ ansible w/o ansible and python-selinux | 17:55 |
| weshay | so.. the script needs to install ansible and python-selinux | 17:55 |
| rlandy | fine | 17:55 |
| rlandy | will do that | 17:55 |
| rlandy | and leave quickstart.sh --install-deps in env setup playbook for the rest | 17:55 |
| * rlandy updates | 17:56 | |
| rlandy | hmmm both python3-libselinux and libselinux-python? | 17:58 |
| rlandy | weshay: ^^ | 17:58 |
| weshay | let's say python2 only atm | 17:58 |
| weshay | for mvb | 17:58 |
| weshay | mvp | 17:58 |
| weshay | mvp | 17:58 |
| rlandy | most valuable player | 17:58 |
| rlandy | weshay: is there a good check for libselinux-python install or we just install | 18:02 |
| rlandy | works fine | 18:02 |
| weshay | rlandy, I don't think this worked TASK [Install remaining dependencies] ******************************************************************************************************************************************************************************* | 18:04 |
| weshay | changed: [localhost] | 18:04 |
| * rlandy checks | 18:04 | |
| weshay | rlandy, atm I have sudo $(command -v dnf || command -v yum) install -y ansible python2-libselinux | 18:04 |
| weshay | but it is best to have two lines | 18:04 |
| weshay | two checks | 18:04 |
| rlandy | ha ... let's be clear here ... | 18:05 |
| rlandy | libselinux-python OR python2-libselinux | 18:05 |
| rlandy | No match for argument: libselinux2-python | 18:06 |
| rlandy | Package python2-libselinux-2.8-1.fc28.x86_64 is already installed, skipping. | 18:06 |
| rlandy | sudo $(command -v dnf || command -v yum) install -y libselinux-python | 18:07 |
| rlandy | ^^ that is the command, no? | 18:07 |
| *** dsneddon has quit IRC | 18:08 | |
| rlandy | I am checking output | 18:08 |
| rlandy | TASK [Install remaining dependencies] ************************************************************************************************************************************************************************* | 18:09 |
| rlandy | task path: /tmp/reproduce-tmp.fIbFO/launcher-env-setup-playbook.yaml:36 | 18:09 |
| rlandy | changed: [localhost] => {"changed": true, "cmd": "bash git/tripleo-quickstart/quickstart.sh --install-deps", "delta": "0:00:02.689630", "end": "2019-01-30 13:09:24.085483", "rc": 0, "start": "2019-01-30 13:09:21.395853", "stderr": "", "stderr_lines": [], "stdout": "NOTICE: installing dependencies\nsudo-1.8.23-1.fc28.x86_64\nLast metadata expiration check: 0:52:37 ago on Wed Jan 30 12:16:44 2019.\nPackage git-core-2.17.2-2.fc28.x8 | 18:09 |
| rlandy | 6_64 is already installed, skipping.\nPackage gcc-8.2.1-6.fc28.x86_64 is already installed, skipping.\nPackage iproute-4.18.0-1.fc28.x86_64 is already installed, skipping.\nPackage libyaml-0.1.7-5.fc28.x86_64 is already installed, skipping.\nPackage libffi-devel-3.1-16.fc28.x86_64 is already installed, skipping.\nPackage openssl-devel-1:1.1.0i-1.fc28.x86_64 is already installed, skipping.\nPackage redhat-rpm-config-110-1.fc28.noa | 18:09 |
| rlandy | rch is already installed, skipping.\nPackage python3-libselinux-2.8-1.fc28.x86_64 is already installed, skipping.\nPackage python2-libselinux-2.8-1.fc28.x86_64 is already installed, skipping.\nPackage python3-PyYAML-3.12-10.fc28.x86_64 is already installed, skipping.\nDependencies resolved.\nNothing to do.\nComplete!\npip:9.0.3 virtualenv:16.0.0 setuptools:39.2.0 install-deps succeeded.", "stdout_lines": ["NOTICE: installing | 18:09 |
| rlandy | dependencies", "sudo-1.8.23-1.fc28.x86_64", "Last metadata expiration check: 0:52:37 ago on Wed Jan 30 12:16:44 2019.", "Package git-core-2.17.2-2.fc28.x86_64 is already installed, skipping.", "Package gcc-8.2.1-6.fc28.x86_64 is already installed, skipping.", "Package iproute-4.18.0-1.fc28.x86_64 is already installed, skipping.", "Package libyaml-0.1.7-5.fc28.x86_64 is already installed, skipping.", "Package libffi-devel-3.1-16.f | 18:09 |
| rlandy | c28.x86_64 is already installed, skipping.", "Package openssl-devel-1:1.1.0i-1.fc28.x86_64 is already installed, skipping.", "Package redhat-rpm-config-110-1.fc28.noarch is already installed, skipping.", "Package python3-libselinux-2.8-1.fc28.x86_64 is already installed, skipping.", "Package python2-libselinux-2.8-1.fc28.x86_64 is already installed, skipping.", "Package python3-PyYAML-3.12-10.fc28.x86_64 is already installed, | 18:09 |
| rlandy | skipping.", "Dependencies resolved.", "Nothing to do.", "Complete!", "pip:9.0.3 virtualenv:16.0.0 setuptools:39.2.0 install-deps succeeded."]} | 18:10 |
| rlandy | oops | 18:10 |
| rlandy | sorry | 18:10 |
| *** saneax has joined #oooq | 18:10 | |
| *** trown is now known as trown|lunch | 18:10 | |
| weshay | k k | 18:12 |
| weshay | I hit TASK [libvirt/setup/overcloud : get a list of MACs to use] ********************************************************************************************************************************************************** | 18:12 |
| weshay | task path: /var/tmp/RECREATE/roles/libvirt/setup/overcloud/tasks/main.yml:22 | 18:12 |
| weshay | fatal: [localhost]: FAILED! => {"msg": "An unhandled exception occurred while templating '[{u'bridge': u'brovc', u'name': u'overcloud'}, {u'bridge': u'brext', u'forward_mode': u'nat', u'name': u'external', u'netmask': u\"{{ external_network_cidr|ipaddr('netmask') }}\", u'nat_port_range': [1024, 65535], u'address': u'{{ external_network_cidr|nthhost(1) }}', u'dhcp_range': [u'{{ external_network_cidr|nthhost(10) }}', u'{{ external_network_cidr|nt | 18:12 |
| weshay | hhost(50) }}']}]'. Error was a <class 'ansible.errors.AnsibleFilterError'>, original message: The ipaddr filter requires python-netaddr be installed on the ansible controller"} | 18:12 |
| weshay | to retry, use: --limit @/var/tmp/RECREATE/launcher-playbook.retry | 18:12 |
| * weshay runs w/ -vv | 18:12 | |
| weshay | -vvvv | 18:12 |
| weshay | again | 18:12 |
| *** jpena is now known as jpena|off | 18:13 | |
| weshay | rlandy, ur right.. it's running | 18:15 |
| rlandy | weshay: https://review.openstack.org/#/c/631067/40/roles/create-zuul-based-reproducer/templates/reproducer-zuul-based-quickstart.sh.j2 - added line 110 - let me know if that does not help | 18:15 |
| rlandy | ok | 18:23 |
| rlandy | still hitting python-netaddr error? | 18:23 |
| rlandy | will add to install | 18:23 |
| weshay | ya.. maybe I need to look at quickstart.sh | 18:23 |
| *** saneax has quit IRC | 18:23 | |
| weshay | atm I'm running w/ sudo $(command -v dnf || command -v yum) install -y ansible python2-libselinux python2-netaddr | 18:23 |
| weshay | talking to gabriele too atm.. so I'll check in a minute | 18:24 |
| *** sshnaidm is now known as sshnaidm|afk | 18:25 | |
| rlandy | https://github.com/openstack/tripleo-quickstart/blob/master/roles/libvirt/setup/overcloud/tasks/main.yml#L12 | 18:26 |
| rlandy | weshay: ^^ should install there | 18:26 |
| weshay | ya.. that one is weird | 18:26 |
| rlandy | can you see the output of the install task? | 18:27 |
| *** dsneddon has joined #oooq | 18:34 | |
| weshay | TASK [libvirt/setup/overcloud : ensure python-netaddr] ************************************************************************************************************************************************************** | 18:34 |
| weshay | task path: /var/tmp/RECREATE/roles/libvirt/setup/overcloud/tasks/main.yml:12 | 18:34 |
| weshay | ok: [localhost] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: python-netaddr"]} | 18:34 |
| weshay | I think it failed earlier before that | 18:34 |
| weshay | seems like a quickstart.sh bug | 18:35 |
| weshay | like we could add it here https://github.com/openstack/tripleo-quickstart/blob/master/quickstart.sh#L123 | 18:38 |
| *** dsneddon has quit IRC | 18:42 | |
| weshay | rlandy, yum is working :) | 18:46 |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053, tripleo-ci-centos-7-containerized-undercloud-upgrades, tripleo-ci-centos-7-undercloud-containers, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ (3 more messages) | 18:48 |
| rlandy | weshay: cool | 18:48 |
| rlandy | happy to hear something works | 18:48 |
| rlandy | panda: are you all set or still want to meet? | 18:53 |
| *** dsneddon has joined #oooq | 19:03 | |
| *** ykarel|away has quit IRC | 19:05 | |
| panda | rlandy: still in my 5th hour of meeting, I'll ping you when I finish | 19:05 |
| rlandy | panda: sure - no rush | 19:07 |
| *** trown|lunch is now known as trown | 19:23 | |
| panda | weshay: https://review.openstack.org/509728 | 19:25 |
| *** panda is now known as panda|off | 19:59 | |
| weshay | I'm officially in love | 20:05 |
| weshay | rlandy, this is a little odd.. my node lost it's name resolution | 20:09 |
| weshay | and failed in midway through the standalone deployment | 20:09 |
| weshay | rlandy, I wonder if we should supplement the dns server config | 20:10 |
| weshay | w/ something other than localhost | 20:10 |
| rlandy | to help your lost node? | 20:12 |
| *** rfolco has quit IRC | 20:13 | |
| weshay | hrm.. it's not a dns issues.. interesting failed ... http://git.openstack.org/cgit/openstack/tripleo-heat-templates/tree/deployment/timesync/chrony-baremetal-ansible.yaml#n120 | 20:14 |
| weshay | the awesome thing is I'm 100% reproducing what I see in the job log :))) | 20:14 |
| weshay | soooo happy | 20:14 |
| rlandy | great | 20:16 |
| rlandy | I've seen lots of chatter about chrony failures | 20:16 |
| rlandy | weshay: any objection to my merging this ... https://review.rdoproject.org/r/#/c/18613/? | 20:18 |
| rlandy | it could break the world | 20:18 |
| rlandy | but launch doens;t run in ci | 20:18 |
| weshay | rlandy, go 4 it | 20:18 |
| rlandy | let's live dangerously | 20:18 |
| *** rfolco has joined #oooq | 20:24 | |
| rlandy | ha - think it may be ssh between the nodes | 20:29 |
| *** vinaykns has joined #oooq | 20:31 | |
| vinaykns | hello folks...I trying to install oooq...but it isn't getting successful. | 20:37 |
| vinaykns | http://pastebin.test.redhat.com/704788 | 20:37 |
| weshay | vinaykns, I need to see which tq call it's failing on | 20:37 |
| vinaykns | weshay: http://pastebin.test.redhat.com/704790 | 20:40 |
| *** rfolco has quit IRC | 20:40 | |
| weshay | vinaykns, oh.. you are just trying to manually ssh in? | 20:43 |
| vinaykns | weshay: nope..I'm not doing any as such | 20:43 |
| weshay | ok.. still not enough context | 20:43 |
| weshay | I'm not in your head yet | 20:44 |
| vinaykns | weshay: hmm..do you wanna tmate in..? | 20:44 |
| weshay | sure | 20:44 |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053, tripleo-ci-centos-7-containerized-undercloud-upgrades, tripleo-ci-centos-7-undercloud-containers, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ (3 more messages) | 20:48 |
| weshay | failing on TASK [environment/setup : Create libvirt networks] ****************************************************************************************************************************************************************** | 20:49 |
| weshay | vinaykns, ok w/ --teardown all? | 20:52 |
| weshay | will remove all vm's, networks etc | 20:52 |
| vinaykns | yeah sure go ahead.! | 20:52 |
| weshay | hrm.. libvirt looks bad | 20:57 |
| weshay | Jan 30 15:31:06 sandbox.usersys.redhat.com libvirtd[1161060]: 2019-01-30 20:31:06.673+0000: 1161060: error : virNetSocketReadWire:1806 : End of file while reading data: Input/output error | 20:57 |
| weshay | rlandy, comments posted to https://review.openstack.org/#/c/631067/40 | 21:07 |
| rlandy | weshay: thanks ... think I found the libvirt problem | 21:07 |
| rlandy | the keys | 21:07 |
| rlandy | can't ssh from one node to the other | 21:07 |
| rlandy | host-key: "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAINDHUUjxEhVn02IQ74PzBo5ClPzgvmLHvq8BmU1eSMvQ" | 21:08 |
| weshay | oh.. for the multinode | 21:08 |
| weshay | nice | 21:08 |
| rlandy | ^^ can from ssh key scan | 21:08 |
| rlandy | and it's added as a known_host | 21:08 |
| weshay | oh REALLLY | 21:08 |
| weshay | that's odd | 21:08 |
| weshay | tmate? | 21:08 |
| weshay | rlandy, can you ping? | 21:09 |
| rlandy | let me check | 21:09 |
| weshay | rlandy, zuul@ip? | 21:09 |
| rlandy | [zuul@subnode-1 .ssh]$ ping 192.168.122.95 | 21:09 |
| rlandy | PING 192.168.122.95 (192.168.122.95) 56(84) bytes of data. | 21:09 |
| rlandy | 64 bytes from 192.168.122.95: icmp_seq=1 ttl=64 time=0.265 ms | 21:09 |
| rlandy | yes ,,, | 21:09 |
| weshay | ssh -vvv | 21:10 |
| rlandy | [zuul@subnode-1 .ssh]$ ssh zuul@192.168.122.95 | 21:10 |
| rlandy | zuul@192.168.122.95's password: | 21:10 |
| rlandy | ^^ should not ask for password | 21:10 |
| weshay | rlandy, can we blue? | 21:15 |
| rlandy | https://github.com/openstack/tripleo-quickstart/blob/364c4ecce906f34fcc3eed7e16d2b79b25497509/roles/tripleo-inventory/tasks/inventory.yml#L24 | 21:16 |
| rlandy | weshay: ack - joining your blue | 21:16 |
| vinaykns | thx weshay.! I guess it would work now.! | 21:32 |
| *** holser_ has quit IRC | 21:36 | |
| jrist | boop https://review.openstack.org/#/c/632019/1 | 21:56 |
| *** trown is now known as trown|outtypewww | 22:07 | |
| *** dtrainor has quit IRC | 22:26 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053, tripleo-ci-centos-7-containerized-undercloud-upgrades, tripleo-ci-centos-7-undercloud-containers, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ (3 more messages) | 22:48 |
| *** vinaykns has quit IRC | 23:05 | |
| *** dtrainor has joined #oooq | 23:06 | |
| *** rascasoft has quit IRC | 23:07 | |
| *** dtrainor has quit IRC | 23:09 | |
| *** dtrainor has joined #oooq | 23:12 | |
| *** dtrainor has quit IRC | 23:20 | |
| *** rlandy is now known as rlandy|bbl | 23:24 | |
| *** dtrainor has joined #oooq | 23:25 | |
| *** rascasoft has joined #oooq | 23:55 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!