| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario007-multinode-oooq-container, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053 @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario007-multinode- (2 more messages) | 00:42 |
|---|---|---|
| *** rlandy has quit IRC | 00:48 | |
| *** agopi has joined #oooq | 01:51 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053 @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-rocky-branch @ https://review.openstack.org/560445, (1 more message) | 02:42 |
| *** chandankumar has quit IRC | 02:57 | |
| *** chandankumar has joined #oooq | 02:59 | |
| *** agopi has quit IRC | 03:01 | |
| *** udesale has joined #oooq | 03:06 | |
| *** apetrich has quit IRC | 03:15 | |
| *** sanjayu_ has joined #oooq | 03:35 | |
| *** agopi has joined #oooq | 03:36 | |
| *** sanjayu_ has quit IRC | 03:39 | |
| *** agopi has quit IRC | 03:43 | |
| *** skramaja has joined #oooq | 03:44 | |
| *** agopi has joined #oooq | 03:46 | |
| *** ykarel|away has joined #oooq | 03:52 | |
| *** ykarel|away is now known as ykarel | 04:25 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-scenario001-multinode-oooq-container @ https://review.openstack.org/602248, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/560445, stable/queens: tripleo-ci-centos-7-scenario000-multinode- (1 more message) | 04:42 |
| *** udesale has quit IRC | 05:01 | |
| *** udesale has joined #oooq | 05:04 | |
| *** udesale has quit IRC | 05:45 | |
| *** udesale has joined #oooq | 05:46 | |
| *** ykarel has quit IRC | 06:02 | |
| *** ykarel has joined #oooq | 06:26 | |
| *** kopecmartin|off is now known as kopecmartin|devc | 06:27 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-scenario001-multinode-oooq-container @ https://review.openstack.org/602248, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/560445, stable/queens: tripleo-ci-centos-7-scenario000-multinode- (1 more message) | 06:42 |
| *** quiquell|off is now known as quiquell | 06:44 | |
| quiquell | https://review.openstack.org/#/c/632695 | 06:47 |
| quiquell | marios: can workflow this? | 06:47 |
| quiquell | I mean if I can | 06:48 |
| marios | quiquell: o/ looking | 06:52 |
| marios | quiquell: well | 06:52 |
| marios | quiquell: i'd rather we discuss/set it up first (see my comment) ssbarnea|rover ? | 06:52 |
| quiquell | https://review.openstack.org/#/c/631840 <- marios sshnaidm|off they are changing the container images | 06:53 |
| marios | quiquell: cos we have to add that to our irc clients | 06:53 |
| quiquell | marios: ack, that's what I thought | 06:53 |
| quiquell | marios: do you have reviews to merge? | 06:54 |
| sshnaidm|off | quiquell|off, with that patch https://review.rdoproject.org/r/#/c/18515/1 zuul UI doesn't work, so better to leave it "stable" and update only nodepool images, it's running now ontests | 06:55 |
| sshnaidm|off | quiquell, we check in CI zuul API, but not UI | 06:55 |
| quiquell | sshnaidm|off: ack, have you see the review a our dockerfile instead of pbrx? | 06:57 |
| quiquell | sshnaidm|off: maybe we can leave online zuul-web with stable and update the others at other review | 06:57 |
| marios | quiquell: no just merged on myself https://review.openstack.org/632972 thanks | 06:57 |
| quiquell | And see what happens | 06:57 |
| quiquell | marios: ack | 06:58 |
| quiquell | sshnaidm|off: I have to check maybe they are tagging them correctly | 07:00 |
| quiquell | Then we dont need rdoci dockerhub | 07:01 |
| sshnaidm|off | quiquell, I don't think it's good idea, zuul code better to be same in all containers | 07:01 |
| sshnaidm|off | quiquell, well, we'll still need to test them and "promote" - for that rdoci is needed | 07:02 |
| quiquell | Not sure | 07:04 |
| quiquell | Well let's talk next week | 07:04 |
| *** jfrancoa has joined #oooq | 07:32 | |
| *** weshay has quit IRC | 07:47 | |
| quiquell | marios: Have you check this ? https://tree.taiga.io/project/morucci-software-factory/us/2245 | 07:56 |
| quiquell | marios: they are doing a ansible playbook version of reproducer no need for whole zuul enchilada | 07:56 |
| marios | quiquell: but how? i mean no zuul? | 07:57 |
| marios | looking | 07:57 |
| quiquell | marios: you just run a zuul command over a server with especial rest API | 07:58 |
| quiquell | marios: I think | 07:58 |
| quiquell | marios: and it runs the job locally | 07:58 |
| marios | quiquell: so it would use upstream zuul | 07:58 |
| marios | cool | 07:58 |
| quiquell | marios: it's WIP now but is like the way to go | 07:58 |
| *** quiquell is now known as quiquell|brb | 08:01 | |
| *** panda|off is now known as panda | 08:04 | |
| *** ccamacho has joined #oooq | 08:13 | |
| *** ykarel is now known as ykarel|lunch | 08:14 | |
| *** quiquell|brb is now known as quiquell | 08:19 | |
| marios | biab | 08:28 |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-scenario001-multinode-oooq-container @ https://review.openstack.org/602248, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/560445, stable/queens: tripleo-ci-centos-7-scenario000-multinode- (1 more message) | 08:43 |
| *** tosky has joined #oooq | 08:49 | |
| *** jpena|off is now known as jpena | 08:51 | |
| *** apetrich has joined #oooq | 09:01 | |
| chandankumar | marios: I have got an idea to unblock os_Tempest tripleoci patch | 09:09 |
| chandankumar | to avoid dependency on os_tempest current review | 09:09 |
| chandankumar | let me send a patch | 09:09 |
| *** udesale has quit IRC | 09:29 | |
| *** udesale has joined #oooq | 09:30 | |
| *** derekh has joined #oooq | 09:31 | |
| marios | chandankumar:ok | 09:32 |
| *** ssbarnea|bkp2 has joined #oooq | 09:38 | |
| *** ssbarnea|rover has quit IRC | 09:39 | |
| *** bogdando has joined #oooq | 09:40 | |
| *** ykarel|lunch is now known as ykarel | 09:40 | |
| arxcruz|ruck | ssbarnea|bkp2: i'm back | 09:46 |
| ssbarnea|bkp2 | arxcruz|ruck: ok in general? | 09:57 |
| arxcruz|ruck | ssbarnea|bkp2: i'm gonna die | 09:57 |
| arxcruz|ruck | eventually | 09:57 |
| arxcruz|ruck | some point in the future | 09:58 |
| ssbarnea|bkp2 | arxcruz|ruck: one of few things that are guaranteed. btw, in few hours i will be out too, till Tuesday. I think i should remember to remove rover suffix, to avoid confusions. | 09:59 |
| *** ssbarnea|bkp2 has quit IRC | 09:59 | |
| *** ssbarnea|rover has joined #oooq | 10:00 | |
| *** ssbarnea|rover has quit IRC | 10:01 | |
| *** ssbarnea has joined #oooq | 10:01 | |
| marios | panda: don't forget the -m PEM -t rsa on the keys you create! (see wednesday note) | 10:02 |
| *** ssbarnea|bkp2 has joined #oooq | 10:04 | |
| sshnaidm|off | quiquell, needed for ovb reproducing: https://review.rdoproject.org/r/#/c/18522/1 | 10:07 |
| *** ssbarnea has quit IRC | 10:07 | |
| quiquell | sshnaidm|off: does not have config merging powers | 10:11 |
| quiquell | sshnaidm|off: going to add some small comments though | 10:11 |
| quiquell | sshnaidm|off: Let's merge this to fix CI(CI) https://review.rdoproject.org/r/#/c/18535 | 10:24 |
| quiquell | sshnaidm|off: have tested and it's working | 10:24 |
| panda | marios what version of openstack client have you used to accept the new images ? | 10:25 |
| panda | marios: I'm trying delorean master on fedora28 but it's giving me a error | 10:26 |
| marios | https://paste.fedoraproject.org/paste/dpz~9Jy6vLJvMr4Ij8W2Bw panda | 10:27 |
| marios | panda: this is latest f29 | 10:27 |
| *** dsneddon has quit IRC | 10:29 | |
| panda | marios: stil not working for me 3.14 btu thanks. | 10:29 |
| panda | getting 'module' object has no attribute 'OpenStackConfigException' | 10:29 |
| marios | check pvt | 10:29 |
| marios | not sure i hit that one though :/ | 10:30 |
| marios | i just confirmed it | 10:32 |
| marios | panda: its the os-cloud thing | 10:32 |
| marios | see pvt | 10:32 |
| *** matbu has quit IRC | 10:32 | |
| *** matbu has joined #oooq | 10:33 | |
| quiquell | panda: did you do --os-cloud rdo-cloud ? | 10:33 |
| quiquell | panda: looks like yoru cloud.yaml is wrong | 10:33 |
| quiquell | panda: btw, we need this to fix CI(CI) we broke it https://review.rdoproject.org/r/#/c/18535/ | 10:34 |
| marios | panda: did it work? | 10:34 |
| marios | quiquell: yeah its that | 10:34 |
| marios | the --os-cloud | 10:34 |
| marios | we should update the readme early next week once this settles | 10:36 |
| marios | especially things like -m-m PEM -t rsa | 10:36 |
| marios | etc | 10:36 |
| marios | quiquell: ^ | 10:36 |
| quiquell | marios: Yep all the pre-requisites have to be there | 10:37 |
| quiquell | marios: suppose is part of the doc taiga task | 10:37 |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-scenario001-multinode-oooq-container @ https://review.openstack.org/602248, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/560445, stable/queens: tripleo-ci-centos-7-scenario000-multinode- (1 more message) | 10:43 |
| panda | sshnaidm|off: quiquell offer expired again | 10:46 |
| panda | sshnaidm|off: quiquell can you share the image again ? | 10:46 |
| quiquell | panda: If you have the openstack-nodepool tenant credentials you can share it to your tenant | 10:46 |
| quiquell | panda: This is what I did | 10:46 |
| marios | quiquell: dind't sshnaidm|off say they were public now? | 10:47 |
| quiquell | panda: openstack --os-cloud openstack image add project <image_id> <tenat_id> | 10:47 |
| marios | quiquell: so do we still ned to import them? | 10:47 |
| quiquell | marios: we have two actual mechanism about images | 10:47 |
| quiquell | marios: image with team pub keys in them <- this is the ones we are sharing | 10:48 |
| quiquell | marios: images that can be injected with pub key from nodepool <- this are the -cloudinit- images and those are publics | 10:48 |
| quiquell | marios: to make cloudinit images working we need this wip from sshnaidm|off https://review.rdoproject.org/r/#/c/18515/ | 10:49 |
| quiquell | marios: ^ this solution scales since we don't need to inject pub keys in the images | 10:49 |
| quiquell | marios: but we have to merge the review | 10:49 |
| marios | quiquell: ack thanks | 10:51 |
| marios | quiquell: so for the pub keys image we have problem since | 10:51 |
| marios | we are all generating new keys now to get it to work | 10:51 |
| quiquell | panda: Do you want me to share or are you doing it yourself ? | 10:51 |
| marios | well i did | 10:51 |
| marios | maybe panda too | 10:51 |
| marios | quiquell: so i won't be able to launch anything nayway | 10:52 |
| quiquell | marios: yep that's why the cloudinit solution is much needed | 10:52 |
| marios | quiquell: ack | 10:52 |
| quiquell | marios: you can do one thing | 10:52 |
| quiquell | marios: put the review in place with the dry_run and see how it goes | 10:52 |
| quiquell | marios: ince launc_job review CI is already launching stuff | 10:52 |
| quiquell | marios: can help you with that if you want | 10:53 |
| marios | panda: still worth getting set up though i mean you can just reimport any image if needed | 10:53 |
| marios | quiquell: sure will try to post later (thats the task for libvirt you mean ) | 10:54 |
| quiquell | marios: yep | 10:54 |
| panda | openstack --os-cloud rdo-cloud image add project 6a6d23d7-65e7-43ea-9307-71b2e17d2ead f6d8d86e1e254c86ba0809af666a4c41 | 11:00 |
| panda | Could not find resource 6a6d23d7-65e7-43ea-9307-71b2e17d2ead | 11:01 |
| panda | mmhh | 11:01 |
| *** dsneddon has joined #oooq | 11:01 | |
| *** jaosorior has quit IRC | 11:07 | |
| *** dsneddon has quit IRC | 11:08 | |
| quiquell | panda: nodepool credentials you need for add project | 11:12 |
| panda | quiquell: nodepool credentials ? | 11:13 |
| quiquell | panda: RDO "openstakc-nodepool" tenant credentials | 11:13 |
| quiquell | panda: this team have them | 11:13 |
| panda | oh | 11:14 |
| quiquell | s/kc/ck/ | 11:14 |
| panda | so this command is run on the source tenant | 11:14 |
| quiquell | panda: yep | 11:14 |
| quiquell | panda: images are there | 11:14 |
| quiquell | panda: and you share it with whatever tenant id you want | 11:14 |
| quiquell | panda: in this case to yours | 11:14 |
| quiquell | panda: then you accept with your tenant credentials | 11:15 |
| ssbarnea|bkp2 | arxcruz|ruck: quiquell panda : can you please help with https://review.rdoproject.org/r/#/c/18517/ ? i want to get the fix deployed before leaks strike again | 11:19 |
| panda | I think I finally have the images \o/ | 11:21 |
| *** dsneddon has joined #oooq | 11:22 | |
| panda | quiquell: ansible-role-tripleo-ci-reproducer : Check image are uploaded fails but images are there, how does the reproducer understand which tenant to use ? | 11:25 |
| marios | panda: quiquell please vote do we know who can merge this? https://review.rdoproject.org/r/18537 | 11:26 |
| *** dsneddon has quit IRC | 11:27 | |
| quiquell | marios: sshnaidm|off can but also people at #rdo | 11:27 |
| quiquell | let's ask | 11:27 |
| marios | ssbarnea|bkp2: arxcruz|ruck panda added you https://review.rdoproject.org/r/#/c/18537/1/resources/gerrit-stream.yaml | 11:27 |
| marios | chandankumar: whats your rdo id ^^ | 11:27 |
| panda | gmail ? | 11:27 |
| marios | chandankumar: add a comment i'll update | 11:27 |
| marios | panda: i just used what came up in rdo gerrit for autocomplete | 11:27 |
| marios | panda: comment if wrong i will update | 11:28 |
| arxcruz|ruck | mine is arxcruz at beloved company dot com | 11:28 |
| arxcruz|ruck | our acruz | 11:28 |
| arxcruz|ruck | or* | 11:28 |
| marios | arxcruz|ruck: add comment | 11:28 |
| marios | please | 11:28 |
| ssbarnea|bkp2 | marios: i hope this is needed only temporary, right:D | 11:28 |
| panda | arxcruz|ruck: the one you use to connect to softwarefactory ? | 11:28 |
| quiquell | panda: you have the log of the error ? | 11:28 |
| marios | arxcruz|ruck: 13:27 < marios> panda: i just used what came up in rdo gerrit for autocomplete | 11:28 |
| panda | arxcruz|ruck: because taht's connected to the github account | 11:28 |
| marios | quiquell: right ssbarnea|bkp2 makes a good point | 11:28 |
| ssbarnea|bkp2 | marios: or our users will not be able to use reproducer. | 11:29 |
| marios | do we need THE WORLD THERE | 11:29 |
| quiquell | panda: you have to be sure that your tenant credentials are at "rdo-cloud" | 11:29 |
| marios | ssbarnea|bkp2: yes i was thinking the same | 11:29 |
| panda | marios: yes, add *@redhat.com | 11:29 |
| marios | done | 11:29 |
| marios | ** | 11:29 |
| marios | *.* | 11:29 |
| arxcruz|ruck | panda: my github account also has the redhat.com | 11:29 |
| panda | marios: connect it to kerberos | 11:29 |
| quiquell | panda: images are by defaultos_centos7_image: openstack-infra-centos-7 | 11:30 |
| quiquell | os_fedora28_image: openstack-infra-fedora-28 | 11:30 |
| panda | quiquell: they are, my clouds.yaml has only that one | 11:30 |
| ssbarnea|bkp2 | this could prove problematic, not sure if the event stream have sensitive info there. also 2nd issue is: what if someone uses his non redhat account upstream. does this mean that only RH staff will be able to run reproducer? | 11:30 |
| marios | arxcruz|ruck: done | 11:30 |
| ssbarnea|bkp2 | ... i know, too many questions | 11:30 |
| quiquell | panda: if you are using upstream-fedora28 new ones you will have to pass the correct ones in the variables | 11:30 |
| panda | quiquell: I have both | 11:31 |
| quiquell | panda: something like this https://paste.fedoraproject.org/paste/MRwdJKHEwYRcvXALhMM-FA | 11:32 |
| quiquell | ssbarnea|bkp2: we have to fix the stream stuff it's not needed at review.openstack.org | 11:33 |
| quiquell | ssbarnea|bkp2: maybe is possible to bypass it with some option at software factory | 11:33 |
| quiquell | panda: Have a question regarding RDO jobs | 11:33 |
| quiquell | panda: looks like RDO base job is not calling "configure-unbound" role so maybe we don't need it at all | 11:34 |
| ssbarnea|bkp2 | quiquell: make a note and continue. we will see how we can address the streams later. | 11:34 |
| quiquell | ssbarnea|bkp2: Yep, let's unblock the images thing first is the main stopper, and the key thing is going to be painful | 11:35 |
| ssbarnea|bkp2 | worst case would be that only RH staff will be able to reproduce jobs run on softwarefactory, but probably we will not endup with that. | 11:35 |
| panda | ssbarnea|bkp2: why does everyone want to always deal with a log file on its own ? | 11:36 |
| quiquell | ssbarnea|bkp2: cleaner review is hard | 11:37 |
| quiquell | ssbarnea|bkp2: it's really tested ? | 11:38 |
| quiquell | marios: have ping people at #rdo let's see what they say | 11:39 |
| quiquell | marios: they will merge it | 11:39 |
| marios | quiquell: thanks but what about it do we need to add *.* there like its a requirement? what do we have to do to fix that | 11:40 |
| marios | quiquell: ssbarnea|bkp2 ^ | 11:40 |
| quiquell | marios: What we really want is that deactivated | 11:40 |
| marios | quiquell: at least lets add a task about it for sprint.next | 11:40 |
| marios | quiquell: well ok | 11:40 |
| marios | so lets start a campaign | 11:41 |
| marios | ;D | 11:41 |
| quiquell | marios: since it's not needed at review.openstack.org don't know why do we need this at RDO | 11:41 |
| marios | harrass.begin() | 11:41 |
| marios | activate | 11:41 |
| quiquell | marios: you are the harrasser | 11:41 |
| quiquell | marios: we are the muscle | 11:41 |
| marios | quiquell: seriously lets add a task/track it talk about it later/monday etc | 11:41 |
| marios | quiquell: not sure how i feel about that :0 | 11:41 |
| * marios recoffeee | 11:41 | |
| panda | quiquell: bua, bua! | 11:42 |
| quiquell | marios: Let's put a review with the dry_run to see how it goes I have found issues with running our base job at RDO images not upstream images | 11:42 |
| quiquell | panda: reproduced ? | 11:42 |
| marios | quiquell: can we please have a 10 minutes call .. in like a little while? or later if you are busy | 11:42 |
| quiquell | marios: yeah totally | 11:42 |
| marios | quiquell: ok. top of this hour works for you in 20 mins? | 11:43 |
| marios | panda: welcome/anyone else. i want to talk about the dry run libvirt task for new zuul repro | 11:43 |
| quiquell | marios: yep | 11:43 |
| marios | quiquell: thank you | 11:43 |
| marios | https://redhat.bluejeans.com/7661925373/ | 11:43 |
| marios | in 17 mins | 11:43 |
| quiquell | 16,5 | 11:44 |
| quiquell | 16 | 11:44 |
| panda | here goes my lunch | 11:44 |
| panda | quiquell: use this time to help me | 11:44 |
| quiquell | panda: sure blue ? | 11:44 |
| quiquell | friday hackaton | 11:45 |
| panda | freakaton | 11:46 |
| panda | freakoutaton | 11:46 |
| panda | quiquell: in the previous command "Get installed images name" I can see the images, so it's the check that is not working | 11:47 |
| quiquell | panda: can you paste the error ? | 11:48 |
| quiquell | marios, ssbarnea|bkp2, panda: stream stuff merged https://review.rdoproject.org/r/#/c/18537/ | 11:48 |
| panda | quiquell: http://paste.openstack.org/show/743455/ | 11:49 |
| marios | quiquell: thanks saw and commented | 11:49 |
| marios | i accidentally informed them of our campaign oops | 11:49 |
| panda | quiquell: found the problem | 11:49 |
| panda | quiquell: it's not openstack-infra, it's upstream-infra now | 11:50 |
| quiquell | panda: that's what I mean before | 11:53 |
| quiquell | panda: the image names are not the default ones in the role | 11:53 |
| panda | quiquell: the tasks taht are dealing with deocker should have "become: true" | 11:55 |
| panda | or you're all running from root ? | 11:56 |
| *** dsneddon has joined #oooq | 11:59 | |
| quiquell | panda: your mean the start of docker service ? | 11:59 |
| quiquell | panda: I don't use root I have wheel permissions at my user | 11:59 |
| quiquell | panda: I think sshnaidm|off was agains using become for that | 11:59 |
| panda | quiquell: meeting | 12:01 |
| panda | quiquell: so it's a requirement | 12:01 |
| *** dsneddon has quit IRC | 12:03 | |
| *** jfrancoa has quit IRC | 12:18 | |
| ssbarnea|bkp2 | bye! see you on Tuesday. | 12:26 |
| panda | wat ? | 12:26 |
| panda | ssbarnea|bkp2: you can't leave | 12:26 |
| ssbarnea|bkp2 | /away PTO | 12:26 |
| ykarel | panda, seems some issue with promoter | 12:27 |
| ykarel | failed: [localhost] (item=neutron-server-ovn) => {"changed": false, "item": "neutron-server-ovn", "msg": "Error pulling trunk.registry.rdoproject.org/tripleorocky/centos-binary-neutron-server-ovn - code: None message: Get https://trunk.registry.rdoproject.org/v2/tripleorocky/centos-binary-neutron-server-ovn/manifests/68348ba00d4f041b4a149b2741eee63987ac4a2d_bf7a0dd6: Get https://trunk.registry.rdoproject.org/openshift/token?account=unused&scope=repository%3A | 12:27 |
| ykarel | tripleorocky%2Fcentos-binary-neutron-server-ovn%3Apull: net/http: request canceled (Client.Timeout exceeded while awaiting headers)"} | 12:27 |
| ykarel | master one is running for long:- 2019-01-25 08:46:04,803 2226 INFO promoter Promoting the container images for dlrn hash f4ae18ea6c700e6220132ba0f04ec9dcb3f28052 on master to current-tripleo | 12:27 |
| ykarel | around 4 hours | 12:28 |
| panda | ykarel: mmhh, sometimes the general lock hangs | 12:28 |
| panda | ykarel: I'll have to kill it | 12:28 |
| ykarel | panda, at a time only one promoter runs, right? | 12:29 |
| panda | ykarel: yes, and creates a lock | 12:29 |
| ykarel | ack, then there is issue because i seen both rocky and master were running together | 12:30 |
| ykarel | i think pike was also ruuning | 12:30 |
| panda | mmhhh, that should not happen | 12:30 |
| panda | the server doesn't have the capacity to run two promotions. Dealing with images from both kills the file system | 12:31 |
| ykarel | okk please clear the issue | 12:32 |
| quiquell | ssbarnea|bkp2: enjoy | 12:32 |
| quiquell | panda, marios: We need to review/merge https://review.rdoproject.org/r/#/c/18535 and https://review.rdoproject.org/r/#/c/18279 | 12:34 |
| marios | quiquell: ack man at this point lets just merge stuff (I mean you and sagi are core there to start with?) | 12:35 |
| marios | quiquell: but sure | 12:35 |
| quiquell | Let me add you as core too | 12:35 |
| marios | i will have a better look but not right now if you are in a rush | 12:35 |
| marios | quiquell: +232, -43 | 12:36 |
| marios | quiquell: noted for next reviews run | 12:36 |
| quiquell | marios RDO email ? | 12:36 |
| marios | maybe later today or worst monday | 12:36 |
| marios | quiquell: qe/ | 12:36 |
| marios | 14:36 < quiquell> marios RDO email ? | 12:36 |
| quiquell | qe ? | 12:37 |
| marios | qe | 12:37 |
| quiquell | Give me your rdo gerrit email so I can add you as core | 12:37 |
| quiquell | Well we will need to merge a config review but they will do it | 12:37 |
| marios | marios AT R H C O M | 12:37 |
| quiquell | marios: ack | 12:37 |
| quiquell | damn that's a cool one | 12:37 |
| marios | quiquell: i don't know if there are any others (marius in upgrades :) ) but when i joined we were like 3k people was easier to ask for stuff like that | 12:38 |
| quiquell | marios: For how long have you being at redhat ? | 12:38 |
| marios | quiquell: ~9yrs | 12:39 |
| marios | man am old | 12:39 |
| quiquell | marios: holy sh... | 12:39 |
| quiquell | marios: didn't know | 12:39 |
| *** dsneddon has joined #oooq | 12:39 | |
| marios | quiquell: as i said already to you today, not sure how to feel about that reaction :) | 12:39 |
| marios | :D | 12:40 |
| quiquell | marios: I feel respect :-) | 12:40 |
| *** skramaja has quit IRC | 12:40 | |
| quiquell | Ok, lunch time or launch time or whatever | 12:41 |
| *** quiquell is now known as quiquell|food | 12:41 | |
| quiquell|food | marios: btw, I think we have mirror issues at our jobs you will find them at dry_run | 12:41 |
| quiquell|food | marios: http://logs.rdoproject.org/79/18279/49/check/tripleo-ci-reproducer-fedora-28-host/ca821c0/etc/pip.conf | 12:42 |
| quiquell|food | marios: no good | 12:42 |
| *** jfrancoa has joined #oooq | 12:42 | |
| marios | quiquell|food: ack will check thx | 12:42 |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-scenario001-multinode-oooq-container @ https://review.openstack.org/602248, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/560445, (1 more message) | 12:43 |
| *** sanjayu_ has joined #oooq | 12:43 | |
| *** dsneddon has quit IRC | 12:44 | |
| *** jpena is now known as jpena|lunch | 12:44 | |
| *** sanjayu_ has quit IRC | 12:53 | |
| *** dsneddon has joined #oooq | 13:12 | |
| *** ccamacho has quit IRC | 13:14 | |
| *** dsneddon has quit IRC | 13:17 | |
| *** quiquell|food is now known as quiquell | 13:27 | |
| *** ccamacho has joined #oooq | 13:29 | |
| *** rlandy has joined #oooq | 13:31 | |
| panda | ykarel: is it unblocked ? | 13:35 |
| ykarel | panda, still same stage | 13:37 |
| ykarel | 2019-01-25 08:46:04,803 2226 INFO promoter Promoting the container images for dlrn hash | 13:37 |
| ykarel | http://38.145.34.55/master.log | 13:37 |
| ykarel | rocky promoted some time back | 13:37 |
| panda | I don't see anythin runnig right now | 13:38 |
| panda | so it's not a problem with the lock | 13:39 |
| quiquell | panda: o/ | 13:41 |
| quiquell | panda: can you +2 the main missing repro reviews ? | 13:41 |
| quiquell | panda: this https://review.rdoproject.org/r/#/c/18535/ | 13:42 |
| quiquell | panda: and this https://review.rdoproject.org/r/#/c/18279/ | 13:42 |
| *** dsneddon has joined #oooq | 13:45 | |
| ykarel | panda, master promoter also not running? | 13:47 |
| *** jpena|lunch is now known as jpena | 13:49 | |
| *** dsneddon has quit IRC | 13:50 | |
| quiquell | ykarel: promotes does not look very healty in the cockpit | 13:50 |
| quiquell | ykarel: humm no they are good | 13:51 |
| panda | ykarel: quiquell cleaning up old docker images in the meantime | 13:51 |
| ykarel | panda, ack | 13:51 |
| panda | definitely something is wrong | 13:55 |
| panda | the dlrn-promoter.sh is not doing anythin | 13:56 |
| quiquell | panda: but I see promotions | 13:56 |
| marios | quiquell: liek dis? https://review.rdoproject.org/r/#/c/18539/1/playbooks/tripleo-ci-reproducer/templates/run.yaml.j2 | 13:56 |
| ykarel | quiquell, master promotions? | 13:56 |
| marios | not sure about the dryrun yet | 13:56 |
| marios | but i don't see us defining or using it anywhere outside of the jinja check if it is defined or not | 13:57 |
| quiquell | ykarel: from 24 | 13:57 |
| ykarel | today is 25 :) | 13:57 |
| panda | let me check journald | 13:57 |
| quiquell | marios: kind of dry_run has to ve a variable at job level | 13:57 |
| *** trown|outtypewww is now known as trown | 13:57 | |
| quiquell | marios: you have to define a new job like tripleo-ci-centos-7-standalone-dryrun | 13:57 |
| quiquell | marios: that has as parent tripleo-ci-centos-7-standalone | 13:58 |
| *** jfrancoa has quit IRC | 13:58 | |
| quiquell | marios: And in this job put dry_run: true | 13:58 |
| quiquell | marios: I think you cannot pass variables at zuul project level | 13:58 |
| quiquell | marios: all have to be inside zuul_yaml variable | 13:59 |
| quiquell | ykarel: ups ok | 14:01 |
| marios | quiquell: oh i see. i thought the job defined there was just for the noop and for this we just define the upstream one | 14:01 |
| marios | quiquell: thanks updating | 14:02 |
| quiquell | marios: It's going to be interesting how this goes | 14:02 |
| marios | quiquell: ah like a var to the inner job i guess | 14:02 |
| quiquell | marios: exactly it, btw you are core now | 14:03 |
| marios | quiquell: i commented there not sure about how that var would be consumed where i currently have it/maybe it would but i don't know | 14:03 |
| marios | k | 14:03 |
| marios | thanks | 14:03 |
| panda | quiquell: how did you setup docker to be run as root:wheel ? | 14:03 |
| rlandy | panda: re your comments https://review.openstack.org/#/c/631067/21/roles/create-zuul-based-reproducer/templates/launcher-playbook.yaml.j2 - did you link the right review? | 14:03 |
| rlandy | neutron spec | 14:03 |
| rlandy | the var to force failures | 14:04 |
| quiquell | panda: One thing is to run docker with normal user (pre.yaml add your user to docker group so you are all good) | 14:04 |
| panda | rlandy: yes, you have to read it and implementing like it's written there | 14:04 |
| panda | rlandy: and don't forger the routers | 14:04 |
| panda | rlandy: (checking) | 14:04 |
| quiquell | panda: and there is also service docker start with normal user, for that I think wheel is the path | 14:04 |
| quiquell | panda, rlandy: still executing "zuul autohold" command is missing there | 14:05 |
| rlandy | quiquell: what's the story with tags? I am running tags all | 14:05 |
| rlandy | to get launcher to run | 14:05 |
| quiquell | rlandy: launc_job review passing we can merge it, and the parent https://review.rdoproject.org/r/#/c/18535/ | 14:05 |
| *** weshay has joined #oooq | 14:06 | |
| quiquell | rlandy: what I usually do is --skip-tags start to just do launch job | 14:06 |
| panda | rlandy: I forgot to add a 2 at the end, the review is https://review.rdoproject.org/api/185032 | 14:06 |
| quiquell | rlandy: in case I don't want to redeploy zuul | 14:06 |
| weshay | quiquell, I have the reproducer working. THanks for finding the issue w/ ssh-keygen -m | 14:06 |
| weshay | that was it :) | 14:06 |
| quiquell | weshay: was hard to find | 14:07 |
| weshay | aye | 14:07 |
| quiquell | weshay: but it's going to be very disruptive :-/ | 14:07 |
| rlandy | quiquell: can skip start - if the users runs first time | 14:07 |
| panda | especially with id_rsa hardcoded | 14:07 |
| quiquell | rlandy: of course not, but if they just want to re launch without waiting for zuul they can skip start | 14:08 |
| quiquell | rlandy: that's what I mean | 14:08 |
| quiquell | rlandy: it's up to the user | 14:08 |
| quiquell | rlandy: I see like a lot of people having zuul all day up, and just running ansible-playbook launncher-playbook.yaml --skip-tags start | 14:09 |
| rlandy | ok - for doc | 14:09 |
| quiquell | rlandy: | 14:09 |
| *** vinaykns has joined #oooq | 14:09 | |
| quiquell | empty comment hehe | 14:09 |
| quiquell | rlandy, weshay: cloudinit public images work like a charm with sagi's change | 14:10 |
| rlandy | quiquell: didn't plan on rewriting that every day we changed stuff | 14:10 |
| quiquell | rlandy: what do you mean ? | 14:10 |
| rlandy | thought we were supposed to be using non-cloud init | 14:10 |
| rlandy | as of yesterday | 14:11 |
| quiquell | rlandy: until cloudinit was prepare | 14:11 |
| quiquell | rlandy: now it works with this review https://review.rdoproject.org/r/#/c/18515/ | 14:11 |
| quiquell | rlandy: I am going to remove the vm_password part | 14:11 |
| quiquell | rlandy: And if passes I merge so no need for image sharing anymroe | 14:11 |
| rlandy | Workflow | 14:11 |
| rlandy | -1 Sagi Shnaidman | 14:11 |
| quiquell | rlandy: I think it's because vm_password thing | 14:12 |
| weshay | cool going to try that | 14:12 |
| quiquell | rlandy: it's really not needed just to connect to console | 14:12 |
| quiquell | weshay, rlandy: Let's merge launch review first please | 14:12 |
| weshay | quiquell, rlandy and I found something on https://review.rdoproject.org/r/#/c/18279/50 | 14:12 |
| weshay | which one? | 14:12 |
| quiquell | weshay: that one, is like the main missing one | 14:13 |
| quiquell | weshay: it depends on the parent review | 14:13 |
| weshay | ya.. | 14:13 |
| weshay | mkdir -p isn't going to hurt anyting | 14:13 |
| quiquell | weshay: doing, thanks ! | 14:14 |
| *** jfrancoa has joined #oooq | 14:14 | |
| quiquell | weshay: mkdir -p is not needed | 14:15 |
| weshay | quiquell, for you | 14:15 |
| quiquell | weshay: this is already an empty cloned repo, is failing at you ? | 14:15 |
| weshay | YES | 14:16 |
| weshay | add it please | 14:16 |
| weshay | it's not going to hurt | 14:16 |
| quiquell | why it's not failing at CI ? | 14:16 |
| weshay | quiquell, that is a fantastic question | 14:16 |
| weshay | one we should talk about and explore | 14:16 |
| quiquell | weshay: CI there is passing and checking job has runn with success | 14:16 |
| quiquell | weshay: fixed | 14:17 |
| quiquell | weshay: we need to merge first the parent https://review.rdoproject.org/r/#/c/18535/ | 14:18 |
| quiquell | weshay: to fix CI(CI) | 14:18 |
| *** dsneddon has joined #oooq | 14:21 | |
| quiquell | weshay: thanks | 14:22 |
| marios | quiquell: updated https://review.rdoproject.org/r/#/c/18539/2/playbooks/tripleo-ci-reproducer/templates/run.yaml.j2 | 14:23 |
| rlandy | quiquell: weshay: did you talk about the testproject dir cloning? | 14:24 |
| rlandy | testproject.path | 14:25 |
| rlandy | clones directly into temp dir | 14:25 |
| weshay | kind of | 14:25 |
| weshay | quiquell, was wondering why the ci wasn't catching the issue.. I agreed w/ him | 14:25 |
| quiquell | marios: yep that's it, is already failing at "host" it's not libvirt but maybe the failure is legit and you don't have to wait for libvirt | 14:26 |
| quiquell | rlandy: yep, don't know why it's failing at weshay without mkdir -p CI is running ok | 14:27 |
| quiquell | rlandy: and we test job at CI | 14:27 |
| rlandy | think it's a preferences thing | 14:27 |
| panda | ykarel: master promoting | 14:27 |
| quiquell | rlandy: with scp ? | 14:28 |
| marios | quiquell: thanks will switch to something else for a bit now | 14:28 |
| quiquell | rlandy: maybe an alias of scp ? | 14:28 |
| marios | quiquell: to revisit thanks for help! | 14:28 |
| panda | ykarel: I restarted the script, but for a while there, I saw some really weird errors | 14:28 |
| panda | ykarel: like configparser wasn't finding the [main] section in the ini file | 14:28 |
| panda | wat ? | 14:28 |
| *** dsneddon has quit IRC | 14:28 | |
| panda | /me shrugs | 14:29 |
| weshay | quiquell, when I clone test project.. that dir is not there | 14:30 |
| weshay | quiquell, what happens when you or ci clones it? | 14:30 |
| quiquell | weshay: launc_job is not your clone | 14:30 |
| quiquell | weshay: I mean it creates a tmpdir and clone there | 14:30 |
| quiquell | weshay: Looks like scp is creating the dir | 14:30 |
| weshay | ya.. no shit | 14:30 |
| weshay | the hooks dir is not present | 14:31 |
| weshay | quiquell, go clone it.. report back to me :) | 14:31 |
| quiquell | weshay: same as in CI I supose | 14:31 |
| weshay | I'll clone it.. and report back to you | 14:31 |
| weshay | there is a diff | 14:31 |
| quiquell | weshay: fix it's working at CI is good to have it | 14:33 |
| quiquell | The next most important stuff for repro is this now https://review.rdoproject.org/r/#/c/18515/ | 14:33 |
| quiquell | to enable cloudinit | 14:33 |
| quiquell | images | 14:33 |
| marios | weshay: panda looks like the panko gate is borked for py37, otherwise scen2 done https://review.openstack.org/#/q/topic:replace-scen2 for replacement of the multinode job on master | 14:34 |
| weshay | quiquell, why does the ci checkout testproject w/ a diff result than me? | 14:34 |
| quiquell | weshay: should be the same and it should have fail at scp | 14:35 |
| quiquell | weshay: btw have you see this ? https://tree.taiga.io/project/morucci-software-factory/us/2245 | 14:37 |
| ykarel | panda, ack, Thanks | 14:37 |
| quiquell | weshay: https://review.openstack.org/#/q/topic:freeze_job+(status:open+OR+status:merged) | 14:38 |
| *** jfrancoa has quit IRC | 14:38 | |
| panda | quiquell: invalid certificate while pulling docker.io/rdoci/gerrit | 14:41 |
| quiquell | panda: that's new | 14:41 |
| quiquell | panda: can you try docker run rdoci/gerrit ? | 14:41 |
| panda | Get https://production.cloudflare.docker.com/registry-v2/docker/registry/v2/blobs/sha256/42/4239240672f4e9c26059500ab02472e744c8152a4c81060aaa22aae789fed2a3/data?verify=1548430245-JqSlfRSq1%2FMyb10ilv%2BULPJE0D4%3D: x509: certificate has expired or is not yet valid | 14:41 |
| panda | /usr/bin/docker-current: manifest for docker.io/rdoci/gerrit:latest not found | 14:42 |
| panda | do I have to setup a particular registry ? | 14:43 |
| marios | can anyone who isn't sshnaidm|off and is here update http://cistatus.tripleo.org/promotion/ to add periodic-tripleo-ci-centos-7-scenario001-standalone-master/ and friends? | 14:43 |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-scenario001-multinode-oooq-container @ https://review.openstack.org/602248, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/560445, (1 more message) | 14:43 |
| quiquell | panda: nope | 14:43 |
| marios | weshay: ? ^ | 14:43 |
| quiquell | panda: can you paste me date from your command line ? | 14:43 |
| quiquell | panda: could be that you have clocks problems there | 14:44 |
| panda | Wed Jan 23 21:12:16 GMT 2019 | 14:44 |
| panda | it's clearly wednesday today | 14:44 |
| quiquell | panda: but we are 25 not 23 | 14:44 |
| panda | quiquell: are you from the future ? | 14:44 |
| marios | panda: are you in a job in a job? i think you forgot to set ntp | 14:44 |
| quiquell | panda: today is 25th not 23th | 14:45 |
| quiquell | panda: They could have update the cert and invalidate old ones like these days | 14:45 |
| panda | quiquell: OMG so i come from the past | 14:45 |
| quiquell | panda: yep | 14:45 |
| quiquell | panda: worst part is that you didn't kno | 14:46 |
| marios | total jobception | 14:46 |
| marios | told ya this would happen | 14:46 |
| quiquell | panda: maybe you will have your salary twice or the like ? | 14:46 |
| quiquell | marios: love your wordings in general | 14:46 |
| quiquell | marios: jobception it is | 14:46 |
| quiquell | marios: reprojobception | 14:46 |
| marios | quiquell: heh i think you said it! | 14:46 |
| panda | quiquell: all I have to do is making my bak account come from the past too | 14:46 |
| panda | AnsiballZ_docker_service.py ? | 14:47 |
| panda | lol nice user name | 14:47 |
| quiquell | XD | 14:47 |
| panda | marios: you have no idea, I'm running a job in gerrit to spawn a VM in openstack with fedora28 that runs the reproducer playbook to create cotainers of a zuul + gerrit to run zuul + gerrit job on openstack | 14:49 |
| panda | we are way beyond inception, it's like a palindrome situation. | 14:49 |
| panda | it's boustrophedon | 14:50 |
| quiquell | panda: and the job can run all that again :-) | 14:51 |
| panda | ooohh, so the neverending story. | 14:51 |
| quiquell | panda: we can end if we replace the standalone dry run with tripleo-ci-reproducer-fedora-28-host we can end discovering black matter | 14:52 |
| rlandy | weshay: quiquell: panda: https://review.openstack.org/#/c/631067/ updated - let me know if I am missing anything else | 14:52 |
| panda | rlandy: did you add the neutron spec ? | 14:52 |
| rlandy | quiquell: lastly, the libvirt joy ... | 14:52 |
| rlandy | panda: of course | 14:52 |
| panda | great | 14:52 |
| rlandy | it's the main feature | 14:52 |
| quiquell | rlandy: centos is sometimes failing at int, no undercloud ip :-/ | 14:52 |
| rlandy | panda; I always take your feedback seriously | 14:52 |
| rlandy | quiquell:ci? | 14:53 |
| rlandy | or manual runs | 14:53 |
| quiquell | rlandy: yep | 14:54 |
| rlandy | quiquell: so ... from tq - we need more than roles in the path, we need libraries etc. | 14:54 |
| rlandy | quiquell, : what changed in centos? | 14:54 |
| quiquell | rlandy: yep we need libraries from it | 14:54 |
| rlandy | ci uses same images as befire | 14:54 |
| panda | mmmh I don't think zuul is going to start | 14:54 |
| rlandy | quiquell: anyways ... | 14:54 |
| panda | ah yes! Status code was 500 and not [200]: HTTP Error 500: Internal Server Error | 14:54 |
| panda | yeeeee | 14:54 |
| quiquell | rlandy: nothing changes is just tthat is the first time we have libvirt at CI so we catch this intermitent errors | 14:54 |
| quiquell | rlandy: At least that I know of | 14:55 |
| marios | seriously only sshnaidm|off can update the http://cistatus.tripleo.org/ ? | 14:55 |
| marios | we should fix that | 14:55 |
| rlandy | panda got an error upgrade | 14:55 |
| marios | i mean its not fair to him either | 14:55 |
| quiquell | marios: I think panda is part of it too | 14:55 |
| marios | rlandy: do you know? 16:55 < marios> seriously only sshnaidm|off can update the http://cistatus.tripleo.org/ ? | 14:55 |
| * marios stares panda | 14:55 | |
| panda | quiquell: marios I'm not | 14:56 |
| quiquell | rlandy: about tq libraries, are we able to load them in the middle of the role ? | 14:56 |
| quiquell | rlandy: don't know if we can do that | 14:56 |
| rlandy | marios: we have push rights to that repo | 14:56 |
| rlandy | weshay and I | 14:56 |
| rlandy | quiquell: yep - needed to add a library path | 14:57 |
| weshay | arxcruz|ruck, you avail in a min? | 14:57 |
| *** dsneddon has joined #oooq | 14:57 | |
| rlandy | the roles path was automatically there | 14:57 |
| arxcruz|ruck | weshay: yes | 14:57 |
| quiquell | rlandy: thing is, no problem with roles this is just loading yaml, librarys are python code I don't know if ansible can "eval" at runtime | 14:57 |
| rlandy | quiquell: correct, my friend .... | 14:58 |
| rlandy | libvirt module was not available | 14:58 |
| rlandy | so we could get o some stuff but not all | 14:58 |
| rlandy | anyways - will go back to that | 14:59 |
| panda | arxcruz|ruck: I had to restart the dlrn-promoter service in the promoter server, it was hanging with weird errors. | 14:59 |
| marios | rlandy: can we please add the periodic-tripleo-ci-centos-7-scenario001-standalone-master/ and friends | 14:59 |
| marios | rlandy: i can post a review if its something like that or a pull request somewhere? | 14:59 |
| arxcruz|ruck | panda: yup, i was seeing your conversation with ykarel earlier :) | 14:59 |
| panda | quiquell: is 500: Internal Server Error the step in which I have to create the new key ? | 14:59 |
| rlandy | marios ack - it's a pull request to https://github.com/sshnaidm/sova | 14:59 |
| rlandy | https://github.com/sshnaidm/sova/graphs/contributors | 15:00 |
| quiquell | panda: could be | 15:00 |
| rlandy | bunch of people have managed it | 15:00 |
| marios | rlandy: https://github.com/sshnaidm/sova/pull/53 | 15:00 |
| marios | rlandy: found an earlier one sorry thanks | 15:00 |
| marios | doing | 15:00 |
| quiquell | ~tripleo-ci-reproducer> docker-compose logs -f scheduler | 15:00 |
| quiquell | ~/tripleo-ci-reproducer | 15:00 |
| *** dsneddon has quit IRC | 15:01 | |
| quiquell | panda: what delimiters did have your keys "RSA" or "OPENSSH" ? | 15:01 |
| rlandy | marios: at least weshay has merge rights there - never actually tested my powers | 15:01 |
| marios | rlandy: thanks gonna post in a sec and will ping you for test :D | 15:01 |
| marios | thanks | 15:01 |
| quiquell | rlandy: do yoiu have the review that install tq at reproducer role ? | 15:02 |
| panda | -----BEGIN OPENSSH PRIVATE KEY----- | 15:02 |
| rlandy | lol - the "OPENSSH" monster again | 15:02 |
| panda | yeee | 15:02 |
| rlandy | quiquell; ^^ we have to doc that in capital letters | 15:02 |
| rlandy | go with RSA | 15:02 |
| rlandy | move on people | 15:02 |
| weshay | rlandy, we can test for it.. in the repro script | 15:02 |
| weshay | next sprint :) | 15:03 |
| rlandy | weshay: this bash script is becoming like a genetic testing screening list | 15:03 |
| rlandy | we test for every possible thing that can go wrong | 15:03 |
| rlandy | and scare you | 15:03 |
| weshay | test key, test for ansible, etc | 15:03 |
| weshay | lolz | 15:03 |
| rlandy | that's what happens when you let the CI team on code development | 15:04 |
| rlandy | is your computer powered on ??? | 15:04 |
| weshay | nanny state | 15:04 |
| rlandy | but we do not track your data | 15:05 |
| weshay | we should! | 15:05 |
| * rlandy amends review to track user password | 15:06 | |
| weshay | rlandy, quiquell cloud-init worked for me :) | 15:07 |
| rlandy | awesomeness | 15:07 |
| rlandy | finally weshay and his key problems solved | 15:08 |
| quiquell | rlandy: commented https://review.openstack.org/#/c/631067 | 15:13 |
| marios | rlandy: little help please? the periodic jobs aren't tracked here https://github.com/marios/sova/blob/a47cc8002cded2d30cb53af984df798765b67821/tripleoci/config.py#L103 | 15:14 |
| quiquell | weshay: yep cloudinit is the way to go | 15:14 |
| quiquell | weshay: let's sagi finish the review so we go for it, I am preparing the playbook to update those images | 15:14 |
| rlandy | quiquell: thanks - will fix | 15:14 |
| quiquell | weshay: no more image sharing needed with that | 15:14 |
| marios | rlandy: so not sure where to add periodic-tripleo-ci-centos-7-scenario001-standalone-master (can't see where any of the periodic jobs are tracked there) | 15:14 |
| rlandy | marios: looking | 15:14 |
| marios | rlandy: thnks | 15:15 |
| rlandy | 'https://github.com/marios/sova/blob/master/tripleoci/periodic.py | 15:15 |
| rlandy | that I guess | 15:15 |
| rlandy | I have added some | 15:15 |
| * rlandy check requests | 15:15 | |
| marios | rlandy: right but there are no job definitions there | 15:17 |
| marios | rlandy: k thanks will revisit monday thanks for checking | 15:17 |
| rlandy | think he just picks them up | 15:17 |
| quiquell | rlandy, weshay: To be able to stream console lojs at command line https://review.rdoproject.org/r/#/c/18475/ | 15:17 |
| rlandy | marios: can looking into it | 15:17 |
| quiquell | so we people can do "finger uuid@localhost" to get the logs of running jobs and not at "Waiting for job to finish..." | 15:18 |
| quiquell | s/lojs/logs/ | 15:18 |
| marios | rlandy: not hight priority but if you have timethanks. been looking at them thinking about wiring them into promotions and finding logs in http://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/ && https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic&job_name=periodic-tripleo-ci-centos-7-scenario001-standalone-master | 15:19 |
| panda | mmh, changed key, removed containers and images, relaunched, still HTTP Error 500: Internal Server Error | 15:19 |
| panda | quiquell: how do I check if the key is the problem ? | 15:20 |
| quiquell | panda: look at your private key | 15:20 |
| quiquell | panda: do they have the "RSA" delimiters or the "OPENSSH" delimiters ? | 15:20 |
| quiquell | panda: -----BEGIN RSA PRIVATE KEY----- | 15:21 |
| quiquell | panda: or | 15:21 |
| quiquell | panda: -----BEGIN OPENSSH PRIVATE KEY----- | 15:21 |
| panda | RSA | 15:21 |
| rlandy | marios; I think sova just scans and picks up periodic | 15:21 |
| quiquell | panda: then you are good | 15:21 |
| rlandy | when the jobs run. we can watch it | 15:21 |
| quiquell | panda: tmate ? | 15:21 |
| quiquell | panda: I want to go to the past | 15:21 |
| panda | ok | 15:21 |
| panda | I'll paste the link here in a second | 15:22 |
| panda | :P | 15:22 |
| quiquell | panda: you don't forget | 15:22 |
| *** ykarel is now known as ykarel|away | 15:22 | |
| panda | and don't forgive | 15:22 |
| quiquell | rlandy: we have to test fedora28 libvirt images :-/ | 15:22 |
| quiquell | rlandy: right now only centos7 jobs can be run | 15:23 |
| quiquell | rlandy: well they will run but it's going to be a lie | 15:23 |
| rlandy | a lie - ha | 15:25 |
| rlandy | CI lies | 15:25 |
| rlandy | quiquell: k - what can we do? | 15:26 |
| quiquell | rlandy: it's not just CI every run of fedora28 at libvirt is going to be a lie | 15:26 |
| quiquell | rlandy: let me look at libvirt roles | 15:26 |
| quiquell | rlandy: I think you can pass the image, but for example "epel" doe not make sense I will update upstream stuff | 15:27 |
| rlandy | marios: to wire into promotions, you had a pipeline job and a criteria job, right? | 15:27 |
| rlandy | quiquell: ci vs manual review | 15:28 |
| rlandy | manual run | 15:28 |
| *** ykarel|away has quit IRC | 15:29 | |
| weshay | ssbarnea|bkp2, fyi https://review.rdoproject.org/r/#/c/18517/ | 15:33 |
| marios | weshay: fyi https://review.rdoproject.org/r/#/c/18091/3//COMMIT_MSG@7 | 15:35 |
| marios | rlandy: no criteria job yet, going to use that ^^^ | 15:35 |
| *** dsneddon has joined #oooq | 15:36 | |
| marios | rlandy: that what you meant? | 15:37 |
| rlandy | yes | 15:37 |
| rlandy | we already approved adding to periodic pipeline | 15:38 |
| rlandy | not sure what else there is | 15:38 |
| marios | rlandy: ack | 15:38 |
| *** dsneddon has quit IRC | 15:41 | |
| *** chem has quit IRC | 15:48 | |
| rlandy | marios: wrt sova, I think we are missing the branch definitions | 15:49 |
| rlandy | consider ... | 15:49 |
| rlandy | https://github.com/sshnaidm/sova/blob/master/tripleoci/config.py#L211 | 15:49 |
| *** chem has joined #oooq | 15:49 | |
| rlandy | so standlone would need "tripleo-ci-centos-7-scenario004-standalone", and "tripleo-ci-centos-7-scenario004-standalone"-master, | 15:49 |
| rlandy | oops | 15:49 |
| marios | gerrit down? | 15:49 |
| rlandy | "tripleo-ci-centos-7-scenario004-standalone-master" | 15:50 |
| rlandy | works for me | 15:50 |
| marios | rlandy: ack thanks i am doing something else now but will revisit | 15:50 |
| rlandy | I think that is it | 15:50 |
| marios | rlandy: yeah i did check one of the branches but there are many | 15:50 |
| rlandy | there is only master now | 15:51 |
| rlandy | OVB has a ton | 15:51 |
| marios | rlandy: for sova | 15:51 |
| rlandy | the script then just parses for a periodic version of those jobs | 15:52 |
| marios | rlandy: https://github.com/sshnaidm/sova check the branches | 15:52 |
| marios | rlandy: there are cont downci alldownci gates gh-pages openshift etc etc | 15:52 |
| *** ykarel|away has joined #oooq | 15:52 | |
| rlandy | marios; that's not the branches I am I talking about | 15:53 |
| rlandy | I am talking about the job definitions | 15:53 |
| rlandy | I can just add this | 15:53 |
| quiquell | drop now have a good weekend !!! | 15:56 |
| *** quiquell is now known as quiquell|off | 15:56 | |
| *** udesale has quit IRC | 15:56 | |
| weshay | oh shit.. I have the standalone deploying | 15:57 |
| weshay | rlandy, ^ | 15:58 |
| rlandy | weshay: is that a bad thing? | 15:58 |
| rlandy | sounds like good news | 15:59 |
| rlandy | weshay: if zuul is already up and running, quiquell|off says we can --skip-tags run so we just launch the job | 16:01 |
| * rlandy hasn't tried that yet | 16:01 | |
| weshay | https://1ce.org/1#SkWJm3dQE | 16:01 |
| weshay | **********************************************************************************************************│2019-01-25 15:49:04 | Deployment successful! | 16:02 |
| weshay | and it worked!!! | 16:03 |
| weshay | \0/ | 16:03 |
| weshay | holy cow | 16:03 |
| jrist | hahaha | 16:03 |
| weshay | I'm on the node.. checking out the logs | 16:03 |
| weshay | wwwweeeeeeeeeeeeeeeeeeeeeeeeeee | 16:03 |
| weshay | :) | 16:03 |
| weshay | jrist, I'm going to blow your mind | 16:03 |
| jrist | likely | 16:04 |
| jrist | it's an easy thing to do | 16:04 |
| rlandy | woohoo - finally!! | 16:05 |
| *** dsneddon has joined #oooq | 16:05 | |
| weshay | rlandy, I am having to recheck the change to get things kicked | 16:10 |
| weshay | but we'll figure that out | 16:10 |
| *** dsneddon has quit IRC | 16:10 | |
| rlandy | weshay: the testproject change or the upstream change? | 16:10 |
| rlandy | if it's the testproject change, that is correct | 16:10 |
| panda | mmhh stil internal server error | 16:10 |
| panda | I'll try again monday. | 16:10 |
| rlandy | by design | 16:10 |
| rlandy | like any other zuul | 16:11 |
| weshay | Add job to launch | 16:11 |
| weshay | change | 16:11 |
| weshay | nothing in tripleo-ci-reproducer/logs | 16:12 |
| rlandy | not sure I understand | 16:13 |
| rlandy | sec - just committing sova change | 16:13 |
| weshay | marios, fyi.. scen1-4 standalone are passing in promotion | 16:16 |
| * weshay thinks we can merge https://review.rdoproject.org/r/#/c/18091/3/ci-scripts/dlrnapi_promoter/config/master.ini | 16:16 | |
| rlandy | weshay: marios: https://github.com/sshnaidm/sova/pull/54 | 16:17 |
| * weshay looks | 16:17 | |
| rlandy | weshay: want me to w+ https://review.rdoproject.org/r/#/c/18091/? | 16:20 |
| weshay | aye | 16:20 |
| weshay | marios++ | 16:20 |
| hubbot1 | weshay: marios's karma is now 6 | 16:20 |
| weshay | rlandy, odd.. my zuul went down | 16:21 |
| weshay | 06a4f8dd62a4 rdoci/zuul:stable "/usr/bin/dumb-ini..." About an hour ago Exite│-rw-------. 1 zuul root 2250 Jan 25 15:16 undercloud-passwords.conf | 16:21 |
| weshay | d (0) About an hour ago | 16:21 |
| rlandy | wierdness | 16:22 |
| *** panda is now known as panda|off | 16:23 | |
| weshay | rlandy, ok.. so you want / need some help w/ libvirt | 16:23 |
| weshay | testing? | 16:23 |
| rlandy | marios: you still have a -1 on https://review.rdoproject.org/r/#/c/18091/ | 16:23 |
| rlandy | we want to w+ that | 16:23 |
| rlandy | weshay: yes - I could use some help ther | 16:23 |
| rlandy | there | 16:23 |
| rlandy | will explain - there are a couple issues | 16:24 |
| weshay | k.. let me reboot | 16:24 |
| weshay | and fix my audio | 16:24 |
| weshay | issue | 16:24 |
| weshay | bb in 3 min | 16:24 |
| rlandy | sure | 16:24 |
| weshay | rlandy, ok.. ready when ever | 16:28 |
| rlandy | joining | 16:28 |
| marios | weshay: rlandy sorry was doing status | 16:39 |
| marios | sec | 16:39 |
| marios | rlandy: removed the -1 thanks | 16:39 |
| rlandy | will merge | 16:40 |
| marios | weshay: re the promotion passing added comment https://review.rdoproject.org/r/#/c/18091/3//COMMIT_MSG@7 scen1 had one fail otherwise yeah 3 green runs each | 16:40 |
| marios | weshay: is why i wanted to add them to sova | 16:40 |
| marios | weshay: but will revisit on monday | 16:40 |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-scenario001-multinode-oooq-container @ https://review.openstack.org/602248, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/560445, (1 more message) | 16:43 |
| marios | status'd | 16:48 |
| marios | have a good friday all its cardioclock | 16:48 |
| *** chem has quit IRC | 16:48 | |
| *** chem has joined #oooq | 16:50 | |
| marios | rlandy: thanks for https://github.com/sshnaidm/sova/pull/54! k now actually bai | 17:05 |
| *** derekh has quit IRC | 17:05 | |
| chandankumar | weshay: on tripleo side, do we support nova config drive feature? | 17:09 |
| weshay | chandankumar, ask in #tripleo and or look in upstream doc | 17:09 |
| *** brault has quit IRC | 17:13 | |
| *** brault has joined #oooq | 17:14 | |
| marios | weshay: fyi https://review.openstack.org/#/c/631491/ 3 out of 4 :) have to check what happens on that scen 3 and sanity check the rest but still... | 17:23 |
| marios | weshay: damn no not yet | 17:23 |
| marios | weshay: here we'll see https://review.openstack.org/#/c/631492/ | 17:23 |
| marios | false positive :/ sorry | 17:23 |
| marios | weshay: is still running | 17:23 |
| weshay | k. .will look in a sec.. working w/ ronelle | 17:24 |
| marios | ok no worries will revisit i just thought they had passed ttyl all | 17:24 |
| marios | (nope they fail anyway bye) | 17:25 |
| marios | (oh well ... ) | 17:25 |
| *** dsneddon has joined #oooq | 17:29 | |
| *** dsneddon has quit IRC | 17:34 | |
| *** dsneddon has joined #oooq | 17:42 | |
| *** jpena is now known as jpena|off | 17:44 | |
| *** chem has quit IRC | 17:48 | |
| *** chem has joined #oooq | 17:49 | |
| *** dsneddon has quit IRC | 17:54 | |
| *** dsneddon has joined #oooq | 17:54 | |
| *** bogdando has quit IRC | 17:59 | |
| *** ykarel|away has quit IRC | 18:02 | |
| weshay | rlandy, https://review.openstack.org/#/c/631067/22/roles/create-zuul-based-reproducer/templates/launcher-playbook.yaml.j2 | 18:02 |
| *** ccamacho has quit IRC | 18:05 | |
| *** trown is now known as trown|lunch | 18:30 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-scenario001-multinode-oooq-container @ https://review.openstack.org/602248, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/560445, (1 more message) | 18:43 |
| *** ska has joined #oooq | 18:45 | |
| ska | When defining the instackenv.json file, is capabilities required? Is that how undercloud assignes roles to nodes? | 18:45 |
| *** chem has quit IRC | 18:49 | |
| *** chem has joined #oooq | 18:50 | |
| rlandy | hIt is also possible (but optional) to set Ironic node capabilities directly in the JSON file. - we assign some in the flavors ... ttps://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/overcloud-prep-flavors/templates/overcloud-prep-flavors.sh.j2#L39 | 19:01 |
| ska | rlandy: is that an ansible style template? | 19:20 |
| rlandy | yes - it is | 19:40 |
| rlandy | weshay: https://github.com/sshnaidm/sova/pull/56 | 19:49 |
| * weshay looks | 19:49 | |
| weshay | rlandy, thanks.. merged! | 19:50 |
| * rlandy hopes I got that right now | 19:52 | |
| *** trown|lunch is now known as trown | 20:01 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/560445, stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ (1 more message) | 20:44 |
| weshay | hrm.. zuul died in the same spot :( | 20:55 |
| ska | Once a Node/Host PXE boots and is configured via undercloud, does it still need to PXE boot on reboots? | 21:04 |
| rlandy | weshay: https://review.openstack.org/#/c/631067 - not yet tested - but that's the basic rewrite | 21:07 |
| weshay | rlandy, cool | 21:10 |
| weshay | rlandy, I'm taking notes setting up on a clean box | 21:10 |
| weshay | so assuming these need to get installed in the repro role | 21:11 |
| weshay | minus ansible | 21:11 |
| weshay | sudo dnf install -y ansible docker-compose git git-review | 21:11 |
| weshay | and I had to run # run newgrp docker; groups | 21:11 |
| weshay | other than that it worked | 21:11 |
| weshay | your review looks good.. | 21:12 |
| rlandy | weshay: cool - so that is user doc | 21:14 |
| rlandy | that should be in both places | 21:14 |
| rlandy | weshay: I am still not sure what the story is with tags | 21:15 |
| rlandy | I needed to run tags all | 21:15 |
| weshay | heh.. it's hot off the presses, we'll settle that next week | 21:15 |
| rlandy | quiquell|off, says not necessary | 21:15 |
| weshay | can't trust that guy | 21:16 |
| weshay | :) | 21:16 |
| rlandy | he's pretty good - you know :) | 21:16 |
| rlandy | it's usually my mistake | 21:16 |
| rlandy | sudo dnf install -y ansible docker-compose git git-review | 21:17 |
| rlandy | ^^ that could go in env setup playbook | 21:17 |
| rlandy | will test more | 21:17 |
| weshay | rlandy, something like https://review.rdoproject.org/r/18545 | 21:22 |
| weshay | rlandy, for the script.. just let's just do yum install ansible | 21:22 |
| weshay | the rest should go through ci I suppose | 21:22 |
| weshay | dam.. ok.. on two systems.. zuul dies | 21:23 |
| weshay | we have a problem | 21:23 |
| weshay | meh.. next week | 21:23 |
| rlandy | well at least it's consistent | 21:37 |
| rlandy | yep - next week ... | 21:37 |
| *** rlandy has quit IRC | 21:37 | |
| *** trown is now known as trown|outtypewww | 21:40 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/560445, stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ (1 more message) | 22:44 |
| *** chem has quit IRC | 22:51 | |
| *** chem has joined #oooq | 22:52 | |
| *** vinaykns has quit IRC | 23:07 | |
| *** kopecmartin|devc is now known as kopecmartin|off | 23:20 | |
| *** chem has quit IRC | 23:52 | |
| *** chem has joined #oooq | 23:53 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!