| *** tosky has quit IRC | 00:06 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario007-multinode-oooq-container, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035, tripleo-ci-fedora-28-standalone @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario007-multinode-oooq-container, (2 more messages) | 00:37 |
|---|---|---|
| *** rlandy has quit IRC | 00:47 | |
| *** weshay is now known as weshay_PTO | 01:26 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario007-multinode-oooq-container, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035, tripleo-ci-fedora-28-standalone @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035, tripleo-ci- (1 more message) | 02:37 |
| *** ykarel has joined #oooq | 02:53 | |
| *** agopi has quit IRC | 02:55 | |
| *** apetrich has quit IRC | 03:14 | |
| *** gkadam has joined #oooq | 03:19 | |
| *** udesale has joined #oooq | 03:30 | |
| *** udesale has quit IRC | 03:35 | |
| *** udesale has joined #oooq | 03:36 | |
| *** gkadam has quit IRC | 04:21 | |
| *** dsneddon has quit IRC | 04:37 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario007-multinode-oooq-container, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035, tripleo-ci-fedora-28-standalone @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci- (2 more messages) | 04:37 |
| *** ykarel has quit IRC | 04:47 | |
| *** dsneddon has joined #oooq | 05:04 | |
| *** ykarel has joined #oooq | 05:04 | |
| *** dsneddon has quit IRC | 05:10 | |
| *** skramaja has joined #oooq | 05:15 | |
| *** saneax has joined #oooq | 05:34 | |
| *** dsneddon has joined #oooq | 05:43 | |
| *** dsneddon has quit IRC | 05:48 | |
| *** dsneddon has joined #oooq | 06:19 | |
| *** dsneddon has quit IRC | 06:24 | |
| *** jfrancoa has joined #oooq | 06:29 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario007-multinode-oooq-container, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035, tripleo-ci-fedora-28-standalone @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci- (2 more messages) | 06:37 |
| *** dsneddon has joined #oooq | 06:52 | |
| *** dsneddon has quit IRC | 06:59 | |
| *** quiquell|off is now known as quiquell | 07:12 | |
| *** ccamacho has joined #oooq | 07:12 | |
| *** holser_ has joined #oooq | 07:14 | |
| *** dsneddon has joined #oooq | 07:20 | |
| *** saneax has quit IRC | 07:20 | |
| quiquell | marios: o/ did you progress with the repro? | 07:21 |
| marios | quiquell: not yet, i am gonna try the keys/image thing today | 07:22 |
| quiquell | Ack | 07:22 |
| quiquell | sshnaidm: I am going help with the images thing now that job launching is almost done | 07:24 |
| quiquell | sshnaidm: it's ok? | 07:24 |
| *** dsneddon has quit IRC | 07:26 | |
| *** saneax has joined #oooq | 07:27 | |
| sshnaidm | quiquell, yeah, sure, need to sync about that | 07:28 |
| sshnaidm | quiquell, but what is really needed urgent imho is OVB reproducing, take a look please in your time: https://review.rdoproject.org/r/#/c/18422/ | 07:29 |
| sshnaidm | quiquell, I'm trying to copy all needed stuff from config repo without secrets and it worked, but jobs didn't start yet.. | 07:30 |
| sshnaidm | quiquell, if you can help with that, would be great | 07:30 |
| *** saneax has quit IRC | 07:30 | |
| chandankumar | sshnaidm: Hey | 07:35 |
| chandankumar | sshnaidm: I didnot understand the last comment on this review https://review.openstack.org/#/c/628421/ | 07:36 |
| quiquell | sshnaidm: cool, the more with take the better, will test it | 07:38 |
| quiquell | sshnaidm: can you help me with the livirt tq review? | 07:38 |
| quiquell | So we can merge libvirt? | 07:38 |
| sshnaidm | quiquell, looking | 07:47 |
| sshnaidm | chandankumar, I wonder if this is required for os tempest role to work, because it adds a lot of dependencies to our requirements file and in this case - dependencies that you don't control | 07:48 |
| chandankumar | sshnaidm: we just need two dependencies | 07:49 |
| sshnaidm | chandankumar, I can imagine situations when we run arbitrary job or just quickstart and have bootstrap fails because of some problem with one of this galaxy role | 07:49 |
| sshnaidm | chandankumar, can we maybe run jobs on these repos? ansible-role-python_venv ansible-config_template ? | 07:51 |
| chandankumar | sshnaidm: yes, we can run the jobs, once we have os_tempest standalone job working | 07:51 |
| *** gkadam has joined #oooq | 07:51 | |
| sshnaidm | chandankumar, ok, would be nice to check them with some standalone | 07:51 |
| chandankumar | sshnaidm: from last run https://review.openstack.org/#/c/627500/ | 07:52 |
| chandankumar | sshnaidm: http://logs.openstack.org/00/627500/65/check/tripleo-ci-centos-7-standalone-os-tempest/198ae77/ we have working os_tempest | 07:52 |
| chandankumar | sshnaidm: only problem here I need to fix is to collect-logs for collecting stackviz and tempest related files at one place | 07:52 |
| chandankumar | sshnaidm: http://logs.openstack.org/00/627500/65/check/tripleo-ci-centos-7-standalone-os-tempest/198ae77/logs/undercloud/var/log/tempest/stestr_results.html.gz | 07:53 |
| chandankumar | sshnaidm: I just need help on manage this part https://review.openstack.org/#/c/628415/33/playbooks/multinode-standalone.yml@68 | 07:54 |
| *** dsneddon has joined #oooq | 07:54 | |
| sshnaidm | chandankumar, I see, part of these vars seems like candidates to our common-vars | 07:55 |
| chandankumar | sshnaidm: yesterday i used undercloud_network_cidr to get these values but it is showing undefined vars | 07:56 |
| chandankumar | sshnaidm: so I hardcoded here, I need someway to share between os_tempest task | 07:56 |
| sshnaidm | quiquell, I see the patch fails in CI https://review.openstack.org/#/c/629839/ - need to fix linters | 07:57 |
| sshnaidm | chandankumar, is it cached variable? I'm afraid not, that's why you get "undefined".. | 07:57 |
| chandankumar | sshnaidm: and one more issue how to handle precommit failure here https://review.openstack.org/#/c/628415/33/playbooks/multinode-standalone.yml@55 | 07:58 |
| chandankumar | sshnaidm: nope not cached, I can make it cache and fix it | 07:58 |
| chandankumar | sshnaidm: in os_tempest run_tempest takes yes or no not true or false but precommit complain here | 07:58 |
| *** dsneddon has quit IRC | 07:58 | |
| sshnaidm | chandankumar, hmm, idk if it's possible there, but ok, we can hardcode them too in this case I think | 07:59 |
| sshnaidm | chandankumar, can you point me to code with "yes" or 'no'? | 07:59 |
| *** dsneddon has joined #oooq | 07:59 | |
| sshnaidm | chandankumar, if it's a string, just use quoted "yes" | 07:59 |
| sshnaidm | chandankumar, look at that: https://github.com/openstack/openstack-ansible-os_tempest/blob/7e6e614ff035c428fa34384169da7dfc00ccb103/tasks/main.yml#L47-L57 | 08:01 |
| sshnaidm | chandankumar, they use "| bool" so you can use string "yes" or "not" in quotes | 08:02 |
| sshnaidm | chandankumar, but tnh it's worth to change it in os_tempest role to use booleans instead, should be trivial patch | 08:02 |
| *** dsneddon has quit IRC | 08:04 | |
| *** ykarel is now known as ykarel|lunch | 08:06 | |
| *** kopecmartin|off is now known as kopecmartin | 08:08 | |
| *** apetrich has joined #oooq | 08:10 | |
| *** jtomasek has joined #oooq | 08:10 | |
| quiquell | sshnaidm: ack | 08:12 |
| quiquell | sshnaidm: can you share the logs with the issue with ovb and cloned config project? | 08:12 |
| sshnaidm | quiquell, there is no logs, job doesn't start if you put it in test1 repo zuul.yaml | 08:26 |
| sshnaidm | quiquell, scheduler complains on syntax error, but it can be anything.. | 08:26 |
| quiquell | sshnaidm: was happening to me yesterday with standalone | 08:26 |
| sshnaidm | quiquell, I don't know what is the way to get the real error message from zuul | 08:27 |
| quiquell | sshnaidm: syntax errors are not fatal, it discard jobs that have them, that's good feature | 08:27 |
| quiquell | sshnaidm: have you check Gerrit comments ? | 08:28 |
| sshnaidm | quiquell, it's usually config error, not "syntax", but not known where exactly | 08:28 |
| quiquell | Usually zuul put stuff there is the job is not correct | 08:28 |
| sshnaidm | quiquell, yeah, nothing is there | 08:28 |
| quiquell | Check the Jobs section in the zuul web | 08:28 |
| quiquell | At tenat | 08:28 |
| sshnaidm | quiquell, maybe because job definition is in zuul-config AND in config/ repo of rdo? then need to think how to bypass it | 08:29 |
| quiquell | You can see all jobs that get correctly configured at startup | 08:29 |
| quiquell | sshnaidm: you can have multiple repos but they cannot collide on job names | 08:29 |
| sshnaidm | quiquell, try to run with my patch and configure tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001 in zuul.yaml for test1 project | 08:30 |
| quiquell | sshnaidm: we have to be very careful with base job we can only have one | 08:30 |
| *** dsneddon has joined #oooq | 08:30 | |
| sshnaidm | quiquell, then we need to exclude config repo of rdo.. | 08:30 |
| sshnaidm | quiquell, it's like with your reproducer job | 08:30 |
| quiquell | Let me take a look, maybe we can extract jus the playbooks or part of the job and regenerate our base | 08:31 |
| quiquell | With the python script | 08:32 |
| quiquell | Instead of using the cloned RDO donfig | 08:32 |
| quiquell | As trusted repo | 08:32 |
| quiquell | Btw cant we do with OBV secrets similar I did with running ci in repro? | 08:33 |
| *** dsneddon has quit IRC | 08:35 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario007-multinode-oooq-container, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035, tripleo-ci-fedora-28-standalone @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci- (2 more messages) | 08:37 |
| *** tosky has joined #oooq | 08:43 | |
| sshnaidm | quiquell, well, currently I just pass secrets to job itself in zuul.yaml for test1, but later I believe we'll detect all credentials for rdo cloud from env and pass them automatically | 08:44 |
| sshnaidm | quiquell, I copy these files because I wouldn't like to maintain two versions of OVB jobs - in rdo config and reproducer | 08:45 |
| sshnaidm | quiquell, if we'll have 2 versions, we'll need to changed definitions in reproducer each time we change something in OVB configs, or maybe periodic jobs.. it will be not convenient | 08:46 |
| sshnaidm | quiquell, I thought to copy all config repo, to remove there secrets, and use it as real config repo.. | 08:46 |
| quiquell | Agree on remove redundancy but maybe only for ovb part | 08:51 |
| *** dtantsur|afk is now known as dtantsur | 08:56 | |
| *** jpena|off is now known as jpena | 08:56 | |
| *** ykarel|lunch is now known as ykarel | 08:57 | |
| *** dsneddon has joined #oooq | 09:09 | |
| *** saneax has joined #oooq | 09:20 | |
| *** bogdando has joined #oooq | 09:29 | |
| *** saneax has quit IRC | 09:31 | |
| *** saneax has joined #oooq | 09:31 | |
| arxcruz|ruck | ykarel: have you seen this http://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-fedora-28-centos-7-containers-standalone-master/54b8e6b/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz ? | 09:34 |
| *** sanjayu_ has joined #oooq | 09:34 | |
| ykarel | arxcruz|ruck, nope, seems new | 09:35 |
| arxcruz|ruck | ykarel: also https://trunk.rdoproject.org/centos7-master/previous-current-tripleo is getting forbidden error | 09:35 |
| ykarel | arxcruz|ruck, yes that link is not existing somehow | 09:35 |
| quiquell | sshnaidm, marios: I think the shared images upstream-foobar have issues and cannot run jobs | 09:35 |
| arxcruz|ruck | ykarel: you know who to contact to fix it ? | 09:35 |
| quiquell | sshnaidm: have you try the OVB with the original images we where using ? | 09:36 |
| ykarel | arxcruz|ruck, ask jpena, it might be related to few days back when weshay_PTO did some manual promotion | 09:36 |
| arxcruz|ruck | achso | 09:37 |
| arxcruz|ruck | jpena: ^ | 09:37 |
| *** saneax has quit IRC | 09:37 | |
| jpena | looking | 09:37 |
| quiquell | sshnaidm: have you check the launcher logs ? | 09:38 |
| sshnaidm | quiquell, centos or fedora? | 09:38 |
| jpena | arxcruz|ruck: yes, the previous-current-tripleo link is broken. It is related to last week's issue, I'll fix that | 09:38 |
| marios | quiquell: ack i will try uploading my own ones today (you mean the ones you shared with us?) | 09:38 |
| quiquell | sshnaidm: fedora | 09:38 |
| arxcruz|ruck | jpena: thanks, do you need me to open a bug or something on rdo side? | 09:38 |
| jpena | arxcruz|ruck: no, it's just a minute | 09:39 |
| quiquell | sshnaidm: humm... wait centos | 09:39 |
| sshnaidm | quiquell, fedora doesn't work yet, I'm preparing the new one | 09:39 |
| arxcruz|ruck | jpena: k, thanks! | 09:39 |
| jpena | do you remember by chance what was the previous hash? | 09:39 |
| sshnaidm | quiquell, centos should work | 09:39 |
| arxcruz|ruck | sshnaidm: you meant fedora-28 periodic job? | 09:39 |
| quiquell | sshnaidm: was not working here | 09:39 |
| quiquell | sshnaidm: will recheck, it was the only difference | 09:39 |
| sshnaidm | arxcruz|ruck, no, it's about reproducer | 09:39 |
| arxcruz|ruck | ok | 09:39 |
| sshnaidm | quiquell, what is the error? | 09:39 |
| quiquell | arxcruz|ruck: sorry arxcruz|ruck | 09:39 |
| quiquell | double nick mention | 09:39 |
| quiquell | sshnaidm: puff at the office it takes more time to startup this thing | 09:40 |
| arxcruz|ruck | quiquell: I forgive you :P | 09:40 |
| sshnaidm | quiquell, do you have an office? :o | 09:40 |
| quiquell | sshnaidm: There is one here in Madrid | 09:40 |
| jpena | arxcruz|ruck: ok, previous-current-tripleo should be ok now | 09:40 |
| marios | quiquell: should i just grab whatever is latest in https://nb02.openstack.org/images/ | 09:41 |
| arxcruz|ruck | jpena: gracias! | 09:41 |
| quiquell | marios: na try those images first, maybe they work for you | 09:41 |
| *** derekh has joined #oooq | 09:41 | |
| marios | quiquell: which ones? you mean the ones you shared we tried yesterday (key issue remember) | 09:42 |
| quiquell | marios: key issue is related to zuul not being able to connect to gerrit | 09:42 |
| quiquell | marios: connection to image is done at nodepool-launcher | 09:43 |
| marios | quiquell: i though it was this https://github.com/paramiko/paramiko/issues/1305 and i was gonna try with key that has no passphrase from start | 09:43 |
| sshnaidm | quiquell, check please that we are talking about centos image with id: d3d4991a-e9ca-4072-bf6a-618874af7f74 | 09:43 |
| marios | sshnaidm: (same as mine fwiw d3d4991a-e9ca-4072-bf6a-618874af7f74 ) | 09:44 |
| sshnaidm | marios, yeah, this is it | 09:44 |
| quiquell | sshnaidm: the f28 image that is not the cloudinit did not have the teams pub keys on it ? | 09:46 |
| sshnaidm | quiquell, f28 image does have, but it requires a selinux relabel afaik, so I need to rerun virt-customize there | 09:47 |
| quiquell | sshnaidm: ack | 09:47 |
| sshnaidm | quiquell, you can use centos, it should work | 09:47 |
| quiquell | marios: use only centos-7 jobs for now | 09:47 |
| quiquell | at your repro | 09:47 |
| marios | quiquell: ok | 09:49 |
| marios | quiquell: fails at check image are uploaded | 09:56 |
| marios | quiquell: so might need to tweak the task? | 09:56 |
| quiquell | marios: what do you mean at check ? | 09:56 |
| marios | quiquell: (I just moved the os_fedora_28_image from my playbook vars) | 09:56 |
| marios | TASK [ansible-role-tripleo-ci-reproducer : Check image are uploaded] ********************************************************* | 09:56 |
| marios | "msg": "Cannot find openstack-infra-fedora-28 at the openstack cloud, you can upload one from\nhttps://nb02.openstack.org/images/ and add your ssh pub key with\nvirt-edit and upload it to your openstack cloud.\n" | 09:56 |
| quiquell | marios: You can keep it | 09:57 |
| quiquell | marios: The only thing you need to do to launch centos-7 jobs is run tripleo-ci-centos-7-standalone instead of fedora-28 | 09:57 |
| quiquell | marios: but you can keep both images at your tenant | 09:57 |
| marios | quiquell: ok sure, but i can't get to that point yet cos of the key issue? | 09:57 |
| marios | quiquell: so i'm confused now | 09:57 |
| arxcruz|ruck | is fedora 28 periodic a promotion blocker? | 09:58 |
| quiquell | marios:You need os_fedora28_image: upstream-infra-fedora-28 | 09:58 |
| quiquell | damn | 09:59 |
| quiquell | and os_centos7_image: upstream-infra-centos-7 | 09:59 |
| quiquell | So it uses the ones I share with you | 09:59 |
| *** sanjayu_ has quit IRC | 10:05 | |
| panda | arxcruz|ruck: which one ? | 10:05 |
| *** sanjayu_ has joined #oooq | 10:05 | |
| panda | arxcruz|ruck: the mixed job yes | 10:05 |
| panda | arxcruz|ruck: the new pipeline no | 10:05 |
| arxcruz|ruck | panda: periodic-tripleo-ci-fedora-28-centos-7-containers-standalone-master | 10:05 |
| panda | arxcruz|ruck: ok, that's the mixed job, and yes it couts towards promotion, except when we decide to ignore it | 10:06 |
| arxcruz|ruck | panda: ok, adding alert tag in the bug | 10:06 |
| panda | arxcruz|ruck: still the nova cell problem ? | 10:06 |
| arxcruz|ruck | panda: nope, now it's an error on heat | 10:06 |
| arxcruz|ruck | panda: https://bugs.launchpad.net/tripleo/+bug/1812837 | 10:06 |
| openstack | Launchpad bug 1812837 in tripleo "periodic fedora 28 job failing with "/bin/sh: line 1: exit: null: numeric argument required" in Run async deployment StandalonePostDeployment step" [Undecided,Triaged] | 10:06 |
| arxcruz|ruck | has anyone seen this 2019-01-22 05:28:36 | Exception: 401 Client Error: Unauthorized for url: https://trunk.registry.rdoproject.org/v2/tripleomasterF28/centos-binary-cron/manifests/tripleo-ci-testing | 10:11 |
| arxcruz|ruck | jpena: ^ | 10:11 |
| panda | arxcruz|ruck: where is it ? | 10:11 |
| arxcruz|ruck | panda: http://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-fedora-28-master-containers-build/0004cd3/logs/undercloud/home/zuul/undercloud_install.log.txt.gz | 10:11 |
| panda | arxcruz|ruck: we changed the password recently, maybe it's not updated there | 10:11 |
| panda | mmhh, that should be the secrets in zuul | 10:12 |
| panda | jpena: do you know if the secret we use was changed properly ? | 10:12 |
| jpena | panda: I'm only aware of a change in the DLRN API password, was the registry password changed too? | 10:13 |
| quiquell | sshnaidm, marios: somone has merge the libvirt patch without merging the depends-on :-( | 10:13 |
| panda | quiquell: I don't think that's possible | 10:13 |
| sshnaidm | quiquell, which patch? | 10:13 |
| panda | jpena: proably not, and anyway that job is highly experimental | 10:13 |
| panda | jpena: I'll dig into it a bit more | 10:14 |
| jpena | panda: ack | 10:14 |
| quiquell | panda, sshnaidm: https://review.rdoproject.org/r/#/c/18121/ | 10:14 |
| panda | arxcruz|ruck: don't worry, that job is really supposed to fail in creative ways | 10:14 |
| quiquell | panda: I created this project with +v superpowers to a subset of us | 10:14 |
| quiquell | panda: that's wrong from my part :-/ | 10:14 |
| sshnaidm | quiquell, we need to remove +2 from users | 10:14 |
| sshnaidm | quiquell, I mean verify +2 | 10:14 |
| panda | quiquell: oooohh, because you bypassed zuul | 10:15 |
| panda | quiquell: ouch | 10:15 |
| sshnaidm | quiquell, it should be only for zuul user | 10:15 |
| quiquell | sshnaidm: yep, damn... | 10:15 |
| quiquell | puff and now my local repro is not working, fucking awful day :-( | 10:15 |
| quiquell | sshnaidm: Let's sort this out | 10:16 |
| panda | quiquell: don't worry it can only get worse | 10:16 |
| quiquell | panda: Thanks :-P | 10:16 |
| ykarel | panda, also looks like the repository name is not accepted tripleomasterF28, running skopeo says:_ FATA[0000] invalid reference format: repository name must be lowercase, just in case u are not aware | 10:17 |
| quiquell | panda: all screw at all fronts | 10:18 |
| panda | ykarel: we merged the changes yesterday. With config repo, merging is the single most effective way to discover problems. | 10:18 |
| quiquell | sshnaidm: removing verify https://review.rdoproject.org/r/18440 | 10:19 |
| ykarel | panda, ack | 10:19 |
| panda | quiquell: well at least the fedora pipeline is currently supposed to blow up | 10:19 |
| quiquell | sshnaidm: please merge sowe don't suffer from it again | 10:19 |
| sshnaidm | quiquell, commented | 10:20 |
| panda | quiquell: verify +2 is very useful in trusted project, where you know you can break things without warning. | 10:20 |
| ykarel | arxcruz|ruck, for that fedora issue: issue is ansible is replaced with ansible-python3 recently, till now ansible was wrongly installed in Fedora | 10:21 |
| panda | ykarel: the one in mixed job ? | 10:22 |
| quiquell | sshnaidm: going to force submit things to unfuck stuff | 10:24 |
| sshnaidm | quiquell, we can just revert too | 10:24 |
| panda | language! | 10:24 |
| * panda captainpanda | 10:24 | |
| quiquell | sshnaidm: doing that and the secrets stuff | 10:25 |
| sshnaidm | panda, is "revert" so bad word? | 10:25 |
| * sshnaidm apologies | 10:25 | |
| panda | sshnaidm: no, but "quiquell" is | 10:25 |
| *** ykarel_ has joined #oooq | 10:25 | |
| *** ykarel has quit IRC | 10:26 | |
| quiquell | panda: Yep I am sorry | 10:26 |
| * ykarel_ got disconnected | 10:26 | |
| *** udesale has quit IRC | 10:28 | |
| *** udesale has joined #oooq | 10:28 | |
| panda | ykarel_: it's even worse than I imagined, in centos before we build the images we install the undercloud. Default undercloud installation uses containers. | 10:31 |
| quiquell | sshnaidm: fixed the verify stuff https://review.rdoproject.org/r/18440 | 10:31 |
| quiquell | sshnaidm: it's ok now ? | 10:31 |
| quiquell | sshnaidm: let me submit the secrets stuff first | 10:31 |
| quiquell | sshnaidm: then we remove the verify | 10:31 |
| panda | ykarel_: in fedora, the containers build job tries to install centos containers in undercloud registry, and fails | 10:31 |
| panda | ykarel_: we're not even in the containers build phase, it fails way before. WE need to use the new method | 10:32 |
| panda | ykarel_: that doesn't need undercloud | 10:32 |
| panda | ykarel_: We'll dig into it in the next sprint probably | 10:32 |
| *** udesale has quit IRC | 10:33 | |
| ykarel_ | panda, u mean the job that Alex added upstream? | 10:34 |
| panda | ykarel_: the method that that job uses. | 10:34 |
| ykarel_ | panda, if it install centos containers why looking for tripleomasterF28 namespace? | 10:34 |
| ykarel_ | panda, ack yes it run openstack overcloud image build i think | 10:35 |
| quiquell | sshnaidm: Don't know why is failing is says that the ptl group does not exists | 10:35 |
| panda | ykarel_: because that's what we specified in the kolla role, but I think he's trying to push containers to that namespace because it considers it the local registry | 10:35 |
| ykarel_ | panda, i think that's the issue using wrong namespace | 10:35 |
| panda | ykarel_: that's not the push phase of the build, that's just the undercloud install | 10:36 |
| ykarel_ | panda, ok confused, it's not mixed job | 10:36 |
| quiquell | sshnaidm: ok going to replicate like DLRN project | 10:36 |
| ykarel_ | so it will build f28 containers, and install those, right | 10:36 |
| ykarel_ | but issue is trying to pull containers which are not ready yet | 10:37 |
| panda | ykarel_: yes, but not at the moment. For this sprint the only thing that mattered was to have the first piece of the pipeline in place | 10:37 |
| sshnaidm | quiquell, just add this new group to line 37 | 10:37 |
| sshnaidm | quiquell, where "groups:" | 10:37 |
| ykarel_ | panda, ack, probably skip the installation part, and just do image build and push | 10:37 |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario007-multinode-oooq-container, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035, tripleo-ci-fedora-28-standalone @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci- (2 more messages) | 10:38 |
| ykarel_ | to be used by other job, but don't know if kolla is ready yet to build containers for f28 | 10:38 |
| ykarel_ | last status i had Alex was working on it | 10:38 |
| quiquell | sshnaidm: added some more stuff like DLRN has | 10:39 |
| panda | ykarel_: it doesn't without patches | 10:39 |
| panda | ykarel_: fetching the doc | 10:39 |
| panda | ykarel_: https://etherpad.openstack.org/p/fedora28-tripleo-containers-build | 10:40 |
| quiquell | sshnaidm: has update the review | 10:40 |
| quiquell | sshnaidm: Please take a look | 10:40 |
| ykarel_ | panda, ack, so the job using those patches ?^^ | 10:40 |
| ykarel_ | if no it doesn't matter we run installation phase or not, just wait | 10:40 |
| ykarel_ | this is different method so probably job don't use those patches | 10:41 |
| panda | ykarel_: no, that job was trying to adapt the old method to fedora | 10:41 |
| ykarel_ | panda, ack | 10:41 |
| panda | ykarel_: using ansible-rdo-kolla-build role from dmsimard | 10:41 |
| ykarel_ | ack got it | 10:41 |
| *** ykarel_ is now known as ykarel | 10:42 | |
| panda | ykarel: but the new method is not yet ready 100%, so we are in a limbo right now. | 10:42 |
| ykarel | panda, ack | 10:43 |
| ykarel | probably should wait then | 10:43 |
| panda | ykarel: yep. | 10:43 |
| panda | ykarel: the thng we were interested now was putting the job, and be sure that it doesn't break anything else. | 10:44 |
| ykarel | okk | 10:44 |
| panda | ykarel: I can reshuffle the words to say the same thing all day. | 10:45 |
| panda | :D | 10:45 |
| ykarel | :) | 10:45 |
| chandankumar | sshnaidm: regarding making undercloud_network_cidr cacheable, this var is coming from extra-common role and is consumed in standalone and which depends on extra-common but os_tempest does not depends on that, DO I need to put a task in standalone or extra-common in order to use set_facts to make it cacheable? | 10:52 |
| *** apetrich has quit IRC | 11:13 | |
| *** udesale has joined #oooq | 11:19 | |
| *** skramaja_ has joined #oooq | 11:24 | |
| *** skramaja has quit IRC | 11:25 | |
| marios | quiquell: difference between upstream_gerrit_key tripleo_ci_gerrit_key ? alternatively for upstream_gerrit key i added the new key i generated to my authorized keys in upstream gerrit. what about for tripleo_ci_gerrit_key? | 11:25 |
| quiquell | marios: nah forget about tripleo_ci_gerrit_key this is to run the CI of repro within repro itself | 11:26 |
| quiquell | marios: you just need to set upstream_gerrit_key and rdo_gerrit_key | 11:26 |
| marios | quiquell: cant cos it fails like scheduler_1 | 2019-01-22 11:23:03,368 - zuul.Scheduler - WARNING - Tenant tripleo-ci-reproducer isn't loaded | 11:27 |
| marios | scheduler_1 | 2019-01-22 11:23:05,943 - gerrit.GerritWatcher - ERROR - Exception on ssh event stream: | 11:27 |
| marios | etc | 11:27 |
| marios | quiquell: :/ | 11:27 |
| quiquell | marios: this is good | 11:27 |
| quiquell | did you see the Invalid Key issue ? | 11:28 |
| marios | quiquell: well scheduler_1 | "Private key file is encrypted" | 11:29 |
| marios | scheduler_1 | paramiko.ssh_exception.PasswordRequiredException: Private key file is encrypted | 11:29 |
| quiquell | marios: look at ~/tripleo-ci-reproducer/etc_zuul/zuul.conf | 11:29 |
| quiquell | Ahh no no | 11:30 |
| marios | quiquell: was about to ask what do look for there. but it looks sane like it copies they keys to /var/ssh right and it has the correct name in zuul.conf | 11:31 |
| marios | like /var/ssh/id_rsa_mykey | 11:31 |
| *** holser_ is now known as holser|food | 11:31 | |
| marios | quiquell: oh except for gerrit it didn't | 11:31 |
| marios | quiquell: like it has correct entry for rdo but no upstream | 11:31 |
| marios | quiquell: so maybe upstream_gerrit_key isn't working or i made an error leme check | 11:32 |
| quiquell | marios: this thing inject keys into docker images | 11:33 |
| quiquell | marios: to be sure do a docker-compose down -v | 11:33 |
| quiquell | marios: before re run the playbook | 11:34 |
| marios | quiquell: ah i didn't do that any time i just ctrl-c rerun | 11:34 |
| marios | quiquell: k | 11:34 |
| quiquell | marios: Even remove ~/tripleo-ci-reproducer | 11:34 |
| quiquell | marios: try first wint down -v (I have implement that at libvirt review) | 11:34 |
| panda | quiquell: what's the command to accept the cloud image again ? | 11:36 |
| panda | quiquell: I'll add it to the doc | 11:36 |
| quiquell | panda: image set --accept I think | 11:36 |
| quiquell | openstack ... image set --accept | 11:36 |
| marios | panda: openstack --os-cloud rdo-cloud image set --accept ID (from my notes when quiquell could remember it properly the other day) | 11:38 |
| panda | marios: ok | 11:38 |
| marios | panda: (assuming you have rdo-cloud as the name in your clouds.yaml) | 11:38 |
| panda | wow, openstack-3 if you're using python3 | 11:39 |
| quiquell | sshnaidm: running https://softwarefactory-project.io/zuul/t/rdoproject.org/stream/7d7cecc7dd0e4334907a494665c5b666?logfile=console.log | 11:42 |
| quiquell | sshnaidm: let's cross fingers | 11:42 |
| panda | mmmhh. Could not find resource d3d4991a-e9ca-4072-bf6a-618874af7f74 | 11:42 |
| quiquell | panda: maybe it timeouts | 11:43 |
| quiquell | panda: give me your tenant id again | 11:43 |
| panda | quiquell: let me check my clouds.yaml config | 11:43 |
| quiquell | panda: ack | 11:43 |
| panda | when I try openstack-3 server list I get module 'openstack.config.exceptions' has no attribute 'OpenStackConfigException' | 11:43 |
| quiquell | panda: You shoul dbe able to list pending images I think | 11:44 |
| quiquell | with image list | 11:44 |
| marios | quiquell: progress? (no key error this time round but now scheduler_1 | sqlalchemy.exc.InternalError: (pymysql.err.InternalError) (1130, "Host '172.18.0.8' is not allowed to connect to this MariaDB server") (Background on this error at: http://sqlalche.me/e/2j85) | 11:44 |
| panda | something is fishy | 11:44 |
| quiquell | panda: why openstack-3 ? | 11:44 |
| quiquell | marios: humm this is weird | 11:44 |
| panda | quiquell: I installed python3-openstackclient | 11:44 |
| quiquell | panda: ack | 11:45 |
| panda | but it's not working very well | 11:45 |
| quiquell | marios: can you do docker-compose ps <- this will show you what's up and what's not | 11:45 |
| panda | oh, I have the image list, so something is working at least | 11:45 |
| panda | quiquell: tenant id f6d8d86e1e254c86ba0809af666a4c41 | 11:46 |
| marios | http://pastebin.test.redhat.com/699769 quiquell "Account 'zuul' is not found or ambiguous\n"" | 11:47 |
| marios | quiquell: (oh thats probably cos it can't find zuul user cos cant connect mariadb) | 11:48 |
| panda | call gozer | 11:48 |
| marios | quiquell: could be selinux? | 11:48 |
| marios | quiquell: has to be enforce 0? | 11:48 |
| quiquell | gerrit config has exit 2 | 11:48 |
| quiquell | argg | 11:48 |
| panda | I think we are going to make quiquell explode. | 11:48 |
| quiquell | marios: docker-compose down -v | 11:48 |
| ykarel | panda, u installed python3-openstackclient from fedora? u can use it from delorean fedora repos, there it should be good | 11:49 |
| quiquell | marios: docker-compose up gerritconfig and look what fails now | 11:49 |
| marios | panda: :D | 11:49 |
| marios | quiquell: thanks | 11:49 |
| quiquell | panda: I would deserve it | 11:49 |
| panda | quiquell: probably, but if you explode we're in trouble with the reproducer. | 11:49 |
| panda | ykarel: what's the difference ? | 11:50 |
| marios | quiquell: ah github key ! | 11:50 |
| ykarel | panda, fedora rpms are out of sync | 11:50 |
| marios | quiquell: oh no sorry but is failing on git clone | 11:50 |
| quiquell | Ok lunch before explode | 11:50 |
| *** quiquell is now known as quiquell|lunch | 11:50 | |
| panda | ykarel: so I've installed a very old version of python3-opesntackclient ? | 11:50 |
| ykarel | panda, https://trello.com/c/wkifSL7A/659-fedora-clients-sync | 11:50 |
| ykarel | panda, yes | 11:51 |
| *** dsneddon has quit IRC | 11:51 | |
| ykarel | number80 is working on to clear clients in fedora, it might be ready by EOM | 11:51 |
| panda | ykarel: ok thanks. SHould I use the current-tripleo/rocky delorean repo ? | 11:51 |
| ykarel | panda, ^^ | 11:51 |
| panda | ykarel: when is he leaving ? | 11:51 |
| ykarel | panda, EOM | 11:52 |
| panda | ykarel: mmhhh, ok. | 11:52 |
| ykarel | panda, u can use master fedora repo | 11:52 |
| panda | AAAAA MASTER AAAA | 11:52 |
| ykarel | panda, http://trunk.rdoproject.org/fedora/current/ | 11:52 |
| * panda runs | 11:52 | |
| panda | ykarel: seems to work thanks. Updated the doc for the reproducer. | 11:57 |
| *** apetrich has joined #oooq | 11:57 | |
| ykarel | ack | 11:58 |
| panda | quiquell|lunch: who said you could go ? :D | 12:01 |
| panda | now I have to go too. | 12:01 |
| *** dsneddon has joined #oooq | 12:22 | |
| *** dsneddon has quit IRC | 12:27 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario007-multinode-oooq-container, tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates, tripleo-ci-fedora-28-standalone @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario000-multinode-oooq- (1 more message) | 12:38 |
| *** jpena is now known as jpena|lunch | 12:43 | |
| *** holser|food is now known as holser_ | 12:48 | |
| *** honza has joined #oooq | 12:53 | |
| *** quiquell|lunch is now known as quiquell | 12:54 | |
| *** dsneddon has joined #oooq | 12:59 | |
| *** dsneddon has quit IRC | 13:04 | |
| *** ykarel is now known as ykarel|afk | 13:08 | |
| *** ykarel|afk has quit IRC | 13:12 | |
| panda | quiquell: ready to be exploded again ? | 13:14 |
| panda | quiquell: where's my shared image ? | 13:14 |
| *** trown|outtypewww is now known as trown | 13:15 | |
| *** chem has quit IRC | 13:16 | |
| *** chem has joined #oooq | 13:17 | |
| quiquell | panda: delivering | 13:17 |
| panda | quiquell: really ? now ? breathe deeply! did the water broke ? there's no rush. OMG I'm gonna be a reproducer. | 13:19 |
| quiquell | Try it now | 13:19 |
| panda | quiquell: ¡bua, bua! | 13:21 |
| quiquell | zuul, zuul | 13:22 |
| panda | I'm pipenving | 13:23 |
| panda | this is so exciting | 13:23 |
| sshnaidm | quiquell, I think I know how to leave all environment after job is finished | 13:23 |
| panda | .... | 13:23 |
| sshnaidm | quiquell, right after it's started we need to stop zk container | 13:23 |
| panda | sshnaidm: wouldn't this create an inconsistencies in zuul nodepool object instance ? | 13:24 |
| quiquell | sshnaidm: holy sh... | 13:24 |
| sshnaidm | panda, which instance? | 13:24 |
| quiquell | sshnaidm: Is not better to fail the job with post ? | 13:24 |
| sshnaidm | quiquell, maybe, but tricky | 13:25 |
| panda | sshnaidm: zuul instance has a nodepool member which hold all the information on the nodes created, stopeed, available. | 13:25 |
| panda | pipenv failed | 13:25 |
| sshnaidm | panda, actually status of nodes is in zookeeper | 13:25 |
| sshnaidm | panda, and if we remove it, nodepool doesn't know what to remove | 13:26 |
| sshnaidm | and all nodes stay where they are | 13:26 |
| sshnaidm | quiquell, panda the other way is to patch a little bit nodepool for holding all nodes by default | 13:27 |
| sshnaidm | and use our version of nodepool in container | 13:27 |
| quiquell | sshnaidm: the later will take time, they told me about adding an optoin to zuul autohold to say that don't filter by job result | 13:27 |
| panda | sshnaidm: so zuul points to nodepool taht points to zookeeper that doesn't respond. How will zuul react ? | 13:27 |
| panda | quiquell: who told you ? | 13:27 |
| sshnaidm | quiquell, that would be ideal | 13:28 |
| sshnaidm | panda, nodepool doesn't care about no connection to zk | 13:28 |
| panda | quiquell: because the comment on the code seems pretty confident "We want to hold only when the job fails" | 13:28 |
| quiquell | sshnaidm: nodepool command did have this in the past but they removed it | 13:28 |
| sshnaidm | panda, and job actually is finished, you don't need them anymore | 13:28 |
| quiquell | sshnaidm: TristanC told me so | 13:28 |
| sshnaidm | quiquell, if it could be configurable, that would be great | 13:29 |
| sshnaidm | I was looking at code there, but not sure how to pass this config.. | 13:29 |
| panda | that's what I'm trying to figure out. Same thing remains for the other solutions, we need to pass a config to our reproducer | 13:30 |
| panda | quiquell: pipenv fails | 13:30 |
| quiquell | more than config would be a CLI thing | 13:30 |
| panda | An error occurred while installing msgpack==0.6.0 | 13:30 |
| quiquell | zuul autohold --skip_results or the like | 13:30 |
| *** dsneddon has joined #oooq | 13:30 | |
| quiquell | sshnaidm, panda: change need is not too much there is a part that read the tuple from autohold we need to add another stuff in that tuple | 13:31 |
| quiquell | there is a linue in the taiga task that point to the code that filter by result | 13:31 |
| *** ykarel|afk has joined #oooq | 13:31 | |
| panda | quiquell: I think we need to update the allowed sha1 in Pipfile.lock | 13:32 |
| *** rlandy has joined #oooq | 13:32 | |
| panda | quiquell: for msgpak | 13:32 |
| quiquell | panda: don't use the pipenv us the pre.yaml playbook from the role | 13:32 |
| panda | quiquell: so should I remove it from the docs ? | 13:33 |
| *** ykarel|afk is now known as ykarel | 13:33 | |
| quiquell | panda: point to pre.yaml from the playbook instead | 13:34 |
| panda | quiquell: ok, removing mentions of pipenv. Where is the pre.yaml playbook ? I don't see it in tripleo-ci-reproducer/ | 13:35 |
| panda | quiquell: ./playbooks/roles/ci-reproducer/files/projects/zuul-config/playbooks/base/pre.yaml | 13:35 |
| panda | this one ^ ? | 13:35 |
| panda | the only pre.yaml that I see in the subtree | 13:35 |
| quiquell | playbooks/tripleo-ci-reproducer/pre.yaml <- | 13:36 |
| panda | quiquell: I just cloned ci-config and it's not there | 13:37 |
| sshnaidm | panda, quiquell we need to patch this: https://github.com/openstack-infra/zuul/blob/2fd688352f5e220fda0dfc72b164144910670d95/zuul/scheduler.py#L1249 | 13:37 |
| panda | quiquell: is tehre a review with it ? | 13:37 |
| quiquell | sshnaidm: You were right the ssh key I generated does not work :-/ | 13:38 |
| panda | sshnaidm: looking into it ...but the comment seems pretty opposing the idea. If we add another status to the filter, we might as well remove the entire method. | 13:39 |
| *** dsneddon has quit IRC | 13:39 | |
| panda | ok not the entire method. | 13:40 |
| panda | spacing is confusing there | 13:40 |
| sshnaidm | need to add option to add "SUCCESS" to this list | 13:41 |
| panda | anyway, I know when proposing a patch, people will ask "what's the use case" to which I'll reply "you know the local zuul reproducer that you don't really like ?" :) | 13:41 |
| quiquell | Ahh forgot the -t rsa | 13:41 |
| panda | quiquell: I don't have a pre.yaml playbook | 13:42 |
| quiquell | sshnaidm: not that easy | 13:42 |
| quiquell | panda: at the role repo playbooks/tripleo-ci-reproducer/pre.yaml | 13:42 |
| quiquell | there it is | 13:42 |
| panda | wow, this documentation is really old | 13:44 |
| panda | quiquell: what is the role repo ? | 13:44 |
| panda | quiquell: I cloned only ci-config | 13:44 |
| quiquell | rdo-infra/ansible-role-tripleo-ci-reproducer | 13:44 |
| quiquell | It has its own repo | 13:45 |
| quiquell | yah forget about the doc, the ROLE has a README inside | 13:45 |
| sshnaidm | quiquell, easy if adding this to zuul.conf, but not easy if we want just a flag for command "autohold" | 13:45 |
| quiquell | We have to put info there | 13:45 |
| sshnaidm | well, better to ask Tristan or somebody else to do it of course :D | 13:45 |
| panda | quiquell: so we could remove the entire tripleo-ci-reproducer subtree from rdo-infra/ci-config | 13:46 |
| panda | ? | 13:46 |
| quiquell | sshnaidm: I we do at zuul.conf we have to restart to get that (I think you can signaling) | 13:46 |
| quiquell | panda: humm is still there ? | 13:46 |
| quiquell | panda: yep yep | 13:46 |
| sshnaidm | quiquell, yeah, it's tricky | 13:46 |
| *** skramaja_ has quit IRC | 13:47 | |
| *** jpena|lunch is now known as jpena | 13:47 | |
| panda | I think we can make the list of statuses to autohold configurable | 13:51 |
| panda | that would add a feature and satisfy all the requirements. | 13:52 |
| panda | yep, scheduler object has configure attribute. We can make that list self.configure.autohold_statuses instead of an hardcoded list of statuses | 13:54 |
| panda | I would have liked to get a reproducer working first, but I think it would make quiquell explode | 13:55 |
| rlandy | doc was written two sprints ago | 13:57 |
| panda | rlandy: I know, I'm trying to update it as I progress | 13:57 |
| panda | rlandy: but I'm pinging quiquell every second | 13:58 |
| rlandy | panda: for contributors or users? | 13:58 |
| panda | rlandy: right now, just to sync it with the current workflow | 13:59 |
| panda | rlandy: do you know why playbooks/tripleo-ci-reproducer/pre.yaml uses primary as host ? | 13:59 |
| rlandy | panda: because this work was done for CI | 13:59 |
| rlandy | panda: for the launcher playbook, I am adding a hosts file | 14:00 |
| panda | rlandy: ok so if I want to launch it locally I have to create an inventory tthat maps primary to localhost | 14:00 |
| panda | ok | 14:00 |
| rlandy | panda: I am at the same place in my reproducer | 14:00 |
| rlandy | panda: if you want o blue for five, I'll fill you in | 14:01 |
| rlandy | may save you some time | 14:01 |
| panda | quiquell: rlandy https://review.rdoproject.org/r/18449 | 14:03 |
| panda | rlandy: I'll ping you at the start of the next step | 14:04 |
| marios | panda: see tripleo | 14:05 |
| quiquell | rlandy, panda, marios: Found issues with key injection, sometimes it mess up the keys | 14:05 |
| quiquell | marios: that's why it's failing :-/ | 14:06 |
| marios | quiquell: so | 14:06 |
| marios | quiquell: i was waiting for you to finish with panda | 14:06 |
| quiquell | marios: have to debug it | 14:06 |
| quiquell | quiquell: like death corpse or the like ? | 14:06 |
| quiquell | marios: I was debuging CI | 14:06 |
| rlandy | quiquell: can we talk about the libvirt workflow ( not host ) | 14:06 |
| *** dsneddon has joined #oooq | 14:07 | |
| quiquell | marios: Yep | 14:07 |
| marios | quiquell: i had to remove the passphrase anyway from the key in order to get passed this error (mariadb connection) connection werror cant connect mariadb http://pastebin.test.redhat.com/699769 , needed to remove passphrase from id_rsa (hard coded?) then docker-compose up gerritconfig OK | 14:07 |
| rlandy | I have the launcher-playbook generated | 14:07 |
| rlandy | no bash needed | 14:07 |
| marios | quiquell: rerun fails like this http://pastebin.test.redhat.com/699776 - i.e. after creating id_rsa_reprozuul new keys and removing passphrase from id_rsa - try net new id_rsa without pass but same | 14:07 |
| quiquell | marios: how about scheduler ? it's working now or do you have Invalid Key ? | 14:07 |
| rlandy | but it needs some stuff around it | 14:07 |
| marios | quiquell: so i even created NEW id_rsa and new gerrit/rdo keys but fails like invalid key ^ http://pastebin.test.redhat.com/699776 | 14:08 |
| *** agopi has joined #oooq | 14:08 | |
| marios | quiquell: so in the end scheduler fails, gerritconfig fine exit 0 | 14:08 |
| marios | quiquell: for invalid key | 14:08 |
| quiquell | marios: we have a bug at the code that inject the key in the containers | 14:08 |
| quiquell | marios: I am trying to figure it out | 14:08 |
| chandankumar | marios: need some help here http://logs.openstack.org/00/627500/66/check/tripleo-ci-centos-7-standalone-os-tempest/0042800/job-output.txt.gz#_2019-01-22_13_02_56_363834 | 14:12 |
| chandankumar | marios: Here is the patch https://review.openstack.org/#/c/628415/ | 14:13 |
| quiquell | marios: got it, it's eating the last new line from keys :-/ weird thing sometimes ssh don't have problem with it | 14:14 |
| marios | chandankumar: are you setting tempest_cidr? otherwise looks like the default has issues here https://review.openstack.org/#/c/628415/34/playbooks/multinode-standalone.yml | 14:15 |
| marios | chandankumar: i mean just from quickly looking at your error message | 14:15 |
| marios | chandankumar: looks like address isn't set and the defaults don't make sense (outside of cide range) | 14:15 |
| chandankumar | marios: https://review.openstack.org/#/c/628415/34/roles/standalone/tasks/main.yml | 14:16 |
| *** dsneddon has quit IRC | 14:16 | |
| chandankumar | marios: will I remove the conditional? | 14:16 |
| quiquell | marios: | 14:17 |
| quiquell | Is that missing end line | 14:17 |
| marios | chandankumar: not sure, i mean it would set the tempest_cidr if you did so might help | 14:17 |
| marios | chandankumar: and looks like is failing for that exactly https://review.openstack.org/#/c/628415/34/playbooks/multinode-standalone.yml@71 | 14:17 |
| marios | quiquell: cool will you update the role | 14:18 |
| quiquell | marios: If the review passes CI then we are good with keys, since CI is failing now because of that | 14:18 |
| chandankumar | quiquell: I have reused the stuff from here https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/validate-tempest/defaults/main.yml#L4 | 14:19 |
| *** sanjayu_ has quit IRC | 14:19 | |
| *** sanjayu_ has joined #oooq | 14:20 | |
| marios | quiquell: thanks which review is it please | 14:21 |
| quiquell | marios: Want to test first with production key, then I put the review in place | 14:21 |
| marios | quiquell: ah k | 14:22 |
| panda | fultonj: you! | 14:22 |
| fultonj | panda: yeah me :) | 14:22 |
| panda | fultonj: commented on your patch, since you started the work we changed everything. | 14:22 |
| fultonj | really? | 14:23 |
| fultonj | i see slagle did some updates. | 14:23 |
| fultonj | i will review (was away last week) | 14:23 |
| marios | rlandy thanks for comments i updated whenever you next have time thanks https://review.rdoproject.org/r/#/q/topic:standalone-scenario-promotion | 14:23 |
| rlandy | marios: looking | 14:24 |
| panda | fultonj: I know, hope you enjoyed yourself. Now back to work ! :) | 14:24 |
| fultonj | (i was in brno getting stuff working on 8) | 14:24 |
| quiquell | rlandy, marios, panda, sshnaidm: https://review.rdoproject.org/r/18450 | 14:24 |
| quiquell | This should do it, if CI passes we are all good with keys | 14:24 |
| sshnaidm | quiquell, ok, so I updated fedora images in openstack-nodepool tenant, now they should work. Send me please your tenant ID again, I'll share it | 14:25 |
| panda | fultonj: yes, I know, as I said, you were wasting time :) | 14:25 |
| marios | quiquell: ack | 14:25 |
| quiquell | sshnaidm: Do i have to remove the old ones ? | 14:25 |
| fultonj | ha | 14:25 |
| quiquell | marios: can you use the review to see if now your key is working fine ? | 14:26 |
| panda | quiquell: does this mean you need to reshare and we need to re accept ? | 14:26 |
| sshnaidm | quiquell, I don't think so.. do you still see them? | 14:26 |
| marios | quiquell: yeah will do in a bit | 14:26 |
| *** sanjayu_ has quit IRC | 14:26 | |
| rlandy | marios: those reviews look good to me now - +2'ed them | 14:29 |
| rlandy | marios: job definitions are pretty harmless if you need to merge them | 14:29 |
| rlandy | there were other +1 votes before my comments yesterday | 14:29 |
| marios | rlandy: thanks i think the idea is to get them merged (don't they then start running?) but we wont wire them into the promotion critieria yet | 14:30 |
| marios | rlandy: do we need to do something else to make them run like add them so some jobs layout? | 14:30 |
| sshnaidm | quiquell, and for using cloud-init image you need to use docker.io/rdoci/nodepool-launcher:patched - until my patch will be merged in nodepool | 14:30 |
| quiquell | sshnaidm: ack... still sruggling with generated ssh key, after fixing new line I can ssh using the injected key in the container | 14:31 |
| quiquell | sshnaidm: but zuul fails... | 14:31 |
| sshnaidm | quiquell, zuul fails where? when connecting to gerrit? | 14:32 |
| rlandy | marios: if you merge job definitions, they don't run anywhere | 14:32 |
| rlandy | marios: you need to add them to check/gate lists | 14:32 |
| rlandy | correct - a layout | 14:32 |
| rlandy | so we can safely merge your definitions | 14:33 |
| quiquell | sshnaidm: yep same as marios Invalid Key, but doing ssh manually to gerrit works fine within the docker container with the injected keys | 14:33 |
| rlandy | and then you can decide where you want them run | 14:33 |
| marios | rlandy: ack ok then yeah lets merge those and i'll followup with a layout change thanks | 14:33 |
| rlandy | marios: you may want to get alex's review on the layout | 14:33 |
| marios | rlandy: ok | 14:33 |
| rlandy | he has pointed out in he past where we added check jobs and not gate jobs | 14:34 |
| rlandy | marios: also - you might want to add them non-voting at first | 14:34 |
| marios | rlandy: right :) same for me when i did the standalone jobs | 14:34 |
| rlandy | marios: ok - merging those changes - you can modify from there | 14:34 |
| marios | rlandy: is the reason i posted that https://review.openstack.org/#/c/631024/3 & looks like http://logs.openstack.org/24/631024/3/check/openstack-tox-docs/2df3a00/html/contributor/ci_primer.html | 14:34 |
| marios | rlandy: (another review for next time you have reviews time :) ) ^ | 14:35 |
| marios | rlandy: thanks for your help | 14:35 |
| *** dsneddon has joined #oooq | 14:35 | |
| rlandy | oh good - someone put that in words | 14:36 |
| rlandy | k - merging your defn changes | 14:36 |
| marios | quiquell: you removed the images? | 14:38 |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario007-multinode-oooq-container, tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates, tripleo-ci-fedora-28-standalone @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario000-multinode-oooq- (1 more message) | 14:38 |
| marios | quiquell: "msg": "Cannot find upstream-infra-fedora-28 at the openstack cloud, you can upload one from\nhttps://nb02.openstack.org/images/ and add your ssh pub key with\nvirt-edit and upload it to your openstack cloud.\n" | 14:38 |
| marios | quiquell: i can only see the centos one in my rdo tenant panda ? | 14:38 |
| marios | quiquell: (am trying to test your change) | 14:39 |
| rlandy | marios: k - your reviews are w+'ed | 14:39 |
| marios | rlandy: thanks | 14:39 |
| * marios shakes fist at quiquell | 14:40 | |
| quiquell | marios: Did they work ? | 14:41 |
| marios | quiquell: no it fails cos the image isn't there anymore the f28 one | 14:41 |
| marios | quiquell: did you delete? | 14:41 |
| rlandy | quiquell: why don't we offer the option of using a stock image with nodepool-setup? | 14:41 |
| quiquell | marios: sshnaidm did it, it's reconstructing it | 14:42 |
| rlandy | save all the dowloading | 14:42 |
| panda | marios: did you accept two images or just one ? | 14:42 |
| quiquell | sshnaidm: send image to marios | 14:42 |
| marios | quiquell: k cos it fails can't find image so didn't get as far as keys | 14:42 |
| marios | panda: two | 14:42 |
| sshnaidm | marios, send me your tenant id | 14:42 |
| rlandy | marios: shoot - your patches are in merge conflict | 14:42 |
| *** dsneddon has quit IRC | 14:43 | |
| rlandy | because they all merged to same file | 14:43 |
| marios | rlandy: no worries will fix later | 14:43 |
| marios | thanks | 14:43 |
| marios | sshnaidm: k sec | 14:43 |
| quiquell | rlandy: Do you want to talk libvirt? I need a break with the ssh key | 14:43 |
| quiquell | driving me crazy | 14:43 |
| rlandy | quiquell:sure - let's give you a break :) | 14:43 |
| rlandy | I'll be quick | 14:43 |
| rlandy | my blue? | 14:44 |
| marios | quiquell: but but community call | 14:44 |
| marios | where we can talk about the key | 14:44 |
| marios | starting now | 14:44 |
| marios | :D | 14:44 |
| marios | (well in a bit) | 14:44 |
| quiquell | I want to close eyes open them and repro working :-/ | 14:45 |
| rlandy | quiquell: let's join the call | 14:45 |
| rlandy | we can talk afterwards | 14:45 |
| quiquell | ack | 14:45 |
| panda | quiquell: \o/ I was able to reproduce the new f28 pipeline, even building containers!! | 14:45 |
| panda | quiquell: that's a miracle! | 14:45 |
| rlandy | marios: I am in the community call alone | 14:46 |
| rlandy | is anyone joining this thing? | 14:46 |
| quiquell | marios: I don't have the event | 14:46 |
| quiquell | panda: \o/ | 14:46 |
| marios | rlandy: oh tripleo didn't finish yet but sec joining quiquell | 14:46 |
| quiquell | panda: I cannot even reproduce my sadness | 14:46 |
| *** saneax has joined #oooq | 14:46 | |
| panda | I'm completely distracted today. | 14:47 |
| rlandy | wait - why did we call the community call if tripleo is not finished yet??? | 14:47 |
| * rlandy is confused | 14:48 | |
| panda | did we ? | 14:48 |
| rlandy | forget it | 14:48 |
| rlandy | quiquell: ok - community call postponed - want to talk libvirt? | 14:51 |
| quiquell | rlandy: ack connecting to your blue | 14:51 |
| marios | ok joining community call now then | 14:56 |
| jaosorior | what's the link? | 14:57 |
| marios | quiquell: still invalid key :/ | 14:58 |
| *** dsneddon has joined #oooq | 15:15 | |
| quiquell | feck feck feck... | 15:18 |
| rlandy | guess missed community call | 15:18 |
| quiquell | marios: Have like one key that works and another (the one we use at CI) that does not :-( | 15:19 |
| *** dsneddon has quit IRC | 15:21 | |
| marios | quiquell: ack gonna do something else for whatever is rest of today will try again tomorrow | 15:23 |
| marios | rlandy: yeah was just me panda jaosorior and we didn't want to disturb you and quiquell so we left it there | 15:24 |
| quiquell | marios: I am droping soon, have to rest my brains | 15:24 |
| marios | quiquell: ack have a good rest | 15:24 |
| quiquell | marios: Like now reproducer is totally not working don't know why :-( | 15:24 |
| quiquell | marios: even CI is broken | 15:24 |
| marios | quiquell: well get there | 15:25 |
| quiquell | yep | 15:25 |
| quiquell | Ok read you tomorrow | 15:25 |
| marios | o/ | 15:25 |
| *** quiquell is now known as quiquell|off | 15:25 | |
| *** holser_ has quit IRC | 15:36 | |
| *** ccamacho has quit IRC | 15:37 | |
| *** holser_ has joined #oooq | 15:37 | |
| *** ccamacho has joined #oooq | 15:37 | |
| *** holser_ is now known as holser|afk | 15:42 | |
| *** holser|afk has quit IRC | 15:42 | |
| *** ykarel is now known as ykarel|away | 15:53 | |
| *** dsneddon has joined #oooq | 15:54 | |
| *** bogdando has quit IRC | 15:56 | |
| *** udesale has quit IRC | 16:02 | |
| *** dsneddon has quit IRC | 16:07 | |
| rlandy | panda: did you hit an error installing python deps in reproducer? | 16:16 |
| panda | rlandy: I'm stuck at FAILED! => {"msg": "The task includes an option with an undefined variable. The error was: 'ansible_user' is undefined\n | 16:21 |
| panda | rlandy: but to my defense I pushed https://review.openstack.org/632498 in the meantime | 16:21 |
| rlandy | panda: k - so your current install is ahead of mine | 16:22 |
| *** trown is now known as trown|lunch | 16:22 | |
| marios | panda rlandy: updated https://review.rdoproject.org/r/#/q/topic:standalone-scenario-promotion if you get a sec thanks (merge conflict from scen1 merge) | 16:22 |
| panda | rlandy: oh, what's your error ? | 16:24 |
| rlandy | panda: I ma just kicking the pre playbook | 16:25 |
| rlandy | it's running python2 | 16:25 |
| rlandy | it should be running python 3 | 16:25 |
| * rlandy changes | 16:25 | |
| panda | rlandy: I think there's a bug somewhere that ansible runs python2 by default and it's hardcoded. I installed my dependencies manually, following the doc | 16:26 |
| rlandy | panda: I see that now | 16:26 |
| rlandy | I was going to define my python | 16:26 |
| panda | rlandy: but apparently the pre playbook already does that, so we may need to update the docs to skip that part | 16:27 |
| rlandy | ansible-playbook /tmp/reproduce-tmp.pLOat/launcher-playbook.yaml -e ansible_python_interpreter="/usr/bin/python3" | 16:27 |
| panda | rlandy: ansible-playbook-3 maybe ? | 16:27 |
| rlandy | panda; I am building the launcher playbook | 16:27 |
| rlandy | defined primary for pre | 16:29 |
| rlandy | but that confuses the rest of the playbook | 16:29 |
| panda | rlandy: confuses ? any log ? | 16:31 |
| panda | marios: gertty hates you | 16:31 |
| panda | marios: searching for the topic does not show anything | 16:31 |
| rlandy | panda: pre is set to run on promoray | 16:32 |
| rlandy | primary | 16:32 |
| rlandy | I defined primary as localhost | 16:33 |
| *** ccamacho has quit IRC | 16:33 | |
| panda | rlandy: and he doesn't like it ? | 16:34 |
| *** dsneddon has joined #oooq | 16:34 | |
| rlandy | panda: well - let me get the latest review ... | 16:34 |
| rlandy | I was just testing the interactions with master | 16:34 |
| rlandy | panda; what review are you working with? | 16:36 |
| panda | rlandy: HEAD | 16:36 |
| rlandy | so master | 16:36 |
| rlandy | then the same | 16:36 |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario007-multinode-oooq-container, tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates, tripleo-ci-fedora-28-standalone @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario000-multinode-oooq- (1 more message) | 16:38 |
| rlandy | failed: [localhost] (item=private) => {"changed": false, "item": "private", "msg": "Network \"private is not found!"} | 16:38 |
| rlandy | panda: ^^ | 16:38 |
| rlandy | there now | 16:39 |
| *** jfrancoa has quit IRC | 16:39 | |
| panda | rlandy: do you have a network called "private" in you rdo-cloud tenant ? | 16:39 |
| rlandy | panda; I don't - was not a requirement before | 16:40 |
| rlandy | not sure why it is required now | 16:40 |
| panda | rlandy: it matches you cloud.yaml I think. But you're way ahead of me now, I had to change ansible_user to ansible_user_id in the playbook code to continue | 16:41 |
| panda | uh, pre playbook completed | 16:41 |
| panda | FAILED! => {"msg": "The task includes an option with an undefined variable. The error was: 'zuul' is undefined | 16:42 |
| rlandy | panda: https://paste.fedoraproject.org/paste/3kK8tkmKCipwDUa-NJPPGA | 16:42 |
| rlandy | that is from a job | 16:42 |
| marios | rlandy: this the right place for the layout? https://review.rdoproject.org/r/18454 | 16:43 |
| sshnaidm | rlandy, it was always a requirement, wasn't mentioned in etherpad maybe | 16:43 |
| marios | panda: see https://review.rdoproject.org/r/#/c/18156 & https://review.rdoproject.org/r/#/c/18093 thanks | 16:43 |
| rlandy | marios:sorry - let me review your previous fixes - sec | 16:43 |
| rlandy | sshnaidm: I managed to run before w/o it | 16:43 |
| marios | rlandy: heh no worries, i'm headed out and is NOT urgent just fyi if you gt time | 16:43 |
| marios | rlandy: thanks | 16:43 |
| sshnaidm | rlandy, not sure how it's possible.. | 16:44 |
| rlandy | sshnaidm: miracle | 16:44 |
| sshnaidm | rlandy, if it was on Hanukka) | 16:44 |
| panda | marios: why 2&3 is separated from 4 ? | 16:45 |
| marios | panda: just posted that way we were doing them individually to start with if you recall (different folks different jobs) | 16:46 |
| panda | marios: aaaah | 16:46 |
| panda | marios: ok | 16:46 |
| panda | marios: does the order of low-meory-usage environment in the list of envs matter ? | 16:47 |
| marios | panda: not in the job definition shouldn't but point to it with comment and i'll check thanks? | 16:47 |
| *** dsneddon has quit IRC | 16:47 | |
| rlandy | marios: you will have the same problem again | 16:47 |
| marios | rlandy: yeah no worries i'll rebase whichever doesn't merge | 16:48 |
| rlandy | marios: ok - I'll merge 2&3 | 16:48 |
| marios | thanks folks /me hometime ttyl | 16:48 |
| marios | have a good day | 16:48 |
| marios | fix everything | 16:48 |
| marios | ! | 16:48 |
| rlandy | ugh - fine - I'll add a private network | 16:49 |
| rlandy | sshnaidm: requirements for private network? | 16:51 |
| rlandy | just added or connected to router> | 16:51 |
| rlandy | subnet_name? | 16:51 |
| rlandy | suggested ip? | 16:51 |
| sshnaidm | rlandy, yeah, network and router | 16:52 |
| marios | rlandy: i created no subnet fwiw | 16:52 |
| marios | rlandy: i mean just created the private network | 16:52 |
| marios | and it got passed that check (assuming is for reproducer) | 16:52 |
| rlandy | marios: command? | 16:52 |
| marios | rlandy: rdo cloud :) web | 16:52 |
| sshnaidm | marios, which IP do you get for your hosts there? | 16:52 |
| panda | marios: booo! | 16:52 |
| sshnaidm | marios, it's cheating :) | 16:53 |
| marios | sshnaidm: didn't get that far still fighting the keys issue with quiquell|off today | 16:53 |
| rlandy | just choosing one | 16:53 |
| sshnaidm | marios, and w/o router and subnet you can't get them | 16:53 |
| rlandy | 172.18.0.0/24? | 16:53 |
| marios | sshnaidm: ack ok thanks will have to change it tomorrow then | 16:53 |
| marios | rlandy: don't listen to me ! | 16:53 |
| marios | :D | 16:53 |
| marios | bai | 16:53 |
| panda | why my local reproducer is expecting the zuul variables to be present ? | 16:54 |
| rlandy | panda;at what stage? | 16:54 |
| panda | rlandy: run.yaml. You're good, your bash script passes it. | 16:54 |
| rlandy | bash? - running playbook only | 16:55 |
| rlandy | don't need run.yaml | 16:55 |
| rlandy | for CI | 16:55 |
| rlandy | ok - now I have a private network, happy??? | 17:00 |
| panda | I think I'll give up for today. I'm trying to use the reproducer as a user, and if it's not in the DOD I'll just stress everyone for nothing. | 17:00 |
| panda | rlandy: did you call it "private" ? | 17:01 |
| rlandy | I did | 17:01 |
| panda | rlandy: then I'm happy. | 17:01 |
| rlandy | starting zuul and friends | 17:01 |
| *** panda is now known as panda|off | 17:02 | |
| sshnaidm | rlandy, do you have shared images in your tenant? | 17:06 |
| rlandy | sshnaidm; no - will get there | 17:07 |
| rlandy | them | 17:07 |
| rlandy | I'm testing the reproducer script in jobs | 17:07 |
| rlandy | which means I will need to add creating private network in launcher playbook | 17:08 |
| rlandy | in production meeting | 17:08 |
| rlandy | will update images in a bit | 17:09 |
| chandankumar | rlandy: panda|off sshnaidm stackviz now working http://logs.openstack.org/00/627500/67/check/tripleo-ci-centos-7-standalone-os-tempest/a076a67/logs/stackviz/#/testrepository.subunit | 17:15 |
| chandankumar | sshnaidm: rlandy now all os_tempest are ready from myside | 17:15 |
| sshnaidm | chandankumar++ | 17:16 |
| hubbot1 | sshnaidm: chandankumar's karma is now 6 | 17:16 |
| *** ccamacho has joined #oooq | 17:16 | |
| panda|off | what are you doing awake ? | 17:16 |
| ssbarnea|bkp2 | in case someone encounteres No module named ssl_match_hostname - you may want to read https://github.com/docker/docker-py/issues/1502#issuecomment-456478142 | 17:16 |
| chandankumar | panda|off: digging one issue on osa side my 3 patches blocked due to that | 17:17 |
| *** dsneddon has joined #oooq | 17:17 | |
| * chandankumar is hiding | 17:20 | |
| *** dtantsur is now known as dtantsur|afk | 17:23 | |
| *** dsneddon has quit IRC | 17:31 | |
| *** trown|lunch is now known as trown | 17:35 | |
| *** ccamacho has quit IRC | 17:51 | |
| *** derekh has quit IRC | 18:00 | |
| *** dsneddon has joined #oooq | 18:00 | |
| *** ssbarnea|rover has joined #oooq | 18:02 | |
| *** dsneddon has quit IRC | 18:04 | |
| *** ssbarnea|bkp2 has quit IRC | 18:04 | |
| *** ykarel|away has quit IRC | 18:09 | |
| *** jpena is now known as jpena|off | 18:22 | |
| *** gkadam has quit IRC | 18:26 | |
| *** dsneddon has joined #oooq | 18:32 | |
| *** kopecmartin is now known as kopecmartin|off | 18:38 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario007-multinode-oooq-container, tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates, tripleo-ci-fedora-28-standalone @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario000-multinode-oooq- (1 more message) | 18:38 |
| *** saneax has quit IRC | 18:38 | |
| *** dsneddon has quit IRC | 18:45 | |
| *** dsneddon has joined #oooq | 18:52 | |
| rlandy | sshnaidm: which images do I need now? | 19:27 |
| rlandy | I have these three: | 19:27 |
| rlandy | openstack-infra-centos-7 Image Active shared No QCOW2 5.70 GB | 19:27 |
| rlandy | openstack-infra-fedora-28 Image Active shared No QCOW2 5.96 GB | 19:27 |
| rlandy | upstream-centos-7-1537943451 Image Active Image from Other Project - Non-Public No RAW | 19:27 |
| *** holser_ has joined #oooq | 19:43 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-fedora-28-standalone @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario007-multinode-oooq-container @ https://review.openstack.org/560445, stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container- (1 more message) | 20:38 |
| rlandy | now zuul wont start | 20:45 |
| *** agopi has quit IRC | 21:47 | |
| *** trown is now known as trown|outtypewww | 22:00 | |
| *** agopi has joined #oooq | 22:23 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-fedora-28-standalone @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario007-multinode-oooq-container @ https://review.openstack.org/560445, stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container- (1 more message) | 22:39 |
| *** holser_ has quit IRC | 23:54 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!