| *** ssbarnea has quit IRC | 00:25 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-fedora-28-standalone @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001 @ https://review.openstack.org/602248, master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-scenario000 (3 more messages) | 00:41 |
|---|---|---|
| *** huynq has joined #oooq | 01:07 | |
| *** agopi|pto is now known as agopi|off | 01:11 | |
| *** dsneddon has quit IRC | 02:34 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-fedora-28-standalone, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053, tripleo-ci-centos-7-multinode-1ctlr-featureset010, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035 @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-ovb- (3 more messages) | 02:41 |
| *** dsneddon has joined #oooq | 02:54 | |
| *** udesale has joined #oooq | 03:56 | |
| *** ykarel|away has joined #oooq | 04:24 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario001-multinode-oooq-container, tripleo-ci-fedora-28-standalone, tripleo-ci-centos-7-multinode-1ctlr-featureset010, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035 @ (3 more messages) | 04:42 |
| *** ykarel|away is now known as ykarel | 04:47 | |
| *** ratailor has joined #oooq | 05:18 | |
| *** ratailor_ has joined #oooq | 05:40 | |
| *** ratailor has quit IRC | 05:43 | |
| *** quiquell|off is now known as quiquell | 06:07 | |
| *** ykarel is now known as ykarel|afk | 06:21 | |
| *** apetrich has quit IRC | 06:27 | |
| *** ykarel|afk has quit IRC | 06:31 | |
| *** ykarel|afk has joined #oooq | 06:31 | |
| *** jfrancoa has joined #oooq | 06:37 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario001-multinode-oooq-container, tripleo-ci-fedora-28-standalone, tripleo-ci-centos-7-multinode-1ctlr-featureset010, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035 @ (3 more messages) | 06:42 |
| quiquell | Good morning | 06:42 |
| *** apetrich has joined #oooq | 06:43 | |
| *** udesale has quit IRC | 06:49 | |
| *** jaganathan has joined #oooq | 07:02 | |
| *** ccamacho has joined #oooq | 07:34 | |
| *** saneax has joined #oooq | 07:40 | |
| *** ratailor__ has joined #oooq | 07:55 | |
| quiquell | marios: How are you doing with reproducer ? | 07:57 |
| *** ratailor_ has quit IRC | 07:57 | |
| marios | quiquell: o | 07:58 |
| marios | o/ | 07:58 |
| quiquell | Need any help review there ? | 07:58 |
| marios | quiquell: sure check the reviews on the cards, sec though i am checking something that rlandy posted last night (main problem we are currently chasing is getting the playbooks) | 07:59 |
| marios | quiquell: so here i am trying to get the zuul variables from the file. it works OK for changes but going down the inheritance path for the playbooks is really messy (we would be reinventing zuul parser) | 08:00 |
| marios | quiquell: so instead rlandy is trying to get them from console here https://review.openstack.org/#/c/614645/ | 08:00 |
| quiquell | marios: console json, isn't it ? | 08:00 |
| quiquell | marios: Looks like we have plenty of stuff there | 08:01 |
| marios | quiquell: sorry, "here i am trying to get the zuul variables from the file." https://review.openstack.org/#/c/613572/ | 08:01 |
| marios | quiquell: and then the other half of it is geting the toci_ scripts rendered and rlandy has reviews check the cards | 08:02 |
| marios | https://tree.taiga.io/project/tripleo-ci-board/task/271?kanban-status=1447278 here quiquell for the scripts rendering | 08:03 |
| marios | thanks | 08:03 |
| quiquell | Next cycle I will look into it | 08:03 |
| quiquell | thanks to you | 08:04 |
| *** amoralej|off is now known as amoralej | 08:07 | |
| *** ykarel|afk is now known as ykarel | 08:11 | |
| *** bogdando has joined #oooq | 08:26 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario001-multinode-oooq-container, tripleo-ci-fedora-28-standalone, tripleo-ci-centos-7-multinode-1ctlr-featureset010, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035 @ (3 more messages) | 08:42 |
| *** ssbarnea has joined #oooq | 09:16 | |
| quiquell | ssbarnea: Good mornig | 09:16 |
| quiquell | ssbarnea: Have coffe already reach your system ? | 09:17 |
| ssbarnea | mo(u)rning! nope, going now for it | 09:18 |
| ssbarnea | we need to talk.... about dnf | 09:18 |
| quiquell | ssbarnea: Yep also found issues with the shyaml stuff uploaded fix | 09:19 |
| quiquell | ssbarnea: and have a review for failfast for package that does not exists a yum | 09:19 |
| quiquell | ssbarnea: looks like yum does not fail if package does not exists in list, | 09:19 |
| quiquell | ssbarnea: let's talk when you are ready and do a roadmap to deliery the f... f28 | 09:20 |
| *** ykarel_ has joined #oooq | 09:20 | |
| *** ykarel has quit IRC | 09:23 | |
| ssbarnea | quiquell https://bluejeans.com/2655417928 | 09:29 |
| *** ykarel_ is now known as ykarel | 09:31 | |
| *** ykarel_ has joined #oooq | 09:34 | |
| *** ykarel_ has quit IRC | 09:35 | |
| *** ykarel has quit IRC | 09:36 | |
| *** dtantsur|afk is now known as dtantsur | 09:48 | |
| ssbarnea | quiquell : yum install python2-urllib3 python-urllib3 | 09:51 |
| *** udesale has joined #oooq | 09:53 | |
| *** matbu|brb has quit IRC | 09:54 | |
| *** ykarel has joined #oooq | 09:55 | |
| chandankumar | panda|off: Hello | 09:57 |
| chandankumar | panda|off: There is no option to move it to done for this card https://tree.taiga.io/project/tripleo-ci-board/issue/306?kanban-status=2027736 | 09:57 |
| chandankumar | the issue is fixed now, I have putted the status to closed is it ok? | 09:58 |
| sshnaidm|off | chandankumar, do you know something about IOError: [Errno 13] Permission denied: '/var/log/tempest/tempest.log' ? https://ci.centos.org/artifacts/rdo/jenkins-tripleo-quickstart-promote-master-current-tripleo-delorean-minimal-423/undercloud/home/stack/tempest.log.gz | 10:02 |
| chandankumar | sshnaidm|off: checking, which FS it uses? | 10:03 |
| chandankumar | oh sorry It is minimal na | 10:03 |
| sshnaidm|off | chandankumar, it's not fs | 10:03 |
| *** panda|off is now known as panda | 10:03 | |
| chandankumar | sshnaidm|off: it is the same issue | 10:04 |
| chandankumar | sshnaidm|off: like fs01 | 10:04 |
| ykarel | sshnaidm|off, kolla logs are still not available:- https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-centos-7-master-containers-build/9ffa1bd/logs/ | 10:04 |
| sshnaidm|off | chandankumar, ok | 10:04 |
| chandankumar | let me update that | 10:04 |
| panda | chandankumar: top right corner, now it says "Closed" | 10:04 |
| quiquell | ssbarnea: https://review.openstack.org/#/c/615072/ | 10:07 |
| *** sshnaidm|off has quit IRC | 10:10 | |
| *** sshnaidm has joined #oooq | 10:14 | |
| *** quiquell is now known as quiquell|brb | 10:17 | |
| *** sshnaidm is now known as sshnaidm|off | 10:24 | |
| ssbarnea | panda: can you please review https://review.openstack.org/#/c/614762/ ? it passed zuul, we need to act. | 10:36 |
| panda | wtf | 10:37 |
| panda | wtf wtf | 10:38 |
| *** agopi|off has quit IRC | 10:39 | |
| *** agopi|off has joined #oooq | 10:39 | |
| panda | WTF | 10:40 |
| *** udesale has quit IRC | 10:40 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario001-multinode-oooq-container, tripleo-ci-fedora-28-standalone, tripleo-ci-centos-7-multinode-1ctlr-featureset010, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035 @ (3 more messages) | 10:42 |
| panda | ssbarnea: -1 | 10:42 |
| panda | ssbarnea: commants incoming | 10:42 |
| ssbarnea | sure, waiting for them to fix them. | 10:42 |
| *** agopi|off has quit IRC | 10:43 | |
| quiquell|brb | ssbarnea: Testing the optinal thing now | 10:44 |
| *** quiquell|brb is now known as quiquell | 10:48 | |
| quiquell | ssbarnea: Damn, now the f28 it's not finding 'requests' after installing python3-requests | 10:50 |
| quiquell | ssbarnea: We are using --system-site-packages... | 10:50 |
| quiquell | ssbarnea: Going to look at this first before adding the fail fast at shyaml | 10:51 |
| *** huynq has quit IRC | 10:55 | |
| panda | ssbarnea: commented | 11:04 |
| panda | ssbarnea: let's work together on this one | 11:12 |
| quiquell | panda: tha optional virtualenv/pip/setuptools ? | 11:12 |
| ssbarnea | panda : give me 5min to read first. | 11:12 |
| panda | quiquell: 614762 | 11:13 |
| quiquell | panda: Yep that's it, did implement something dumbier this morning, didn't know you were doing it | 11:14 |
| quiquell | panda: Let's go with Sorin's change this validate thing is better | 11:15 |
| panda | it needs some adjustme but the idea is sane | 11:15 |
| quiquell | panda Yep someting like ./configure at linux tarballs but for python | 11:16 |
| quiquell | panda: It has to exercise some python import we need (no all for sure) | 11:16 |
| ssbarnea | panda bj? https://bluejeans.com/2655417928 | 11:17 |
| panda | ssbarnea: not while I'm listening to U2 | 11:18 |
| quiquell | panda, ssbarnea: the python3-heat-aget is not correct is intalling python2 binaries :-( that's why is failing now | 11:36 |
| quiquell | Will try to fix | 11:36 |
| quiquell | panda: I have move from done to in progress a card, was missing some pinning at fedora relase file | 11:46 |
| *** trown|outtypewww is now known as trown | 12:25 | |
| rlandy | marios: hey | 12:36 |
| rlandy | marios: worked with panda last night ... | 12:36 |
| ssbarnea | panda https://seashells.io/v/fpEAM48N | 12:37 |
| rlandy | I was not able to rsync the log-output directly from the log root | 12:37 |
| rlandy | panda: ^^ | 12:37 |
| rlandy | not permissable | 12:37 |
| panda | rlandy: what about a simple copy ? | 12:38 |
| rlandy | marios: panda: I can cat the file, save it to ansible var and grep that | 12:38 |
| rlandy | http://logs.openstack.org/74/614874/5/check/tripleo-ci-fedora-28-standalone/c2f1cf8/job-output.txt.gz | 12:38 |
| marios | rlandy: o/ panda | 12:41 |
| marios | rlandy: so it failed for https://review.openstack.org/#/c/614645/5/roles/create-reproducer-script/tasks/main.yml console_log_data | 12:41 |
| marios | rlandy: i updated mine https://review.openstack.org/#/c/613572/9/roles/create-reproducer-script/templates/reproducer-quickstart.sh.j2 but i expect that will fail too | 12:41 |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario010-multinode-oooq-container, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario001-multinode-oooq-container, tripleo-ci-fedora-28-standalone, tripleo-ci-centos-7-scenario004-multinode-oooq-container, tripleo- (3 more messages) | 12:42 |
| marios | rlandy: i am currently looking at this: https://review.openstack.org/#/c/613572/9/roles/collect-logs/tasks/publish.yml though i am wondering if i can just use the console.txt created on line 12 there :) | 12:42 |
| marios | panda: ^ | 12:42 |
| marios | rlandy: i could just grep that if it is the right one :) i hadn't noticed it before | 12:42 |
| rlandy | lookup('env', 'BUILD_URL') != "" | 12:43 |
| *** quiquell is now known as quiquell|lunch | 12:43 | |
| marios | rlandy: ? | 12:44 |
| marios | rlandy: oh it doesn't fire.... | 12:44 |
| marios | ? | 12:44 |
| rlandy | it does where we build images | 12:44 |
| * rlandy thinks that's ovb | 12:45 | |
| rlandy | checking | 12:45 |
| rlandy | we can change that | 12:45 |
| marios | rlandy: well i was wondering as i couldn't find that console.txt but yeah can we use the same logic to retrieve it | 12:45 |
| marios | rlandy: ideally json but well take txt at this point! | 12:45 |
| ssbarnea | panda https://stackoverflow.com/questions/2990414/echo-that-outputs-to-stderr | 12:46 |
| marios | rlandy: maybe not so simple | 12:46 |
| marios | rlandy: i mean when/what sets that BUILD_URL and will it always have the logs we want | 12:47 |
| rlandy | marios: yeah - you can remove the when condition and try it | 12:48 |
| *** ratailor__ has quit IRC | 12:49 | |
| ssbarnea | panda https://seashells.io/v/aNSZdSPe | 12:51 |
| rlandy | marios: wrt https://review.openstack.org/#/c/614837/5/playbooks/tripleo-ci/post.yaml | 12:52 |
| rlandy | I think I set fact here on localhost | 12:52 |
| rlandy | which is why it's not accessible | 12:53 |
| rlandy | will check | 12:53 |
| panda | rlandy: yep | 12:59 |
| panda | rlandy: have you ried a simple copy ? | 12:59 |
| rlandy | panda: can try that | 12:59 |
| panda | for a single file is not a big problem | 12:59 |
| rlandy | but the error wasn't rsync | 12:59 |
| rlandy | it was from zuul | 12:59 |
| panda | oh | 13:00 |
| quiquell|lunch | panda, ssbarnea: the optional packages review is working with f28 http://logs.openstack.org/28/615128/1/check/tripleo-ci-fedora-28-standalone/8f6e4e2/job-output.txt.gz | 13:00 |
| rlandy | Syncing files to outside the working dir is prohibited | 13:00 |
| rlandy | panda; ^^ | 13:00 |
| quiquell|lunch | panda, ssbarnea: fails beyond package installation | 13:00 |
| panda | quiquell|lunch: why are you working on the same thing ? | 13:00 |
| panda | we* | 13:00 |
| *** quiquell|lunch is now known as quiquell | 13:00 | |
| quiquell | panda: I am using it to find issues at fedora28 job | 13:01 |
| quiquell | panda: And also this morning didn't know of the existence of the review | 13:01 |
| quiquell | panda: So I created something similar but we can discard it | 13:01 |
| quiquell | panda: https://tree.taiga.io/project/tripleo-ci-board/task/310?kanban-status=1447275 | 13:02 |
| quiquell | panda: Do you have a minute for blue ? | 13:02 |
| panda | rlandy: I might have a solution: set_fact: console_log in localhost, then copy in undercloud using content: {{ console_log }} | 13:03 |
| panda | rlandy: hoping the copy module is not completely blocked | 13:03 |
| rlandy | panda: you have a patch to try that? | 13:03 |
| *** dsneddon has quit IRC | 13:03 | |
| panda | rlandy: no | 13:03 |
| rlandy | panda: at the moment I am looking at these two reviews - maybe you can decipher ... | 13:04 |
| rlandy | https://review.openstack.org/#/c/614837/5/playbooks/tripleo-ci/post.yaml | 13:04 |
| rlandy | seems to capture the json data ( no failure) | 13:04 |
| rlandy | but | 13:04 |
| panda | but you're setting the fact in localhost | 13:05 |
| rlandy | https://review.openstack.org/#/c/614645/5/roles/create-reproducer-script/tasks/main.yml | 13:05 |
| rlandy | correct | 13:05 |
| rlandy | has the var as undefined | 13:05 |
| panda | did you try to use host_vars[localhost] then ? | 13:05 |
| rlandy | right - I forget the exact syntax | 13:05 |
| rlandy | host_vars[localhost].console_log_output? | 13:06 |
| rlandy | host_vars[localhost].console_log_data | 13:06 |
| rlandy | loaclhost in ''? | 13:07 |
| rlandy | think so | 13:07 |
| panda | rlandy: yes | 13:07 |
| marios | rlandy: panda will this work: | 13:07 |
| panda | marios: no | 13:07 |
| marios | oh | 13:07 |
| panda | :) | 13:07 |
| marios | https://review.openstack.org/#/c/613572/10/roles/collect-logs/tasks/publish.yml | 13:07 |
| marios | https://review.openstack.org/#/c/613572/10/roles/create-reproducer-script/templates/reproducer-quickstart.sh.j2 | 13:07 |
| rlandy | marios: really> you have ci results? | 13:08 |
| marios | rlandy: no i do not i just posted i am asking if it will | 13:09 |
| marios | :) | 13:09 |
| marios | rlandy: i updated my review to try and get the console from there using the existing zuul variables since i use them already for changes | 13:09 |
| rlandy | marios: no idea - only CI tells | 13:09 |
| marios | rlandy: so i can get the log root but | 13:09 |
| rlandy | nice thought that | 13:09 |
| marios | rlandy: i don't know if it is the 'right' place | 13:09 |
| panda | marios: will not work | 13:09 |
| panda | marios: because when you run collect logs, you run it in undercloud, so "localhost" doesn't have log_root | 13:10 |
| marios | panda: can you spare any more than that? or is it like i have to pay you now per word | 13:10 |
| marios | panda: panda ack. this is what i was worried about | 13:10 |
| panda | marios: gimme 50 bucks | 13:10 |
| marios | panda: i mean if it was 'right place' | 13:10 |
| *** ykarel_ has joined #oooq | 13:10 | |
| panda | going to grab some lunch, brb | 13:10 |
| *** panda is now known as panda|lunch | 13:10 | |
| *** ykarel has quit IRC | 13:10 | |
| ssbarnea | panda marios this is the first one that we need to merge: https://review.openstack.org/#/c/614884/ | 13:11 |
| ssbarnea | the big one depends on that. | 13:11 |
| ssbarnea | rlandy ^ | 13:11 |
| rlandy | ssbarnea: why shyaml only for p3? | 13:13 |
| *** ccamacho has quit IRC | 13:14 | |
| *** ccamacho has joined #oooq | 13:14 | |
| quiquell | rlandy: there is no python-shyaml at centos7 so we install it with pip | 13:15 |
| quiquell | rlandy: it's installed later on if not present | 13:15 |
| rlandy | marios: well, here's a low tech solution ... | 13:18 |
| rlandy | http://logs.openstack.org/74/614874/6/check/tripleo-ci-fedora-28-standalone/be9d252/job-output.txt.gz#_2018-11-02_13_03_19_915994 | 13:19 |
| marios | rlandy: cool, i mean whatever works at this point | 13:19 |
| marios | rlandy: checking the review | 13:19 |
| rlandy | marios: it was my worst case scenario try | 13:20 |
| *** dsneddon has joined #oooq | 13:20 | |
| rlandy | - name: cat json file | 13:20 |
| rlandy | shell: "cat {{ zuul.executor.log_root }}/job-output.json | grep playbook" | 13:20 |
| rlandy | register: json_output | 13:20 |
| marios | rlandy: well fine | 13:20 |
| rlandy | that is it | 13:20 |
| marios | rlandy: i mean at that point we can assemble the list and just pass the list | 13:20 |
| rlandy | nothing more | 13:20 |
| weshay | ssbarnea, quiquell are there any jobs upstream for f28 that get very far? | 13:20 |
| marios | rlandy: pass the list into collect-logs | 13:20 |
| weshay | I know quiquell has stuff deploying locally | 13:20 |
| rlandy | marios: yep | 13:20 |
| rlandy | marios: you can clean up the grep | 13:21 |
| rlandy | to return a cleaner list | 13:21 |
| rlandy | some of the solutions may be slicker | 13:21 |
| marios | rlandy: sure i think "*RUN START" case sensitive will do it :) | 13:21 |
| rlandy | waiting on ci for my other review | 13:21 |
| quiquell | weshay: squashed stuff http://logs.openstack.org/00/615100/3/check/tripleo-ci-fedora-28-standalone/a93fbb6/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz | 13:21 |
| rlandy | marios: if you grep the txt file | 13:21 |
| quiquell | weshay: did have to update the hash to get the latest tripleoclient changes | 13:21 |
| rlandy | I grep'ed the json | 13:21 |
| rlandy | ^^ cleaner | 13:22 |
| quiquell | weshay: also some heat agents are running python2 :-( | 13:22 |
| marios | rlandy: oh ok this is json finde | 13:22 |
| ssbarnea | weshay: i seen job running deployment (and failing during) yesterday but was one bundling open CRs. | 13:22 |
| rlandy | marios: the lines below get the json data | 13:22 |
| rlandy | and try work with that | 13:22 |
| quiquell | ssbarnea: I am squasing all our changes to test them together and look for f28 bugs we have to fix | 13:22 |
| rlandy | I get a strange error there | 13:22 |
| marios | rlandy: ok so can i grab this https://review.openstack.org/#/c/614874/6/playbooks/tripleo-ci/run-v3.yaml and put it somewhere else then or you gonna update it? | 13:22 |
| quiquell | I think changin DLRN hash has some not desied effects | 13:23 |
| rlandy | marios: feel free to take it | 13:23 |
| rlandy | it was just a test idea | 13:23 |
| rlandy | and that should be in post | 13:23 |
| rlandy | not runv3 | 13:23 |
| marios | rlandy: right | 13:23 |
| rlandy | was just playing with it there | 13:23 |
| rlandy | panda had te initial idea | 13:24 |
| rlandy | quicker testing | 13:24 |
| rlandy | http://logs.openstack.org/74/614874/6/check/tripleo-ci-fedora-28-standalone/be9d252/job-output.txt.gz#_2018-11-02_13_03_21_151029 | 13:24 |
| rlandy | ^^ I get that error on trying to work with the json directly | 13:24 |
| rlandy | the grep passes | 13:24 |
| ssbarnea | question regarding the cat json: cant we convert it to yaml instead to make it human readable? lie in cat foo.json | json2yaml ? | 13:27 |
| rlandy | marios: that grep picks up some other lines ... should make it more specific grep '"playbook":' | 13:27 |
| rlandy | and get rid of matching | 13:27 |
| rlandy | panda: we got one working solution (not pretty one, though) | 13:27 |
| marios | rlandy: ack | 13:28 |
| rlandy | marios: still testing accessing the localhost data within collect_logs | 13:29 |
| rlandy | https://review.openstack.org/#/c/614645/6/roles/create-reproducer-script/tasks/main.yml | 13:29 |
| rlandy | rather patch 7 | 13:29 |
| marios | rlandy: cool that would be nice | 13:29 |
| quiquell | rlandy, marios: To copy stuff locally ansible has slurp https://docs.ansible.com/ansible/2.6/modules/slurp_module.html | 13:31 |
| quiquell | rlandy, marios: Maybe this one does not give you problems | 13:31 |
| quiquell | I mean in remote | 13:32 |
| rlandy | quiquell: zuul objects :( | 13:32 |
| rlandy | we can try | 13:32 |
| rlandy | maybe it will get by | 13:32 |
| rlandy | but tristan confirmed it | 13:32 |
| quiquell | rlandy: _inheritance ? | 13:32 |
| rlandy | talk to marios about that one :) | 13:33 |
| quiquell | rlandy: It's not at output.json ? | 13:34 |
| weshay | quiquell, ssbarnea http://pastebin.test.redhat.com/665712 | 13:36 |
| weshay | have you seen this? | 13:36 |
| marios | quiquell: inheritance_path ends very messy. we have to parse it then go find the files and parse them to find the jobs and parse those to find the pre | 13:37 |
| marios | quiquell: #cool story bro | 13:37 |
| quiquell | marios: Parsing from the cloned repos at /home/zuul/src pufff | 13:38 |
| quiquell | marios: It's not at outpu.json ? | 13:38 |
| marios | quiquell: yes this is rlandy idea we are trying to make it work not so simple | 13:38 |
| rlandy | quiquell: it is | 13:38 |
| marios | quiquell: but yes we are planning to use console | 13:38 |
| rlandy | quiquell; we have about 7 patches out there | 13:38 |
| rlandy | working on that idea | 13:39 |
| marios | quiquell: for list of playbooks. at the point we are executing this it means we will only get the pres | 13:39 |
| quiquell | rlandy: looks like you cannot copy files out of workspace of job, have tried it before | 13:39 |
| marios | but that is enough | 13:39 |
| rlandy | quiquell: ack - zull says no | 13:39 |
| rlandy | zuul | 13:39 |
| rlandy | quiquell: you have to cat out the data | 13:39 |
| rlandy | or capture the json | 13:39 |
| rlandy | not slick | 13:39 |
| rlandy | bit it works | 13:39 |
| rlandy | but | 13:39 |
| quiquell | weshay: I don't see this issue at the f28 squash | 13:39 |
| quiquell | rlandy: "computer says no" little britain :-) | 13:40 |
| ssbarnea | weshay not yet, this is the first time i seend that error. | 13:40 |
| quiquell | rlandy: raw 'cp' not working either ? | 13:40 |
| rlandy | quiquell: marios: we should get zuul to run on some other social media platforms - protects its data like nothing I've seen | 13:40 |
| *** d0ugal has quit IRC | 13:40 | |
| rlandy | although out little cat solution is a data breach | 13:41 |
| quiquell | rlandy: could be that the output file is still "in progress" that's why does allow to access it | 13:41 |
| quiquell | rlandy: password and stuff there ? | 13:41 |
| ssbarnea | weshay : can you please give some context on pyyaml install failure with py36? i see pip, and I know we do have a pyyaml rpm for it, so my first question would , be where? why we do not have the rpm. | 13:42 |
| rlandy | https://git.zuul-ci.org/cgit/zuul/tree/tests/remote/test_remote_action_modules.py | 13:42 |
| rlandy | quiquell: ^^ | 13:42 |
| rlandy | <tristanC> rlandy_: the zuul.log_root and being able to copy files from executor to test instances is only available from zuul job playbook contexts | 13:43 |
| weshay | ssbarnea, quiquell resolved | 13:44 |
| quiquell | weshay: ack, what it was ? | 13:44 |
| weshay | quiquell, meh.. fucking universe | 13:45 |
| quiquell | weshay: not just galaxy ? | 13:46 |
| marios | rlandy: https://review.openstack.org/#/c/615191/1/playbooks/tripleo-ci/post.yaml used in the updated https://review.openstack.org/#/c/613572/14/roles/collect-logs/tasks/publish.yml | 13:48 |
| *** panda|lunch is now known as panda | 13:48 | |
| marios | rlandy: (also added an echo in the reproducer https://review.openstack.org/#/c/613572/14/roles/create-reproducer-script/templates/reproducer-quickstart.sh.j2 ) | 13:49 |
| marios | rlandy: let's see | 13:49 |
| marios | :/ | 13:49 |
| quiquell | marios, rlandy: Maybe it would be nice to directly pack everything in a tarball with already cloned stuff from /home/zuul/src/ | 13:50 |
| marios | quiquell: right that is the other idea we considered | 13:50 |
| marios | quiquell: instead of reproduer.sh it would be reproducer directory with all the stuff we want in there | 13:50 |
| quiquell | marios: more than directory autoexecutable tarball | 13:51 |
| quiquell | marios: download and run it will unpack stuff | 13:51 |
| marios | quiquell: nice but again it would be better if we can get this simple case to work | 13:51 |
| quiquell | marios: sure sure | 13:51 |
| marios | the biggest problem is the feedback loop | 13:51 |
| marios | quiquell: can't test this locally | 13:52 |
| quiquell | marios: You need a local zuul to do so | 13:52 |
| *** ykarel_ is now known as ykarel | 13:53 | |
| rlandy | quiquell: I'd like to vote on https://review.openstack.org/#/c/614884/ | 13:58 |
| rlandy | quiquell: your comment says it works on f28 | 13:58 |
| quiquell | rlandy: works in the part it should work, f28 is still not passing | 13:59 |
| quiquell | rlandy: I miss adding logs | 13:59 |
| rlandy | quiquell: pls - that's the critical point | 13:59 |
| rlandy | 2018-11-02 09:33:47.105795 | primary | Package python3-PyYAML-3.12-10.fc28.x86_64 is already installed, skipping. | 13:59 |
| rlandy | 2018-11-02 09:33:47.120220 | primary | Error: Unable to find a match | 13:59 |
| marios | panda: the invite doesn't have a bluejeans we using weshay ? | 13:59 |
| marios | rlandy: ^ | 14:00 |
| rlandy | ^^ still see that in the logs | 14:00 |
| panda | marios: rlandy weshay yes, let's use wehay | 14:00 |
| rlandy | joined | 14:00 |
| rlandy | only one there | 14:00 |
| rlandy | oh - there you all are | 14:01 |
| *** vinaykns has joined #oooq | 14:02 | |
| quiquell | rlandy: Damn you are right | 14:02 |
| quiquell | Checked wront review I think | 14:04 |
| quiquell | rlandy: damn wait it depends on the optional thing | 14:10 |
| quiquell | rlandy: dnf.conf issue excluding pip/virtualenv/setuptools | 14:10 |
| *** rfolco is now known as rfolco|off | 14:11 | |
| quiquell | rlandy: added the dependency now, we should see shyaml working at f28 now | 14:13 |
| *** amoralej is now known as amoralej|lunch | 14:27 | |
| *** d0ugal has joined #oooq | 14:27 | |
| *** dsneddon has quit IRC | 14:31 | |
| *** dsneddon has joined #oooq | 14:34 | |
| quiquell | weshay, ssbarnea: I think we are starting to hit legit errors at the f28 squash | 14:34 |
| weshay | k | 14:35 |
| weshay | quiquell, ssbarnea and you guys pinged alex I saw right? | 14:35 |
| weshay | quiquell, I'd like to see the legit failure in ci | 14:35 |
| quiquell | weshay: yep he will fix the issue after update the hash | 14:35 |
| weshay | k | 14:35 |
| quiquell | weshay: so we can pin the hash again | 14:35 |
| quiquell | weshay: need to update hash to update tripleo-client | 14:36 |
| weshay | k | 14:36 |
| *** ccamacho has quit IRC | 14:36 | |
| weshay | quiquell, I see you chatting there, /me following | 14:38 |
| weshay | thanks | 14:38 |
| quiquell | weshay: ack, let's merge stuff to have this working | 14:39 |
| quiquell | weshay: Don't have clear idea how we are going to select hashes :-/ | 14:39 |
| weshay | quiquell, so.. current-tripleo is not sufficient atm for fedora | 14:40 |
| weshay | ? | 14:40 |
| weshay | it's about 3 days old | 14:40 |
| weshay | now | 14:40 |
| quiquell | weshay: Then it's good | 14:40 |
| quiquell | weshay: ack | 14:40 |
| weshay | 5 days old | 14:40 |
| weshay | https://trunk.rdoproject.org/centos7-master/current-tripleo/ | 14:40 |
| weshay | 3ed8ac0e93367a02ad53d9fa93467057724b6621_fd8eb74b | 14:40 |
| * weshay looks at fedora dlrn | 14:40 | |
| quiquell | weshay: The criteria would be centos7 standalone working | 14:41 |
| quiquell | weshay: maybe we can make it report to fedora DLRN too | 14:41 |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario010-multinode-oooq-container, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario001-multinode-oooq-container, tripleo-ci-fedora-28-standalone, tripleo-ci-centos-7-scenario004-multinode-oooq-container, tripleo- (3 more messages) | 14:42 |
| quiquell | have to leave | 14:44 |
| quiquell | weshay and all have a good week | 14:45 |
| *** quiquell is now known as quiquell|off | 14:45 | |
| weshay | thanks you too | 14:45 |
| *** dsneddon has quit IRC | 14:45 | |
| *** ccamacho has joined #oooq | 14:53 | |
| panda | ssbarnea: quiquell|off hit me with review to review | 14:57 |
| panda | oh no, not the compliance training again ... | 15:01 |
| ssbarnea | panda https://review.openstack.org/#/c/614884/ | 15:02 |
| ssbarnea | panda haha, i did it the day it came, someone new we will be needing it soon :D | 15:02 |
| rlandy | panda: lol - just saw that wonder if it's different this year - after this week's events | 15:02 |
| rlandy | quiquell|off: thanks | 15:03 |
| rlandy | panda: can you post the review you showed during meeting? | 15:04 |
| rlandy | the one with hostvars? | 15:04 |
| ssbarnea | anyone stressed about compulsory training, ping me on private, i may help. | 15:05 |
| *** amoralej|lunch is now known as amoralej | 15:05 | |
| panda | rlandy: https://review.openstack.org/614811 | 15:05 |
| arxcruz | weshay: https://review.openstack.org/#/c/615133/ standalone with tempest below 1 hour | 15:08 |
| *** ssbarnea has quit IRC | 15:08 | |
| weshay | arxcruz, k.. we still have a long pole here http://logs.openstack.org/33/615133/2/check/tripleo-ci-centos-7-standalone/9ac46cc/logs/stackviz/#/testrepository.subunit/timeline?test=tempest.api.compute.admin.test_volumes_negative.VolumesAdminNegativeTest.test_update_attached_volume_with_nonexistent_volume_in_body | 15:09 |
| weshay | oops | 15:09 |
| weshay | http://logs.openstack.org/33/615133/2/check/tripleo-ci-centos-7-standalone/9ac46cc/logs/stackviz/#/testrepository.subunit/timeline | 15:09 |
| weshay | arxcruz, any clues re: that? | 15:09 |
| arxcruz | weshay: i think i just realize the problem | 15:10 |
| *** ssbarnea has joined #oooq | 15:10 | |
| arxcruz | weshay: so, all of these tests are part of a particular scenario | 15:10 |
| arxcruz | so it seems all tests inside a scenario, run in one single worker | 15:10 |
| weshay | which? | 15:10 |
| arxcruz | weshay: TestNetworkBasicOps have 10 tests | 15:11 |
| arxcruz | all these tests on these scenarios run in one single worker | 15:11 |
| arxcruz | I don't know if this is a configuration to run in that way or not | 15:12 |
| arxcruz | same with other scenarions | 15:12 |
| arxcruz | weshay: need to dig, perhaps resources for those scenarios are low then it runs in parallel, while api who doesn't waste too much resoures can run in parallel | 15:13 |
| arxcruz | let me check | 15:13 |
| weshay | k.. I think that is the right issue to attack | 15:13 |
| weshay | not what we are running | 15:13 |
| *** chandankumar is now known as chkumar|off | 15:14 | |
| weshay | arxcruz, not only that.. but why not run those first | 15:14 |
| panda | rlandy: damn I have to correct my patch | 15:15 |
| weshay | arxcruz, I see basic ops run, then boot pattern | 15:15 |
| panda | "{{ zuul.executor.log_root|to_json }}" | 15:15 |
| panda | doesn't mean anything | 15:15 |
| panda | I needed to lookup the file | 15:15 |
| rlandy | panda: ack | 15:15 |
| rlandy | panda: like in https://review.openstack.org/#/c/614837/5/playbooks/tripleo-ci/post.yaml | 15:16 |
| rlandy | wait - is it possible to set_var on another node? | 15:16 |
| *** dsneddon has joined #oooq | 15:17 | |
| arxcruz | weshay: i think the scheduler that decides that | 15:17 |
| arxcruz | weshay: I can digg into it if you want, but not sure how long would take, maybe create an issue | 15:18 |
| *** fhubik has joined #oooq | 15:18 | |
| arxcruz | but right now, i don't know what else could I do | 15:18 |
| *** fhubik has left #oooq | 15:18 | |
| panda | rlandy: set_var ? | 15:22 |
| panda | you mean specify varsl in the play for plrimary taking data from localhost ? | 15:22 |
| rlandy | panda: I am sudo coding | 15:22 |
| panda | ... | 15:22 |
| rlandy | panda: something like that | 15:22 |
| rlandy | a girl can dream, right???? | 15:22 |
| weshay | arxcruz, so some kind of a bug in ostestr | 15:23 |
| weshay | arxcruz, are these tests decorated some how to override concurrency | 15:23 |
| arxcruz | weshay: maybe is a bug in ostestr, and no, nothing decorated to override concurrency | 15:24 |
| weshay | arxcruz, meh.. nothing here http://logs.openstack.org/33/615133/2/check/tripleo-ci-centos-7-standalone/9ac46cc/logs/undercloud/home/zuul/tempest/etc/tempest.conf.txt.gz | 15:28 |
| arxcruz | weshay: yeah, i search on tempest config.py | 15:28 |
| *** dsneddon has quit IRC | 15:28 | |
| arxcruz | i don't think it is related to tempest | 15:28 |
| arxcruz | i ping mtreinish hope he can anser | 15:28 |
| weshay | arxcruz, does running w/ debug: true change how it's executed? | 15:29 |
| arxcruz | weshay: no, just the output on tempest.log | 15:29 |
| weshay | omg.. lolz https://arxcruz.net/ | 15:29 |
| weshay | this shit is not working | 15:29 |
| * weshay loves it | 15:29 | |
| arxcruz | weshay: :D | 15:29 |
| rlandy | panda: I am still not understanding https://review.openstack.org/#/c/614811/9/playbooks/tripleo-ci/run-v3.yaml | 15:31 |
| rlandy | that would have to happen in post, right? | 15:32 |
| panda | rlandy: it's the same, I just put the POC in run to get the results faster | 15:33 |
| rlandy | k | 15:33 |
| panda | rlandy: I could even probably skip some other pre-run | 15:33 |
| rlandy | panda: no - this is good | 15:33 |
| rlandy | we want to see the pre playbooks listed | 15:33 |
| rlandy | I can test that same idea in post | 15:33 |
| panda | ssbarnea: where's the -extras big review again | 15:41 |
| panda | ssbarnea: got it | 15:41 |
| *** dsneddon has joined #oooq | 15:45 | |
| ssbarnea | going out for ~40 mins. | 15:51 |
| ssbarnea | btw, if someone can tell if an attempt to convert the job-output.json to a human readably console output (yaml-ish) would be appreciated, please say so (+/-1) | 15:53 |
| weshay | ssbarnea, vs. the txt version? | 15:57 |
| *** jaosorior has quit IRC | 15:58 | |
| panda | ssbarnea: -1, human have the text console, we need the json for the machines | 16:00 |
| *** jfrancoa has quit IRC | 16:01 | |
| *** jaganathan has quit IRC | 16:06 | |
| *** dsneddon has quit IRC | 16:11 | |
| ssbarnea | panda : maybe you don't know what I am referring to, is not about replacing the build-result.json, is about avoiding to cat it on console! on console we need something that is human friendly, brief and without "\n" and lots of double quotes around... Someone posted a link to like in run_v3 that was doing the ugly cat, this is my problem. the json file is good and useful as long nobody is dumping it to my console/log ;) | 16:15 |
| panda | ssbarnea: don't talk about solution until after we've disccused the problem | 16:16 |
| ssbarnea | sure, let me create a bug named "help, i got blinded by the logs" pointing to http://logs.openstack.org/91/610491/26/check/tripleo-ci-fedora-28-standalone/d1f01e3/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz#_2018-10-31_21_01_59 | 16:19 |
| panda | bah 51 minutes to have a syntax error on a missing ) | 16:20 |
| marios | panda: i had something similar earlier https://review.openstack.org/#/c/613572/8/roles/collect-logs/tasks/publish.yml@88 | 16:22 |
| marios | panda: and of course this stuff is happening at the very end of the run | 16:23 |
| *** dsneddon has joined #oooq | 16:24 | |
| panda | if we only had something to reproduce out job ... | 16:24 |
| ssbarnea | marios panda : now we have a bug lets gather info there https://bugs.launchpad.net/tripleo/+bug/1801388 | 16:24 |
| openstack | Launchpad bug 1801388 in tripleo "build output is impossible to read due to nested json dumps" [Undecided,New] | 16:24 |
| panda | ssbarnea: wha ... whe ... who .... | 16:24 |
| panda | ssbarnea: I'm sure it's a duplicate | 16:24 |
| panda | ssbarnea: and it's not our to fix | 16:24 |
| panda | ssbarnea: this is the ansible of the deployment | 16:25 |
| ssbarnea | is directly affecting everyone ability to read logs, including us. i am not saying that we created it, but we need to find a way to avoid it, even if this means chasing someone else to make a fix. | 16:26 |
| panda | ssbarnea: let me summary the situation with you once more: we are surrounded by the fires of the hell itself, and we have plastic water guns. Please choose carefully how you want to use your reserve of water. | 16:27 |
| ssbarnea | good point | 16:27 |
| ssbarnea | i am going to build a dam in my garden, to have enough water. | 16:28 |
| ssbarnea | anyway, the "b'" prefix looks like a py3 specific issue if I rememeber well. | 16:28 |
| *** dsneddon has quit IRC | 16:29 | |
| ssbarnea | panda weshay rlandy : this one is waiting for our reviews https://review.openstack.org/#/c/614762 | 16:32 |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario010-multinode-oooq-container, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario001-multinode-oooq-container, tripleo-ci-fedora-28-standalone, tripleo-ci-centos-7-scenario004-multinode-oooq-container, tripleo- (3 more messages) | 16:42 |
| *** dsneddon has joined #oooq | 16:45 | |
| *** ykarel is now known as ykarel|away | 16:48 | |
| *** dsneddon has quit IRC | 16:53 | |
| *** ykarel_ has joined #oooq | 17:10 | |
| panda | rlandy: it works! http://logs.openstack.org/11/614811/10/check/tripleo-ci-fedora-28-standalone/c9b6150/logs/console_log.json | 17:11 |
| panda | json console is on the undercloud | 17:12 |
| panda | I can go home happy now | 17:12 |
| *** ykarel|away has quit IRC | 17:13 | |
| rlandy | panda++ | 17:16 |
| hubbot1 | rlandy: panda's karma is now 7 | 17:16 |
| marios | panda: niice | 17:19 |
| *** dtantsur is now known as dtantsur|afk | 17:19 | |
| marios | panda: rlandy will have a closer look monday have a good weekend | 17:20 |
| rlandy | marios: ack - have a good weekend | 17:20 |
| *** trown is now known as trown|lunch | 17:21 | |
| *** bogdando has quit IRC | 17:22 | |
| *** ccamacho has quit IRC | 17:27 | |
| *** panda is now known as panda|off | 17:36 | |
| *** dsneddon has joined #oooq | 17:39 | |
| *** vinaykns has quit IRC | 17:41 | |
| ssbarnea | re: panda weshay rlandy : this one is waiting for our reviews https://review.openstack.org/#/c/614762 | 17:46 |
| * weshay looks | 17:46 | |
| weshay | ssbarnea, did that pass ci.centos tests on any of the recent patch sets? | 17:47 |
| weshay | ssbarnea, would be nice to figure out how to not fail on cico errors | 17:49 |
| * weshay weeps to myself | 17:49 | |
| weshay | ssbarnea, do you want help validating it? | 17:50 |
| ssbarnea | weshay i a confident on this one, just want to put it to merge queue. | 17:50 |
| ssbarnea | because it blocks the other two. | 17:50 |
| ssbarnea | panda made the changes, I tested locally, we looked at zuul, seems legit. | 17:51 |
| weshay | ssbarnea, so post the logs of your local run man | 17:51 |
| ssbarnea | weshay does https://seashells.io/v/eVegREBK count? | 17:52 |
| ssbarnea | weshay : look at https://ci.centos.org/job/tripleo-quickstart-gate-master-delorean-quick-basic/7738/consoleFull - search for " install-deps succeeded" | 17:54 |
| *** amoralej is now known as amoralej|off | 17:55 | |
| ssbarnea | i asked centos guis to enable line numbering plugin: they denied: "need to assert the risk"... | 17:55 |
| * weshay hates this | 17:56 | |
| weshay | 13:06:23 + /home/jenkins/workspace/tripleo-quickstart-gate-master-delorean-quick-basic/bin/cico node get --arch x86_64 --release 7 --count 1 --retry-count 6 --retry-interval 60 -f csv | 17:56 |
| weshay | 13:12:24 The requested operation failed as no inventory is available. | 17:56 |
| *** dsneddon has quit IRC | 17:56 | |
| *** vkapalav has joined #oooq | 17:56 | |
| ssbarnea | weshay for jenkins I know how to do something: change config to abandon build on specific failures and make it not vote -1 on abandons. | 17:57 |
| ssbarnea | so when infra fails, you wound not get a vote, just a message. | 17:57 |
| ssbarnea | quite useful as it informs you without forcing you to "investigate". | 17:58 |
| ssbarnea | weshay any idea on how to escalate https://bugs.centos.org/view.php?id=15373 ? | 17:59 |
| weshay | #centos-devel | 18:00 |
| ssbarnea | that is what I did, and this is where I got the explanation about risks on making logs bigger.... | 18:02 |
| ssbarnea | but no comment on bugs which i really hate as nobody is able to trace irc logs | 18:02 |
| ssbarnea | maybe if we get more people to confirm the bug/request they would re-consider it. enabling it takes seconds. | 18:03 |
| ssbarnea | but i also feel for them because I know what jenkins down means, with any config change there is some risk involved. | 18:03 |
| ssbarnea | if i am the only one asking for this feature, is unlikely to happen | 18:04 |
| ssbarnea | this issue with jenkins is why I consider that the CI people need to own it (run the instance), without this the admin will always work against the team, trying to avoid any change. one of the reasons why downstream jenkins is not managed by central-ci team. | 18:06 |
| ssbarnea | i know that we cannot change this in our case | 18:06 |
| ssbarnea | so we can only use community approach to drive changes | 18:07 |
| -openstackstatus- NOTICE: OpenStack infra's mirror nodes stopped accepting connections on ports 8080, 8081, and 8082. We will notify when this is fixed and jobs can be rechecked if they failed to communicate with a mirror on these ports. | 18:08 | |
| ssbarnea | weshay: is it normal for me not to be able to edit or add wiki pages on taiga? is this by design? | 18:11 |
| ssbarnea | i also need a new tag on taiga named "external", for tracking external issues. | 18:17 |
| *** vkapalav has quit IRC | 18:31 | |
| *** vinaykns has joined #oooq | 18:32 | |
| *** trown|lunch is now known as trown | 18:33 | |
| ykarel_ | weshay, can u review https://review.openstack.org/#/c/615123/1, i cannot see similar failures in rocky phase1, not sure if {% if release not in ['newton', 'ocata', 'pike', 'queens'] -%} is correct | 18:37 |
| *** ykarel_ is now known as ykarel | 18:37 | |
| ykarel | can see the issue only in master phase1 minimal job | 18:37 |
| ykarel | i see similar patches are fired for multiple featuresets | 18:38 |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario010-multinode-oooq-container, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-scenario001-multinode-oooq-container, tripleo-ci-fedora-28-standalone, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053, tripleo-ci-centos-7-scenario004-multinode-oooq-container, tripleo- (4 more messages) | 18:42 |
| *** dsneddon has joined #oooq | 18:47 | |
| -openstackstatus- NOTICE: The firewall situation with ports 8080, 8081, and 8082 on mirror nodes has been resolved. You can recheck jobs that have failed to communicate to the mirrors on those ports now. | 18:52 | |
| *** ykarel is now known as ykarel|away | 18:55 | |
| *** ykarel|away has quit IRC | 19:01 | |
| *** dsneddon has quit IRC | 19:09 | |
| *** dsneddon has joined #oooq | 19:13 | |
| *** apetrich has quit IRC | 19:24 | |
| *** apetrich has joined #oooq | 19:38 | |
| rlandy | weshay: ping | 19:59 |
| weshay | shalom | 19:59 |
| rlandy | weshay: upshift stage looking better :) | 19:59 |
| weshay | oh ya | 19:59 |
| weshay | that's cool | 19:59 |
| rlandy | weshay: I couldn't get through the whole test | 19:59 |
| weshay | rlandy, hey... I have some news.. I should run by you | 19:59 |
| rlandy | flavors not big enough | 19:59 |
| rlandy | but IPMI works | 19:59 |
| weshay | nice | 19:59 |
| rlandy | news?? | 20:00 |
| rlandy | more news | 20:00 |
| weshay | ya.. nothing exciting | 20:00 |
| rlandy | I think I'm maxed out this week | 20:00 |
| rlandy | but ok - go4it | 20:00 |
| weshay | but should get your input and thoughts on day #1 | 20:00 |
| rlandy | ok | 20:00 |
| weshay | blue for 5min tops | 20:00 |
| rlandy | k | 20:00 |
| *** saneax has quit IRC | 20:05 | |
| ssbarnea | weshay or rlandy : still have some time time before going home? | 20:09 |
| weshay | ssbarnea, what's up | 20:10 |
| ssbarnea | weshay i want some feedback on https://github.com/ssbarnea/mooolecule -- it helped me find a possible bug in qs.sh | 20:10 |
| ssbarnea | i wonder if I should add a -1 on the change before it merges. | 20:11 |
| weshay | ssbarnea, /me reads | 20:14 |
| weshay | ssbarnea, can you add the project to https://review.rdoproject.org/sf/welcome.html in the near future | 20:15 |
| weshay | ssbarnea, I'll give it whirl too | 20:15 |
| weshay | ssbarnea, I think we would eventually want a ci job that runs this right? | 20:16 |
| ssbarnea | weshay I already started this process, don't worry. but progress is slow as f28 takes prio. There is an already existing CR on tq that adds the new job, but the does is not functional yet. | 20:17 |
| rlandy | ssbarnea: ack - just finishing off upshift testing - will look in a bit | 20:17 |
| weshay | k cool | 20:17 |
| ssbarnea | it was only a temporary convenience measure to put code on github, to isolate it from tq work I have in progress and to test my changes with it. it will be shorted lived. | 20:19 |
| ssbarnea | check https://seashells.io/v/RwjxzzuV | 20:19 |
| ssbarnea | this is why few moments ago I added -1 Worflow on that change, to prevent it from merging. | 20:19 |
| weshay | ssbarnea, this looks really nice | 20:24 |
| weshay | ssbarnea, so it's late in your day, but ya.. I'd like to chat about the future w/ this | 20:25 |
| ssbarnea | once it helps you find bugs in ~20s is hard not to love it | 20:25 |
| ssbarnea | sure | 20:25 |
| weshay | right | 20:28 |
| weshay | ssbarnea++ | 20:28 |
| hubbot1 | weshay: ssbarnea's karma is now 4 | 20:28 |
| ssbarnea | weshay : practical question: on vanilla centos 7 there is no existing pip package available. none. what to do? | 20:32 |
| weshay | ssbarnea, yum install easy-install; easy-install pip | 20:32 |
| weshay | no? | 20:32 |
| ssbarnea | ok,... | 20:32 |
| ssbarnea | openstack repos do have pip as rpm, but not the default repos in centos. | 20:33 |
| ssbarnea | so the right statement is "pip may not be available as an rpm". i could do a search for it, and install it if found, and to fallback to easy_install. | 20:34 |
| ssbarnea | only because we were too tired to publish it | 20:34 |
| weshay | ssbarnea, bluejeans | 20:35 |
| ssbarnea | joined | 20:36 |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario010-multinode-oooq-container, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario001-multinode-oooq-container, tripleo-ci-fedora-28-standalone, tripleo-ci-centos-7-scenario004-multinode-oooq-container, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035 @ (4 more messages) | 20:42 |
| weshay | yum install -y https://trunk.rdoproject.org/centos7-master/build-deps/latest/noarch/python-pip-8.1.2-1.el7.noarch.rpm | 20:45 |
| weshay | ssbarnea, | 20:45 |
| weshay | [delorean-master-build-deps] | 20:49 |
| weshay | name=dlrn-master-build-deps | 20:49 |
| weshay | baseurl=https://trunk.rdoproject.org/centos7-master/build-deps/latest/ | 20:49 |
| weshay | enabled=1 | 20:49 |
| weshay | gpgcheck=0 | 20:49 |
| ssbarnea | weshay: i am inclined to let to merge in its current form to unblock the rest and make another CR to add the improvent, as it concerns only non CI usage. | 21:24 |
| weshay | non-ci? | 21:25 |
| weshay | ssbarnea, sorry where do you think we would hit the issue | 21:25 |
| weshay | not following what you are saying | 21:25 |
| ssbarnea | the problem with missing python-pip rpm is visible only if you try to run qs.sh on a vanilla centos, and on CI we do not ever do this. | 21:26 |
| weshay | ssbarnea, sure we do | 21:26 |
| weshay | that is exactly what the ci and jobs in ci.centos are for.. also the internal jenkins jobs | 21:27 |
| ssbarnea | "vanila" = pure image, ones where we do not do any kind of preparatory work. | 21:27 |
| weshay | ssbarnea, the nodes in ci.centos are vanilla | 21:28 |
| weshay | internally.. qs is installed on fedora 26/28 | 21:29 |
| ssbarnea | weshay i miss something because i am failing to see that error ony builds posting at https://review.openstack.org/#/c/614762/ | 21:30 |
| ssbarnea | i see failures, not unrelated to this known issue. | 21:30 |
| weshay | right.. /me checking https://ci.centos.org/artifacts/rdo/jenkins-tripleo-quickstart-gate-master-delorean-quick-basic-7738/console.txt.gz | 21:31 |
| weshay | again | 21:31 |
| ssbarnea | even the two rdo ci failures, did pass the "install-deps" step and failed later. maybe i miss somethign. | 21:31 |
| weshay | maybe because it's on a slave | 21:31 |
| weshay | in ci.centos | 21:31 |
| weshay | and the deps are already met | 21:31 |
| weshay | ssbarnea, however breaking users is not ok | 21:32 |
| rlandy | ssbarnea: what reviews are left? | 21:33 |
| weshay | ssbarnea, this worked for me though http://chunk.io/weshayutin/72b12fc0efa2414b988a02445423f76e | 21:33 |
| weshay | ssbarnea, ah duh | 21:34 |
| weshay | I'm on f28 | 21:34 |
| weshay | ssbarnea, I could run from one of my centos boxes | 21:35 |
| ssbarnea | weshay updated https://review.openstack.org/#/c/614762/ -- back to zero | 21:39 |
| weshay | ssbarnea, ok.. rlandy please review the latest change and I'll monitor it over the weekend https://review.openstack.org/#/c/614762/8..9/quickstart.sh | 21:39 |
| ssbarnea | thanks, I will ping others tomorrow, after we get the gate passing. | 21:40 |
| weshay | ssbarnea, tomorrow is saturday, sshnaidm|off will be online on sunday | 21:40 |
| ssbarnea | probably I will find others to review it | 21:40 |
| rlandy | weshay: kind if out of time today. will be on line tomorrow night/sunday if still needed | 21:40 |
| *** rlandy has quit IRC | 21:41 | |
| ssbarnea | have a nice weekend! time to grab something to drink | 21:41 |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario010-multinode-oooq-container, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario001-multinode-oooq-container, tripleo-ci-fedora-28-standalone, tripleo-ci-centos-7-scenario004-multinode-oooq-container, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035 @ (4 more messages) | 22:42 |
| *** vinaykns has quit IRC | 23:01 | |
| *** jtomasek has quit IRC | 23:31 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!