| *** jbadiapa has joined #oooq | 00:02 | |
| hubbot | FAILING CHECK JOBS on master: tripleo-quickstart-extras-gate-newton-delorean-full-minimal @ https://review.openstack.org/560445, stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224 | 00:48 |
|---|---|---|
| *** links has joined #oooq | 01:33 | |
| *** vvaldez has joined #oooq | 01:34 | |
| hubbot | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224, master: tripleo-quickstart-extras-gate-newton-delorean-full-minimal @ https://review.openstack.org/560445 | 02:48 |
| *** vinaykns has joined #oooq | 03:00 | |
| *** vvaldez has quit IRC | 03:01 | |
| *** yolanda has quit IRC | 03:12 | |
| *** skramaja has joined #oooq | 03:21 | |
| *** vinaykns has quit IRC | 03:38 | |
| *** ykarel|away has joined #oooq | 03:49 | |
| *** udesale has joined #oooq | 03:52 | |
| *** jaganathan has joined #oooq | 04:17 | |
| *** pgadiya has joined #oooq | 04:40 | |
| *** pgadiya has quit IRC | 04:40 | |
| *** ykarel|away is now known as ykarel | 04:44 | |
| hubbot | FAILING CHECK JOBS on master: tripleo-quickstart-extras-gate-newton-delorean-full-minimal @ https://review.openstack.org/560445, stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224 | 04:48 |
| *** skramaja_ has joined #oooq | 05:10 | |
| *** skramaja has quit IRC | 05:10 | |
| *** ratailor has joined #oooq | 05:18 | |
| *** ccamacho has quit IRC | 05:21 | |
| *** links has quit IRC | 05:32 | |
| *** yolanda has joined #oooq | 05:34 | |
| *** udesale_ has joined #oooq | 05:36 | |
| *** udesale has quit IRC | 05:37 | |
| *** bogdando has joined #oooq | 05:40 | |
| *** quiquell|off is now known as quiquell|rover | 05:53 | |
| quiquell|rover | ykarel: Do you know if someone has change the promoter ? | 05:54 |
| ykarel | quiquell|rover, no | 05:54 |
| *** links has joined #oooq | 05:56 | |
| quiquell|rover | arxcruz: Good morning | 06:00 |
| quiquell|rover | arxcruz, ykarel: Thre is something to fix here ? https://review.openstack.org/#/c/568869/ | 06:01 |
| quiquell|rover | Use healthcheck api to determine swift service | 06:01 |
| ykarel | quiquell|rover, i think the patch is not stable as it's failing consistently, | 06:02 |
| ykarel | i mean in CI | 06:02 |
| quiquell|rover | ykarel: I see, will try to reproduce | 06:02 |
| ykarel | we need to make it work multiple times before considering it merge, otherwise it may lead to merge failures in subsequent patches | 06:02 |
| ykarel | looks like enabling swift tests are causing the issue | 06:03 |
| quiquell|rover | ykarel: So it's a race condition problem ? | 06:03 |
| ykarel | may be performance | 06:03 |
| ykarel | or race, but not sure, | 06:03 |
| quiquell|rover | ykarel: Maybe there are some timemouts than can be adjusted | 06:03 |
| ykarel | yes, but those should be considered before merging the patch | 06:04 |
| ykarel | we need to see what's are the requirement for swift tests | 06:05 |
| ykarel | and consider those in jobs | 06:05 |
| ykarel | for packstack one, we can try: https://review.openstack.org/#/c/577041/ | 06:09 |
| ykarel | this should improve little | 06:10 |
| quiquell|rover | ykarel: we didn't have a master promotion because of this ? | 06:12 |
| ykarel | quiquell|rover, only one job failed yesterday with the temp patch, i havent' looked into detail but it didn't look real issue | 06:13 |
| ykarel | so currently we are blocked because of tempestconf patch | 06:13 |
| quiquell|rover | ykarel: Will look | 06:13 |
| ykarel | okk | 06:23 |
| *** ykarel is now known as ykarel|afk | 06:23 | |
| *** ccamacho has joined #oooq | 06:24 | |
| arxcruz | quiquell|rover: ykarel|afk hello, too morning in the morning for me :) | 06:26 |
| quiquell|rover | arxcruz: We can wait until coffe kicks in at your body (or beer) | 06:28 |
| arxcruz | come on, beer now? are you crazy? | 06:28 |
| arxcruz | my wife would kill me :D | 06:28 |
| arxcruz | ykarel|afk: what's the difference between scenario000 and 002 for example? do you know? before I dig into it | 06:30 |
| *** sanjayu__ has joined #oooq | 06:31 | |
| *** ykarel|afk has quit IRC | 06:33 | |
| *** sanjayu_ has joined #oooq | 06:33 | |
| *** ykarel|afk has joined #oooq | 06:34 | |
| *** matbu has quit IRC | 06:36 | |
| *** sanjayu__ has quit IRC | 06:36 | |
| *** ykarel|afk is now known as ykarel | 06:46 | |
| hubbot | FAILING CHECK JOBS on master: tripleo-quickstart-extras-gate-newton-delorean-full-minimal @ https://review.openstack.org/560445, stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224 | 06:48 |
| ykarel | arxcruz, just the deployment configuration, like scenario 002 has no ceilometer, etc | 06:48 |
| ykarel | some services are disable in scenario002 and service-workers is configured=2 | 06:49 |
| ykarel | arxcruz, but we are seeing failures in devstack too, so need to consider there as well | 06:50 |
| arxcruz | ykarel: i think the problem is in http://logs.openstack.org/69/568869/37/gate/python-tempestconf-tempest-devstack-demo/2212040/job-output.txt.gz#_2018-06-20_23_51_51_665559 | 06:50 |
| arxcruz | ykarel: as you can see, it's generating 10 accounts | 06:50 |
| arxcruz | and the quota is 12 so, if we get a concurrency where two tests are creating accounts... for example | 06:51 |
| arxcruz | not sure if the quota is 12 though | 06:51 |
| ykarel | arxcruz, so good to check quota | 06:52 |
| quiquell|rover | arxcruz: Can we group some test to run sequencially ? | 06:52 |
| quiquell|rover | arxcruz: Or maybe we miss some teardown that cleans the accounts ? | 06:53 |
| arxcruz | ykarel: i set concurrency for 1, should create half of the accounts | 06:53 |
| arxcruz | let's see how it will behave | 06:53 |
| ykarel | arxcruz, i think in packstack it's not configurable yet | 06:54 |
| ykarel | but why we are only seeing in tempestconf job, are accounts not used at other places in devstack swift jobs | 06:55 |
| ykarel | ok let's see how your patch goes | 06:57 |
| *** apetrich has quit IRC | 06:58 | |
| *** apetrich has joined #oooq | 06:58 | |
| *** kopecmartin has joined #oooq | 06:59 | |
| ykarel | arxcruz, isn't it required tempest_conscurrency to be same when running tempest and when creating accounts file | 06:59 |
| *** rfolco_ has joined #oooq | 06:59 | |
| arxcruz | ykarel: no idea | 06:59 |
| ykarel | arxcruz, ack | 07:00 |
| ykarel | ok let's see | 07:00 |
| arxcruz | ykarel: https://github.com/openstack/tempest/blob/master/tempest/cmd/account_generator.py#L88 | 07:00 |
| * ykarel looks | 07:01 | |
| *** rfolco has quit IRC | 07:02 | |
| *** quiquell|rover is now known as quique|rover|afk | 07:03 | |
| ykarel | hmm looks like this will help | 07:04 |
| arxcruz | after this patch, i think the best is disable swift, and add a tripleo job running swift | 07:04 |
| *** gkadam has joined #oooq | 07:05 | |
| *** apetrich has quit IRC | 07:06 | |
| ykarel | arxcruz, but tempestconf used only in tripleo? | 07:06 |
| arxcruz | ykarel: no | 07:06 |
| arxcruz | ykarel: nevermind, just realize swift was always enabled, we just disable the tests | 07:07 |
| *** jtomasek has joined #oooq | 07:07 | |
| ykarel | okk | 07:07 |
| *** skramaja_ has quit IRC | 07:08 | |
| *** skramaja_ has joined #oooq | 07:10 | |
| *** matbu has joined #oooq | 07:10 | |
| *** tesseract has joined #oooq | 07:11 | |
| *** udesale__ has joined #oooq | 07:14 | |
| *** skramaja has joined #oooq | 07:15 | |
| *** udesale_ has quit IRC | 07:16 | |
| *** skramaja_ has quit IRC | 07:17 | |
| *** links has quit IRC | 07:21 | |
| *** links has joined #oooq | 07:21 | |
| ykarel | looks like we need to increase accounts, instead of decreasing | 07:24 |
| *** amoralej|off is now known as amoralej | 07:25 | |
| *** tosky has joined #oooq | 07:33 | |
| *** quique|rover|afk is now known as quiquell|rover | 07:39 | |
| *** sshnaidm|off has quit IRC | 07:40 | |
| *** links has quit IRC | 07:42 | |
| *** links has joined #oooq | 07:55 | |
| ykarel | arxcruz, tosky it failed again | 07:56 |
| ykarel | Details: Insufficient number of users provided. SecGroupIPv6Test have allocated all the credentials for this allocation request | 07:56 |
| tosky | ykarel: I was reading that | 07:57 |
| ykarel | arxcruz, tosky also was there any reason not changing https://github.com/openstack/python-tempestconf/blob/master/etc/default-overrides.conf#L65 in the current patch? | 07:58 |
| ykarel | it should be reseller_admin_role | 07:58 |
| tosky | forgotten? There is another patch floating around to remove default-overrides.conf, so it may have been forgotten | 07:59 |
| ykarel | looks like it's required atleast if other patch is not merged | 08:01 |
| ykarel | so good to consider before merging this | 08:01 |
| *** ykarel is now known as ykarel|lunch | 08:02 | |
| arxcruz | i'm out of options | 08:02 |
| ykarel|lunch | hmm someone from tempest need to look at it | 08:04 |
| ykarel|lunch | me too don't know much there | 08:04 |
| arxcruz | just found this https://bugs.launchpad.net/tempest/+bug/1455561 | 08:05 |
| openstack | Launchpad bug 1455561 in tempest "Tempest leaking account credentials" [Medium,Fix released] - Assigned to Matthew Treinish (treinish) | 08:05 |
| tosky | arxcruz: does it make sense to change that value in default-overrides, in the meantime? | 08:05 |
| arxcruz | latest comment | 08:05 |
| arxcruz | tosky: no, the defaut is being used, and non used config, doesn't matter | 08:05 |
| arxcruz | imho | 08:05 |
| ykarel|lunch | need to change here as well config_tempest/tests/services/test_object_storage.py | 08:07 |
| arxcruz | ykarel|lunch: change what ? | 08:07 |
| ykarel|lunch | tempest.conf contains reseller_admin while tempest is looking for reseller_admin_role, arxcruz isn't this wrong? | 08:07 |
| tosky | could it be related to what kopecmartin mentioned few days ago, not many users in accounts.yaml? | 08:08 |
| ykarel|lunch | tosky, yes currently this is the issue | 08:08 |
| ykarel|lunch | in accounts.yaml there are 4 Member roles, 2 ResellerAdmin, and 4 others, so increasing would help | 08:09 |
| ykarel|lunch | arxcruz, after reducing tempest_conscurrency in PS38 the users reduced to half, so should try by increasing concurrency | 08:10 |
| * ykarel|lunch going for lunch | 08:10 | |
| arxcruz | ykarel|lunch: you mean, up concurrency to 3 for example ? | 08:11 |
| tosky | arxcruz: I don't get why we can't set reseller_admin_role: the code in the patch is indeed checking it's value, and creating the role | 08:13 |
| arxcruz | tosky: we can | 08:13 |
| arxcruz | no problem | 08:13 |
| tosky | if the point is "it won't change anything", well, maybe, but... | 08:14 |
| *** sshnaidm has joined #oooq | 08:18 | |
| arxcruz | tosky: updating concurrency to 3 to see if will pass | 08:20 |
| arxcruz | and changint to reseller_admin_role | 08:20 |
| arxcruz | changing* | 08:22 |
| arxcruz | okay, from what I saw, yes, we must increase concurrency | 08:26 |
| arxcruz | what's happening is, it checks the accounts.yaml file | 08:26 |
| arxcruz | and get one user from the accounts.yaml matching the role | 08:27 |
| arxcruz | and write a file with hash | 08:27 |
| arxcruz | if the hash already exists, means it's being used | 08:27 |
| arxcruz | so it trows the error | 08:27 |
| arxcruz | so increasing the number of users in accounts.yaml it will be able to get another role | 08:28 |
| *** apetrich has joined #oooq | 08:30 | |
| tosky | arxcruz: oh, right, because the generated users depends on the concurrency | 08:31 |
| tosky | but not in the demo case, iirc | 08:32 |
| arxcruz | tosky: the problem is exactly in the demo case, because we use the accounts.yaml | 08:36 |
| *** holser_ has joined #oooq | 08:37 | |
| tosky | arxcruz: so more users, and reduce the concurrency so that it matches that number of users | 08:38 |
| tosky | or simply reduce the concurrency to 1 as you did originally, not just increase it | 08:38 |
| arxcruz | tosky: reduce concurrency means 1 | 08:38 |
| arxcruz | tosky: there are two concurrencies | 08:38 |
| arxcruz | one, from tempest tests to run in parallel | 08:38 |
| arxcruz | and the concurrency in accounts.yaml | 08:38 |
| arxcruz | i reduce the one in accounts.yaml | 08:38 |
| tosky | and I was talking about tempest execution concurrency | 08:39 |
| arxcruz | tosky: reduce to 1 will increase the time of the job | 08:39 |
| tosky | sure, but at least it can show if the problem is fixed by working in that direction (concurrency matching the number of accounts) | 08:40 |
| arxcruz | tosky: gotcha | 08:40 |
| arxcruz | tosky: can we wait this run first? | 08:40 |
| arxcruz | then we can decrease the concurrency only for demo | 08:40 |
| kopecmartin | wait concurrency of tempest run has to be equal to concurrency the account.yaml is generated with, doesn't it? | 08:41 |
| tosky | that's probably the point | 08:42 |
| tosky | oh, this is confusing for me | 08:43 |
| tosky | I'm still convinced that, in case of demo, we manually create accounts.yaml, but it's not true | 08:43 |
| tosky | I'm only convinced that it should be the thing to do, but no, that's not the case | 08:43 |
| tosky | so the demo case is exactly when tempest account-generator is called, which means that increasing the users generated in accounts.yaml, but without changing the users needed to run tempest tests, may fix it | 08:44 |
| tosky | obvious as it can sound, I'm writing it down so that I won't forget it again | 08:45 |
| arxcruz | lol | 08:45 |
| arxcruz | shit, i didn't change the test | 08:47 |
| arxcruz | tox fail, but i'll let it finishes so we can see if it will work | 08:47 |
| tosky | yep | 08:48 |
| hubbot | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224 | 08:48 |
| quiquell|rover | sshnaidm: Last changes from RR Cockpit https://review.rdoproject.org/r/14318 | 08:49 |
| arxcruz | tosky: one demo pass | 08:55 |
| arxcruz | let's see the other one | 08:55 |
| tosky | if it passes, when you change the job to fix the unit test, please also add a BIG comment when the concurrency is set for tempest account-generator | 08:56 |
| tosky | and add yourself as Co-Authored-By in that patch | 08:57 |
| *** panda|off is now known as panda | 09:24 | |
| sshnaidm | quiquell|rover, you need to configure your editor to remove trailing spaces automatically | 09:25 |
| sshnaidm | quiquell|rover, https://review.rdoproject.org/r/#/c/14318/11/ci-scripts/ruck-rover-cockpit/grafana/ruck-rover.dashboard.json | 09:26 |
| *** sshnaidm is now known as sshnaidm|lnch | 09:26 | |
| arxcruz | tosky: it passes | 09:27 |
| arxcruz | \o/ | 09:27 |
| quiquell|rover | sshnaidm|lnch: This is not edited with an editor | 09:27 |
| quiquell|rover | sshnaidm|lnch: generated from grafana | 09:27 |
| *** ykarel|lunch is now known as ykarel | 09:29 | |
| tosky | arxcruz: remember to add yourself as co-author! | 09:29 |
| tosky | I wonder why the tripleo test failed this time in DLRN | 09:29 |
| sshnaidm|lnch | quiquell|rover, ack | 09:30 |
| arxcruz | tosky: what should I put ? | 09:31 |
| arxcruz | name, nick, or email ? | 09:32 |
| ykarel | arxcruz, yes i meant concurrency to be greater than 2 | 09:33 |
| arxcruz | ykarel: yup, now is working :) | 09:33 |
| arxcruz | awesome | 09:33 |
| ykarel | Great | 09:33 |
| arxcruz | now we can finally merge this code | 09:33 |
| arxcruz | ykarel: how long for the next periodic? | 09:33 |
| ykarel | 3.5 hours | 09:33 |
| ykarel | sorry 2.5 hours | 09:34 |
| ykarel | 12:10 UTC | 09:34 |
| arxcruz | ykarel: don't think will be able to get the patch | 09:34 |
| arxcruz | ykarel: may ask admins to kill the run ? | 09:34 |
| arxcruz | since it will be a waste of resources anyway | 09:34 |
| ykarel | arxcruz, yup we can do that | 09:34 |
| ykarel | arxcruz, yes, also they would be resuming zuulv3 migration today | 09:34 |
| ykarel | so we might be affected | 09:35 |
| tosky | arxcruz: Co-Authored-By: Name Surname <email> | 09:37 |
| arxcruz | tosky: done | 09:40 |
| *** zoli is now known as zoli|lunch | 09:40 | |
| *** dtantsur|afk is now known as dtantsur | 09:41 | |
| arxcruz | that's interesting https://arstechnica.com/information-technology/2018/06/backdoored-images-downloaded-5-million-times-finally-removed-from-docker-hub/ | 09:46 |
| arxcruz | imagine one backdoor like that in a cirros image :P | 09:46 |
| tosky | AHaHAHha | 09:48 |
| tosky | ahha | 09:48 |
| tosky | ah! | 09:48 |
| tosky | :/ | 09:48 |
| panda | what was the image ? | 09:57 |
| arxcruz | dunno | 10:30 |
| *** dtrainor has quit IRC | 10:30 | |
| *** dtrainor has joined #oooq | 10:31 | |
| arxcruz | for god sake, fails again | 10:31 |
| arxcruz | well, now a different error | 10:32 |
| arxcruz | tosky: i'll abandon and restore just to kill the run | 10:32 |
| *** apetrich has quit IRC | 10:38 | |
| hubbot | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224 | 10:48 |
| *** zoli|lunch is now known as zoli | 10:57 | |
| panda | wow [Errno 5] [Errno 12] Cannot allocate memory | 11:03 |
| *** quiquell|rover is now known as quique|rover|lch | 11:03 | |
| *** sshnaidm|lnch is now known as sshnaidm | 11:04 | |
| quique|rover|lch | Puff some scenarios jobs are failing now :-( | 11:05 |
| *** udesale__ has quit IRC | 11:20 | |
| *** udesale__ has joined #oooq | 11:21 | |
| *** udesale__ has quit IRC | 11:27 | |
| *** ratailor has quit IRC | 11:30 | |
| arxcruz | okay, now we need to wait the tripleo job | 11:50 |
| arxcruz | tosky: ^ | 11:50 |
| *** quique|rover|lch is now known as quiquell|rover | 11:51 | |
| tosky | arxcruz: oh, interesting move | 11:51 |
| tosky | (abandon and restore to kill the jobs) | 11:51 |
| panda | Error: Package: qpid-dispatch-router-0.8.0-1.el7.x86_64 (@delorean-master-testing)", \n " Requires: libqpid-proton.so.10()(64bit)", | 11:53 |
| panda | quiquell|rover: is this a known problem ? Seeing it just on a test job | 11:53 |
| arxcruz | tosky: learn from EmilienM_PTO :) | 11:53 |
| quiquell|rover | panda: https://bugs.launchpad.net/tripleo/+bug/1778040 | 11:53 |
| openstack | Launchpad bug 1778040 in tripleo "Error at overcloud_prep_containers Package: qpid-dispatch-router-0.8.0-1.el7.x86_64 (@delorean-master-testing)", " Requires: libqpid-proton.so.10()(64bit)" [Critical,New] - Assigned to Quique Llorente (quiquell) | 11:53 |
| panda | quiquell|rover: ok, thanks | 11:54 |
| quiquell|rover | panda: Looks like they are changing the version of qpid in rdo | 11:54 |
| quiquell|rover | number80 | 11:54 |
| panda | quiquell|rover: ok , ok , so my test job is at par with how the others are failing | 11:54 |
| panda | good news | 11:54 |
| quiquell|rover | panda: You are totally right the worng results :-) | 11:55 |
| quiquell|rover | panda: Can you give some love here ? https://review.rdoproject.org/r/#/c/14084/ | 11:56 |
| quiquell|rover | panda: The are unit tests for the promoter | 11:56 |
| *** trown|outtypewww is now known as trown | 12:05 | |
| *** amoralej is now known as amoralej|lunch | 12:10 | |
| *** atoth has joined #oooq | 12:11 | |
| *** apetrich has joined #oooq | 12:12 | |
| *** rlandy has joined #oooq | 12:14 | |
| rlandy | panda: trown: quiquell|rover: requesting review: https://review.openstack.org/#/c/576632/ (reproducer fix for periodic) | 12:18 |
| marios | o/ # Q where is toci_vxlan_networking_multinode.sh.j2 ?? not here https://github.com/openstack/tripleo-quickstart-extras/tree/master/roles/undercloud-setup/templates | 12:22 |
| marios | i am looking at https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/undercloud-setup/tasks/toci_vxlan_networking.yml | 12:23 |
| marios | rlandy: panda trown * please thanks if you have a sec ^ | 12:23 |
| trown | marios: hmm I think toci_vxlan_networking_multinode is false everywhere... some old cold that can probably just get removed | 12:25 |
| marios | trown: i see, so the last two tasks using the toci_vxlan_networking.sh are the ones we use/need (curentlY) | 12:26 |
| quiquell|rover | rlandy: ack | 12:26 |
| marios | trown: thanks | 12:26 |
| trown | marios: yep | 12:26 |
| panda | marios: it was removed in d4c4290355d12ebab2869d0b02d26a9590ee7621 | 12:27 |
| marios | thank you panda | 12:30 |
| quiquell|rover | rlandy: Comment | 12:31 |
| rlandy | looking | 12:31 |
| rlandy | quiquell|rover: I am happy to make a change, question is do you want to accept this review and make the change on top (periodic reproducer is broken now) or make the change in this review go through another round of ci? | 12:35 |
| weshay|ruck | quiquell|rover, thanks for getting this open https://bugs.launchpad.net/tripleo/+bug/1778040 | 12:36 |
| openstack | Launchpad bug 1778040 in tripleo "Error at overcloud_prep_containers Package: qpid-dispatch-router-0.8.0-1.el7.x86_64 (@delorean-master-testing)", " Requires: libqpid-proton.so.10()(64bit)" [Critical,Triaged] - Assigned to Quique Llorente (quiquell) | 12:36 |
| quiquell|rover | weshay|ruck: Will have a review in a moment | 12:36 |
| quiquell|rover | weshay|ruck: We are deactivating opstool wrongly | 12:37 |
| trown | rlandy: quiquell|rover: I think we should go with the change as is since it is a bug fix, where as the suggestion is more of a refactor/optimization | 12:42 |
| trown | quiquell|rover: can always put your refactor on top | 12:42 |
| *** links has quit IRC | 12:43 | |
| quiquell|rover | trown: Ok, ItÂ's also more future proof | 12:43 |
| rlandy | quiquell|rover; I'll put in a patch on top of that one with your suggested refactor | 12:44 |
| trown | quiquell|rover: ya it is a good suggestion, I just prefer merging bug fixes ASAP, and improvements as a secondary priority | 12:44 |
| panda | quiquell|rover: rlandy jinjia does not support the any function | 12:45 |
| weshay|ruck | panda, when you have 5min, I need your assistance | 12:45 |
| quiquell|rover | rlandy, trown: Agree, also panda has the last word on it. | 12:46 |
| weshay|ruck | panda, context is https://review.openstack.org/#/c/576987/ https://review.rdoproject.org/jenkins/job/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset002-master-upload/320/console | 12:46 |
| quiquell|rover | I went to pythonic :-/ | 12:46 |
| quiquell|rover | s/to/too/ | 12:46 |
| rlandy | yeah - panda's suggestion seems reasonable | 12:46 |
| rlandy | I just fixed the obvious bug and moved on | 12:46 |
| panda | rlandy: approved | 12:46 |
| rlandy | thanks - will refactor in new patch | 12:47 |
| panda | weshay|ruck: I have now | 12:47 |
| quiquell|rover | rlandy: Let me know if 'any' works, | 12:47 |
| panda | quiquell|rover: tested it, it doesn't | 12:48 |
| quiquell|rover | panda: ack | 12:48 |
| weshay|ruck | panda, k.. hit my blue | 12:48 |
| panda | quiquell|rover: UndefinedError: 'any' is undefined | 12:48 |
| hubbot | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224 | 12:49 |
| rlandy | myoung|off: when you are back, I want to ff pike ... safe to do so? | 12:50 |
| rlandy | queens and master are green | 12:50 |
| quiquell|rover | rlandy: Talking about phase2, how do we update ansible in the jenkins jobs ? | 12:50 |
| quiquell|rover | rlandy: we may be hitting an ansible issue | 12:51 |
| rlandy | on the slave? | 12:51 |
| rlandy | quiquell|rover: which job and which error? | 12:51 |
| quiquell|rover | rlandy: I think... https://bugs.launchpad.net/tripleo/+bug/1773569 | 12:51 |
| openstack | Launchpad bug 1773569 in tripleo "TQE: ovb-create-stack.yml role overrides are not working as expected" [High,New] - Assigned to Quique Llorente (quiquell) | 12:51 |
| quiquell|rover | rlandy: Have tested locally just the -e part of the ansible execution and it's working fine | 12:52 |
| quiquell|rover | rlandy: Just to discard ansible versioning | 12:52 |
| rlandy | quiquell|rover: so there is probably a version of ansible installed on the slave, you can update that | 12:54 |
| quiquell|rover | rlandy: Loging and update ? | 12:54 |
| rlandy | quiquell|rover: is this still happening? | 12:54 |
| rlandy | I would guess there was a stale workspace or something but I'll send you the slav ekeys | 12:55 |
| quiquell|rover | rlandy: Yep that could be too, but looks like is persistent. | 12:55 |
| rlandy | there are two possible slaves - checking | 12:57 |
| rlandy | quiquell|rover: sending in pvt | 12:57 |
| quiquell|rover | rlandy: there is a execution from yesterday with the same issue | 12:57 |
| rlandy | check both slaves | 13:00 |
| ykarel | tosky, +1 From zuul https://review.openstack.org/#/c/568869/ | 13:00 |
| ykarel | kopecmartin, arxcruz ^^ | 13:01 |
| arxcruz | tosky: kopecmartin please +2+w this patch | 13:02 |
| arxcruz | :) | 13:02 |
| kopecmartin | done | 13:02 |
| *** vvaldez has joined #oooq | 13:04 | |
| panda | weshay|ruck: sorry, rabbitmq implements AMQP, so it's entirely possible it's rabbitmq like you said, but I've seen qpid packages flying too. | 13:06 |
| panda | weshay|ruck: in the package list for the log you pointed me at, I see puppet-rabbitmq, but no rabbitmq package, and I see qpid-proton package instead | 13:07 |
| panda | so maybe they're switching back once again. | 13:08 |
| weshay|ruck | panda, why would it work on the other jobs though? | 13:08 |
| arxcruz | tosky: | 13:08 |
| arxcruz | tosky: please, could you +2 +w ? | 13:08 |
| quiquell|rover | weshay|ruck: Addind some reviews to the bug, but still the dependencies are wrong | 13:08 |
| weshay|ruck | quiquell|rover, which one? | 13:08 |
| rlandy | trown: https://trello.com/c/1gb8UuYM/825-create-the-inventory-so-that-the-multinode-playbook-can-work-properly - not sure if I did the right thing for the review on this card. Pls comment and I'll correct it | 13:09 |
| *** myoung|off is now known as myoung | 13:09 | |
| quiquell|rover | weshay|ruck: qpid-dispatch-router-0.8.0-1.el7.x86_64 depends on libqpid-proton.so.10 | 13:10 |
| weshay|ruck | quiquell|rover, we need to spend a few minutes today working on the ara config for the undercloud | 13:10 |
| quiquell|rover | weshay|ruck: but libqpid-proton.so.11 is installed | 13:10 |
| weshay|ruck | right k | 13:10 |
| quiquell|rover | weshay|ruck: Yep, timeout is getting more present | 13:10 |
| weshay|ruck | quiquell|rover, amoralej|lunch is helping? | 13:10 |
| quiquell|rover | weshay|ruck: yep also number80 at #rdo | 13:11 |
| myoung | rlandy: haven't parsed scrollback/context yet, but it's always 'safe' to rebase stable branches, can rebase them back. If want a preview of what will happen can look at sbtest-* jobs defined. (aside: IMO it would make sense to revisit these to remove jobs that are no longer safe enough to vote there) | 13:11 |
| trown | rlandy: ya I commented in the review | 13:11 |
| quiquell|rover | weshay|ruck: About the bug, we were installing opstools always | 13:11 |
| myoung | rlandy: ...but if we want to pick up TQ changes to config for BM we should just rebase them methinks | 13:12 |
| trown | rlandy: its right in the way I have tested on libvirt... I am going to test with an inventory that looks like CI now that we know what CI will look like | 13:12 |
| rlandy | trown: apologies, I just checked the card and not the review for comments - looking | 13:12 |
| rlandy | trown: ok - agreed - it should look more that the CI inventory if possible | 13:13 |
| rlandy | if that works with your test run | 13:14 |
| weshay|ruck | quiquell|rover, fyi.. /me is trying to fix the master fs002 upload issue | 13:14 |
| weshay|ruck | atm | 13:14 |
| trown | rlandy: yep, testing that now | 13:14 |
| trown | rlandy: but the way you did it is how I was thinking to do it | 13:15 |
| weshay|ruck | panda, lolz it's ovb we can't update zuul :) | 13:15 |
| quiquell|rover | weshay|ruck: ack, let me finish this other | 13:15 |
| rlandy | myoung: ok - I'll ff pike - let's see what shakes out - bm is broken without latest change on pike | 13:15 |
| weshay|ruck | quiquell|rover, aye | 13:15 |
| quiquell|rover | weshay|ruck: I still don't know why opstools breaks ups | 13:16 |
| panda | weshay|ruck: /me facepalm | 13:17 |
| quiquell|rover | weshay|ruck: btw, did you change the promoter yesterday ? | 13:20 |
| quiquell|rover | Need some reviews for a blocker https://review.openstack.org/#/c/577145/ | 13:21 |
| quiquell|rover | myoung: Do you have a brain cycle for me ? | 13:32 |
| sshnaidm | weshay|ruck, what's problem with ovb? | 13:34 |
| weshay|ruck | sshnaidm, atm fs002 is failing in image prep | 13:34 |
| panda | quiquell|rover: FYI some featuresets will remain broken, because they explicitly enable opstools | 13:35 |
| quiquell|rover | panda: Yep, I am already talking with RDO guys, nothing we can do about it. | 13:36 |
| quiquell|rover | panda: But the ones that don't will be fixed | 13:36 |
| quiquell|rover | panda: What is opstools used for ? | 13:37 |
| panda | quiquell|rover: approved. Frankly I have not idea :) | 13:37 |
| panda | quiquell|rover: first mention of opstools is in d389e35ea4835fac1296cf04d552fb5d2b006032, and it served as alternative for packages like collectd | 13:38 |
| myoung | quiquell|rover: i do! but i'm prepping for CI squad scrum in 20...is this about the LP i updated yesterday? | 13:39 |
| sshnaidm | weshay|ruck, hmm.. AMQPLAIN login refused: user 'guest' - invalid credentials | 13:39 |
| quiquell|rover | myoung: Damn the scrum, Do I have to assist ? | 13:39 |
| sshnaidm | weshay|ruck, maybe still leftovers after this big puppet change where were removed rabbitmq configs | 13:41 |
| weshay|ruck | ya.. odd thing is that other jobs in the same hash work | 13:42 |
| * weshay|ruck looking at it w/ bogdan | 13:42 | |
| sshnaidm | when we'll remove this puppet already.. | 13:43 |
| *** amoralej|lunch is now known as amoralej | 13:44 | |
| sshnaidm | and glance has the password: https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset002-master-upload/2d572f1/undercloud/var/log/config-data/glance_api/etc/glance/glance-api.conf.txt.gz | 13:45 |
| myoung | quiquell|rover: if you want to hop into my BJ I'll be there in a minute...context? | 13:46 |
| quiquell|rover | myoung: All the notes are in the bug, want to update ansible in the slaves | 13:46 |
| quiquell|rover | myoung: Has just try manually and default gets overrided correctly | 13:46 |
| quiquell|rover | myoung: that's all, I am trying to update ansible in the slaves now | 13:46 |
| quiquell|rover | weshay|ruck: I am going to start with the ansible.cfg we need to dig into the timout thing | 13:48 |
| weshay|ruck | quiquell|rover, thanks | 13:48 |
| *** vinaykns has joined #oooq | 13:53 | |
| *** ccamacho1 has joined #oooq | 13:54 | |
| *** ccamacho has quit IRC | 13:54 | |
| *** skramaja has quit IRC | 13:54 | |
| quiquell|rover | myoung, rlandy: slaves of https://rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/label/gate_oooq_ovb updates with ansible 2.5.5 | 13:56 |
| rlandy | cool - let's see what shakes out | 13:57 |
| sshnaidm | myoung, I think https://review.openstack.org/#/c/576834/3 is good to go | 13:57 |
| quiquell|rover | rlandy: is safe to click on "REebuild Last" ? | 13:58 |
| rlandy | yeah | 13:58 |
| rlandy | or just click on the job you want to rebuild | 13:58 |
| quiquell|rover | Damn: ansible-playbook 2.4.1.0 | 14:00 |
| quiquell|rover | ok, is the tq requirements.txt | 14:01 |
| myoung | quiquell|rover: this woudl not be the first time a point release of ansible has caused issues :) | 14:01 |
| quiquell|rover | myoung: going to update the requirements.txt | 14:01 |
| myoung | ci squad scrum | 14:01 |
| weshay|ruck | I'm triple booked :( | 14:02 |
| myoung | ack | 14:03 |
| *** quiquell|rover is now known as quiquell|off | 14:09 | |
| marios | trown: can you hear me guessing not? | 14:10 |
| *** ykarel is now known as ykarel|away | 14:12 | |
| marios | panda: what is jjb in https://trello.com/c/THLXPAqw/806-migrate-rdo-sf-jobs-to-zuulv3 please | 14:15 |
| marios | jenkins job builder | 14:16 |
| marios | jolly jiving buffalo? | 14:16 |
| *** links has joined #oooq | 14:17 | |
| panda | marios: jenkins job builder | 14:20 |
| panda | marios: jokers junk bash | 14:20 |
| marios | panda: thanks | 14:20 |
| weshay|ruck | lolz | 14:21 |
| weshay|ruck | jolly jiving buffalo | 14:21 |
| *** links has quit IRC | 14:23 | |
| myoung | marios: https://docs.openstack.org/infra/jenkins-job-builder/ | 14:46 |
| hubbot | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224 | 14:49 |
| marios | myoung: thnks | 14:54 |
| myoung | marios: dated, but might be useful... https://docs.google.com/presentation/d/1SLF4oJrXgZU1xjjXqVNoM6AgUshdLVtvRbCIpwuct4M/edit#slide=id.p | 14:55 |
| myoung | marios: increasingly we don't care about JJB :) but for internal stuff running on internal jenkins (and ci.centos jobs running on jenkins today) it's still relevant. Here's where our internal jobs are defined, if curious and/or want a working example: http://git.app.eng.bos.redhat.com/git/tripleo-environments.git/tree/jenkins/jobs/tripleo-quickstart | 14:57 |
| marios | myoung: ack thanks | 14:57 |
| *** jaganathan has quit IRC | 15:02 | |
| *** apetrich has quit IRC | 15:05 | |
| sshnaidm | panda, rfolco_ which channel? | 15:06 |
| myoung | weshay|ruck: arxcruz tempest standup? or should martin and I do this quickly ? | 15:06 |
| panda | rfolco_: sshnaidm bj/gcerami | 15:06 |
| *** bogdando has quit IRC | 15:07 | |
| *** ratailor has joined #oooq | 15:08 | |
| myoung | arxcruz, weshay|ruck, going once..going twice... | 15:08 |
| arxcruz | myoung: logging | 15:09 |
| arxcruz | sorry | 15:09 |
| rfolco_ | panda, https://softwarefactory.usersys.redhat.com/logs/11/111/7/check/tripleo-ci-centos-7-undercloud-oooq/820b0fd/job-output.txt.gz#_2018-06-21_14_12_03_204437 | 15:12 |
| *** gkadam has quit IRC | 15:17 | |
| rasca | sshnaidm, hey, I was looking at https://review.openstack.org/#/c/558121/ do you think you can wait me to try to run this patch on a baremetal env? Just to be a reasonable +1 | 15:17 |
| sshnaidm | rasca, of course | 15:17 |
| sshnaidm | rasca, please tell me if everything works for you | 15:17 |
| *** jaganathan has joined #oooq | 15:20 | |
| *** jaganathan has quit IRC | 15:23 | |
| *** jaganathan has joined #oooq | 15:23 | |
| rasca | sshnaidm, testing at once, I'll comment the review | 15:27 |
| sshnaidm | rasca, cool, thanks | 15:27 |
| arxcruz | ykarel|away: merged!!! | 15:29 |
| arxcruz | tosky: weshay|ruck ^ | 15:29 |
| arxcruz | now just wait the rpms and master should be good to go | 15:29 |
| tosky | arxcruz: commented elsewhere | 15:30 |
| *** Goneri has joined #oooq | 15:30 | |
| arxcruz | tosky: ? | 15:33 |
| tosky | arxcruz: see #refstack | 15:34 |
| arxcruz | :) | 15:35 |
| ykarel|away | arxcruz, Great, but RDO is not consistent, we need to wait for it to be consistent | 15:38 |
| ykarel|away | https://review.rdoproject.org/r/#/q/topic:rdo-FTBFS+status:open | 15:38 |
| rfolco_ | sshnaidm, which one | 15:39 |
| sshnaidm | rfolco_, I meant to stay there :) | 15:39 |
| panda | lol | 15:40 |
| sshnaidm | rfolco_, ok, jump to mine :) | 15:40 |
| rfolco_ | I reconnected now | 15:40 |
| rfolco_ | lol | 15:40 |
| sshnaidm | rfolco_, https://bluejeans.com/u/sshnaidm | 15:40 |
| *** ykarel|away has quit IRC | 15:47 | |
| weshay|ruck | arxcruz, which patch? | 15:51 |
| arxcruz | weshay|ruck: https://review.openstack.org/#/c/568869/ | 15:52 |
| weshay|ruck | arxcruz, chandankumar kopecmartin tosky nice work, thanks all! | 15:53 |
| weshay|ruck | nice to see all those green jobs and tripleo job | 15:54 |
| weshay|ruck | s | 15:54 |
| tosky | the good thing is that the patch is clean enough and should not need refactoring in the future, hopefully | 15:55 |
| panda | weshay|ruck: you make it sound like the two groups don't intersect | 15:56 |
| panda | green jobs on one side, tripleo jobs on the other | 15:56 |
| weshay|ruck | panda, no.. tripleo jobs don't always intersect w/ non-tripleo projects | 15:56 |
| weshay|ruck | I'm thankful to see the tripleo jobs there | 15:57 |
| weshay|ruck | regardless of status | 15:57 |
| myoung | weshay|ruck: status on the various cards for tempest squad are all in trello | 15:58 |
| weshay|ruck | myoung, thanks | 15:59 |
| rfolco_ | sshnaidm, https://softwarefactory.usersys.redhat.com/r/#/c/111/ | 16:00 |
| myoung | quiquell|off: sorry missed ya. I'm on early tomorrow if need to chat still | 16:03 |
| sshnaidm | rfolco_, and where is this repo where you showed me "blob" file? | 16:08 |
| *** ratailor_ has joined #oooq | 16:08 | |
| *** tesseract has quit IRC | 16:09 | |
| *** sanjayu_ has quit IRC | 16:09 | |
| *** ratailor has quit IRC | 16:12 | |
| *** ratailor_ has quit IRC | 16:12 | |
| *** panda is now known as panda|off | 16:13 | |
| *** d0ugal has quit IRC | 16:23 | |
| *** trown is now known as trown|lunch | 16:27 | |
| *** amoralej is now known as amoralej|off | 16:29 | |
| *** zoli is now known as zoli|gone | 16:32 | |
| *** arxcruz is now known as arxcruz|off | 16:36 | |
| *** ccamacho1 has quit IRC | 16:36 | |
| *** holser_ has quit IRC | 16:43 | |
| hubbot | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224 | 16:49 |
| *** kopecmartin has quit IRC | 16:52 | |
| *** dtantsur is now known as dtantsur|afk | 16:52 | |
| rfolco_ | sshnaidm, https://softwarefactory.usersys.redhat.com/r/gitweb?p=config.git;a=blob;f=playbooks/upstream/pre.yaml;h=380133eaaa594c6705a375c4add57e60c9fe3e92;hb=HEAD | 16:56 |
| rfolco_ | sshnaidm, config repo | 16:56 |
| *** trown|lunch is now known as trown | 17:31 | |
| *** amoralej|off is now known as amoralej | 17:33 | |
| *** apetrich has joined #oooq | 18:16 | |
| *** Goneri has quit IRC | 18:30 | |
| *** jaganathan has quit IRC | 18:32 | |
| *** gkadam has joined #oooq | 18:36 | |
| hubbot | adarazs: An error has occurred and has been logged. Please contact this bot's administrator for more information. | 18:49 |
| sshnaidm | myoung, trown 70 lines in total: https://review.openstack.org/#/c/576834/ (all the rest is removing files) - do we need to split it | 18:54 |
| sshnaidm | ? | 18:54 |
| *** gkadam has quit IRC | 18:55 | |
| trown | sshnaidm: 70 lines seems reasonable | 18:56 |
| sshnaidm | great | 18:56 |
| *** amoralej is now known as amoralej|off | 18:59 | |
| *** holser_ has joined #oooq | 19:03 | |
| *** weshay|ruck is now known as weshay_PTO | 19:12 | |
| *** tcw has joined #oooq | 19:13 | |
| *** tcw has quit IRC | 19:14 | |
| *** tcw has joined #oooq | 19:14 | |
| *** tcw has quit IRC | 19:15 | |
| *** tcw has joined #oooq | 19:20 | |
| rlandy | quiquell|off: hmm ... stack create is no longer working :( | 19:23 |
| *** tcw1 has joined #oooq | 19:35 | |
| *** tcw has quit IRC | 19:35 | |
| *** tcw1 has quit IRC | 19:41 | |
| *** tcw has joined #oooq | 19:42 | |
| *** holser_ has quit IRC | 19:53 | |
| *** holser_ has joined #oooq | 19:54 | |
| *** yolanda_ has joined #oooq | 20:07 | |
| *** yolanda has quit IRC | 20:11 | |
| *** trown is now known as trown|outtypewww | 20:48 | |
| hubbot | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224 | 20:49 |
| *** myoung is now known as myoung|off | 21:30 | |
| *** vvaldez has quit IRC | 21:38 | |
| *** d0ugal_ has joined #oooq | 21:44 | |
| *** holser_ has quit IRC | 21:49 | |
| *** vinaykns has quit IRC | 22:00 | |
| *** holser_ has joined #oooq | 22:18 | |
| hubbot | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224 | 22:49 |
| *** holser_ has quit IRC | 22:58 | |
| *** rlandy has quit IRC | 23:09 | |
| *** tosky has quit IRC | 23:34 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!