| *** yolanda__ has joined #oooq | 00:09 | |
| *** yolanda_ has quit IRC | 00:13 | |
| hubbot | FAILING CHECK JOBS on stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: legacy-tripleo-ci-centos-7-container-to-container-upgrades-master @ https://review.openstack.org/560445, stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224 | 00:59 |
|---|---|---|
| *** yolanda_ has joined #oooq | 01:51 | |
| *** agopi has joined #oooq | 01:54 | |
| *** yolanda__ has quit IRC | 01:54 | |
| *** yolanda__ has joined #oooq | 01:59 | |
| *** yolanda_ has quit IRC | 02:02 | |
| *** dtrainor_ has quit IRC | 02:21 | |
| *** dtrainor_ has joined #oooq | 02:21 | |
| *** udesale has joined #oooq | 02:55 | |
| hubbot | FAILING CHECK JOBS on stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: legacy-tripleo-ci-centos-7-container-to-container-upgrades-master @ https://review.openstack.org/560445, stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224 | 02:59 |
| *** dtrainor_ has quit IRC | 03:28 | |
| *** dtrainor_ has joined #oooq | 03:29 | |
| *** dtrainor_ has quit IRC | 03:29 | |
| *** dtrainor_ has joined #oooq | 03:30 | |
| *** ykarel|away has joined #oooq | 04:18 | |
| *** ykarel|away is now known as ykarel | 04:18 | |
| *** jaganathan has quit IRC | 04:32 | |
| *** jaganathan has joined #oooq | 04:32 | |
| hubbot | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224, stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: legacy-tripleo-ci-centos-7-container-to-container-upgrades-master @ https://review.openstack.org/560445 | 04:59 |
| *** skramaja has joined #oooq | 05:03 | |
| *** dtrainor_ has quit IRC | 05:12 | |
| *** dtrainor_ has joined #oooq | 05:12 | |
| *** dtrainor_ has quit IRC | 05:14 | |
| *** ratailor has joined #oooq | 05:22 | |
| *** quiquell|off is now known as quiquell | 05:35 | |
| chkumar|ruck | %gatestatus | 05:37 |
| hubbot | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224, stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: legacy-tripleo-ci-centos-7-container-to-container-upgrades-master @ https://review.openstack.org/560445 | 05:37 |
| *** pgadiya has joined #oooq | 05:41 | |
| *** pgadiya has quit IRC | 05:41 | |
| quiquell | marios: Are you there ? | 05:46 |
| *** yolanda_ has joined #oooq | 05:48 | |
| quiquell | ykarel: Going to cry https://review.openstack.org/#/c/579426/ | 05:50 |
| quiquell | ykarel: merged after more than one week :-) | 05:50 |
| *** yolanda__ has quit IRC | 05:52 | |
| ykarel | yeah finally merged :) | 05:54 |
| ykarel | and no need tor cry, :) | 05:54 |
| quiquell | good openning for a Friday :-) | 05:54 |
| ykarel | yup | 05:54 |
| * quiquell has a tissue already in the hand | 05:54 | |
| Tengu | :) | 05:55 |
| marios | quiquell: o/ | 05:57 |
| quiquell | marios: Was looking at the toci_gate_test refactoring we need to do | 05:57 |
| quiquell | marios: Maybe we can cleanup the upgrade part already done at emit_releases | 05:57 |
| quiquell | marios: But don't know what to do with the tags | 05:58 |
| marios | quiquell: sure, where you looking /me opens vim | 05:58 |
| quiquell | marios: https://github.com/openstack-infra/tripleo-ci/blob/master/toci_gate_test.sh#L101-L124 | 05:59 |
| quiquell | marios: most of what is here is already done here https://github.com/openstack-infra/tripleo-ci/blob/master/scripts/emit_releases_file/emit_releases_file.py | 05:59 |
| quiquell | marios: What is missing is the TAGS | 06:00 |
| marios | quiquell: hm, right. well it doesn't really belong in emit_release | 06:01 |
| quiquell | marios: Nope, but is emit releases the one that read the upgrade stuff from featureset... | 06:02 |
| quiquell | marios: Maybe we can read the output from emit_releases and put the correct tags | 06:03 |
| marios | quiquell: not sure how easy that would be if you mean, deduce from the releases what tags we need? | 06:04 |
| marios | quiquell: like mixed upgrade means no undercloud-upgrade for example | 06:04 |
| quiquell | marios: undercloud-upgrade, overcloud-upgrade and overcloud-upgrade | 06:05 |
| quiquell | marios: the last is overcloud-update | 06:05 |
| quiquell | marios: Yep, thats already resolved in the emit_releases_script | 06:06 |
| quiquell | marios: Maybe we can have a tags.sh with all the tags that different part of code discover | 06:06 |
| marios | quiquell: thats sounds good though i don't fully follow the proposal/how we'll discover the tags based on the releases file | 06:07 |
| quiquell | marios: Reading the releases file is no good, no. | 06:08 |
| marios | quiquell: am staring at emit_release trying to understand :) | 06:09 |
| quiquell | marios: Maybe we can split it, the emit_releases_file has two parts | 06:09 |
| quiquell | marios: Discover the kind of upgrade/update and the emitting | 06:09 |
| quiquell | marios: Taking out the discovering and have two emitters | 06:10 |
| quiquell | marios: One for releases and other for tags | 06:10 |
| marios | quiquell: yeah or just relase releases alone and make a new one for emit_tags for eg | 06:10 |
| quiquell | marios: Look first the unit tests, is easier to learn what it does | 06:10 |
| marios | quiquell: but still, how are we discovering tags | 06:10 |
| marios | quiquell: oh i see, the tags are literally set based on the featureset | 06:11 |
| quiquell | marios: featureset and toci_jobtype | 06:11 |
| quiquell | marios: the ones related to upgrades/updates are from featureset | 06:12 |
| marios | yea | 06:12 |
| quiquell | marios: The job is duplicated in the bash script and in the python script | 06:12 |
| quiquell | marios: So maybe we can split emit_releases_script.py | 06:13 |
| quiquell | marios: And have a discover_upgrades that get you the kind of upgrades | 06:14 |
| quiquell | marios: and this is the input for emit_releases and emit_upgrade_tags or similar | 06:14 |
| quiquell | marios: Don't know | 06:14 |
| *** jfrancoa has joined #oooq | 06:14 | |
| marios | quiquell: sounds good, what about the cases like singlenode where we don't have upgrades stuff | 06:15 |
| marios | quiquell: i mean it sounds reasonable, we need a mechanism to get the tags if we are removing the one in toci_gate_test | 06:16 |
| quiquell | marios: THis is just coverint the tags related to upgrdaes | 06:16 |
| marios | quiquell: and python is a good choice especially since we want to move there and we have the emit_releases example | 06:16 |
| quiquell | marios: We already have duplicated code | 06:16 |
| quiquell | marios: singlenode, multinode, ovb is another animal | 06:16 |
| marios | quiquell: so yeah maybe put out a wip something to point at for further discussion | 06:16 |
| marios | quiquell: ok makes sense to concentrate on something for now. it might be that the 'discover_upgrades_tags' becomes a more general 'discover_tags' latter | 06:17 |
| quiquell | marios: Do you wannt to try som tmate to do it ? | 06:17 |
| marios | quiquell: sure, but can we do it in like 1 hour or so, didnt even clear inbox yet :) and i need to do some code reviews but that sounds great idea | 06:18 |
| quiquell | marios: I will have to leave my kid at kindergarden soo, let's go back later | 06:18 |
| marios | quiquell: please :) | 06:18 |
| marios | quiquell: ah ok | 06:18 |
| marios | quiquell: well lets do it now then | 06:18 |
| quiquell | marios: I mean we can do it in an hour | 06:18 |
| quiquell | marios: no problem, that's even better. | 06:19 |
| marios | quiquell: ok great talk here in bit | 06:19 |
| marios | thanks | 06:19 |
| quiquell | marios: sure | 06:19 |
| *** sanjayu_ has joined #oooq | 06:37 | |
| *** dmellado has joined #oooq | 06:46 | |
| *** quiquell is now known as quiquell|bbl | 06:58 | |
| hubbot | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224, stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: legacy-tripleo-ci-centos-7-container-to-container-upgrades-master @ https://review.openstack.org/560445 | 06:59 |
| *** sanjayu_ has quit IRC | 07:01 | |
| *** ccamacho has quit IRC | 07:08 | |
| *** ccamacho has joined #oooq | 07:08 | |
| *** saneax has joined #oooq | 07:11 | |
| *** yolanda_ is now known as yolanda | 07:12 | |
| chkumar|ruck | %gatestatus | 07:13 |
| hubbot | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224, stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: legacy-tripleo-ci-centos-7-container-to-container-upgrades-master @ https://review.openstack.org/560445 | 07:13 |
| *** ykarel is now known as ykarel|lunch | 07:26 | |
| chkumar|ruck | quiquell|bbl: hello | 07:29 |
| chkumar|ruck | quiquell|bbl: logs for legacy jobs i mean zuul v3 migrated jobs are not showing in the graphana dashboard | 07:29 |
| chkumar|ruck | quiquell|bbl: please have a look | 07:30 |
| *** tosky has joined #oooq | 07:36 | |
| *** quiquell|bbl is now known as quiquell | 07:39 | |
| quiquell | chkumar|ruck: good morning | 07:39 |
| quiquell | chkumar|ruck: Working fine here, pass me the link to the dashboard | 07:40 |
| quiquell | To the panel I mean | 07:40 |
| *** bogdando has joined #oooq | 07:41 | |
| chkumar|ruck | quiquell: http://38.145.34.131:3000/d/pgdr_WVmk/cockpit?orgId=1&var-launchpad_tags=alert&var-promotion_names=current-tripleo&var-promotion_names=current-tripleo-rdo&var-promotion_names=current-tripleo-rdo-testing&var-releases=master&var-releases=queens&var-releases=pike&var-releases=ocata&var-influxdb_filter=job_name%7C!~%7C%2Fovb%2F | 07:42 |
| chkumar|ruck | quiquell: check Failing check jobs (noop) section | 07:42 |
| chkumar|ruck | quiquell: got that if cloud and region is null then there is no logs | 07:43 |
| chkumar|ruck | ykarel|lunch: msg": "SSH Error: data could not be sent to remote host \"38.145.34.157\". Make sure this host can be reached over ssh", | 07:45 |
| chkumar|ruck | ykarel|lunch: http://logs.rdoproject.org/85/564285/27/openstack-check/legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-pike/42d6b3e/job-output.txt.gz | 07:45 |
| chkumar|ruck | ykarel|lunch: is it a known issue? | 07:45 |
| chkumar|ruck | ykarel|lunch: it is from stable pike noop jobs | 07:45 |
| quiquell | chkumar|ruck: Looks like it's a POST_FAILURE | 07:46 |
| quiquell | https://softwarefactory-project.io/zuul/api/tenant/rdoproject.org/builds?change=564285&job_name=legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset022-pike | 07:46 |
| quiquell | "log_url": "finger://ze03.softwarefactory-project.io:7979/fbf72bd29bb4458cae19cbc02e9c5ce7" | 07:46 |
| *** amoralej|off is now known as amoralej | 07:46 | |
| chkumar|ruck | quiquell: thanks, I will wait for next run then | 07:51 |
| *** gkadam has joined #oooq | 07:52 | |
| quiquell | chkumar|ruck: 200~https://softwarefactory-project.io/zuul/api/tenant/rdoproject.org/builds?change=564285&patchset=27&job_name=legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset022-pike | 07:52 |
| quiquell | chkumar|ruck: btw going to change it to show the real failure | 07:53 |
| *** yolanda_ has joined #oooq | 07:54 | |
| *** yolanda has quit IRC | 07:56 | |
| *** florianf has joined #oooq | 07:58 | |
| chkumar|ruck | quiquell: you mean the real failure will be shown in next run? | 07:59 |
| quiquell | POST_FAILURE for example | 08:00 |
| quiquell | chkumar|ruck: check it out now | 08:02 |
| chkumar|ruck | quiquell: yup it passed | 08:03 |
| *** holser_ has joined #oooq | 08:03 | |
| *** ykarel|lunch is now known as ykarel | 08:16 | |
| chkumar|ruck | quiquell: I am not sure it is a issue http://38.145.34.131:3000/d/pgdr_WVmk/cockpit?orgId=1&from=now%2Fd&to=now%2Fd | 08:18 |
| chkumar|ruck | under skipped pike promotions 2018-07-06 00:03:17 we are seeing the jobs which are runned on 5th July | 08:19 |
| quiquell | chkumar|ruck: We can go back on time with the uper right filter | 08:20 |
| quiquell | chkumar|ruck: I only see 00:07:50 the last of 6 th of july | 08:23 |
| quiquell | chkumar|ruck: It was at the beginning of the day | 08:23 |
| quiquell | Maybe the filter works different for you at different time zone ? | 08:23 |
| chkumar|ruck | quiquell: yup might be | 08:24 |
| quiquell | I don't see the entry, so looks like depends on the time zone | 08:24 |
| quiquell | It show you stuff at happend at your geo place | 08:25 |
| quiquell | That's good and bad | 08:25 |
| quiquell | Sure we can make it work at UTZ | 08:25 |
| *** ratailor has quit IRC | 08:28 | |
| *** ratailor has joined #oooq | 08:29 | |
| jfrancoa | hello! may I throw a question? | 08:34 |
| jfrancoa | I've been trying to deploy pike containerized with tripleo-quickstart, and I realized that by default we deploy pike in baremetal when using oooq https://github.com/openstack/tripleo-quickstart/blob/master/config/general_config/minimal.yml#L17 | 08:35 |
| jfrancoa | why is that? I though that from pike, the containerized deployment was the de-facto | 08:37 |
| chkumar|ruck | quiquell: weshay All pike promotions jobs are getting affected by this bug https://bugs.launchpad.net/tripleo/+bug/1780403 | 08:38 |
| openstack | Launchpad bug 1780403 in tripleo "[pike promotions] overcloud deploy is failing at Ensure system is NTP time synced step" [Critical,Triaged] | 08:38 |
| chkumar|ruck | quiquell: Do i need to add alert for the same? | 08:38 |
| chkumar|ruck | or wait till weshay comes? | 08:38 |
| quiquell | chkumar|ruck: There is already an alert for pike promotions | 08:39 |
| quiquell | jfrancoa: It's not from queens ? | 08:46 |
| *** dtantsur|afk is now known as dtantsur | 08:57 | |
| hubbot | FAILING CHECK JOBS on stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-quickstart-extras-gate-newton-delorean-full-minimal, legacy-tripleo-ci-centos-7-container-to-container-upgrades-master @ https://review.openstack.org/560445, stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ (1 more message) | 08:59 |
| *** Goneri has joined #oooq | 09:17 | |
| *** panda|off is now known as panda | 09:40 | |
| panda | lucasagomes: ping | 09:51 |
| chkumar|ruck | panda: which jobs runs fs21? | 09:52 |
| panda | chkumar|ruck: gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset021-* in rdo sf | 09:53 |
| *** dtantsur is now known as dtantsur|brb | 09:55 | |
| chkumar|ruck | panda: thanks :-) | 10:01 |
| *** matbu has quit IRC | 10:02 | |
| *** gkadam_ has joined #oooq | 10:33 | |
| *** gkadam has quit IRC | 10:37 | |
| quiquell | panda: It's safe to run ci-config/infra-setup/provision-all.yml at my tenant ? | 10:51 |
| quiquell | just commenting the server I don't want ? | 10:51 |
| panda | quiquell: it's safe. What are you going to test ? | 10:52 |
| quiquell | panda: Going to move the new ruck rover monitoring tool there | 10:52 |
| quiquell | panda: I want to test it first at my tenant | 10:52 |
| panda | quiquell: ah, ok | 10:53 |
| quiquell | panda: btw, I am trying to test zuul's encryption mechanism | 10:54 |
| quiquell | panda: Could be nice to the reproducer job | 10:55 |
| quiquell | https://review.rdoproject.org/r/14677 | 10:55 |
| quiquell | panda: But cannot be tested without merge it :-/ | 10:55 |
| hubbot | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224, stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-quickstart-extras-gate-newton-delorean-full-minimal, legacy-tripleo-ci-centos-7-container-to-container-upgrades-master @ (1 more message) | 10:59 |
| chkumar|ruck | ykarel: we have started hitting timededout on rdo ci jobs also http://logs.rdoproject.org/24/567224/50/openstack-check/legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-queens/bd84fcf/job-output.txt.gz | 10:59 |
| chkumar|ruck | at same step | 10:59 |
| ykarel | chkumar|ruck, what's taking most time | 11:00 |
| chkumar|ruck | ykarel: 2018-07-06 09:12:01.690747 | tripleo-ovb-centos-7 | overcloud-prep-containers : Prepare for the containerized deployment -- 466.25s | 11:05 |
| chkumar|ruck | 2018-07-06 09:12:01.690880 | tripleo-ovb-centos-7 | overcloud-prep-images : Prepare the overcloud images for deploy ------- 349.46s | 11:05 |
| chkumar|ruck | undercloud-deploy : Install the undercloud --------------------------- 1045.49s | 11:05 |
| ykarel | chkumar|ruck, u need to compare with the passing ones | 11:05 |
| ykarel | and found the difference | 11:06 |
| *** yolanda_ is now known as yolanda | 11:12 | |
| chkumar|ruck | quiquell: can we track this one also /legacy-periodic-tripleo-centos-7-master-containers-build | 11:22 |
| quiquell | chkumar|ruck: It doesn't appear ? | 11:22 |
| quiquell | chkumar|ruck: We are tracking all the jobs | 11:22 |
| quiquell | chkumar|ruck: for tripleo projects, we are not filtering by jobs | 11:23 |
| chkumar|ruck | quiquell: nope | 11:23 |
| chkumar|ruck | quiquell: sorry i can see in last jobs | 11:24 |
| quiquell | chkumar|ruck: It's ? | 11:24 |
| *** ratailor has quit IRC | 11:25 | |
| quiquell | chkumar|ruck: Has filter by it and it appears | 11:25 |
| Tengu | (12 | 11:25 |
| Tengu | woops | 11:25 |
| panda | (13 | 11:29 |
| panda | I win | 11:29 |
| Tengu | :] | 11:29 |
| chkumar|ruck | quiquell: panda https://bugs.launchpad.net/tripleo/+bug/1780427 | 11:35 |
| openstack | Launchpad bug 1780427 in tripleo "[periodic][master] legacy-periodic-tripleo-centos-7-master-containers-build /etc/nodepool/primary_node_private: No such file or directory" [Critical,Triaged] | 11:35 |
| chkumar|ruck | please have a look on this one | 11:35 |
| *** matbu has joined #oooq | 11:35 | |
| panda | mmhh, legacy ? | 11:36 |
| panda | oh, yeah, because of the zuulv3 migration | 11:36 |
| panda | chkumar|ruck: so /etc/nodepool is a directory taht is not used anymore in zuulv3 | 11:37 |
| chkumar|ruck | panda: we need to fix that then | 11:37 |
| quiquell | panda: We get that info from inventory ? | 11:37 |
| panda | chkumar|ruck: unless it is migrated correclty, we need a legacy playbook to still create it | 11:37 |
| chkumar|ruck | All the legacy migrated periodic jobs are RED | 11:38 |
| panda | quiquell: we COULD but we have to change the workflow first, so until we fix that part, we need to still provide that directory | 11:38 |
| panda | chkumar|ruck: that was expected | 11:38 |
| chkumar|ruck | panda: above one is blocking promotion | 11:39 |
| quiquell | chkumar|ruck, panda: We don't have a role already to create that legacy stuff ? | 11:40 |
| *** quiquell is now known as quiquell|lunch | 11:43 | |
| panda | quiquell|lunch: we have a playbook copied from default zuul definitions | 11:45 |
| chkumar|ruck | panda: only master and queens got affected rest is ok currently | 11:45 |
| panda | chkumar|ruck: have you identified the legacy code that is run in that job ? | 11:46 |
| chkumar|ruck | from https://review.rdoproject.org/zuul3/builds.html | 11:46 |
| chkumar|ruck | panda: Are you talking this one https://github.com/rdo-infra/rdo-jobs/blob/master/playbooks/legacy/periodic-tripleo-centos-7-master-containers-build/run.yaml | 11:47 |
| panda | chkumar|ruck: yes, more or less | 11:49 |
| panda | chkumar|ruck: I see some additional tasks to add some /etc/nodepool files\ | 11:49 |
| panda | chkumar|ruck: you'll need to add this somewhere https://github.com/openstack-infra/tripleo-ci/blob/master/playbooks/openstack-zuul-jobs/legacy/pre.yaml | 11:51 |
| panda | chkumar|ruck: this is what creates the file that is missing | 11:51 |
| panda | chkumar|ruck: you may add the above playbook as pre-run in the job conficuration here https://github.com/rdo-infra/rdo-jobs/blob/master/zuul.d/jobs.yaml | 11:52 |
| panda | chkumar|ruck: how long until you EOD ? going to lunch now | 11:52 |
| chkumar|ruck | panda: 2 more hours | 11:53 |
| chkumar|ruck | panda: preparing the patch | 11:53 |
| panda | chkumar|ruck: ok | 11:53 |
| *** panda is now known as panda|lunch | 11:53 | |
| *** saneax has quit IRC | 11:53 | |
| chkumar|ruck | panda|lunch: i think it is already there in https://github.com/rdo-infra/rdo-jobs/blob/master/playbooks/legacy/pre.yaml | 11:59 |
| *** saneax has joined #oooq | 12:05 | |
| panda|lunch | chkumar|ruck: mmhh, is is called/added somewhere in the job ? | 12:07 |
| panda|lunch | ah eyah, I see it | 12:08 |
| *** quiquell|lunch is now known as quiquell | 12:08 | |
| panda|lunch | log time | 12:08 |
| weshay | panda|lunch, you have some time? | 12:13 |
| chkumar|ruck | weshay: Hey | 12:13 |
| weshay | hey chkumar|ruck | 12:13 |
| chkumar|ruck | weshay: we need some attension on these two bugs https://bugs.launchpad.net/tripleo/+bug/1780403 and https://bugs.launchpad.net/tripleo/+bug/1780427 | 12:13 |
| openstack | Launchpad bug 1780403 in tripleo "[pike promotions] overcloud deploy is failing at Ensure system is NTP time synced step" [Critical,Triaged] | 12:13 |
| openstack | Launchpad bug 1780427 in tripleo "[periodic][master][queens] legacy-periodic-tripleo-centos-7-master-containers-build /etc/nodepool/primary_node_private: No such file or directory" [Critical,Triaged] | 12:13 |
| chkumar|ruck | panda|lunch: is looking into nodepool one | 12:14 |
| *** holser_ has quit IRC | 12:14 | |
| chkumar|ruck | weshay: https://review.rdoproject.org/etherpad/p/ruckrover-sprint16 check line: 75 to 114 | 12:14 |
| weshay | chkumar|ruck, k k | 12:15 |
| weshay | chkumar|ruck, k.. thank you | 12:15 |
| *** panda|lunch is now known as panda | 12:19 | |
| weshay | panda, k.. blue | 12:20 |
| weshay | quiquell, I will be late | 12:20 |
| panda | weshay: I'm there | 12:21 |
| *** holser_ has joined #oooq | 12:22 | |
| *** rlandy has joined #oooq | 12:37 | |
| *** udesale has quit IRC | 12:37 | |
| *** udesale has joined #oooq | 12:38 | |
| quiquell | panda, weshay: https://review.openstack.org/#/c/576417/ merged | 12:39 |
| quiquell | we can start to merge the n -> n + 1 stuff | 12:39 |
| rlandy | panda: hello - fif you manage to reach the tmate and env I left? | 12:39 |
| quiquell | weshay: ack, ping me when you are ready | 12:39 |
| rlandy | did | 12:39 |
| arxcruz | weshay: chkumar|ruck will we have meeting? | 12:46 |
| chkumar|ruck | arxcruz: nope, i think | 12:46 |
| chkumar|ruck | martin is not available | 12:47 |
| *** tcw has quit IRC | 12:52 | |
| chkumar|ruck | brb | 12:54 |
| *** yolanda_ has joined #oooq | 12:54 | |
| *** yolanda has quit IRC | 12:57 | |
| hubbot | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224, stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-quickstart-extras-gate-newton-delorean-full-minimal, legacy-tripleo-ci-centos-7-container-to-container-upgrades-master @ (1 more message) | 12:59 |
| *** amoralej is now known as amoralej|lunch | 13:02 | |
| *** skramaja has quit IRC | 13:06 | |
| panda | rlandy: yes, so , I asked lucasagomes to do some debug, but he's not sure if he has time today, and he's off next week, so if he realizes he can't look at it, we can point this environment to dalvarez or ajo | 13:07 |
| rlandy | ok | 13:07 |
| lucasagomes | rlandy, panda finishing something up here first before I leave for next week but I will try my best to take a look at it today | 13:09 |
| lucasagomes | if I can't finish it today, dalvarez, u think u have some time to investigate the problem with that tempest failure in ooo ^ ? | 13:10 |
| rlandy | lucasagomes: panda: thanks ... I was looking at one other possible setup issue ... I may rebuild the env - if I do, I will send a new tmate session | 13:10 |
| panda | rlandy: oooh what other setup issue ? | 13:11 |
| lucasagomes | rlandy, ack | 13:11 |
| dalvarez | lucasagomes: i can try, ping ajo/me on monday panda | 13:11 |
| rlandy | panda: https://github.com/openstack-infra/zuul-jobs/blob/master/roles/multi-node-bridge/tasks/common.yaml#L54 | 13:11 |
| rlandy | ^^ think this is false | 13:11 |
| rlandy | dalvarez: thank you | 13:12 |
| ajo | dalvarez: panda ack | 13:12 |
| panda | rlandy: aaaaaaah the firewall | 13:12 |
| rlandy | panda: checking if we ran that step or not - sec | 13:12 |
| panda | rlandy: I was circling back on that on wednesday, but I was blind to that variable | 13:13 |
| rlandy | this is debugging with a sledgehammer | 13:13 |
| rlandy | line by line | 13:13 |
| lucasagomes | ouch | 13:13 |
| rlandy | panda:idk yet - looking if we executed that | 13:13 |
| panda | I just said "firewall is fin, this is just allowing traffic" | 13:13 |
| *** tcw has joined #oooq | 13:13 | |
| lucasagomes | rlandy, panda FWIW, we run that test in the networking-ovn' upstream gate against the same version of OVS/OVN u are using and it works... so it likely be something in the env indeed | 13:13 |
| panda | rlandy: but if it's the firewall, how are we able to connect to the instance in scenario008 then ? | 13:14 |
| rlandy | lucasagomes: yeah - we think it is a set up issue as the old tests work just fine | 13:14 |
| panda | the mistery thickens | 13:14 |
| rlandy | the issue is to find what we missed | 13:14 |
| rlandy | don;t know yet - just checking our execution steps | 13:16 |
| weshay | quiquell, 3min | 13:29 |
| *** dtantsur|brb is now known as dtantsur | 13:29 | |
| quiquell | weshay: Connecting | 13:32 |
| quiquell | weshay: We can skip the 1-1 if you are busy | 13:34 |
| weshay | quiquell, always busy | 13:35 |
| panda | "that's my secret captain, I'm always busy" | 13:36 |
| *** Goneri has quit IRC | 13:39 | |
| *** ykarel is now known as ykarel|away | 13:51 | |
| weshay | panda, heh.. ready to chat again when you want to | 13:54 |
| weshay | actually 3min break for me | 13:54 |
| *** agopi has quit IRC | 13:56 | |
| weshay | ok | 14:00 |
| panda | weshay: waiting for you | 14:01 |
| *** amoralej|lunch is now known as amoralej | 14:04 | |
| rlandy | panda: I'm testing with bridge_authorize_internal_traffic set to true. There are a few other changes we need to make to the zuul-jobs role. Like the role changes the mtu but does not right that change out to ifcfg-* file. So the value will revert if we reboot | 14:19 |
| *** agopi has joined #oooq | 14:34 | |
| *** ykarel|away has quit IRC | 14:34 | |
| *** jfrancoa has quit IRC | 14:40 | |
| *** jfrancoa has joined #oooq | 14:42 | |
| rlandy | hrybacki: hello there - | 14:44 |
| rlandy | you own https://beaker.engineering.redhat.com/view/rdo-ci-fx2-02-s3.v101.rdoci.lab.eng.rdu.redhat.com#essentials | 14:44 |
| rlandy | can you switch that owner to marios? | 14:45 |
| marios | o/ thanks rlandy hrybacki 'mandreou' | 14:45 |
| *** quiquell is now known as quiquell|off | 14:47 | |
| hubbot | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224, stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-quickstart-extras-gate-newton-delorean-full-minimal, legacy-tripleo-ci-centos-7-container-to-container-upgrades-master @ (1 more message) | 14:59 |
| *** yolanda__ has joined #oooq | 15:00 | |
| *** yolanda_ has quit IRC | 15:03 | |
| *** ykarel|away has joined #oooq | 15:14 | |
| *** ykarel|away is now known as ykarel | 15:17 | |
| rfolco | weshay, I am in your room. The door was open so I did not knock | 15:30 |
| weshay | rfolco, ah thanks | 15:30 |
| weshay | rfolco, /me grabs some coffee and be right there | 15:30 |
| *** saneax has quit IRC | 15:36 | |
| *** saneax has joined #oooq | 15:37 | |
| *** bogdando has quit IRC | 15:39 | |
| weshay | rfolco, https://docs.google.com/forms/d/e/1FAIpQLScwGY5RDNkxxcXB1nEQ5hBtyA94pISvNi084X5Q7YxX7V5Ocg/viewform | 15:42 |
| rlandy | ugh - no luck on the tempest failure | 15:48 |
| marios | rlandy: can you check https://review.openstack.org/#/c/579587/ when you get a sec please maybe this way is better (and see comments in reply to yours thanks) | 15:54 |
| rlandy | will do | 15:55 |
| *** trown is now known as trown|lunch | 15:55 | |
| marios | weshay: can you please check https://review.openstack.org/#/c/578081/ when you get a chance anyone else please reviews appreciated | 15:55 |
| weshay | panda, hey | 15:58 |
| weshay | panda, can you join my blue? | 15:58 |
| weshay | w/ rfolco | 15:58 |
| *** saneax has quit IRC | 15:58 | |
| panda | weshay: coming | 16:01 |
| *** lucasagomes is now known as lucas-afk | 16:01 | |
| *** gkadam_ has quit IRC | 16:05 | |
| arxcruz | weshay: panda https://review.openstack.org/#/c/580384/ before the weekend pretty please? :D | 16:05 |
| weshay | panda, rlandy huh https://review.openstack.org/#/c/579653/ | 16:07 |
| weshay | check out scen007 / 008 | 16:07 |
| *** ykarel has quit IRC | 16:07 | |
| panda | rlandy: hold on | 16:08 |
| arxcruz | lucas-afk: hey, let me know when you're around :) | 16:08 |
| rlandy | lol | 16:09 |
| *** jfrancoa has quit IRC | 16:09 | |
| rlandy | what happened??? | 16:09 |
| rlandy | weshay: did you do something??? | 16:10 |
| * rlandy checks if I can reproduce that | 16:10 | |
| weshay | rlandy, panda just took scen007 / 008 out of the new method | 16:10 |
| rlandy | ok right | 16:11 |
| rlandy | I thought the problem was resolved :( | 16:11 |
| rlandy | vxlan networking | 16:12 |
| rlandy | works like a charm | 16:12 |
| *** ykarel has joined #oooq | 16:13 | |
| panda | rlandy: any luck with the firewall ? | 16:13 |
| rlandy | panda: no show | 16:13 |
| rlandy | same error | 16:13 |
| panda | ok, we'll ping dalvarez monday | 16:13 |
| rlandy | TASK [multi-node-bridge : Authorize the multi-node-bridge network] *********************************************************************************************** | 16:14 |
| rlandy | changed: [primary] | 16:14 |
| rlandy | changed: [secondary] | 16:14 |
| rlandy | ^^ not skipped this time | 16:14 |
| rlandy | panda: really I was just pulling at straws | 16:14 |
| *** tcw1 has joined #oooq | 16:22 | |
| *** tcw has quit IRC | 16:22 | |
| weshay | rfolco, https://review.openstack.org/#/c/579854/ | 16:29 |
| *** panda is now known as panda|off | 16:36 | |
| *** dtantsur is now known as dtantsur|afk | 16:37 | |
| weshay | rfolco, https://review.openstack.org/#/c/579154/ | 16:39 |
| weshay | https://review.openstack.org/#/c/579161/ | 16:39 |
| *** yolanda_ has joined #oooq | 16:46 | |
| *** yolanda__ has quit IRC | 16:49 | |
| *** ykarel_ has joined #oooq | 16:49 | |
| *** ykarel has quit IRC | 16:53 | |
| hubbot | FAILING CHECK JOBS on stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-quickstart-extras-gate-newton-delorean-full-minimal, legacy-tripleo-ci-centos-7-container-to-container-upgrades-master @ https://review.openstack.org/560445, stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ (1 more message) | 16:59 |
| *** amoralej is now known as amoralej|off | 17:04 | |
| *** ykarel_ is now known as ykarel|away | 17:06 | |
| *** udesale has quit IRC | 17:08 | |
| rlandy | arxcruz: how can I prevent tempest from deleting the instance after it runs? | 17:10 |
| arxcruz | rlandy: automatically? there's no way :/ | 17:10 |
| rlandy | ;( | 17:10 |
| arxcruz | rlandy: you can either add a breakpoint and not run the cleanup | 17:11 |
| rlandy | arxcruz: I am on the undercloud | 17:11 |
| arxcruz | rlandy: you want me to go through with you ? | 17:12 |
| rlandy | arxcruz: the instance is not reachable | 17:12 |
| rlandy | I want to see why | 17:12 |
| rlandy | arxcruz: see tmate session | 17:13 |
| arxcruz | rlandy: ok | 17:13 |
| *** florianf has quit IRC | 17:22 | |
| *** holser_ has quit IRC | 17:22 | |
| *** yolanda__ has joined #oooq | 17:29 | |
| *** yolanda_ has quit IRC | 17:32 | |
| *** ykarel|away has quit IRC | 17:39 | |
| *** trown|lunch is now known as trown | 17:41 | |
| *** yolanda_ has joined #oooq | 18:07 | |
| *** yolanda__ has quit IRC | 18:10 | |
| *** ccamacho has quit IRC | 18:17 | |
| hubbot | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224, stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-quickstart-extras-gate-newton-delorean-full-minimal, legacy-tripleo-ci-centos-7-container-to-container-upgrades-master @ (1 more message) | 18:59 |
| *** matbu has quit IRC | 19:35 | |
| *** matbu has joined #oooq | 19:37 | |
| *** matbu has quit IRC | 19:42 | |
| *** matbu has joined #oooq | 19:43 | |
| *** yolanda__ has joined #oooq | 19:56 | |
| *** yolanda_ has quit IRC | 19:59 | |
| *** yolanda_ has joined #oooq | 20:22 | |
| *** yolanda__ has quit IRC | 20:26 | |
| *** yolanda__ has joined #oooq | 20:31 | |
| *** yolanda_ has quit IRC | 20:34 | |
| *** Goneri has joined #oooq | 20:35 | |
| *** jaganathan has quit IRC | 20:47 | |
| hubbot | FAILING CHECK JOBS on stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-quickstart-extras-gate-newton-delorean-full-minimal, legacy-tripleo-ci-centos-7-container-to-container-upgrades-master @ https://review.openstack.org/560445, stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ (1 more message) | 20:59 |
| *** trown is now known as trown|outtypewww | 21:15 | |
| *** agopi has quit IRC | 21:41 | |
| rlandy | weshay: hello - still around? | 21:43 |
| rlandy | zuul-jobs is using openvswitch from ocata ... | 21:44 |
| rlandy | https://github.com/openstack-infra/zuul-jobs/blob/master/roles/multi-node-bridge/vars/RedHat.yaml#L4 | 21:44 |
| rlandy | where do I get the latest rpm | 21:44 |
| weshay | rlandy, hey | 21:44 |
| weshay | rlandy, oh man | 21:45 |
| rlandy | yeah | 21:45 |
| rlandy | ovs 2.6 as opposed to 2.9 | 21:45 |
| rlandy | I am still debugging stupid 007 08 failure :( | 21:45 |
| weshay | https://repos.fedorapeople.org/repos/openstack/openstack-queens/ | 21:45 |
| weshay | there may be another place | 21:46 |
| weshay | via centos | 21:46 |
| * weshay looks | 21:46 | |
| rlandy | I found that | 21:46 |
| rlandy | but really we need to be using the one being tested | 21:46 |
| rlandy | or at least latest | 21:46 |
| rlandy | when running master we will use ovs from queens | 21:47 |
| rlandy | which is better than ocata, I grant you | 21:47 |
| rlandy | https://trunk.rdoproject.org/centos7-master/current-passed-ci/ we don't build release rpms | 21:47 |
| weshay | rlandy, current-passed-ci is == current-tripleo-rdo | 21:50 |
| rlandy | yeah where is the release rpm I don;t see | 21:50 |
| weshay | rlandy, let me see what you are doing | 21:50 |
| weshay | why do you need a release file | 21:50 |
| weshay | release rpm | 21:50 |
| weshay | oh I see | 21:51 |
| rlandy | https://github.com/openstack-infra/zuul-jobs/blob/master/roles/multi-node-bridge/vars/RedHat.yaml#L4 | 21:51 |
| rlandy | ^^ what do I overwrite that with? | 21:51 |
| weshay | rlandy, I think we can try to change https://github.com/openstack-infra/zuul-jobs/blob/master/roles/multi-node-bridge/tasks/common.yaml#L8-L19 | 21:51 |
| weshay | rlandy, just to be sure.. | 21:52 |
| weshay | what is the package name | 21:52 |
| rlandy | so the thing is ... | 21:52 |
| *** yolanda_ has joined #oooq | 21:53 | |
| rlandy | here we are trying to set up the bridge before we get on the undercloud | 21:53 |
| weshay | rlandy, I would perfer us use https://trunk.rdoproject.org/centos7-master/deps/latest/x86_64/ | 21:53 |
| weshay | https://trunk.rdoproject.org/centos7-master/deps/latest/ | 21:54 |
| rlandy | well, iiuc, this should change per what we are testing | 21:54 |
| weshay | rlandy, the notion of release rpms.. didn't really catch on | 21:54 |
| weshay | rlandy, we should just use the deps repos | 21:54 |
| weshay | for each release | 21:54 |
| weshay | that's what it is there for | 21:54 |
| rlandy | yeah - will need to adjust that role then | 21:54 |
| weshay | yup | 21:54 |
| weshay | rlandy, and the packaging team should test it when they update it | 21:55 |
| rlandy | ok - I'll mess with it more on my local reproducer | 21:55 |
| rlandy | but this is a problem | 21:55 |
| weshay | rock on | 21:55 |
| weshay | rlandy, we should be able to update that role | 21:55 |
| rlandy | because we are not testing ovs from the right place | 21:55 |
| weshay | especially for our packaging | 21:55 |
| rlandy | ie:if it changes | 21:55 |
| weshay | ya | 21:55 |
| rlandy | the bridge is set up before we get there | 21:55 |
| weshay | \0/ upstream packagin | 21:55 |
| weshay | g | 21:55 |
| rlandy | ugh - need to go - will report more results on monday | 21:56 |
| *** yolanda__ has quit IRC | 21:56 | |
| rlandy | but yeah - that's a possible cause of our bridge troubles | 21:56 |
| weshay | rlandy, k | 21:56 |
| weshay | everything seems to be openvswitch problems lately | 21:57 |
| rlandy | trying again with quuens one - at least that will be closer | 22:01 |
| *** Goneri has quit IRC | 22:33 | |
| *** holser_ has joined #oooq | 22:39 | |
| *** holser_ has quit IRC | 22:55 | |
| *** saneax has joined #oooq | 22:55 | |
| hubbot | FAILING CHECK JOBS on stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-quickstart-extras-gate-newton-delorean-full-minimal, legacy-tripleo-ci-centos-7-container-to-container-upgrades-master @ https://review.openstack.org/560445, stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ (1 more message) | 22:59 |
| *** yolanda__ has joined #oooq | 23:04 | |
| *** yolanda_ has quit IRC | 23:06 | |
| *** yolanda has joined #oooq | 23:07 | |
| *** yolanda__ has quit IRC | 23:09 | |
| rlandy | weshay: finally ... | 23:15 |
| rlandy | 2018-07-06 23:13:06 | ====== | 23:15 |
| rlandy | 2018-07-06 23:13:06 | Totals | 23:15 |
| rlandy | 2018-07-06 23:13:06 | ====== | 23:15 |
| rlandy | 2018-07-06 23:13:06 | Ran: 1 tests in 83.0000 sec. | 23:15 |
| rlandy | 2018-07-06 23:13:06 | - Passed: 1 | 23:15 |
| rlandy | 2018-07-06 23:13:06 | - Skipped: 0 | 23:15 |
| rlandy | 2018-07-06 23:13:06 | - Expected Fail: 0 | 23:15 |
| rlandy | 2018-07-06 23:13:06 | - Unexpected Success: 0 | 23:15 |
| rlandy | 2018-07-06 23:13:06 | - Failed: 0 | 23:15 |
| rlandy | 2018-07-06 23:13:06 | Sum of execute time for each test: 74.2932 se | 23:15 |
| rlandy | tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_network_basic_ops [74.293208s] ... ok | 23:15 |
| rlandy | panda|off: ^^ got it now | 23:15 |
| *** rlandy has quit IRC | 23:16 | |
| *** yolanda_ has joined #oooq | 23:43 | |
| *** yolanda has quit IRC | 23:46 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!