| *** rlandy has quit IRC | 00:13 | |
| *** ykarel|away has joined #oooq | 01:58 | |
| *** apetrich has quit IRC | 02:10 | |
| *** dsneddon has quit IRC | 03:50 | |
| *** dsneddon has joined #oooq | 04:01 | |
| *** dsneddon has quit IRC | 04:08 | |
| *** udesale has joined #oooq | 04:13 | |
| *** ykarel|away has quit IRC | 04:22 | |
| *** surpatil has joined #oooq | 04:29 | |
| *** soniya29 has joined #oooq | 04:37 | |
| *** dsneddon has joined #oooq | 04:39 | |
| *** ykarel|away has joined #oooq | 04:41 | |
| *** saneax has joined #oooq | 04:46 | |
| *** ykarel|away is now known as ykarel | 04:56 | |
| *** marios has joined #oooq | 05:15 | |
| *** ratailor has joined #oooq | 05:26 | |
| *** akahat has joined #oooq | 05:39 | |
| *** skramaja has joined #oooq | 05:43 | |
| *** ykarel is now known as ykarel|meeting | 05:44 | |
| *** ykarel_ has joined #oooq | 05:48 | |
| *** ratailor_ has joined #oooq | 05:48 | |
| *** ykarel|meeting has quit IRC | 05:50 | |
| *** ratailor has quit IRC | 05:52 | |
| *** dsneddon has quit IRC | 06:05 | |
| *** surpatil has quit IRC | 06:10 | |
| *** surpatil has joined #oooq | 06:10 | |
| *** surpatil has quit IRC | 06:12 | |
| *** surpatil has joined #oooq | 06:13 | |
| *** jfrancoa has joined #oooq | 06:15 | |
| *** dsneddon has joined #oooq | 06:20 | |
| *** jaosorior has joined #oooq | 06:21 | |
| *** holser has joined #oooq | 06:24 | |
| *** dsneddon has quit IRC | 06:25 | |
| *** holser has quit IRC | 06:28 | |
| *** surpatil has quit IRC | 06:30 | |
| *** surpatil has joined #oooq | 06:34 | |
| *** saneax has quit IRC | 06:34 | |
| *** dsneddon has joined #oooq | 06:35 | |
| *** dsneddon has quit IRC | 06:40 | |
| *** holser has joined #oooq | 06:46 | |
| *** bogdando has joined #oooq | 06:49 | |
| chandankumar | marios: you are too fast :-) | 06:52 |
|---|---|---|
| marios | chandankumar: don't worry about updating it man lets just wait for zuul | 06:53 |
| *** dsneddon has joined #oooq | 06:54 | |
| *** udesale has quit IRC | 06:58 | |
| *** udesale has joined #oooq | 06:59 | |
| *** akahat has quit IRC | 07:00 | |
| *** udesale has quit IRC | 07:02 | |
| *** udesale has joined #oooq | 07:02 | |
| *** dsneddon has quit IRC | 07:05 | |
| *** dsneddon has joined #oooq | 07:06 | |
| *** tesseract has joined #oooq | 07:12 | |
| *** tosky has joined #oooq | 07:16 | |
| *** ykarel_ is now known as ykarel|meeting | 07:24 | |
| *** ratailor__ has joined #oooq | 07:35 | |
| *** ykarel_ has joined #oooq | 07:36 | |
| *** ykarel|meeting has quit IRC | 07:36 | |
| *** ykarel_ is now known as ykarel | 07:37 | |
| *** ratailor_ has quit IRC | 07:37 | |
| *** amoralej|off is now known as amoralej | 07:39 | |
| *** dsneddon_ has joined #oooq | 07:39 | |
| *** surpatil is now known as surpatil|lunch | 07:41 | |
| *** dsneddon has quit IRC | 07:42 | |
| *** udesale has quit IRC | 07:48 | |
| *** udesale has joined #oooq | 07:48 | |
| bogdando | o/ please merge https://review.opendev.org/#/q/topic:ci-repro+(status:open+OR+status:merged) | 07:49 |
| bogdando | this is needed for zuul-based reproducer and libvirt multinode jobs | 07:49 |
| *** jpena|off is now known as jpena | 07:49 | |
| *** ykarel is now known as ykarel|lunch | 07:49 | |
| *** kopecmartin|off is now known as kopecmartin | 07:53 | |
| *** akahat has joined #oooq | 07:54 | |
| *** jaosorior has quit IRC | 08:19 | |
| *** panda|off is now known as panda | 08:20 | |
| *** recheck_ has quit IRC | 08:22 | |
| *** recheck has joined #oooq | 08:22 | |
| *** recheck has quit IRC | 08:25 | |
| *** recheck has joined #oooq | 08:25 | |
| *** recheck has quit IRC | 08:27 | |
| *** recheck has joined #oooq | 08:27 | |
| *** ccamacho has joined #oooq | 08:31 | |
| *** jbadiapa has joined #oooq | 08:34 | |
| zbr|ruck | panda: ping! | 08:40 |
| *** ykarel|lunch is now known as ykarel | 08:40 | |
| *** derekh has joined #oooq | 08:42 | |
| panda | zbr|ruck: ICMP TTL exceeded. | 08:42 |
| panda | I just want to die. | 08:42 |
| zbr|ruck | panda: your wish will be granted, but not necessarily at a time you want. | 08:45 |
| zbr|ruck | panda: https://review.rdoproject.org/r/#/c/22822/ | 08:45 |
| zbr|ruck | i am wondering why emilien is added automatically to each review, i wonder if this is caused by my git-review or if is server side | 08:46 |
| zbr|ruck | apparently I do not have the feature enabled, so it must be server side. | 08:49 |
| panda | zbr|ruck: I'm going to block any change on that repo until we agree on a plan to move forward. I'm proposing to either branch the repo, or clone the promoter role directory so we have an older promoter code and a new promoter code. The old promoter part shall not accept any change at all, as we have no way to test the old code, any change put there will just break it. TOo fragile, just fork it, leave it alone | 08:50 |
| panda | and then kill it. | 08:50 |
| panda | marios: ^ | 08:51 |
| zbr|ruck | panda: you are better looking at sprint tasks. there is too much pressure now and not enough time. | 08:52 |
| marios | panda: ack i think we will be discussing this in depth @ scrum ... 'new' directory sounds clean if we can get away without branching it | 08:52 |
| panda | zbr|ruck: not enough time for ? | 08:53 |
| zbr|ruck | marios: yeah, I like marios proposal, we can have a "promoter2" role, for a while. | 08:53 |
| panda | marios: zbr|ruck taht also means that we need to update all the code that we have taht is not merged yet | 08:53 |
| marios | zbr|ruck: its panda proposal :D | 08:53 |
| zbr|ruck | panda propose new branch, but new role seems easier to deal with. | 08:54 |
| panda | zbr|ruck: I proposed new dir too. | 08:54 |
| zbr|ruck | as long you do not copy the entire ci-scripts it should be fine. | 08:55 |
| panda | ok since we are all in agreement, I'll make the changes | 08:56 |
| zbr|ruck | panda: can you please tell me why my pipefail change broke molecule-delegated. at https://review.rdoproject.org/r/#/c/22763/ | 08:56 |
| panda | not sure where ... I have a stack of 6 changes now all floating | 08:57 |
| panda | ah yes, I have it. | 08:57 |
| zbr|ruck | panda: make small ones and i will be glad to help you merge them. | 08:57 |
| panda | zbr|ruck: I already have small ones, only 200 lines each | 08:57 |
| panda | well one is 500 | 08:57 |
| panda | this is the smallest they can get | 08:57 |
| chandankumar | marios: any thing more needed on this https://review.opendev.org/#/c/684260/ as rocky job is passing | 08:58 |
| *** jbadiapa has quit IRC | 08:59 | |
| marios | ack revoted chandankumar | 09:05 |
| marios | panda: could i add a pip install dlrn here https://review.rdoproject.org/r/#/c/22772/6/playbooks/molecule-delegated-promote-images.yml | 09:05 |
| chandankumar | marios: thanks! | 09:05 |
| marios | panda: i tried added there https://review.rdoproject.org/r/#/c/22772/6/playbooks/molecule-delegated-promote-images-pre.yml but it didn't like it | 09:05 |
| panda | marios: remove it from here https://review.rdoproject.org/r/#/c/22772/6/playbooks/molecule-delegated-promote-images-pre.yml@28 and you should be fine. dlrn is not an rpm package AFAIK just python package | 09:11 |
| marios | panda: k (trying with pip but will remove it if you think not necessary... needed it in my local virtual env thats why i added it there) | 09:11 |
| zbr|ruck | i fixed the venv on promoter, next run should pass, it failed to install docker because some files were owned by root inside promoter_venv | 09:12 |
| panda | do not fixe anything on the promoter | 09:15 |
| panda | marios: pip is fine, just try not to install as an rpm package, because it doesn't exist | 09:16 |
| panda | zbr|ruck: nobody should touch the promoter at all at this point, it worked until now, we revert any change that we have, and leave it alone. If you want to fix something fix it on the new code | 09:17 |
| marios | panda: ack also added docker there (via pip)... the scenario is going to fail didn't deal with the promote-images.sh images yet justt trying to get the ci to the same point as my local | 09:17 |
| zbr|ruck | marios: can you help me debug something? i want to find out why that image was not found on the image server. http://promoter.rdoproject.org/centos7_stein.log | 09:32 |
| zbr|ruck | that line: The promoted hash was not found: https://images.rdoproject.org/centos7/stein/rdo_trunk | 09:33 |
| panda | zbr|ruck: how old is this hash ? | 09:34 |
| marios | zbr|ruck: weird ... i was going to say maybe the upload job failed for that hash but then the other jobs wouldn't run if it was the case | 09:34 |
| marios | panda: but the log is from today | 09:35 |
| panda | how old is this hash ? | 09:35 |
| panda | the log may be from today, but the hash is what matters | 09:35 |
| panda | there is a script that delets old images in the images server | 09:35 |
| marios | panda: yeah i mean why would it promote an old hash but running today | 09:35 |
| panda | if they don't have any links attached to them | 09:35 |
| panda | so it's entirely possible the images were uploaded 5 days ago, and since no promotion happened, they were erased | 09:36 |
| panda | every small action on the promoter has a devastating cascading effect ... | 09:37 |
| *** udesale has quit IRC | 09:53 | |
| *** udesale has joined #oooq | 09:54 | |
| *** surpatil has joined #oooq | 09:56 | |
| *** surpatil|lunch has quit IRC | 09:56 | |
| *** jaosorior has joined #oooq | 09:59 | |
| *** jaosorior has quit IRC | 10:06 | |
| zbr|ruck | tomorrow i am going in PTO, never being so happy about that. | 10:09 |
| *** dsneddon_ has quit IRC | 10:09 | |
| zbr|ruck | panda: please check https://review.rdoproject.org/r/#/c/22822/ | 10:09 |
| panda | zbr|ruck: what would this accomplish ? | 10:13 |
| zbr|ruck | panda: is stated clearly in the message. also we do have a rule about no non-voting jobs for long time. | 10:14 |
| zbr|ruck | i had a case where one of my changes broke it and I almost ignored it, because I did (wrongly) assume that the nv jobs broken. | 10:14 |
| zbr|ruck | if is working and not flaky it must be voting. | 10:15 |
| panda | zbr|ruck: let me summarize what it's happening there right now. We have the new code that has tests. We also have a production promoter server using old code that CANNOT be tested in any way other than manually pulling changes and see if we can block the promotions for 5 days. You want to make voting a job taht tests the new code only, that would also potentiall block changes to the old code for no reasons. | 10:16 |
| marios | panda: wdyt https://review.rdoproject.org/r/#/c/22655/8/ci-scripts/dlrnapi_promoter/tests/staging-setup/staging_environment.py@157 | 10:17 |
| marios | brb | 10:17 |
| zbr|ruck | panda: your sentence makes no sense to me. | 10:18 |
| panda | zbr|ruck: take a look ata https://review.rdoproject.org/r/22805, patches 1 and 2 | 10:20 |
| panda | this was needed to fix the promoter, as it was using ansible in side the virtualenv | 10:20 |
| panda | and not working at all | 10:20 |
| panda | zbr|ruck: tdo-tox-molecule was failing because now the ansibe-playbook was not inside the virtualenv | 10:21 |
| panda | zbr|ruck: so a job that is testing the new promoter code, was blocking a change that was needed for the old promoter code, that one that uses ansible from package | 10:21 |
| panda | the solution here was to remove part of the new code, to make promoter work again, but the test was testing that the new code works | 10:22 |
| panda | complete mismatch | 10:22 |
| zbr|ruck | bj in 5? i need to get a coffee, maybe we can clarify few more things before i get insane | 10:23 |
| *** jaosorior has joined #oooq | 10:24 | |
| panda | I'm already insane | 10:25 |
| *** dsneddon_ has joined #oooq | 10:26 | |
| zbr|ruck | https://bluejeans.com/2655417928 | 10:30 |
| panda | marios: what you're asking is partially here https://review.rdoproject.org/r/#/c/22664/ | 10:31 |
| *** dsneddon_ has quit IRC | 10:33 | |
| marios | panda: k will check thx but i think we need it anyway i.e. it should match the real one. might be a bit messy to 'fix' it now i wonder how many places reference the path as /centos/ already | 10:34 |
| *** saneax has joined #oooq | 10:45 | |
| *** jaosorior has quit IRC | 10:51 | |
| *** jpena is now known as jpena|lunch | 11:00 | |
| *** dsneddon_ has joined #oooq | 11:03 | |
| *** udesale has quit IRC | 11:17 | |
| amoralej | zbr|ruck, we've updated ovs version in queens https://review.rdoproject.org/r/#/c/22769/ you should start getting it updated in jobs in some minutes | 11:29 |
| amoralej | let us know if you observe anything abnormal | 11:29 |
| zbr|ruck | amoralej: sure, if I get pass the other broken stuff. | 11:33 |
| amoralej | or if this update breaks anything that was working now :) | 11:34 |
| amoralej | hopefully not, it passed ci | 11:35 |
| *** tosky_ has joined #oooq | 11:44 | |
| *** tosky has quit IRC | 11:46 | |
| *** soniya29 has quit IRC | 11:47 | |
| *** dsneddon_ has quit IRC | 11:47 | |
| *** ratailor__ has quit IRC | 11:50 | |
| *** saneax has quit IRC | 11:54 | |
| *** amoralej is now known as amoralej|lunch | 11:55 | |
| *** akahat has quit IRC | 11:58 | |
| EmilienM | zbr|ruck: no worries, I'll remove myself | 11:59 |
| weshay | zbr|ruck, can you please kick a testproject that builds the stein images (upload featureset02) for 2019-10-03 04:07:30,949 30687 ERROR promoter ERROR: The promoted hash was not found: https://images.rdoproject.org/centos7/stein/rdo_trunk/a361c5edd5734320f781e10358a602f4ba57fc04_647b08e4 | 11:59 |
| weshay | amoralej|lunch, anything abnormal w/ queens.. lolz | 12:00 |
| weshay | now I have to revalidate :( | 12:01 |
| *** tosky_ is now known as tosky | 12:01 | |
| weshay | zbr|ruck, rhel-8 promoted :) | 12:04 |
| weshay | stein missing overcloud images | 12:04 |
| rfolco | panda, hi o/ | 12:05 |
| rfolco | panda, what happened w/ delegated job failing on last check ? | 12:05 |
| *** dsneddon_ has joined #oooq | 12:06 | |
| *** dsneddon_ has quit IRC | 12:12 | |
| *** jpena|lunch is now known as jpena | 12:12 | |
| *** soniya29 has joined #oooq | 12:15 | |
| zbr|ruck | weshay: like https://review.rdoproject.org/r/#/c/22797/ but for stein? | 12:16 |
| weshay | zbr|ruck, ya.. but this time ur going to have to specify the hash please | 12:17 |
| weshay | 2019-10-03 04:07:30,949 30687 ERROR promoter ERROR: The promoted hash was not found: https://images.rdoproject.org/centos7/stein/rdo_trunk/a361c5edd5734320f781e10358a602f4ba57fc04_647b08e4 | 12:17 |
| rfolco | panda, marios I see some attempts to fix delegated container-push test... not sure why this started failing though... | 12:18 |
| rfolco | https://review.rdoproject.org/r/#/c/22656/5..8/ci-scripts/infra-setup/roles/promoter/molecule/container-push/playbook.yml | 12:18 |
| zbr|ruck | weshay: takes time, as I need to dig how to do it | 12:20 |
| weshay | np | 12:20 |
| marios | rfolco: not from me (I just rebased for conflict today ) | 12:20 |
| rfolco | marios, ok, I'm running it locally to find out what broke it | 12:25 |
| weshay | zbr|ruck, easiest way might be a dnm depends-on patch that adjusts a promotion job release file | 12:27 |
| weshay | panda, rfolco so.. rhel8 master promoted afaict.. but I don't see a big message saying so in http://promoter.rdoproject.org/redhat8_master.log | 12:28 |
| weshay | 2019-10-03 05:46:06,773 9109 INFO promoter SUCCESS promoting tripleo-ci-testing as current-tripleo ({'timestamp': 1570018326, 'distro_hash': '9e5988125e88f803ba20743be7aa99079dd275f2', 'promote_name': 'tripleo-ci-testing', 'user': 'review_rdoproject_org', 'repo_url': 'https://trunk.rdoproject.org/rhel8-master/96/8c/968c9526ff2eeafaa13207a791f61173db604fff_9e598812', 'full_hash': '968c9526ff2eeafaa13207a791f61173db604fff_9e598812', | 12:28 |
| weshay | 'repo_hash': '968c9526ff2eeafaa13207a791f61173db604fff_9e598812', 'commit_hash': '968c9526ff2eeafaa13207a791f61173db604fff'}) | 12:28 |
| weshay | ah.. I guess that | 12:28 |
| weshay | panda, zbr|ruck feature request :) | 12:28 |
| panda | ? | 12:29 |
| weshay | a line of \0/\0/\0/\0/\0/\0/\0/\0/\0/\0/\0/\0/\0/\0/\0/\0/\0/\0/\0/\0/\0/\0/\0/\0/\0/\0/\0/\0/\0/\0/\0/ | 12:29 |
| weshay | panda, ya.. sorry.. was just hard to find | 12:29 |
| panda | weshay: after we have the test we can turn the promoter inside out, starting from the logging. | 12:30 |
| weshay | a line of \0/ \0/ \0/ \0/ \0/ \0/ \0/ \0/ \0/ \0/ \0/ \0/ \0/ \0/\0/ \0/ \0/ \0/ \0/ \0/ | 12:30 |
| weshay | k | 12:30 |
| panda | rfolco: which one ? | 12:30 |
| rfolco | weshay, lets add some ascii art to the logs w/ a nice pic like a trophy or a big smile | 12:30 |
| panda | rfolco: wonderful, but only AFTER we have test merged | 12:30 |
| panda | rfolco: and the server is using the tested code | 12:31 |
| panda | we have a bit to discuss during the scrum | 12:31 |
| chandankumar | weshay: time for 1:1 | 12:31 |
| rfolco | panda, delegated container-push is failing since patchset 6 in your top patch | 12:31 |
| chandankumar | weshay: https://meet.google.com/jfx-pwrz-meb | 12:31 |
| panda | rfolco: waht is failing right now ? | 12:31 |
| rfolco | https://review.rdoproject.org/r/#/c/22656/5..8/ci-scripts/infra-setup/roles/promoter/molecule/container-push/playbook.yml | 12:31 |
| rfolco | ps5 --> molecule-delegated SUCCESS in 4m 55s (non-voting) | 12:32 |
| rfolco | ps6 --> molecule-delegated FAILURE in 4m 33s (non-voting) | 12:32 |
| panda | WTH | 12:33 |
| panda | what's "playbook" ? | 12:33 |
| rfolco | nothing changed from 5 to 6 | 12:33 |
| rfolco | in the playbook/test itself | 12:33 |
| rfolco | panda, check if any image left behind task is failing... looking for what broke it | 12:34 |
| *** rlandy has joined #oooq | 12:34 | |
| panda | we need to regroup | 12:36 |
| panda | rfolco: the new conditional with |length == 0 is not working as expected | 12:37 |
| rfolco | panda@radio: stick together team. fire in the hole! | 12:37 |
| rfolco | roger that | 12:37 |
| *** dsneddon_ has joined #oooq | 12:38 | |
| rfolco | panda, it was in ps5... | 12:38 |
| *** amoralej|lunch is now known as amoralej | 12:44 | |
| panda | rfolco: mmmmhh | 12:48 |
| panda | rfolco: I know what it is | 12:49 |
| panda | rfolco: it's the validation container | 12:49 |
| panda | rfolco: the one that validates the login | 12:49 |
| rfolco | panda, whhhaaaaat? looks a harmless test | 12:49 |
| panda | rfolco: a harmless test taht adds a new container to the registry wihtout deleting it | 12:50 |
| rfolco | aaaaaaah | 12:50 |
| panda | rfolco: so when then check "did we remove everything ?" comes, it finds the nop container still and it says: you forgot to delete something | 12:50 |
| rfolco | panda, perfect. I thought we were deleting that test in the other patch | 12:51 |
| panda | rfolco: we have to reaorganize the patches now or we'll not be able to merge anything | 12:52 |
| rfolco | panda, ok good lets chat in scrum, will grab coffee and some bad words to insult people breaking my patch | 12:53 |
| beagles | btw am aware that octavia scenario 10 job is giving grief on master right now http://zuul.openstack.org/builds?job_name=tripleo-ci-centos-7-scenario010-standalone | 13:00 |
| beagles | looking into it | 13:00 |
| *** skramaja has quit IRC | 13:01 | |
| weshay | rlandy, rfolco https://review.rdoproject.org/r/#/c/22800/ https://review.rdoproject.org/r/#/c/22801/ | 13:01 |
| chandankumar | rlandy: needs +w on this https://review.opendev.org/#/c/684260/ | 13:07 |
| rlandy | looking | 13:08 |
| rlandy | weshay: https://github.com/rdo-infra/review.rdoproject.org-config/blob/a370e1e788f47088f8768901add4091671abb4af/zuul.d/upstream.yaml#L21 | 13:15 |
| weshay | rlandy, thanks | 13:16 |
| *** aakarsh has quit IRC | 13:17 | |
| zbr|ruck | weshay: ping when you can. | 13:25 |
| weshay | zbr|ruck, hola | 13:25 |
| marios | rfolco: (cos we discussed before fyi that is blocked https://tree.taiga.io/project/tripleo-ci-board/task/1315?kanban-status=1447274 until we merge https://review.rdoproject.org/r/#/c/22656/ which wires up the manifest push... until then we have no _manifest in rdo registry | 13:25 |
| marios | rfolco: err in docker.io/quay i mean | 13:25 |
| zbr|ruck | weshay: https://meet.google.com/mtg-nhiw-tdd | 13:26 |
| weshay | zbr|ruck, I'm in scrum.. can chat live at the top of the hour | 13:26 |
| zbr|ruck | ok, because I am kinda clueless regarding building that specific image | 13:27 |
| *** udesale has joined #oooq | 13:33 | |
| weshay | zbr|ruck, for stein.. fs02 upload images does it | 13:35 |
| weshay | zbr|ruck, can you please jump off the promotion server.. I'm limiting access to just myself at this time | 13:36 |
| weshay | thanks | 13:36 |
| zbr|ruck | i am out | 13:36 |
| weshay | zbr|ruck, thanks.. periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset002-stein-upload | 13:37 |
| weshay | is the job.. that builds the images, tests them.. and uploads | 13:37 |
| * weshay checks the image server.. this should not have promoted in the first place w/ that missing | 13:37 | |
| zbr|ruck | yeah but you asked me to promote a specific hash, that is what did not know. | 13:37 |
| weshay | https://github.com/rdo-infra/ci-config/blob/master/ci-scripts/dlrnapi_promoter/config/CentOS-7/stein.ini#L20 | 13:38 |
| weshay | ah.. rlandy we can specify a hash in testproject review right? ^ | 13:39 |
| rlandy | ack | 13:39 |
| rlandy | weshay: see how we do that in reproducer | 13:40 |
| weshay | zbr|ruck, | 13:41 |
| weshay | vars: | 13:41 |
| weshay | featureset_override: | 13:41 |
| weshay | dlrn_hash_tag: a317d33dbcda3681a967ce1ed983020ff3512311_58c6e4ba | 13:41 |
| weshay | indented correctly of course | 13:41 |
| zbr|ruck | thanks! | 13:42 |
| zbr|ruck | weshay: https://review.rdoproject.org/r/#/c/22868/1/.zuul.yaml ? | 13:45 |
| *** aakarsh has joined #oooq | 13:50 | |
| *** surpatil has quit IRC | 13:54 | |
| *** soniya29 has quit IRC | 13:54 | |
| weshay | 2019-10-03 09:23:44.917571 | primary | +(./toci_quickstart.sh:46): main(): export 'QUICKSTART_DEFAULT_RELEASE_ARG=--extra-vars @/home/zuul/workspace/.quickstart/config/release/tripleo-ci/RedHat-8/master.yml' | 13:55 |
| weshay | dlrn_hash_tag: current-tripleo | 13:55 |
| *** rnoriega- has joined #oooq | 14:06 | |
| *** Vorrtex has joined #oooq | 14:07 | |
| *** rnoriega_ has quit IRC | 14:09 | |
| rlandy | weshay: can you post the job link please | 14:11 |
| weshay | http://logs.rdoproject.org/53/682853/13/openstack-check/tripleo-ci-rhel-8-scenario004-standalone-rdo/3dd6834/job-output.txt | 14:12 |
| rlandy | 2019-10-03 09:33:56 | tag: current-tripleo | 14:15 |
| *** Goneri has joined #oooq | 14:17 | |
| weshay | rlandy, proxy issue http://mirror.regionone.rdo-cloud-tripleo.rdoproject.org:8080/rdo/rhel8-master/current-tripleo/delorean.repo | 14:19 |
| rlandy | interesting - commit is correct | 14:21 |
| weshay | zbr|ruck, ok.. proxy issues in rdo :) | 14:29 |
| weshay | zbr|ruck, https://bugs.launchpad.net/tripleo/+bug/1846504 | 14:30 |
| openstack | Launchpad bug 1846504 in tripleo "rdo mirror/proxy not working correctly" [Critical,Triaged] - Assigned to wes hayutin (weshayutin) | 14:30 |
| weshay | zbr|ruck, looks ok now.. | 14:30 |
| weshay | zbr|ruck, I can chat now | 14:30 |
| rfolco | panda, pls let me know when you get validation nop fixed / removed or if you want me to do that | 14:30 |
| bogdando | hi. please merge https://review.opendev.org/#/q/topic:ci-repro+(status:open+OR+status:merged) | 14:31 |
| bogdando | this is needed for zuul-based reproducer and libvirt multinode jobs | 14:31 |
| bogdando | rlandy, weshay: ^^ | 14:31 |
| weshay | bogdando, cool /me looks | 14:31 |
| bogdando | thanks! | 14:32 |
| *** ykarel is now known as ykarel|afk | 14:33 | |
| *** rfolco is now known as rfolco|bbl | 14:35 | |
| rlandy | looking | 14:35 |
| weshay | rlandy, I think we can merge both of those? | 14:54 |
| rlandy | weshay: ack | 14:54 |
| *** jpena is now known as jpena|off | 14:59 | |
| panda | rfolco|bbl: delegated job fixed | 15:00 |
| bogdando | rlandy, weshay: thanks! | 15:03 |
| weshay | panda, this is an anchor definition right? &scenario004 | 15:08 |
| weshay | when one uses & | 15:08 |
| weshay | zbr|ruck, ^ | 15:09 |
| zbr|ruck | yeag | 15:10 |
| rlandy | weshay: just fyi - to fox failing master/stein bm ... https://code.engineering.redhat.com/gerrit/182659 Add overcloud_container_cli to match OVB | 15:11 |
| zbr|ruck | weshay: this explains it ok-ish: https://confluence.atlassian.com/bitbucket/yaml-anchors-960154027.html | 15:11 |
| zbr|ruck | simple case: to define &foo. and to use/expand &foo | 15:11 |
| zbr|ruck | but if you need to override the expansion, you will see <<: *foo construct | 15:12 |
| zbr|ruck | it does not work with lists, only with dictionaries | 15:12 |
| *** ykarel|afk is now known as ykarel|away | 15:13 | |
| panda | weshay: yes, the *scenario004 will be the reference to that definition | 15:13 |
| panda | weshay: rlandy rfolco|bbl marios all patches are rebased and are passing voting tests. I'll start to merge stuff with your blessing now. | 15:17 |
| marios | panda: ack | 15:18 |
| rlandy | panda: sec - I need to change staging/playbook | 15:19 |
| rlandy | will update | 15:19 |
| *** jbadiapa has joined #oooq | 15:19 | |
| panda | rlandy: yesterday I forgot to mention that you can test the test_promoter_test.py locally wiht the command "tox -epy27 -epy35 ci-scripts/dlrnapi_promoter/tests/staging-setup/test_promoter_test.py" it will emulate what the job upstream is doing, but locally and only for the test that concerns you | 15:20 |
| rlandy | panda: https://review.rdoproject.org/r/#/c/22804/ fails a lot of tests - not sure we want to merge that yet | 15:21 |
| panda | rlandy: I'll start merging my 4 patches in the stack, no rush | 15:21 |
| rlandy | panda: sorry - got caught up in bm stuff | 15:21 |
| marios | panda: i think we need to tweak the images setup a bit more, not just version | 15:23 |
| *** Vorrtex has quit IRC | 15:23 | |
| marios | panda: for example add a current-tripleo there too | 15:23 |
| marios | panda: but we can do it on top | 15:23 |
| marios | panda: i'll explain tomorrow, but basically the promote-images.sh is expecting an existing 'current-tripleo' that is can rename to previous-current | 15:24 |
| marios | panda: you around tomorrow? | 15:24 |
| panda | marios: yes | 15:24 |
| panda | marios: I think I fixed that in the latest refactor | 15:25 |
| marios | panda: ok lets do a sync in our morning then if you have time ... the skeleton is there now https://review.rdoproject.org/r/#/c/22772/11/ci-scripts/infra-setup/roles/promoter/molecule/promote-images/playbook.yml have a look if you get time | 15:25 |
| rlandy | panda: no worries - I'll merge the test stuff when it passes - won't bock anyone else's work | 15:25 |
| panda | marios: the stage_info.yaml file now contains all the commits with some special pointer | 15:25 |
| panda | marios: for example you can point directly at the currently promoted commit,, the previous promoted commit, and the one that is supposed to be promoted | 15:26 |
| panda | rlandy: let me know if I can help | 15:26 |
| *** ykarel|away has quit IRC | 15:26 | |
| marios | panda: see comment there and logs https://review.rdoproject.org/r/#/c/22772/11/ci-scripts/infra-setup/roles/promoter/molecule/promote-images/playbook.yml (for more on what i mean, otherwise we talk tomorrow anyway thanks) | 15:27 |
| rlandy | panda: hmmm ... looks like linters are failing on unrelated changes | 15:33 |
| rlandy | http://logs.rdoproject.org/04/22804/2/check/tox-linters/dffcb2d/job-output.txt | 15:34 |
| *** udesale has quit IRC | 15:35 | |
| *** udesale has joined #oooq | 15:35 | |
| panda | rlandy: look for timestamp 2019-10-03 15:23:50.780284 | 15:36 |
| *** tesseract has quit IRC | 15:36 | |
| rlandy | k - fixing | 15:36 |
| *** dsneddon_ has quit IRC | 15:41 | |
| *** bogdando has quit IRC | 15:42 | |
| weshay | rlandy, fyi https://review.opendev.org/686430 | 15:42 |
| rlandy | will look in a sec | 15:43 |
| rlandy | 019-10-03 15:23:50.781394 | rdo-centos-7 | ci-scripts/dlrnapi_promoter/tests/staging-setup/test_promoter_test.py:72:60: E226 missing whitespace around arithmetic operator | 15:46 |
| rlandy | 2019-10-03 15:23:50.781446 | rdo-centos-7 | mock_urllib.side_effect = url_lib.HTTPError(*[None]*5) | 15:46 |
| rlandy | panda: ^^ fixed all error except above. what is your intention here? | 15:47 |
| weshay | marios, panda need reviews on this.. and an update to promotion criteria https://review.rdoproject.org/r/#/c/22867/ | 15:48 |
| marios | rfolco|bbl: rlandy panda setup call tomorrow during 'scrum' time? (should i) | 15:49 |
| rlandy | marios: ack - thanks | 15:49 |
| *** ykarel|away has joined #oooq | 15:49 | |
| *** jbadiapa has quit IRC | 15:50 | |
| panda | marios: yes | 15:51 |
| panda | ... if you don't see me tomorrow is probably because storm lorenzo wiped out my home. | 15:52 |
| marios | weshay: ack | 15:54 |
| weshay | thanks | 15:54 |
| weshay | rfolco|bbl, ugh | 15:55 |
| weshay | rfolco|bbl, ping me when ur back | 15:55 |
| panda | rfolco|bbl: ping also me when ur back | 15:57 |
| panda | :) | 15:57 |
| *** rfolco|bbl is now known as rfolco | 16:00 | |
| rfolco | weshay, here o/ | 16:00 |
| rfolco | back from lunch | 16:00 |
| *** marios has quit IRC | 16:01 | |
| rfolco | ok gimme 10 min brb | 16:04 |
| zbr|ruck | panda: ouch, on saturday I have a flight, and guess where I am supposed to fly from... https://www.google.com/maps/@49.663661,-11.8838656,5z/data=!3m1!4b1!4m3!15m2!1m1!1s%2Fg%2F11fnqbt298 | 16:05 |
| weshay | zbr|ruck, I think you could probaby fix this one https://bugs.launchpad.net/tripleo/+bug/1846523 it's either kolla or a bad rpm spec I think | 16:06 |
| openstack | Launchpad bug 1846523 in tripleo "tripleo rhel8 swift-proxy-server container fails to build due to python2 package in spec" [Critical,Triaged] | 16:06 |
| zbr|ruck | weshay: sure, looking now. | 16:07 |
| panda | zbr|ruck: oh happy days | 16:07 |
| weshay | zbr|ruck, btw.. I wanted to chat w/ you re: the docker login test.. so as you have time | 16:07 |
| zbr|ruck | weshay: now. | 16:08 |
| weshay | zbr|ruck, ok | 16:09 |
| weshay | zbr|ruck, https://bluejeans.com/u/whayutin | 16:09 |
| *** jbadiapa has joined #oooq | 16:10 | |
| *** dsneddon_ has joined #oooq | 16:14 | |
| *** jbadiapa has quit IRC | 16:27 | |
| weshay | zbr|ruck, https://review.opendev.org/#/c/685177/ | 16:27 |
| weshay | https://review.opendev.org/#/c/685184/2/roles/create-reproducer-script/templates/reproducer-quickstart.sh.j2 | 16:27 |
| *** udesale has quit IRC | 16:28 | |
| *** ccamacho has quit IRC | 16:31 | |
| chandankumar | see ya tomorrow | 16:34 |
| chandankumar | weshay: please have a look at emit relase review https://review.opendev.org/684260 when free! | 16:34 |
| *** chandankumar is now known as raukadah | 16:34 | |
| zbr|ruck | weshay: very weird python2-ceilometermiddleware is nowhere to be found. but the other one is. | 16:36 |
| raukadah | :-( | 16:36 |
| weshay | raukadah, don't be sad... just need more reviews :) | 16:36 |
| weshay | same thing I said on the mtg | 16:36 |
| raukadah | weshay: I will bring in tuesday tripleo meeting if not get merged then | 16:37 |
| weshay | ++ | 16:37 |
| ykarel|away | zbr|ruck, python2- packages are removed for rhel8 and fedora | 16:39 |
| ykarel|away | u seeing issues somewhere? | 16:39 |
| weshay | ykarel|away, ya.. container builds for rhel8 | 16:39 |
| ykarel|away | weshay, nothing should rely on python2 packages now | 16:39 |
| ykarel|away | swift is full python3 | 16:40 |
| weshay | aye | 16:40 |
| zbr|ruck | ykarel|away: we agree here, but we need to figureout where this comes from: http://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-rhel-8-master-containers-build-push/7a736e5/logs/containers-consolidated-builds.log.txt.gz | 16:40 |
| * ykarel|away looks | 16:40 | |
| raukadah | ykarel|away: time to revisit rhel8 workaround patch in kolla | 16:40 |
| raukadah | may be we can get rid of that | 16:40 |
| zbr|ruck | ouch... i bet that patch is again the cause | 16:41 |
| *** jpena|off is now known as jpena | 16:41 | |
| ykarel|away | if workaroud is related to py2 packages then yes need to get rid of it | 16:41 |
| raukadah | arxcruz|ruck: weshay we got some patches for f21 skip list autmoation https://review.rdoproject.org/r/22862 and https://review.rdoproject.org/r/22870 please review when free | 16:42 |
| zbr|ruck | found! | 16:43 |
| ykarel|away | ues https://review.opendev.org/#/c/671183/9/docker/swift/swift-proxy-server/Dockerfile.j2 | 16:43 |
| zbr|ruck | done. updated review. | 16:46 |
| ykarel|away | ack | 16:50 |
| *** dsneddon_ has quit IRC | 16:53 | |
| *** dsneddon_ has joined #oooq | 16:54 | |
| *** derekh has quit IRC | 16:55 | |
| *** dsneddon_ has quit IRC | 16:59 | |
| weshay | rlandy can you please workflow this.. I may be wrong about how to do it.. but using it to test. https://review.opendev.org/#/c/682853/ | 17:00 |
| rlandy | looking | 17:00 |
| *** dsneddon_ has joined #oooq | 17:00 | |
| weshay | rlandy, oh wait | 17:01 |
| weshay | sorry sorry | 17:01 |
| rlandy | weshay: openstack/tripleo-heat-templates | 17:01 |
| rlandy | I can +1 vote | 17:01 |
| weshay | "standalone_container_cli": "podman", | 17:01 |
| rlandy | scenario004 runs and passes | 17:01 |
| rlandy | so that is ok | 17:02 |
| rlandy | podman is correct for where we don;t use pacemaker | 17:02 |
| weshay | podman is correct for rhel 8 period, w/ or w/o pacemaker | 17:03 |
| ykarel|away | zbr|ruck, re. stein-upload job https://review.rdoproject.org/r/#/c/22868/1/.zuul.yaml? from where u got that hash | 17:03 |
| ykarel|away | i think a361c5edd5734320f781e10358a602f4ba57fc04_647b08e4 is needed to have promotion | 17:04 |
| weshay | rlandy, hrm.. it's still using docker | 17:04 |
| weshay | 2019-10-03 10:54:43 | fatal: [standalone]: FAILED! => {"changed": false, "checksum": "ef190563a3dbc423ae8c85c94f1dec85274be531", "msg": "Destination directory /etc/docker does not exist"} | 17:04 |
| weshay | but container_cli is set to podman | 17:04 |
| weshay | that's where I'm getting mixed up | 17:04 |
| * rlandy looks | 17:06 | |
| rlandy | weshay: which test did you look at? | 17:07 |
| weshay | http://logs.rdoproject.org/53/682853/13/openstack-check/tripleo-ci-rhel-8-scenario004-standalone-rdo/773f529/ | 17:07 |
| rlandy | that should be podman | 17:07 |
| rlandy | I think we went down this road | 17:08 |
| rlandy | getting reviews | 17:08 |
| *** dsneddon_ has quit IRC | 17:09 | |
| weshay | rlandy, ah look at that | 17:09 |
| weshay | http://logs.rdoproject.org/53/682853/13/openstack-check/tripleo-ci-rhel-8-scenario004-standalone-rdo/773f529/logs/undercloud/home/zuul/standalone_parameters.yaml.txt.gz | 17:09 |
| weshay | ContainerCli: podman | 17:09 |
| rlandy | ContainerCli: podman | 17:09 |
| rlandy | yep | 17:09 |
| weshay | seems like a boog | 17:10 |
| *** dsneddon_ has joined #oooq | 17:10 | |
| weshay | rlandy, so the passed default_params are not overriding https://opendev.org/openstack/tripleo-heat-templates/src/branch/master/ci/environments/scenario004-standalone.yaml#L106 | 17:10 |
| rlandy | think it's in the heat templates | 17:10 |
| zbr|ruck | ykarel|away: from wes | 17:11 |
| ykarel|away | zbr|ruck, good to cross check than | 17:11 |
| ykarel|away | https://trunk-primary.rdoproject.org/api-centos-stein/api/civotes_detail.html?commit_hash=a361c5edd5734320f781e10358a602f4ba57fc04&distro_hash=647b08e47a72f7533142c09074f227628f08f9fa | 17:12 |
| rlandy | https://review.opendev.org/#/c/679308/ | 17:14 |
| weshay | rlandy, perhaps the bug is here.. for not switching on container_cli https://opendev.org/openstack/ansible-role-container-registry/src/branch/master/tasks/main.yml | 17:14 |
| rlandy | but that was an ovb failure | 17:14 |
| *** tosky has quit IRC | 17:14 | |
| *** dsneddon_ has quit IRC | 17:14 | |
| rlandy | https://opendev.org/openstack/tripleo-heat-templates/src/branch/master/ci/environments/scenario004-standalone.yaml#L106 - so correct | 17:15 |
| rlandy | I remember scenarios004 is an iisue | 17:16 |
| rlandy | look in zuul | 17:16 |
| rlandy | https://github.com/rdo-infra/rdo-jobs/blob/master/zuul.d/standalone-jobs.yaml#L449 | 17:18 |
| rlandy | featureset_override: *scenario004_podman | 17:18 |
| rlandy | ^^ does that work?? | 17:18 |
| rlandy | weshay: since when do we allow featureset_override: *scenario004_podman? | 17:19 |
| weshay | rlandy, ya.. the issue is not w/ ci afaict | 17:19 |
| weshay | ci has podman not docker | 17:19 |
| weshay | if I were to try to fix this... | 17:19 |
| weshay | I would set container_registry_deploy_docker to false in http://logs.rdoproject.org/53/682853/13/openstack-check/tripleo-ci-rhel-8-scenario004-standalone-rdo/773f529/logs/undercloud/home/zuul/standalone_parameters.yaml.txt.gz | 17:20 |
| rlandy | you can try - there is speculative testing | 17:20 |
| rlandy | but strange now | 17:21 |
| weshay | rlandy, do we have a osp-15 or 16 job? | 17:21 |
| weshay | standalone? | 17:21 |
| rlandy | https://review.rdoproject.org/zuul/builds?job_name=tripleo-ci-rhel-8-scenario004-standalone-rdo | 17:22 |
| rlandy | weshay: ack - we have 15 | 17:22 |
| rlandy | getting | 17:22 |
| weshay | rlandy, I don't think this is invoked in https://opendev.org/openstack/ansible-role-container-registry in podman | 17:22 |
| rlandy | weshay: https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/builds?job_name=tripleo-ci-rhel-8-standalone-rhos-15 | 17:23 |
| weshay | thanks for the assist | 17:23 |
| rlandy | https://sf.hosted.upshift.rdu2.redhat.com/logs/periodic/code.engineering.redhat.com/openstack/tripleo-ci-internal-jobs/master/tripleo-ci-rhel-8-standalone-rhos-15/274fdf9/logs/undercloud/home/zuul/standalone_parameters.yaml.txt.gz | 17:24 |
| rlandy | that says podman | 17:24 |
| panda | weshay: which ansible version we want to use in the promoter ? 2.4 from default repo only ? | 17:24 |
| *** dsneddon_ has joined #oooq | 17:24 | |
| rlandy | panda: dig me out of my misery here ... how do I line break this to make python + linters happy? commit['full_hash'] == stage_info['promotions']['promotion_candidate']['full_hash']: | 17:30 |
| panda | rlandy: commit['full_hash'] = \ | 17:31 |
| panda | rlandy: and then indent 4 the next line | 17:31 |
| weshay | panda, what we have is working.. I 2.4 did not work when we tried I think | 17:31 |
| weshay | I think we have 2.8.x | 17:32 |
| panda | rlandy: or promotion_candidate = stage_info['promotions']['promotion_candidate'] ... then ... commit['full_hash'] = promotion_candidate['full_hash'] | 17:32 |
| weshay | ansible-2.8.4-1.el7.noarch | 17:32 |
| rlandy | panda: I tried option one - linters not happy | 17:32 |
| weshay | panda, if we want to freeze it.. we should jinja2 the /etc/yum/config | 17:32 |
| panda | oh wait | 17:32 |
| rlandy | trying option two | 17:32 |
| panda | it's a conditional, then use the second | 17:32 |
| rlandy | ack | 17:32 |
| panda | weshay: ansible 2.8 comes from epel | 17:33 |
| panda | weshay: and they both work ... we just need to agree on the level of compatibility in the tasks .. for example if this needs to run in ansible 2.4 I need to use with_items instead of "loop" | 17:34 |
| rfolco | panda, https://review.rdoproject.org/r/#/c/22737/ is ready to merge IMO | 17:35 |
| rfolco | panda, tripleo-common integration test | 17:35 |
| weshay | panda, think it makes sense to use 2.8.4 | 17:36 |
| rfolco | now will adapt the same workflow for the zuul end-to-end job | 17:36 |
| panda | rfolco: that's what YOU think | 17:36 |
| weshay | and freeze the rpm config on the box w/ yum config | 17:36 |
| *** amoralej is now known as amoralej|off | 17:36 | |
| weshay | rfolco, we need to chat when you have a sec | 17:36 |
| panda | weshay: to use 2.8 from package we need to install epel too then | 17:36 |
| panda | too tired to do it now, next set of patches ... | 17:37 |
| panda | rfolco: please rebase over 22664 ... it's about to merge | 17:37 |
| panda | rfolco: and it's changeing a few things | 17:37 |
| weshay | panda, epel is fine outside of tripleo work.. but locking is a good idea | 17:37 |
| rfolco | panda, you make my life so ... so... exciting | 17:38 |
| panda | weshay: ok, good to know we have the green light for ansible 28 but ONLY from package. focusing on moving the integration test forward, hope we have time to add that too | 17:38 |
| panda | rfolco: I know right ? | 17:38 |
| rfolco | weshay, can I ping you later ? maybe in 1 hour | 17:39 |
| weshay | rfolco, ya.. that's fine.. | 17:39 |
| rfolco | weshay, thx | 17:39 |
| rlandy | tox-linters success - yay! | 17:41 |
| *** ykarel|away has quit IRC | 17:43 | |
| panda | nevermind, we absolutely need ansible2.8 | 17:48 |
| panda | installing EPEL | 17:48 |
| *** rfolco has quit IRC | 17:52 | |
| *** holser has quit IRC | 17:56 | |
| weshay | rlandy, apparently this fixes it https://review.rdoproject.org/r/#/c/22876/3/zuul.d/standalone-jobs.yaml | 18:01 |
| rlandy | ha - ok | 18:01 |
| weshay | https://github.com/openstack/tripleo-heat-templates/blob/master/environments/podman.yaml | 18:02 |
| rlandy | we should see it tum | 18:03 |
| rlandy | run | 18:03 |
| weshay | rlandy, doesn't matter it's boosted | 18:04 |
| weshay | and nonvoting | 18:04 |
| rlandy | charming | 18:04 |
| * rlandy loves non-voting jobs | 18:04 | |
| rlandy | almost as much as linters | 18:04 |
| *** jpena is now known as jpena|off | 18:05 | |
| *** ykarel|away has joined #oooq | 18:11 | |
| panda | maerged 22664 | 18:14 |
| weshay | rlandy, it's being worked on.. | 18:14 |
| panda | we are in business again | 18:14 |
| panda | from here we can merge all the other stuff | 18:14 |
| *** recheck has quit IRC | 18:34 | |
| *** recheck has joined #oooq | 18:34 | |
| *** recheck has quit IRC | 18:37 | |
| *** recheck has joined #oooq | 18:37 | |
| weshay | need to increase the ovb timeout | 18:37 |
| weshay | 2019-09-25 19:14:18 | Ansible timed out at 3768 seconds. | 18:37 |
| rlandy | weshay: I just did that for ovb | 18:38 |
| rlandy | deploy timeout | 18:38 |
| rlandy | had the same error | 18:38 |
| weshay | is that set in tripleo-ci/env/ovb? | 18:39 |
| weshay | rlandy, oh.. you mean .. it immediately overran the new timeout | 18:39 |
| rlandy | weshay: there is job timeout and deploy timeout | 18:39 |
| rlandy | I increased deploy_timeout | 18:39 |
| weshay | ya.. that is what I'm looking at | 18:39 |
| weshay | 2019-09-25 19:14:18 | Ansible timed out at 3768 seconds. | 18:40 |
| weshay | in the overcloud-deploy | 18:40 |
| rlandy | I override it in env_settings | 18:40 |
| rlandy | it was 90 | 18:40 |
| rlandy | I have it at 150 | 18:40 |
| rlandy | wiggle room | 18:40 |
| rlandy | performance regression???? | 18:40 |
| weshay | hrm.. 3768 is 63min | 18:40 |
| rlandy | weshay: check the time deploy actually ran | 18:41 |
| weshay | timeout is still set to 90 | 18:41 |
| weshay | --libvirt-type qemu --timeout 90 | 18:41 |
| rlandy | from the top of the log ti the bottom | 18:41 |
| weshay | http://logs.rdoproject.org/openstack-periodic-latest-released/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-stein/d83fc7e/logs/undercloud/home/zuul/overcloud-deploy.sh | 18:41 |
| weshay | k | 18:41 |
| rlandy | so then is the whole test timing out? | 18:41 |
| weshay | 90 min | 18:44 |
| weshay | it hasn't change | 18:44 |
| weshay | where is your change | 18:44 |
| zbr|ruck | weshay: can we get rid of kolla cherry-picks? is the 2nd time in less than two weeks when we had to rebase them | 18:46 |
| weshay | zbr|ruck, we loose rhel 8 if we get rid of it dude | 18:46 |
| zbr|ruck | gettting rid == merging them somehpw | 18:46 |
| rlandy | weshay: my change? | 18:47 |
| zbr|ruck | somehow I have the impression that we are all fighting to promote some cherry picks, i am not sure we can "scale" well this way. it was supposed to be a short term hack, more of a POC | 18:48 |
| weshay | zbr|ruck, the cherry picks are there until infra and centos get 8 | 18:48 |
| weshay | and probably kolla | 18:49 |
| rlandy | panda: think this is staging env error ... http://logs.rdoproject.org/04/22804/12/check/tox-py36/ee33b00/tox/reports.html | 18:49 |
| rlandy | weshay: I just kicked stein and master bm to compare | 18:55 |
| *** rfolco has joined #oooq | 18:57 | |
| rfolco | panda, I'll check what broke my test | 18:58 |
| *** holser has joined #oooq | 19:00 | |
| rlandy | https://review.rdoproject.org/r/#/c/22804/ | 19:12 |
| rlandy | panda: ^^ ok - tests are passing | 19:12 |
| rlandy | enough to merge | 19:12 |
| *** rlandy is now known as rlandy|brb | 19:13 | |
| *** zbr|ruck is now known as zbr | 19:14 | |
| *** ykarel|away has quit IRC | 19:17 | |
| weshay | rlandy|brb, zbr rfolco please wf https://review.rdoproject.org/r/#/c/22876/ | 19:18 |
| rfolco | looking | 19:18 |
| weshay | rfolco, job just passed in zuul | 19:19 |
| rfolco | weshay, link handy? | 19:19 |
| weshay | rfolco, https://review.rdoproject.org/zuul/stream/df419e82a08647cc974c46aaf544ee1b?logfile=console.log | 19:21 |
| *** holser has quit IRC | 19:22 | |
| weshay | rfolco, do you have wf? | 19:24 |
| rfolco | weshay, no :( | 19:24 |
| weshay | zbr, do you have wf? https://review.rdoproject.org/r/#/c/22876/ | 19:25 |
| *** holser has joined #oooq | 19:31 | |
| rfolco | weshay, wanna chat? | 19:54 |
| weshay | panda++ | 19:56 |
| weshay | zbr, you have the queens patch up? | 19:58 |
| weshay | zbr, see it, I'm blind.. duh | 20:12 |
| panda | rfolco: I needed to modify molecule-delegated-pre a bit to not use ansible from virtualenv | 20:22 |
| panda | rfolco: maybe that's what it's breaking your patch | 20:22 |
| rfolco | panda, no, full_hash has been changed | 20:23 |
| rfolco | I fixed and rebased | 20:23 |
| rfolco | waiting for results, it works local | 20:23 |
| rfolco | panda, hope you don't make me rebase on anything else | 20:23 |
| panda | rfolco: ah yea, now it comes from stage_info['promotions']['promotion_cadidate']['full_hash'] | 20:23 |
| panda | rfolco: I'm done with the patches | 20:24 |
| rfolco | panda, as soon as I get +1 from zuul, please do a review and 2+ w+ if possible | 20:25 |
| rfolco | improvements can be done later, lets get something in for ths sprint | 20:25 |
| rfolco | panda, green light https://review.rdoproject.org/r/#/c/22737/ | 20:30 |
| rfolco | panda, http://logs.rdoproject.org/37/22737/23/check/molecule-tripleo-common/4fb614a/job-output.txt | 20:31 |
| *** rlandy|brb is now known as rlandy | 20:38 | |
| rlandy | Could not resolve host: mirror.regionone.rdo-cloud.rdoproject.org | 20:39 |
| rlandy | hmmm | 20:39 |
| rfolco | rlandy, panda w+ if no objections please https://review.rdoproject.org/r/#/c/22737/ | 20:40 |
| panda | rfolco: are you using any information from the staging environment script ? | 20:41 |
| rlandy | oh what now? | 20:41 |
| rfolco | panda, I rely on artifacts like registries... containers etc | 20:41 |
| rlandy | https://review.rdoproject.org/r/#/c/22804/ | 20:42 |
| rfolco | panda, and I use /tmp/stage-info file as well | 20:42 |
| rlandy | hoe does that pass in check and fail in gate??? | 20:42 |
| rlandy | how | 20:42 |
| rfolco | thats weird | 20:43 |
| rlandy | recheck I guess | 20:43 |
| rlandy | panda: wrt your comment, I will uncomment the log check | 20:44 |
| rlandy | but wanted to get in what was there atm | 20:44 |
| *** Goneri has quit IRC | 20:46 | |
| *** tosky has joined #oooq | 20:50 | |
| rlandy | I may need this ... https://review.opendev.org/#/c/686430/ | 20:52 |
| weshay | rlandy, ugh.. so i think I screwed up the order | 20:55 |
| weshay | -e /usr/share/openstack-tripleo-heat-templates/environments/low-memory-usage.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/podman.yaml -e /usr/share/openstack-tripleo-heat-templates/ci/environments/scenario004-standalone.yaml \ | 20:55 |
| weshay | podman.yml probably needs to be last? | 20:55 |
| weshay | request for preemptive wf https://review.rdoproject.org/r/#/c/22889/1/zuul.d/standalone-jobs.yaml | 20:57 |
| *** aakarsh has quit IRC | 20:59 | |
| panda | rlandy: commented on your patch | 21:06 |
| panda | rfolco: merged your patch | 21:06 |
| rfolco | panda, \o/ | 21:06 |
| panda | rfolco: probably option 3 is the easiest as you don't need dlrn db at all during tests, as we are mocking dlrnapi response | 21:07 |
| panda | rfolco: merged with comments ... :) | 21:08 |
| panda | I meant rlandy: probably option 3 is the easiest as you don't need dlrn db at all during tests, as we are mocking dlrnapi response | 21:08 |
| panda | but if they're really parallel, then other things may step on each other's toes, not just the db | 21:11 |
| weshay | pretty please https://review.rdoproject.org/r/#/c/22889/ | 21:21 |
| rlandy | ok | 21:21 |
| *** jfrancoa has quit IRC | 21:21 | |
| panda | rlandy: I'm seeing if I can get the tests to run serially intead of parallal | 21:22 |
| rlandy | panda: looking into option 3 | 21:22 |
| panda | so we don't spawn 4 staging env setup | 21:22 |
| rlandy | weird that they pass check and not gate | 21:23 |
| rlandy | should be the same | 21:23 |
| panda | rlandy: since parallelization is arbitrary, it'entirely possible that in check, the 4 tests were assigned to the same executor, and they ran in fact in serial | 21:23 |
| rlandy | ah ok | 21:23 |
| panda | rlandy: testing serialization in https://review.rdoproject.org/r/22890 | 21:24 |
| rlandy | weshay: ok to wf https://review.rdoproject.org/r/#/c/22889/ - not sure how preemptive | 21:29 |
| weshay | rlandy, ya.. the job that is running isn't even testing scen004 | 21:29 |
| weshay | so it just doesn't matter | 21:29 |
| weshay | rlandy, more interested in | 21:29 |
| weshay | your thoughts on the order | 21:29 |
| weshay | I think the last one wins | 21:29 |
| weshay | rlandy, but it's safe to wf | 21:29 |
| weshay | for sure | 21:29 |
| rlandy | weshay: always with yaml files | 21:29 |
| rlandy | k - w+ - we can revert | 21:30 |
| weshay | oh. that's how all yaml files work? | 21:30 |
| weshay | last wins | 21:30 |
| weshay | guess that makes sense | 21:30 |
| panda | crap, 150 nova reviews on rdo, nothing will ever ever merge now. | 21:32 |
| rlandy | panda: those aren't real | 21:33 |
| rlandy | they just show up for some reason every now and again | 21:34 |
| rlandy | nova doesn | 21:34 |
| rlandy | 't trigger on rdo | 21:34 |
| panda | I know, but the wait is real ... | 21:35 |
| panda | rlandy: the serialization works with my tests. Le'ts see if I can undertand how to activate it on uinttest class. | 21:35 |
| rlandy | there is something about 5pm ish that makes rdo go nuts | 21:35 |
| rlandy | panda: great | 21:35 |
| rlandy | at least that explains the sporadic failures | 21:36 |
| rlandy | https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/stream/07a1dd98adb648ed95571649e302355c?logfile=console.log | 21:36 |
| rlandy | yay- master is getting somewhere | 21:36 |
| rlandy | weshay: ^^ we can compare that time to ovb when it starts deploying | 21:37 |
| panda | rlandy: seems to be working, mind if I upload new PS ? | 21:40 |
| panda | rlandy: otherwise it's really a 2 lines change | 21:42 |
| panda | rfolco: you should limit the tripleo-common job to the relevant files | 21:46 |
| panda | rfolco: mmhh, and also molecule-delegated | 21:46 |
| rfolco | panda, hmm ok, just like staging job ? | 21:46 |
| panda | rfolco: possibly, yes, but you should include also the molecule files themselves | 21:48 |
| rfolco | panda, ok will do it later tonight | 21:49 |
| panda | rfolco: you're working on the 2-components integration test now ? | 21:49 |
| rfolco | panda, yes, and on the improvements for the molecule one | 21:49 |
| rfolco | panda, am I blocking you somehow ? | 21:50 |
| panda | rfolco: not at all | 21:51 |
| panda | rfolco: keep calm and carry on | 21:51 |
| rfolco | panda, cool, let it go | 21:52 |
| rlandy | panda: go4it | 22:02 |
| rlandy | no point in multiple patch sets | 22:03 |
| rlandy | 2019-10-03 21:44:12.960456 | rdo-centos-7 | ci-scripts/dlrnapi_promoter/tests/staging-setup/test_staging_environment.py:98:1: E302 expected 2 blank lines, found 1 | 22:04 |
| rlandy | 2019-10-03 21:44:12.960527 | rdo-centos-7 | @pytest.mark.serial | 22:04 |
| rlandy | panda: ^^ just fyi - your patch has a linter failure | 22:04 |
| rlandy | lucky you :( | 22:04 |
| panda | rlandy: already fixed | 22:05 |
| panda | rlandy: and I've now push new PS in your change | 22:06 |
| panda | pushed* | 22:06 |
| rlandy | panda: thanks | 22:08 |
| rlandy | nova jobs are gone | 22:09 |
| panda | they went nova. | 22:09 |
| rlandy | it's some sort of a zuul hitch | 22:09 |
| rlandy | they show up everywhere | 22:09 |
| rlandy | and then gather themselves back to the nova homeland | 22:10 |
| panda | rlandy: py27 and py36 passes, when jobs complete, I'm going to merge | 22:10 |
| rlandy | panda: ack | 22:10 |
| rlandy | I will put the log parser and clean up changes on top of whatever merges | 22:11 |
| rlandy | makes life a bit easier | 22:11 |
| weshay | panda, rlandy I have something that will make our lives a little easier.. haven't tested it.. and I'm probably doing it wrong.. but you'll get the drift | 22:11 |
| weshay | patch in a few | 22:11 |
| rlandy | amazing | 22:11 |
| rlandy | love living on the edge | 22:11 |
| rlandy | haven't tested it.. and I'm probably doing it wrong.. | 22:12 |
| panda | rlandy: when marios' work merges tomorrow (morning I hope) you'll have your logs and stuff to gnaw. | 22:13 |
| rlandy | ok | 22:13 |
| rlandy | really I can test it off the real logs | 22:13 |
| rlandy | then we should see the test kick on playbooks/staging | 22:14 |
| *** ksambor has quit IRC | 22:16 | |
| weshay | rlandy, panda something like https://review.rdoproject.org/r/22892 | 22:17 |
| panda | weshay: as I said, once we have the tests in place, we can turn the logging inside out. we can even put an audio library to hear a fanfare | 22:19 |
| weshay | lolz :) | 22:19 |
| weshay | totally fine.. just wanted to plat a seed and watch it grow :) | 22:19 |
| panda | weshay: or a give with you dancing | 22:19 |
| panda | gif* | 22:19 |
| weshay | panda, aye.. you see the point though of providing the link to the dlrn_api html report | 22:20 |
| weshay | as I wait for queens to promote | 22:20 |
| rlandy | logger.info('\0/ \0/ \0/ woot!!!') | 22:22 |
| rlandy | cute | 22:22 |
| rlandy | decent idea | 22:23 |
| rlandy | https://review.rdoproject.org/r/#/c/22804/ aya | 22:23 |
| rlandy | yay | 22:23 |
| panda | weshay: we already have the information of what passed and what failed. We can put it in the log together with that link for further investigation | 22:26 |
| weshay | panda, it's much easier to see the entire history of the runs on a hash | 22:26 |
| weshay | panda, we don't provide that | 22:26 |
| weshay | panda, for example | 22:26 |
| weshay | https://trunk-primary.rdoproject.org/api-centos-master-uc/api/civotes_detail.html?commit_hash=b53ec95f1f048d235cf9ad0d9cc5d672cb9fa24a&distro_hash=3ec6d70a4dff9654f09b0ae697ce9714fa9aae72 | 22:26 |
| weshay | panda, please add that feature at the appropriate time :) | 22:27 |
| panda | weshay: we could in the same logs, we have all the infor needed, and we could even use a nice table format for that. Anyway, ywah food for thoughts. | 22:28 |
| weshay | panda, dude | 22:28 |
| weshay | if you could render it as a link.. that would cool.. and no | 22:28 |
| weshay | don't render it as a table.. | 22:28 |
| weshay | it still won't be the entire history | 22:29 |
| weshay | panda, do what I'm asking please | 22:29 |
| rlandy | panda++ | 22:30 |
| rlandy | looking better on gates | 22:30 |
| * rlandy has to go ... back later | 22:31 | |
| *** rlandy is now known as rlandy|bbl | 22:31 | |
| panda | weshay: when I'm finished with the promoter you won't even recognize it. It will even solve problems automatically and make the coffee while you relax. | 22:32 |
| weshay | panda, that's also not what I want to hear | 22:32 |
| weshay | I want an html link.. that's it | 22:32 |
| panda | weshay: you're still listening to me ? I'm dropping on the keyboard ... the long night of the merges has ended ... and we finally merged everything. Tomorrow will be a big day, we'll probably have the promotion job running end to end finally with all the components. | 22:34 |
| weshay | that is very cool!! | 22:34 |
| weshay | \0/ | 22:34 |
| weshay | well done :) | 22:34 |
| panda | completionist out. Peace. | 22:35 |
| weshay | :) | 22:35 |
| *** panda is now known as panda|off | 22:35 | |
| recheck | [rdo-blog] RDO is ready to ride the wave of CentOS Stream → https://blogs.rdoproject.org/2019/10/rdo-centos-stream/ | 22:50 |
| weshay | say what??? | 22:50 |
| *** jtomasek has quit IRC | 23:07 | |
| *** holser has quit IRC | 23:08 | |
| *** tosky has quit IRC | 23:23 | |
| *** jtomasek has joined #oooq | 23:41 | |
| *** jtomasek_ has joined #oooq | 23:52 | |
| *** jtomasek_ has quit IRC | 23:54 | |
| *** jtomasek has quit IRC | 23:54 | |
| *** jtomasek has joined #oooq | 23:54 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!