| *** altlogbot_3 has quit IRC | 00:46 | |
| *** altlogbot_0 has joined #oooq | 00:47 | |
| *** bhagyashris has joined #oooq | 01:11 | |
| *** Vorrtex has joined #oooq | 01:23 | |
| *** d0ugal has quit IRC | 01:52 | |
| *** d0ugal has joined #oooq | 02:06 | |
| *** apetrich has quit IRC | 02:10 | |
| weshay | rlandy|ruck|bbl we may have an unstable update job http://zuul.openstack.org/builds?job_name=tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates | 03:28 |
|---|---|---|
| weshay | meh.. failed on infra in your run 2019-08-19 20:31:41.029825 | primary | Data could not be sent to remote host "10.4.70.40". Make sure this host can be reached over ssh: ssh: connect to host 10.4.70.40 port 22: No route to host | 03:29 |
| *** gkadam has joined #oooq | 03:33 | |
| weshay | the buildah failure looks like a fluke http://zuul.openstack.org/builds?job_name=tripleo-build-containers-centos-7-buildah&change=677063 | 03:34 |
| *** gkadam has quit IRC | 03:40 | |
| *** skramaja has joined #oooq | 03:53 | |
| *** rlandy|ruck|bbl is now known as rlandy|ruck | 03:56 | |
| rlandy|ruck | https://review.opendev.org/#/c/677227/ could w+ | 03:57 |
| rlandy|ruck | anyone pls vote | 03:57 |
| *** udesale has joined #oooq | 04:02 | |
| *** rlandy|ruck has quit IRC | 04:02 | |
| *** ykarel has joined #oooq | 04:07 | |
| *** ratailor has joined #oooq | 04:46 | |
| *** ykarel is now known as ykarel|afk | 04:49 | |
| *** ykarel|afk has quit IRC | 04:53 | |
| *** Vorrtex has quit IRC | 05:04 | |
| *** raukadah is now known as chkumar|rover | 05:15 | |
| *** ykarel|afk has joined #oooq | 05:19 | |
| *** ykarel|afk is now known as ykarel | 05:19 | |
| *** jaosorior has joined #oooq | 05:32 | |
| *** sanjayu_ has joined #oooq | 05:32 | |
| *** ykarel is now known as ykarel|meeting | 05:33 | |
| *** ratailor_ has joined #oooq | 05:35 | |
| *** kopecmartin|off is now known as kopecmartin | 05:37 | |
| *** ykarel has joined #oooq | 05:37 | |
| *** ratailor__ has joined #oooq | 05:38 | |
| *** ykarel|meeting has quit IRC | 05:38 | |
| *** ratailor has quit IRC | 05:38 | |
| *** ratailor_ has quit IRC | 05:40 | |
| *** surpatil has joined #oooq | 06:25 | |
| *** ykarel is now known as ykarel|meeting | 06:26 | |
| *** udesale has quit IRC | 06:33 | |
| *** udesale has joined #oooq | 06:34 | |
| *** udesale has quit IRC | 07:01 | |
| *** udesale has joined #oooq | 07:02 | |
| *** amoralej|off is now known as amoralej | 07:19 | |
| *** pierreprinetti has joined #oooq | 07:33 | |
| *** bhagyashris has quit IRC | 07:39 | |
| *** jpena|off is now known as jpena | 07:41 | |
| *** ykarel has joined #oooq | 07:50 | |
| *** ratailor has joined #oooq | 07:50 | |
| *** ratailor__ has quit IRC | 07:50 | |
| *** ykarel is now known as ykarel|lunch | 07:51 | |
| *** ykarel|meeting has quit IRC | 07:52 | |
| *** apetrich has joined #oooq | 08:13 | |
| *** jaosorior has quit IRC | 08:16 | |
| *** panda has quit IRC | 08:18 | |
| *** panda has joined #oooq | 08:20 | |
| *** ratailor_ has joined #oooq | 08:24 | |
| *** bhagyashris has joined #oooq | 08:25 | |
| *** ratailor has quit IRC | 08:27 | |
| zbr | panda: morning. can you wf https://review.opendev.org/#/c/677227/2 ? | 08:27 |
| *** derekh has joined #oooq | 08:28 | |
| *** ykarel|lunch is now known as ykarel | 08:29 | |
| *** yolanda has quit IRC | 08:30 | |
| *** yolanda has joined #oooq | 08:43 | |
| ksambor | hey, I'm trying to run no-deprecated tripleo ci job reproducer and it failied on task: 'Wait for job to start' with error: https://gist.github.com/mrKisaoLamb/492eea518b8a43ba05d4c0849a6c6725 Any ideas/tips what I'm doing wrong? | 08:46 |
| panda | zbr: chandan's testing patch on that is not passing | 08:50 |
| zbr | panda: chkumar|rover path is failing to get image, not due to out of disk but lets see what he knows. | 08:54 |
| zbr | @oooq: can we please start uding "Needed-By:" or "Tested-By:" messages in commits? I do find hard to find testing patches hidden in some gerrit comments. These do not have any speacial meaning to zuul but they are much more visible. Makes sense? | 08:56 |
| panda | chkumar|rover: even more than that, we are running 28 integration jobs on that patch, and none is providing any feedback on the patch itself ? | 09:00 |
| zbr | panda: that was exactly the question I asked wes yesterday. no answer. | 09:02 |
| chkumar|rover | zbr: sorry which patch? | 09:06 |
| chkumar|rover | zbr: https://review.rdoproject.org/r/#/c/20832/ this one? | 09:08 |
| chkumar|rover | ok I got the failure issue | 09:08 |
| zbr | chkumar|rover: i guess you already answered my question about: why is better to put link tot testing patches directly in the commit. | 09:11 |
| panda | zbr: I don't think the starting patch was clear in the first place | 09:12 |
| zbr | panda: yeah. but by reading the bug i figured out a little bit. | 09:13 |
| zbr | in fact the question would why not changing the default from 10GB to 12GB instead? | 09:14 |
| chkumar|rover | zbr: panda sshnaidm|afk https://review.rdoproject.org/r/#/c/21889/ | 09:15 |
| panda | zbr: I see overcloud_cinder_lvm_loop_device_size: 12288 | 09:15 |
| chkumar|rover | to remove fedora bits | 09:15 |
| ykarel | chkumar|rover, panda is rhel8 job need to be run on stable/stein as well? | 09:16 |
| ykarel | or just master | 09:16 |
| chkumar|rover | ykarel: nope, just master | 09:17 |
| panda | chkumar|rover: why not stein ? | 09:17 |
| ykarel | chkumar|rover, i just noticed it's running in stable/stein https://review.rdoproject.org/zuul/builds | 09:17 |
| ykarel | so asked if it's intentional or not | 09:17 |
| chkumar|rover | ykarel: panda currently dlrn provides train packages only so master is targetted | 09:18 |
| panda | chkumar|rover: and in the future ? | 09:18 |
| zbr | i really HATE the newer zuul output page... it was already hard to find the error but now it takes it to a new level. | 09:18 |
| chkumar|rover | ykarel: In future, donot know may be weshay has the answer | 09:19 |
| zbr | i keep encountering The task includes an option with an undefined variable. The error was: 'featureset' is undefined\ --- everywhere. | 09:19 |
| ykarel | chkumar|rover, panda so is rhel8 will continue even after centos8 is ready? | 09:19 |
| ykarel | i guess not | 09:20 |
| chkumar|rover | zbr: https://review.opendev.org/#/c/677063 | 09:20 |
| panda | ykarel: I guess yes | 09:20 |
| ykarel | panda, ack but only few jobs, right? | 09:20 |
| chkumar|rover | the plan is to use RHEL-8 for third party job and upstream CI will use CentOS 8 | 09:20 |
| ykarel | otherwise it will be duplicated too much | 09:20 |
| ykarel | chkumar|rover, ack, is the plan documented somewhere, so i can also follow | 09:21 |
| zbr | tripleo-ci: another pieace of code that is full of functional tests, not. | 09:21 |
| panda | ykarel: there' a general move of QE upstream, having rhel8 jobs ensure we are getting problems at early development stages | 09:21 |
| zbr | and that was in... common role. | 09:21 |
| zbr | cost of that bug: days of not being able to test/merge other work. | 09:22 |
| ykarel | panda, ack noted | 09:22 |
| chkumar|rover | ykarel: https://tree.taiga.io/project/tripleo-ci-board/epic/940 | 09:22 |
| chkumar|rover | zbr: above featureset patch is merged now | 09:23 |
| zbr | hurrah! | 09:23 |
| ykarel | chkumar|rover, ack Thanks | 09:23 |
| panda | featureset patch ? | 09:23 |
| chkumar|rover | ykarel: https://review.rdoproject.org/r/#/c/21889/ | 09:23 |
| ykarel | chkumar|rover, Done | 09:24 |
| * ykarel checks why it timed out both for fedora and rhel in last periodic | 09:24 | |
| sshnaidm|afk | arxcruz, zbr did you check if 2.8 and removing "hostvars" from job works? | 09:24 |
| arxcruz | sshnaidm|afk: i did not, i'm waiting weshay decision if we will move everything to 2.8, including upstream jobs, or just the rdo jobs | 09:25 |
| *** apetrich has quit IRC | 09:25 | |
| zbr | sshnaidm|afk: no, because of prev bug prevented testing. but i am doing it. | 09:25 |
| sshnaidm|afk | arxcruz, if it doesn't work, all this move doesn't make sense | 09:25 |
| sshnaidm|afk | zbr, which bug? | 09:26 |
| arxcruz | sshnaidm|afk: hostvars or ansible_python_interpreter ? | 09:26 |
| sshnaidm|afk | arxcruz, why should weshay take this decision?? | 09:26 |
| sshnaidm|afk | arxcruz, it's the same | 09:26 |
| arxcruz | sshnaidm|afk: well, i asked yesterday in the meeting and he said he would take a look before a decision being made | 09:26 |
| zbr | sshnaidm|afk++ yep, we should not WAIT for others. | 09:26 |
| arxcruz | not saying he will make the decision, but he said he would take a look | 09:27 |
| arxcruz | so... | 09:27 |
| sshnaidm|afk | arxcruz, anyway, we need to be sure it helps, otherwise it's not relevant | 09:27 |
| * ykarel hmm rhel and fedora build were available after 40 minutes, check was for 30 minutes | 09:27 | |
| zbr | we are engineers, we investigate and make a decision based on what we discover, hopefully a good one. | 09:27 |
| panda | trust us, we are engineers | 09:27 |
| *** apetrich has joined #oooq | 09:28 | |
| sshnaidm|afk | arxcruz, if you take wrong decision, that's fine, we'll -2 it | 09:28 |
| arxcruz | sshnaidm|afk: yes, i know, but we did not get a concensus if we should move all the jobs to 2.8, or only rdo jobs | 09:28 |
| arxcruz | zbr wants to move everything, but that wasn't discussed | 09:28 |
| zbr | sshnaidm|afk: practical question on which repo should I add the test change for ovb ansible change? the one at https://review.opendev.org/#/c/677256/2 | 09:28 |
| sshnaidm|afk | arxcruz, I think we forgot what was the initial task, it's about testing 2.8 w/o python interpreter configured | 09:29 |
| arxcruz | sshnaidm|afk: so, only rdo jobs, not upstream jobs, that was my question, because I did a patch yesterday, and zbr asked me why not everything 2.8 | 09:30 |
| arxcruz | then i wanted to discuss | 09:30 |
| sshnaidm|afk | zbr, yeah, but worth to run an ovb job with dep on this patch to see if it really uses 2.8 | 09:31 |
| sshnaidm|afk | arxcruz, I think testing this should make our picture more clear, if it doesn't work - all this move is not relevant, nothing to discuss | 09:32 |
| arxcruz | sshnaidm|afk: fine by me | 09:32 |
| chkumar|rover | arxcruz: https://review.opendev.org/#/c/676439/ | 09:33 |
| chkumar|rover | thanks! | 09:33 |
| arxcruz | chkumar|rover: the documentation for this config_drive is vague... | 09:33 |
| arxcruz | Enable special configuration drive with metadata. | 09:34 |
| zbr | a feature extreamly useful for zuul would be to be able to declare a list of extra jobs to trigger on a commit, directly inside the commit message. | 09:40 |
| panda | zbr: you can already modify zuul.d dir to your liking | 09:45 |
| *** jaosorior has joined #oooq | 09:47 | |
| zbr | arxcruz: ouch, i found blocker for upgrading ansible, i am looking now where it comes from | 09:48 |
| zbr | ERROR! Unexpected Exception, this is probably a bug: 'PlaybookCLI' object has no attribute 'options' | 09:48 |
| zbr | they changed api | 09:48 |
| sshnaidm|afk | zbr, this is a common error from old ara with 2.8 | 09:49 |
| *** sshnaidm|afk is now known as sshnaidm | 09:49 | |
| zbr | i do not see anything related to ara, but clearly sounds like one I got with it in the past.https://review.rdoproject.org/zuul/stream/242964d298854cebba8aa44737c9f462?logfile=console.log | 09:50 |
| *** ratailor__ has joined #oooq | 09:53 | |
| zbr | sshnaidm: do you know where these errors are coming from? i am still trying to figure out their source. | 09:56 |
| *** ratailor_ has quit IRC | 09:56 | |
| sshnaidm | zbr, console is gone | 09:58 |
| zbr | https://etherpad.openstack.org/p/ssbarnea | 09:58 |
| chkumar|rover | sshnaidm: tempest fix for rhel-8 https://review.opendev.org/#/c/677191/ is merged, feel free to run the build container first then ovb job | 09:58 |
| sshnaidm | chkumar|rover, hmm.. now we need to promote them | 09:59 |
| *** akahat has joined #oooq | 09:59 | |
| sshnaidm | zbr, can you try it on regular multinode job? | 10:01 |
| chkumar|rover | sshnaidm: then we need to wait for another 3 hrs. | 10:01 |
| zbr | i am 99% sure that is caused by rdo zuul itself and will happening with any job trying to use ansible 2.8, in fact I can easily double check that. | 10:02 |
| sshnaidm | chkumar|rover, that's fine, anyway patches for ovb are stuck in queue.. | 10:02 |
| sshnaidm | zbr, just curious if upstream job will work with 2.8 | 10:03 |
| zbr | i am going to test this too | 10:03 |
| sshnaidm | zbr, if it is, then something in rdo, yeah.. I'm inclined to blame ara old version | 10:03 |
| zbr | i bet the fix is easy but how zuul it fails in that case is... epic. | 10:03 |
| zbr | i bet they pre-install ara somewhere on rdo, and that is what is breaks it. likely not happening upstream. | 10:04 |
| *** bhagyashris has quit IRC | 10:06 | |
| *** pierreprinetti has quit IRC | 10:07 | |
| *** udesale has quit IRC | 10:16 | |
| *** udesale has joined #oooq | 10:17 | |
| chkumar|rover | arxcruz: https://trello.com/c/ZvV5ul81/1029-cixlp1836046tripleociproa-tempestscenariotestnetworkbasicopstestnetworkbasicops-failing-on-queens and commented on review/bug itself | 10:23 |
| chkumar|rover | sshnaidm: panda https://review.rdoproject.org/r/#/c/21804/ and https://review.rdoproject.org/r/#/c/21891/ | 10:26 |
| chkumar|rover | needs review and workflow | 10:26 |
| sshnaidm | chkumar|rover, does tripleo-ci-rhel-8-standalone-rdo run on master only too? | 10:27 |
| chkumar|rover | sshnaidm: yes | 10:28 |
| chkumar|rover | sshnaidm: let me add the branches there | 10:28 |
| chkumar|rover | sshnaidm: we need to do the same for all RHEl-8 jobs | 10:30 |
| *** ykarel is now known as ykarel|afk | 10:34 | |
| *** ykarel|afk is now known as ykarel | 10:35 | |
| chkumar|rover | sshnaidm: https://review.rdoproject.org/r/#/c/21893/ | 10:37 |
| *** gkadam has joined #oooq | 10:38 | |
| chkumar|rover | panda: zbr https://review.rdoproject.org/r/#/c/21787/ left a comment on scneario 1-4 jobs | 10:41 |
| panda | chkumar|rover: you're right | 10:43 |
| chkumar|rover | panda: we found the issue, right now in another jobs | 10:43 |
| zbr | panda: chkumar|rover : updated https://review.rdoproject.org/r/#/c/21787/ (added branches like askes) | 10:51 |
| zbr | sshnaidm: sadly upgrades failed on recheck of https://review.opendev.org/#/c/676497/ | 10:57 |
| sshnaidm | zbr, I see it's passing so far | 10:58 |
| *** gkadam is now known as gkadam-afk | 11:00 | |
| *** tesseract has joined #oooq | 11:08 | |
| *** ykarel is now known as ykarel|afk | 11:13 | |
| sshnaidm | zbr, do you have dummy patch that runs these jobs? https://review.rdoproject.org/r/#/c/21787/ | 11:18 |
| *** udesale has quit IRC | 11:18 | |
| *** amoralej is now known as amoralej|lunch | 11:22 | |
| zbr | sshnaidm: can you please rebase https://review.rdoproject.org/r/#/c/21860/ on top of d8d059288278bfb1c3be67bc23e9ea6a1a917994 ? | 11:32 |
| zbr | that was the testing change but it got outdated, and I am unable to rebase because was made by marios | 11:32 |
| chkumar|rover | zbr: panda sshnaidm weshay http://38.145.32.151/usage.txt.gz files list on rdo logserver | 11:33 |
| chkumar|rover | it will help to trim some more unwanted files | 11:33 |
| chkumar|rover | to decrease the load on rdo log server | 11:33 |
| zbr | chkumar|rover: wth is .stackviz/lib/ harvested? | 11:34 |
| chkumar|rover | zbr: it is not needed | 11:35 |
| zbr | also ./logs/undercloud/var/lib/mistral/overcloud/.git/ | 11:37 |
| zbr | stackviz folder itself counts for 81M out of total 211M. | 11:39 |
| chkumar|rover | zbr: I have remove .stackviz last week | 11:39 |
| chkumar|rover | that one was gone | 11:39 |
| chkumar|rover | zbr: sshnaidm panda weshay more better link http://file.pnq.redhat.com/chkumar/logserver/usage.txt | 11:40 |
| *** gkadam-afk is now known as gkadam | 11:42 | |
| *** akahat has quit IRC | 11:49 | |
| *** akahat has joined #oooq | 11:50 | |
| chkumar|rover | arxcruz: can we get a bug https://review.opendev.org/#/c/673400/ for fs01 os_tempest issue so that we can get more attension, thanks! | 11:54 |
| arxcruz | chkumar|rover: what i need is a reproduced environment, that's all i want :) | 11:54 |
| arxcruz | but i tried 4 times and fails the deployment | 11:54 |
| chkumar|rover | arxcruz: Does the issue faced is known from ruck/rover etherpad | 11:55 |
| chkumar|rover | ? | 11:55 |
| arxcruz | chkumar|rover: no | 11:55 |
| chkumar|rover | or is it something related to reproducer? | 11:55 |
| arxcruz | chkumar|rover: it fails in different steps every time | 11:55 |
| arxcruz | :/ | 11:55 |
| arxcruz | i'll try one more time | 11:55 |
| chkumar|rover | arxcruz: if needed, feel free to hold one of the rdocloud node, debug it from there | 11:56 |
| chkumar|rover | arxcruz: ack! | 11:56 |
| arxcruz | chkumar|rover: is that possible to get the one from the check job ? | 11:56 |
| chkumar|rover | arxcruz: yes ping fbo on #sf-ops with the review link and job name to put on hold | 11:57 |
| chkumar|rover | sshnaidm: What is the default timeout for ovb periodic job? | 11:59 |
| chkumar|rover | sshnaidm: for example this https://github.com/rdo-infra/rdo-jobs/blob/master/zuul.d/ovb-jobs.yaml#L518 | 12:00 |
| sshnaidm | chkumar|rover, look at the parent | 12:00 |
| chkumar|rover | yes got it https://github.com/rdo-infra/review.rdoproject.org-config/blob/master/zuul.d/tripleo-rdo-base.yaml#L6 | 12:01 |
| chkumar|rover | it is 2 layer below | 12:01 |
| chkumar|rover | thanks! | 12:01 |
| *** rlandy has joined #oooq | 12:02 | |
| *** rlandy is now known as rlandy|ruck | 12:03 | |
| chkumar|rover | rlandy|ruck: Hey ruck! | 12:05 |
| rlandy|ruck | chkumar|rover: hey | 12:05 |
| rlandy|ruck | scared to look | 12:05 |
| chkumar|rover | rlandy|ruck: featureset fix patch merged | 12:05 |
| chkumar|rover | ci goes green donot know how | 12:05 |
| rlandy|ruck | periodic-tripleo-ci-reproducer-centos-7-libvirt-standalone-vexxhost periodic-tripleo-ci-reproducer-centos-7-openstack-standalone | 12:05 |
| rlandy|ruck | still failing | 12:05 |
| rlandy|ruck | two went green | 12:06 |
| rlandy|ruck | chkumar|rover: great - could not get that past gates yesterday | 12:06 |
| rlandy|ruck | chkumar|rover: did you get your kolla patch in | 12:06 |
| chkumar|rover | rlandy|ruck: kolla patches both in | 12:06 |
| rlandy|ruck | great | 12:06 |
| chkumar|rover | rlandy|ruck: gate goes green in morning | 12:06 |
| rlandy|ruck | chkumar|rover: k - going to merge https://review.opendev.org/#/c/677227/ | 12:07 |
| chkumar|rover | rlandy|ruck: yes | 12:07 |
| ykarel|afk | chkumar|rover, log size is collected in job itslef | 12:07 |
| ykarel|afk | iirc it used to be 80MB some time back | 12:08 |
| rlandy|ruck | chkumar|rover: two of the reproducer tests are ok | 12:08 |
| rlandy|ruck | will see whats still with the other two | 12:08 |
| *** ykarel|afk is now known as ykarel | 12:08 | |
| rlandy|ruck | o nvm - same tests | 12:09 |
| rlandy|ruck | passing now | 12:09 |
| *** surpatil has quit IRC | 12:10 | |
| chkumar|rover | rlandy|ruck: we also found that container build, standlaone. ovb jobs are running on stbale/branch, now fixed | 12:10 |
| chkumar|rover | rlandy|ruck: https://review.opendev.org/#/c/676439/ is also good to go | 12:10 |
| rlandy|ruck | chkumar|rover: k - raised the unauthorized issue to a cix | 12:10 |
| rlandy|ruck | chkumar|rover: k - +2'ed it - will w+ when we see ci on fs001 | 12:11 |
| chkumar|rover | rlandy|ruck: cool, thanks! | 12:12 |
| rlandy|ruck | rhel8 hasn't promoted yet | 12:12 |
| chkumar|rover | rlandy|ruck: fs037 https://bugs.launchpad.net/tripleo/+bug/1840763 | 12:12 |
| openstack | Launchpad bug 1840763 in tripleo "[queens][fs037] Upgrade jobs are failiing while doing overcloud minor update with ssh error " [Critical,Triaged] | 12:12 |
| chkumar|rover | rlandy|ruck: yes due to promotion to testing hash issue fixed now | 12:12 |
| chkumar|rover | rlandy|ruck: https://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-7-master-promote-consistent-to-tripleo-ci-testing/53b2776/job-output.txt.gz#_2019-08-20_00_23_26_053962 and fix merged https://review.rdoproject.org/r/#/c/21889/ | 12:13 |
| rlandy|ruck | chkumar|rover: weshay: we have seen this mor than pnce | 12:13 |
| rlandy|ruck | once | 12:13 |
| rlandy|ruck | https://bugs.launchpad.net/tripleo/+bug/1840763 | 12:13 |
| rlandy|ruck | putting back promotion-blocker | 12:13 |
| rlandy|ruck | chkumar|rover is right there | 12:13 |
| rlandy|ruck | I saw it yesterday | 12:13 |
| rlandy|ruck | and was going to create the same bug if we saw it again | 12:13 |
| chkumar|rover | rlandy|ruck: all the fs037 jobs are failing frm very long time | 12:13 |
| rlandy|ruck | chkumar|rover: they are part of promotion criteria | 12:14 |
| chkumar|rover | rlandy|ruck: do we have something to raise in tripleo meeting | 12:14 |
| rlandy|ruck | they nede to be fixed or removed | 12:14 |
| chkumar|rover | checking | 12:14 |
| rlandy|ruck | chkumar|rover: what happened with pike and ocata? | 12:14 |
| chkumar|rover | rlandy|ruck: not checked with that, we are doing eol soon | 12:15 |
| chkumar|rover | may be tonight or tomorrow | 12:15 |
| rlandy|ruck | chkumar|rover: so other than the gate failures, i don't think so | 12:15 |
| chkumar|rover | rlandy|ruck: https://review.rdoproject.org/r/#/c/21893/ needs +w | 12:16 |
| rlandy|ruck | weshay: ^^ anything you want us to raise at tripleo meeting? | 12:16 |
| rlandy|ruck | done | 12:16 |
| chkumar|rover | rlandy|ruck: do we want to reraise the unauthorized issue in meeting | 12:16 |
| rlandy|ruck | I fixed the image build the same way | 12:16 |
| rlandy|ruck | isk - we raised it last week | 12:17 |
| rlandy|ruck | idk | 12:17 |
| ksambor | hey sshnaidm , rlandy|ruck I have question: Hey, I'm trying to run new reproducer and unfortunately I got: https://gist.github.com/mrKisaoLamb/492eea518b8a43ba05d4c0849a6c6725 Any ideas/tips what I'm doing wrong? | 12:17 |
| rlandy|ruck | no news this week | 12:17 |
| *** akahat has quit IRC | 12:17 | |
| ykarel | rlandy|ruck, i added info regarding ocata and pike eol on tripleo meeting etherpad | 12:18 |
| rlandy|ruck | ksambor: can you tell us at what point you hit that error - also what platform you are running on | 12:18 |
| *** akahat has joined #oooq | 12:18 | |
| rlandy|ruck | ykarel: thanks | 12:18 |
| ksambor | rlandy|ruck: yeah it was during task: Wait for job to start | 12:19 |
| chkumar|rover | rlandy|ruck: as per the logs from this patch https://review.opendev.org/#/c/674919/ for unauthorized it is facing issue while skopeo copy for rsyslog container image in image prepare role https://storage.gra1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/logs_19/674919/21/check/tripleo-ci-centos-7-containers-multinode/adb279d/logs/undercloud/var/log/tripleo-container-image-prepare.log.txt.gz | 12:20 |
| chkumar|rover | rlandy|ruck: https://bugs.launchpad.net/tripleo/+bug/1840763 is not in queens promotion list, I think we can remove promotion blocker? /me donot know | 12:22 |
| openstack | Launchpad bug 1840763 in tripleo "[queens][fs037] Upgrade jobs are failiing while doing overcloud minor update with ssh error " [Critical,Triaged] | 12:22 |
| rlandy|ruck | chkumar|rover: I don't think I am going to raise it again at the tripleo meeting - discussed with weshay - created CIX - will raise it at escalation meeting with prod chain infra | 12:22 |
| chkumar|rover | rlandy|ruck: ok | 12:23 |
| rlandy|ruck | done | 12:24 |
| rlandy|ruck | removed | 12:24 |
| ksambor | rlandy|ruck: so i'm runing this on fedora and hit error during task: Wait for job to start | 12:26 |
| rlandy|ruck | ksambor: iiuc, you mean when the local job is to start? we have seen the job start have a hitch the first time - can you open local gerrit do you see the job created? can you rerun it from there? | 12:26 |
| chkumar|rover | rlandy|ruck: I am increasing timeout for fs035 | 12:26 |
| rlandy|ruck | chkumar|rover: which timeout? | 12:26 |
| rlandy|ruck | deploy or overall job? | 12:26 |
| chkumar|rover | rlandy|ruck: deploy | 12:26 |
| chkumar|rover | during overcloud | 12:26 |
| rlandy|ruck | chkumar|rover: so I asked that yesterday | 12:26 |
| rlandy|ruck | and consenus was not | 12:27 |
| rlandy|ruck | I put that as a question in the card | 12:27 |
| chkumar|rover | rlandy|ruck: ok | 12:27 |
| rlandy|ruck | I think we can try spin on it | 12:27 |
| rlandy|ruck | chkumar|rover: that was my first suggestion | 12:27 |
| rlandy|ruck | if container pulls are taking longer | 12:27 |
| *** ratailor__ has quit IRC | 12:27 | |
| rlandy|ruck | chkumar|rover: I think we should increase the time and spin on it | 12:27 |
| rlandy|ruck | not merge it yet | 12:28 |
| chkumar|rover | rlandy|ruck: ok putting the patch | 12:28 |
| rlandy|ruck | then maybe we can prove it needs longer question is why? | 12:28 |
| rlandy|ruck | if it's just that there are more containers ok | 12:28 |
| rlandy|ruck | if we have a performance issue, we are covering it up | 12:28 |
| zbr | chkumar|rover: do you see a rebase button on https://review.rdoproject.org/r/#/c/21860/ page? | 12:32 |
| weshay | rfolco few min | 12:32 |
| chkumar|rover | zbr: yes | 12:32 |
| *** gkadam has quit IRC | 12:32 | |
| chkumar|rover | zbr: will i rebase it? | 12:32 |
| zbr | chkumar|rover: please press it and put 18c675f32a07deb82df6f8b6129ad237f18c19d1 there. | 12:33 |
| chkumar|rover | zbr: done | 12:33 |
| zbr | chkumar|rover: also maybe you know who I need to tip to get core on rdo? | 12:33 |
| chkumar|rover | zbr: you mean on rdo-jobs? | 12:34 |
| zbr | yeah. | 12:34 |
| zbr | or all | 12:34 |
| chkumar|rover | zbr: will ok, will drop an email to rdo-list | 12:34 |
| zbr | thanks. | 12:34 |
| rfolco | weshay, I'll do bug triage offline, don't worry. | 12:38 |
| weshay | rfolco can you open https://drive.google.com/file/d/12u484rjPgOmr4p1hMYeM9RjgTOIxwpzm/view?usp=sharing | 12:41 |
| rfolco | weshay, yes this one opens | 12:41 |
| weshay | ok.. from there I think you can open draw.io directly | 12:41 |
| zbr | chkumar|rover: also rebase https://review.rdoproject.org/r/#/c/21861/ to 9b7681c255f90423117dd562a6c4fca7cc6aae34 | 12:42 |
| weshay | chkumar|rover rfolco let's triage for a few | 12:42 |
| chkumar|rover | zbr: now sayin merge conflict | 12:42 |
| chkumar|rover | zbr: needs a rebase manually | 12:43 |
| *** jaosorior has quit IRC | 12:43 | |
| zbr | chkumar|rover: are you doing it? i think i may be able to do it myself manually. not sure why rebase is missing from the UI for non cores, as they can always update a review from CLI | 12:44 |
| chkumar|rover | zbr: feel free to do it | 12:44 |
| zbr | probably bad ACL template | 12:44 |
| sshnaidm | ksambor, sometimes it take for me time to wait until tenant will be ready (f29), just check it after 10-15 mins in http://localhost:9000/t/tripleo-ci-reproducer/status | 12:52 |
| sshnaidm | ksambor, when it's ready just recheck your patch in gerrit: http://localhost:8080/q/status:open | 12:53 |
| rlandy|ruck | chkumar|rover: periodic-tripleo-rhel-8-buildimage-overcloud-full-master is green :)) | 12:56 |
| rlandy|ruck | so happy to see that job back | 12:56 |
| *** jaosorior has joined #oooq | 12:59 | |
| weshay | https://etherpad.openstack.org/p/tripleo-meeting-items | 13:02 |
| rlandy|ruck | chkumar|rover: anything else to watch after you log off? | 13:02 |
| chkumar|rover | rlandy|ruck: nope | 13:03 |
| rlandy|ruck | chkumar|rover: cool - quieter day :) | 13:04 |
| *** amoralej|lunch is now known as amoralej | 13:04 | |
| chkumar|rover | rlandy|ruck: Feel free to remove f28 related bugs | 13:06 |
| chkumar|rover | or close it | 13:06 |
| rlandy|ruck | chkumar|rover: from the cockpit or the bugs themselves? | 13:06 |
| rlandy|ruck | oh - the bugs themselves - ok | 13:07 |
| rlandy|ruck | can take care of that this afternoon | 13:07 |
| rlandy|ruck | easy enough | 13:07 |
| rlandy|ruck | sure | 13:07 |
| chkumar|rover | rlandy|ruck: I have closed few of them | 13:07 |
| weshay | chkumar|rover https://bugs.launchpad.net/tripleo/+bug/1815744 | 13:08 |
| openstack | Launchpad bug 1815744 in tripleo "build-test-package does not fail on partial builds" [High,In progress] | 13:08 |
| rlandy|ruck | np - I'll finish the rest | 13:10 |
| rlandy|ruck | chkumar|rover: ^^ | 13:10 |
| chkumar|rover | rlandy|ruck: I am taking care as a rover part. | 13:10 |
| sshnaidm | chkumar|rover, I see new containers have been built now for rhel8, does it contain your fix? | 13:22 |
| sshnaidm | chkumar|rover, if so, I'll rerun the ovb job to test | 13:23 |
| chkumar|rover | sshnaidm: yes it should as it clones stuff from master | 13:23 |
| sshnaidm | chkumar|rover, great | 13:24 |
| panda | rfolco: ping | 13:24 |
| rfolco | panda, o/ | 13:24 |
| rfolco | panda, ace | 13:25 |
| rfolco | 15-0 | 13:25 |
| panda | rfolco: you never played table tennis. | 13:25 |
| rfolco | :) | 13:26 |
| panda | rfolco: anyway, I was looking at the staging setup and I ended up doing some work on https://tree.taiga.io/project/tripleo-ci-board/task/1246?kanban-status=1447275 | 13:26 |
| panda | too | 13:26 |
| panda | rfolco: so before you pick it up, let's sync | 13:26 |
| rlandy|ruck | weshay: can we close this out? https://bugs.launchpad.net/tripleo/+bug/1740928 | 13:26 |
| openstack | Launchpad bug 1740928 in tripleo "Fedora Support for TripleO-Quickstart" [Low,In progress] - Assigned to Alex Schultz (alex-schultz) | 13:26 |
| rfolco | panda, lets sync after community call | 13:27 |
| panda | rfolco: ok | 13:27 |
| rfolco | panda, I did some work in a rdo cloud vm, like excluding volume from the role when staging | 13:29 |
| rfolco | panda, and docker storage overlay2 isn't needed anymore... | 13:30 |
| rfolco | panda, among other things | 13:30 |
| panda | rfolco: becaue it's default ? | 13:30 |
| rfolco | panda, no, there is a task there that injects that to docker json config | 13:31 |
| *** akahat has quit IRC | 13:31 | |
| rfolco | ping community call | 13:32 |
| rfolco | zbr, weshay arxcruz chkumar|rover ^ | 13:32 |
| chkumar|rover | rfolco: /me is skipping the community call directly drop to tripleo meeting sorry. | 13:33 |
| *** ykarel is now known as ykarel|away | 13:33 | |
| rfolco | zbr, weshay joing community call? | 13:33 |
| zbr | sure... | 13:34 |
| weshay | rfolco I have a manager on duty call | 13:34 |
| rfolco | ack weshay | 13:34 |
| *** udesale has joined #oooq | 13:35 | |
| zbr | chkumar|rover: re scen1-4 on rhel8: test job fail due to missing containers as seen on rehttps://review.rdoproject.org/r/#/c/21860/ -- i guess that is expected, right. | 13:39 |
| *** ykarel|away has quit IRC | 13:40 | |
| *** Goneri has joined #oooq | 13:41 | |
| *** udesale has quit IRC | 13:42 | |
| *** udesale has joined #oooq | 13:43 | |
| weshay | rlandy|ruck ya.. anything f28 can be closed | 13:43 |
| *** aakarsh has quit IRC | 13:45 | |
| *** udesale has quit IRC | 13:50 | |
| chkumar|rover | zbr: checking | 13:51 |
| chkumar|rover | zbr: https://review.opendev.org/#/c/676497/ and https://review.opendev.org/#/c/676474/ will fix the issue | 13:54 |
| chkumar|rover | zbr: those containers never built for rhel | 13:55 |
| *** ykarel|away has joined #oooq | 13:59 | |
| panda | rfolco: https://review.rdoproject.org/r/21873 | 13:59 |
| *** ykarel|away is now known as ykarel | 13:59 | |
| *** aakarsh has joined #oooq | 14:14 | |
| chkumar|rover | rlandy|ruck: weshay sshnaidm: dmsimard is looking on this https://bugs.launchpad.net/tripleo/+bug/1839532 feel free to poke him | 14:28 |
| openstack | Launchpad bug 1839532 in tripleo "tripleo gate jobs are failing to pull containers when running on ovh provider with "UNAUTHORIZED" error" [Critical,Triaged] | 14:28 |
| rlandy|ruck | sshnaidm: ok - let's try this again ... have a few minutes to meet about sova updates? | 14:37 |
| *** skramaja has quit IRC | 14:39 | |
| sshnaidm | rlandy|ruck, yeah, let's meet now :) | 14:39 |
| *** ykarel is now known as ykarel|away | 14:40 | |
| sshnaidm | rlandy|ruck, https://bluejeans.com/u/sshnaidm/ | 14:40 |
| rlandy|ruck | sshnaidm: thanks - joining | 14:40 |
| *** openstackstatus has quit IRC | 14:58 | |
| *** openstack has joined #oooq | 14:59 | |
| *** ChanServ sets mode: +o openstack | 14:59 | |
| *** PagliaccisCloud has joined #oooq | 15:03 | |
| chkumar|rover | rlandy|ruck: see ya tomorrow | 15:08 |
| *** chkumar|rover is now known as raukadah | 15:08 | |
| rlandy|ruck | k - thanks | 15:08 |
| weshay | rlandy|ruck can you open a bug on that we lost our footer in upstream logs | 15:36 |
| rlandy|ruck | weshay: ack | 15:37 |
| weshay | rlandy|ruck please set "alert" | 15:37 |
| weshay | zbr 2019-08-20 13:16:59 | "ModuleNotFoundError: No module named 'pyngus'", | 15:58 |
| weshay | http://logs.rdoproject.org/60/21860/3/check/tripleo-ci-rhel-8-scenario003-standalone-rdo/aa120a6/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz | 15:59 |
| zbr | sshnaidm: are you aware of baildah failure on https://review.opendev.org/#/c/676497/ ? | 15:59 |
| *** udesale has joined #oooq | 16:00 | |
| zbr | weshay: no f.. idea what pygus was, i had to google... yet another "mainstream" project: https://github.com/kgiusti/pyngus | 16:01 |
| zbr | it has even less stars than my selinux joke | 16:01 |
| zbr | weshay: joke aside, this seems like something else we need to package | 16:04 |
| weshay | zbr ya.. we're missing a dep somewhere | 16:04 |
| weshay | zbr may need to compare to an osp log | 16:04 |
| weshay | which we have | 16:04 |
| weshay | zbr it's not in 15 https://sf.hosted.upshift.rdu2.redhat.com/logs/63/178463/6/check/tripleo-ci-rhel-8-standalone-rhos-15/d9d8a83/logs/undercloud/var/log/extra/rpm-list.txt.gz | 16:05 |
| weshay | zbr perhaps it's a new train/master requirement we need uploaded to the rhui | 16:06 |
| zbr | weshay: is not clear to me who needs to do this. it may be only one of many others. | 16:08 |
| weshay | zbr, yes.. it may be.. I suspect the other scenarios will get further on one more try w/ the right depends on that chandan pointed out | 16:09 |
| weshay | don't think those two patches landed | 16:09 |
| *** sanjayu__ has joined #oooq | 16:10 | |
| *** sanjayu_ has quit IRC | 16:13 | |
| *** jpena is now known as jpena|off | 16:14 | |
| rlandy|ruck | weshay: sorry - back - can you link for "weshay> rlandy|ruck can you open a bug on that we lost our footer in upstream logs" | 16:15 |
| rlandy|ruck | so I can log bug | 16:15 |
| weshay | rlandy|ruck ya.. perhaps infra already knows.. that the footers are gone and they have a plan.. maybe they dont | 16:16 |
| *** Vorrtex has joined #oooq | 16:18 | |
| weshay | zbr does.. https://review.rdoproject.org/r/#/c/21860/ have https://review.opendev.org/#/c/676497/ and https://review.opendev.org/#/c/676474/ in the dep tree? | 16:23 |
| weshay | rlandy|ruck not sure how many issues you are fire fighting atm | 16:25 |
| weshay | rlandy|ruck we could use a standalone rhos-16 job to help debug what zbr is hitting | 16:25 |
| rlandy|ruck | weshay: two - give me 10 minutes | 16:25 |
| weshay | we have some missing deps | 16:25 |
| rlandy|ruck | weshay: np - will get to it this afternoon | 16:26 |
| weshay | thanks | 16:26 |
| weshay | imho.. we would only ever want / need .. n and n-1 | 16:27 |
| rlandy|ruck | so get rid of 14? | 16:31 |
| zbr | sshnaidm: weshay : another abandon/restore on https://review.opendev.org/#/c/676497/ ? infra is already asking about that bug. | 16:32 |
| weshay | zbr context? | 16:32 |
| weshay | that bug | 16:32 |
| raukadah | weshay: rlandy|ruck https://github.com/rdo-packages/oslo-messaging-distgit/blob/rpm-master/python-oslo-messaging.spec#L206 | 16:33 |
| weshay | sshnaidm correct me if I'm wrong.. | 16:33 |
| zbr | weshay: https://zuul.opendev.org/t/openstack/build/0d3a2390eb574917bd74dfc31a88b2bd/log/job-output.txt#1729 | 16:33 |
| weshay | sshnaidm https://review.rdoproject.org/r/#/c/21860/ needs https://review.opendev.org/#/c/676497/ https://review.opendev.org/#/c/676474/ | 16:34 |
| weshay | zbr me looks | 16:34 |
| raukadah | weshay: rlandy|ruck also in deps https://github.com/redhat-openstack/rdoinfo/blob/31d10438d789aaa547ded4ee750a850c2e3348b6/buildsys-tags/cloud7-openstack-common-testing.yml#L991 | 16:34 |
| weshay | zbr I saw another patch to fix that error | 16:34 |
| * weshay is slightly out of the loop | 16:35 | |
| * weshay looks | 16:35 | |
| zbr | weshay: do not ask me why there is no LP bug on 'featureset' is undefined on that. | 16:35 |
| zbr | i only enconered it as a "user". | 16:35 |
| sshnaidm | zbr, if they fix the proxy, it will be merged.. win/win | 16:35 |
| weshay | sshnaidm we're working it in #tripleo | 16:35 |
| sshnaidm | zbr, buildah error seems not related to that patch | 16:36 |
| zbr | sshnaidm: i know, it worked in check. | 16:36 |
| sshnaidm | zbr, I think I saw something similar, rlandy|ruck do we have a bug about buildah errors in this cycle..? | 16:38 |
| rlandy|ruck | sshnaidm: no alex just mentioned it | 16:38 |
| rlandy|ruck | oggng that after I complete current bug | 16:38 |
| rlandy|ruck | logging | 16:39 |
| rlandy|ruck | <mwhahaha> rlandy|ruck: fyi the buildah error log script doesn't work, https://openstack.fortnebula.com:13808/v1/AUTH_e8fd161dc34c421a979a9e6421f823e9/logs_97/676497/6/check/tripleo-build-containers-centos-7-buildah/fb1714a/logs/containers-build-errors.log.txt.gz | 16:39 |
| rlandy|ruck | sshnaidm: ^^ | 16:39 |
| rlandy|ruck | if that is what you mean | 16:39 |
| rlandy|ruck | on my list | 16:39 |
| sshnaidm | rlandy|ruck, ack, I saw similar errors before, when buildah fails, wasn't sure if we have a bug about it | 16:41 |
| rlandy|ruck | not yet | 16:41 |
| rlandy|ruck | to do | 16:41 |
| sshnaidm | it shows Built...................... 135 Expected................... 135 but failed | 16:41 |
| rlandy|ruck | weshay: https://bugs.launchpad.net/tripleo/+bug/1840818 - does this capture what you are after wrt upstream logs? | 16:44 |
| openstack | Launchpad bug 1840818 in tripleo "Headers and footers no longer appear in upstream logss" [Undecided,New] | 16:44 |
| weshay | rlandy|ruck I think it's just the footer | 16:45 |
| weshay | meh | 16:45 |
| rlandy|ruck | weshay: footers are missing from the logs dir https://105ce45b0fa3ad9a1033-fc6b0fdbb44c8a933c3daf9bbf32644a.ssl.cf1.rackcdn.com/676439/4/check/tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades/cd02ee1/logs/ | 16:47 |
| rlandy|ruck | that we can fix I think | 16:47 |
| weshay | k | 16:48 |
| rlandy|ruck | looking at that now | 16:49 |
| *** derekh has quit IRC | 16:59 | |
| rlandy|ruck | ha | 17:00 |
| rlandy|ruck | https://opendev.org/openstack/tripleo-ci/src/branch/master/docs/tripleo-quickstart-logs.html | 17:00 |
| *** udesale has quit IRC | 17:04 | |
| *** tesseract has quit IRC | 17:10 | |
| *** ykarel|away has quit IRC | 17:18 | |
| *** kopecmartin is now known as kopecmartin|off | 17:19 | |
| *** ykarel|away has joined #oooq | 17:22 | |
| weshay | WOOT http://logs.rdoproject.org/49/21649/39/check/periodic-tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001-master/547cbde/ | 17:32 |
| raukadah | weshay: we fixed the tempest issue from samsung tv blog post | 17:40 |
| weshay | ? | 17:41 |
| weshay | there is a joke there that I'm not getting | 17:41 |
| raukadah | weshay: if you do google search, related file not found issue, it will brought home automation issue that point to samsung tv running app blog | 17:41 |
| weshay | heh | 17:42 |
| rlandy|ruck | weshay: zbr: https://bugs.launchpad.net/tripleo/+bug/1840828 | 17:42 |
| openstack | Launchpad bug 1840828 in tripleo "standalone scenario 3 on RedHat 8 - python3-pyngus (used for amqp1) is missing" [High,Triaged] | 17:42 |
| *** ykarel|away has quit IRC | 17:42 | |
| raukadah | i also learned about subprocess.run from that | 17:43 |
| raukadah | which is py36 | 17:43 |
| *** amoralej is now known as amoralej|off | 17:43 | |
| raukadah | *is in | 17:43 |
| rlandy|ruck | ^^ where do want this assigned? | 17:43 |
| *** sshnaidm is now known as sshnaidm|afk | 17:44 | |
| raukadah | rlandy|ruck: as per Alex, we test both rabbitmq and amqp, we have should have one job for ampq, we need to find out first, then switch to assign this bug to DF i think | 17:44 |
| *** tesseract has joined #oooq | 17:45 | |
| raukadah | but any case we will be getting the new deps in rhel8 trunk master deps | 17:45 |
| *** sanjayu__ has quit IRC | 17:46 | |
| weshay | nice | 17:46 |
| weshay | zbr see.. not so hard :) | 17:46 |
| rlandy|ruck | zbr: you can edit/comment on https://bugs.launchpad.net/tripleo/+bug/1840828 | 17:48 |
| openstack | Launchpad bug 1840828 in tripleo "standalone scenario 3 on RedHat 8 - python3-pyngus (used for amqp1) is missing" [High,Triaged] | 17:48 |
| rlandy|ruck | going to set up the rhos-16 jobs now so yo can compare | 17:48 |
| rlandy|ruck | give me half an hour | 17:48 |
| raukadah | weshay: rlandy|ruck can we this be a part of https://opendev.org/openstack/tripleo-ci/src/branch/master/docs/tripleo-quickstart-logs.html of collect-logs itself? | 17:51 |
| raukadah | as OSA, will be using it they can make it custom | 17:51 |
| raukadah | using jinja | 17:51 |
| raukadah | just an idea | 17:51 |
| weshay | raukadah have they confirmed they are going to pick it up and use it? | 17:51 |
| raukadah | arxcruz: ^^ can tell | 17:51 |
| weshay | raukadah that last bit is done outside of collect logs | 17:52 |
| weshay | has to be done in infra | 17:52 |
| weshay | but they can follow the same method we use | 17:52 |
| weshay | we have it doc'd some where | 17:53 |
| raukadah | weshay: http://eavesdrop.openstack.org/meetings/openstack_ansible_meeting/2019/openstack_ansible_meeting.2019-08-20-16.14.log.html | 17:53 |
| weshay | I could add how to to do it to the footer itself | 17:53 |
| * weshay looks | 17:53 | |
| weshay | nice | 17:54 |
| weshay | arxcruz++ | 17:54 |
| arxcruz | not my fault! | 17:55 |
| arxcruz | what? :) | 17:55 |
| rlandy|ruck | raukadah: weshay: only upstream | 17:57 |
| rlandy|ruck | not in rdocloud etc. | 17:57 |
| weshay | rlandy|ruck ? | 17:58 |
| rlandy|ruck | the footer | 17:58 |
| rlandy|ruck | https://opendev.org/openstack/tripleo-ci/src/branch/master/docs/tripleo-quickstart-logs.html | 17:58 |
| weshay | aye | 17:59 |
| rlandy|ruck | we don't apply it | 17:59 |
| * rlandy|ruck goes back to rhos-16 | 17:59 | |
| weshay | I need more words | 17:59 |
| weshay | but it's probably not a need | 18:00 |
| rlandy|ruck | raukadah: can we merge https://review.opendev.org/#/c/676439/ and close the related card? | 18:05 |
| rlandy|ruck | http://logs.rdoproject.org/39/676439/4/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001/70501dc/ failed though | 18:05 |
| raukadah | rlandy|ruck: /me does not have +w rights, feel free to merge it | 18:05 |
| raukadah | it pass in last run | 18:06 |
| raukadah | rlandy|ruck: it is an overcloud deploy failure | 18:06 |
| rlandy|ruck | yeah - ok | 18:06 |
| rlandy|ruck | will w+ | 18:06 |
| rlandy|ruck | tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001 SUCCESS in 3h 09m 59s | 18:07 |
| rlandy|ruck | updated card - thanks | 18:08 |
| raukadah | rlandy|ruck: anything to do with this https://trello.com/c/dq5ntFjS/1084-cixlp1840763tripleociproa-queensfs037-upgrade-jobs-are-failiing-while-doing-overcloud-minor-update-with-ssh-error | 18:09 |
| rlandy|ruck | raukadah: I am editing the card now | 18:10 |
| rlandy|ruck | saying it's not blocking promotions | 18:10 |
| rlandy|ruck | raukadah: weshay: I moved the card to failing jobs | 18:11 |
| rlandy|ruck | not critical outage | 18:12 |
| rlandy|ruck | bur upgrades team should be aware anyways | 18:12 |
| rlandy|ruck | will bring it up at meeting tomorrow | 18:12 |
| raukadah | rlandy|ruck: feel free to add ci tag to all the bugs filed by ruck/rover so that we can see in cockpit | 18:12 |
| rlandy|ruck | raukadah: iiuc, we only track bugs there we are responsible for | 18:13 |
| rlandy|ruck | ie: the ci tag only goes on bugs we are required to fix | 18:13 |
| raukadah | weshay: ^^ | 18:13 |
| raukadah | it will be easier for closing down as it appears in cockpit | 18:14 |
| rlandy|ruck | upgrades bug | 18:14 |
| rlandy|ruck | weshay: zbr: https://code.engineering.redhat.com/gerrit/179007 Add rhos-16 release file | 18:19 |
| rlandy|ruck | test job coming up | 18:19 |
| rlandy|ruck | does that have to be scenario-003 | 18:21 |
| rlandy|ruck | or just a regular standalone job? | 18:21 |
| rlandy|ruck | zbr: ^^? | 18:21 |
| weshay | raukadah the alert tag | 18:22 |
| raukadah | rlandy|ruck: qdrouterd-container-puppet.yaml needs to be used | 18:22 |
| weshay | although maybe I can check to see if CI is still overused | 18:22 |
| weshay | and not helpful | 18:22 |
| rlandy|ruck | regular standlaone first - let's get rhos-16 bits in order - then scenraio003 | 18:23 |
| weshay | hrm.. not so bad | 18:24 |
| raukadah | rlandy|ruck: ack! | 18:24 |
| weshay | rlandy|ruck !!! http://dashboard-ci.tripleo.org/d/YRJtmtNWk/cockpit?orgId=1&fullscreen&panelId=231 | 18:34 |
| weshay | check it out | 18:34 |
| rlandy|ruck | oh dear | 18:35 |
| rlandy|ruck | did rdocloud go away? | 18:35 |
| weshay | lolz | 18:35 |
| weshay | no Allan pushed on it again | 18:35 |
| rlandy|ruck | or did someone actually answer our request? | 18:35 |
| rlandy|ruck | oh how sweet | 18:35 |
| rlandy|ruck | we should send him flowers | 18:36 |
| rlandy|ruck | weshay: zbr: https://code.engineering.redhat.com/gerrit/179009 Add rhos-16 standalone job | 18:37 |
| rlandy|ruck | k - let's let that run - adding scenario003 job | 18:37 |
| sshnaidm|afk | rlandy|ruck, pushed fixes to sova, we'll see later results | 19:14 |
| rlandy|ruck | sshnaidm|afk++ | 19:14 |
| rlandy|ruck | thanks | 19:14 |
| *** aakarsh has quit IRC | 19:53 | |
| rlandy|ruck | standalone_container_cli: docker | 19:56 |
| rlandy|ruck | zbr: weshay: ^^ is that correct for rhel8? | 19:57 |
| rlandy|ruck | https://review.rdoproject.org/r/#/c/21787/9/zuul.d/standalone-jobs.yaml | 19:57 |
| * weshay looks | 19:58 | |
| rlandy|ruck | weshay: zbr: https://code.engineering.redhat.com/gerrit/#/c/179009/ | 20:01 |
| rlandy|ruck | ^^ include scenario003 | 20:01 |
| weshay | I not familiar enough yet w/ anchors to be a good judge | 20:01 |
| rlandy|ruck | inheritance was interesting | 20:02 |
| rlandy|ruck | weshay: there were some deps issues already ... | 20:02 |
| rlandy|ruck | https://sf.hosted.upshift.rdu2.redhat.com/logs/09/179009/2/check/tripleo-ci-rhel-8-standalone-rhos-16/6040c49/logs/undercloud/var/log/tripleo-container-image-prepare.log.txt.gz | 20:03 |
| rlandy|ruck | idk - how far we want this used | 20:03 |
| rlandy|ruck | https://bugs.launchpad.net/tripleo/+bug/1840828 - not sure who takes that assignment | 20:03 |
| openstack | Launchpad bug 1840828 in tripleo "standalone scenario 3 on RedHat 8 - python3-pyngus (used for amqp1) is missing" [High,Triaged] | 20:03 |
| * rlandy|ruck needs to address buildah jobs now | 20:04 | |
| weshay | huh.. osp-16 weird | 20:06 |
| weshay | I don't see python3-pyngus | 20:06 |
| weshay | might be a new dep | 20:06 |
| rlandy|ruck | see where it's added | 20:06 |
| rlandy|ruck | links in the bug | 20:06 |
| weshay | ya.. added 6 months ago | 20:08 |
| weshay | so we have two seperate issues atm.. python3-pyngus and the osp-16 standalone | 20:09 |
| rlandy|ruck | correct | 20:09 |
| weshay | and that was a vanilla deployment of standaone not scen03 right? | 20:09 |
| rlandy|ruck | just want to log the buildah bug | 20:09 |
| rlandy|ruck | it was a vanilla standalone | 20:10 |
| rlandy|ruck | scen003 just added | 20:10 |
| rlandy|ruck | chceking if it ran | 20:10 |
| rlandy|ruck | config issue - resubmitted | 20:11 |
| rlandy|ruck | job just started | 20:12 |
| weshay | rlandy|ruck maybe we can ask jjoyce tomorrow | 20:14 |
| rlandy|ruck | https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/stream/297a4718db95497d806f8acc0b468833?logfile=console.log | 20:14 |
| rlandy|ruck | about osp-16? | 20:15 |
| rlandy|ruck | python3-pyngus? | 20:15 |
| weshay | https://brewweb.engineering.redhat.com/brew/search?match=glob&type=package&terms=python3-pyngus | 20:15 |
| weshay | that's interesting | 20:15 |
| rlandy|ruck | sure - the scenario003 run is still in progress though | 20:15 |
| weshay | maybe it's not supposed to be installed as alfreado was saying | 20:15 |
| weshay | I don't think that rpm is built .. period | 20:16 |
| weshay | hrm.. looks like it's in fedora | 20:17 |
| rlandy|ruck | where did they come up with this thing? | 20:17 |
| jjoyce | What's up? | 20:17 |
| rlandy|ruck | all of a sudden | 20:17 |
| rlandy|ruck | jjoyce: hey there | 20:17 |
| weshay | jjoyce looking for a missing dep.. https://bugs.launchpad.net/tripleo/+bug/1840828 | 20:18 |
| openstack | Launchpad bug 1840828 in tripleo "standalone scenario 3 on RedHat 8 - python3-pyngus (used for amqp1) is missing" [High,Triaged] | 20:18 |
| weshay | python3-pyngus | 20:18 |
| jjoyce | For 16 or 15? | 20:18 |
| rlandy|ruck | showed up in master | 20:18 |
| weshay | for rhel8.. it could be something is misconfigured btw | 20:18 |
| jjoyce | Or both | 20:18 |
| rlandy|ruck | also checking rhos-16 | 20:18 |
| weshay | jjoyce train | 20:18 |
| weshay | so 16 | 20:18 |
| weshay | jjoyce https://brewweb.engineering.redhat.com/brew/search?match=glob&type=package&terms=python3-pyngus | 20:18 |
| rlandy|ruck | got a rhos-16 build going now | 20:19 |
| weshay | I see it in fedora build system only atm | 20:19 |
| jjoyce | https://brewweb.engineering.redhat.com/brew/packageinfo?packageID=49199 | 20:19 |
| jjoyce | But looks super old | 20:19 |
| weshay | ya | 20:19 |
| rlandy|ruck | no rhos-16 | 20:20 |
| rlandy|ruck | all 7 | 20:20 |
| weshay | I suspect we're hitting it because something is misconfigured | 20:20 |
| weshay | and it shouldn't be there | 20:20 |
| weshay | it was added 6 months ago in src | 20:20 |
| jjoyce | Yeah we would of seen it by now if it were missing. | 20:20 |
| weshay | well.. the service that reqs it | 20:20 |
| weshay | aye | 20:20 |
| weshay | https://github.com/openstack/tripleo-heat-templates/blame/master/ci/environments/scenario003-standalone.yaml#L15 | 20:21 |
| rlandy|ruck | we didn;t know either until we ran scen003 | 20:21 |
| weshay | rlandy|ruck that is the only scenario where it exists though http://codesearch.openstack.org/?q=rpc-qdrouterd-container-puppet&i=nope&files=&repos= | 20:22 |
| rlandy|ruck | https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/stream/297a4718db95497d806f8acc0b468833?logfile=console.log - ha - actually still running | 20:22 |
| rlandy|ruck | total job hack | 20:23 |
| weshay | rlandy|ruck compare that to rabbitmq-messaging | 20:23 |
| weshay | rlandy|ruck I bet it should be 54 OS::TripleO::Services::OsloMessagingRpc: ../deployment/rabbitmq/rabbitmq-messaging-rpc-container-puppet.yaml | 20:23 |
| weshay | rlandy|ruck let's try a patch to scen003 w/ that | 20:24 |
| jjoyce | We don't have it in any of the composes from what I can tell, so it must show up as a require in the spec. | 20:24 |
| weshay | it would replace qdrouterd w/ rabbit | 20:24 |
| weshay | jjoyce aye... ya doesn't seem like a packaging issue | 20:24 |
| rlandy|ruck | config | 20:24 |
| weshay | that's for the assist! | 20:24 |
| weshay | thanks | 20:24 |
| weshay | rlandy|ruck shall we try? | 20:25 |
| rlandy|ruck | weshay: ack | 20:25 |
| rlandy|ruck | https://review.rdoproject.org/r/#/c/21787/ is rdo-jobs | 20:25 |
| rlandy|ruck | so we can test on review | 20:25 |
| jjoyce | weshay: Let me know if you need me to dig more. | 20:26 |
| weshay | rlandy|ruck k | 20:26 |
| weshay | jjoyce thanks man | 20:26 |
| rlandy|ruck | weshay: k - you putting in the tht patch or am I? | 20:26 |
| jjoyce | any time | 20:27 |
| * rlandy|ruck does it | 20:27 | |
| weshay | meh.. I'm just a pesky manager.. | 20:27 |
| jjoyce | We should start a manager coding support group | 20:27 |
| weshay | jjoyce totally... TODAY.... WE INTRODUCE "THE BUBBLE SORT" | 20:28 |
| weshay | rlandy|ruck k.. ya http://codesearch.openstack.org/?q=OS%3A%3ATripleO%3A%3AServices%3A%3AOsloMessagingRpc&i=nope&files=&repos= | 20:28 |
| weshay | that pretty much confirms it | 20:28 |
| *** Goneri has quit IRC | 20:28 | |
| weshay | OsloMessagingRpc is configured to use rabbit everywhere else | 20:28 |
| jjoyce | HAHAHAHA | 20:28 |
| weshay | rlandy|ruck ya.. think this will work fine | 20:29 |
| rlandy|ruck | sec - trying to paste the right line :) | 20:29 |
| weshay | rlandy|ruck I wonder centos was carrying a python-pyngus package | 20:29 |
| weshay | probably | 20:29 |
| rlandy|ruck | weshay: https://review.opendev.org/677562 Replace rpc-qdrouterd with rabbitmq-messaging-rpc | 20:34 |
| rlandy|ruck | ok - let's try this dep with scenario003 on rhel | 20:35 |
| weshay | rlandy|ruck looks right to me | 20:36 |
| weshay | rlandy|ruck ya.. confirmed https://cbs.centos.org/koji/buildinfo?buildID=26196 | 20:37 |
| rlandy|ruck | https://review.rdoproject.org/r/#/c/21787/ | 20:37 |
| rlandy|ruck | ok - that runs again | 20:37 |
| weshay | rlandy|ruck that is not running scen03 | 20:38 |
| rlandy|ruck | ugh - it doesn't run | 20:38 |
| rlandy|ruck | hold on | 20:38 |
| rlandy|ruck | adding it | 20:38 |
| weshay | https://review.rdoproject.org/r/#/c/21861/ | 20:39 |
| weshay | rlandy|ruck ^ | 20:39 |
| rlandy|ruck | ah - saves me recreating a job | 20:40 |
| rlandy|ruck | thanks | 20:40 |
| weshay | rlandy|ruck I think we can just recheck 21861 | 20:40 |
| rlandy|ruck | need to rebase it | 20:41 |
| rlandy|ruck | there is no rebase button | 20:42 |
| rlandy|ruck | have to do it via reviews | 20:42 |
| rlandy|ruck | weshay: k - here we go | 20:57 |
| rlandy|ruck | with all that extra nova stuff around :( | 20:58 |
| weshay | cool | 20:59 |
| weshay | thanks | 20:59 |
| rlandy|ruck | ugh - that change does not kick scenario003 on centos7 | 21:05 |
| rlandy|ruck | how are these test set up??? | 21:05 |
| rlandy|ruck | weshay: ... maybe we need those test as they were .... <mwhahaha> we have a job to test amqp | 21:07 |
| rlandy|ruck | <raukadah> amoralej: scenario1-4 rhel8 based jobs are running in periodic right now not in prod | 21:07 |
| rlandy|ruck | <mwhahaha> the rest rabbitmq | 21:07 |
| weshay | rlandy|ruck well tru for centos | 21:07 |
| weshay | not so tru for rhel8 | 21:07 |
| rlandy|ruck | https://review.opendev.org/#/c/677562/ can't merge | 21:08 |
| weshay | let's see if it works first.. then sort out what we need to cover.. if osp doesn't ship w/ support for amqp.. then I think we may be able to | 21:08 |
| rlandy|ruck | k - I left it w-1 | 21:08 |
| weshay | rlandy|ruck when centos8 comes in that may be it for train + amqp | 21:09 |
| rlandy|ruck | osp is a hole other issue - leaving that for now | 21:10 |
| rlandy|ruck | at least redhat8_master is promoting now | 21:11 |
| rlandy|ruck | one thing | 21:11 |
| weshay | rlandy|ruck really? | 21:17 |
| weshay | woot | 21:18 |
| weshay | 2019-08-20 15:42:02,717 23311 INFO promoter SUCCESS promoting tripleo-ci-testing as current-tripleo ({'timestamp': 1566303568, 'distro_hash': 'd0e11ceb3684ffd1daa672e98c5a82697f730733', 'promote_name': 'tripleo-ci-testing', 'user': 'review_rdoproject_org', 'repo_url': 'https://trunk.rdoproject.org/rhel8-master/a4/47/a447a10b12efed2e989ed61de5d0 | 21:18 |
| weshay | d1562a2919ea_d0e11ceb', 'full_hash': 'a447a10b12efed2e989ed61de5d0d1562a2919ea_d0e11ceb', 'repo_hash': 'a447a10b12efed2e989ed61de5d0d1562a2919ea_d0e11ceb', 'commit_hash': 'a447a10b12efed2e989ed61de5d0d1562a2919ea'}) | 21:18 |
| *** aakarsh has joined #oooq | 21:22 | |
| rlandy|ruck | yep | 21:24 |
| rlandy|ruck | http://zuul.openstack.org/builds?job_name=tripleo-build-containers-centos-7-buildah | 21:29 |
| rlandy|ruck | buildah failure is not consistent | 21:29 |
| rlandy|ruck | weshay: k - so abandon https://review.opendev.org/#/c/677562/? | 21:31 |
| *** tesseract has quit IRC | 21:32 | |
| rlandy|ruck | just move scen003 to non-voting? | 21:32 |
| weshay | rlandy|ruck well technically all 1-4 should e nv until stable | 21:32 |
| weshay | sounds like 3 will stay that way on rhel | 21:32 |
| rlandy|ruck | so abandon patch? | 21:32 |
| rlandy|ruck | weshay: ^^? | 21:33 |
| weshay | meh.. I guess .. in theory centos should follow rhel.. but that doesn't always seem to be the case .. pisses me off | 21:34 |
| weshay | I don't think we'll get this package in centos8 | 21:34 |
| rlandy|ruck | I don;t want alex to have a fit at this patch | 21:34 |
| rlandy|ruck | rather get rid of it | 21:35 |
| rlandy|ruck | weshay: one more for you ... buildah errors - sporadic http://zuul.openstack.org/builds?job_name=tripleo-build-containers-centos-7-buildah | 21:42 |
| rlandy|ruck | alex claims the buildah error log script is not working | 21:42 |
| rlandy|ruck | https://storage.bhs1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/logs_19/674919/24/check/tripleo-build-containers-centos-7-buildah/75e6db3/logs/containers-build-errors.log.txt.gz | 21:43 |
| weshay | rlandy|ruck I think the version of buildah in centos-7 is old | 21:43 |
| rlandy|ruck | can't say I know what its' supposed to do | 21:43 |
| weshay | I wonder if we can convince Emilien to drop that job.. since we have the rhel8 version now | 21:44 |
| weshay | rlandy|ruck can you put up a change to mark it non-voting, note we have an active rhel8 version and put Emilien on the review please? | 21:44 |
| rlandy|ruck | yep | 21:45 |
| weshay | thanks | 21:45 |
| rlandy|ruck | checking if the log is working in rhel 8 | 21:49 |
| rlandy|ruck | weshay: http://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-rhel-8-master-containers-build-push/f740eae/logs/containers-build-errors.log.txt.gz | 21:56 |
| rlandy|ruck | ^^ not sure it works in rhel-8 | 21:56 |
| rlandy|ruck | either | 21:56 |
| rlandy|ruck | http://logs.rdoproject.org/00/21800/4/check/periodic-tripleo-rhel-8-master-containers-build-push/2d613ea/logs/containers-build-errors.log.txt.gz | 21:57 |
| rlandy|ruck | maybe that works | 21:57 |
| weshay | I'll be back later.. have to keep the kids in line | 22:07 |
| *** zbr has quit IRC | 22:14 | |
| *** zbr has joined #oooq | 22:15 | |
| *** ksambor has quit IRC | 22:43 | |
| *** ksambor has joined #oooq | 22:44 | |
| *** PagliaccisCloud has quit IRC | 22:55 | |
| *** zbr has quit IRC | 23:22 | |
| *** zbr has joined #oooq | 23:23 | |
| *** Vorrtex has quit IRC | 23:40 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!