Friday, 2020-02-28

*** tosky has quit IRC00:18
*** rlandy has quit IRC00:49
*** beagles is now known as beagles_pto01:55
*** saneax has quit IRC02:24
*** apetrich has quit IRC03:11
*** apetrich has joined #oooq03:15
*** ykarel|away is now known as ykarel03:57
ykarelnice ^^ \o/04:09
*** pojadhav|away is now known as pojadhav04:41
*** udesale has joined #oooq05:13
*** skramaja has joined #oooq05:19
*** saneax has joined #oooq05:37
*** ratailor has joined #oooq06:07
*** jfrancoa has joined #oooq06:51
*** ykarel is now known as ykarel|afk06:55
*** marios|ruck has joined #oooq06:57
*** dpawlik has joined #oooq07:09
*** ykarel|afk is now known as ykarel07:30
*** holser__ has joined #oooq07:58
*** tesseract has joined #oooq08:05
marios|ruckpanda: liar liar bum's on fire https://review.rdoproject.org/r/#/c/24699/19/ci-scripts/dlrnapi_promoter/dlrn_interface.py08:14
*** tosky has joined #oooq08:21
*** holser__ has quit IRC08:28
*** ykarel is now known as ykarel|lunch08:37
*** jpena|off is now known as jpena08:38
*** saneax has quit IRC08:45
*** skramaja has quit IRC08:45
*** skramaja has joined #oooq08:50
*** amoralej|off is now known as amoralej08:54
*** saneax has joined #oooq08:59
*** holser has joined #oooq09:03
pandamarios|ruck: ?09:07
*** ykarel|lunch is now known as ykarel09:08
pandamarios|ruck: ah09:09
pandalol09:09
marios|ruck:)09:11
marios|rucki mean ignoring would be better09:11
marios|ruck(you can still feel free to panda but don't think getting those votes back will be a problem)09:11
*** saneax has quit IRC09:11
pandamarios|ruck: sometimes I hae to decide, do it and forget to report it, or report it and forget to do it, my mind can't process two things at a time09:12
*** saneax has joined #oooq09:13
*** saneax has quit IRC09:13
pandamarios|ruck: now that I think09:13
panda(previously I didnt')09:14
pandamarios|ruck: it's correct to have the : at the end09:14
pandabecause what follows is the list of successful jobs09:14
*** saneax has joined #oooq09:14
pandamarios|ruck: closing WONTFIX09:14
marios|ruckpanda: well "commit: 360d335e94246d7095672c5aa92b59afa380a059, distro: 9e5988125e88f803ba20743be7aa99079dd275f2"09:16
marios|ruck11:14 < panda> because what follows is the list of successful jobs09:16
*** saneax has quit IRC09:16
marios|ruckpanda: orly09:16
pandamarios|ruck: :P09:16
marios|ruckpanda: :D fine09:16
*** saneax has joined #oooq09:23
ykarelmarios|ruck, hi09:26
ykarelmarios|ruck, u planning to try virtualenv fixes today?09:26
ykarelif yes let me know before u post recheck on review09:26
ykareli think i got the issue with jenkins job which never passed09:26
ykareli would like to try something09:27
ykareli mean https://ci.centos.org/job/tripleo-quickstart-gate-master-delorean-quick-basic/ job09:27
ykarelweshay|ruck, fyi ^^09:32
marios|ruckykarel: yeah was gonna switch in a bit and check that some more and look at your comment. that error from debug is weird09:32
marios|rucki recall zbr mentioning something similar do you remember ""virtualenv' is a package > and cannot be directly executed"09:33
marios|ruckykarel: ok are you updating the review? https://review.opendev.org/#/c/708791/5/roles/build-test-packages/tasks/main.yml09:33
pandapython -m virtualenv09:34
ykarelmarios|ruck, /me not updating, u can update09:34
pandafor python309:34
ykareljust before posting recheck on other review, let me know09:34
marios|ruckykarel: ok09:34
zbrmarios|ruck: if you see that error, you have an ancient virtualenv, as in one that eons old.09:34
marios|ruckzbr: ack i think that is what ykarel was saying ... and i think we have mix of pip and package versions of things. might be cleaner to just remove them and reinstall in that task ykarel09:40
zbrmarios|ruck: be careful that is a mined field09:40
*** derekh has joined #oooq09:40
zbrone you endup touching system python with pip, you are on YOLO land.09:41
marios|rucklol09:41
zbrpip is ok for --user or virtualenvs, not problem with that. but not system.09:41
*** skramaja_ has joined #oooq09:41
ykarelmarios|ruck, me not saying to remove and reinstall09:41
ykareljust install with --user09:41
*** skramaja has quit IRC09:42
ykarelremoving might have some other side effects so good to avoid09:42
marios|ruckykarel: ack09:42
amoralejmarios|ruck, fyi https://lists.rdoproject.org/pipermail/dev/2020-February/009311.html10:07
amoraleji expect i will not break anything but let me know if you notice something10:08
marios|ruckok thanks amoralej10:08
*** tesseract has quit IRC10:11
*** tesseract has joined #oooq10:12
*** tesseract has quit IRC10:16
*** tesseract has joined #oooq10:16
*** ykarel is now known as ykarel|afk10:25
marios|ruckykarel|afk: o/ want to update https://review.opendev.org/#/c/708791/5/roles/build-test-packages/tasks/main.yml in a bit re 11:34 < ykarel> just before posting recheck on other review, let me know10:30
marios|ruckykarel|afk: let me know when you're back i can hold for a bit10:30
*** skramaja_ has quit IRC10:42
marios|ruckykarel|afk: i'd like to get it going soon recheck at https://review.opendev.org/#/c/708264/ incoming10:50
marios|ruckykarel|afk: so can run maybe once more today if needed10:50
marios|ruckground control to major tom10:51
zbri need link to one repo where ansible-lint fails to install with six error11:09
zbri do have a potential hotfix and i want to test it11:09
zbrthe bug report at https://bugs.launchpad.net/tripleo/+bug/1865000 links to logs not repos. and i am not able to see the failure on the repos.11:10
openstackLaunchpad bug 1865000 in tripleo "openstack-tox-linters failing with ModuleNotFoundError: No module named 'six'" [Critical,Triaged]11:10
*** ykarel|afk is now known as ykarel11:11
marios|ruckykarel: i just hit recheck few mins ago11:12
ykarelmarios|ruck, ack, zipp downgrade shouldn't be needed now11:13
ykarelmarios|ruck, okk i updated on jenkins side11:13
ykarellet me see if it worked or not11:13
marios|ruckykarel: yeah i added explicit pip virtualenv as discussed lets see11:13
ykarelmarios|ruck, looks like it helped, but failed at other step, will look more11:16
ykarelhttps://ci.centos.org/job/tripleo-quickstart-gate-master-delorean-quick-basic/79/consoleText11:17
marios|ruckykarel: hmm k will check there in a minute too... lets see what happens with queens11:20
*** holser has quit IRC11:29
*** holser has joined #oooq11:30
marios|ruckykarel: ack on recheck but please lets let it run? at least once so we can see q?11:37
ykarelmarios|ruck, recheck shouldn't affect current runs11:38
marios|ruckykarel: hmm well it is stil queued... not sure if that's cos the queue is long... looking11:40
marios|ruckykarel: yeah it says it was in there for 41 mins so looks like you're right11:40
marios|ruckykarel: ack11:41
marios|ruckrecheck away11:41
marios|ruck:)11:41
ykarel:)11:41
marios|ruckfoooood brb11:53
arxcruzraukadah: will we have tempest meeting today ?11:54
pandaweshay|ruck: https://review.opendev.org/710441, someone from upgrades should chack.11:56
*** jfrancoa has quit IRC12:12
*** ratailor has quit IRC12:14
*** saneax has quit IRC12:16
*** saneax has joined #oooq12:17
ykarelmarios|ruck, it failed, due to list [--user --upgrade12:19
ykareli will revise it12:19
marios|ruckykarel: hmm12:19
marios|ruckykarel: looking12:19
marios|ruckykarel: why didn't it like --upgrade12:19
marios|ruckis it -upgrade or sthing12:19
ykarelmarios|ruck, should be string12:19
ykareli think it don't like list12:20
marios|ruckykarel: k let me update that sec12:20
ykarelokk12:20
marios|ruckykarel: i want to see the logs first12:20
ykarelmarios|ruck, https://66183e98f39c24f931a5-6295bbf84e3f5d8ca4e2258728f7c9c2.ssl.cf2.rackcdn.com/708264/11/check/tripleo-ci-centos-7-containers-multinode-queens/2991d02/job-output.txt12:20
marios|ruckykarel: gimme couple mins12:20
ykarelextra_args: '--user --upgrade' should work12:21
marios|ruckyah thats what i'd adding ykarel12:22
ykarelack12:22
marios|ruckykarel: strange it should be able to parse that list but whatever12:22
*** jfrancoa has joined #oooq12:27
*** cgoncalves has quit IRC12:29
*** jpena is now known as jpena|lunch12:33
*** ChanServ sets mode: +o panda12:34
*** panda changes topic to "Docs: https://docs.openstack.org/tripleo-quickstart/latest/ || 28th Feb promoter patch to review https://review.rdoproject.org/r/24694 || next promoter patch in the chain https://review.rdoproject.org/r/24714"12:35
*** panda sets mode: -o panda12:35
marios|ruckykarel: damn same problem12:54
marios|ruckykarel: maybe we need to try the setuptools way12:54
ykarelmarios|ruck, queens failed :(12:55
ykarelmarios|ruck, sorry what setuptools way?12:55
marios|ruckykarel: https://review.opendev.org/#/c/708791/4/roles/build-test-packages/tasks/main.yml@102 but i see it is from yesterday12:56
zbrykarel: marios|ruck i see you both are having the same kind of fun as me, around setuptools....12:56
marios|ruckykarel: will try the contextlib2 thing but we tried that before12:56
marios|ruckykarel: that was the first one we tried12:56
marios|ruckykarel: but maybe we didn't use '--user' lets try it12:57
ykarelmarios|ruck, u tried with become last time?12:57
marios|ruckykarel: can't remember let me find it12:57
marios|ruckhttps://review.opendev.org/#/c/708791/1/roles/build-test-packages/tasks/main.yml12:57
marios|ruckzbr: https://www.youtube.com/watch?v=WpmILPAcRQo12:58
ykarelmarios|ruck, ack this is different ^^12:58
ykarelu tried in venv and the failure there was expected12:58
marios|ruckykarel: ack ok updating12:58
ykarelmarios|ruck, try the same that i commented12:59
zbri am at conf, so i will need to watch it later.12:59
marios|ruckzbr: k pycon?12:59
marios|ruckzbr: its super important make sure you watch it12:59
zbrjust had a talk with john skeet12:59
ykarelmarios|ruck, till then we can check how to handle the 1st way12:59
marios|ruckzbr: (don't waste mobile data on it though)12:59
marios|ruckzbr: the title is "Dirty Dancing - Time of my Life (Final Dance) - High Quality"12:59
marios|ruck;)12:59
*** cgoncalves has joined #oooq12:59
marios|ruckykarel: ack doing12:59
*** rlandy has joined #oooq12:59
weshay|ruckah shit.. I missed the tempest call.. arxcruz, soniya29 kopecmartin sorry I thought it was at 6am.. :(13:00
rlandyweshay|ruck: arxcruz: panda: canceling today's centos 8 sync13:01
rlandyrfolco is out and so is chandan13:01
weshay|ruckhttps://meet.google.com/oiv-geho-mai?authuser=1 ?13:01
arxcruzweshay|ruck: it was me and kopecmartin we sync some things we are doing, and done...13:01
weshay|ruckmarios|ruck, how are we love?13:02
* weshay|ruck watched a movie w/ british actors last night13:02
rlandywow - you two got close13:02
weshay|ruckwe are green13:02
weshay|ruckgate green13:02
marios|ruckweshay|ruck: alright pet13:02
weshay|rucklolz13:02
rlandyhappy dance13:02
marios|ruckweshay|ruck: yeah not too shabby but lets not dwell on that too much13:02
weshay|ruckrlandy, where are you meet?13:03
marios|ruckweshay|ruck: i refreshed multiple times when i saw the promotions on grafana13:03
rlandyweshay|ruck: canceled ...13:03
rlandynobody here13:03
weshay|ruckI am13:03
rlandysorry13:03
rlandyok - we meet13:03
weshay|ruckpanda, is13:03
rlandyhttps://meet.google.com/oiv-geho-mai?authuser=113:03
rlandypanda: ^^13:03
rlandyforget previous announcements13:04
weshay|ruckhttps://review.opendev.org/#/c/710177/6/zuul.d/standalone-jobs.yaml13:05
weshay|ruckpanda, https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_d85/710441/1/check/tripleo-ci-centos-7-containers-multinode-train/d855b0c/logs/emit_releases_file.log13:12
*** udesale_ has joined #oooq13:13
*** udesale has quit IRC13:15
zbrhttps://github.com/ansible/ansible-lint/pull/688 is ready for review (that is fixing six)13:26
zbris big because the issue is big, six was only the tip of the aisberg13:27
zbrin fact packaging was a known issue with PR ready13:27
weshay|ruckpanda, https://hackmd.io/HrQd03c9SxOMtFPFrq50tg13:29
ysandeepweshay|ruck, reminder our 1:113:29
weshay|ruckysandeep, coming13:32
ysandeepweshay|ruck, thanks!13:32
*** udesale_ has quit IRC13:33
*** jpena|lunch is now known as jpena13:38
*** tesseract has quit IRC13:40
*** tesseract has joined #oooq13:42
*** ysandeep is now known as ysandeep|away13:48
marios|ruckooof zuul queues growing again ? :/13:49
marios|ruckzbr: so did you do something for https://bugs.launchpad.net/tripleo/+bug/1865000 ? a lot of green there http://zuul.openstack.org/builds?job_name=openstack-tox-linters13:53
openstackLaunchpad bug 1865000 in tripleo "openstack-tox-linters failing with ModuleNotFoundError: No module named 'six'" [Critical,Triaged]13:53
marios|ruckzbr: and there https://review.rdoproject.org/zuul/builds?job_name=tox-linters13:53
zbrahve you cliked the link? PR is green, waiting for reviews.13:53
zbrtime to ping #ansible-galaxy and #ansible-devel channels to do something about it13:53
marios|ruckzbr: no i have not. but then why is the job green13:53
marios|ruckzbr: wondering if something else fixed it?13:54
zbrthat error appears on certain cases only version mix of setuptools&co13:54
zbrsomething that cannot be controlled by deps13:55
zbrbecause the issue is around the build-env, not the destination env.13:55
zbri also wanted to add zuul jobs for ansible-lint to test it and avoid such issue, see https://github.com/ansible/ansible-lint/pull/65713:57
zbris more than a month old, we need some peer pressure to make them do something about these, ask on #ansible-devel13:58
*** Goneri has joined #oooq14:00
weshay|ruckpanda, can you review and vote on https://review.opendev.org/#/c/710230/14:10
weshay|ruckpanda, rebase required https://review.rdoproject.org/r/#/c/24714/14:13
rlandyykarel: hi ... please review https://review.opendev.org/#/c/710177/ again ...14:14
rlandysee changes to templates/branches14:14
ykarelrlandy, ack looking14:14
marios|ruckyup something is up again the upstream zuul queue time is growing up to 11 hours now14:15
*** saneax has quit IRC14:16
marios|ruckbut our jobs look relatively green don't know14:16
* marios|ruck talking to self yup its the last few days of ruck|rover14:16
weshay|ruckmarios|ruck, lolz14:16
weshay|ruckmarios|ruck, use the cockpit luke.. http://dashboard-ci.tripleo.org/d/cockpit/cockpit?orgId=1&fullscreen&panelId=6114:18
weshay|ruck3 failures, 1 timeout since yesterdays outtage14:18
marios|ruckweshay|ruck: yeah it looks pretty green on the main page too14:19
* marios|ruck relax14:19
weshay|ruck:)14:19
marios|ruckmaybe just the US coming online14:19
marios|ruckcan you please go back to sleep14:19
* weshay|ruck checking the three failed jobs14:19
weshay|ruckUS!!! GO TO SLEEP.. GET REST BEFORE THE CORNAVIRUS KILLS YOUR FAMILY!14:19
marios|ruckyeah :/ is it like that on the news you mean?14:20
rlandyyeah - big news here14:20
rlandythey are disinfecting the subways and everything14:20
marios|ruck:/14:21
weshay|ruck    ERROR:root:Image prepare failed: [Errno 17] File exists14:23
weshay|ruck    Traceback (most recent call last):14:23
weshay|ruck      File "/usr/bin/tripleo-container-image-prepare", line 138, in <module>14:23
weshay|ruck        lock=lock)14:23
weshay|ruck      File "/usr/lib/python2.7/site-packages/tripleo_common/image/kolla_builder.py", line 233, in container_images_prepare_multi14:23
weshay|ruck        uploader.upload()14:23
weshay|ruck      File "/usr/lib/python2.7/site-packages/tripleo_common/image/image_uploader.py", line 276, in upload14:23
weshay|ruck        uploader.run_tasks()14:23
weshay|ruck      File "/usr/lib/python2.7/site-packages/tripleo_common/image/image_uploader.py", line 2349, in run_tasks14:23
weshay|ruck        for result in p.map(upload_task, self.upload_tasks):14:23
weshay|ruck      File "/usr/lib/python2.7/site-packages/concurrent/futures/_base.py", line 605, in result_iterator14:23
weshay|ruck        yield future.result()14:23
weshay|ruck      File "/usr/lib/python2.7/site-packages/concurrent/futures/_base.py", line 422, in result14:23
weshay|ruck        return self.__get_result()14:23
weshay|ruck      File "/usr/lib/python2.7/site-packages/concurrent/futures/_base.py", line 381, in __get_result14:23
weshay|ruck        raise exception_type, self._exception, self._traceback14:23
weshay|ruck    OSError: [Errno 17] File exists14:23
weshay|ruck  stderr_lines: <omitted>14:23
weshay|ruckok.. two jobs failing on similiar issues here14:23
weshay|ruck    INFO:tripleo_common.image.image_uploader:[docker.io/tripleomaster/centos-binary-heat-engine:b5ef03c9c939db551b03e9490edc6981ff582035_76ebc465] Completed modify and upload for image14:23
weshay|ruckdiff containers14:23
weshay|ruck2 gate failures ^14:23
marios|ruckweshay|ruck: ack ok14:24
marios|ruckweshay|ruck: looking & will file LP14:24
weshay|ruckmarios|ruck, poke at that code prior to bug.. why would it fail if FILE EXISTS14:24
marios|ruckweshay|ruck: ack yeah getting logs now14:25
marios|ruckweshay|ruck: where are you seeing that one14:26
marios|ruckweshay|ruck: i see tempest fail at https://2f631210719fff44d0f5-e9cc619bd684627a8607e78f73e48fa0.ssl.cf2.rackcdn.com/709999/1/gate/tripleo-ci-centos-7-standalone/aec767e/job-output.txt14:26
marios|ruckweshay|ruck: or is it TIMED_OUT14:26
marios|ruckme looking at FAILURE14:26
marios|ruckhttp://dashboard-ci.tripleo.org/d/cockpit/cockpit?orgId=1&fullscreen&panelId=6114:27
marios|ruckfrom that list ^14:27
marios|ruckah hold on weshay|ruck i was only looking at the 'red' ones ...14:27
marios|ruck:/14:27
weshay|ruckthe color indicates the time14:27
ykarelrlandy, apart from branches: regex for c7 and missing environment/podman.yaml in c8 004 scenario LGTM14:28
marios|ruckweshay|ruck: ack seen one now at https://f0a6031bc8a0bf8f57a7-5ff3dddcf26e50a416e8b904201353d9.ssl.cf2.rackcdn.com/709094/2/gate/tripleo-ci-centos-7-containers-multinode/35a0497/logs/undercloud/home/zuul/overcloud_deploy.log14:28
*** amoralej is now known as amoralej|lunch14:30
rlandyykarel:  "missing environment/podman.yaml in c8 004 scenario" ... np - adding that ... pls explain "branches: regex for c7"14:33
ykarelrlandy, what about using branch regex for c7 ^(stable/(stein|train)).*$14:33
ykarelrlandy, with master there we will have issue when we branch ussuri14:33
ykarelthis job will start triggering than14:34
rlandyyeah I guess - now that it's defined, we could do that14:34
ykarelweshay|ruck, marios|ruck so somehow i get https://ci.centos.org/job/tripleo-quickstart-gate-master-delorean-quick-basic/86/console working14:37
ykarelreached undercloud install14:37
weshay|ruckykarel, w/o patches?14:37
marios|ruckykarel: nice :D14:37
marios|ruckykarel: still waiting for queens in the queue14:37
ykarelweshay|ruck, will propose patch in ci-config after some time14:37
ykarelyes hacked the job config and slave config14:38
weshay|ruckah k14:38
weshay|ruckykarel++ thank you14:38
ykarelweshay|ruck, issue was pip cache on slaves14:38
ykareltripleo-quickstart-extras was used very old14:39
weshay|ruckykarel, ya.. that's always been so... f.. frustrating14:39
weshay|ruckhad a hard time convincing people that was not working and the src was out of date14:39
ykarelwhich people :)14:40
ykarelit's clear in logs it's outdated14:40
weshay|ruckykarel, ya.. probably haven't talked about that for about a year14:41
weshay|ruckso details are fuzzy on who14:41
ykarelweshay|ruck, ack14:42
rlandyykarel: weshay|ruck: one more time, pls ... https://review.opendev.org/#/c/710177/14:42
ykarelrlandy, ack14:45
ykarelrlandy, commented14:55
*** ykarel is now known as ykarel|away14:59
* ykarel|away will check later from home15:00
rlandyweshay|ruck: what do you want to do about adding non-voting jobs scenario 001 and 004 to gate? ykarel correctly points out we shouldn't do that ( in general) but we plan for these to be non-voting in the short term15:00
rlandyugh - removed them from gate15:03
rlandynvm - we'll add them back when they are voting15:03
weshay|ruckrlandy, technically he is right15:04
rlandyyep - I did it15:04
weshay|ruckrlandy, but it's fine in this case15:04
rlandyit's fine15:04
rlandylet's just get this patch in15:04
weshay|ruckwill make it easier to flip15:04
weshay|ruckk15:04
rlandywe can add them back15:04
rlandyI am not stopping centos-8 in check over 2 jobs in gate15:05
*** tesseract has quit IRC15:05
rlandyhttps://review.opendev.org/#/c/710177/ - hopefully this goes now15:05
weshay|ruckmarios|ruck, rlandy fyi https://review.rdoproject.org/r/2552515:06
*** tesseract has joined #oooq15:07
marios|ruckweshay|ruck: ack15:07
rlandyweshay|ruck: I'm -1 on decreasing the pipeline frequency15:11
rlandyget rid of useless jobs, yes15:12
rlandybut with any luck, we will merge multinode soon15:12
rlandyso then we will start to be a viable testing15:12
marios|ruckrlandy: maybe we need a new pipeline then? replied there... i thin kthat is meant to apply to centos7 only15:13
marios|ruckrlandy: https://review.rdoproject.org/r/#/c/25525/1/zuul.d/upstream.yaml15:13
weshay|ruckrlandy, IT"S PINNED :)15:13
*** amoralej|lunch is now known as amoralej15:13
rlandywe just have to rememver to undo that15:13
weshay|ruckrlandy, we will :)15:14
rlandythen I am bring the component pipeline to every day15:14
rlandyweshay|ruck: ok then15:14
rlandymarios|ruck: have you seen this swap error elsewhere on rdocloud?15:16
rlandy2020-02-28 02:27:37.823492 | primary | mkswap: error: /root/swapfile is mounted; will not make swapspace15:16
rlandy2020-02-28 02:27:38.114499 | primary | ERROR15:16
rlandyhttps://logserver.rdoproject.org/openstack-component-compute/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-compute-master/f428cdc/job-output.txt15:16
marios|ruckweshay|ruck: looks like ramishra beat us to it https://bugs.launchpad.net/tripleo/+bug/186495315:16
openstackLaunchpad bug 1864953 in tripleo "Image prepare failed: [Errno 17] File exists" [Medium,Triaged]15:16
weshay|ruckaye15:17
rlandynever saw the swap issue yesterday15:19
marios|ruckweshay|ruck: updated the description added more links15:19
rlandyhttps://logserver.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-master/278d951/job-output.txt15:19
rlandy^^ there it is in master pipeline15:20
weshay|ruck2020-02-28 09:17:03.044762 | primary | mkswap: error: /root/swapfile is mounted; will not make swapspace15:23
rlandysince when????15:23
weshay|ruckrlandy, that's in 2020-02-28 09:16:46.178375 | PRE-RUN START: [untrusted : opendev.org/openstack/tripleo-ci/playbooks/tripleo-ci/pre.yaml@master]15:23
rlandyyeah15:23
rlandythe scenario jobs git by it though15:24
* rlandy will rerun15:24
weshay|ruckprimary | included: /var/lib/zuul/builds/278d9512175e407a97da35ec036fc2c1/untrusted/project_4/opendev.org/openstack/openstack-zuul-jobs/roles/configure-swap/tasks/root.yaml15:24
weshay|ruck2020-02-28 09:16:50.567615:24
weshay|ruckand that.. we run an include task15:24
rlandyhappened in component pipeline as well15:25
rlandyyeah - been passing forever15:25
weshay|ruckhttps://opendev.org/openstack/tripleo-ci/src/branch/master/roles/prepare-node/tasks/main.yaml#L1-L315:25
weshay|ruckrlandy, only hit it once?15:25
rlandyweshay|ruck: in integration pipeline15:26
rlandyand twice in component15:26
rlandyrunning testproject now15:28
marios|ruckrlandy: havent seen anything like swap error15:29
marios|ruckrlandy: looking at your link15:29
rlandymarios|ruck; I think it's only centos-815:29
rlandymaybe node changed15:29
marios|ruckrlandy: k15:29
rlandyI am running testproject15:29
* rlandy doesn't like having the pipeline run infrequently for this exact reason15:30
rlandystuff other than the product itself break15:30
marios|ruckrlandy: but i think he meant centos7 right weshay|ruck ?15:30
marios|ruckrlandy: like reduce 1/day for centos715:30
marios|ruckrlandy: but for centos8 make it 3/day15:30
marios|ruckrlandy: weshay|ruck: so we might need to split pipeline?15:31
rlandyit's the whole pipeline15:31
marios|ruckweshay|ruck: rlandy: openstack-periodic-master7 master815:31
weshay|ruckmarios|ruck, to be clear.. master / ussuri is pinned15:31
weshay|ruckdoesn't matter if it's centos-7 or 815:31
weshay|ruckonce a day.. augmented by test project should be fine15:31
marios|ruckweshay|ruck: wut. i thought the whole point of pinning is cos python 215:32
weshay|ruckmarios|ruck, it is...15:32
marios|ruckweshay|ruck: so for centos8 we didn't have to pin cos python315:32
weshay|ruckbut the rdo team has no way to delineate the pin per distro15:32
marios|ruckapparently that is a wrong assumption15:32
marios|ruckweshay|ruck: oh i see15:32
weshay|ruckmarios|ruck, that's what I thought as well15:32
marios|ruckwell fantastic15:32
weshay|ruckmarios|ruck, rlandy which is why I'm comfortable sending centos-8 jobs to vote / gate15:33
marios|ruckrlandy: weshay|ruck: let's split the difference and call it 2/day then15:33
rlandywe need a testproject pipeline15:33
rlandymarios|ruck: I am not fighting this15:33
marios|ruckbut if its pinned15:33
marios|ruckwhats the point15:33
marios|ruckjust that sometimes they fail on unrelated/transient/tempest/intermittent errors15:33
weshay|ruckmarios|ruck, only incoming changes would be from tripleo and random failures15:33
marios|ruckso 2 /day is something like 1 /day really15:33
marios|ruckwdyt rlandy weshay|ruck ? ^15:33
weshay|ruckmarios|ruck, the only thing master promotions are doing atm.. is allowing the rpm update on nodes and containers to run faster for tripleo changes15:34
weshay|ruckmarios|ruck, and I just blew away tempest verififcation15:34
rlandyweshay|ruck: at what point will they unpin ussuri?15:35
weshay|ruckso.. it's going to pass once a day :)15:35
rlandywhen we can run how many jobs in centos-8?15:35
weshay|ruckrlandy, when we're done w/ upstream centos-815:35
rlandydefine done15:35
rlandyall jobs gone??15:35
weshay|rucknot update15:35
rlandythis seems wrong15:35
weshay|rucklolz15:35
rlandylet's make all centos-7 non-voting and unpin15:35
weshay|ruckit's going to land a like turd when it's unpinned15:36
weshay|ruckrlandy, no no15:36
rlandynow or later15:36
weshay|ruckrlandy, I think once multinode and fs001 come alive.. we'll have enough coverage15:36
rlandymarios|ruck is like, no, wait until my shift is over15:36
weshay|ruckrlandy, the shitty part was that it was not discussed15:37
marios|ruckrlandy: ;)15:37
rlandycommunication is over-rated15:37
weshay|ruckplease15:37
rlandyI am not saying I agree - it's just how things run15:37
rlandyhttps://review.rdoproject.org/zuul/stream/8a92e7fbcc6d43a49b4d40acdf2a2b3c?logfile=console.log15:37
rlandyswap error again15:37
rlandyHouston we have a problem15:38
rlandysshnaidm updated the image but I thought that was reproducer only15:39
rlandyasking #rhos-ops15:39
sshnaidmrlandy, which swap error?15:42
sshnaidmrlandy, in which jobs?15:42
rlandysshnaidm: centos-8 standalone jobs ...15:42
rlandy2020-02-28 09:17:02.781007 | TASK [configure-swap : Make swapfile]15:42
rlandy2020-02-28 09:17:03.044762 | primary | mkswap: error: /root/swapfile is mounted; will not make swapspace15:42
sshnaidmrlandy, huh15:42
sshnaidmrlandy, I was reproducing this too yesterday15:43
sshnaidmrlandy, but I thought it's reproducers issue15:43
rlandysshnaidm: this is new15:43
rlandywe were running green yesterday15:43
sshnaidmrlandy, and it happens in regular jobs sometimes?15:43
rlandyit happened in the standalone jobs we have in component and integration pipelines15:44
rlandysomehow the scenario jobs didn't hit it15:44
rlandyunless they ran before the change ... seee15:44
sshnaidmrlandy, cool, I can say reproducer works great..15:44
rlandythat's one thing15:44
rlandyhttps://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-master%2015:44
rlandy^^ look at the centos-8 jobs there15:45
weshay|ruckrlandy, just in case https://review.opendev.org/71047015:47
rlandyweshay|ruck: looks like we may call the playbook twice15:48
rlandysee rhos-ops15:48
rlandyhttps://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-standalone-master15:48
weshay|ruckrlandy, it's getting called twice15:49
rlandytestproject run passed15:49
weshay|ruck2020-02-28 09:13:53.265362 | primary | included: /var/lib/zuul/builds/278d9512175e407a97da35ec036fc2c1/untrusted/project_4/opendev.org/openstack/openstack-zuul-jobs/roles/configure-swap/tasks/root.yaml15:49
rlandyyeah - looking at how15:49
weshay|ruck2020-02-28 09:16:50.540771 | primary | included: /var/lib/zuul/builds/278d9512175e407a97da35ec036fc2c1/untrusted/project_4/opendev.org/openstack/openstack-zuul-jobs/roles/configure-swap/tasks/root.yaml15:49
rlandyand it's not called twice in centos-7??15:49
weshay|ruckrlandy, looks like we can remove it from ours maybe 2020-02-28 09:16:46.178375 | PRE-RUN START: [untrusted : opendev.org/openstack/tripleo-ci/playbooks/tripleo-ci/pre.yaml@master]15:49
weshay|ruck2020-02-28 09:13:48.223569 | PRE-RUN START: [untrusted : opendev.org/openstack/tripleo-ci/playbooks/tripleo-ci/pre.yaml@master]15:51
rlandyonly once in centso-715:51
weshay|ruckrlandy, we have a partenting issue15:51
rlandyyep15:51
weshay|ruckrlandy, we're calling pre: twice15:51
weshay|ruckpre.yaml15:51
rlandyweshay|ruck: chat?15:51
weshay|rucksure15:51
weshay|ruckhttps://meet.google.com/weu-oamj-mmg?authuser=115:52
sshnaidmzbr, can you please fix linters in https://review.rdoproject.org/r/#/c/25464/ ?15:54
*** ykarel|away is now known as ykarel15:56
weshay|ruckrlandy, https://logserver.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-master/278d951/job-output.txt15:59
raukadahok I am back for few hours, anything I can help16:07
marios|ruckoooci-jobs.sh now with centos-8 promotion criteria https://github.com/marios/tripleo_ruck_job_tool/commit/a6bdeee5de2c7ef6f5d244d45db785d246db4cab  ... i realise I'm the only one using that but fyi, i *am* still using it, and in case folks forgot about that16:10
*** tesseract has quit IRC16:17
marios|ruckykarel: weshay|ruck: looks like it works \o/ https://review.opendev.org/#/c/708791/9/roles/build-test-packages/tasks/main.yml16:19
marios|ruckykarel: weshay|ruck: at last it got passed the error http://zuul.openstack.org/stream/bb2de517f51244a79ebcca0c202f94b0?logfile=console.log16:19
ykarelmarios|ruck, hmm queens  job passed that point16:19
marios|ruckykarel: yeah16:19
weshay|ruckNICE16:20
marios|ruckweshay|ruck: ykarel: still running i'll update if it finishes green and remove wip etc16:23
marios|ruckykarel:++ looks like contextlib2>0.5.0,<0.6.0 was the winner... and the debug message also changes now so it seems that also fixes the 'ancient virtualenv' issue too16:24
marios|ruckykarel: no longer 'virtualenv' is a package and cannot be directly executed"16:24
ykarelmarios|ruck, ack for unblocking, but i think we can check for more better way16:25
marios|ruckykarel: but now like it passes fine rc': 0, 'failed': False, u'stderr': u'virtualenv 20.0.7 from /usr/lib/python2.7/site-packages/virtualenv/__init__.pyc', u'delta': u'0:00:00.159649'16:25
ykarellike to figure out 1st way didn't worked16:25
marios|ruckykarel: honestly its queens so at this point i'd tidy that up with pointers to the bug and merge it16:25
ykareland fix that, as --user is more safer16:25
marios|ruckykarel: and if we can make it more beautiful later bonus16:25
marios|ruckimho16:26
ykarelmarios|ruck, okk16:26
marios|ruckykarel: cos phase 1 is red for almost 1 month16:26
marios|ruckon that issue16:26
ykarelmarios|ruck, ack for me on unblocking that16:26
* ykarel checks if ci.centos job finished16:26
*** cgoncalves has quit IRC16:27
*** cgoncalves has joined #oooq16:28
ykareltripleo-quickstart-gate-master-delorean-quick-basic SUCCESS16:31
*** beagles_pto is now known as beagles16:32
weshay|ruckykarel, NICE16:34
weshay|ruckmarios|ruck, are you updating for line 60 https://review.opendev.org/#/c/708791/9/roles/build-test-packages/tasks/main.yml16:36
marios|ruckweshay|ruck: yeah was waiting for it to complete16:37
marios|ruckweshay|ruck: if you mean debug16:37
marios|ruckweshay|ruck: also need to remove layout.yaml https://review.opendev.org/#/c/708264/16:37
marios|ruckweshay|ruck: blocking to prevent accident16:38
weshay|ruckaye16:38
weshay|rucknop16:38
weshay|rucknp16:38
weshay|ruckwhat a pita that bug was16:38
marios|ruckyeah16:38
marios|ruckweshay|ruck: it came down to noticing that virtualenv was being installed in queens but not th rest (i.e. the task was firing like it was saying 'there is no venv'16:38
weshay|rucknow I have to create a centos-8 version of that job lolz16:38
marios|ruckbut actually is was like 'virtualenv' is a package and cannot be directly executed"16:38
weshay|ruckOH REEELY16:39
marios|ruckweshay|ruck: and zbr told us that was ancient16:39
marios|ruckweshay|ruck: so first we tried re-installing virtualenv itself but nop16:39
marios|ruckthen this contextlib2 (we tried that v1 if you recall ? https://review.opendev.org/#/c/708791/1/roles/build-test-packages/tasks/main.yml16:39
marios|ruckweshay|ruck: but we tried for the venv, today we added it for system and particular versions 'cos ykarel said' and he was right ;(16:40
marios|ruck:)16:40
rlandyweshay|ruck: looking better with https://review.rdoproject.org/r/#/c/25537/16:42
rlandyduplicate job definition must have been smooshed together by zuul16:43
weshay|ruckrlandy, ya. ur passed it16:44
weshay|ruckrlandy, :)16:44
rlandywherever chandan is, he's merging patches16:44
weshay|ruckraukadah, you'll want to move 3rd party dep jobs to use centos-816:44
weshay|rucknext week16:45
*** jfrancoa has quit IRC16:48
*** ykarel is now known as ykarel|afk16:48
raukadahweshay|ruck, yes16:55
marios|ruckweshay|ruck: ykarel|afk: fyi summary there https://bugs.launchpad.net/tripleo/+bug/1863632/comments/816:58
openstackLaunchpad bug 1863632 in tripleo "lastest virtenv workarounds broke native tripleo-quickstart install" [Critical,Triaged]16:58
raukadahrlandy, i am here16:58
* marios|ruck starting shutdown sequence weshay|ruck 16:59
rlandyraukadah: what happened to the conference? yesterday?16:59
weshay|ruckmarios|ruck, rock on.. thanks for help16:59
marios|ruckweshay|ruck: did you see email about monday?16:59
marios|ruckopenstack-pcci weshay|ruck17:00
* weshay|ruck looks17:00
weshay|ruckmarios|ruck, ya.. we got u17:00
marios|ruckweshay|ruck: ack thx17:00
raukadahrlandy, it was today and tomorrow17:02
raukadahrlandy, I attended it, it was fun17:02
rlandycool17:02
rlandyraukadah: we got multinode passing17:03
raukadahrlandy, yes cool17:03
raukadahrlandy, I also saw panda fix release emit script thanks you17:03
rlandyraukadah: funny - we didn't change anything :)17:03
rlandyraukadah: and undercloud job still hangs - looking into that17:03
rlandyweird, right?17:04
raukadahrlandy, then undercloud will also work then?17:04
*** jfrancoa has joined #oooq17:04
raukadahrlandy, yes if not working17:04
rlandyraukadah: some voodoo there17:04
rlandyinvestigating17:04
rlandybut once the depends on patches go in, we will merge multinode17:05
rlandyand it should go upstream as well17:05
pandaraukadah: weshay|ruck: It's not fixed yet, and with integrtion tests taking more than 1 hour to start, I don't think I'll be able fix that patch today. Sorry.17:07
rlandyhttps://f0a7a1b83275e65c79fc-5b3bf89a3c3f441361bcb66efd5a8121.ssl.cf2.rackcdn.com/710230/1/check/tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates/7ba1337/job-output.txt17:07
rlandywe got a legit error17:07
weshay|ruckpanda, aye.. so it goes17:07
rlandyweshay|ruck: raukadah: ^^ not going in17:07
*** tosky has quit IRC17:09
rlandyfixing https://review.opendev.org/#/c/71023017:10
*** marios|ruck is now known as marios|out17:18
rlandyweshay|ruck: raukadah: https://review.opendev.org/#/c/710177/ - got at +1 from yakrel :)17:23
rlandyykarel17:23
rlandylet's go17:23
weshay|ruckrlandy, aye.. 1.5 hours until they kick17:24
rlandyunderstood17:24
weshay|ruckrlandy, here's a blast from the past.. https://review.rdoproject.org/r/#/c/25541/17:28
weshay|ruckrlandy, k.. between periodic master and https://review.rdoproject.org/r/#/c/25536/3/zuul.d/tripleo.yaml you should have space to run components nightly17:30
* rlandy turns back time17:31
rlandyweshay|ruck: ack - putting in that review17:31
weshay|ruckrlandy, hrm.. hit this upstream as well in docker.io 2020-02-28 17:07:35 | tripleo_common.image.exception.ImageNotFoundException: Not found image: docker://trunk.registry.rdoproject.org/tripleomaster/centos-binary-neutron-server:0481ffa881dbc1aa61d9810533d8522917:37
rlandysame as component jobs17:38
rlandywhat's up with that?17:38
rlandyit should use docker.io soon though17:39
rlandyruns current-tripleo17:39
*** ykarel|afk is now known as ykarel17:39
*** holser has quit IRC17:44
*** jpena is now known as jpena|off17:50
*** derekh has quit IRC18:02
*** marios|out has quit IRC18:19
*** amoralej is now known as amoralej|off18:22
*** ccamacho has quit IRC18:24
rlandydocker://trunk.registry.rdoproject.org/tripleomaster/centos-binary-collectd:047fa91541bc92aa0845fab138b43f3318:31
rlandyweshay|ruck; what was the final resolution with the /centos-binary-collectd container?18:31
weshay|ruckrlandy, I pushed it18:31
weshay|rucknothing since18:31
rlandyweshay|ruck: docker or rdo?18:31
weshay|ruckdocker https://hub.docker.com/layers/tripleomaster/centos-binary-collectd/0481ffa881dbc1aa61d9810533d85229/images/sha256-be6d2ca2138be2d7e2b155eb49c08e9bfb4bf85520866cef7a4add2133fbba71?context=explore18:32
rlandymissing in rdo18:32
weshay|ruckrlandy, check hash_x86_6418:32
rlandyhttps://console.registry.rdoproject.org/registry#/images/tripleomaster/centos-binary-collectd:0481ffa881dbc1aa61d9810533d8522918:33
rlandybut not https://console.registry.rdoproject.org/registry#/images/tripleomaster/centos-binary-collectd:047fa91541bc92aa0845fab138b43f3318:34
rlandymaybe this will settle down18:35
ykarelrlandy, https://review.opendev.org/#/c/709701/ is needed for collectd18:36
ykarelit depends on other epel patch18:36
rlandyah ok18:36
*** ykarel is now known as ykarel|away18:48
*** EmilienM is now known as EvilienM19:30
*** dpawlik has quit IRC19:43
*** dpawlik has joined #oooq19:43
weshay|ruckrlandy, we have to get the hash yet again?20:03
weshay|ruckhttps://review.opendev.org/#/c/710230/2/roles/container-prep/tasks/main.yml20:04
rlandyweshay|ruck: yep :(20:04
rlandychandan had that review in but it was failing gates20:04
weshay|ruckok... we're def writing a python module and an ansible module for this20:04
rlandyso I just fixed up the when cases20:04
rlandyit may or may not get by now20:04
rlandyweshay|ruck: I am trying to get multinode in https://review.rdoproject.org/r/#/c/25414/20:05
rlandybut it has depends-on20:05
weshay|ruckrlandy, btw.. Alex found an issue w/ the undercloud install on centos-8 and is working it20:05
rlandyweshay|ruck: going to do the same upstream so at least we have a review there20:05
rlandyweshay|ruck: do we have a patch?20:05
rlandyso we can test with https://review.rdoproject.org/r/#/c/25366/20:06
* rlandy wonders how multinode passes then20:07
weshay|ruckrlandy, don't see a patch up yet20:07
rlandyok20:07
weshay|ruckI think it may be inconsistent.. not sure tbh20:07
rlandyalways fun20:08
rlandyhttps://opendev.org/openstack/tripleo-ci/src/branch/master/roles/run-test/templates/oooq_common_functions.sh.j2#L16220:10
rlandyget_build_command20:10
rlandymaybe panda's patch fixes that20:11
weshay|ruckrlandy, when something is used that often.. that should trigger a pattern of writing the logic in python, creating a ansible and python module if possible20:12
rlandyabsolutely20:13
rlandythis is nightmare20:13
weshay|ruckheh ya20:13
rlandyI am literally dreaming of centos 8 issues these days20:13
weshay|ruckoh boy20:14
weshay|ruckrlandy, perhaps take monday off :)20:14
rlandyweshay|ruck: when this is all over, yeah20:15
rlandywe have things to do!!!!20:15
weshay|ruckrlandy, ya.. you and I both20:15
rlandypurim20:15
rlandyweshay|ruck: tracing back the get_extra_vars_from_release - panda's patch may fix it - we may not need chandan's patch - testing20:27
weshay|ruckgosh20:38
zbrrlandy: you are not the only one having centos-8 dreams, does this info makes you feel better?20:41
rlandyzbr: in a way :)20:42
rlandyso we're all crazy, together20:42
zbrthe not so funny part is that sometimes "i see dead packages"20:44
weshay|rucklolz20:44
weshay|ruckdo they know they're dead?20:45
zbrweshay|ruck: they were moved, or moved...20:45
zbrremoved20:45
weshay|ruckam I -e'd?20:45
zbrtox removed,  re2(-devel) moved into EPEL.20:46
zbrand re2-devel is areal issue because pre-commit may need it, and we cannot install it using bindep20:46
zbrbecause bindep cannot enable epel-release and install a package, it works only for already enabled repos.20:47
zbri am wondering how we will address that.20:47
zbra temporary workaround is to use the tools/test-setup.sh script to enable epel and install the package, but is ugly as hell.20:48
weshay|ruckzbr, tox removed it as a dep?20:48
zbrtox removed is not a big deal, the upstream ensure-tox installs it, i fixed it.20:49
zbrinstalls it using pip, but works. that was merged this week.20:49
weshay|ruckzbr, create a clone of bindep for centos that enables epel.. installs packages via the same pattern.. and then removes epel20:49
weshay|ruckdone20:49
weshay|ruckbindep-epel20:50
weshay|ruckbindepel20:50
*** jfrancoa has quit IRC20:50
zbrupstream would not oppose me patching bindep itself20:51
zbrbut i am not sure how we can define that in the bindep file format.20:51
zbrre bindep, I think that upstream proposed using a role to enable epel, so not changing bindep itself21:01
weshay|ruckzbr, we just need to be careful.. to ensure epel is turned off before any tripleo bits start21:01
zbrmainly because it would be too complex and it would not be a generic solution, some repos are installed using a RPM (url), not a package21:01
zbrweshay|ruck: in this case is about linters, so we would not affect other jobs.21:02
weshay|ruckk21:02
zbri will investigate more, I want to see who really needs the fb-re2 wheel, which is the one that tries to compile some C code that needs re2-devel.21:03
zbronly the "fb-" prefix gives me creeps, even if I know is only regex library.21:03
zbri wish all a nice weekend21:16
zbrweshay|ruck: the zuul feature I implemented is the "build history" link from the job row, as in https://zuul.opendev.org/t/openstack/build/98e431e23a3d4a8abfe1afc6c84e09ca21:19
zbrwhich opens a query about history of that specific job on the same repository.21:19
zbrsaving lots of clicks.21:19
weshay|ruckah .. ya.. I saw marios using that21:27
weshay|ruckthat is nice21:27
*** jmasud has joined #oooq21:31
rlandyweshay|ruck: OMG ...22:17
rlandyhttps://review.rdoproject.org/r/#/c/25366/22:17
rlandylook - singlenode22:17
rlandyDeployment successful!22:18
rlandywoohoo22:18
rlandyI'm out ... good shabbos!22:19
*** rlandy has quit IRC22:21
*** tosky has joined #oooq22:24
*** EvilienM is now known as EmilienM22:56
*** dpawlik has quit IRC23:13
weshay|rucknice!23:31

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!