Thursday, 2019-05-09

*** dsneddon has quit IRC01:15
*** dsneddon has joined #oooq01:20
*** dsneddon has quit IRC01:29
*** dsneddon has joined #oooq01:39
*** apetrich has quit IRC01:56
*** rlandy|bbl is now known as rlandy02:04
*** rlandy has quit IRC02:04
*** ykarel|away has joined #oooq03:04
*** dsneddon has quit IRC03:43
*** dsneddon has joined #oooq03:45
*** dsneddon has quit IRC03:49
*** udesale has joined #oooq03:51
*** dsneddon has joined #oooq04:10
*** ratailor has joined #oooq04:14
*** ykarel|away is now known as ykarel04:24
*** raukadah is now known as chandankumar04:30
*** dsneddon has quit IRC04:37
*** ykarel is now known as ykarel|afk04:40
*** ykarel|afk has quit IRC04:44
*** ykarel|afk has joined #oooq04:59
*** ykarel_ has joined #oooq05:01
*** ykarel|afk has quit IRC05:04
*** jaosorior has joined #oooq05:12
*** ykarel_ is now known as ykarel05:32
*** dsneddon has joined #oooq05:34
*** marios has joined #oooq05:58
*** ksambor has joined #oooq06:02
*** udesale has quit IRC06:13
*** udesale has joined #oooq06:14
*** holser_ has joined #oooq06:15
*** saneax has joined #oooq06:31
*** sanjayu_ has joined #oooq06:35
*** saneax has quit IRC06:35
*** sanjayu__ has joined #oooq06:37
*** sanjayu_ has quit IRC06:40
*** marios has quit IRC06:40
*** marios has joined #oooq06:42
*** quiquell|off is now known as quiquell06:47
*** dsneddon has quit IRC06:49
*** udesale has quit IRC06:53
*** udesale has joined #oooq06:53
*** kopecmartin|off is now known as kopecmartin06:54
*** dsneddon has joined #oooq06:58
*** apetrich has joined #oooq06:59
*** jfrancoa has joined #oooq07:00
*** tesseract has joined #oooq07:09
*** dsneddon has quit IRC07:11
*** sanjayu__ has quit IRC07:12
*** sanjayu__ has joined #oooq07:12
*** dsneddon has joined #oooq07:15
*** panda|off is now known as panda07:19
*** ykarel is now known as ykarel|lunch07:21
*** jaosorior has quit IRC07:27
*** skramaja has joined #oooq07:28
*** dsneddon has quit IRC07:45
chandankumarquiquell: Hello07:45
chandankumarquiquell: on fedora 30 I reached here https://bugs.launchpad.net/tripleo/+bug/1828172/comments/707:45
openstackLaunchpad bug 1828172 in tripleo "No match for argument: python2-dnf on tripleo reproducer libvirt Fedora 30" [High,Triaged]07:45
chandankumarbut job is not able to start07:45
chandankumarhow to debug there?07:45
*** jpena|off is now known as jpena07:45
quiquellchandankumar: first thing is do a docker-compose ps07:47
quiquellto check what's the state of services07:47
chandankumarquiquell: http://paste.openstack.org/show/751142/07:49
chandankumargerritconfig exit with 107:49
chandankumarsorry with )07:49
chandankumar007:49
quiquellthats ok07:51
quiquellmaybe it's jus taking more time in your machine to start it up07:51
quiquelltry07:51
quiquellcurl http://localhost:9000/api/tenant/tripleo-ci-reproducer/status07:51
chandankumarhttp://paste.openstack.org/show/751143/07:53
chandankumarwill I increase the wait for job time to start?07:54
quiquellYep increase it07:57
quiquellzuul looks ok07:57
chandankumarnodes are up07:57
quiquellmayeb you can put a patch to parameterize retries and time between retries at role level07:57
quiquellso people can increase it in case they have slower mnachine connections07:58
chandankumarquiquell: sure07:58
*** dsneddon has joined #oooq08:21
*** ykarel|lunch is now known as ykarel08:29
*** dsneddon has quit IRC08:36
*** jtomasek has joined #oooq08:37
*** bogdando has joined #oooq08:53
*** panda has quit IRC08:55
*** panda has joined #oooq08:55
chandankumarquiquell: still not working09:03
chandankumarquiquell: stuck at same place09:03
* chandankumar leaves the experiment focus on other work09:05
quiquellservices started ?09:06
pandachandankumar: same thing that happened to me09:06
quiquelldid you increase retries ?09:06
chandankumarYes09:06
chandankumarit made it to 6009:06
quiquelldman09:06
quiquellthey are 60 now09:06
chandankumarhttp://paste.openstack.org/show/751150/09:07
quiquellhttps://github.com/rdo-infra/ansible-role-tripleo-ci-reproducer/blob/master/tasks/start.yaml#L2909:07
chandankumaron zuul dashboard it is saying something went wrong09:07
quiquellthis place09:07
quiquellwait this is different09:08
quiquellthis means job has not started09:08
quiquell:-/09:08
chandankumarhttps://review.rdoproject.org/r/#/c/20608/09:08
chandankumarI send pr and made same changes in /tmp created directory invoked the playbook09:08
*** dsneddon has joined #oooq09:13
*** dsneddon has quit IRC09:17
*** marios has quit IRC09:19
*** marios has joined #oooq09:38
*** dsneddon has joined #oooq09:44
*** dsneddon has quit IRC09:48
*** Tengu has quit IRC10:13
*** Tengu has joined #oooq10:13
mariosarxcruz: chandankumar kopecmartin o/ folks please see comment #5 https://bugs.launchpad.net/tripleo/+bug/182786310:33
openstackLaunchpad bug 1827863 in tripleo "(stable/stein) standalone-upgrade job failing tempest post upgrade - test_server_security_groups" [Undecided,Triaged]10:33
marioschandankumar: ^ is that right? should we backport that change?10:33
marioschandankumar: otherwise, do we need to think about changing the way we run tempest on that job. maybe only run it once at the end.10:33
*** jaosorior has joined #oooq10:34
marioschandankumar: but i don't think that is a good solution. it sounds like a bug in tempestconf as far as i understood.. on the 2nd run it doesn't create the flavor and it has changed in the meantime10:34
*** jaosorior has quit IRC10:35
arxcruzmarios: i though chandankumar already did the backport or update the tempestconf version in rock to have this patch10:35
*** jaosorior has joined #oooq10:35
mariosarxcruz: i didn't see something in the bug (and was out last week, only now revisiting).10:35
chandankumarmarios: https://review.rdoproject.org/r/#/c/20579/10:36
chandankumarmarios: once promotion happens, new packages will be available10:36
mariosarxcruz: and it still hits it today on stein see https://review.opendev.org/#/c/656967 hm well may 7th http://logs.openstack.org/67/656967/5/check/tripleo-ci-centos-7-standalone-upgrade/12a149b/logs/tempest.html.gz10:36
chandankumarmarios: temepstconf 2.2.0 will fix the issue10:36
marioschandankumar: ack thanks added on bug10:37
mariosjfrancoa: ^10:37
* chandankumar also added10:38
chandankumarmarios: is the same kind of job in periodic10:38
chandankumarwe can test it right now?10:38
chandankumar*.10:38
jfrancoamarios: great, is there a way to pin that version to check if the job would pass?10:39
*** tesseract has quit IRC10:40
*** tesseract has joined #oooq10:41
mariosjfrancoa: see chandankumar example https://review.rdoproject.org/r/#/c/20574/10:42
mariosjfrancoa: no sorry thats not what you want10:43
mariosjfrancoa: dno't know chandankumar can we test specific version of tempest on a check job?10:43
*** dsneddon has joined #oooq10:44
*** tesseract has quit IRC10:45
*** tesseract has joined #oooq10:45
chandankumarI think there was a patch in toci tjhough which we use to do that10:46
*** holser_ is now known as holser|burger10:46
chandankumarwe use to do something like this https://review.opendev.org/#/c/637482/2/roles/common/vars/main.yaml10:47
mariosjfrancoa: chandankumar well looks like there is master promotion today so hopefully we see the fix soon10:49
*** dsneddon has quit IRC10:49
jfrancoamarios: ok, in that case let's just wait for the promotion10:50
mariosjfrancoa: if its green fyi https://review.opendev.org/#/c/655661/10:53
mariosjfrancoa: s/if/when ;)10:53
jfrancoamarios: h00t h00t! looking forward that :-)10:54
jfrancoamarios: is there any policy to make the job voting? like it should be green for 3 full moons...or something like that?10:55
mariosjfrancoa: not official policy no but yes it needs to be voting and stable. in this case though we make an exception because it is time sensitive. we need it voting before the world starts backporting all their stuff to stein10:55
mariosjfrancoa: err green and stable10:56
*** amoralej is now known as amoralej|lunch10:57
*** panda is now known as panda|lunch11:02
*** skramaja_ has joined #oooq11:02
*** skramaja has quit IRC11:02
chandankumarack!11:04
*** udesale has quit IRC11:15
*** d0ugal has quit IRC11:16
*** d0ugal has joined #oooq11:16
*** ykarel is now known as ykarel|afk11:37
*** jpena is now known as jpena|lunch11:39
*** dsneddon has joined #oooq11:44
*** dsneddon has quit IRC11:48
*** rf0lc0 is now known as rfolco|ruck11:49
*** d0ugal has quit IRC11:51
*** d0ugal has joined #oooq11:52
rfolco|ruckchandankumar, did you check tempestconf issue that ykarel|afk raised ?12:05
chandankumarrfolco|ruck: it is for queens, I am looking on it12:05
rfolco|ruckchandankumar, do we have a bug ? let me know how I can help here12:06
*** panda|lunch is now known as panda12:06
chandankumarrfolco|ruck: no bug12:08
chandankumarrfolco|ruck: for queens and rocky issue, I have reused the same bug for skip list12:09
rfolco|ruckchandankumar, ok, I see the related-bug.12:10
*** rlandy has joined #oooq12:20
weshaymorning12:28
*** jpena|lunch is now known as jpena12:31
*** amoralej|lunch is now known as amoralej12:33
rfolco|ruckweshay, o/12:34
weshayrfolco|ruck see my freak out email last night?12:35
rfolco|ruckweshay, I see the dissenter email ?12:35
weshayopenstack-dev man12:36
*** quiquell is now known as quiquell|lunch12:37
rfolco|ruckweshay, I see some green ones... could be infra transient issue?12:41
weshayya.. was the patches I think12:46
weshayrfolco|ruck please work w/ chandankumar to make sure we have all our pre-planning mtgs for next sprint12:46
chandankumarmarios: weshay rfolco|ruck https://review.rdoproject.org/r/#/c/20608/12:47
*** udesale has joined #oooq12:50
chandankumarweshay: rfolco|ruck CentOS 8 https://wiki.centos.org/About/Building_812:53
rfolco|ruckchandankumar, weshay panda pre-planning scheduled for tomorrow morning 2pm UTC12:54
*** ratailor has quit IRC12:54
chandankumarrfolco|ruck: accepted thanks!12:55
*** ykarel|afk is now known as ykarel12:55
*** sanjayu__ has quit IRC12:57
*** sanjayu__ has joined #oooq12:57
*** holser_ has joined #oooq13:00
chandankumarpanda: weshay marios rfolco|ruck rlandy kopecmartin arxcruz quiquell|lunch scrum time13:01
arxcruzchandankumar:you're the one missed :P13:01
quiquell|lunchchandankumar: I am going to try to finish stuff on the zuul repro13:03
quiquell|lunchchandankumar: before I leave13:03
chandankumarquiquell|lunch: ack!13:03
chandankumararxcruz: I am there13:03
rfolco|ruckweshay, connection issues13:04
rfolco|ruckneed to reconnect here13:04
*** Goneri has joined #oooq13:18
*** quiquell|lunch is now known as quiquell13:25
*** tesseract has quit IRC13:35
*** tesseract has joined #oooq13:35
*** vkapalav has joined #oooq13:42
*** altlogbot_1 has quit IRC13:43
*** altlogbot_2 has joined #oooq13:44
*** dsneddon has joined #oooq13:45
*** tcw has joined #oooq13:48
*** tcw has quit IRC13:48
*** tcw has joined #oooq13:48
*** dsneddon has quit IRC13:50
*** quiquell is now known as quiquell|brb13:55
mariospanda: sorry for the spam on the taiga task :/ i wrote it a while ago13:57
*** Vorrtex has joined #oooq14:20
*** ccamacho has joined #oooq14:21
*** tesseract has quit IRC14:24
*** ykarel is now known as ykarel|away14:31
*** skramaja_ has quit IRC14:48
*** tesseract has joined #oooq14:50
*** tesseract has quit IRC14:51
*** tesseract has joined #oooq14:51
*** ratailor has joined #oooq14:52
*** tesseract has quit IRC15:01
*** quiquell|brb is now known as quiquell|off15:01
*** ykarel|away has quit IRC15:02
*** tesseract has joined #oooq15:03
*** aakarsh has quit IRC15:10
*** tesseract has quit IRC15:11
rlandyweshay: https://cloud.centos.org/centos/7/images/ for the md5sum for the latest centos image (1901) ... I see sha256sum.txt.asc but no md5sum.txt.asc. expected?15:14
chandankumarweshay: do you have ascineema os_tempest recording somewhere https://asciinema.org/a/rm7LDAs6RAR1xh7oQrp07LaeR?15:16
*** tesseract has joined #oooq15:16
chandankumarit got expired15:16
chandankumarpeople are asking about it downstream15:16
*** tesseract has quit IRC15:18
*** tesseract has joined #oooq15:20
*** dtrainor has joined #oooq15:20
mariosrlandy: o/ i updated the readme quite a bit - in particular removed the list of options - fyi (cos you wrote that) grateful for thoughts if you have time thanks https://review.opendev.org/#/c/657779/3/roles/create-zuul-based-reproducer/templates/README-reproducer-zuul-based-quickstart.html.j215:21
mariosweshay: will you consider re-adding your vote please. I lost it when I updated that thanks https://review.opendev.org/#/c/657779/15:21
mariosweshay: update was to the readme https://review.opendev.org/#/c/657779/3/roles/create-zuul-based-reproducer/templates/README-reproducer-zuul-based-quickstart.html.j215:22
rlandyok15:22
*** jaosorior has quit IRC15:23
chandankumarrlandy: https://review.opendev.org/#/c/657964/ please merge this once the gate passed15:23
chandankumarit will bring promotion jobs back15:23
* chandankumar is logging out15:23
chandankumarsee ya tomorrow15:23
chandankumarrlandy: regarding introducing praveenkumar, I talked with him15:24
chandankumarhis main query was does the any projects wants to run zuul based CI system does they provide the infra having capacity more than 30 GB15:24
rlandyok - watching that15:24
chandankumarproject may be outside RH that's why I said no15:25
weshaymarios done15:25
rlandychandankumar: so the bm nodes may have that capacity15:25
rlandylet me check15:25
chandankumarbut do we provide the infra for them?15:25
chandankumarjust like openstack ci provides infra for openstack servcies not other projects15:26
mariosweshay: thanks. i aimed for making it more copy-paste friendly15:26
mariosweshay: you can see it once the jobs run it will get rendered15:26
rlandychandankumar: not outside of what we are using - but he can ask the scale.HA/performance people15:27
chandankumarrlandy: ok, I will let him know.15:27
chandankumarrlandy: thanks :-)15:27
*** chandankumar is now known as raukadah15:28
rlandymarios: maybe you know, wrt md5 for centos images, iiuc, these used to be provided? but now I only see sha256sum.txt.asc15:29
mariosrlandy: not something am aware of any change there etc15:33
* marios shutdown sequence15:33
*** ykarel has joined #oooq15:35
*** dsneddon has joined #oooq15:39
weshaymarios fyi.. sorry to hit you late.. but it can be a tomorrow thing..  https://review.opendev.org/#/c/655661/5  the recheck on stein standalone upgrade is also failing15:49
* weshay will have a look15:50
weshaymerged your doc patches, thank you for rocking those out15:50
mariosweshay: ack i will have to check that tomorrow15:51
mariosweshay: i mean the standalone stein thing15:51
weshayaye15:51
weshaythanks15:51
mariosweshay: anyway we need the fixes all merged and promotion for tempest thing15:51
*** kopecmartin is now known as kopecmartin|off15:51
mariosweshay: thank you15:51
*** dsneddon has quit IRC15:52
weshaymarios probably easy15:54
weshayTASK [Purge /var/lib/docker] ***************************************************[00m15:54
weshay2019-05-09 14:25:11 | 2019-05-09 14:25:11.781 131581 WARNING tripleoclient.v1.tripleo_upgrade.Upgrade [-] fatal: [standalone]: FAILED! => {"changed": false, "msg": "rmtree failed: [Errno 16] Device or resource busy: '/var/lib/docker/containers/807daa15424a4e74ad86b124e3977415540e9cca1f78cf5c19e0b90134b9e43c/shm'"}[00m15:54
weshaywill raise a bug15:54
mariosweshay: that one is fixed... i mean that is the master issue15:54
weshayoh.. the stein job is hitting right now15:54
mariosweshay: maybe didn't merge yet15:55
weshayhttp://logs.openstack.org/61/655661/5/check/tripleo-ci-centos-7-standalone-upgrade-stein/8d30c36/logs/undercloud/home/zuul/standalone_upgrade.log.txt.gz#_2019-05-09_14_25_1115:55
weshayfor tomorrow15:55
mariosweshay: see https://review.opendev.org/#/c/656967 merge 37 mins ago15:55
mariosweshay: that is the fix15:55
mariosweshay: so after this it should work15:55
weshayah k15:55
mariosweshay: all info in https://tree.taiga.io/project/tripleo-ci-board/task/98915:56
marios(at least i hope it works, assuming the tempest issue is fixed)15:56
* marios out ttyl will pickup tomorrow15:56
*** marios has quit IRC15:56
weshaythanks15:56
*** tesseract has quit IRC16:01
*** ratailor has quit IRC16:03
*** ykarel is now known as ykarel|away16:11
*** aakarsh has joined #oooq16:14
*** bogdando has quit IRC16:15
rlandyweshay: hey ... still on about the md5 question16:16
weshayk.. what's up16:16
rlandy<rlandy> weshay: https://cloud.centos.org/centos/7/images/ for the md5sum for the latest centos image (1901) ... I see sha256sum.txt.asc but no md5sum.txt.asc. expected?16:16
rlandy md5 for centos images, iiuc, these used to be provided? but now I only see sha256sum.txt.asc16:16
weshayrlandy you have to download and then run it to calculate it16:17
rlandyweshay: ok16:17
*** vkapalav is now known as vinaykns16:19
weshaypanda can you please fix the linter issues on  https://code.engineering.redhat.com/gerrit/#/c/167729/16:24
*** dtantsur has quit IRC16:26
*** jpena is now known as jpena|off16:42
*** irclogbot_3 has quit IRC16:46
*** irclogbot_2 has joined #oooq16:46
*** dsneddon has joined #oooq16:49
*** dsneddon has quit IRC16:54
*** dsneddon has joined #oooq17:06
*** udesale has quit IRC17:13
*** holser_ has quit IRC17:46
*** vinaykns has quit IRC17:55
*** vinaykns has joined #oooq17:58
*** jtomasek has quit IRC18:07
*** jtomasek has joined #oooq18:12
*** jtomasek has quit IRC18:16
*** ykarel|away has quit IRC18:17
*** amoralej is now known as amoralej|off18:18
rlandyfyi - internal zuul looks down19:00
rlandy"Something went wrong."19:12
rlandyhow informative zuul19:12
rfolco|ruck:-/19:14
rlandyback now19:15
*** jfrancoa has quit IRC19:36
*** aakarsh has quit IRC19:57
*** Vorrtex has quit IRC20:25
*** dsneddon has quit IRC20:39
*** Goneri has quit IRC21:13
*** aakarsh has joined #oooq21:17
*** vinaykns has quit IRC22:19
*** dsneddon has joined #oooq23:44
*** dsneddon has quit IRC23:51

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!