Tuesday, 2019-09-17

*** Goneri has quit IRC01:31
*** weshay has quit IRC01:56
*** apetrich has quit IRC02:11
*** rfolco has quit IRC02:28
*** ykarel|away has joined #oooq02:28
*** jschlueter has quit IRC02:50
*** jschlueter has joined #oooq02:56
*** jschlueter has quit IRC03:18
*** jschlueter has joined #oooq03:24
*** aakarsh|2 has quit IRC03:40
*** surpatil has joined #oooq04:13
*** ratailor has joined #oooq04:29
*** ykarel|away is now known as ykarel04:49
*** udesale has joined #oooq05:01
*** skramaja has joined #oooq05:03
*** soniya29 has joined #oooq05:14
*** marios has joined #oooq05:16
*** ykarel has quit IRC05:18
*** dsneddon has quit IRC05:21
*** dsneddon has joined #oooq05:28
*** dsneddon has quit IRC05:34
*** sshnaidm|afk is now known as sshnaidm|pto05:35
*** holser has joined #oooq05:37
*** ykarel has joined #oooq05:38
*** Tengu has joined #oooq05:53
*** sanjayu_ has joined #oooq05:54
*** sanjayu_ has quit IRC06:00
*** saneax has joined #oooq06:01
*** jtomasek has joined #oooq06:23
*** holser has quit IRC06:25
*** kopecmartin|off is now known as kopecmartin06:31
*** akahat has joined #oooq07:00
*** jbadiapa has quit IRC07:08
*** ratailor has quit IRC07:21
*** ratailor has joined #oooq07:22
*** jbadiapa has joined #oooq07:24
*** tosky has joined #oooq07:27
mariosarxcruz: sync in 2 mins as per calendar k?07:27
mariosbrb07:27
*** saneax has quit IRC07:28
arxcruzmarios: 2 min for a coffee07:30
mariosarxcruz: ack joining take yr time07:30
arxcruzmarios: joined07:32
*** apetrich has joined #oooq07:40
zbrmorning07:41
*** jpena|off is now known as jpena07:46
*** ykarel is now known as ykarel|lunch07:48
chandankumarzbr: morning07:59
chandankumarzbr: https://review.rdoproject.org/r/#/c/22305/ please have a look at this patch07:59
zbrfixing now08:00
chandankumarthanks!08:02
*** amoralej|off is now known as amoralej08:02
*** bogdando has joined #oooq08:10
*** holser has joined #oooq08:14
*** derekh has joined #oooq08:33
zbrchandankumar: marios : i updated ^ / ready08:53
marioszbr: noting will check in a bit09:06
*** ykarel|lunch is now known as ykarel09:08
*** ratailor has quit IRC09:23
*** ratailor has joined #oooq09:24
*** saneax has joined #oooq09:24
*** sanjayu_ has joined #oooq09:25
*** saneax has quit IRC09:29
*** dtantsur|afk is now known as dtantsur10:07
*** surpatil has quit IRC10:13
*** recheck has quit IRC10:17
*** recheck has joined #oooq10:18
*** recheck has quit IRC10:22
*** recheck has joined #oooq10:23
*** recheck has quit IRC10:24
*** ratailor has quit IRC10:25
*** recheck has joined #oooq10:25
*** ratailor has joined #oooq10:26
*** recheck has quit IRC10:28
*** recheck has joined #oooq10:29
*** recheck has quit IRC10:31
*** recheck has joined #oooq10:31
*** udesale has quit IRC10:34
*** udesale has joined #oooq10:34
*** recheck has quit IRC10:37
*** recheck has joined #oooq10:38
*** recheck has quit IRC10:41
*** recheck has joined #oooq10:41
recheck[gh-ansible] v2.8.5 → https://github.com/ansible/ansible/releases/tag/v2.8.511:09
recheck[gh-ansible] v2.9.0b1 → https://github.com/ansible/ansible/releases/tag/v2.9.0b111:09
recheck[gh-ansible] stable-2.9-branchpoint → https://github.com/ansible/ansible/releases/tag/stable-2.9-branchpoint11:09
zbrso it worked, cool.11:09
chandankumarrecheck: recheck11:10
chandankumarzbr: Can I abandon this review https://review.rdoproject.org/r/22299 in favor of https://review.rdoproject.org/r/22305 ?11:13
*** jpena is now known as jpena|lunch11:17
*** amoralej is now known as amoralej|lunch11:17
zbrchandankumar: done workflowed, but if you could also check that it works as desired is would be cool.11:22
chandankumarzbr: sure11:23
*** ratailor has quit IRC11:27
*** ratailor has joined #oooq11:27
chandankumarzbr: I think we screwed up again11:31
*** brault has joined #oooq11:31
*** pierreprinetti has joined #oooq11:33
*** weshay has joined #oooq11:33
weshaypanda, please renick11:35
weshaypanda, stein is falsely going to red.. http://rhos-release.virt.bos.redhat.com:3030/rhosp11:36
weshaypanda,  please get that fixed11:36
*** brault has quit IRC11:36
*** dsneddon has joined #oooq11:37
*** panda is now known as panda|ruck11:41
*** dsneddon has quit IRC11:42
*** sanjayu_ has quit IRC11:44
recheck[gh-pytest-molecule] h2o → https://github.com/pycontribs/pytest-molecule/releases/tag/1.2.011:51
recheck[gh-pytest-molecule] 1.1.0 → https://github.com/pycontribs/pytest-molecule/releases/tag/1.1.011:51
recheck[gh-pytest-molecule] 1.1.0rc6: Add option to add markers for unavailable drivers → https://github.com/pycontribs/pytest-molecule/releases/tag/1.1.0rc611:51
recheck[gh-molecule] Molecule Modularized → https://github.com/ansible/molecule/releases/tag/3.0a111:51
recheck[gh-molecule] 3.0a0: azure: assure we respect locally configured region (#2243) → https://github.com/ansible/molecule/releases/tag/3.0a011:51
recheck[gh-molecule] 2.22 → https://github.com/ansible/molecule/releases/tag/2.2211:51
recheck[gh-selinux] 0.1.6 → https://github.com/pycontribs/selinux/releases/tag/0.1.611:51
recheck[gh-selinux] 0.1.5 → https://github.com/pycontribs/selinux/releases/tag/0.1.511:51
recheck[gh-selinux] 0.1.5rc5 → https://github.com/pycontribs/selinux/releases/tag/0.1.5rc511:51
weshaypanda|ruck, gate queue is also at 18hr11:59
weshaywell 1511:59
weshayhrm.. and it just reset12:00
*** rfolco has joined #oooq12:04
panda|ruckweshay: deploys are timing out on overcloud. There is no particular step that takes too much, everything seems to proceed very slowly.12:05
weshaypanda|ruck,  k.. we have ara on the overcloud, but it doesn't render due to the new logging12:06
weshayara would help us see what the issue could be.. although still painful12:06
chandankumarweshay: panda|ruck https://review.rdoproject.org/r/#/c/22333/ please merge this12:06
weshaypanda|ruck, we may want to compare job run times to the same job in rdo 3rd party12:06
weshaychandankumar,  all I have is +112:07
panda|ruckweshay: stein promoting now, I thought I fixed this yesterday, but I needed to change the timestamp on the report a bit.12:07
weshaypanda|ruck, k.. thank12:07
weshaythanks12:07
panda|ruckchandankumar: reverted12:08
chandankumarpanda|ruck: yes, please +w it12:08
*** matbu has quit IRC12:12
*** rlandy has joined #oooq12:18
*** matbu has joined #oooq12:18
weshayya.. fyi all master promotions have been blocked for a couple days now..12:19
weshay2019-09-17 00:45:19.809305 | primary |   "msg": "Logging into trunk.registry.rdoproject.org for user unused failed - 401 Client Error: Unauthorized (\"Get https://trunk.registry.rdoproject.org/v2/: unauthorized: authentication required\")"12:19
rlandyzbr: hey12:19
rlandythanks for the molecule fix12:19
rlandytrying the tests again12:19
rlandyoh - I see you rechecked12:21
chandankumarweshay: it is a known issue12:21
weshayaye12:22
weshayyes12:22
chandankumarzbr: https://review.rdoproject.org/r/#/c/22334/12:23
*** soniya29 has quit IRC12:26
*** openstackstatus has quit IRC12:28
*** openstack has joined #oooq12:29
*** ChanServ sets mode: +o openstack12:29
*** sanjayu_ has joined #oooq12:33
rlandyzbr: I thought we made molecule tests voting???12:36
*** jpena|lunch is now known as jpena12:37
*** Goneri has joined #oooq12:39
*** ratailor has joined #oooq12:43
*** amoralej|lunch is now known as amoralej12:44
chandankumarrlandy: panda|ruck https://review.rdoproject.org/r/#/c/22334/12:46
rlandyrfolco: ok - got passing jobs here ... https://review.rdoproject.org/r/#/c/22255/ - you can rebase on that12:48
rfolcorlandy, cool thank you12:59
mariosrfolco: rlandy13:00
mariossync13:00
rfolcogoing13:00
rfolcobj :(13:01
*** ratailor has quit IRC13:08
*** dsneddon has joined #oooq13:13
*** akahat has quit IRC13:20
*** aakarsh|2 has joined #oooq13:21
weshaycommunity call13:32
sshnaidm|ptohttps://www.phoronix.com/scan.php?page=news_item&px=CentOS-8.0-Next-Week13:42
sshnaidm|pto2019-09-16 The final release will be on September 24.13:43
weshayaye13:47
*** sanjayu_ has quit IRC13:48
chandankumarweshay: rlandy panda|ruck https://review.rdoproject.org/r/#/c/22334/13:50
*** ykarel is now known as ykarel|afk13:56
*** migi has joined #oooq14:05
*** beagles has joined #oooq14:06
rfolcomigi, I think that last link is for the 1st level vm, which for CI jobs is not used.... nodepool spawns the vm for us with whatever is available in that node. If nested is enabled, kvm virt will be used.14:07
rfolcomigi, ci upstream I mean... I am not quite sure if centos.ci has baremetal or other jobs that runs that code... maybe rlandy can tell14:08
migirfolco: right but if 1st level is not used then second level won't have knowledge as well, right?14:09
rfolcomigi, the first link I sent you is the set-libvirt-type role, which runs in the ci job upstream (inside the vm) and tries to enable nested/kvm for the 2nd level vms14:10
rfolcoif available14:10
rlandychandankumar: is https://review.rdoproject.org/r/#/c/22334/ tested anywhere we can see the login?14:10
rlandynevermind config14:10
rlandyok - we go on faith14:10
rfolcomigi, in the upstream case w/ zuul, some cloud providers support nested/kvm, others don't. So we set qemu as default and set kvm as virt type when available/enabled.14:11
rfolcomigi, we cannot modify host's module/cpu flags when it's a nodepool vm14:12
rlandyrfolco: migi - baremetal only run a virt undercloud14:12
rfolcorlandy, in which case this runs ? https://github.com/openstack/tripleo-quickstart/blob/master/roles/parts/kvm/tasks/main.yml#L3314:15
rlandyif you select kvm14:18
rlandyI assume14:18
rfolcorlandy, bm jobs only right?14:18
rlandywhere does bm come into this?14:18
rfolcorlandy, this seems to be first level, meant to be run on the host, not inside a nodepool vm14:19
rfolcorlandy, not first level, host level I mean14:20
rlandyso then bm would not kick it14:20
rlandythe overcloud is deployed flat on the baremetal14:21
rlandyrfolco: marios: fyi - testing out using meta for local registry now14:24
rlandywill update patch https://review.rdoproject.org/r/#/c/22255/14:25
*** Vorrtex has joined #oooq14:27
rfolcomigi, I am not quite sure where that code runs.... the other code I know, overcloud deploy for ci jobs running in zuul, which will enable nested virt if available/enabled in that provider.14:28
*** ykarel|afk is now known as ykarel14:28
zbrrlandy: rfolco hi! did my new release of pytest-molecule fixed your job?14:30
rlandyzbr++14:30
rlandyyes - thank you14:30
rfolcozbr, you da man14:30
rfolcozbr++14:31
rfolcoits a pitty bot is not recording kudos14:31
zbrrlandy:  yeah... being able to break so many things with a single release of: molecule, pytest-molecule or selinux gives me nitghmares.14:31
rlandyzbr: what it is to have power?!14:32
zbrmaybe i should invite few of you to these repos, so you can backup me in case something happens to me14:34
rfolcorlandy, can I rebase now or wait for next ps ?14:37
rfolcorlandy, will rebase anyway, can fix minor conflicts later14:38
rlandyrfolco: go ahead an rebase now14:39
rfolcothx14:39
rlandyI'll ping you when the next ps is ready14:40
rfolcorlandy, ack thanks14:40
*** dtantsur is now known as dtantsur|afk14:52
weshaypanda|ruck,  need a few more min15:02
panda|ruckchandankumar: where's the new dlrn password ?15:04
panda|ruckchandankumar: promotion is failing without15:05
weshaypanda|ruck, I'm in15:06
panda|ruckaaaahh, google meet ...15:08
panda|ruck-_-15:08
panda|ruckgimme a sec15:08
*** ykarel is now known as ykarel|away15:08
weshaynp15:08
*** udesale has quit IRC15:08
weshayI can switch if you want15:08
*** Vorrtex has quit IRC15:09
weshaychandankumar, you around? we need the new registry ssh key in the infra doc15:17
chandankumarweshay: yes15:21
chandankumarweshay: https://review.rdoproject.org/r/2230215:21
chandankumarpanda|ruck: please merge this https://review.rdoproject.org/r/2233415:22
panda|ruckchandankumar: I need the plain text15:22
weshaychandankumar,  that is salted15:22
chandankumarpanda|ruck: may be jpena can help15:22
beaglesI'm running into a problem with quickstart where it is running out of space in the image when running "modify-image: Run virt-customize on the provided image" - ring a bell with anyone?15:37
weshayFYI.. centos 7.7 was released15:37
weshaywe need a master / stein promotion15:37
rlandyrdo-tox-molecule queued (non-voting)15:38
*** aakarsh|2 is now known as aakarsh15:38
rlandyrfolco: ^^ hmmm - can't get a node to run molecule15:38
rlandyworked on my local run though15:38
rfolcodammn15:39
rlandyrfolco: I think it's a safe change15:40
rlandyrun on rdo vm looks ok15:40
rlandydoing namespace check now15:41
*** ykarel_ has joined #oooq15:48
*** ykarel|away has quit IRC15:50
*** ykarel_ has quit IRC15:54
rlandypanda|ruck: can we chat for 5 minutes before you log off - question about adding namespace15:56
beaglesrlandy, chandankumar: does that modify-image error seem familiar?15:57
beaglesmentioned ^^15:57
chandankumarjpena: zbr https://review.rdoproject.org/r/22334 please have a look15:57
weshaychandankumar, jpena we need promotion jobs back online asap.. as 7.7 is dropping in.. it hasn't hit mirrors yet15:58
weshay<weshay> hrm.. I don't think the centos base updates for 7.7 have hit the openstack mirrors yet15:58
weshay<weshay> http://mirror.regionone.fortnebula.opendev.org/centos/7/os/x86_64/Packages/NetworkManager-1.12.0-6.el7.x86_64.rpm15:58
weshay<weshay> http://mirror.regionone.fortnebula.opendev.org/centos/7/cr/x86_64/Packages/NetworkManager-1.18.0-5.el7.x86_64.rpm15:58
chandankumarbeagles: sorry do we have logs?15:58
beagles1s15:58
zbrchandankumar: jpena: based on my testing quiet did noting, neither the loop control label name. both were supposed to avoid leaking the loop vars15:58
beagleschandankumar: http://paste.openstack.org/show/777173/15:59
zbri was not able find any way to secure content of vars from loop with ansible, maybe only if we build a censored version in another task, one with no_log.15:59
beagleschandankumar: the log from the ansible task is http://paste.openstack.org/show/777174/16:00
chandankumarbeagles: as per the error, it appears that there is not much space on the system16:03
*** marios is now known as marios|out16:03
weshayhrm.. this is in the main repo though.. http://mirror.centos.org/centos/7/os/x86_64/Packages/centos-release-7-7.1908.0.el7.centos.x86_64.rpm16:04
beagleschandankumar: mm..16:05
chandankumarrlandy: ^^ please have a look at above issue16:06
beagleschandankumar: that would be the filesystem in the image itself wouldn't it?16:06
chandankumarmight be16:07
beaglesit was actually fine sometime last week - I wonder if an rpm dep got changed and its pulling in a bunch of extra stuff16:07
beaglesbox is one of those dsal rigs should be lots of space16:09
chandankumarzbr: can you take a look at beagles issue?16:10
zbrbeagles: where is the _home_stack_repo_setup.sh.log ?16:12
beagleszbr: do you mean this output http://paste.openstack.org/show/777173/ - do you need the entire log?16:12
rlandylooking16:12
*** bogdando has quit IRC16:13
zbrbeagles: ahh, ok should be enough. how does you fs looks like?16:13
beagleszbr http://paste.openstack.org/show/777175/16:14
rlandychandankumar: the mirror issue?16:14
zbrconsidering that you downlaod ~371MB, i suppose that you will need at least 1GB free. i suppose that task runs in a VM which has a small disk.16:15
zbri do not see any reason to worry about space on you host.16:15
*** marios|out has quit IRC16:15
zbrthe image you are trying to customize is too small, is not your host disk.16:15
beaglesright16:16
zbri need to see more than this, as this image is running out of space. no idea where its size is determined from.16:18
zbrbut i remember something about image size being bumped at some point,16:19
beagleszbr: after I ran into this the first time - I wiped out /var/cache/tripleo-quickstart and reran to try and make sure it was pullin gthe latest image down16:19
rlandyrfolco: I'm adding the namespace pieces now16:19
rlandydo you want me to do that in a separate patch16:19
* rlandy thinks so16:19
rfolcorlandy, cool, I rebased and local molecule is working, running tox16:19
rfolcorlandy, nooo, let simplify this into these 2 patches16:20
zbrbeagles: see if you can pot a longer logs so i can figure out where it comes from.16:20
rlandyrfolco: ok - will put in a new commit on the patch I have16:20
rlandycoming up16:20
zbri am a bit surprised to see that modify-image role does not have anything related to diks size, lots of features but not related to image size.16:20
rfolcorlandy, ok, so I can rebase on it again later16:21
rlandyrebooting16:29
chandankumarjpena: after discussing with zbr we will be merging https://review.rdoproject.org/r/#/c/22299/ this review16:30
*** rlandy has quit IRC16:30
chandankumarzbr: please add the votes on ^^16:30
jpenachandankumar: ack for me. Let's test it before merging the secret change16:30
zbrjpena: that is what I wanted to say. let if be bad, so we can see it failing first.16:31
* beagles wonders if I'm caught in a centos7.7 related snag16:33
beaglesthis is a yum update16:33
beaglesand is updating the world - the image is really supposed to be huge so should have lots of room16:34
beagles(well 4G isn't huge but...)16:34
*** rlandy has joined #oooq16:35
*** derekh has quit IRC16:41
recheck[gh-molecule] Molecule Modularized → https://github.com/ansible/molecule/releases/tag/3.0a216:49
chandankumarzbr: jpena this time it works, without leaking secrets16:51
*** rfolco is now known as rfolco|dentist16:51
*** brault has joined #oooq16:52
chandankumarsshnaidm|pto: zbr: jpena thanks for all the help16:56
*** jpena is now known as jpena|off16:59
*** amoralej is now known as amoralej|off17:00
rlandyrfolco|dentist: fyi ... https://review.rdoproject.org/r/#/c/22255/12..1317:03
rlandynamespace added17:03
chandankumarzbr: https://logs.rdoproject.org/90/22290/7/check/tripleo-ci-rhel-8-standalone-rdo/7167bce/job-output.txt.gz#_2019-09-17_16_56_16_74143317:04
chandankumarthis one does not looks good17:04
*** skramaja has quit IRC17:05
*** kopecmartin is now known as kopecmartin|off17:06
zbrchandankumar: seems secured to me, but I do not see the later printing of failure17:09
*** brault has quit IRC17:10
rlandymake that patch 1417:11
*** recheck has quit IRC17:49
*** recheck has joined #oooq17:53
*** holser has quit IRC18:30
*** Vorrtex has joined #oooq18:30
*** brault has joined #oooq18:42
*** pierreprinetti has quit IRC19:09
*** pierreprinetti has joined #oooq19:09
*** brault has quit IRC19:16
rlandyhmmm ...19:19
rlandyno node for rdo-molecule :(19:20
rlandyzbr: what node does the molecule job run on?19:26
*** brault has joined #oooq19:26
rlandythe report gives no information on thta19:26
rlandythat19:27
rlandy    tox-linters SUCCESS in 3m 00s19:31
rlandy    tripleo-ci-promotion-staging SUCCESS in 3m 51s (non-voting)19:31
rlandy    rdo-tox-molecule rdo-tox-molecule : NODE_FAILURE in 0s (non-voting)19:31
rlandy    tox-jjb SUCCESS in 2m 03s19:31
rlandy    tox-py27 SUCCESS in 3m 27s19:31
rlandy    tox-py36 SUCCESS in 3m 52s19:31
rlandy^^ node failure19:31
rlandyrfolco|dentist: ^^ fyi19:32
*** brault has quit IRC19:32
rlandyalso no molecule job on https://review.rdoproject.org/r/#/c/21872/19:33
rlandytopic updated ... https://review.rdoproject.org/r/#/q/topic:staging-promoter+(status:open+OR+status:merged)19:34
*** holser has joined #oooq19:49
*** jtomasek has quit IRC20:11
zbrrlandy: runs on fedora nodes.20:18
rlandyzbr: hmmm ... we're fresh out of those :(20:18
zbrbut once we have centos-8 we can switch to it.20:18
zbrc7 does not have good py3, and the docker on it is too buggy to be trusted.20:19
zbrbut in general it could be any machine with decent docker version.20:19
panda|ruckthank god, stein finally promoted20:22
panda|ruckrfolco|dentist: promoter server is fubar, I will need to speak to you urgently20:23
weshaypanda|ruck,  perfect :)   we may have the root cause to the gate failures20:25
weshaypanda|ruck,  get out of here20:25
panda|ruckweshay: I was able to create this for ara https://review.opendev.org/68267920:26
panda|ruckweshay: I'm not sure if this is what we're missing, but the command to create the reports are still in collect logs20:26
panda|ruckweshay:  and that seems the vairable that enables them20:27
weshaypanda|ruck,  ah nice.. we had the old code :)20:27
weshayhrm20:28
panda|rucknot sure if it's enough20:28
weshayit's not20:28
*** holser has quit IRC20:29
weshaypanda|ruck, get some rest.. start looking tomorrow20:29
panda|ruckall right, good night20:29
*** panda|ruck is now known as panda|ruck|off20:29
*** weshay is now known as weshay|ruck20:32
weshay|ruckrhel 8 standalone running in 1 hour 11min on that patch20:34
weshay|ruckmaybe centos8 will fix all our problems ;)20:35
*** Goneri has quit IRC20:46
*** brault has joined #oooq20:48
*** brault has quit IRC20:52
*** brault has joined #oooq20:57
*** brault has quit IRC21:01
*** Vorrtex has quit IRC21:21
*** rfolco|dentist is now known as rfolco21:23
rfolcopanda|ruck|off, I didn't want to wake you up... if there is anything urgent you need to tell me about promoter, drop me a msg21:30
rfolcoweshay|ruck, you know what panda means by promoter is fubar?21:31
weshay|ruckrfolco,  ya.. the auth key changed.. and we didn't get it changed in all the places it needed to be21:33
weshay|ruckit's working now21:33
rfolcoweshay|ruck, ok thx21:34
rfolcorlandy, zbr what about centos7 node w/ docker-ce ? we are installing docker-ce anyway for the staging env21:35
rlandyidk - anything that will run atm21:35
rlandyrfolco: I have just moved on to starting the tests21:35
rlandywhile we wait for available nodes21:35
rlandywill try again later tonight21:36
rfolcorlandy, ok, will try on centos7 tonight21:36
*** aakarsh has quit IRC21:54
*** tosky has quit IRC23:11
*** dsneddon has quit IRC23:35
*** rlandy has quit IRC23:51

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!