| *** dsneddon has quit IRC | 00:34 | |
| *** dsneddon has joined #oooq | 01:02 | |
| *** bhagyashris has joined #oooq | 01:12 | |
| *** Goneri has quit IRC | 01:35 | |
| *** apetrich has quit IRC | 02:10 | |
| *** dsneddon has quit IRC | 02:24 | |
| *** dsneddon has joined #oooq | 02:56 | |
| *** dsneddon has quit IRC | 03:00 | |
| *** gkadam has joined #oooq | 03:19 | |
| *** dsneddon has joined #oooq | 03:29 | |
| *** dsneddon has quit IRC | 03:34 | |
| *** dsneddon has joined #oooq | 04:01 | |
| *** raukadah is now known as chkumar|rover | 04:04 | |
| *** dsneddon has quit IRC | 04:05 | |
| *** surpatil has joined #oooq | 04:22 | |
| *** udesale has joined #oooq | 04:23 | |
| *** ratailor has joined #oooq | 04:34 | |
| *** dsneddon has joined #oooq | 04:34 | |
| *** dsneddon has quit IRC | 04:39 | |
| *** skramaja has joined #oooq | 04:55 | |
| *** jaosorior has joined #oooq | 05:01 | |
| *** dsneddon has joined #oooq | 05:13 | |
| *** surpatil has quit IRC | 05:17 | |
| *** dsneddon has quit IRC | 05:18 | |
| *** apetrich has joined #oooq | 05:42 | |
| *** dsneddon has joined #oooq | 05:45 | |
| *** jbadiapa has quit IRC | 05:47 | |
| *** jbadiapa has joined #oooq | 05:48 | |
| *** dsneddon has quit IRC | 05:50 | |
| *** aakarsh has quit IRC | 05:51 | |
| *** aakarsh has joined #oooq | 05:51 | |
| *** udesale has quit IRC | 05:55 | |
| *** kopecmartin|off is now known as kopecmartin | 05:55 | |
| *** udesale has joined #oooq | 05:55 | |
| *** surpatil has joined #oooq | 06:02 | |
| *** udesale has quit IRC | 06:06 | |
| *** skramaja has quit IRC | 06:08 | |
| *** skramaja has joined #oooq | 06:11 | |
| *** saneax has joined #oooq | 06:18 | |
| *** dsneddon has joined #oooq | 06:19 | |
| *** dsneddon has quit IRC | 06:24 | |
| *** surpatil has quit IRC | 06:36 | |
| *** gkadam has quit IRC | 06:43 | |
| *** amoralej|off is now known as amoralej | 06:50 | |
| *** jpena|off is now known as jpena | 06:59 | |
| *** dsneddon has joined #oooq | 07:00 | |
| *** dsneddon has quit IRC | 07:07 | |
| *** jtomasek has joined #oooq | 07:29 | |
| *** chem` has joined #oooq | 07:36 | |
| *** chem has quit IRC | 07:37 | |
| *** dsneddon has joined #oooq | 07:37 | |
| *** dsneddon has quit IRC | 07:42 | |
| *** jtomasek has quit IRC | 07:42 | |
| *** aakarsh has quit IRC | 08:08 | |
| *** aakarsh has joined #oooq | 08:09 | |
| *** dtantsur|afk is now known as dtantsur | 08:10 | |
| *** dsneddon has joined #oooq | 08:12 | |
| *** dsneddon has quit IRC | 08:17 | |
| *** aakarsh|2 has joined #oooq | 08:28 | |
| chkumar|rover | panda: Hello | 08:29 |
|---|---|---|
| panda | chkumar|rover: hu | 08:29 |
| chkumar|rover | panda: please have a look at this review https://review.opendev.org/#/c/678008/ thanks! | 08:29 |
| *** aakarsh has quit IRC | 08:29 | |
| panda | chkumar|rover: Why just a related-bug ? | 08:34 |
| chkumar|rover | panda: it is a workaround to fix the test | 08:37 |
| chkumar|rover | that's why it is related-bug | 08:37 |
| *** ksambor has quit IRC | 08:39 | |
| *** ksambor has joined #oooq | 08:39 | |
| *** dsneddon has joined #oooq | 08:44 | |
| *** ksambor has quit IRC | 08:47 | |
| *** ksambor has joined #oooq | 08:47 | |
| *** dsneddon has quit IRC | 08:48 | |
| *** dsneddon has joined #oooq | 09:22 | |
| *** dsneddon has quit IRC | 09:27 | |
| chkumar|rover | amoralej: does stein phase 1 works or still stuck at tempest? | 09:51 |
| *** dsneddon has joined #oooq | 10:00 | |
| *** dsneddon has quit IRC | 10:05 | |
| zbr | i remember someone used to have a script to query zuul about current patch build status, who was? | 10:06 |
| zbr | i kinda need one. | 10:06 |
| chkumar|rover | zbr: marios, check the tool | 10:06 |
| *** jaosorior has quit IRC | 10:10 | |
| chkumar|rover | kopecmartin: https://review.rdoproject.org/r/#/c/21879/ commented here | 10:13 |
| kopecmartin | ok, thanks | 10:15 |
| chkumar|rover | zbr: https://github.com/marios/tripleo_ruck_job_tool/blob/master/oooci-jobs.sh#L120 | 10:17 |
| zbr | ahh, not really what i was looking for. | 10:23 |
| zbr | it was more of "show me the build status of current patch", like if jobs are in queue, building or already ended. | 10:23 |
| *** jaosorior has joined #oooq | 10:27 | |
| *** bhagyashris has quit IRC | 10:30 | |
| *** dsneddon has joined #oooq | 10:31 | |
| *** dsneddon has quit IRC | 10:36 | |
| amoralej | sorry chkumar|rover nop, i was in a call | 10:42 |
| amoralej | chkumar|rover, is it affecting oooq jobs also? | 10:42 |
| amoralej | i understood not | 10:42 |
| amoralej | only p-o-i | 10:43 |
| chkumar|rover | amoralej: oooq jobs are fine with some other failures but not related to tempest timeout | 10:43 |
| amoralej | ack | 10:44 |
| chkumar|rover | kopecmartin: commented back | 10:57 |
| rfolco | panda, o/ | 11:04 |
| rfolco | panda, are you working on the credentials part of the provisioning ? | 11:05 |
| *** apetrich has quit IRC | 11:08 | |
| *** apetrich has joined #oooq | 11:09 | |
| *** dsneddon has joined #oooq | 11:09 | |
| *** rfolco is now known as rfolco|brb | 11:10 | |
| *** tesseract has joined #oooq | 11:12 | |
| panda | rfolco|brb: yes, wanna sync ? | 11:13 |
| *** dsneddon has quit IRC | 11:14 | |
| *** skramaja has quit IRC | 11:16 | |
| *** rfolco|brb is now known as rfolco | 11:17 | |
| rfolco | panda, sure | 11:18 |
| rfolco | panda, my bj? | 11:18 |
| panda | ok | 11:20 |
| *** apetrich has quit IRC | 11:24 | |
| *** apetrich has joined #oooq | 11:25 | |
| *** jaosorior has quit IRC | 11:26 | |
| chkumar|rover | kopecmartin: arxcruz Feel free to add/remove/update the agenda for today's tempest meeting https://etherpad.openstack.org/p/tripleo-tempest-squad-meeting | 11:29 |
| *** jpena is now known as jpena|lunch | 11:32 | |
| *** d0ugal has quit IRC | 11:33 | |
| *** dsneddon has joined #oooq | 11:43 | |
| *** jaosorior has joined #oooq | 11:44 | |
| chkumar|rover | zbr: Hello | 11:45 |
| chkumar|rover | zbr: for scenario 1 standalone RHEL-8 https://review.opendev.org/#/c/668851/ | 11:46 |
| chkumar|rover | otherworkaround is to remove this deprecated file from tht | 11:46 |
| chkumar|rover | https://github.com/openstack/tripleo-heat-templates/blob/master/ci/environments/scenario001-standalone.yaml#L30 | 11:47 |
| chkumar|rover | or set this to OS::Heat::None | 11:47 |
| chkumar|rover | will fix the issue | 11:47 |
| chkumar|rover | for fluented missing container | 11:48 |
| *** dsneddon has quit IRC | 11:48 | |
| *** d0ugal has joined #oooq | 11:49 | |
| zbr | chkumar|rover: thanks. is martin still working on that change? seems far from ready, all jobs fail. | 12:04 |
| chkumar|rover | zbr: I poked martin internally, waiting for the reply | 12:11 |
| chkumar|rover | zbr: in mean time, we can add the workaround and let it go | 12:12 |
| weshay | 0/ | 12:12 |
| weshay | couldn't log into my irc.. phew | 12:12 |
| zbr | chkumar|rover: is this one lines enough? | 12:13 |
| chkumar|rover | zbr: I think so | 12:13 |
| chkumar|rover | zbr: we can interate over it | 12:13 |
| zbr | let me try to make a change for it | 12:13 |
| chkumar|rover | anyway check job is red | 12:14 |
| chkumar|rover | due to broken standalone scenario 1 | 12:14 |
| weshay | chkumar|rover the fix is NOT to remove fluentd | 12:14 |
| chkumar|rover | https://review.opendev.org/#/c/674556/ | 12:14 |
| weshay | ah.. maybe they are going to do it | 12:15 |
| chkumar|rover | weshay: yes | 12:15 |
| chkumar|rover | weshay: waiting for rlandry then we can sync | 12:15 |
| weshay | chkumar|rover ok.. note fluentd is in scen1,2,3 | 12:15 |
| weshay | zbr chkumar|rover is rover.. zbr you should be driving this | 12:15 |
| zbr | weshay: i do not see it in 3, in config. | 12:15 |
| *** surpatil has joined #oooq | 12:15 | |
| weshay | zbr hrm.. ok.. it's early for me.. maybe it's not in 3 | 12:16 |
| zbr | but i see it in 1+2, and I suspect i need to diable it in all. | 12:16 |
| weshay | ya.. any where we find it.. correct | 12:16 |
| zbr | including multinode | 12:16 |
| chkumar|rover | weshay: zbr http://codesearch.openstack.org/?q=..%2F..%2Fdeployment%2Fdeprecated%2Flogging%2Ffluentd-container-puppet.yaml&i=nope&files=&repos= | 12:17 |
| zbr | so mainly we disable the service instead of removing it, as is much faster. | 12:17 |
| *** rlandy has joined #oooq | 12:20 | |
| *** rlandy is now known as rlandy|ruck | 12:20 | |
| rlandy|ruck | chkumar|rover: hi - welcome back! | 12:21 |
| chkumar|rover | rlandy|ruck: :-) | 12:21 |
| rlandy|ruck | chkumar|rover: ok - if I merge https://review.opendev.org/#/c/678008/? | 12:21 |
| chkumar|rover | rlandy|ruck: please wait | 12:21 |
| chkumar|rover | rlandy|ruck: let this https://review.opendev.org/#/c/674556/ gets merged first | 12:21 |
| chkumar|rover | check/gate rhel-7 standalone scenario1 is broken | 12:22 |
| *** dsneddon has joined #oooq | 12:22 | |
| rlandy|ruck | chkumar|rover: k - I can only +1 there | 12:22 |
| chkumar|rover | it will take around 2 and half hour to get it mergedf | 12:23 |
| chkumar|rover | rlandy|ruck: weshay want to sync? | 12:23 |
| rlandy|ruck | chkumar|rover: sure | 12:23 |
| chkumar|rover | rlandy|ruck: weshay https://bluejeans.com/1571313919 | 12:24 |
| *** dsneddon has quit IRC | 12:27 | |
| zbr | weshay: fyi, my base-py2 / base-py3 atom addition to bindep got merged. it should make updating bindep files easier. https://review.opendev.org/#/c/639951/ | 12:38 |
| weshay | nice | 12:39 |
| zbr | weshay: i guess we should merge https://review.opendev.org/#/c/678038/ right? even if we cannot enable scenarios for other reasons. | 12:41 |
| weshay | zbr as long as the jobs are non-voting.. the zuul changes can merge | 12:46 |
| weshay | rlandy|ruck chkumar|rover FYI.. centos-7.7 http://mirror.regionone.rdo-cloud-tripleo.rdoproject.org/centos/7/rt/x86_64/Packages/ | 12:47 |
| weshay | rlandy|ruck chkumar|rover it's active now.. | 12:47 |
| *** udesale has joined #oooq | 12:48 | |
| weshay | arxcruz after you update the skip list for fs21 | 12:48 |
| *** jpena|lunch is now known as jpena | 12:49 | |
| panda | rfolco: local tests completed successfully, now fixing zuul execution | 12:54 |
| rfolco | panda, you da man | 12:54 |
| panda | rfolco: I'll ping you so we can sync on the next steps | 12:54 |
| chkumar|rover | arxcruz: kopecmartin tempest meeting | 12:54 |
| *** dsneddon has joined #oooq | 12:54 | |
| kopecmartin | (y) | 12:54 |
| rfolco | panda, just call my name, I'll be there | 12:54 |
| panda | rfolco: youuuu'll beee theeere, oh yes, you would, coz I got a friend ? | 12:55 |
| chkumar|rover | arxcruz: kopecmartin https://bluejeans.com/1571313919 waiting . | 12:56 |
| rfolco | panda, you got a friend in me (toy story) | 12:57 |
| rfolco | panda, from now on, I'll just answer with songs | 12:58 |
| panda | anybody remembers if there's a way to run playbook sin zuul directly as root ? | 12:58 |
| rlandy|ruck | depending on what the playbook does you may or may not be able to | 12:59 |
| *** dsneddon has quit IRC | 12:59 | |
| rlandy|ruck | on the executor, you will fail out | 12:59 |
| panda | rlandy|ruck: no, root as the ansible_user | 12:59 |
| chkumar|rover | arxcruz: we are waiting for you. | 13:00 |
| panda | the promoter role was historically run as root | 13:00 |
| rlandy|ruck | hmmm | 13:00 |
| panda | I'm workign aound by putting become: everywhere | 13:00 |
| *** amoralej is now known as amoralej|lunch | 13:00 | |
| panda | and become_user: for the "root" runs | 13:01 |
| weshay | arxcruz joining? | 13:01 |
| rlandy|ruck | panda: like setting ansible_user on the hostvar in the zuul job? | 13:01 |
| panda | rlandy|ruck: I can try that | 13:02 |
| rlandy|ruck | panda: see the way we set ansible_python_interpreter in rhel8 jobs | 13:03 |
| *** ratailor has quit IRC | 13:04 | |
| rlandy|ruck | use the same constructs | 13:04 |
| zbr | 7.7 adds pure py3, no longer scl stuff. for centos was not a big deal due to epel, but for rhel is something really good. SCL was a big ***.. | 13:09 |
| zbr | hmm, 7.7 was not released yet. | 13:13 |
| *** jaosorior has quit IRC | 13:14 | |
| *** aakarsh|2 has quit IRC | 13:15 | |
| rlandy|ruck | zbr: you still don't have core ob ci-config? | 13:15 |
| rlandy|ruck | on | 13:15 |
| zbr | rlandy|ruck: on ci-config i have, but not on other repos from rdo (like rdo-jobs) | 13:17 |
| rlandy|ruck | zbr that's https://review.rdoproject.org/r/#/admin/groups/2801,members, I think | 13:17 |
| zbr | yeah, i think that is the group the unlocks the doors of rdo. | 13:17 |
| chkumar|rover | rlandy|ruck: wait, we will do the paper work first, then make him. | 13:18 |
| rlandy|ruck | k - so that is config-core | 13:18 |
| rlandy|ruck | chkumar|rover: no worries I am just checking | 13:18 |
| rlandy|ruck | chkumar|rover: besides zbr says he has +2 on ci-config | 13:18 |
| rlandy|ruck | which was the ask, no? | 13:19 |
| rlandy|ruck | zbr: chkumar|rover: ^^ pls confirm what the ask is here | 13:19 |
| rlandy|ruck | then I will know what process we are going down | 13:19 |
| rlandy|ruck | config-core is a diff group | 13:20 |
| *** Goneri has joined #oooq | 13:20 | |
| *** amoralej|lunch is now known as amoralej | 13:22 | |
| zbr | rlandy|ruck: there is also https://review.rdoproject.org/r/#/admin/groups/6,members | 13:22 |
| zbr | personally I think it would be a good idea to consolidate the two groups into one, but this channel is not the right place to check this. | 13:23 |
| rlandy|ruck | zbr: again, what is being asked for | 13:23 |
| rlandy|ruck | there are reason for those doff groups | 13:24 |
| rlandy|ruck | diff | 13:24 |
| zbr | rlandy|ruck: i am not the right person to answer this question, i do not know the history. | 13:24 |
| rlandy|ruck | chkumar|rover, asked if you had +2 on ci-config - you say you do | 13:24 |
| rlandy|ruck | zbr: although I don;t see your name | 13:25 |
| rlandy|ruck | oh - here you are https://review.rdoproject.org/r/#/admin/groups/3370,members | 13:25 |
| rlandy|ruck | so then we are all set | 13:25 |
| *** dsneddon has joined #oooq | 13:27 | |
| *** ksambor has quit IRC | 13:27 | |
| *** ksambor has joined #oooq | 13:28 | |
| *** dsneddon has quit IRC | 13:31 | |
| zbr | rlandy|ruck: as a note, by looking at rdo changes made by our team in general, ~10-15% are on ci-config, the rest are on: rdo-jobs config *reproducer. | 13:32 |
| rlandy|ruck | hmmm - I spend most of my time editing review.rdoproject.org/config | 13:34 |
| chkumar|rover | zbr: weshay: rlandy|ruck https://review.opendev.org/#/c/668851/ patch updated for fluented | 13:35 |
| *** surpatil has quit IRC | 13:42 | |
| chkumar|rover | rlandy|ruck: zbr is already core at config and ci-config | 13:45 |
| chkumar|rover | he is just missing at rdo-jobs | 13:45 |
| rlandy|ruck | chkumar|rover: so the group there is https://review.rdoproject.org/r/#/admin/groups/6,members | 13:47 |
| * rlandy|ruck asks on #sf-ops | 13:47 | |
| chkumar|rover | rlandy|ruck: we need to send a patch to config repo to update the acls | 13:47 |
| rlandy|ruck | weshay: ^^ ok - this is a diff ask iiuc than what as mentioned this morning | 13:47 |
| rlandy|ruck | chkumar|rover: we can just add to group | 13:48 |
| rlandy|ruck | at least I can | 13:48 |
| *** aakarsh|2 has joined #oooq | 13:48 | |
| zbr | chkumar|rover: correction: i am not core on config, only on ci-config. | 13:51 |
| zbr | that is because i only member of tripleo-ci-core and not config-core: https://review.rdoproject.org/r/#/admin/groups/6,members | 13:51 |
| rlandy|ruck | yep - this is confusing | 13:53 |
| rlandy|ruck | there are a bunch of diff groups here | 13:53 |
| zbr | i do not even see marios listed there and if i remember well, someone was supposed to propose him on rdo-list. | 13:53 |
| rlandy|ruck | yes - that was me | 13:53 |
| zbr | i think that config-core is the main one, giving access to all/most repos. | 13:53 |
| * rlandy|ruck waits to chat with weshay re: all these groups and permissions | 13:54 | |
| rlandy|ruck | make sure everyone is in the right group and propose changes | 13:54 |
| zbr | base on group ID, (6) i suspect this was one of the first groups created on rdo gerrit. | 13:55 |
| rlandy|ruck | weshay: ^^ when you have a moment to review these permission groups | 13:55 |
| rlandy|ruck | not exactly | 13:56 |
| rlandy|ruck | some were rdo core groups we got added to | 13:56 |
| rlandy|ruck | it's s bit like the tripleo situation | 13:56 |
| rlandy|ruck | if added to some groups, you will get core on some sf stuff | 13:56 |
| rlandy|ruck | I don't often use that right | 13:57 |
| *** dsneddon has joined #oooq | 14:06 | |
| *** d0ugal has quit IRC | 14:06 | |
| *** d0ugal has joined #oooq | 14:06 | |
| *** dsneddon has quit IRC | 14:11 | |
| *** dsneddon has joined #oooq | 14:15 | |
| chkumar|rover | rlandy|ruck: weshay I am logging out, I will keep an eye on centos7.7 stuff | 14:24 |
| *** chkumar|rover is now known as raukadah | 14:24 | |
| rlandy|ruck | sure | 14:25 |
| raukadah | Happy Weekend Team, see ya on monday :-) | 14:25 |
| rlandy|ruck | bye | 14:30 |
| amoralej | rlandy|ruck, raukadah i've done some progress with the issue in stein | 14:39 |
| amoralej | but not found RC yet | 14:39 |
| amoralej | i think it's related to watcher | 14:39 |
| amoralej | i'm going on pto, jpena will continue working on this on monday | 14:39 |
| rlandy|ruck | amoralej: thanks | 14:42 |
| *** amoralej is now known as amoralej|off | 14:47 | |
| *** ksambor_ has joined #oooq | 14:50 | |
| *** kopecmartin is now known as kopecmartin|off | 14:51 | |
| *** amoralej|off has quit IRC | 14:51 | |
| *** ksambor has quit IRC | 14:53 | |
| *** ksambor_ is now known as ksambor | 14:53 | |
| rlandy|ruck | wow - something is wrong on rdo | 15:10 |
| rlandy|ruck | panda: you're working on rdocloud post-failure issue? | 15:14 |
| panda | rlandy|ruck: no | 15:14 |
| panda | rlandy|ruck: what's wrong ? | 15:14 |
| rlandy|ruck | <weshay> it's the network issue panda was working on afaict | 15:14 |
| rlandy|ruck | not true | 15:14 |
| rlandy|ruck | will look | 15:14 |
| panda | rlandy|ruck: maybe weshay meant the conflicting IPs | 15:15 |
| weshay | ya | 15:15 |
| weshay | no action required.. just looking at logreduce | 15:15 |
| rlandy|ruck | is that causing the post failures now? | 15:15 |
| rlandy|ruck | periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset002-stein-uploadpost_failure | 15:16 |
| rlandy|ruck | periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-steinpost_failur | 15:16 |
| rlandy|ruck | ^^ that???4 | 15:16 |
| rlandy|ruck | conflicting ips | 15:16 |
| rlandy|ruck | idk so | 15:17 |
| *** dsneddon has quit IRC | 15:18 | |
| rlandy|ruck | 2019-08-23 14:18:25.876731 | TASK [upload-logs : Upload logs to log server] | 15:19 |
| rlandy|ruck | 2019-08-23 15:18:10.114572 | POST-RUN END RESULT_TIMED_OUT: [trusted : review.rdoproject.org/config/playbooks/base-minimal/post-logs.yaml@master] | 15:19 |
| rlandy|ruck | ^^ that is not caused by deploy failure | 15:20 |
| panda | rlandy|ruck: definitely no | 15:24 |
| rlandy|ruck | finally - someone agrees with me | 15:24 |
| panda | rlandy|ruck: and also it's not in our repo | 15:25 |
| panda | the role from base minimal is failing | 15:25 |
| panda | mmmhhh | 15:25 |
| panda | unless, we consumed all the space | 15:25 |
| panda | in the log server | 15:25 |
| rlandy|ruck | panda: see #sf-ops | 15:25 |
| rlandy|ruck | 20 jobs sending logs concurrently | 15:26 |
| *** jpena is now known as jpena|off | 15:32 | |
| rlandy|ruck | weshay: when you are done with the log reduce stuff , I could use some more info on fs021 and centos-7.7 | 15:34 |
| rlandy|ruck | ie: where to confirm that | 15:35 |
| rlandy|ruck | looking around yum files | 15:35 |
| rlandy|ruck | 2019-08-23 13:30:20 | ** Performing Heat stack create.. ** | 15:36 |
| rlandy|ruck | 2019-08-23 13:30:21 | Exception: ERROR: resources.UndercloudServiceChain<file:///home/zuul/tripleo-heat-installer-templates/common/services/undercloud-role.yaml>: The Resource Type (OS::TripleO::LoggingConfiguration) could not be found. | 15:36 |
| panda | rfolco: role working \o/ | 15:40 |
| panda | rfolco: wanna sync ? | 15:40 |
| rfolco | role sheet | 15:41 |
| panda | ?? | 15:41 |
| rfolco | panda, yep | 15:41 |
| rfolco | panda, my blue | 15:41 |
| panda | we always go in your blue | 15:42 |
| rfolco | panda, my blue, my rules | 15:42 |
| panda | rfolco: ok, let's meet each on his own blue | 15:43 |
| *** dsneddon has joined #oooq | 15:48 | |
| rlandy|ruck | weshay: you're looking at overcloud deploy failures? | 15:51 |
| *** Vorrtex has joined #oooq | 15:51 | |
| *** dsneddon has quit IRC | 15:53 | |
| weshay | rlandy|ruck in that log reduce report? | 15:53 |
| weshay | rfolco ready | 15:53 |
| rlandy|ruck | 2019-08-23 15:11:38 | ERROR: configuration of safe defaults failed. | 15:53 |
| rlandy|ruck | weshay: ^^ looking at that error | 15:53 |
| weshay | in logreduce? | 15:53 |
| rlandy|ruck | yep | 15:53 |
| rfolco | weshay, now I need more time, syncing with panda | 15:53 |
| weshay | rfolco ok... just patch me in... and we'll 1-1 after | 15:54 |
| rlandy|ruck | weshay: guess we can sync this afternoon when you have more time | 15:55 |
| rlandy|ruck | re: this and fs021 | 15:55 |
| weshay | rlandy|ruck ya.. I'm not in context yet | 15:55 |
| rlandy|ruck | k - will ping later | 15:56 |
| *** udesale has quit IRC | 15:57 | |
| *** udesale has joined #oooq | 15:58 | |
| *** hamzy has quit IRC | 15:59 | |
| weshay | rfolco can I jump into your call? or wont be helpful | 16:00 |
| *** dtantsur is now known as dtantsur|afk | 16:00 | |
| weshay | rlandy|ruck rdo jobs are all red | 16:01 |
| weshay | is that the timeout? | 16:02 |
| rlandy|ruck | weshay: lol | 16:02 |
| rlandy|ruck | was kicking and creaming about that about half hour ago | 16:03 |
| weshay | sorry.. lots of mtgs | 16:03 |
| rlandy|ruck | weshay: log server was over jammed | 16:03 |
| weshay | bah.... thanks | 16:03 |
| rlandy|ruck | jpena killed some processes for us | 16:03 |
| rlandy|ruck | looking better now | 16:03 |
| rlandy|ruck | on it | 16:03 |
| rlandy|ruck | weshay: we still have some rdocloud failures | 16:04 |
| rlandy|ruck | but for the most part the master promotion is looking more stable now | 16:05 |
| rlandy|ruck | check jobs - jury is still out | 16:05 |
| weshay | k.. thanks for tracking it down | 16:05 |
| rlandy|ruck | overcloud deploy errors are a real thing | 16:05 |
| rlandy|ruck | looking at that | 16:06 |
| rlandy|ruck | brb - lunch | 16:06 |
| *** tesseract has quit IRC | 16:11 | |
| zbr | rlandy|ruck: do you know anything about this failure? https://object-storage-ca-ymq-1.vexxhost.net/v1/86bbbcfa8ad043109d2d7af530225c72/logs_06/678206/1/check/tripleo-ci-centos-7-scenario001-standalone/94bb951/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz | 16:28 |
| *** dsneddon has joined #oooq | 16:28 | |
| zbr | Error: unable to exec into ceph-mon-standalone: no container with name or ID ceph-mon-standalone found: no such container | 16:28 |
| rlandy|ruck | looking | 16:29 |
| *** Vorrtex has quit IRC | 16:29 | |
| zbr | logstash reports ~31 failures in the last 48h, started at midnight. | 16:30 |
| rlandy|ruck | there was a ceph fix | 16:30 |
| rlandy|ruck | oh scenario001 | 16:31 |
| rlandy|ruck | yeah | 16:31 |
| rlandy|ruck | the fix just merged | 16:31 |
| rlandy|ruck | abr: ^^ | 16:31 |
| rlandy|ruck | zbr: ^^ | 16:31 |
| *** dsneddon has quit IRC | 16:32 | |
| zbr | rlandy|ruck: thanks. i rechecked https://review.opendev.org/#/c/678206/1 -- we may want to workflow is if it works. | 16:35 |
| rlandy|ruck | zbr: yep | 16:37 |
| *** Vorrtex has joined #oooq | 16:39 | |
| *** saneax has quit IRC | 16:43 | |
| *** saneax has joined #oooq | 16:45 | |
| rlandy|ruck | k - so now we are looking at overcloud deploy errors | 16:51 |
| rlandy|ruck | 2019-08-23 15:59:04 | Ansible timed out at 3807 seconds. | 16:52 |
| *** saneax has quit IRC | 16:56 | |
| rlandy|ruck | 2019-08-23 15:38:22 | changed: [overcloud-controller-1 -> 192.168.24.21] => { | 16:58 |
| rlandy|ruck | 2019-08-23 15:38:22 | "changed": true, | 16:58 |
| rlandy|ruck | 2019-08-23 15:59:04 | "cmd": "/bin/rsync --delay-updates -F --compress --archive --out-format=<<CHANGED>>%i %n%L /opt/puppetlabs/Overcloud configuration failed. | 16:58 |
| rlandy|ruck | 2019-08-23 15:59:04 | /var/lib/container-puppet/puppetlabs/", | 16:58 |
| rlandy|ruck | 2019-08-23 15:59:04 | "rc": 0 | 16:58 |
| rlandy|ruck | 2019-08-23 15:59:04 | } | 16:58 |
| rlandy|ruck | ^^ 20 mins | 16:58 |
| *** dsneddon has joined #oooq | 17:01 | |
| *** saneax has joined #oooq | 17:01 | |
| rfolco | weshay, ping me if anything, working on the multi-arch tagging patch. Can skip 1x1 today if nothing urgent. | 17:02 |
| *** udesale has quit IRC | 17:05 | |
| *** udesale has joined #oooq | 17:06 | |
| *** dsneddon has quit IRC | 17:06 | |
| *** udesale has quit IRC | 17:11 | |
| *** udesale has joined #oooq | 17:12 | |
| *** dsneddon has joined #oooq | 17:12 | |
| *** saneax has quit IRC | 17:13 | |
| *** saneax has joined #oooq | 17:15 | |
| *** chem` has quit IRC | 17:26 | |
| *** saneax has quit IRC | 17:31 | |
| panda | zbr: does prepare.yml commit changes on the image ? | 17:34 |
| panda | zbr: I still have some leftovers from previous tests even after I destroy | 17:34 |
| zbr | panda: which driver? | 17:35 |
| panda | zbr: docker | 17:35 |
| *** udesale has quit IRC | 17:35 | |
| *** hamzy has joined #oooq | 17:35 | |
| zbr | with docker you should not have any leftovers but maybe destroy did not run? | 17:36 |
| zbr | tbh, I did not use a custom prepare.yml so far, I only using the container building and converge after. still, using prepare should work similar to converge. | 17:36 |
| panda | zbr: I think prepare commits ... | 17:42 |
| panda | zbr: I have a loop device associated to file in the base image centos:7 now | 17:42 |
| panda | zbr: I'm moving everything to converge now | 17:43 |
| zbr | well, if you start to mount stuff in the container, that is how you will start getting stuff persistent. | 17:43 |
| panda | mmmhhh | 17:44 |
| raukadah | panda: rfolco sshnaidm|afk arxcruz feel free to reply on this https://lists.rdoproject.org/pipermail/dev/2019-August/009126.html thanks! | 17:44 |
| panda | zbr: I did not mount /dev .... | 17:45 |
| panda | zbr: or whereeve is losetup storing information | 17:46 |
| zbr | panda: do you have a PR that i can download and play with myself? | 17:47 |
| zbr | i may be able to figure out few things by trying it myself. | 17:48 |
| panda | zbr: I have a gerrit change | 17:48 |
| zbr | yeah, sent it. i will try to run it myself. | 17:48 |
| panda | zbr: https://review.rdoproject.org/r/21873 | 17:48 |
| panda | zbr: no sorry | 17:48 |
| panda | zbr: https://review.rdoproject.org/r/21909 this one | 17:48 |
| panda | losetup association is still on my host ... and I cannot remove it | 17:49 |
| panda | does docker mount /dev from host by default ? I don't get it ... | 17:53 |
| panda | why are these things on my host ? I ran them on the container ... | 17:53 |
| zbr | panda: i found one bug! you do not have destroy listedo in your custom sequence. you need it. | 17:57 |
| zbr | it does not destroy because is not listed. so it will do it only if you do it manually. | 17:57 |
| panda | zbr: I explicitly called destroy multiple times | 17:57 |
| panda | zbr: but an operation taht I thought was done on the container, ended up messing my host :( | 17:58 |
| panda | zbr: let's see if it appens again, or I just misplaced a delegate_to | 17:58 |
| zbr | one reason why i prefer running as non root without sudo. | 17:58 |
| panda | zbr: I'm not root, and I don't use sudo | 17:59 |
| zbr | got first error: msg: 'Error while getting device information with parted script: ''/sbin/parted -s -m /dev/vdb -- unit ''KiB'' print''' | 17:59 |
| zbr | promoter : Mount external volume - create a partition | 17:59 |
| panda | zbr: that should have been corrected. | 18:00 |
| panda | zbr: but at that point the loop device is already created | 18:00 |
| panda | zbr: and it happened again | 18:00 |
| panda | zbr: I have again loop device associated in my host | 18:01 |
| rlandy|ruck | weshay: can we sync? | 18:01 |
| panda | zbr: even if I'm running on container | 18:01 |
| zbr | i am inclined that partioning is not something you can do in containers (or should not) | 18:01 |
| zbr | cannot we skip these steps for testing? | 18:02 |
| panda | zbr: these are part of the real server ... if we make a mistake and it's skipped, we could merge broken code | 18:03 |
| zbr | in this case: stop using containers. | 18:03 |
| zbr | when using containers you do not have full isolation from host, we can easily mess things in containers. | 18:04 |
| zbr | just use the openstack driver and provision a node with it. | 18:04 |
| panda | zbr: you can't run dcker on contaier, you can't partition on containers ... :( | 18:04 |
| panda | zbr: what's the status of local vm drivers ? like vagrant ? | 18:04 |
| zbr | panda: i had the impression that you knew chroot limitations, docker is not very far from it. | 18:05 |
| zbr | panda: people are using it, what i do not like is that default backend for vagrant is virtualbox which has some conflicts with libvirt. | 18:06 |
| panda | zbr: any libvirt driver ? | 18:06 |
| zbr | but there is a libvirt plugin for vagrant, you can try to use that. still you need to install it and is bit experimental. probably will work fine on centos. (on mac it had some problems) | 18:07 |
| zbr | sadly not yet. | 18:07 |
| zbr | this is why I suggest: use a cloud. | 18:07 |
| zbr | openstack one, should be much easier/flexible. | 18:08 |
| panda | zbr: mmmhhh | 18:08 |
| zbr | less headaches | 18:08 |
| panda | zbr: what does it take to write a new driver ? | 18:08 |
| zbr | not to mention, someone else will pay the electricity ;) | 18:08 |
| zbr | panda: is not very hard but not is the worst possible moment for doing it, need to wait ~1month. | 18:09 |
| zbr | we started migrating all drivers to make them plugins. once done it should be much easier to add new driver. | 18:09 |
| zbr | and tbh, I think that libvirt would be very useful. | 18:10 |
| zbr | my recomandation (in order): a) try openstack driver b) try vagrant + libvirt c) swear at me if none works. | 18:11 |
| panda | zbr: since I can't do almost anything with docker, it's not just useful, it's necessary | 18:11 |
| panda | zbr: can I swear preventivley ? | 18:11 |
| rlandy|ruck | panda: was auth turned on for rdo containers? | 18:11 |
| panda | rlandy|ruck: no | 18:12 |
| panda | rlandy|ruck: what auth ? the login to the registry ? | 18:12 |
| rlandy|ruck | yep | 18:12 |
| rlandy|ruck | trying to track down why deploy is taking so long | 18:12 |
| rlandy|ruck | all of a sudden | 18:12 |
| panda | rlandy|ruck: no, the last comment from alex requires a refacotring. Either I move everything to rdo-jobs and goodnight, or I have to split the role in to docker and podman, the put all podman related stuff to rdo-jobs anyway | 18:13 |
| zbr | partitioning, docker-in-docker are exactly the kind of stuff for which containers are to be avoided. | 18:13 |
| rlandy|ruck | ok | 18:13 |
| panda | rlandy|ruck: then I'm not your culprit ... I haven't marged anything lately | 18:13 |
| panda | zbr: and I found them immediately | 18:14 |
| rlandy|ruck | panda: you are absolved | 18:14 |
| *** pierreprinetti has joined #oooq | 18:56 | |
| *** hamzy has quit IRC | 19:01 | |
| *** pierreprinetti has quit IRC | 19:21 | |
| *** hamzy has joined #oooq | 19:26 | |
| *** weshay is now known as weshay_MOD | 20:10 | |
| weshay_MOD | rlandy|ruck ack.. I have time now | 20:15 |
| weshay_MOD | or we can push to monday.. ur choice | 20:15 |
| rlandy|ruck | weshay_MOD: nothing urgent | 20:21 |
| rlandy|ruck | isn't it MOWE | 20:21 |
| weshay_MOD | manager on weekend escalation? | 20:21 |
| weshay_MOD | catchy | 20:21 |
| rlandy|ruck | wicked existence | 20:22 |
| rlandy|ruck | but yes the former | 20:22 |
| rlandy|ruck | yay https://review.opendev.org/#/c/678008/ merged - maybe fs020 has a shot now | 20:24 |
| *** Vorrtex has quit IRC | 20:47 | |
| *** aakarsh|2 has quit IRC | 22:10 | |
| *** rlandy|ruck has quit IRC | 22:21 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!