Thursday, 2020-12-10

*** Chaserjim has quit IRC00:05
*** Chaserjim has joined #openstack-kolla00:06
*** dcapone2004 has quit IRC00:20
*** maharg101 has joined #openstack-kolla00:20
*** adrian-a has quit IRC00:25
*** maharg101 has quit IRC00:26
*** calbers has quit IRC00:27
*** calbers has joined #openstack-kolla00:28
*** kevko has quit IRC00:30
*** brinzhang has joined #openstack-kolla00:38
*** Chaserjim has quit IRC00:45
*** Chaserjim has joined #openstack-kolla00:45
*** benj_- has joined #openstack-kolla00:53
*** benj_ has quit IRC00:54
*** benj_- is now known as benj_00:54
*** wuchunyang has joined #openstack-kolla00:57
*** Chaserjim has quit IRC01:00
*** Chaserjim has joined #openstack-kolla01:00
*** LinPeiWen has joined #openstack-kolla01:09
*** sorin-mihai has quit IRC01:18
*** Chaserjim has quit IRC01:23
*** Chaserjim has joined #openstack-kolla01:23
*** Chaserjim has quit IRC01:34
*** Chaserjim has joined #openstack-kolla01:34
*** ricolin has quit IRC01:35
*** Chaserjim has quit IRC01:47
*** Chaserjim has joined #openstack-kolla01:47
*** dking has joined #openstack-kolla01:55
*** maharg101 has joined #openstack-kolla02:21
*** maharg101 has quit IRC02:26
*** k_mouza has joined #openstack-kolla02:28
*** k_mouza has quit IRC02:33
*** Wasaac has quit IRC02:33
*** Chaserjim has quit IRC02:35
*** Chaserjim has joined #openstack-kolla02:36
*** Chaserjim has quit IRC02:57
*** Chaserjim has joined #openstack-kolla02:57
*** priteau has quit IRC03:03
*** Chaserjim has quit IRC03:22
*** openstackgerrit has quit IRC03:22
*** Chaserjim has joined #openstack-kolla03:23
*** zijlboot has quit IRC03:27
*** Wasaac has joined #openstack-kolla03:53
*** dave-mccowan has quit IRC03:54
*** Chaserjim has quit IRC04:00
*** Chaserjim has joined #openstack-kolla04:00
*** wuchunyang has quit IRC04:01
*** bsanjeewa has joined #openstack-kolla04:10
*** maharg101 has joined #openstack-kolla04:22
*** bsanjeewa has quit IRC04:23
*** maharg101 has quit IRC04:27
*** Chaserjim has quit IRC04:54
*** Chaserjim has joined #openstack-kolla04:54
*** Chaserjim has quit IRC05:00
*** Chaserjim has joined #openstack-kolla05:01
*** skramaja has joined #openstack-kolla05:06
*** zzzeek has quit IRC05:06
*** zzzeek has joined #openstack-kolla05:08
*** Chaserjim has quit IRC05:08
*** Chaserjim has joined #openstack-kolla05:09
*** skramaja has quit IRC05:11
*** dasp has quit IRC05:18
*** Chaserjim has quit IRC05:26
*** Chaserjim has joined #openstack-kolla05:27
*** skramaja has joined #openstack-kolla05:29
*** evrardjp has quit IRC05:33
*** evrardjp has joined #openstack-kolla05:33
*** JamesBenson has quit IRC05:33
*** dasp has joined #openstack-kolla05:34
*** JamesBenson has joined #openstack-kolla05:35
*** JamesBenson has quit IRC05:41
*** bsanjeewa has joined #openstack-kolla05:43
*** bsanjeewa has quit IRC05:52
*** bsanjeewa has joined #openstack-kolla05:55
*** Chaserjim has quit IRC05:59
*** Chaserjim has joined #openstack-kolla05:59
*** bsanjeewa has quit IRC06:00
*** bsanjeewa has joined #openstack-kolla06:02
*** bsanjeewa has quit IRC06:02
*** rpittau|afk has quit IRC06:11
*** rpittau|afk has joined #openstack-kolla06:12
*** zzzeek has quit IRC06:17
*** JamesBenson has joined #openstack-kolla06:17
*** zzzeek has joined #openstack-kolla06:18
*** bsanjeewa has joined #openstack-kolla06:20
*** JamesBenson has quit IRC06:22
*** bsanjeewa has quit IRC06:25
*** Chaserjim has quit IRC06:25
*** Chaserjim has joined #openstack-kolla06:26
*** jaicaa has quit IRC06:31
*** Chaserjim has quit IRC06:33
*** Chaserjim has joined #openstack-kolla06:34
*** jaicaa has joined #openstack-kolla06:34
*** cah_link has joined #openstack-kolla06:45
*** Chaserjim has quit IRC06:49
*** evrardjp has quit IRC06:49
*** evrardjp_ has joined #openstack-kolla06:49
*** Chaserjim has joined #openstack-kolla06:49
*** openstackgerrit has joined #openstack-kolla06:52
openstackgerritMichal Nasiadka proposed openstack/kolla stable/victoria: Support CentOS 8.3  https://review.opendev.org/c/openstack/kolla/+/76642106:52
*** bsanjeewa has joined #openstack-kolla06:56
*** bsanjeewa_ has joined #openstack-kolla06:58
*** bsanjeewa has quit IRC07:00
*** bsanjeewa_ has quit IRC07:01
*** bsanjeewa_ has joined #openstack-kolla07:01
*** sorin-mihai has joined #openstack-kolla07:03
*** zijlboot has joined #openstack-kolla07:09
*** sorin-mihai has quit IRC07:28
*** sorin-mihai_ has joined #openstack-kolla07:28
*** sorin-mihai_ has quit IRC07:34
openstackgerritMichal Nasiadka proposed openstack/kolla stable/ussuri: Support CentOS 8.3  https://review.opendev.org/c/openstack/kolla/+/76640907:40
*** Chaserjim has quit IRC07:42
*** Chaserjim has joined #openstack-kolla07:42
*** maharg101 has joined #openstack-kolla07:44
openstackgerritMichal Nasiadka proposed openstack/kolla stable/train: Support CentOS 8.3  https://review.opendev.org/c/openstack/kolla/+/76641007:45
*** nikparasyr has joined #openstack-kolla07:45
openstackgerritMichal Nasiadka proposed openstack/kolla stable/train: Support CentOS 8.3  https://review.opendev.org/c/openstack/kolla/+/76641007:47
*** jbadiapa has joined #openstack-kolla07:48
openstackgerritMichal Nasiadka proposed openstack/kolla stable/victoria: Support CentOS 8.3  https://review.opendev.org/c/openstack/kolla/+/76642107:53
*** Chaserjim has quit IRC07:54
openstackgerritMichal Nasiadka proposed openstack/kolla stable/ussuri: Support CentOS 8.3  https://review.opendev.org/c/openstack/kolla/+/76640907:54
*** Chaserjim has joined #openstack-kolla07:54
*** rgogunskiy has joined #openstack-kolla07:58
*** Chaserjim has quit IRC08:00
*** Chaserjim has joined #openstack-kolla08:01
*** gfidente has joined #openstack-kolla08:04
yoctozeptomnasiadka: why did not you wait for the merge to happen?08:07
mnasiadkayoctozepto: which merge?08:08
*** bengates has joined #openstack-kolla08:08
yoctozeptomnasiadka: centos 8.3...08:09
yoctozeptoyou proposed unmerged backport08:09
yoctozepto(no cherrypick stamp)08:09
mnasiadkawill update later08:10
mnasiadkanow I'm trying to follow the whole stack08:10
mnasiadkawhich is some nightmare08:10
mnasiadkahttps://review.opendev.org/c/openstack/kolla-ansible/+/76618108:11
mnasiadkaand this won't merge without adding centos 8.3 inside08:11
mnasiadkawell, not inside, but in kolla :)08:11
*** andrewbonney has joined #openstack-kolla08:11
mnasiadkabut kolla-ansible jobs will fail, so seems we have a cyclic dependency08:11
yoctozeptomnasiadka: why not08:11
yoctozeptoit was passing yesterday08:12
yoctozeptojust had some minor gate failure08:12
yoctozeptowell, minor08:12
yoctozeptodocker limits08:12
mnasiadkakolla-ansible-centos8-source is building images08:12
mnasiadkaand it's failing08:12
mnasiadkawhy the heck is it building images?08:13
mnasiadkain gate but not in check? :D08:13
yoctozeptomnasiadka: hmm, queue quirks unfortunately08:14
*** Chaserjim has quit IRC08:14
yoctozeptolet's make sure nothing gates in kolla08:14
yoctozepto(for the time being)08:14
yoctozeptofocus on having k-a train merge now08:15
*** Chaserjim has joined #openstack-kolla08:15
yoctozeptodo not recheck any others08:15
yoctozeptothen make sure k-a ussuri etc. merge08:15
mnasiadkait will get +1 from Zuul in a minute and go to gate08:15
yoctozeptothen move onto kolla08:15
mnasiadkalet's see what happens then08:15
yoctozepto++08:15
*** rpittau|afk is now known as rpittau08:17
*** Chaserjim has quit IRC08:18
*** Chaserjim has joined #openstack-kolla08:18
*** JamesBenson has joined #openstack-kolla08:18
mnasiadkayoctozepto: seems getting those changes merged will take some time, due to deps08:20
*** vrdel has quit IRC08:20
*** vrdel has joined #openstack-kolla08:21
*** JamesBenson has quit IRC08:23
*** e0ne has joined #openstack-kolla08:23
yoctozeptomnasiadka: yup08:24
yoctozeptomnasiadka: I also did not want to backport c8.3 before backporting the CI slim down08:24
yoctozeptoand waited with all this for the k-a stack to merge08:24
mnasiadkayoctozepto: I did w-1, once we're ready - I'll update the changes.08:24
yoctozeptohence why I was a bit disappointed with your action :-)08:25
yoctozeptook, just try to be lightweight on the CI08:25
yoctozeptodo you remember when was the last time we had only a single CI issue? :D08:25
*** bsanjeewa__ has joined #openstack-kolla08:25
yoctozeptobecause nowadays I think 2+ is a norm08:25
*** bsanjeewa_ has quit IRC08:25
mnasiadkawell, the docker version check is our bug, but centos 8.3 was imposed on us ;)08:26
mnasiadkaso I think it's a tie08:26
*** sorin-mihai has joined #openstack-kolla08:26
yoctozeptoyeah, but how come it is pretty much the same day :-)08:28
yoctozeptoplus that virtualenv update with new pip that broke l-c08:29
mnasiadkawell, life can't be easy08:29
yoctozeptoalso our "bug"08:29
yoctozeptoindeed08:29
*** kevko has joined #openstack-kolla08:30
mnasiadkaafter analysing the mail threads on centos-devel yesterday, I'm getting to the point nobody wants to run unstable centos stream in production :)08:31
yoctozeptoI have just gate-guarded the train patch08:32
yoctozeptoI will release the w-1 once the gate is clear08:32
yoctozeptomnasiadka: it looks like it08:32
yoctozeptomnasiadka: I don't mind for our use case08:32
yoctozeptobut I understand folks can be pissed off ;-)08:32
mnasiadkawell, without a local mirror with frozen content - it will be hard to run a consistent env08:33
*** Chaserjim has quit IRC08:33
*** Chaserjim has joined #openstack-kolla08:33
openstackgerritwanghongxu proposed openstack/kolla master: keepalived: Remove checkers.pid file for keepalived  https://review.opendev.org/c/openstack/kolla/+/76641308:35
yoctozeptomnasiadka: I have been freezing it this way even with the point releases; they do change too so this point is moot08:35
yoctozeptothe shorter cadence08:35
yoctozeptoand vendors relying on point releases08:36
yoctozeptothese are valid08:36
*** bsanjeewa__ has quit IRC08:36
mnasiadkayoctozepto: I don't think CentOS updates a lot in the point releases, unless you use -CR08:36
*** zzzeek has quit IRC08:36
*** Chaserjim has quit IRC08:37
*** Chaserjim has joined #openstack-kolla08:38
*** zzzeek has joined #openstack-kolla08:39
openstackgerritMerged openstack/kolla stable/stein: [CI] Stop testing lower-constraints and fix py27 job  https://review.opendev.org/c/openstack/kolla/+/76624308:42
yoctozeptomnasiadka: a lot not but for real reproducibility you need to freeze anyway08:42
mgoddardyoctozepto, mnasiadka: morning08:42
yoctozeptobut I get the point about this much more dynamic now08:42
yoctozeptomgoddard: morning!08:43
mgoddardthe fun continues08:43
yoctozeptoindeed08:43
mgoddardso is kolla in the gate because it has a depends-on to the k-a patch? Has the W+1 been removed from the kolla patch?08:43
*** bsanjeewa has joined #openstack-kolla08:44
*** dougsz has joined #openstack-kolla08:45
yoctozeptomgoddard: the patch has merged08:45
yoctozeptoit is all right now08:45
yoctozeptowe have a common gate08:45
mgoddardok08:45
mgoddardwe do08:45
yoctozeptowhich triggers the build logic08:45
yoctozeptoand mnasiadka pointed out the order can cause failures now08:45
yoctozeptoso we have to be slightly careful08:46
yoctozeptochecks are safe as they are "queue-less"08:46
yoctozeptowell, the proper term is "independent pipeline"08:46
yoctozeptobut you get the point08:46
yoctozeptoI have updated the summary on the whiteboard08:47
mgoddardthanks08:47
openstackgerritRadosÅ‚aw Piliszek proposed openstack/kolla stable/train: CI: stop checking on aarch64  https://review.opendev.org/c/openstack/kolla/+/76626408:48
yoctozeptowhat are we doing about ^ ?08:49
yoctozeptoI guess hrw's word is final here :P08:49
openstackgerritMark Goddard proposed openstack/kolla master: update packages to python3 for ovs-dpdk contrib template  https://review.opendev.org/c/openstack/kolla/+/76461408:49
yoctozeptomgoddard: ^^^^08:56
mnasiadkayoctozepto: does that mean we are not going to publish aarch64 images anymore on Docker Hub? That might mean we should remove the old images to avoid confusion...08:56
yoctozeptomnasiadka: yeah, I believe we should also drop some comm about this on the ml08:56
mnasiadkayup08:57
mgoddardthis is just for Train, right?08:57
mgoddardit's fine by me08:58
mgoddardML update sounds sensible08:58
mnasiadkawell, Train is not EM yet, so we should at least make an announcement08:58
mgoddardwe have said many times we will remove images from dockerhub, but we never build that tool :)08:58
mnasiadkaYeah, we need to - because there's like years of history in there, it would be good to clean that up08:59
mgoddardagreed08:59
mnasiadkawe could just run a playbook that removes images by tags that we don't support anymore, that shouldn't be too complicated08:59
mgoddardit's not impossible, someone just needs to do it :)09:00
mgoddardprobably better would be to remove anything over X days old09:00
mgoddardthat catches a bunch of stuff09:01
yoctozeptoover 6 months old sounds plausible09:03
*** muhaha has joined #openstack-kolla09:03
mgoddard+109:04
mgoddardwouldn't cover the aarch64 case though, for 6 months09:04
yoctozeptoyeah, but every other cruft :-)09:04
mnasiadkaopendev.org certificate expired? geez09:05
mnasiadkaah, only on some gitea instances09:05
yoctozeptothere is some quirk regarding gerrit/zuul picking up changes to gate09:07
yoctozeptoif you ever see nothing happens09:07
yoctozeptodo the w-1 w+1 dance I am doing09:07
yoctozepto(and repeat as necessary)09:07
yoctozeptook, k-a docker 20.10 train-ussuri-victoria gating09:07
yoctozeptofingers crossed09:08
*** Chaserjim has quit IRC09:09
*** Chaserjim has joined #openstack-kolla09:10
yoctozeptoINFO:kolla.common.utils.base:[91mError: Unable to find a match: centos-release-openstack-victoria09:13
yoctozeptolovely09:13
yoctozeptomnasiadka: ^ c8.3 in its ducking glory09:13
yoctozeptoINFO:kolla.common.utils.kuryr-libnetwork:[91mERROR: Could not find a version that satisfies the requirement jsonschema<3.0.0,>=2.6.0 (from kuryr-libnetwork)09:14
yoctozepto^ this one is new pip09:14
yoctozeptoduck my life09:14
yoctozepto😂09:15
yoctozeptothat last one was train09:16
yoctozeptoussuri looks green though09:16
hrwops. I removed publish. dropped patch09:16
*** k_mouza has joined #openstack-kolla09:17
openstackgerritMark Goddard proposed openstack/kayobe master: DNM: Test Swift  https://review.opendev.org/c/openstack/kayobe/+/76461709:17
yoctozeptohrw: xD09:18
*** Chaserjim has quit IRC09:25
*** Chaserjim has joined #openstack-kolla09:25
*** Chaserjim has quit IRC09:31
*** Chaserjim has joined #openstack-kolla09:32
*** zzzeek has quit IRC09:32
*** zzzeek has joined #openstack-kolla09:33
*** wuchunyang has joined #openstack-kolla09:34
*** devfaz has quit IRC09:34
*** dswebb has joined #openstack-kolla09:37
*** Chaserjim has quit IRC09:37
*** Chaserjim has joined #openstack-kolla09:38
hrwyoctozepto: so now we are waiting for docker2010 going through gates of hell, then centos 8.3, l-c and then can finally start looking at patches09:39
hrw?09:39
*** Chaserjim has quit IRC09:41
*** dswebb has quit IRC09:41
*** Chaserjim has joined #openstack-kolla09:41
yoctozeptohrw: pretty much, yes; see https://etherpad.opendev.org/p/KollaWhiteBoard09:43
yoctozeptowhere I am tracking this duckery09:43
hrwyoctozepto: marked l-c there as merged09:50
mnasiadkayoctozepto: I think mariadb failed in the gates of hell on mixed job - https://review.opendev.org/c/openstack/kolla-ansible/+/766181/09:59
*** Chaserjim has quit IRC10:01
*** Chaserjim has joined #openstack-kolla10:02
yoctozeptomnasiadka: yup10:02
yoctozeptoso it's all lost10:03
*** numans has quit IRC10:04
*** Chaserjim has quit IRC10:04
*** Chaserjim has joined #openstack-kolla10:05
yoctozeptorecheck only that one now10:05
*** Chaserjim has quit IRC10:09
*** Chaserjim has joined #openstack-kolla10:09
openstackgerritwanghongxu proposed openstack/kolla master: keepalived: Remove pid files for keepalived  https://review.opendev.org/c/openstack/kolla/+/76641310:12
*** zzzeek has quit IRC10:13
*** wuchunyang has quit IRC10:13
*** zzzeek has joined #openstack-kolla10:14
openstackgerritwanghongxu proposed openstack/kolla master: keepalived: Remove pid files for keepalived  https://review.opendev.org/c/openstack/kolla/+/76641310:16
*** JamesBenson has joined #openstack-kolla10:19
yoctozeptorecheck issued10:23
yoctozeptoplease do not do any other rechecks or w+1s10:23
*** JamesBenson has quit IRC10:23
yoctozeptoif it fails twice more (meaning we don't get it merged by the end of the working day), I'll propose to slim down the CIs already10:32
yoctozeptothat's the plan10:32
yoctozeptoand now onto other things10:32
*** zzzeek has quit IRC10:39
*** zzzeek has joined #openstack-kolla10:41
*** rgogunskiy has quit IRC10:45
*** k_mouza has quit IRC10:46
*** rgogunskiy has joined #openstack-kolla10:47
*** k_mouza has joined #openstack-kolla10:48
*** Chaserjim has quit IRC10:55
*** Chaserjim has joined #openstack-kolla10:55
*** adrian-a has joined #openstack-kolla11:08
*** Chaserjim has quit IRC11:14
*** Chaserjim has joined #openstack-kolla11:15
*** Chaserjim has quit IRC11:16
*** Chaserjim has joined #openstack-kolla11:16
*** Chaserjim has quit IRC11:29
*** Chaserjim has joined #openstack-kolla11:30
*** wuchunyang has joined #openstack-kolla11:31
*** wuchunyang has quit IRC11:35
*** Chaserjim has quit IRC11:37
*** Chaserjim has joined #openstack-kolla11:38
*** JamesBenson has joined #openstack-kolla12:00
*** skramaja has quit IRC12:05
*** JamesBenson has quit IRC12:06
*** JamesBenson has joined #openstack-kolla12:06
*** Chaserjim has quit IRC12:10
*** Chaserjim has joined #openstack-kolla12:10
*** zzzeek has quit IRC12:14
bsanjeewaCould someone please give me a guide to configure external ceph with kayobe?12:14
bsanjeewaussuri12:15
*** zzzeek has joined #openstack-kolla12:17
mgoddardbsanjeewa: https://docs.openstack.org/kolla-ansible/ussuri/reference/storage/external-ceph-guide.html12:17
mgoddardbsanjeewa: some paths need to be modified for kayobe12:17
mgoddardi.e. /etc/kolla -> <kayobe-config>/etc/kayobe/kolla/12:18
*** Chaserjim has quit IRC12:18
bsanjeewamgoddard: Thanks12:18
*** Chaserjim has joined #openstack-kolla12:19
*** Chaserjim has quit IRC12:20
*** Chaserjim has joined #openstack-kolla12:21
*** Chaserjim has quit IRC12:22
*** Chaserjim has joined #openstack-kolla12:22
*** vrdel has quit IRC12:24
*** lxkong has quit IRC12:26
*** vrdel has joined #openstack-kolla12:27
openstackgerritMark Goddard proposed openstack/kolla-ansible master: Fix mariadb_recovery when mariadb container is missing  https://review.opendev.org/c/openstack/kolla-ansible/+/76645812:28
openstackgerritMark Goddard proposed openstack/kolla-ansible master: DNM: Testing for bug 1907658  https://review.opendev.org/c/openstack/kolla-ansible/+/76645912:30
openstackbug 1907658 in kolla-ansible wallaby "Mariadb recovery fails if mariadb container does not exist" [Medium,Triaged] https://launchpad.net/bugs/190765812:30
yoctozeptomgoddard: I would hold with proposing new changes too to avoid hitting dockerhub limits12:34
yoctozepto<gerrit nazi>12:35
mnasiadkadid we finally change to publish weekly?12:35
*** bsanjeewa has quit IRC12:36
yoctozeptowe did not but it will not help in here anyhow12:36
yoctozeptowithout using a caching mirror12:36
mnasiadkaok, so that part is missing12:37
yoctozeptomhm12:38
mnasiadkaanyway, let's merge all of that crap that is underway12:38
yoctozeptoI12:38
yoctozeptoamd12:38
yoctozeptosorry, keyboard12:38
yoctozeptoI am working on it12:38
*** Chaserjim has quit IRC12:39
*** Chaserjim has joined #openstack-kolla12:39
*** zzzeek has quit IRC12:40
mgoddardwe do use a caching mirror12:40
mgoddardsee tests/templates/globals-default.j212:41
*** zzzeek has joined #openstack-kolla12:41
yoctozeptomgoddard: but it seems to still hit limits; if it really "cached", it would not12:42
yoctozeptoI believe it is because it still calls dockerhub each time12:42
mgoddardyoctozepto: images are updated every day12:42
yoctozeptoand then it's irrelevant whether these are weekly images or not12:42
yoctozeptomgoddard: so? we should still be under limit if caching happened12:43
mgoddard3 distros, 2 types, many many images12:43
mgoddardeasily over the limit12:43
yoctozeptohmm, I think I've done the math so that it would be unlikely to hit it12:44
mgoddardif caching did not happen we would get only a few jobs passing per hour12:44
yoctozeptonot really because we have different ip addresses12:44
yoctozeptoso it's not that 1:112:44
mgoddardthe mirror's network's gateway will be the IP12:45
yoctozeptobut it still does not look current caching is enough; I suspect it just still hits dockerhub at least from time to time and it triggers limits12:45
yoctozeptobut maybe the math was wrong12:46
mgoddardI think so12:46
mgoddardif we have ~30 containers per job12:46
mgoddard3 distros is almost 100 pulls already12:47
yoctozeptoexactly - almost12:47
yoctozeptoit is below 10012:47
mgoddardwhat about branches12:47
mgoddardbinary/source12:47
yoctozeptohahahaha, I forgot about branches obviously; it is the same namespace in dockerhub12:47
yoctozeptopoor me12:47
mgoddardif we ran more jobs we'd be failing everywhere12:47
yoctozepto<facepalm>12:47
mgoddardplus scenarios12:48
*** rgogunskiy has quit IRC12:48
yoctozeptoscenarios overlap12:48
mgoddardto an extent12:48
mgoddardwe have hundreds of images. I don't know how many we test, but quite a few12:48
yoctozeptoyes, this makes sense12:48
yoctozeptoso caching is happening12:48
mgoddardyes12:48
mgoddardwe just get lucky sometimes12:49
yoctozeptoI hope we get lucky this time12:49
mgoddardweekly publish would help a lot12:49
mgoddardand jobs would get more reliable throughout the week :D12:49
yoctozeptoexcept for weekend jobs12:49
yoctozeptobut then again there are not many of them12:49
yoctozeptoindeed12:49
yoctozeptoso, maybe just switch for now?12:49
yoctozepto(and cry later)12:50
mgoddardto weekly? yes12:50
yoctozeptoack12:50
mgoddardespecially for older branches12:50
yoctozeptowe should have stopped stein12:51
yoctozeptoand we did not12:51
yoctozeptoI am stopping it12:51
openstackgerritRadosÅ‚aw Piliszek proposed openstack/kolla stable/stein: [CI] EM - Stop periodics  https://review.opendev.org/c/openstack/kolla/+/76646112:53
yoctozeptomgoddard, mnasiadka, hrw: ^12:53
*** ricolin has joined #openstack-kolla12:56
*** adrian-a has quit IRC12:57
*** dave-mccowan has joined #openstack-kolla12:57
hrw+212:58
*** wuchunyang has joined #openstack-kolla12:59
*** adrian-a has joined #openstack-kolla13:00
*** adrian-a has quit IRC13:01
*** adrian-a has joined #openstack-kolla13:01
hrwslowly cleans13:03
*** wuchunyang has quit IRC13:03
*** priteau has joined #openstack-kolla13:10
*** vrdel has quit IRC13:22
*** vrdel has joined #openstack-kolla13:24
*** k_mouza_ has joined #openstack-kolla13:33
*** k_mouza has quit IRC13:33
*** hjensas has quit IRC13:35
*** muhaha has quit IRC13:38
*** hjensas has joined #openstack-kolla13:56
*** wuchunyang has joined #openstack-kolla14:04
*** adrian-a has quit IRC14:17
*** TrevorV has joined #openstack-kolla14:19
*** wuchunyang has quit IRC14:20
*** adrian-a has joined #openstack-kolla14:21
*** bsanjeewa has joined #openstack-kolla14:21
openstackgerritMerged openstack/kolla-ansible stable/train: Fix prechecks with Docker 20.10.0 and py27 job  https://review.opendev.org/c/openstack/kolla-ansible/+/76618114:27
mnasiadkayay14:28
mnasiadkayoctozepto: ^^14:28
mnasiadkadid a recheck on next one in the row14:29
*** wuchunyang has joined #openstack-kolla14:30
yoctozeptomnasiadka: great, thanks14:31
mnasiadkaso train done, ussuri -> victoria -> master14:31
mnasiadkaand then we can deal with the kolla side14:31
yoctozepto++14:33
*** wuchunyang has quit IRC14:36
*** k_mouza_ has quit IRC14:38
*** k_mouza has joined #openstack-kolla14:39
*** cah_link has quit IRC14:44
hrwand on tuesday we can restart development ;D14:48
mnasiadkamaybe wed14:49
mnasiadka:)14:49
hrwhttps://review.opendev.org/c/openstack/kolla/+/765807 anyone? tier 2 stuff14:51
hrwzuul gave -1 only on l-c (now removed)14:51
hrwhas added 'test' image which fails but as it is tier 2 image whole run is success14:52
*** cah_link has joined #openstack-kolla14:52
*** k_mouza has quit IRC14:52
*** k_mouza has joined #openstack-kolla14:56
adrian-awhat's the logic on which a commit gets on master, why after stable releases? is that defined somewhere?15:00
*** wuchunyang has joined #openstack-kolla15:00
adrian-aand "Fix prechecks with Docker 20.10.0" is taking days to reach Victoria because automatic tests take too much and are unreliable?15:02
*** wuchunyang has quit IRC15:04
*** e0ne has quit IRC15:07
*** e0ne has joined #openstack-kolla15:08
yoctozeptoadrian-a: yes, that's it; we have been hit by a bunch of independent issues15:16
yoctozeptoplus any random instabilities15:16
yoctozeptowe have upgrade jobs which forced us to reverse the merge strategy for this particular fix15:16
adrian-aok15:20
*** devfaz has joined #openstack-kolla15:23
*** k_mouza has quit IRC15:24
hrwadrian-a: but note that patches were backported from master to stable ones15:24
hrwyoctozepto: do you plan to backport ci slim once we get back to green?15:25
yoctozeptohrw: yes (it's on the whiteboard)15:26
hrwupdated it a bit15:28
hrwso tomorrow we should have docker 2010 - each branch is up to 6 (3+3) h on zuul which means today we get ussuri in. victoria during night, master in the morning15:30
*** wuchunyang has joined #openstack-kolla15:34
*** wuchunyang has quit IRC15:39
adrian-aok, nice15:41
adrian-aI'm wating for the CCLA to be processed (I have submitted the signed PDF and I assume it will show in Gerrit). I have a small docs fix.15:42
adrian-athis https://prnt.sc/vzvrfr and a sudo mkdir /etc/ansible after pip install 'ansible<2.10'15:44
adrian-aThis virtualenv install piece was removed a while ago https://review.opendev.org/c/openstack/kolla-ansible/+/735894/7/doc/source/user/quickstart.rst#40 but python3 -m venv env does no work without the python3-virtualenv package15:45
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: elasticsearch-curator: drop botocore workaround  https://review.opendev.org/c/openstack/kolla/+/76650015:48
*** k_mouza has joined #openstack-kolla15:49
*** nikparasyr has left #openstack-kolla15:53
*** adrian-a has quit IRC15:57
*** frickler has joined #openstack-kolla16:00
yoctozeptohrw: if it goes smoothly ;-)16:01
yoctozeptothanks for the update16:01
hrwif ;D16:02
fricklernoob question: does anyone have a comprehensive comparison of kolla with e.g. osa or others, helping to decide which one to use?16:03
*** Chaserjim has quit IRC16:04
yoctozeptofrickler: I know mgoddard has some old comparison16:04
yoctozeptoit is probably inaccurate now16:05
yoctozeptowe are happy to help construct it, along with the osa team16:05
*** muhaha has joined #openstack-kolla16:14
*** kevko has quit IRC16:15
openstackgerritPierre Riteau proposed openstack/kolla-ansible master: Fix failure during Monasca Grafana upgrade  https://review.opendev.org/c/openstack/kolla-ansible/+/76650316:17
*** dougsz has quit IRC16:17
openstackgerritPierre Riteau proposed openstack/kolla-ansible master: Fix failure during Monasca Grafana upgrade  https://review.opendev.org/c/openstack/kolla-ansible/+/76650316:18
*** e0ne has quit IRC16:28
*** dougsz has joined #openstack-kolla16:31
*** cah_link has quit IRC16:32
*** k_mouza has quit IRC16:33
*** bsanjeewa has quit IRC16:34
*** k_mouza has joined #openstack-kolla16:35
*** bengates has quit IRC16:55
*** Trevor_V has joined #openstack-kolla17:05
*** TrevorV has quit IRC17:06
*** adrian-a has joined #openstack-kolla17:19
*** dougsz has quit IRC17:31
*** Chaserjim has joined #openstack-kolla17:31
*** dave-mccowan has quit IRC17:33
*** dave-mccowan has joined #openstack-kolla17:34
*** dave-mccowan has quit IRC17:39
JamesBensonyoctozepto: The patch for the docker update, is that getting ported to the ussuri branch?17:42
hrwJamesBenson: it goes to all branches17:43
yoctozeptoJamesBenson: yes, it already is17:43
yoctozeptoJamesBenson: we are suffering multiple issues concurrently17:43
yoctozeptolike17:43
hrwussuri one is now at gates17:43
yoctozeptogerrit is down at the very moment17:43
JamesBensonah okay, I'm still getting failures from the docker in pre-checks that why I was wondering.17:44
yoctozeptoI would tell you to apply a fix from a link but it is not going to load17:45
yoctozeptohttps://review.opendev.org/c/openstack/kolla-ansible/+/76618217:45
JamesBensonokay17:45
yoctozeptowe are drowning in a general misery17:45
hrwgrab fix from stable/train ;D17:52
yoctozepto^ yeas, opendev.org itself should work for that17:54
yoctozeptohttps://opendev.org/openstack/kolla-ansible/commit/c34bf4529d4c4bbe57944e871e408a4b26486ed717:54
yoctozeptoJamesBenson: ^17:54
yoctozeptoyou only need that one regexp fixed17:55
JamesBensonah, okay. Cool, thanks :-)17:55
*** jbadiapa has quit IRC17:57
hrwand we wait for zuul...17:57
yoctozeptohrw: we will lose the results most likely17:58
yoctozeptoas gerrit will not get them17:58
yoctozeptobeing down...17:58
hrwits up again18:04
*** rpittau is now known as rpittau|afk18:04
yoctozeptoyeas18:08
hrwit is gating ;d18:10
yoctozepto<318:21
*** k_mouza has quit IRC18:48
*** bsanjeewa has joined #openstack-kolla18:51
*** gfidente is now known as gfidente|afk18:53
openstackgerritMerged openstack/kolla-ansible stable/ussuri: Fix prechecks with Docker 20.10.0  https://review.opendev.org/c/openstack/kolla-ansible/+/76618219:03
hrwtime for victioria recheck?19:05
hrwdone19:06
*** cah_link has joined #openstack-kolla19:07
yoctozeptoack, updated the wb19:13
*** adrian-a has quit IRC19:20
*** adrian-a has joined #openstack-kolla19:24
openstackgerritMerged openstack/kolla stable/stein: [CI] EM - Stop periodics  https://review.opendev.org/c/openstack/kolla/+/76646119:27
*** maharg101 has quit IRC19:33
*** dciabrin_ has joined #openstack-kolla19:40
*** dciabrin has quit IRC19:43
openstackgerritRadosÅ‚aw Piliszek proposed openstack/kolla-ansible stable/stein: [CI] EM - Stop periodics  https://review.opendev.org/c/openstack/kolla-ansible/+/76653919:48
*** cah_link has quit IRC19:51
openstackgerritRadosÅ‚aw Piliszek proposed openstack/kolla-ansible stable/stein: [CI] EM - Stop periodics  https://review.opendev.org/c/openstack/kolla-ansible/+/76653919:53
*** muhaha has quit IRC19:53
yoctozeptomgoddard, mnasiadka, hrw: just noticed this is very important in lessening the morning CI stress on dockerhub -> https://review.opendev.org/c/openstack/kolla-ansible/+/76653919:53
yoctozepto(and CI itself of course)19:54
yoctozeptoI will approve it myself if I can't see you come to it19:54
*** muhaha has joined #openstack-kolla19:58
hrw+219:58
*** andrewbonney has quit IRC20:05
*** cah_link has joined #openstack-kolla20:05
yoctozeptohrw thx20:06
adrian-aI'm curious, are there some IF-s in testing code to not test everything on every commit? E.g. on docs change, there's no point to run any code tests.20:10
yoctozeptoadrian-a: there are20:11
yoctozeptojust we are not changing docs here :-)20:11
adrian-a:D20:11
yoctozeptoour CI is quite comprehensive nowadays20:12
yoctozeptoand not so fast20:12
mgoddardyoctozepto: we should just kill those periodics20:12
yoctozeptoso it takes time20:12
adrian-ayeah, the logic applies for multiple "code partitioning"20:12
yoctozeptomgoddard: kill all?20:12
mgoddardyoctozepto: yes20:12
mgoddardthey eat our precious pulls, and we rarely check them20:12
mgoddardnot to mention infra resources20:13
yoctozeptomgoddard: speak for yourself, I do check them20:14
mgoddardkeep them if you like20:14
yoctozeptobut I agree we went a bit overboard with them20:14
yoctozeptoas a side note, tripleo has more ;p20:14
yoctozeptoI think drastically reducing the amount of them makes sense20:15
mgoddardno doubt20:15
yoctozeptowell, otoh we still do lots of backports usually20:15
mgoddarduntil we resolve the pull issue, those pulls are better spent on real reviews20:15
yoctozeptoso we still get to see most of these results20:15
yoctozeptomgoddard: and that!20:16
yoctozeptook, I am removing them all20:16
yoctozeptoI will only leave publishes for now20:16
mgoddardhow many dockerhub user accounts would we need to guarantee pulls succeed?20:16
yoctozeptoxD20:16
yoctozeptowell, account gives us 200 pulls20:17
yoctozeptoone per branch should be fine20:17
mgoddardusername=<job>-<branch>-<cloud>20:17
mgoddardthat should do it20:17
yoctozeptoI think one per branch is enough; my calculations forgot branches20:18
yoctozeptobut now with 200 per branch20:18
yoctozeptoit should be plenty20:18
yoctozeptowell, except for these clouds20:18
yoctozeptohow many cloud endpoints do we have? ;D20:18
yoctozeptolet's leave that as a future idea; an exercise for a reader20:18
mgoddardyeah. hoping that periodic-weekly helps20:19
yoctozeptoas well20:19
openstackgerritRadosÅ‚aw Piliszek proposed openstack/kolla-ansible master: [CI] Drop periodics  https://review.opendev.org/c/openstack/kolla-ansible/+/76654320:22
yoctozeptomgoddard: ^20:22
openstackgerritRadosÅ‚aw Piliszek proposed openstack/kolla-ansible stable/victoria: [CI] Drop periodics  https://review.opendev.org/c/openstack/kolla-ansible/+/76642820:23
yoctozeptodoes not make sense to wait for merge, this is not user-facing20:24
openstackgerritRadosÅ‚aw Piliszek proposed openstack/kolla-ansible stable/ussuri: [CI] Drop periodics  https://review.opendev.org/c/openstack/kolla-ansible/+/76642920:24
openstackgerritRadosÅ‚aw Piliszek proposed openstack/kolla-ansible stable/train: [CI] Drop periodics  https://review.opendev.org/c/openstack/kolla-ansible/+/76643020:25
openstackgerritMerged openstack/kolla-ansible stable/stein: [CI] EM - Stop periodics  https://review.opendev.org/c/openstack/kolla-ansible/+/76653920:26
openstackgerritRadosÅ‚aw Piliszek proposed openstack/kolla-ansible stable/train: [CI] Drop periodics and multinode gating  https://review.opendev.org/c/openstack/kolla-ansible/+/76643020:29
yoctozeptomgoddard: Train with a twist ^20:29
yoctozeptothis should never have made it into gate20:30
yoctozeptowe already have two centoses in train20:30
yoctozeptoa multinode job in gate is asking for trouble20:30
yoctozeptoall right; this all looks much brighter now20:30
openstackgerritMark Goddard proposed openstack/kayobe master: Fixes for CentOS 8.3  https://review.opendev.org/c/openstack/kayobe/+/76620320:31
yoctozeptomgoddard: workaholism too?20:31
mgoddardyoctozepto: went for a haircut earlier. Try not to work late normally20:32
mgoddardand out \o20:34
yoctozeptobye20:35
*** cah_link has quit IRC21:01
*** lxkong has joined #openstack-kolla21:02
openstackgerritMerged openstack/kolla-ansible master: [CI] Drop periodics  https://review.opendev.org/c/openstack/kolla-ansible/+/76654321:02
openstackgerritMerged openstack/kolla-ansible stable/ussuri: [CI] Drop periodics  https://review.opendev.org/c/openstack/kolla-ansible/+/76642921:04
openstackgerritMerged openstack/kolla-ansible stable/victoria: [CI] Drop periodics  https://review.opendev.org/c/openstack/kolla-ansible/+/76642821:09
openstackgerritAdrian Andreias proposed openstack/kolla-ansible master: [docs] Add python3-venv package to quickstart.rst  https://review.opendev.org/c/openstack/kolla-ansible/+/76655821:15
*** Trevor_V has quit IRC21:16
*** muhaha has quit IRC21:30
openstackgerritMerged openstack/kolla-ansible stable/train: [CI] Drop periodics and multinode gating  https://review.opendev.org/c/openstack/kolla-ansible/+/76643022:03
*** vrdel has quit IRC22:11
*** vrdel has joined #openstack-kolla22:13
*** gfidente|afk has quit IRC22:33
*** adrian-a has quit IRC22:38
*** vrdel has quit IRC22:45
*** sorin-mihai has quit IRC22:57
*** bsanjeewa has quit IRC23:09
openstackgerritMerged openstack/kolla-ansible stable/victoria: Fix prechecks with Docker 20.10.0  https://review.opendev.org/c/openstack/kolla-ansible/+/76618323:19
*** maharg101 has joined #openstack-kolla23:30
*** maharg101 has quit IRC23:35
*** adeberg has quit IRC23:53
*** adeberg has joined #openstack-kolla23:53
*** sorin-mihai has joined #openstack-kolla23:57

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!