Monday, 2020-02-03

*** tetsuro_ has quit IRC00:00
*** tetsuro has joined #openstack-infra00:01
*** sreejithp has joined #openstack-infra00:08
*** sreejithp has quit IRC00:19
*** sreejithp has joined #openstack-infra00:20
*** sreejithp has quit IRC01:22
openstackgerritIan Wienand proposed openstack/project-config master: Remove publish-zuul-python-branch-tarball and release-zuul-python  https://review.opendev.org/70536701:22
kevinzianw: ping01:28
kevinzianw: sorry I just back from holiday01:28
*** tetsuro_ has joined #openstack-infra01:58
*** tetsuro has quit IRC02:01
*** ramishra has joined #openstack-infra02:05
*** dSrinivas has joined #openstack-infra02:22
dSrinivasHi All, Can anyone please provide the document link to configure new Openstack CI with Zuul new version. Thank you.02:23
*** rcernin has joined #openstack-infra02:27
ianwkevinz: hey, no worries, how's it going02:33
kevinzianw: thx. I tried upload a ubuntu bionic cloud image with admin user in remote node, it works fine02:34
ianwkevinz: i'll have to just context switch back in, but i think our issue was nb03 -> linaro-us connectivity, let me see02:36
ianwkevinz: ok, checking the logs right now, first i'm seeing02:37
ianwopenstack.exceptions.HttpException: HttpException: 503: Server Error for url: https://uk.linaro.cloud:9292/v2/images, 503 Service Unavailable: No server is available to handle this request.02:37
kevinzianw: uk?02:38
kevinzI thought it should be us.linaro.cloud02:38
ianwkevinz: yes, i think this is a new issue :/02:39
ianwhttp://paste.openstack.org/show/789042/ is the log02:39
ianwok, we are *also* seeing the same failures to connect to us.linaro.cloud -> http://paste.openstack.org/show/789043/02:40
ianwso, which one to debug first :)02:41
dSrinivas@ianw Hi02:41
ianwhello02:42
*** jamesmcarthur has quit IRC02:43
dSrinivasianw: Can you please provide me the document link to configure the new Openstack CI with new Zuul version02:44
ianwdSrinivas: unfortunately, step-by-step instructions for setting up a zuulv3 third-party CI system are not currently available02:44
dSrinivasianw: Ok. Can I use this link https://opendev.org/opendev/puppet-openstackci/src/branch/master/doc/source/third_party_ci.rst to configure Openstack CI and then later upgrade the Zuul version to V302:46
ianwdSrinivas: there is a spec available @ https://docs.opendev.org/opendev/infra-specs/latest/specs/zuulv3-3rd-party-ci.html that covers a lot of the topics02:46
kevinzianw: sorry I think the glance service in linaro london is dead, now glance is recovered. Could you retry?02:47
ianwdSrinivas: you probably could, but the upgrade will be significant.  TBH i'd try something like https://zuul-ci.org/docs/zuul/tutorials/quick-start.html#quick-start02:48
ianwthis is not focused at 3rd party ci, but will lead you in the right direction02:48
dSrinivasi02:50
ianwkevinz: checking ...02:50
dSrinivasianw: Ok. Thanks for the information02:50
ianwdSrinivas: we would love to get 3rd-party ci focused documentation updated; if you're in a position to contribute to that as you progress that would be great02:51
ianwperhaps read through that spec above, and if you have ideas bring them up02:52
*** jamesmcarthur has joined #openstack-infra02:53
dSrinivasianw: Ok But Can i follow old document puppet modules and system-config inorder to setup the jenkins, nodepool etc. except zuul02:53
ianwdSrinivas: well that's the thing; the architecture has changed significantly if you're using zuulv3.  the extant puppet is really a dead end, it is not being maintained, let alone improved02:54
ianwkevinz: | 0000086134 | 0000000626 | linaro-london        | ubuntu-xenial-arm64  | None                            | None                                 | uploading | 00:00:10:17  |02:56
ianwlooks like we're uploading02:56
ianwkevinz: so that brings us back to why nb03.openstack.org can't talk to us.linaro.cloud:500002:57
openstackgerritIan Wienand proposed zuul/zuul-jobs master: upload-afs: rename to upload-afs-roots; add afs-upload-synchronize  https://review.opendev.org/70536803:07
openstackgerritIan Wienand proposed zuul/zuul-jobs master: upload-afs: rename to upload-afs-roots; add afs-upload-synchronize  https://review.opendev.org/70536803:18
openstackgerritIan Wienand proposed openstack/project-config master: Remove publish-zuul-python-branch-tarball and release-zuul-python  https://review.opendev.org/70536703:19
openstackgerritIan Wienand proposed openstack/project-config master: Migrate to upload-afs-roots role  https://review.opendev.org/70537003:19
openstackgerritIan Wienand proposed opendev/base-jobs master: Migrate to upload-afs-roots role  https://review.opendev.org/70537103:19
*** jamesmcarthur has quit IRC03:22
kevinzianw: could you add my pub.key to that node nb03.openstack.org , so that I can check03:24
openstackgerritIan Wienand proposed zuul/zuul-jobs master: Remove deprecated upload-afs role  https://review.opendev.org/70537303:24
dSrinivasianw: Can i configure the Openstack CI with the Old configuration such that Our CI will be running for the driver and later we can upgrade the zuul to new version. Is it ok03:24
*** jamesmcarthur has joined #openstack-infra03:28
*** rcernin has quit IRC03:31
ianwdSrinivas: i'm not sure i can really tell you what to do ... i would not really recommend starting fresh with a largely unsupported software stack.  there's the quickstart guide i mentioned, or you might like to investigate other options like https://www.softwarefactory-project.io/docs/3.4/operator/quickstart.html#third-party-ci-quickstart03:37
dSrinivasianw: OK. Thank you for the information03:39
openstackgerritIan Wienand proposed openstack/project-config master: upload-artifacts-with-afs: use upload-afs-synchronzie role  https://review.opendev.org/70537403:41
*** jamesmcarthur has quit IRC03:46
openstackgerritIan Wienand proposed zuul/zuul-jobs master: upload-afs: rename to upload-afs-roots; add afs-upload-synchronize  https://review.opendev.org/70536803:48
openstackgerritIan Wienand proposed zuul/zuul-jobs master: Remove deprecated upload-afs role  https://review.opendev.org/70537303:48
*** jamesmcarthur has joined #openstack-infra03:51
ianwkevinz: http://paste.openstack.org/show/789046/ ... traceroute from nb03 -> us.linaro.cloud03:54
*** ykarel|away is now known as ykarel04:07
*** tetsuro_ has quit IRC04:07
*** tetsuro has joined #openstack-infra04:08
*** psachin has joined #openstack-infra04:29
*** jamesmcarthur has quit IRC04:50
*** raukadah is now known as chkumar|rover04:55
*** dSrinivas has quit IRC05:20
*** hongbin has joined #openstack-infra05:23
*** dSrinivas has joined #openstack-infra05:26
*** evrardjp has quit IRC05:33
*** evrardjp has joined #openstack-infra05:34
*** icarusfactor has joined #openstack-infra05:36
*** factor has quit IRC05:38
*** soniya29 has joined #openstack-infra05:45
*** udesale has joined #openstack-infra05:49
*** viks___ has joined #openstack-infra05:54
*** dSrinivas has quit IRC05:58
*** jamesmcarthur has joined #openstack-infra06:01
*** icarusfactor has quit IRC06:02
*** dSrinivas has joined #openstack-infra06:04
*** jamesmcarthur has quit IRC06:06
*** dSrinivas has quit IRC06:12
*** dSrinivas has joined #openstack-infra06:16
*** hongbin has quit IRC06:16
*** mattw4 has joined #openstack-infra06:17
*** dSrinivas has quit IRC06:20
*** ricolin has quit IRC06:24
*** dSrinivas has joined #openstack-infra06:28
*** tetsuro_ has joined #openstack-infra06:35
*** tetsuro has quit IRC06:39
dSrinivasianw: Is it mandatory to use Zuul V3 now or later we can change06:43
dSrinivasit06:43
*** mattw4 has quit IRC06:43
*** Lucas_Gray has joined #openstack-infra06:44
ianwdSrinivas: it is not mandatory to use zuul v3; the old zuul codebase can still listen to gerrit and make changes06:45
*** dSrinivas has quit IRC06:46
*** lmiccini has joined #openstack-infra06:48
*** roman_g has joined #openstack-infra07:05
*** AJaeger has quit IRC07:06
*** AJaeger has joined #openstack-infra07:08
*** lpetrut has joined #openstack-infra07:09
*** dSrinivas has joined #openstack-infra07:10
*** dpawlik has joined #openstack-infra07:12
*** dpawlik has quit IRC07:13
*** dpawlik has joined #openstack-infra07:14
*** tetsuro_ has quit IRC07:14
*** tetsuro has joined #openstack-infra07:14
*** dtantsur|afk is now known as dtantsur07:20
*** dSrinivas has quit IRC07:30
*** tkajinam has quit IRC07:36
*** hwoarang has quit IRC07:36
*** hwoarang has joined #openstack-infra07:37
*** Adri2000 has joined #openstack-infra07:37
*** tkajinam has joined #openstack-infra07:38
*** slaweq has joined #openstack-infra07:38
*** pcaruana has joined #openstack-infra07:43
*** iurygregory has joined #openstack-infra07:51
*** ykarel is now known as ykarel|lunch07:51
*** gfidente has joined #openstack-infra07:52
*** Lucas_Gray has quit IRC07:58
*** ralonsoh has joined #openstack-infra07:59
*** piotrowskim has joined #openstack-infra08:00
*** kopecmartin has joined #openstack-infra08:00
*** tkajinam_ has joined #openstack-infra08:01
*** Lucas_Gray has joined #openstack-infra08:03
*** tkajinam has quit IRC08:04
*** rpittau|afk is now known as rpittau08:05
*** dchen has quit IRC08:10
*** tesseract has joined #openstack-infra08:14
*** apetrich has joined #openstack-infra08:16
*** yboaron has joined #openstack-infra08:25
*** tetsuro has quit IRC08:27
*** tosky has joined #openstack-infra08:29
*** yboaron_ has joined #openstack-infra08:30
*** yboaron has quit IRC08:33
*** dSrinivas has joined #openstack-infra08:33
*** dSrinivas has quit IRC08:36
*** yboaron_ has quit IRC08:37
*** yboaron_ has joined #openstack-infra08:37
*** tkajinam_ has quit IRC08:38
*** dSrinivas has joined #openstack-infra08:39
*** ociuhandu has joined #openstack-infra08:46
*** tetsuro has joined #openstack-infra08:48
*** ociuhandu has quit IRC08:50
*** lucasagomes has joined #openstack-infra08:55
*** ociuhandu has joined #openstack-infra09:02
*** yolanda has joined #openstack-infra09:03
*** ociuhandu has quit IRC09:07
*** rchanter has joined #openstack-infra09:07
openstackgerritFelix Schmidt proposed zuul/zuul master: Implement github checks API  https://review.opendev.org/70516809:18
*** tetsuro has quit IRC09:21
*** Lucas_Gray has quit IRC09:23
*** ykarel|lunch is now known as ykarel09:29
*** xek has joined #openstack-infra09:30
*** sshnaidm|off is now known as sshnaidm09:30
*** pkopec has joined #openstack-infra09:35
*** derekh has joined #openstack-infra09:39
*** Lucas_Gray has joined #openstack-infra09:39
*** Tengu has quit IRC09:46
*** Tengu has joined #openstack-infra09:48
*** Tengu has quit IRC09:54
*** gshippey has joined #openstack-infra09:54
*** lucasagomes_ has joined #openstack-infra09:55
*** Tengu has joined #openstack-infra09:55
*** lucasagomes has quit IRC09:55
*** lucasagomes_ is now known as lucasagomes09:56
*** jaosorior has joined #openstack-infra10:01
*** Tengu has quit IRC10:03
*** Tengu has joined #openstack-infra10:03
*** Tengu has quit IRC10:08
*** Lucas_Gray has quit IRC10:09
*** Tengu has joined #openstack-infra10:13
*** Tengu_ has joined #openstack-infra10:14
*** Tengu_ has quit IRC10:15
*** ociuhandu has joined #openstack-infra10:18
*** ociuhandu has quit IRC10:23
*** rchanter has quit IRC10:27
*** lucasagomes has quit IRC10:27
*** lucasagomes has joined #openstack-infra10:41
*** ociuhandu has joined #openstack-infra10:47
*** Lucas_Gray has joined #openstack-infra10:59
*** Lucas_Gray has quit IRC11:09
*** Lucas_Gray has joined #openstack-infra11:10
*** tkajinam has joined #openstack-infra11:28
*** Lucas_Gray has quit IRC11:31
*** yboaron_ has quit IRC11:35
mordredmorning folks11:38
*** Lucas_Gray has joined #openstack-infra11:41
AJaegergood morning, mordred! feeling already at home or do you live in boxes?11:42
*** rpittau is now known as rpittau|bbl11:42
mordredAJaeger: a little of both - we're out of most of the boxes, but there's still a bunch of organization left. internet works though! :)11:43
*** ociuhandu has quit IRC11:44
*** ociuhandu has joined #openstack-infra11:45
*** hashar has joined #openstack-infra11:48
AJaegergreat!11:48
*** ociuhandu has quit IRC11:48
*** ociuhandu has joined #openstack-infra11:48
*** ociuhandu has quit IRC11:51
*** yboaron_ has joined #openstack-infra11:59
*** Lucas_Gray has quit IRC12:05
*** ociuhandu has joined #openstack-infra12:09
*** Lucas_Gray has joined #openstack-infra12:11
mordredianw, dSrinivas: dtroyer has been working on a windmill-based 3PCI Zuul v3 setup and mentioned maybe writing up some docs on how to do that12:12
*** ociuhandu has quit IRC12:15
*** ociuhandu has joined #openstack-infra12:16
dSrinivasmordred: How can I configure new openstack CI freshly without using puppet modules as it is going to move to ansible.12:21
*** rfolco has joined #openstack-infra12:24
mordreddSrinivas: yeah - that would be what dtroyer has been working on - he's using windmill (ansible) to setup the zuul - I'm not sure there's any docs on it yet12:26
dSrinivasmordred: Ok12:34
*** soniya29 has quit IRC12:41
*** rosmaita has left #openstack-infra12:47
*** rosmaita has joined #openstack-infra12:47
noonedeadpunkmornings folks. Can I ask for a hold of https://zuul.opendev.org/t/openstack/build/70dca1b262264ec98c5e623892cfa8a1 ?  I'm really puzzled why it's failing as this dir should be created according to logs https://zuul.opendev.org/t/openstack/build/70dca1b262264ec98c5e623892cfa8a1/log/job-output.txt#1753812:49
noonedeadpunkAlso I can't reproduce this in the sandbox...12:50
*** ramishra has quit IRC12:51
*** ociuhandu has quit IRC12:54
*** ociuhandu has joined #openstack-infra12:54
*** udesale_ has joined #openstack-infra12:55
*** udesale has quit IRC12:57
*** rh-jelabarre has joined #openstack-infra12:58
*** ociuhandu has quit IRC13:00
*** rlandy has joined #openstack-infra13:01
*** rosmaita has quit IRC13:05
openstackgerritMerged zuul/zuul master: Dockerfile: create a zuul user with uid 10001  https://review.opendev.org/65024613:09
*** rosmaita has joined #openstack-infra13:10
*** ramishra has joined #openstack-infra13:10
openstackgerritMonty Taylor proposed opendev/system-config master: Add Apache to Ansible for Gerrit  https://review.opendev.org/70449813:11
*** psachin has quit IRC13:16
*** yboaron_ has quit IRC13:19
*** jamesmcarthur has joined #openstack-infra13:21
*** derekh has quit IRC13:24
*** rpittau|bbl is now known as rpittau13:34
*** jamesmcarthur has quit IRC13:34
*** ykarel is now known as ykarel|afk13:36
*** ykarel|afk is now known as ykarel|away13:39
*** Goneri has joined #openstack-infra13:40
*** jamesmcarthur has joined #openstack-infra13:46
*** mrmartin has joined #openstack-infra13:55
openstackgerritMonty Taylor proposed opendev/system-config master: Add Apache to Ansible for Gerrit  https://review.opendev.org/70449813:57
*** ykarel|away is now known as ykarel14:04
*** Lucas_Gray has quit IRC14:04
*** Lucas_Gray has joined #openstack-infra14:05
*** ahosam has joined #openstack-infra14:07
ttxWeird thing happened on kata-dev ML this morning with mass unsubscription at 9UTC sharp14:09
ttxabout 35% of the subscribers14:10
ttxThe timing sounds consistent with mailman monthly tasks...14:10
ttxNotification of the unsubscription did not give any detail on why14:11
ttxAny idea of what could trigger that?14:14
ttx(I was personally unsubscribed and did nothing to trigger it)14:14
*** sreejithp has joined #openstack-infra14:14
*** tkajinam has quit IRC14:16
*** derekh has joined #openstack-infra14:17
*** ykarel is now known as ykarel|mtg14:19
*** ahosam has quit IRC14:19
*** jamesmcarthur has quit IRC14:21
*** jamesmcarthur has joined #openstack-infra14:23
*** Lucas_Gray has quit IRC14:31
*** Lucas_Gray has joined #openstack-infra14:32
*** yboaron_ has joined #openstack-infra14:34
*** jamesmcarthur has quit IRC14:35
*** zxiiro has joined #openstack-infra14:36
*** chkumar|rover is now known as raukadah14:37
*** eharney has joined #openstack-infra14:41
*** smarcet has joined #openstack-infra14:46
*** dtantsur is now known as dtantsur|brb14:50
*** ykarel|mtg is now known as ykarel14:52
*** Sundar has joined #openstack-infra15:04
*** artom has joined #openstack-infra15:04
*** apetrich has quit IRC15:07
*** ociuhandu has joined #openstack-infra15:07
SundarHello all, regarding https://zuul.opendev.org/t/openstack/builds?job_name=openstack-tox-py37&result=TIMED_OUT, we are trying to find a good way to determine the cause of the Zoom timeout in Cyborg. The test reports only indicate what worked. So: (a) how can we find out which tests timed out, if any? (b) Is there a message that Zuul sends indicating15:11
Sundartimeout?15:11
*** jamesmcarthur has joined #openstack-infra15:12
*** Lucas_Gray has quit IRC15:14
SundarIn previous discussions http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2020-01-22.log.html#t2020-01-22T15:22:32, it was said that perhaps Cyborg has some race conditions. While we looked at it in general, this is too general. We need to know which tests timed out..15:14
*** lpetrut has quit IRC15:18
*** mattw4 has joined #openstack-infra15:18
*** Lucas_Gray has joined #openstack-infra15:20
AJaegerSundar: One way: recheck and follow the output - go to zuul.opendev.org, find your change and click on it - and you get a terminal15:20
*** mattw4 has quit IRC15:25
*** eharney has quit IRC15:25
*** zbr has quit IRC15:30
*** smarcet has quit IRC15:31
*** armax has joined #openstack-infra15:31
*** sgw has joined #openstack-infra15:32
openstackgerritTobias Henkel proposed zuul/zuul master: Cap cheroot to fix issues with concurrent requests  https://review.opendev.org/70545915:32
*** eharney has joined #openstack-infra15:33
sgwMorning all, anyone awake yet?  How long does it take to propogate a SSH Key change from Ubuntu One -> review.opendev.org?  Also once a OneID is created should will an owner: search on review.opendev.org match or does that person have to have submitted a code review?15:34
*** zbr has joined #openstack-infra15:34
*** hashar has quit IRC15:35
* sgw says nevermind!15:39
*** ahosam has joined #openstack-infra15:41
*** ahosam has quit IRC15:43
*** rfolco is now known as rfolco|eats15:43
*** ociuhandu has quit IRC15:44
*** ykarel is now known as ykarel|away15:44
*** ociuhandu has joined #openstack-infra15:45
*** electrofelix has joined #openstack-infra15:47
*** ociuhandu has quit IRC15:52
*** yboaron_ has quit IRC15:52
*** eharney has quit IRC15:59
clarkbttx: is it possible that is related to the recent pbl (un)listing? maybe mailman does unsubs for bounces periodically?16:00
clarkbsgw we dont import keys from ubuntuone16:01
*** dSrinivas has quit IRC16:02
sgwYeah, we found the problem, partly our starlingx setup docs, partly human factors!16:04
*** gyee has joined #openstack-infra16:04
*** dtantsur|brb is now known as dtantsur16:05
*** yboaron_ has joined #openstack-infra16:07
*** nicolasbock has joined #openstack-infra16:10
*** ociuhandu has joined #openstack-infra16:23
*** eharney has joined #openstack-infra16:23
*** Lucas_Gray has quit IRC16:26
*** sshnaidm is now known as sshnaidm|afk16:27
*** mattw4 has joined #openstack-infra16:31
*** mattw4 has quit IRC16:32
*** eernst has joined #openstack-infra16:32
*** udesale_ has quit IRC16:32
*** jamesmcarthur has quit IRC16:33
*** rpittau is now known as rpittau|afk16:33
*** ociuhandu has quit IRC16:34
*** jamesmcarthur has joined #openstack-infra16:37
*** zxiiro has quit IRC16:45
*** ramishra has quit IRC16:48
*** mattw4 has joined #openstack-infra16:51
*** rfolco|eats is now known as rfolco16:51
*** lmiccini has quit IRC16:54
*** ociuhandu has joined #openstack-infra16:55
*** mattw4 has quit IRC16:56
*** eernst has quit IRC16:57
*** eernst has joined #openstack-infra16:58
ttxclarkb: mayyybe16:59
clarkbttx: elsewhere jbryce seems to have attributed it to bounces17:02
*** eernst has quit IRC17:03
*** lucasagomes has quit IRC17:07
*** yboaron_ has quit IRC17:07
*** mattw4 has joined #openstack-infra17:08
*** yboaron_ has joined #openstack-infra17:08
*** ociuhandu has quit IRC17:09
*** ociuhandu has joined #openstack-infra17:10
*** yboaron_ has quit IRC17:13
*** ociuhandu has quit IRC17:15
*** ociuhandu has joined #openstack-infra17:17
*** eharney has quit IRC17:19
*** ociuhandu has quit IRC17:20
*** ociuhandu has joined #openstack-infra17:21
openstackgerritMerged opendev/gear master: Add BSD/Darwin support.  https://review.opendev.org/67167417:29
*** eharney has joined #openstack-infra17:32
openstackgerritBrian Haley proposed openstack/project-config master: Update Octavia failure rate dashboard  https://review.opendev.org/70549617:32
*** evrardjp has quit IRC17:33
openstackgerritMerged opendev/gear master: Ignore keepalive on unsupported platforms  https://review.opendev.org/70281617:34
*** evrardjp has joined #openstack-infra17:34
*** yboaron_ has joined #openstack-infra17:35
openstackgerritTristan Cacqueray proposed zuul/nodepool master: Dockerfile: create a nodepool user with uid 10001  https://review.opendev.org/70549717:38
*** smarcet has joined #openstack-infra17:38
*** yboaron_ has quit IRC17:39
openstackgerritTristan Cacqueray proposed zuul/nodepool master: Dockerfile: remove the uid_entrypoint service  https://review.opendev.org/70524117:41
*** TViernion has joined #openstack-infra17:42
*** smarcet_ has joined #openstack-infra17:47
*** smarcet has quit IRC17:48
*** smarcet_ is now known as smarcet17:48
* mordred goes to find some lunch17:50
*** Sundar has quit IRC17:53
*** ociuhandu_ has joined #openstack-infra17:54
*** ociuhandu has quit IRC17:57
*** ociuhandu_ has quit IRC17:59
*** gfidente is now known as gfidente|afk17:59
*** derekh has quit IRC18:00
*** electrofelix has quit IRC18:06
*** yboaron_ has joined #openstack-infra18:08
*** drifterza has joined #openstack-infra18:09
*** jamesmcarthur has quit IRC18:12
*** jamesmcarthur has joined #openstack-infra18:19
*** weshay is now known as weshay|ruck18:30
openstackgerritTobias Henkel proposed zuul/zuul master: Add foreground option  https://review.opendev.org/63564918:30
openstackgerritTobias Henkel proposed zuul/zuul master: Deprecate -d switch for running in foreground  https://review.opendev.org/70518518:30
openstackgerritTobias Henkel proposed zuul/zuul master: Don't enforce foreground with -d switch  https://review.opendev.org/70518918:30
*** drifterza has quit IRC18:33
*** smarcet has quit IRC18:42
*** smarcet has joined #openstack-infra18:47
*** ralonsoh has quit IRC18:51
*** smarcet has quit IRC18:51
openstackgerritGhanshyam Mann proposed openstack/hacking master: [ussuri][goal] Drop python 2.7 support and testing  https://review.opendev.org/70551418:52
clarkbmordred: when you return from lunch can you point me in the right direction for web proxy setup in the docker-compose world of ours? re https://review.opendev.org/#/c/705258/18:54
*** jamesmcarthur has quit IRC18:54
clarkbmordred: I think being consistent there with review and everything else would be nice if we can manage it18:55
mordredclarkb: uhm ... I'm not sure if I can point you to the RIGH?T direction18:57
mordredclarkb: https://review.opendev.org/#/c/704498/ is the direction I've been going with gerrit18:57
clarkblooks like that doesn't use a container'd web server but instead runs it on the host?18:58
mordredyeah. I was thinking that it might be nicer to go container - but hadn't gotten there yet18:58
clarkbI can go that route too. I think that was mostly what I was looking at whether or not it made sense to put that in a container or not18:58
clarkbI think what gets tricky is a host level apache can route on port 443 based on sni names18:58
clarkbbut if we containerized them we cannot18:59
mordredwhy not? if it's using host networking there should be no difference18:59
mordredmy main concern with containerizing them is whether or not graceful would still work properly18:59
clarkbright it presumes a single apache18:59
clarkbwith host networking that works18:59
clarkbwithout it you'd be running many apaches (though we don't deploy like that today)19:00
mordredyeah. I wouldn't think multiple apaches - I was mostly thinking containerized just so it could be in the compose file19:00
clarkbgotcha19:00
mordredbut - this is just me thinking - I'm probably wrong about everything :)19:00
clarkbwhen you run apache on the host its a bit more explicit there is a single apache19:00
mordredyeah. it's also not hard to run an apache on the host19:01
clarkbok after lunch I'll try to get started on similar for refstack and LE it up19:01
mordredotoh - if we shift in the future from compose to k8s - we'll need to sort out containerized apache - but maybe we just sort that at that point in time19:01
*** eharney has quit IRC19:02
*** diablo_rojo has joined #openstack-infra19:02
mordredclarkb: I'm thinking similarly launching a new review-dev, putting it on opendev and LEing it up19:02
clarkbhowever, I'm beginning to worry that the fixes to make the docker image build at all are just going to sit unreviewed.19:02
clarkband that makes me wonder how much more effort I should invest in this19:02
clarkbits been a great learning experience up to this point and helped to fix some bugs in our container testing19:02
mordredclarkb: is there a core team?19:03
mordredclarkb: yeah - I'd be very surprised if you get any reviews, given the core team19:03
clarkbya, and ttx and diablo_rojo are in it, so probably going to be up to them19:03
clarkbnot sure if anyone else is active. maybe tosky19:03
openstackgerritGhanshyam Mann proposed openstack/os-performance-tools master: [ussuri][goal] Drop python 2.7 support and testing  https://review.opendev.org/70552019:04
*** pkopec has quit IRC19:07
*** pkopec has joined #openstack-infra19:08
toskyclarkb: uh, where?19:09
openstackgerritGhanshyam Mann proposed openstack/os-testr master: [ussuri][goal] Drop python 2.7 support and testing  https://review.opendev.org/70552219:09
diablo_rojoclarkb, I can take a look, but I had little context when I got given +2 power..19:09
clarkbtosky: diablo_rojo https://review.opendev.org/#/c/705265/ is the bottom of the stack. I don't think it is urgent, but it would be helpful to understand if work like that is useful or if the project is effectively dead and maybe we need to think about not hosting it anymore rather than updating its deployment19:11
* diablo_rojo opens tab19:11
*** eharney has joined #openstack-infra19:14
openstackgerritTristan Cacqueray proposed zuul/zuul-operator master: Add tenant reconfiguration when main.yaml changed  https://review.opendev.org/70363119:15
*** stevebaker has joined #openstack-infra19:15
*** eharney has quit IRC19:17
toskyclarkb: oh, the refstack server19:21
toskyI believe that diablo_rojo may provide more details on the Foundation position on it; I've got the +2 from work on refstack-client (and tempestclient), but I can help with reviews19:22
*** hashar has joined #openstack-infra19:26
* clarkb pop out for late breakfast/early lunch19:30
openstackgerritMerged zuul/zuul master: Add gcloud_service auth option for Gerrit driver  https://review.opendev.org/70490419:30
diablo_rojotosky, at the board meeting in shanghai it sounded like Refstack is till an important effort to maintain, but there wasn't a lot of progress in terms of getting board members to HELP support the effort.19:31
diablo_rojotosky, when hogepodge left the foundation ttx and I were given +2 power on the repos so that we could add people when they came along I think.19:32
diablo_rojoThat's about all I got for info :)19:32
openstackgerritJames E. Blair proposed zuul/zuul master: Gerrit: poll for merged changes if no stream events  https://review.opendev.org/70505319:38
openstackgerritJames E. Blair proposed zuul/zuul master: Add google-cloud-storage to executor ansible  https://review.opendev.org/70527919:38
openstackgerritTristan Cacqueray proposed zuul/zuul-operator master: Add CONTRIBUTE file  https://review.opendev.org/70553519:43
*** smarcet has joined #openstack-infra19:46
*** gshippey has quit IRC19:50
*** dtantsur is now known as dtantsur|afk19:53
*** gfidente|afk is now known as gfidente19:56
*** tesseract has quit IRC20:01
*** irclogbot_1 has quit IRC20:05
*** irclogbot_3 has joined #openstack-infra20:06
openstackgerritAndreas Jaeger proposed openstack/os-performance-tools master: Fix docs building  https://review.opendev.org/70554320:18
openstackgerritTobias Henkel proposed zuul/zuul master: DNM: Debug container build  https://review.opendev.org/70421520:19
*** jamesmcarthur has joined #openstack-infra20:20
*** fresta has quit IRC20:22
openstackgerritTobias Henkel proposed zuul/zuul master: Use apt mirror infrastructure during zuul-quick-start  https://review.opendev.org/64944820:22
openstackgerritTobias Henkel proposed zuul/zuul master: DNM: Debug container build  https://review.opendev.org/70421520:22
openstackgerritRadosÅ‚aw Piliszek proposed openstack/project-config master: Add RP to devstack  https://review.opendev.org/70554720:23
openstackgerritTobias Henkel proposed zuul/zuul master: Enable ansible cleanup  https://review.opendev.org/63601520:24
*** fresta has joined #openstack-infra20:24
*** artom has quit IRC20:26
toskydiablo_rojo: and I think one of you added me and Chandan, I think; ok, I can't guarantee I can do many stuff, but I can review patches20:28
*** gfidente has quit IRC20:28
openstackgerritAndreas Jaeger proposed openstack/os-performance-tools master: Fix docs building  https://review.opendev.org/70554320:28
diablo_rojotosky, reviewing is what we need right now with clarkb's patch so that's perfect :)20:29
toskyclarkb: so, just to summarize, is this part of the effoert for making sure refstack (server) works with a more modern stack, or do you want to continue with the containerization? Is there some additional document about this (maybe the logs of the -infra meetings, or an email I may have missed on the lists?)20:29
openstackgerritTristan Cacqueray proposed zuul/nodepool master: Dockerfile: create a nodepool user with uid 10001  https://review.opendev.org/70549720:29
openstackgerritAndreas Jaeger proposed openstack/os-performance-tools master: Fix docs building  https://review.opendev.org/70554320:29
openstackgerritAndreas Jaeger proposed openstack/os-performance-tools master: Fix docs building  https://review.opendev.org/70554320:31
clarkbtosky: this started because we need to upgrade the server that refstack currently runs on. When I looked at the puppet I realized that much of it needed updating anyway so rather than invest in that I looked at conatinerizing it using our new deployment tooling20:34
clarkbtosky: this only started on like friday so no meeting stuff yet20:34
clarkbbut it is related to the server upgrades topic that we've been having over recent meetings20:35
mordredyah - but in general we're rolling out more containerized stuff across opendev20:35
clarkbMy concern is that I don't want to invest in modifying a lot of stuff for refstack if things on the development side are dead20:35
clarkbI've basically shown what needs to be done to make refstack deployable in a modern state. But if we aren't able to maintain the software we should probably sunset the service instead of updating its deployment20:36
toskyso containerize it as much as it is, like a fat application containerized, and not really a set of "micro"services?20:36
clarkbtosky: no I've undone the fat application containerizing that is already in refstacki20:36
clarkbin part because that dockerfile does not work20:36
mordredyah. we like container-per-process20:36
clarkbhttps://review.opendev.org/#/c/705258/ is what it currently looks like in system-config and that has simple testing that is working20:36
toskybut haven't you mentioned some issues with apache earlier?20:36
toskyI guess I should re-read it20:36
clarkbtosky: I need to add apache to 25820:37
clarkbbut it will not be added to the refstack dockerfile. that is orthogonal20:37
AJaegertosky: I think in this case a fat monolith is superior - and the right step.20:37
clarkbthe TODO add HTTPS is another way of saying "add apache"20:37
AJaegerBtw. https://changelog.com/posts/monoliths-are-the-future is worth reading20:37
clarkbAJaeger: you think a single container with mysql, apache, and refstack is better?20:38
*** jamesmcarthur has quit IRC20:38
AJaegerclarkb: no - a microservice would be 10 more container ;)20:38
toskyI guess this is the vi vs emacs for the '20s20:38
AJaegerclarkb: those three are "normal"20:38
toskybut anyway20:38
toskyis the set of non-WIP patches good enough already for a proper review, or may it change if the deployment part is changed?20:39
clarkbtosky: the first two changes I expect are good enough for review. The third I think may be conversation worthy.20:39
toskyoki20:40
mordredAJaeger: I think there may be different definitions of "fat monolith" around here :)20:40
clarkbI've sort of punted on the third by putting the dockerfile in system-config20:40
AJaegermordred: yeah ;)20:40
clarkbbeacuse I don't actually care how refstack the project wants to build its images as long as I can build one myself. And for that I need to fix bindep (second change)20:40
mordredclarkb: ++20:40
AJaegermordred: btw. could you find some reviewers for https://review.opendev.org/686590 , please?20:41
mordredAJaeger: nice article20:41
openstackgerritIan Wienand proposed zuul/zuul-jobs master: upload-afs: rename to upload-afs-roots; add afs-upload-synchronize  https://review.opendev.org/70536820:42
openstackgerritIan Wienand proposed zuul/zuul-jobs master: Remove deprecated upload-afs role  https://review.opendev.org/70537320:42
clarkbtosky: from my side I just don't awnt to invest more time and resources in this if the end result is refstack is a dead end20:43
toskyclarkb: I think there are still users, and what reported by diablo_rojo seems to confirm that20:43
openstackgerritMerged zuul/zuul master: Gerrit: poll for merged changes if no stream events  https://review.opendev.org/70505320:43
clarkbso far I've justified the work because I've learned a lot and its been useful in that way, but I think I've got to the point where its going to be cookiecutter service replacement/upgrade that I've done a million times over so less interested in continuing unless upstream is maintained20:43
clarkbtosky: yes there are users, but that doesn't matter if th softwrae isn't maintained and can't be made deployable20:44
toskyclarkb: that's right, so let's try to fix it, I'd say20:44
diablo_rojoclarkb, an update on figuring how to test the new server..I talked to egle and she said she didn't know what has been done in the past and referred me to Mark who also doesn't know what was done in the past but is digging into it. I also just realized hogepodge may have left details in his giant google doc status thing he wrote before he left that I will take a look at.20:45
clarkbok. The two basic things I need are correct workable bindep file (so that I can install deps in docker image build) and python3.7 support because python3.7 is what our base python images are built on20:45
clarkbI think the second change covers ^20:45
toskyclarkb: is this (bindep,py37) something that can be tested also in the normal gates, without docker?20:46
clarkbdiablo_rojo: thanks thats the second TODO on my change above :) I'm hoping we can automate that at least for this aspect (then we'll likely do it by hand once deployed in prod)20:46
tosky(I guess yes)20:46
clarkbtosky: bindep should already be tested because the unittests use it. Python37 would require adding a new job (doable but not yet done)20:47
toskydiablo_rojo: do you think that part of those content (the non-restricted ones) could be put $somewhere?20:47
diablo_rojotosky, yeah, once I have info I plan to add it to docs somewhere20:47
*** yboaron_ has quit IRC20:48
*** eharney has joined #openstack-infra20:53
openstackgerritMerged zuul/zuul master: Cap cheroot to fix issues with concurrent requests  https://review.opendev.org/70545920:54
openstackgerritAndreas Jaeger proposed openstack/os-performance-tools master: Fix docs building  https://review.opendev.org/70554320:59
openstackgerritAndreas Jaeger proposed openstack/os-performance-tools master: Fix docs building  https://review.opendev.org/70554321:04
openstackgerritAndreas Jaeger proposed openstack/os-performance-tools master: Fix docs building  https://review.opendev.org/70554321:14
*** Goneri has quit IRC21:16
*** smarcet has quit IRC21:17
clarkbinfra-root can I get reviews on https://review.opendev.org/#/c/705134/ I updated it to remove the no longer necessary auth_url line21:19
clarkbI can watch that this afternoon if it goes in21:19
clarkbonce cloud setup is complete the next step will be to dobule check quotas and then add to nodepool config21:19
clarkbthanks I've approved it21:24
*** harlowja has quit IRC21:25
*** nicolasbock has quit IRC21:29
johnsomIs there a path forward for stable branch log files that are in gzip format? Example: https://zuul.opendev.org/t/openstack/build/3f8e35e4e7fe4b18bd4a86609cf7d191/log/logs/devstacklog.txt.gz21:33
johnsomOh, I see that was from the 25th, so maybe fixed?21:35
openstackgerritClark Boylan proposed zuul/zuul-jobs master: Use unique loop vars to avoid conflicts  https://review.opendev.org/70533721:35
clarkbjohnsom: I sent email at the beginning of the year to oepnstack-discuss.Long story short stop explicitly compressing in the jobs21:36
clarkbyou should only compress when the file is a compressed file type like a .tar.gz tarball21:36
johnsomYeah, saw that, but I thought it said "we are fixing thise"21:36
johnsomOh, I am a dork. Sorry, that was a neutron-lbaas job, so pre-zuul3 probably.21:37
johnsomFun with deprecated code and stable branches.21:37
clarkbno, I didn't promise to fix things. I think we thought devstack zuulv3 was already good to go21:37
clarkbjust legacy jobs and we expect people to fix those by deleting them21:37
openstackgerritMerged zuul/zuul master: Add google-cloud-storage to executor ansible  https://review.opendev.org/70527921:38
fungitosky: clarkb: mordred: to be clear, hosting for refstack *probably* will always remain an openstack-infra thing and not an opendev thing, since it's clearly very openstack-specific21:46
fungialso, yes, deletion is the ultimate "fix" for all legacy pre-zuulv3 jobs21:47
clarkbfungi: yes, but if the software isnt maintaned nor deployable it wont be an opemstack-infra thing either21:48
clarkbthat is my concern, I dont want to invest further in deploying it unless it is maintained and that seems to be a bit in thr air21:48
fungioh, totally21:49
*** artom has joined #openstack-infra21:49
fungiwas just trying to clarify with mordred's (i think) comment about opendev switching to containery things21:49
fungiit logically follows that openstack-infra probably should follow suit, but we have to take care not to conflate the two21:50
clarkbfungi: fwiw I got it as far as running the service via docker compose last week. Need to add web server proxy with ssl termination as well as some functional testing21:50
fungiawesome!21:50
fungisorry i've been useless over the weekend21:50
clarkbI have zero interest in updating the puppet for this service if someone else wants to take that apprpach they may do so21:50
clarkbI looked an realizedut needed some big updates for nodejs stuff21:50
fungiyeah, i caught some of that21:51
fungiexcellent work21:51
clarkbbut my progress requires updates to refstack21:51
fungii'm willing to pitch in there once i get home, recover and file expense reports21:52
*** dpawlik has quit IRC21:53
*** jamesmcarthur has joined #openstack-infra21:53
openstackgerritMerged opendev/system-config master: Add airship CI cloud  https://review.opendev.org/70513421:59
*** slaweq has quit IRC22:02
*** mattw4 has quit IRC22:09
*** mattw4 has joined #openstack-infra22:09
*** slaweq has joined #openstack-infra22:11
*** slaweq has quit IRC22:16
mordredfungi: yes - good point re: not conflating opendev/openstack-infra22:17
*** sreejithp has quit IRC22:18
clarkbtosky: diablo_rojo the functional testing is the other bit beacuse I don't want to do a fairly major upgrade and not be able to confrim it is working afterwords. I think what we want is some fake data we can submit both in our gate testing and in production as well as directions on how to do that (I'm assuming the actual submission process is documented for users so I can find that, but I don't have22:21
clarkbacloud to generate the data against)22:21
*** dchen has joined #openstack-infra22:21
toskyI need to recheck, but I think it's not difficult to gather some test data22:22
diablo_rojoclarkb, yeah I think that info of adding a submission is documented. Danny can probably get you that if you can't find it anywhere.22:24
*** mattw4 has quit IRC22:29
*** mattw4 has joined #openstack-infra22:29
*** xek has quit IRC22:29
*** jamesmcarthur has quit IRC22:32
*** rfolco has quit IRC22:36
openstackgerritTristan Cacqueray proposed zuul/zuul-jobs master: use-buildset-registry: disable docker userland proxy  https://review.opendev.org/70275322:36
*** dciabrin has quit IRC22:38
*** dciabrin has joined #openstack-infra22:43
openstackgerritMerged zuul/zuul master: Fix github app authentication to work with checks API endpoints  https://review.opendev.org/70516722:46
openstackgerritTristan Cacqueray proposed zuul/zuul-operator master: Add tenant reconfiguration when main.yaml changed  https://review.opendev.org/70363122:46
openstackgerritTristan Cacqueray proposed zuul/zuul-operator master: Add CONTRIBUTE file  https://review.opendev.org/70553522:46
*** tkajinam has joined #openstack-infra22:53
openstackgerritMerged zuul/zuul master: Add some debug lines for provides/requires  https://review.opendev.org/70531323:00
clarkbour run_all.sh loop seems to be getting slow again. Current run is at 2 hours23:08
clarkblogs seem to jump from 21:01:46 to 23:01:4723:09
clarkbFrom PLAY [Base: set up users and base package repositories on all hosts] *********** to PLAY [Bridge: configure the bastion host] **************************************23:10
clarkbI guess we aren't logging what it does to setup users and base package repos?23:10
clarkbdoes ^ make sense to anyone23:10
clarkbthe current run_all.sh script started at 21:00 fwiw23:10
clarkb(so not two loops there)23:10
*** slaweq has joined #openstack-infra23:11
*** jamesmcarthur has joined #openstack-infra23:11
ianwclarkb: that should be graphed ...23:14
ianwhttp://grafana.openstack.org/d/qzQ_v2oiz/bridge-runtime?orgId=1 looks disappointingly empty :/23:15
*** slaweq has quit IRC23:15
openstackgerritTristan Cacqueray proposed zuul/zuul-jobs master: DNM: install-docker: enable setting docker userland proxy  https://review.opendev.org/70275323:16
openstackgerritTristan Cacqueray proposed zuul/zuul-operator master: Add OpenShift SCC and functional test  https://review.opendev.org/70275823:16
*** jamesmcarthur has quit IRC23:17
ianweverything looks disappointingly empty ... it's either me or something up with graphite23:17
clarkbno its sad for me too23:18
clarkbI get "no data"23:18
ianwi'm guessing graphite.opendev.org has crashed ... let me see if any console23:20
openstackgerritJames E. Blair proposed zuul/zuul-jobs master: Add upload-logs-gcs role  https://review.opendev.org/70371123:20
openstackgerritJames E. Blair proposed zuul/zuul-jobs master: Add index_links option to zuul manifest  https://review.opendev.org/70558023:20
ianwyeah, stuff about hung i/o processes ... kicking it now23:22
clarkbthanks23:22
*** pkopec has quit IRC23:22
*** hashar has quit IRC23:26
ianwhttp://grafana.openstack.org/d/qzQ_v2oiz/bridge-runtime?orgId=1&from=now-60d&to=now ... definitely periods where something goes astray, but a very slow upward trend23:26
ianw#status log rebooted graphite.opendev.org due to it being hung with storage i/o errors23:26
openstackstatusianw: finished logging23:26
ianwlooks like pretty standard rax-migrated-but-it-didn't-really-work crash23:27
*** harlowja has joined #openstack-infra23:28
*** jaosorior has quit IRC23:34
openstackgerritJames E. Blair proposed zuul/zuul master: web: link to index.html if index_links is set  https://review.opendev.org/70558523:45
ianwcorvus: so looking, i think that having afs_target set with {{ zuul.project.name}} in https://review.opendev.org/#/c/705374/1/playbooks/publish/openstack-artifacts-with-afs.yaml should be sufficient to stop any users of the publish jobs from writing elsewhere.  they can't override this?23:47
*** jaosorior has joined #openstack-infra23:49
clarkbianw: I wonder if the run_all.sh stalled out on reporting to graphite?23:53
clarkbthats udp though and should be fire and forget?23:53
ianwclarkb: yep, udp send for that ... fire and hope it gets there :)23:54

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!