Friday, 2019-08-30

*** noorul has joined #zuul00:03
*** threestrands has quit IRC00:03
*** jamesmcarthur has joined #zuul00:12
*** noorul has quit IRC00:20
*** jamesmcarthur has quit IRC00:22
*** jamesmcarthur has joined #zuul00:25
*** jamesmcarthur has quit IRC00:37
*** jamesmcarthur has joined #zuul00:37
*** jamesmcarthur has quit IRC00:46
*** jamesmcarthur has joined #zuul00:50
*** jamesmcarthur has quit IRC00:55
*** jamesmcarthur has joined #zuul01:06
*** jamesmcarthur has quit IRC01:26
*** jamesmcarthur has joined #zuul01:27
*** jamesmcarthur has joined #zuul01:32
*** bhavikdbavishi has joined #zuul02:00
*** bhavikdbavishi1 has joined #zuul02:03
*** bhavikdbavishi has quit IRC02:04
*** bhavikdbavishi1 is now known as bhavikdbavishi02:04
*** bhavikdbavishi has quit IRC02:15
*** jamesmcarthur has quit IRC02:36
*** jamesmcarthur has joined #zuul02:37
*** jamesmcarthur has quit IRC02:45
*** bhavikdbavishi has joined #zuul03:14
*** bhavikdbavishi1 has joined #zuul03:17
*** bhavikdbavishi has quit IRC03:19
*** bhavikdbavishi1 is now known as bhavikdbavishi03:19
*** rfolco has quit IRC04:03
*** jhesketh has quit IRC04:20
*** raukadah is now known as chandankumar05:02
*** badboy has joined #zuul05:27
*** igordc has quit IRC05:46
*** bhavikdbavishi has quit IRC06:01
*** tosky has joined #zuul07:29
*** jpena|off is now known as jpena07:36
*** jhesketh has joined #zuul07:38
*** bhavikdbavishi has joined #zuul07:39
*** bhavikdbavishi1 has joined #zuul07:42
*** bhavikdbavishi has quit IRC07:43
*** bhavikdbavishi1 is now known as bhavikdbavishi07:43
mhuhey all, I forgot to mention that fbo and me are going to give a presentation about Zuul @ the Open Source Summit Europe in October (just before the Shanghai Summit): https://osseu19.sched.com/event/TLMB/stop-merging-broken-code-an-introduction-to-zuul-matthieu-huin-fabien-boucher-red-hat08:12
mhushould we send an announcement on the zuul-discuss ML?08:13
mhualso, how can we get stickers to give to attendees? They were a huge hit at FOSDEM this year08:13
openstackgerritTobias Henkel proposed zuul/zuul master: Add support for smart reconfigurations  https://review.opendev.org/65211408:37
corvusmhu: yes, please feel free to send to list; we can also put something on the homepage; maybe clarkb or fungi can help with stickers?08:37
*** hashar has joined #zuul08:45
zbris there a way to query zuul about job parenting, so I cam build a dependency graph with jobs?09:43
zbri want to build a diagram with a script and as long as I can obtain the dependencies, it should be ok.09:45
evrardjpclarkb: ofc. I just didn't think we exposed it long enough, but you're right, one day is enough. I can abandon this if you prefer.09:59
evrardjp(it's about https://review.opendev.org/#/c/676393/1)09:59
*** sgw has quit IRC10:23
*** clayg has quit IRC10:23
*** smyers has quit IRC10:23
*** tobiash has quit IRC10:23
*** zer0c00l has quit IRC10:23
*** pleia2 has quit IRC10:23
*** tobias-urdin has quit IRC10:23
*** flaper87 has quit IRC10:23
*** swest has quit IRC10:24
*** SpamapS has quit IRC10:24
*** Diabelko has quit IRC10:24
*** yoctozepto has quit IRC10:24
*** evrardjp has quit IRC10:24
*** weshay_pto has quit IRC10:24
*** clayg has joined #zuul10:24
*** irclogbot_3 has quit IRC10:25
*** smyers has joined #zuul10:25
*** tobiash has joined #zuul10:27
*** evrardjp has joined #zuul10:27
*** yoctozepto has joined #zuul10:27
*** irclogbot_0 has joined #zuul10:29
*** Diabelko has joined #zuul10:40
*** SpamapS has joined #zuul10:41
*** bhavikdbavishi has quit IRC10:59
*** hashar has quit IRC11:02
*** jpena is now known as jpena|lunch11:33
*** badboy has quit IRC12:02
*** rlandy has joined #zuul12:07
*** rlandy is now known as rlandy|ruck12:07
*** hashar has joined #zuul12:09
*** jamesmcarthur has joined #zuul12:10
*** gtema has joined #zuul12:11
*** yolanda has quit IRC12:12
*** gtema has quit IRC12:14
*** rfolco has joined #zuul12:25
*** jamesmcarthur has quit IRC12:27
*** jeliu_ has joined #zuul12:34
*** jpena|lunch is now known as jpena12:40
*** yolanda has joined #zuul12:41
*** jamesmcarthur has joined #zuul13:05
mordredzbr: if you look at, for instance, http://zuul.opendev.org/t/openstack/job/openstack-tox-py36-with-oslo-master - you'll see the "parent" field.13:06
mordredzbr: you can see that in the json output if you do something like: curl 'http://zuul.opendev.org/api/tenant/openstack/job/openstack-tox-py36-with-oslo-master' -H 'DNT: 1' -H 'Accept-Encoding: gzip, deflate' -H 'Accept-Language: en-US,en;q=0.9,nl;q=0.8' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.100 Safari/537.36' -H 'Accept:13:06
mordredapplication/json, text/plain, */*' -H 'Referer: http://zuul.opendev.org/t/openstack/job/openstack-tox-py36-with-oslo-master' -H 'Connection: keep-alive' --compressed --insecure13:06
zbrmordred: thats a good start but i need to be careful not to hammer zuul while building a dependency tree. thanks, probably i will try something during the weekend and implement some caching.13:12
zbrmordred: i want to do something like https://mermaidjs.github.io/mermaid-live-editor/#/edit/eyJjb2RlIjoiZ3JhcGggVERcbnNvbWUtam9iIC0tPiB0cmlwbGVvLWNpLWJhc2VcbnRyaXBsZW8tY2ktYmFzZSAtLT4gYmFzZSIsIm1lcm1haWQiOnsidGhlbWUiOiJkZWZhdWx0In1913:13
fungimhu: i have a handful but other osf may have some extra packs of them on hand or may be able to get some printed quickly and shipped... also i believe the template/graphic is available if you want to get your own printed. will try to find out options today13:14
zbrso if I save the dependency graph it should be very easy to generate the diagram from it, and also include hyperlinks. (mermaid is pure js)13:14
openstackgerritMerged zuul/zuul master: Add release note for bug fix to test correct commit  https://review.opendev.org/67928513:21
mordredzbr: might also be easier to just do http://zuul.opendev.org/api/tenant/openstack/jobs ... it gets the whole set in one go and is basically names and parents13:28
*** edmondsw_ has joined #zuul13:32
*** hashar has quit IRC13:35
tobiashexecutors now finally moved from ceph to nvme and the performance gain is *massive* :)13:35
mordredtobiash: good to know!13:36
openstackgerritTristan Cacqueray proposed zuul/zuul master: web: fix connections list github base_url  https://review.opendev.org/67895013:36
tobiashwhen at quota we had job startup times up to 25min, now at the same load <3min13:37
corvustobiash: wow.  both of those numbers are amazing13:37
tobiashalso we had many problems with lagging ansible (per task overhead of random 1-5 seconds)13:37
tobiashnow <50ms13:37
tobiashand at the same time we reduced the number of executors from 42 to 2013:38
corvuszbr: also, you can filter the jobs page, which shows the inheritance tree by default http://zuul.openstack.org/jobs (try entering "tripleo-ci-centos-7-containers-multinode")13:38
*** edmondsw_ has quit IRC13:41
openstackgerritTristan Cacqueray proposed zuul/zuul-jobs master: Add phoronix-test-suite job  https://review.opendev.org/67908213:45
*** hashar has joined #zuul13:48
*** EmilienM is now known as EvilienM13:49
zbrcorvus: nice! now this is raising a new question regarding something I discovered this morning on rdo.13:50
zbrin my attempt to make more jobs use ansible 2.8, I attempted to add ansible-version: 2.8 to "base" job from *rdo* zuul config.13:51
zbrobviously that jpena raise a big warning sign, as this had the potential of breaking *lots* of jobs, so I proposed creating a `base-rolling` job that is more open to bumping ansible than the base, aiming to re-parent some jobs on that one.13:52
zbrbut soon I realised that most of our jobs from tripleo are defined upstream, so I could not really change parenting to something that does not exist there.13:53
zbri wonder what would be a better approach for this.13:54
openstackgerritTristan Cacqueray proposed zuul/zuul master: executor: add merge_jobs options to disable gearman merge jobs  https://review.opendev.org/67697413:54
mordredzbr: pabelanger started working on a slightly different base job construct including a base that is not trusted on top of a base-minimal that is the trusted one13:56
zbrmaybe we can even have a base-rolling on zuul too? see how I defined it: https://review.rdoproject.org/r/#/c/21979/2/zuul.d/jobs.yaml13:56
*** jamesmcarthur has quit IRC13:57
mordredzbr: I think he's using that approach over in ansible zuul but hasn't had time to finish in openstack13:57
pabelangeryes, would suggest making base untrusted and parent to base-minimal trusted. Works well for us in testing alot of changes.  In our case, when we moved from 2.7 to 2.8 ansible, we we're able to properly test pre-merge.13:58
*** jamesmcarthur has joined #zuul13:58
pabelangerhttps://github.com/ansible/ansible-zuul-jobs/blob/master/zuul.d/jobs.yaml#L213:58
pabelangerzbr: mordred: rdoproject is also setup like this14:00
pabelangerso, changing base in rdo-jobs should be fine14:00
pabelangernot base-minimal in config-project14:00
pabelangeror did this change?14:01
zbrpabelanger: mainly because there were concerns that some unexpected jobs may break if we do bump ansible to 2.8 in base job. in fact if you look at previous changeset, this was my first attempt.14:01
pabelangerzbr: yah, https://review.rdoproject.org/21979/ isn't needed14:02
pabelangerjust update base to 2.8, don't commit then use depends-on14:02
pabelangeryou can test all your jobs14:02
pabelangerwithout landing anything14:02
pabelangerbecause rdo-jobs is untrusted14:02
zbrnot sure what we mean by "test all your jobs". there are other teams using rdo, there are jobs I have no idea about. not sure how to evaluate impact on others.14:03
zbrafaik we don't have problem using 2.8 on tripleo ones, we already switched ovb ones.14:03
pabelangeryou'd need to open reviews for those projects, with depends-on14:03
fungithe way we usually do it in opendev is announce when we plan to change something potentially disruptive and include information on how users of that feature can test to see whether they will be broken ahead of the change going live14:04
zbrthe question was more about what would be the impact on the ones we do not know about.14:04
pabelangerit will be the same process as updating all jobs to parent to base-rolling14:04
fungiand then leave it up to the users to determine whether they'll be broken14:04
pabelangersome review need to be created14:04
zbrfungi: maybe you are right, adding the bump and sending an invitation message for others to use depends-on to evaluate the risks and in one week we do the bump.14:05
pabelangerI would no create 2 base jobs, but uplift all jobs to latest ansible version, then help teams that don't want it pin their jobs to the old release14:05
zbrthey will have enough time to override the new value in their job if they need.14:05
pabelangers/help/have14:05
zbrpabelanger: fungi : thanks for the input on this, seems resonable approach.14:05
pabelangerbut for the most part, I haven't seen any breaking changes with 2.814:05
zbrme neither but people do still have healing wounds from previous bumps ;)14:06
pabelangeryup, I think ML post, encourage them to test is right approach. Then schedule day for cut over, and pin any jobs that fail14:07
zbrone thing that I learnt is that more pain using an outdated ansible version that the pain introduced by newer version, especially since ~2.4.14:07
pabelangerit might be better to focus on getting openstack tenant on ansible 2.8 first, they will catch a lot of things, before rdoproject14:08
pabelangerso far, zuul.a.c and zuul tenant for opendev is 2.814:09
pabelangerI _think_ tobiash might be too?14:09
pabelangernot sure about SpamapS14:09
pabelangernow that stable-2.9 is cut, once beta1 goes out I'd like to keep ontop of ansible 2.9 release14:09
tobiashpabelanger: we're still on 2.714:10
*** hashar has quit IRC14:14
*** noorul has joined #zuul14:14
*** hashar has joined #zuul14:14
noorulHi!14:18
noorulI am struggling to understand how Zuul works14:19
pabelangeryou've come to the right place :D14:19
openstackgerritMonty Taylor proposed zuul/zuul master: Write errors from ansible execution into json log  https://review.opendev.org/67672314:26
*** jamesmcarthur has quit IRC14:27
noorulThis is my current setup https://etherpad.openstack.org/p/rWL36RmF6W14:30
noorulYou can see my project definition from line number 20714:31
noorulIn the post-run phase I am expecting playbooks/base/post.yaml to be called as it is part of base14:32
noorulBut it is not getting called14:32
*** jamesmcarthur has joined #zuul14:33
*** jamesmcarthur has quit IRC14:35
noorulCan someone help me to understand whether my expectation is correct or wrong?14:35
*** jamesmcarthur has joined #zuul14:36
noorulpabelanger: Do you have any idea?14:41
pabelangerlooking14:41
pabelangernoorul: do any in post-run fire14:42
noorulYes14:43
pabelangerso, maybe possible one of the post-run playbooks is bad14:44
pabelangerstopping others from running14:44
pabelangerif you look in executor logs, you should be able to debug14:44
noorulhttp://paste.openstack.org/show/767646/14:44
noorulpabelanger: That is from the job console14:44
pabelangeryah, ara-report is failing14:45
pabelangerso playbooks are running14:45
pabelangerjust not successful14:45
noorulpabelanger: No, post.yaml is not called14:45
pabelangerhowever14:45
pabelangerwell14:45
pabelangerI was just going to say14:45
noorulAfter post-ssh.yaml , post-logs.yaml is called14:45
pabelangeris post.yaml hosts entry matching?14:45
pabelangerwhat does post.yaml look like14:46
pabelangerthere is likely a syntax error or hosts: all is not working14:46
noorulLine 19614:46
pabelangernoorul: that is called playbook/base/post.yaml14:47
pabelangerbut you have post-run: playbooks/base/post.yaml14:47
pabelangeris that a typo or just issue with pastebin14:48
* noorul is looking14:48
noorultypo in etherpad14:49
pabelangerkk14:49
noorulbubble wrap command: http://paste.openstack.org/show/767647/14:56
pabelangeryah, so it is trying to run14:56
pabelangerfor fails14:56
pabelangeror output isn't logged14:56
pabelangerif you follow the debug log, what do you see?14:56
pabelangerwhat is exit code14:56
noorulpabelanger: You are right14:59
noorulrsyn is failing http://paste.openstack.org/show/767648/14:59
pabelangerso, this is a few times now I have seen this14:59
openstackgerritMerged zuul/zuul master: Make json outfile seeking clearer  https://review.opendev.org/67874015:00
pabelangerflapper87 had this issue too15:00
pabelangerI think it might be because of different uids?15:00
pabelanger"Permission denied, please try again15:01
noorulpabelanger: In my case scheduler executor nodepool are running on same instance15:01
noorulpabelanger: Also the same node is used as ansible remote host15:02
pabelanger1 sec, looking for keep command15:02
noorulpabelanger: Using static driver15:02
pabelangerso you can inspect work_dir15:02
pabelangerI think it is 'zuul-executor keep'15:03
pabelangeryah, so if you run that, you'll have access to the work directory15:03
pabelangerthen you can try to debug why it is failing15:03
pabelangeryou also may want to node the node15:03
pabelangerand compare file permissiosn15:03
noorulwhich is the work_dir?15:04
pabelangerfor exmaple: /var/lib/zuul/builds/3963c049f2bc47318f09def3a4acf818/work/logs/15:05
pabelangerfrom your pb above15:05
pabelangerthat will be on your zuul-executor15:05
noorulThere is no such folder15:06
pabelangeryou need to use zuul-executor keep15:07
pabelangerotherwise we delete it15:07
pabelangerat the end of each jobs run15:07
noorulOh I see15:07
*** michael-beaver has joined #zuul15:07
noorulWhen I try to use ssh as zuul user I get permission error15:07
noorulzuul@aruba-virtual-machine:~$ ssh zuul@10.29.12.16115:07
noorulidentity_sign: private key /var/lib/zuul/.ssh/id_rsa contents do not match public15:07
pabelangeryou are using one time ssh keys right?15:08
pabelangeryah add-build-sshkey15:08
pabelangerso the job is going to delete access for the zuul key15:08
noorulIs $HOME/.ssh/id_rsa generated everytime15:09
noorulpabelanger: I have to step out now. I will be back.15:10
pabelangernoorul: when you use add-build-sshkey yes15:12
pabelangerack15:12
Shrewstobiash: i was looking at how you might test 671704 and i *think* you might be able to use something similar to FakeLaunchAndDeleteFailCloud, used in test_launchNode_delete_error.15:13
*** noorul has quit IRC15:16
tobiashShrews: thanks, I'll look into that, but probably in a few weeks. I'll be on vacation the next three weeks so I won't be around much.15:22
Shrewstobiash: ooh, enjoy vacation! I'm going to -1 that until you have a chance to look at it. Or I might get bored and do it for you.  :)15:23
tobiashShrews: thanks :)15:23
SpamapSpabelanger: I'm on whatever ansible 3.10.1 has as default15:48
pabelangerthanks! ansible 2.715:48
*** chandankumar is now known as raukadah15:58
*** sgw has joined #zuul16:02
SpamapSBTW, for perspective on scaling down from tobiash's level .. we run 2 executors.. and about 100 jobs per day. :-P16:02
SpamapSLanding 2 - 10 gated PR's per day16:03
*** jangutter has quit IRC16:04
mordredtobiash: I'm also on vacation the new 2 weeks. yay vacation!16:05
tobiashmordred: have fun!16:06
*** jpena is now known as jpena|off16:07
mordredtobiash: you too!16:14
*** noorul has joined #zuul16:22
noorulpabelanger: hi16:22
noorulIs it best practice to use add-build-sshkey role?16:30
*** igordc has joined #zuul16:39
openstackgerritTristan Cacqueray proposed zuul/zuul-jobs master: Add phoronix-test-suite job  https://review.opendev.org/67908216:43
*** hashar has quit IRC16:44
toskyphoronix-test-suite o.O16:48
*** jeliu_ has quit IRC16:50
noorulIs ARA installed as part of zuul?16:54
sgwMorning folks, this might be more of an ansible question, is there a way to get a certain component in an item.path?  Currently I am using {{ item.path.split('/')[4] }} I don't want to hard code the 4 since it can change, I want it to be the second to last path component16:55
clarkbuse [-2] to index from the end17:09
clarkbsgw ^17:09
Shrewsnoorul: you can see the list of packages zuul depends on for installation in the requirements.txt file17:19
ShrewsI feel like the "search repository" feature of opendev.org just does not work sometimes17:28
ShrewsFor example, search for "zuul" at https://opendev.org/zuul/zuul-jobs and no results are returned17:29
Shrewsyet that code base is littered with that term17:29
ShrewsI guess my expectations do not match reality17:30
Shrewscorvus: is that something we don't have turned on for gitea? ^^17:32
Shrews(if it's even an option to do so)17:33
ShrewsREPO_INDEXER_ENABLED: false: Enables code search (uses a lot of disk space, about 6 times more than the repository size).17:36
Shrewsmaybe the disk space issue is the reason we have that turned off  :/17:36
fungiShrews: yep, known problem upstream for gitea17:39
fungiuse codesearch.o.o in the meantime17:40
fungithere's an open issue about implementing better keyword indexing and an elasticsearch backend for it17:40
Shrewscoolio17:40
fungiopen issue for upstream gitea i mean17:40
fungino idea what their timeline might be on that though (and i have a feeling they don't know either)17:41
fungimight be worth some discussion with them in shanghai though17:41
fungiideally someone using opendev gets annoyed enough to ragecode a gitea pr to redo their search feature ;)17:42
*** jamesmcarthur has quit IRC18:18
*** jamesmcarthur has joined #zuul18:36
*** jamesmcarthur has quit IRC18:39
*** jamesmcarthur has joined #zuul18:42
*** jamesmcarthur has quit IRC18:45
openstackgerritRonelle Landy proposed zuul/zuul-jobs master: Only use RHEL8 deps repo on Red Hat systems newer than 7  https://review.opendev.org/67912618:47
fungizuul-maint: see discussion with johnsom in #openstack-infra about trouble loading some (but not all) object storage-backed logs for a change via chrome: https://zuul.opendev.org/t/openstack/build/1bcb6753acee47cf938f142cc6c4a286/log/controller/logs/octavia-amphora_log.txt.gz18:54
fungistill no smoking gun, but possibly more useful data points18:54
johnsomAh, I just posted this in the infra channel:18:54
johnsomoctavia-amphora_log.txt.gz:1 Access to XMLHttpRequest at 'https://openstack.fortnebula.com:13808/v1/AUTH_e8fd161dc34c421a979a9e6421f823e9/logs_58/679358/2/check/octavia-v2-dsvm-scenario/1bcb675/controller/logs/octavia-amphora_log.txt.gz' from origin 'https://zuul.opendev.org' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource.18:54
johnsomFrom the chrome developer tools log18:54
fungithat does indeed suggest that some of our cors stuff might not be complete18:55
openstackgerritRonelle Landy proposed zuul/zuul-jobs master: Only use RHEL8 deps repo on Red Hat systems newer than 7  https://review.opendev.org/67912619:05
*** jamesmcarthur has joined #zuul19:10
openstackgerritRonelle Landy proposed zuul/zuul-jobs master: Only use RHEL8 deps repo on Red Hat systems newer than 7  https://review.opendev.org/67912619:11
*** jeliu_ has joined #zuul19:11
*** noorul has quit IRC19:26
*** jamesmcarthur has quit IRC19:28
*** jamesmcarthur has joined #zuul19:29
*** jamesmcarthur has quit IRC19:33
*** jeliu_ has quit IRC19:42
*** jamesmcarthur has joined #zuul19:52
*** armstrongs has joined #zuul20:04
*** pleia2 has joined #zuul20:06
*** armstrongs has quit IRC20:16
*** michael-beaver has quit IRC21:07
*** jamesmcarthur has quit IRC21:47
*** rlandy|ruck has quit IRC22:11
*** tosky has quit IRC23:14
*** jhesketh has quit IRC23:18

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!