| *** goldyfruit_ has joined #openstack-infra | 00:06 | |
| clarkb | https://review.opendev.org/#/c/682755/1 has a +1 from zuul now. Thanks for getting that fix in | 00:18 |
|---|---|---|
| *** lbragstad has quit IRC | 00:19 | |
| *** lbragstad has joined #openstack-infra | 00:19 | |
| *** zhurong has quit IRC | 00:38 | |
| *** jamesmcarthur has joined #openstack-infra | 00:42 | |
| *** zhurong has joined #openstack-infra | 00:49 | |
| *** gyee has quit IRC | 00:50 | |
| *** xenos76 has quit IRC | 00:53 | |
| *** jamesmcarthur has quit IRC | 01:07 | |
| *** markvoelker has quit IRC | 01:08 | |
| *** slaweq has joined #openstack-infra | 01:11 | |
| *** slaweq has quit IRC | 01:16 | |
| *** rkukura has quit IRC | 01:20 | |
| *** markvoelker has joined #openstack-infra | 01:20 | |
| openstackgerrit | Akihiro Motoki proposed openstack/project-config master: Stop gerrit notification to neutron-stadium specific channels https://review.opendev.org/682771 | 01:40 |
| *** yamamoto has joined #openstack-infra | 01:56 | |
| openstackgerrit | Mohammed Naser proposed zuul/zuul-operator master: Create zookeeper operator https://review.opendev.org/676458 | 02:08 |
| *** apetrich has quit IRC | 02:10 | |
| openstackgerrit | Merged openstack/project-config master: Add the gerrit checks plugin repo to Zuul https://review.opendev.org/682752 | 02:17 |
| *** yamamoto has quit IRC | 02:21 | |
| *** yamamoto has joined #openstack-infra | 02:21 | |
| openstackgerrit | Merged openstack/project-config master: Add upstream Gerrit projects to OpenDev Zuul tenant https://review.opendev.org/682754 | 02:21 |
| openstackgerrit | Merged openstack/project-config master: Add system-config and zuul to OpenDev tenant for jobs https://review.opendev.org/682755 | 02:21 |
| *** jamesmcarthur has joined #openstack-infra | 02:22 | |
| *** roman_g has quit IRC | 02:35 | |
| *** jamesmcarthur has quit IRC | 02:38 | |
| *** jamesmcarthur has joined #openstack-infra | 02:47 | |
| *** larainema has joined #openstack-infra | 02:49 | |
| ianw | dmsimard: fwiw i've had good luck importing svg's into tinkercad and extruding, then stl into cura | 02:49 |
| ianw | and i can't remember what it was, something microsoft related, you can run an stl file through it to make it "watertight" and that worked well for me once too | 02:50 |
| ianw | was this -> https://tools3d.azurewebsites.net/ | 02:51 |
| dmsimard | ianw: thanks for the advice ! I'll check tinkercad out tomorrow | 02:55 |
| *** zhangfei has joined #openstack-infra | 02:57 | |
| ianw | yeah i've found it enough to do all manner of simple to moderate designs, with a very shallow learning curve | 02:58 |
| *** markvoelker has quit IRC | 03:09 | |
| *** slaweq has joined #openstack-infra | 03:11 | |
| *** slaweq has quit IRC | 03:16 | |
| *** jamesmcarthur has quit IRC | 03:24 | |
| *** weshay|ruck has quit IRC | 03:24 | |
| *** jamesmcarthur has joined #openstack-infra | 03:36 | |
| *** dave-mccowan has quit IRC | 03:36 | |
| *** rh-jelabarre has quit IRC | 03:47 | |
| *** rh-jelabarre has joined #openstack-infra | 03:47 | |
| *** zhangfei has quit IRC | 03:49 | |
| *** jamesmcarthur has quit IRC | 03:50 | |
| *** ricolin has joined #openstack-infra | 03:54 | |
| openstackgerrit | Ian Wienand proposed zuul/zuul master: zuul_console: fix python 3 support https://review.opendev.org/682556 | 03:59 |
| openstackgerrit | Ian Wienand proposed zuul/zuul master: Support nodes setting 'auto' python-path https://review.opendev.org/682275 | 03:59 |
| *** ramishra has joined #openstack-infra | 04:01 | |
| *** zhangfei has joined #openstack-infra | 04:07 | |
| auristor | ianw: was the delay between last volume update and release enabled for mirror.opensuse ? | 04:07 |
| ianw | auristor: no, i've only done that for the fedora script at the moment | 04:08 |
| ianw | honestly i haven't had a chance to check back in on it ... let me see | 04:08 |
| auristor | ok. mirror.opensuse has been released at least twice today and the last release started almost six hours ago | 04:09 |
| ianw | 2019-09-17 21:06:17 | Starting ForwardMulti from 536871007 to 536871007 on afs02.dfw.openstack.org (as of Tue Sep 17 13:03:58 2019). | 04:09 |
| ianw | 2019-09-18 03:34:27 | updating VLDB ... done | 04:09 |
| ianw | that's a good 7+ hours :/ | 04:10 |
| auristor | it completed in the last few minutes | 04:10 |
| ianw | 2019-09-17 13:04:11 | Starting ForwardMulti from 536871007 to 536871007 on afs02.dfw.openstack.org (as of Tue Sep 17 05:04:17 2019). | 04:10 |
| ianw | 2019-09-17 18:51:20 | updating VLDB ... done | 04:10 |
| ianw | ^ that one was a "zero" update (rsync didn't touch anything) | 04:11 |
| ianw | for opensuse, we merged https://review.opendev.org/#/c/682576/ which brought in some extra repo bits ... | 04:12 |
| ianw | i think, if i'm reading the logs correctly, sent 23,618 bytes received 4,694,912,393 bytes 5,219,495.29 bytes/sec | 04:13 |
| ianw | an extra 5gb or so added to the volume | 04:13 |
| ianw | http://files.openstack.org/mirror/logs/rsync-mirrors/opensuse.log | 04:14 |
| ianw | i think it might be better if these files rotate weekly | 04:14 |
| *** ykarel has joined #openstack-infra | 04:24 | |
| *** ociuhandu has joined #openstack-infra | 04:30 | |
| *** auristor has quit IRC | 04:33 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: logrotate: better rotation options https://review.opendev.org/682786 | 04:33 |
| *** janki has joined #openstack-infra | 04:34 | |
| *** auristor has joined #openstack-infra | 04:34 | |
| *** ociuhandu has quit IRC | 04:35 | |
| AJaeger | config-core, https://review.opendev.org/#/c/682215/ and https://review.opendev.org/682216 should be ready - please review | 04:37 |
| *** whoami-rajat has joined #openstack-infra | 04:38 | |
| *** lajoskatona has joined #openstack-infra | 04:42 | |
| *** lajoskatona has left #openstack-infra | 04:44 | |
| *** udesale has joined #openstack-infra | 04:45 | |
| *** pcaruana has joined #openstack-infra | 04:46 | |
| *** jaosorior has quit IRC | 04:57 | |
| *** jaosorior has joined #openstack-infra | 04:57 | |
| *** markvoelker has joined #openstack-infra | 05:10 | |
| *** slaweq has joined #openstack-infra | 05:11 | |
| *** jtomasek has joined #openstack-infra | 05:13 | |
| *** e0ne has joined #openstack-infra | 05:14 | |
| *** e0ne has quit IRC | 05:14 | |
| *** markvoelker has quit IRC | 05:14 | |
| *** slaweq has quit IRC | 05:16 | |
| *** e0ne has joined #openstack-infra | 05:16 | |
| *** ramishra has quit IRC | 05:39 | |
| *** e0ne has quit IRC | 05:50 | |
| *** kopecmartin|off is now known as kopecmartin | 05:54 | |
| openstackgerrit | Ian Wienand proposed zuul/nodepool master: Set default python-path to "auto" https://review.opendev.org/682797 | 05:57 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: logrotate: better rotation options https://review.opendev.org/682786 | 06:02 |
| *** slaweq has joined #openstack-infra | 06:11 | |
| *** pgaxatte has joined #openstack-infra | 06:13 | |
| *** slaweq has quit IRC | 06:16 | |
| *** igordc has joined #openstack-infra | 06:17 | |
| *** igordc has quit IRC | 06:19 | |
| *** e0ne has joined #openstack-infra | 06:21 | |
| *** xek_ has joined #openstack-infra | 06:22 | |
| *** e0ne has quit IRC | 06:26 | |
| *** xek_ has quit IRC | 06:30 | |
| *** zhangfei has quit IRC | 06:32 | |
| *** shachar has quit IRC | 06:35 | |
| *** snapiri has joined #openstack-infra | 06:35 | |
| *** armax has joined #openstack-infra | 06:39 | |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Add horizon to translation jobs https://review.opendev.org/682808 | 06:54 |
| *** slaweq has joined #openstack-infra | 06:55 | |
| *** armax has quit IRC | 06:56 | |
| *** rcernin has quit IRC | 06:57 | |
| *** zhangfei has joined #openstack-infra | 06:57 | |
| *** apetrich has joined #openstack-infra | 06:59 | |
| *** roman_g has joined #openstack-infra | 07:00 | |
| openstackgerrit | Andreas Jaeger proposed openstack/openstack-zuul-jobs master: Use on-disk horizon dir for translations https://review.opendev.org/682809 | 07:03 |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Add horizon to translation jobs https://review.opendev.org/682808 | 07:03 |
| openstackgerrit | Andreas Jaeger proposed openstack/openstack-zuul-jobs master: Use on-disk horizon dir for translations https://review.opendev.org/682809 | 07:07 |
| AJaeger | config-core, please review 682808 and 682809 to fix train translations - and make them more robust. | 07:07 |
| *** trident has quit IRC | 07:08 | |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Add horizon to translation jobs https://review.opendev.org/682808 | 07:08 |
| *** tosky has joined #openstack-infra | 07:18 | |
| *** trident has joined #openstack-infra | 07:19 | |
| *** sshnaidm|pto is now known as sshnaidm|rover | 07:24 | |
| sshnaidm|rover | before this big logs change I could leave ara.sqlite of my nested ansible in my custom ara-report directory and have nice html reports, is it possible to get this now in some way? | 07:31 |
| *** beagles has quit IRC | 07:31 | |
| *** mmedvede has quit IRC | 07:31 | |
| *** b3nt_pin has joined #openstack-infra | 07:32 | |
| *** rpittau|afk is now known as rpittau | 07:32 | |
| *** arxcruz has quit IRC | 07:32 | |
| *** mmedvede has joined #openstack-infra | 07:34 | |
| *** arxcruz has joined #openstack-infra | 07:36 | |
| *** jpena|off is now known as jpena | 07:41 | |
| *** ykarel is now known as ykarel|lunch | 07:44 | |
| AJaeger | mordred: https://review.opendev.org/682809 removes a "TODO(mordred)", want to review that together with https://review.opendev.org/682808, please? | 07:46 |
| AJaeger | sshnaidm|rover: best discuss later with corvus | 07:47 |
| sshnaidm|rover | AJaeger, ack | 07:48 |
| *** gfidente has joined #openstack-infra | 07:50 | |
| *** gfidente has quit IRC | 07:51 | |
| *** e0ne has joined #openstack-infra | 07:57 | |
| *** dchen has quit IRC | 08:03 | |
| *** ralonsoh has joined #openstack-infra | 08:04 | |
| *** tkajinam has quit IRC | 08:04 | |
| *** AJaeger has quit IRC | 08:14 | |
| ianw | sshnaidm|rover: the answer is currently no, because that relied on the log server running wsgi middleware to read the sqlite file. now the files are served directly from buckets by the hosting providers | 08:14 |
| ianw | sshnaidm|rover: the current practice for nested ansible is to switch ara to static html generation. it creates a lot of little files, but for one project it should be OK | 08:15 |
| *** gfidente has joined #openstack-infra | 08:15 | |
| ianw | as AJaeger mentioned, future options are under discussion ... there is a thread i think on infra list | 08:15 |
| sshnaidm|rover | ianw, ack, thanks | 08:15 |
| sshnaidm|rover | ianw, btw, can I generate something as zuul artifact so I can show this in build logs interface? | 08:16 |
| sshnaidm|rover | like file or link to something | 08:16 |
| *** xenos76 has joined #openstack-infra | 08:17 | |
| ianw | sshnidm|rover: yes ... i'm trying to think of an example | 08:18 |
| ianw | i think if drop it in the "artifacts" directory; i.e. sibling to logs. codesearching for things like that you'll probably find some examples | 08:18 |
| *** AJaeger has joined #openstack-infra | 08:27 | |
| frickler | https://zuul-ci.org/docs/zuul/user/jobs.html#return-artifacts | 08:27 |
| openstackgerrit | Merged zuul/zuul master: Add no-jobs reporter action https://review.opendev.org/681278 | 08:36 |
| *** ociuhandu has joined #openstack-infra | 08:37 | |
| *** kaiokmo has joined #openstack-infra | 08:38 | |
| *** dciabrin_ has quit IRC | 08:38 | |
| *** derekh has joined #openstack-infra | 08:40 | |
| *** ociuhandu has quit IRC | 08:41 | |
| *** iurygregory has joined #openstack-infra | 08:48 | |
| openstackgerrit | Merged zuul/zuul master: Add report time to item model https://review.opendev.org/681323 | 08:50 |
| *** ociuhandu has joined #openstack-infra | 08:52 | |
| openstackgerrit | Merged openstack/cookiecutter master: Update URLs https://review.opendev.org/679998 | 08:52 |
| *** ociuhandu has quit IRC | 08:53 | |
| *** KeithMnemonic1 has joined #openstack-infra | 08:55 | |
| *** KeithMnemonic has quit IRC | 08:55 | |
| *** KeithMnemonic1 has quit IRC | 08:55 | |
| *** KeithMnemonic1 has joined #openstack-infra | 08:56 | |
| *** pcaruana has quit IRC | 08:57 | |
| *** ociuhandu has joined #openstack-infra | 08:58 | |
| *** ramishra has joined #openstack-infra | 08:59 | |
| openstackgerrit | Merged openstack/project-config master: Add horizon to translation jobs https://review.opendev.org/682808 | 09:00 |
| *** pcaruana has joined #openstack-infra | 09:01 | |
| *** priteau has joined #openstack-infra | 09:07 | |
| openstackgerrit | Merged zuul/zuul master: Add Item.formatStatusUrl https://review.opendev.org/681324 | 09:11 |
| stephenfin | I assume there's a reason that rechecks on approved patches run both check and gate jobs? | 09:13 |
| sshnaidm|rover | stephenfin, firstly check jobs should run | 09:14 |
| openstackgerrit | Merged openstack/openstack-zuul-jobs master: Use on-disk horizon dir for translations https://review.opendev.org/682809 | 09:19 |
| *** kjackal has joined #openstack-infra | 09:21 | |
| stephenfin | sshnaidm|rover: Once, sure, but if they've already passed and it's just a flaky gate failure, why do we need to run them again? | 09:22 |
| sshnaidm|rover | stephenfin, I assume to be sure that it's just flaky gate failure and not some bug.. But I also find it not convenient. | 09:24 |
| *** ramishra_ has joined #openstack-infra | 09:25 | |
| *** ramishra has quit IRC | 09:26 | |
| *** ramishra has joined #openstack-infra | 09:26 | |
| *** yamamoto has quit IRC | 09:27 | |
| *** ramishra_ has quit IRC | 09:30 | |
| *** panda|ruck|off is now known as panda|ruck | 09:37 | |
| *** tesseract has joined #openstack-infra | 09:44 | |
| AJaeger | infra-root, any idea why the translation jobs now fail with 682809 merged, see http://zuul.opendev.org/t/openstack/build/60c2674924914f57a176b3e5f69bea9f | 09:45 |
| AJaeger | it works fine here http://zuul.opendev.org/t/openstack/build/0978b7bb113945d8ac70f1e20e234ab6/log/job-output.txt#1289 | 09:46 |
| AJaeger | but fails here http://zuul.opendev.org/t/openstack/build/60c2674924914f57a176b3e5f69bea9f/log/job-output.txt#1008 with horizon checked out via required-projects | 09:48 |
| AJaeger | found it "horizon/horizon" | 09:48 |
| openstackgerrit | Andreas Jaeger proposed openstack/openstack-zuul-jobs master: Fix horizon translation setup https://review.opendev.org/682850 | 09:49 |
| *** dtantsur|afk is now known as dtantsur | 09:49 | |
| AJaeger | mordred: could you +2, please? ^ | 09:49 |
| mordred | AJaeger: whoops! +3d it. (that's an obvious error) | 09:50 |
| * mordred feels silly for missing that | 09:50 | |
| openstackgerrit | Merged openstack/project-config master: Announce opendev/project-config changes in channels https://review.opendev.org/682757 | 09:50 |
| * AJaeger feels silly for leaving it in - thanks, mordred | 09:51 | |
| AJaeger | thanks, frickler ! | 09:51 |
| *** yamamoto has joined #openstack-infra | 09:58 | |
| *** markvoelker has joined #openstack-infra | 09:59 | |
| openstackgerrit | Fabien Boucher proposed zuul/zuul master: Pagure - fix wrong commit gitweb url https://review.opendev.org/679946 | 10:02 |
| openstackgerrit | Fabien Boucher proposed zuul/zuul master: Pagure - handle initial comment change event https://review.opendev.org/680310 | 10:02 |
| openstackgerrit | Fabien Boucher proposed zuul/zuul master: Pagure - handle Pull Request tags (labels) metadata https://review.opendev.org/681050 | 10:02 |
| openstackgerrit | Fabien Boucher proposed zuul/zuul master: Pagure - reference pipelines add open: True requirement https://review.opendev.org/681252 | 10:02 |
| openstackgerrit | Fabien Boucher proposed zuul/zuul master: Pagure - handles pull-request.closed event https://review.opendev.org/681279 | 10:02 |
| *** pgaxatte has quit IRC | 10:03 | |
| *** markvoelker has quit IRC | 10:04 | |
| *** openstackgerrit has quit IRC | 10:06 | |
| *** ociuhandu has quit IRC | 10:07 | |
| *** yamamoto has quit IRC | 10:07 | |
| *** openstackgerrit has joined #openstack-infra | 10:10 | |
| openstackgerrit | Merged openstack/openstack-zuul-jobs master: Fix horizon translation setup https://review.opendev.org/682850 | 10:10 |
| *** zbr has quit IRC | 10:11 | |
| *** dciabrin has joined #openstack-infra | 10:12 | |
| *** zbr has joined #openstack-infra | 10:12 | |
| *** kjackal has quit IRC | 10:12 | |
| *** whoami-rajat has quit IRC | 10:13 | |
| *** kjackal has joined #openstack-infra | 10:13 | |
| *** pgaxatte has joined #openstack-infra | 10:14 | |
| *** ociuhandu has joined #openstack-infra | 10:14 | |
| *** markvoelker has joined #openstack-infra | 10:16 | |
| *** dciabrin has quit IRC | 10:17 | |
| *** ociuhandu_ has joined #openstack-infra | 10:18 | |
| AJaeger | yeah, fixed - just saw first successful run | 10:18 |
| *** kjackal has quit IRC | 10:19 | |
| *** ramishra has quit IRC | 10:20 | |
| *** markvoelker has quit IRC | 10:20 | |
| *** ociuhandu has quit IRC | 10:22 | |
| openstackgerrit | Fabien Boucher proposed zuul/zuul master: Pagure - add support for git.tag.creation event https://review.opendev.org/679938 | 10:30 |
| *** pgaxatte has quit IRC | 10:30 | |
| *** ykarel|lunch is now known as ykarel | 10:30 | |
| mordred | AJaeger: \o/ | 10:31 |
| *** whoami-rajat has joined #openstack-infra | 10:34 | |
| *** dciabrin has joined #openstack-infra | 10:37 | |
| *** yamamoto has joined #openstack-infra | 10:42 | |
| *** exsdev0 has joined #openstack-infra | 10:46 | |
| *** exsdev has quit IRC | 10:47 | |
| *** exsdev0 is now known as exsdev | 10:47 | |
| *** AJaeger has quit IRC | 10:55 | |
| openstackgerrit | Merged zuul/zuul master: Pagure - fix wrong commit gitweb url https://review.opendev.org/679946 | 10:57 |
| *** udesale has quit IRC | 11:01 | |
| *** AJaeger has joined #openstack-infra | 11:01 | |
| *** udesale has joined #openstack-infra | 11:02 | |
| *** yamamoto has quit IRC | 11:12 | |
| *** lucasagomes has joined #openstack-infra | 11:13 | |
| *** pcaruana has quit IRC | 11:19 | |
| *** e0ne_ has joined #openstack-infra | 11:20 | |
| *** e0ne has quit IRC | 11:23 | |
| *** dave-mccowan has joined #openstack-infra | 11:23 | |
| *** armax has joined #openstack-infra | 11:24 | |
| *** psachin has joined #openstack-infra | 11:25 | |
| *** pcaruana has joined #openstack-infra | 11:28 | |
| *** iurygregory_ has joined #openstack-infra | 11:29 | |
| *** iurygregory has quit IRC | 11:29 | |
| *** jpena is now known as jpena|lunch | 11:35 | |
| *** zhangfei has quit IRC | 11:42 | |
| *** udesale has quit IRC | 11:44 | |
| ttx | mgoddard: recent removal of #openstack-kayobe from accessbot is confusing irc-meetings, since you still have #openstack-kayobe in https://opendev.org/opendev/irc-meetings/src/branch/master/meetings/kayobe-team-meeting.yaml | 11:49 |
| ttx | mgoddard: should we remove / change that meeting? | 11:50 |
| *** yamamoto has joined #openstack-infra | 11:59 | |
| fungi | i generally don't recommend removing channels from accessbot unless the channel owners really want infra to stop managing their access controls | 12:00 |
| fungi | there's nothing wrong with keeping access lists up to date on defunct channels, in my opinion | 12:01 |
| AJaeger | fungi, the channel redirects now to another one, so is dead | 12:01 |
| *** yamamoto has quit IRC | 12:02 | |
| *** yamamoto has joined #openstack-infra | 12:02 | |
| AJaeger | fungi: it redirects to #openstack-kolla - don't we remove it in that case? | 12:02 |
| *** pgaxatte has joined #openstack-infra | 12:03 | |
| fungi | what if they want to undo the redirect in the future but we've switched which account we use to manage channels? | 12:03 |
| fungi | it's not as if accessbot joins any channels, it just checks/updates chanserv acls, so additional channels aren't a significant overhead for it | 12:04 |
| AJaeger | fungi: Ah! Wasn't aware that the redirect is only limited in this way and therefore accessbot is useful. Let me propose a re-add | 12:05 |
| *** markvoelker has joined #openstack-infra | 12:05 | |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Revert "Remove accessbot from #openstack-kayobe" https://review.opendev.org/682881 | 12:06 |
| AJaeger | fungi: ^ | 12:06 |
| AJaeger | mgoddard: sorry for wrong advise on the accessbot removal ^ | 12:06 |
| *** janki has quit IRC | 12:07 | |
| * fungi disappears for morning exercise, back in a while | 12:08 | |
| *** jamesmcarthur has joined #openstack-infra | 12:09 | |
| *** rh-jelabarre has joined #openstack-infra | 12:10 | |
| *** weshay has joined #openstack-infra | 12:12 | |
| *** goldyfruit_ has quit IRC | 12:15 | |
| *** jamesmcarthur has quit IRC | 12:16 | |
| *** armax has quit IRC | 12:20 | |
| *** jamesmcarthur has joined #openstack-infra | 12:22 | |
| *** derekh has quit IRC | 12:24 | |
| *** rlandy has joined #openstack-infra | 12:25 | |
| *** armax has joined #openstack-infra | 12:28 | |
| *** jamesmcarthur has quit IRC | 12:31 | |
| *** jpena|lunch is now known as jpena | 12:31 | |
| openstackgerrit | Merged openstack/project-config master: Revert "Remove accessbot from #openstack-kayobe" https://review.opendev.org/682881 | 12:31 |
| openstackgerrit | David Shrewsbury proposed zuul/zuul master: Add scheduler config options for hold expiration https://review.opendev.org/682675 | 12:42 |
| mgoddard | AJaeger: np | 12:44 |
| *** xenos76 has quit IRC | 12:46 | |
| *** fdegir has quit IRC | 12:47 | |
| *** georgk has quit IRC | 12:47 | |
| *** fdegir has joined #openstack-infra | 12:47 | |
| *** georgk has joined #openstack-infra | 12:47 | |
| openstackgerrit | David Shrewsbury proposed zuul/zuul master: Add scheduler config options for hold expiration https://review.opendev.org/682675 | 12:48 |
| *** jamesmcarthur has joined #openstack-infra | 12:51 | |
| *** mriedem has joined #openstack-infra | 12:51 | |
| AJaeger | mgoddard: did you see ttx' ping for the IRC meeting? | 12:52 |
| *** janki has joined #openstack-infra | 12:52 | |
| *** iurygregory has joined #openstack-infra | 12:53 | |
| *** yamamoto has quit IRC | 12:53 | |
| *** janki has quit IRC | 12:53 | |
| *** iurygregory_ has quit IRC | 12:55 | |
| *** yamamoto has joined #openstack-infra | 12:57 | |
| *** jcoufal has joined #openstack-infra | 13:01 | |
| *** ramishra has joined #openstack-infra | 13:01 | |
| *** JorgeFranco has joined #openstack-infra | 13:03 | |
| *** derekh has joined #openstack-infra | 13:04 | |
| AJaeger | fungi: a few more repos for your system-config change: https://review.opendev.org/#/c/682771/ - see also email thread | 13:04 |
| *** ociuhandu_ has quit IRC | 13:06 | |
| *** ricolin_ has joined #openstack-infra | 13:08 | |
| *** Goneri has joined #openstack-infra | 13:09 | |
| *** ricolin has quit IRC | 13:11 | |
| *** ricolin_ is now known as ricolin | 13:12 | |
| *** eharney has joined #openstack-infra | 13:17 | |
| *** ociuhandu has joined #openstack-infra | 13:18 | |
| *** eharney has quit IRC | 13:19 | |
| mgoddard | AJaeger: oh no I didn't, thanks | 13:20 |
| mgoddard | ttx: I'll remove the kayobe meeting from irc-meetings | 13:20 |
| *** eharney has joined #openstack-infra | 13:20 | |
| openstackgerrit | Mark Goddard proposed opendev/irc-meetings master: Remove kayobe team meeting https://review.opendev.org/682894 | 13:22 |
| *** goldyfruit_ has joined #openstack-infra | 13:30 | |
| *** whoami-rajat has quit IRC | 13:33 | |
| *** psachin has quit IRC | 13:33 | |
| *** aaronsheffield has joined #openstack-infra | 13:34 | |
| *** armax has quit IRC | 13:35 | |
| *** ociuhandu has quit IRC | 13:35 | |
| *** armax has joined #openstack-infra | 13:36 | |
| ttx | mgoddard: thanks! | 13:37 |
| mgoddard | ttx: np | 13:38 |
| AJaeger | config-core, please review https://review.opendev.org/682216 and https://review.opendev.org/682215 to finish switching specs.o.o publishing to promote jobs | 13:41 |
| *** yamamoto has quit IRC | 13:49 | |
| *** xenos76 has joined #openstack-infra | 13:55 | |
| openstackgerrit | Merged openstack/project-config master: Stop gerrit notification to neutron-stadium specific channels https://review.opendev.org/682771 | 14:02 |
| *** ykarel is now known as ykarel|afk | 14:04 | |
| *** xek_ has joined #openstack-infra | 14:06 | |
| openstackgerrit | David Shrewsbury proposed zuul/zuul master: Add scheduler config options for hold expiration https://review.opendev.org/682675 | 14:07 |
| *** dtantsur is now known as dtantsur|afk | 14:10 | |
| openstackgerrit | David Shrewsbury proposed zuul/zuul master: Mark nodes as USED when deleting autohold https://review.opendev.org/664060 | 14:12 |
| openstackgerrit | David Shrewsbury proposed zuul/zuul master: Auto-delete expired autohold requests https://review.opendev.org/663762 | 14:12 |
| openstackgerrit | David Shrewsbury proposed zuul/zuul master: Add autohold delete/info commands to web API https://review.opendev.org/679057 | 14:12 |
| openstackgerrit | David Shrewsbury proposed zuul/zuul master: Remove outdated TODO https://review.opendev.org/682421 | 14:12 |
| *** kjackal has joined #openstack-infra | 14:17 | |
| *** openstackgerrit has quit IRC | 14:21 | |
| *** pgaxatte has quit IRC | 14:27 | |
| *** yamamoto has joined #openstack-infra | 14:28 | |
| *** kjackal has quit IRC | 14:29 | |
| *** ociuhandu has joined #openstack-infra | 14:30 | |
| mnaser | infra-root: i think openstackgerrit bot died? or maybe its still joining | 14:36 |
| *** openstackgerrit has joined #openstack-infra | 14:36 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: DNM: test prepare-workspace-git base-test https://review.opendev.org/682912 | 14:36 |
| *** yamamoto has quit IRC | 14:37 | |
| corvus | mnaser: ^ seems to work -- is it not working in some other channel? | 14:37 |
| mnaser | corvus: it literally joined ONE second before you made that change | 14:37 |
| mnaser | lol | 14:37 |
| mnaser | unless it joins-on-demand or something post restart (since 682771 merged) | 14:37 |
| corvus | mnaser: it leaves and joins channels based on activity | 14:37 |
| mnaser | aaaaaaaah | 14:37 |
| *** yamamoto has joined #openstack-infra | 14:37 | |
| mnaser | ok that explains it, sorry i just saw it leave and never join again so i figured i'd let people know | 14:38 |
| mnaser | sorry for my human nagios failure :) | 14:38 |
| corvus | mnaser: you're still ahead of real nagios :) | 14:38 |
| *** yamamoto has quit IRC | 14:40 | |
| *** yamamoto has joined #openstack-infra | 14:40 | |
| fungi | mnaser: yeah, it's a workaround for freenode's limit of 120 simultaneous channels for a single account | 14:42 |
| mnaser | neat-o | 14:43 |
| fungi | gerritbot leaves less-active channels when it needs to join others to announce things, in order to keep its joined channel count at or below 120 at all times | 14:43 |
| fungi | slightly more efficient than just sharding the configuration across instances for separate freenode accounts | 14:44 |
| *** priteau has quit IRC | 14:47 | |
| *** priteau has joined #openstack-infra | 14:51 | |
| mnaser | ++ | 14:51 |
| *** ykarel|afk is now known as ykarel | 14:54 | |
| *** priteau has quit IRC | 14:56 | |
| openstackgerrit | Merged opendev/irc-meetings master: Remove kayobe team meeting https://review.opendev.org/682894 | 14:57 |
| *** jaosorior has quit IRC | 14:57 | |
| *** priteau has joined #openstack-infra | 14:58 | |
| openstackgerrit | Merged zuul/zuul master: Pagure - handle initial comment change event https://review.opendev.org/680310 | 14:59 |
| *** eernst has joined #openstack-infra | 15:04 | |
| *** iurygregory has quit IRC | 15:05 | |
| openstackgerrit | Merged zuul/zuul-website master: Update to page titles and Users https://review.opendev.org/680459 | 15:06 |
| *** arxcruz is now known as arxcruz|ruck | 15:08 | |
| *** gyee has joined #openstack-infra | 15:08 | |
| *** mattw4 has joined #openstack-infra | 15:09 | |
| *** TxGirlGeek has joined #openstack-infra | 15:10 | |
| *** xek__ has joined #openstack-infra | 15:12 | |
| *** zul has joined #openstack-infra | 15:13 | |
| *** xek_ has quit IRC | 15:14 | |
| clarkb | mgoddard: fwiw ovs is namespaced and you can run a separate vswitchd in a different namespace aiui | 15:15 |
| clarkb | if however you are using the host network namespace then ya I think you have to defer to the host | 15:15 |
| clarkb | similar to how you wouldnt run a competing dhcpcd in a container | 15:16 |
| openstackgerrit | Merged zuul/zuul master: zuul_console: fix python 3 support https://review.opendev.org/682556 | 15:17 |
| *** michael-beaver has joined #openstack-infra | 15:19 | |
| clarkb | stephenfin: "clean check" was applied to openstack's pipelines by sdague in an effort to keep flaky changes out of the gate | 15:20 |
| *** goldyfruit_ has quit IRC | 15:20 | |
| clarkb | stephenfin: there were two major issues he was trying to addres with this (though this is all from memory so maybe I remwmber wrong) | 15:20 |
| fungi | basically if your change fails in the gate, then there's an increased chance it's introducing nondeterministic failure behavior so forcing it to go through check again helps make that more apparent | 15:21 |
| clarkb | stephenfin: first was people approving changes that hadno hope of ever passing due to pep8 or other trivial errors. The change would then enter the gate and causehavoc wih resets | 15:21 |
| clarkb | and ya second thing is what fungi points out | 15:21 |
| fungi | well, we also had a stale check guard in place for a while to help deal with that first point | 15:22 |
| clarkb | we root caused how a handful of gate instability bugs managed to merge and most ot them were the result of gettibg lucky after a number of reverifies | 15:22 |
| fungi | because people would approve changes which had been sitting around with a 6-month-old passing check | 15:22 |
| fungi | and then wreck the gate as they went through unable to pass the current state of jobs for the repo | 15:23 |
| corvus | clarkb: can you join #opendev? | 15:23 |
| stephenfin | Ah, those are all good point. I hadn't considered the knock on effect to the queue of jobs in the gate | 15:24 |
| clarkb | stephenfin: this is why whenever this topic of queue times comes up I push so hard on starting with gate stability | 15:27 |
| clarkb | any other changes we make will be completely wiped out by a flaky gate | 15:27 |
| clarkb | and a flaky gate implies our software is flaky | 15:27 |
| clarkb | fixing it is win win | 15:27 |
| stephenfin | yup, for sure | 15:27 |
| stephenfin | well, _someone's_ software is flaky | 15:27 |
| stephenfin | my software is perfect(TM) | 15:28 |
| * stephenfin kids, obviously | 15:28 | |
| clarkb | we prioritize the gate because they are changes in a state reviewed by humans to both be "good" and ready to merge | 15:29 |
| clarkb | when the gate resets we stop all jobs behind the rest point and start them all again with a higher priority than all check jobs | 15:30 |
| clarkb | this takes resources from your changes in check | 15:30 |
| *** liuyulong has joined #openstack-infra | 15:30 | |
| clarkb | start doing that every 15-45 minutes and you never free up resources for check | 15:31 |
| clarkb | this has been our sitaution all through feature freeze with openstack integrated and tripleo trading off on who gets to reset | 15:32 |
| *** iurygregory has joined #openstack-infra | 15:34 | |
| *** markvoelker has quit IRC | 15:35 | |
| *** david-lyle has quit IRC | 15:35 | |
| *** dklyle has joined #openstack-infra | 15:35 | |
| *** zbr has quit IRC | 15:36 | |
| *** zbr has joined #openstack-infra | 15:36 | |
| *** zzehring has quit IRC | 15:37 | |
| *** sshnaidm|rover is now known as sshnaidm | 15:39 | |
| *** dtantsur|afk is now known as dtantsur | 15:39 | |
| *** panda|ruck is now known as panda | 15:39 | |
| *** zzehring has joined #openstack-infra | 15:40 | |
| *** trident has quit IRC | 15:41 | |
| liuyulong | Hi guys, how to make the running instance new NIC get the right IPv6 address with NetworkManager? Or it is impossible? | 15:44 |
| clarkb | liuyulong: are you asking about our test instances? | 15:44 |
| liuyulong | clarkb, no, Donny told me, our CI uses glean. | 15:45 |
| *** jamesmcarthur has quit IRC | 15:45 | |
| liuyulong | I mean upstream CI. | 15:45 |
| liuyulong | So why not NetworkManager? | 15:46 |
| openstackgerrit | Andreas Jaeger proposed openstack/infra-manual master: Document "clean check" requirement https://review.opendev.org/682933 | 15:46 |
| AJaeger | stephenfin, fungi, clarkb , what about this ^ ? | 15:46 |
| clarkb | liuyulong: we do use network manager on centos and fedora. Glean is the piece between reading cloud metadata and network manager | 15:47 |
| *** ociuhandu has quit IRC | 15:47 | |
| *** jamesmcarthur has joined #openstack-infra | 15:47 | |
| clarkb | on other platforms glean sits between metadata and the platform specifc network config /etc/network/interfaces on debuntu, systemd-networkd on gentoo and so on | 15:47 |
| *** ociuhandu has joined #openstack-infra | 15:48 | |
| clarkb | for ipv6 specifically some platforms never had static configuration of ipv6 addresses added to glean | 15:48 |
| openstackgerrit | James E. Blair proposed openstack/project-config master: Symlink /opt/git to /opt/git/opendev.org https://review.opendev.org/682934 | 15:49 |
| liuyulong | clarkb, great news, we are tunning our local image NetworkManager settings. But everytime the running instances' new NIC will have an incorrect IPv6 address. | 15:49 |
| openstackgerrit | James E. Blair proposed openstack/project-config master: Cache repos in /opt/git/opendev.org https://review.opendev.org/682935 | 15:49 |
| *** ociuhandu has quit IRC | 15:49 | |
| clarkb | they function if there are RAs but not if you are expected to manually co figure the IPs | 15:49 |
| liuyulong | The address is not the port fixed IPv6 addr. | 15:49 |
| *** ociuhandu has joined #openstack-infra | 15:49 | |
| *** jamesmcarthur has quit IRC | 15:49 | |
| liuyulong | The image is CentOS7.5. FYI | 15:50 |
| *** ykarel is now known as ykarel|away | 15:50 | |
| clarkb | if we arent talking about our images then I wouldnt know where to start. Are you receiving router advertisements? | 15:50 |
| AJaeger | clarkb, could you please review https://review.opendev.org/682216 and https://review.opendev.org/682215 to finish switching specs.o.o publishing to promote jobs? | 15:51 |
| clarkb | or are you statically configuring the IP? | 15:51 |
| *** jamesmcarthur has joined #openstack-infra | 15:52 | |
| liuyulong | RA packets are received. IP was configured automatically. | 15:52 |
| *** trident has joined #openstack-infra | 15:53 | |
| *** markvoelker has joined #openstack-infra | 15:53 | |
| *** kjackal has joined #openstack-infra | 15:53 | |
| liuyulong | IPv4 address is configured correctly. IPv6 address has the right perfix. | 15:54 |
| openstackgerrit | James E. Blair proposed openstack/project-config master: Cache repos in /opt/git/opendev.org https://review.opendev.org/682935 | 15:54 |
| *** ramishra has quit IRC | 15:55 | |
| clarkb | liuyulong: I think that points at a problem in neutron then? if it thinks the port has one mac address but another is used? The other thing to check is whether or not hardware address obfuscation is enabled on your image | 15:55 |
| clarkb | fungi: ^ I forget how to check that but you probably remember | 15:55 |
| liuyulong | clarkb, but the entire IPv6 address was incorrect. So, since the MAC is correct, we may want to see if NetworkManager can use the EUI64 to generate and config the address. | 15:56 |
| clarkb | liuyulong: you said it has the right prefix | 15:57 |
| *** jamesmcarthur has quit IRC | 15:57 | |
| liuyulong | clarkb, yes, the our prefix len is /48, the Neutron subnet pool size is /64. | 15:58 |
| clarkb | that sounds like a misconfigured neutron then? or something else is sending RAs too? | 15:58 |
| *** armax has quit IRC | 15:58 | |
| *** TxGirlGeek has quit IRC | 15:58 | |
| *** jamesmcarthur has joined #openstack-infra | 15:59 | |
| *** eernst has quit IRC | 16:00 | |
| liuyulong | clarkb, 240e:980:xxxx::/48 is for the subnet pool with a default /64 prefix len for our subnets. | 16:03 |
| *** rpittau is now known as rpittau|afk | 16:03 | |
| liuyulong | clarkb, Then I have a subnet with prefix 240e:980:xxxx:8009::/64, and a port 240e:980:xxxx:8009:f816:3eff:fee1:4af1 with MAC fa:16:3e:e1:4a:f1. | 16:04 |
| liuyulong | 10: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 | 16:04 |
| liuyulong | link/ether fa:16:3e:e1:4a:f1 brd ff:ff:ff:ff:ff:ff | 16:04 |
| liuyulong | inet 192.168.1.30/24 brd 192.168.1.255 scope global noprefixroute eth1 | 16:04 |
| liuyulong | valid_lft forever preferred_lft forever | 16:04 |
| liuyulong | inet6 240e:980:xxxx:8009:fb5d:500:74bd:2700/64 scope global noprefixroute dynamic | 16:04 |
| liuyulong | This is the new NIC inside the VM. | 16:04 |
| clarkb | liuyulong: please use a paste service like http://paste.openstack.org for pasting large chunks of lines in the future. It helps with readability of the channel | 16:05 |
| *** kjackal has quit IRC | 16:05 | |
| *** ykarel|away has quit IRC | 16:05 | |
| clarkb | so ya that looks like you are getting the correct prefix | 16:05 |
| clarkb | I think that may be caused by hardware obfuscation? | 16:06 |
| *** yamamoto has quit IRC | 16:06 | |
| clarkb | I want to say it is a sysctl setting to disable that | 16:06 |
| liuyulong | clarkb, yes, the prefix is correct. Just have no idea how this "240e:980:xxxx:8009:fb5d:500:74bd:2700" was coming form. | 16:06 |
| *** zbr is now known as zbr|ruck | 16:07 | |
| liuyulong | net.ipv6.conf.*.use_tempaddr and net.ipv6.conf.*.addr_gen_mode ? | 16:07 |
| clarkb | liuyulong: https://tools.ietf.org/html/rfc4941 is the rfc | 16:07 |
| openstackgerrit | Andreas Jaeger proposed openstack/infra-manual master: Document "clean check" requirement https://review.opendev.org/682933 | 16:08 |
| clarkb | liuyulong: ya what is the value of use_tempaddr for eth1? | 16:08 |
| *** zzehring has quit IRC | 16:09 | |
| openstackgerrit | James E. Blair proposed openstack/project-config master: Symlink /opt/git to /opt/git/opendev.org https://review.opendev.org/682934 | 16:09 |
| openstackgerrit | James E. Blair proposed openstack/project-config master: Cache repos in /opt/git/opendev.org https://review.opendev.org/682935 | 16:09 |
| liuyulong | clarkb, this will be the problem, the eth1 was new added to the instance, it can not be set. This value will inherit the 'default' and 'all' ? | 16:10 |
| fungi | correct, you'll need to disable ipv6 tempaddr support via sysctl configuration | 16:10 |
| liuyulong | net.ipv6.conf.all.use_tempaddr = 0 | 16:11 |
| fungi | neutron doesn't work with it as far as i know | 16:11 |
| *** xek__ has quit IRC | 16:11 | |
| clarkb | liuyulong: what is it on your running instance? you were able to check the ip addr output, can you check the sysctl for that same eth1 interface? | 16:11 |
| haleyb | https://docs.openstack.org/neutron/latest/admin/config-ipv6.html#configuring-interfaces-of-the-guest | 16:11 |
| fungi | thanks haleyb! | 16:12 |
| *** kjackal has joined #openstack-infra | 16:12 | |
| haleyb | windows guests trip over that a lot, but lately linux distros have been enabling it as well | 16:12 |
| *** gfidente has quit IRC | 16:12 | |
| *** efried is now known as efried_pto | 16:13 | |
| *** Zara has joined #openstack-infra | 16:13 | |
| AJaeger | corvus: want to WIP 682935 ? | 16:13 |
| AJaeger | corvus: Ah, the depends-on has WIP - so, we're good. Ignore me... | 16:14 |
| liuyulong | for new added NIC on running instance, the value is net.ipv6.conf.*.use_tempaddr = 0 | 16:14 |
| openstackgerrit | Merged opendev/irc-meetings master: Updated the Diversity WG to monthly https://review.opendev.org/682727 | 16:15 |
| liuyulong | # uname -r | 16:15 |
| liuyulong | 3.10.0-862.el7.x86_6 | 16:15 |
| liuyulong | 3.10.0-862.el7.x86_64 | 16:15 |
| fungi | liuyulong: one possibility is, depending on how you set your sysctl variables at boot, interface autoconfiguration could be racing whatever disables privacy extensions and opaque identifiers | 16:16 |
| *** kjackal has quit IRC | 16:16 | |
| fungi | so by the time those variables get set back to 0 the interface has already set an address? | 16:17 |
| openstackgerrit | Merged openstack/openstack-zuul-jobs master: Switch project-config to promote jobs [5]: Update specs, cleanup https://review.opendev.org/682215 | 16:17 |
| spotz | thanks ttx! | 16:18 |
| fungi | liuyulong: is addr_gen_mode being set to 0 too? | 16:19 |
| clarkb | liuyulong: fungi it is also possible that network manager configures it separetly of sysctl | 16:19 |
| fungi | (per the document haleyb linked above) | 16:19 |
| liuyulong | fungi, sure, let me test it again to see if it has race... | 16:19 |
| *** kjackal has joined #openstack-infra | 16:20 | |
| fungi | clarkb: oh, yep there's that as well, we've seen kernel interface autoconfiguration and network-manager race each other to decide who gets to set the interface's v6 addresses | 16:20 |
| *** zzehring has joined #openstack-infra | 16:22 | |
| liuyulong | The naked eye cannot distinguish...It is too quick... | 16:23 |
| *** gmann is now known as gmann_afk | 16:23 | |
| liuyulong | net.ipv6.conf.eth3.use_tempaddr is 0 all the time before the address added. | 16:24 |
| haleyb | liuyulong: and NM hasn't set addr-gen-mode or ip6-privacy? https://developer.gnome.org/NetworkManager/unstable/settings-ipv6.html says it defaults the first to privacy | 16:27 |
| liuyulong | [connection] | 16:28 |
| liuyulong | ipv4.route-metric=0 | 16:28 |
| liuyulong | ipv6.method=auto | 16:28 |
| liuyulong | ipv6.ip6-privacy=0 | 16:28 |
| liuyulong | ipv6.addr-gen-mode=0 | 16:28 |
| liuyulong | addr-gen-mode NM_SETTING_IP6_CONFIG_ADDR_GEN_MODE_EUI64 (0) | 16:28 |
| *** goldyfruit_ has joined #openstack-infra | 16:28 | |
| liuyulong | ip6-privacy -1: unknown, 0: disabled, 1: enabled (prefer public address), 2: enabled | 16:29 |
| liuyulong | This is the settings of /etc/NetworkManager/NetworkManager.conf | 16:29 |
| clarkb | liuyulong: haleyb fungi as a sanity check liuyulong may want to boot our centos 7 image and see if it fares any better? | 16:29 |
| *** kopecmartin is now known as kopecmartin|off | 16:29 | |
| openstackgerrit | Merged openstack/project-config master: Symlink /opt/git to /opt/git/opendev.org https://review.opendev.org/682934 | 16:30 |
| clarkb | liuyulong: https://nb02.openstack.org/images/centos-7-0000054888.qcow2 there are md5sums and sha256sums in that dir too | 16:30 |
| liuyulong | clarkb, cool | 16:30 |
| clarkb | glean will configure the root user with your supplied ssh key | 16:30 |
| clarkb | liuyulong: on my desktop the nm gui configurator tool does have options for enabling privacy extensions | 16:33 |
| clarkb | so it could be that nm is doing this independent of your sysctl values | 16:33 |
| liuyulong | clarkb, yes, nmcli connection show the real value does not match the "NetworkManager.conf". | 16:34 |
| *** dtantsur is now known as dtantsur|afk | 16:38 | |
| *** derekh has quit IRC | 16:41 | |
| liuyulong | Thank you guys, the image download is a bit slow. I will update my testing here. | 16:45 |
| liuyulong | It's 0:45 in Beijing now. I had to go bed now. | 16:45 |
| *** liuyulong has quit IRC | 16:47 | |
| *** tosky has quit IRC | 16:49 | |
| *** e0ne_ has quit IRC | 16:50 | |
| mnaser | out of curiosity | 16:52 |
| mnaser | hows nodepool coping with centos 7.7 | 16:52 |
| *** tesseract has quit IRC | 16:53 | |
| fungi | i'm not sure we know yet | 16:55 |
| clarkb | mnaser: haven't seen any complaints yet, but also haven't checked that we've built and uploaded it anywhere either | 16:55 |
| mnaser | yeah i wonder if its just been failing to buidl for a while :P | 16:55 |
| clarkb | well the image I linked liuyulong to is from today | 16:55 |
| mnaser | oh ok | 16:56 |
| clarkb | about 10 hours ago | 16:56 |
| mnaser | so at least we know they're still building | 16:56 |
| clarkb | ya should be building fine | 16:56 |
| mnaser | but maybe mirrors havent syncd yet too | 16:56 |
| mnaser | the infra ones at least | 16:56 |
| fungi | yeah, here's the image build log: https://nb02.openstack.org/centos-7-0000054888.log | 16:56 |
| pabelanger | I'm seeing mirror issues in zuul.a.c for centos-7, but haven't dug into it yet. | 16:56 |
| fungi | 2019-09-18 06:24:22.626 | Build completed successfully | 16:56 |
| fungi | so we started uploading it to providers around 10.5 hours ago | 16:57 |
| mnaser | sudo rpm --dbpath=/var/lib/rpm --root /opt/dib_tmp/dib_build.am3pYeSd/mnt --nodeps -ivh /tmp/tmp.IFHWiSygY6/basesystem-10.0-7.el7.centos.noarch.rpm /tmp/tmp.IFHWiSygY6/centos-release-7-7.1908.0.el7.centos.x86_64.rpm /tmp/tmp.IFHWiSygY6/filesystem-3.2-25.el7.x86_64.rpm /tmp/tmp.IFHWiSygY6/setup-2.8.71-10.el7.noarch.rpm | 16:57 |
| mnaser | it is installing centos-release-7.7.1908 | 16:57 |
| mnaser | cool! | 16:57 |
| *** xenos76 has quit IRC | 16:57 | |
| openstackgerrit | Merged zuul/zuul master: Add support for the Gerrit checks plugin https://review.opendev.org/680778 | 16:59 |
| fungi | mnaser: http://zuul.opendev.org/t/openstack/build/0a18d82c6ceb495ba6b1d1e1b9225f69/log/logs/ansible/facts-all.log.txt.gz#1215-1221 | 17:04 |
| fungi | yep | 17:04 |
| mnaser | wewt prefect | 17:04 |
| fungi | that build was a successful centos-7 job for openstack-ansible btw | 17:05 |
| fungi | just grabbed at random from the zuul builds page | 17:05 |
| mnaser | who else but us and tripleo mostly using centos-7 :p | 17:06 |
| fungi | right, and i wasn't 100% sure a tripleo job would provide a good example | 17:06 |
| fungi | so i filtered by openstack-ansible | 17:06 |
| openstackgerrit | Merged zuul/zuul master: Update gerrit pagination test fixtures https://review.opendev.org/682114 | 17:08 |
| *** jamesmcarthur_ has joined #openstack-infra | 17:09 | |
| *** jamesmcarthur has quit IRC | 17:12 | |
| *** jpena is now known as jpena|off | 17:19 | |
| *** whoami-rajat has joined #openstack-infra | 17:20 | |
| *** armax has joined #openstack-infra | 17:26 | |
| *** ricolin has quit IRC | 17:27 | |
| *** armax has quit IRC | 17:31 | |
| openstackgerrit | Merged openstack/project-config master: Switch project-config to promote jobs [6]: Remove publish-openstack-specs https://review.opendev.org/682216 | 17:31 |
| AJaeger | config-core, a couple of open changes for review, please: https://review.opendev.org/682639 https://review.opendev.org/680830 https://review.opendev.org/682613 https://review.opendev.org/#/c/682581/ | 17:33 |
| *** yamamoto has joined #openstack-infra | 17:33 | |
| AJaeger | clarkb: what's the status of https://review.opendev.org/#/c/681322/ ? did we restart Zuul with the dependency merged and thus can merge the cleanup playbook? | 17:34 |
| clarkb | AJaeger: on https://review.opendev.org/#/c/680830/ I asked pabelanger if he wanted to use one of the popular galaxy roles for ES instead and didn't really get a firm answer either way. if we can confirm that using one of those is undesireable then we should go ahead and merge it ( thats mostly me just double checking stuff and from an opendev perspective the change is fine ) | 17:35 |
| clarkb | AJaeger: good question I believe we did restart with the necessary change. I'll rerun my test now | 17:35 |
| *** iurygregory has quit IRC | 17:35 | |
| openstackgerrit | Merged zuul/zuul master: Support HTTP-only Gerrit https://review.opendev.org/681936 | 17:35 |
| *** rkukura has joined #openstack-infra | 17:35 | |
| AJaeger | clarkb: ok, let's wait for pabelanger to confirm. | 17:36 |
| pabelanger | clarkb: AJaeger: yah, ones on galaxy are to agressive in what they configure. Looking to create a very thing shim to package install / config file laydown | 17:36 |
| *** zbr|ruck is now known as zbr | 17:36 | |
| clarkb | pabelanger: can you check the comment on that change I noticed one other small thing you should confirm or update before we merge | 17:37 |
| pabelanger | sure | 17:38 |
| *** yamamoto has quit IRC | 17:38 | |
| openstackgerrit | Paul Belanger proposed openstack/project-config master: Add windmill-ops / ansible-role-elasticsearch https://review.opendev.org/680830 | 17:40 |
| *** armax has joined #openstack-infra | 17:42 | |
| *** jtomasek has quit IRC | 17:43 | |
| openstackgerrit | Paul Belanger proposed openstack/project-config master: Set requireContributorAgreement false for windmill roles https://review.opendev.org/682954 | 17:44 |
| AJaeger | clarkb: see what you triggered ;) ^ | 17:45 |
| clarkb | AJaeger: :P | 17:46 |
| denaitre | Hey everyone! I am quite new regarding Zuul and the OpenStack testing process, and I hope this chan is the right place to ask my questions. I would like to know how many times OpenStack is deployed per day/week by the CI process. If I understood correctly, OpenStack is deployed with devstack for each integration tests. Given that nearly 2500 changes are commited each week, I guess OpenStack is de | 17:47 |
| denaitre | ployed a large number of times. Is it possible to get this number from Zuul? Maybe by investigating the parent job http://zuul.openstack.org/job/devstack-base ? | 17:47 |
| openstackgerrit | Merged openstack/project-config master: Update gentoo image to current profile https://review.opendev.org/682639 | 17:47 |
| *** e0ne has joined #openstack-infra | 17:48 | |
| fungi | denaitre: there are many different ways openstack can be deployed by jobs (devstack, openstack-ansible, kolla/loci, openstack-helm, tripleo, puppet-openstack...) so that complicates identifying a solid count | 17:49 |
| *** e0ne has quit IRC | 17:49 | |
| mordred | yeah. but - the data should ultimately be able to be mined :) | 17:49 |
| openstackgerrit | Clark Boylan proposed zuul/zuul-jobs master: DO NOT MERGE test cleanup phase playbook https://review.opendev.org/680178 | 17:49 |
| fungi | i agree, it would just take a fair amount of knowing those sorts of details | 17:49 |
| clarkb | I think the last time I counted devstack-gate executions it was on the order of 10k/day | 17:50 |
| mordred | yeah. that said - taking a first stab at finding all of the jobs that have the devstack base job as a base job in their hierarchy | 17:50 |
| clarkb | but this was a couple years ago so things may have changed | 17:50 |
| AJaeger | team, FYI, specs publishing works http://specs.openstack.org/openstack/keystone-specs/ just got updated via promote job. So, all those changes are tested and look fine. | 17:50 |
| mordred | well - actually - you'd want devstack and devstack-gate base jobs (to get modern and legacy versions) | 17:50 |
| mordred | it wouldn't be _perfect_ - but it would be a good first number | 17:51 |
| mordred | then adding the list of base jobs like fungi mentions as a second pass to improve the numbers | 17:51 |
| clarkb | denaitre: mordred fungi one way to get a quick count is to ask logsatsh for an indicator of a deployment. With the right string you'll be able to do it for all th deployment methods (though I have no idea if such a string exists) | 17:52 |
| mordred | clarkb: oh - that's a good idea | 17:52 |
| mordred | clarkb: I mean - you could start by asking logstash for invocations of "./stack.sh" | 17:53 |
| mordred | that would get all the devstacks | 17:53 |
| clarkb | mordred: ya though multinode jobs run stack.sh multiple times per job :/ | 17:53 |
| mordred | that's a good point | 17:53 |
| mordred | there's probably some initial keystone bootstrap that always happens but only happens once | 17:53 |
| mordred | kmalloc cmurphy: ^^ ? | 17:53 |
| fungi | then there's the problem of defining "openstack" for purposes of this analysis. not all jobs deploy the same combinations of services... does a functional test job which just starts one or a handful of services count as an openstack deployment? | 17:54 |
| openstackgerrit | Merged openstack/project-config master: Enable Review-Priority flag for Blazar https://review.opendev.org/682613 | 17:54 |
| openstackgerrit | Merged openstack/project-config master: Fix release ACL for Kayobe https://review.opendev.org/682581 | 17:54 |
| denaitre | sounds great! and I just found the link to logstash in the doc | 17:55 |
| mordred | fungi: I'd argue that, for the purposes of understanding how many times a week the CI system installs openstack, any combination of keystone + a catalog + a service would likely be good enough to count as the CI system doing such work | 17:55 |
| clarkb | denaitre: note we have a hole in our data for the ~3 days prior to about 24 hours ago | 17:55 |
| clarkb | denaitre: there was an elasticsearch outage | 17:55 |
| fungi | mordred: yeah, that would be a reasonable definition | 17:56 |
| clarkb | but data from about 24 hours ago through the present and forward should be accurate | 17:56 |
| mordred | fungi: I agree with you in the more specific sense | 17:56 |
| *** ociuhandu has quit IRC | 17:56 | |
| fungi | i mainly wanted to point out that methodology makes a big difference on what you'll ultimately be counting | 17:56 |
| mordred | fungi: and a more refined report might want to show a rollup report reporting total number of "clouds" deployed, then breakdowns of the set of services in each "cloud" | 17:56 |
| *** sshnaidm is now known as sshnaidm|bbl | 17:57 | |
| mordred | "10k clouds, 9k clouds with keystone, nova, neutron, cinder, glance, 4k clouds with keystone, nova, neutron., cinder, glance, octavia ..." | 17:57 |
| *** armax has quit IRC | 17:57 | |
| mordred | denaitre: good job of nerdsniping us to all ponder this :) | 17:58 |
| * kmalloc looks | 17:59 | |
| clarkb | AJaeger: corvus I think 681322 is ready http://paste.openstack.org/show/777443/ shows my test results (I'll leave that on the change too) | 17:59 |
| openstackgerrit | Merged zuul/zuul master: Add autogenerated tag to Gerrit reviews https://review.opendev.org/682473 | 17:59 |
| kmalloc | mordred: bootstrap should also be idempotent and relatively fast if the data exists | 18:00 |
| *** armax has joined #openstack-infra | 18:00 | |
| kmalloc | but bootstrap should also be light-weight... ideally devstack should know if keystone is bootstrapped up | 18:00 |
| clarkb | mordred: kmalloc: fwiw its incredibly slow with osc. That accounts for about 5-7 minutes of every devstack job | 18:00 |
| kmalloc | and just never run rather than ghe get-or-create method | 18:00 |
| mordred | kmalloc: in general would bootstrap be a decent thing to troll for in logstash logs to find instances of "I installed an openstack" ? | 18:00 |
| *** armax has quit IRC | 18:00 | |
| kmalloc | sure. | 18:01 |
| mordred | kmalloc: and it should be a thing that would be run by non-devstacks too, yeah? | 18:01 |
| kmalloc | clarkb: right, i think stack.sh is inefficient vs keystone's manage-bootstrap mechanism. | 18:01 |
| *** e0ne has joined #openstack-infra | 18:01 | |
| kmalloc | stack.sh does the get-or-create method over and over, which is slow(ish) | 18:01 |
| *** priteau has quit IRC | 18:01 | |
| kmalloc | we might need to write out some state stuff ... or something else to know things have been done and can just reload the info cleanly | 18:01 |
| clarkb | if keystone knows of a magical way to amke that better I'm sure we would all be happy for patches | 18:02 |
| kmalloc | it's def. not magical | 18:02 |
| mordred | clarkb: we had a discussion with johnsom yesterday about an issue octavia had with their devstack plugin due to an output format change for complex data from osc which they parse with awk and whether or not it would be a good idea to replace their whole plugin with python using sdk | 18:02 |
| kmalloc | it's a lot of re-writing | 18:02 |
| clarkb | but bnemec has been poking at using the osc as a service idea | 18:02 |
| mordred | clarkb: because they have a bunch of complex logic | 18:02 |
| AJaeger | clarkb: nice | 18:02 |
| clarkb | osc as a service can reuse tokens and remember some data so it helps | 18:03 |
| kmalloc | yeah, it might help a bunch | 18:04 |
| spotz | fungi can I get topic permissions on openstack-uc please? | 18:05 |
| denaitre | clarkb: ok thanks for the warning | 18:05 |
| mordred | clarkb: https://opendev.org/openstack/octavia/src/branch/master/devstack/plugin.sh#L391-L399 | 18:05 |
| mordred | enjoy line 399 | 18:05 |
| kmalloc | mordred: my eyes are bleeding | 18:05 |
| johnsom | It isn't that bad... lol | 18:06 |
| clarkb | mordred: fwiw my sdk rewrite wasn't small, but ya I think in many ways its mroe maintanable | 18:06 |
| * AJaeger needs a larger monitor or two | 18:06 | |
| fungi | spotz: done, i have you the same permissions in #openstack-uc as mrhillsman has | 18:06 |
| mordred | johnsom: :) | 18:07 |
| spotz | fungi: Thank you Sir just got pointed out the unupdated meeting time is there | 18:07 |
| bnemec | I'm a bit hung up on figuring out how to disable the osc service hack when testing osc changes. | 18:07 |
| kmalloc | johnsom: well i guess it matters what you compare it to | 18:07 |
| bnemec | I briefly looked at the dependency resolution code in one of the other roles and it was...daunting. | 18:07 |
| kmalloc | i've written worse in my career...usually in perl | 18:08 |
| mordred | clarkb: that's this yeah? https://review.opendev.org/#/c/673018/ <-- kmalloc, johnsom | 18:08 |
| johnsom | I pasted a readable version for the reviewers. That is a bit "minimized". | 18:08 |
| clarkb | mordred: johnsom kmalloc the child https://review.opendev.org/#/c/673108/6 is much more intresting | 18:08 |
| clarkb | bnemec: did you see my comment on the mailing list about using a proxy executable called openstack that uses osc as is or the service depending on what is being executed? | 18:09 |
| clarkb | bnemec: if you did that you could also have a flag to pass through to real osc when testing osc | 18:09 |
| clarkb | or maybe it was on the change. I know I posted that idea somewhere | 18:09 |
| bnemec | clarkb: I think I already did that. The problem is figuring out when to disable the service. | 18:10 |
| bnemec | If osc or any of its deps are anywhere in the depends on for the change we shouldn't use the service. | 18:10 |
| clarkb | bnemec: that info is in the zuul dict in the jobs iirc | 18:10 |
| bnemec | I'm sure it's doable, it's just not trivial and I haven't had the time to really dig into it. | 18:11 |
| clarkb | if 'openstack/openstackclient' in zuul.projects: type of condition | 18:11 |
| *** ociuhandu has joined #openstack-infra | 18:11 | |
| corvus | clarkb: 681322+3 | 18:12 |
| bnemec | It needs to be all of the deps too though. | 18:12 |
| clarkb | corvus: tyty | 18:12 |
| bnemec | Although maybe we could get around that by restarting the service to pick up test versions of the deps after they get installed. | 18:13 |
| clarkb | bnemec: well any python imports would just work because they'd find the new code via the python path | 18:13 |
| clarkb | the only thing we have to worry about is cli invocation right? | 18:13 |
| bnemec | I was hoping to avoid that rabbit hole for the initial version, but maybe I can't. | 18:13 |
| clarkb | I guess the worry is that if there is a break to the cli input/outpu (like what johnsom is running into) then we'd not catch any of those cases | 18:14 |
| bnemec | Right now osc is installed from pip so it gets pip deps too. | 18:14 |
| bnemec | Which ianw correctly noted means we might not actually test a change if it was to one of the things pulled in by the osc pip install. | 18:15 |
| clarkb | we are still running osc though right? | 18:15 |
| clarkb | we are only changing the interface to interact with osc | 18:15 |
| *** ociuhandu has quit IRC | 18:16 | |
| openstackgerrit | Merged openstack/project-config master: Add windmill-ops / ansible-role-elasticsearch https://review.opendev.org/680830 | 18:18 |
| *** xenos76 has joined #openstack-infra | 18:19 | |
| kmalloc | i think going to a single instance of OSC per standup vs the package_resource_load (it's expensive... for sure) a bazillion times is worth it | 18:19 |
| openstackgerrit | Merged opendev/base-jobs master: Add cleanup playbook to all base jobs https://review.opendev.org/681322 | 18:19 |
| kmalloc | it should absolutely speed things up | 18:19 |
| openstackgerrit | Merged zuul/zuul master: Use robot_comments in Gerrit https://review.opendev.org/682487 | 18:19 |
| kmalloc | the only other thing i would consider is writing out a state and sourcing it in/exporting env for the values set, so even the OSC bits aren't needed once it's been run once for keystone. | 18:20 |
| kmalloc | basically run once, save somewhere, if values are set trust the values. | 18:20 |
| bnemec | I guess my concern is that as it works right now, if someone pushes a patch to osc-lib we won't actually use that test patch in the service because it uses a pip installed version. | 18:21 |
| bnemec | Unless we restart the service after installing the osc-lib patch being tested, which is something we'll need to do for projects that use a plugin to osc anyway. | 18:22 |
| clarkb | bnemec: I see. Are you changing how osc is installed in the change then? | 18:23 |
| clarkb | or is that already a problem? | 18:23 |
| bnemec | No, I added a pip install of osc before running devstack so I could start the service in Ansible. | 18:23 |
| bnemec | But maybe I should move that into devstack itself and just have the Ansible set a flag. | 18:24 |
| clarkb | ya I would expect devstack to do that (it is already doing it for regular osc cli interaction) | 18:24 |
| clarkb | it could be the first "service" it installs | 18:24 |
| *** kjackal has quit IRC | 18:31 | |
| *** ociuhandu has joined #openstack-infra | 18:33 | |
| *** openstackgerrit has quit IRC | 18:37 | |
| *** ociuhandu has quit IRC | 18:38 | |
| *** goldyfruit___ has joined #openstack-infra | 18:38 | |
| *** goldyfruit_ has quit IRC | 18:41 | |
| clarkb | fwiw there were 863 completed devstack runs on a host called "controller" in the last hour or so | 18:41 |
| clarkb | I've asked for the count for the last 24 hours but I've given logstash a sad doing so | 18:42 |
| clarkb | but 863 in an hour makes me think we are still in at least the 10k/day range | 18:42 |
| clarkb | once you factor in daily work rhythm and such | 18:42 |
| *** mattw4 has quit IRC | 18:44 | |
| clarkb | oh wait was that for 6 hours ugh logstash slowness making it hard to know what you got resutls for and what you didn't get results for | 18:46 |
| * clarkb keeps fiddling with it | 18:46 | |
| clarkb | ok ya its 1550 ish per 12 hours so 863 must've been 6 hours | 18:48 |
| clarkb | 24 hours now returns and says 3566 | 18:48 |
| clarkb | just devstack jobs then is 3.5k/day | 18:49 |
| clarkb | message:"controller | DevStack Component Timing" AND filename:"job-output.txt" | 18:49 |
| clarkb | that is my query | 18:49 |
| clarkb | denaitre: ^ | 18:49 |
| clarkb | then you need to count triplo, osa, etc | 18:49 |
| *** jcoufal has quit IRC | 18:56 | |
| clarkb | message:"export TOCI_JOBTYPE=" AND filename:"job-output.txt" for tripleo gives us another 857 | 18:57 |
| *** lucasagomes has quit IRC | 18:58 | |
| clarkb | message:"Bootstrap the OSA AIO" AND filename:"job-output.txt" for OSA another 277 | 18:59 |
| clarkb | ballpark figure at least 4.7k jobs per day | 18:59 |
| clarkb | if the last 24 hours are representative | 18:59 |
| AJaeger | clarkb: do legacy jobs use controller as well? | 19:03 |
| clarkb | AJaeger: oh good question. I don't think they do. They use primary | iirc | 19:04 |
| * clarkb checks | 19:04 | |
| clarkb | hrm that string doesn't exist I'll need to find a legacy job log I can read | 19:05 |
| AJaeger | it should be primary | 19:06 |
| AJaeger | clarkb: http://zuul.opendev.org/t/openstack/build/a2e91aa3d5d44768b63f7ba067209ef1/log/job-output.txt#53 | 19:06 |
| clarkb | thanks | 19:07 |
| clarkb | message:"primary | Mapping projects to workspace" AND filename:"job-output.txt" should capture d-g and that shows another 1157 | 19:12 |
| clarkb | takes us to 5.8k range | 19:12 |
| mordred | clarkb: you got nerdsniped good | 19:14 |
| clarkb | mordred: yup. But its also useful show off data so figured was worth poking at a bit | 19:14 |
| mordred | agree | 19:14 |
| mordred | it's a good problem to dig in to | 19:15 |
| *** mriedem has quit IRC | 19:15 | |
| AJaeger | infra-root, I documented the "clean check" requirement in the infra-manual, enhancement suggestions welcome: https://review.opendev.org/682933 | 19:15 |
| *** mriedem has joined #openstack-infra | 19:16 | |
| AJaeger | infra-root, https://review.opendev.org/#/c/681353/1 documents our renames from Monday, please approve for bookkeeping | 19:17 |
| *** openstackgerrit has joined #openstack-infra | 19:18 | |
| openstackgerrit | Merged opendev/project-config master: Add record of 20190916 gerrit project renames https://review.opendev.org/681353 | 19:18 |
| AJaeger | thanks, mordred ^ | 19:18 |
| mordred | AJaeger: I think the manual patch looks good - I left a +2- feel free to land it - or keep it open if you want more comments | 19:19 |
| clarkb | infra-root https://review.opendev.org/#/c/682732/ is also related to our recent renames if you want to take a look | 19:19 |
| clarkb | mordred: corvus ^ in particular since you were involved with the change this restores back to | 19:19 |
| AJaeger | mordred: I'll +A tomorrow in case others want to improve. thanks | 19:20 |
| clarkb | Not sure I called it out in the meeting yseterday but my journey to ansiblefest actually starts on friday so I'll be out starting then | 19:21 |
| clarkb | I'm making a pitstop to attend a wedding on my way | 19:21 |
| mordred | clarkb: are you going to a random wedding? | 19:22 |
| clarkb | not a random one no | 19:22 |
| mordred | ok. I mean - I kind of like the idea that you have a hobby of stopping by weddings, so you figured you catch on on the way to ansiblefest ... | 19:22 |
| *** eernst has joined #openstack-infra | 19:23 | |
| *** ociuhandu has joined #openstack-infra | 19:24 | |
| corvus | yeah, i was waiting for an xkcd "my hobby..." | 19:29 |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Remove obsolete docs publish jobs https://review.opendev.org/682972 | 19:34 |
| *** prometheanfire has quit IRC | 19:36 | |
| corvus | clarkb: comment on the setup-org change | 19:37 |
| *** prometheanfire has joined #openstack-infra | 19:37 | |
| clarkb | corvus: good idea I'll do that | 19:38 |
| *** armax has joined #openstack-infra | 19:38 | |
| *** eharney has quit IRC | 19:39 | |
| *** ociuhandu has quit IRC | 19:40 | |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Restore setup-org.yaml https://review.opendev.org/682732 | 19:41 |
| clarkb | corvus: mordred ^ that better? | 19:41 |
| openstackgerrit | David Shrewsbury proposed zuul/nodepool master: Reduce upload threads in tests from 4 to 1 https://review.opendev.org/682977 | 19:41 |
| * clarkb finds lunch | 19:41 | |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Remove obsolete docs publish jobs https://review.opendev.org/682972 | 19:42 |
| *** tosky has joined #openstack-infra | 19:43 | |
| *** jbadiapa has quit IRC | 19:44 | |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Remove obsolete docs publish jobs https://review.opendev.org/682972 | 19:46 |
| *** armax has quit IRC | 19:47 | |
| kmalloc | oh i ... i think that should be a thing: Random wedding stopping on the way to ansiblefest | 19:49 |
| *** mattw4 has joined #openstack-infra | 19:50 | |
| mordred | kmalloc: I think I hear you suggesting that everyone who is going to ansiblefest should stop by a random wedding | 19:51 |
| mordred | that wouldn't be a weird tradition to start at all | 19:52 |
| *** sshnaidm|bbl is now known as sshnaidm | 19:52 | |
| *** Goneri has quit IRC | 19:52 | |
| kmalloc | that is the correct interpretation of what I'm saying | 19:53 |
| kmalloc | for sure! | 19:53 |
| *** ralonsoh has quit IRC | 19:56 | |
| *** jbadiapa has joined #openstack-infra | 19:57 | |
| * dtroyer isn't going to ansiblefest so will settle for an engagement party | 19:58 | |
| dtroyer | While I'm here, I need a bit of assistance with an Intel 3rd party CI account in Gerrit… | 19:58 |
| dtroyer | I'm looking for the email address for intel_zuul in Gerrit | 19:59 |
| fungi | the gerrit rest api will let you query that, anonymously even | 20:02 |
| fungi | i'd look it up but i'm about to disappear to grab dinner | 20:02 |
| dtroyer | aha! teaching me to fish… no worries, thanks fungi | 20:02 |
| fungi | mmm, now i want fish | 20:02 |
| *** larainema has quit IRC | 20:06 | |
| *** sthussey has joined #openstack-infra | 20:07 | |
| * jrosser wonders again about the debian buster updates repo..... | 20:08 | |
| *** pcaruana has quit IRC | 20:10 | |
| *** jamesmcarthur_ has quit IRC | 20:12 | |
| *** jamesmcarthur has joined #openstack-infra | 20:17 | |
| *** jamesmcarthur has quit IRC | 20:17 | |
| *** jamesmcarthur has joined #openstack-infra | 20:18 | |
| rpioso | Please remind me. Does a clean rebase of a change remove its existing approvals to merge? | 20:26 |
| clarkb | rpioso: yes | 20:29 |
| rpioso | clarkb: Thank you! | 20:29 |
| clarkb | jrosser: I'm not sure where we ended up with that. The job was definitely using the updates repo | 20:29 |
| clarkb | fungi was looking at reprepro further | 20:30 |
| clarkb | jrosser: I don't know much about apt's repo format or why packages may be superceded or not so will have to defer to others. The entire config for reprepro is in config management (and I can dig up links to that if people want) and the mirrors themselves are publicly accessible (they shouldn't be used or relied on outside of CI but you can poke around them to see how they are set up | 20:31 |
| jrosser | yes thats right - i couldnt see anything particularly wrong with the job setup | 20:32 |
| *** xenos76 has quit IRC | 20:32 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: RFC: Generic cache implementation https://review.opendev.org/682992 | 20:38 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: RFC: Generic cache implementation https://review.opendev.org/682992 | 20:39 |
| clarkb | jrosser: comparing http://ftp.us.debian.org/debian/pool/main/e/exim4/ to http://mirror.dfw.rax.openstack.org/debian/pool/main/e/exim4/ it seems we've got the right packages (latest for each release) | 20:40 |
| clarkb | I think | 20:40 |
| * clarkb downloads the contents file | 20:40 | |
| clarkb | er I want the packages file | 20:41 |
| *** eharney has joined #openstack-infra | 20:43 | |
| *** goldyfruit___ has quit IRC | 20:44 | |
| clarkb | neither the buster-updates nor the buster-backports packages files seems to point at the up to date exim4-base package and the package that buster proper points to isn't present | 20:44 |
| clarkb | is the problem we need to be mirror a buster security repo too maybe? | 20:45 |
| clarkb | exim4 had updates due to security issues /me pokes around in upstream mirror | 20:45 |
| *** Goneri has joined #openstack-infra | 20:45 | |
| clarkb | the upstream mirror points to the package files we do have in its packages file | 20:47 |
| clarkb | so ya reprepro must be producing an invalid packages file | 20:47 |
| clarkb | anyone know about reprepro and where to start looking for why that may happen? | 20:47 |
| corvus | infra-root: i'd like to restart zuul and, well, frankly see if it can still talk to gerrit after all the changes to the gerrit driver which just landed | 20:52 |
| corvus | i think we should do a full restart, just because i think the executors and mergers might switch from using ssh to http | 20:53 |
| corvus | for git ops | 20:53 |
| corvus | so basically, the plan would be a full restart, watch some scheduler/executor logs, and if things don't work, reinstall from an older commit on all systems and restart again | 20:54 |
| corvus | i'll identify the revert commit now | 20:54 |
| *** ociuhandu has joined #openstack-infra | 20:56 | |
| corvus | actually, the revert commit should be the 3.10.2 tag. so if things go poorly, i'll just reinstall that tag. | 20:56 |
| *** goldyfruit___ has joined #openstack-infra | 20:56 | |
| corvus | oh, we might be able to do this in multiple phases -- we can probably restart a small number of mergers first and make sure they work okay, then do all the mergers... if those work out okay, then proceed to do the whole system. | 20:57 |
| clarkb | corvus: can probably do the executors in a similar way? | 20:58 |
| corvus | clarkb: yeah, but i think the mergers would be sufficient test coverage for that | 20:58 |
| corvus | so i think we can skip it and not lose anything | 20:58 |
| clarkb | fwiw our reprepro options have export=changed set. reading the man page that seems to mean export the package indices for dists that have changed. Which I think means reprepro should've updated those package files | 20:58 |
| clarkb | the reprepro log shows when the package we point at was replaced, but I don't see naythign in the months of logs there about it updating packages.gz | 21:00 |
| *** signed8bit has quit IRC | 21:01 | |
| *** signed8bit has joined #openstack-infra | 21:02 | |
| corvus | i've stopped all mergers and restarted zm01 | 21:07 |
| corvus | it hasn't picked up a job yet :/ | 21:07 |
| corvus | apparently most of what our mergers do is fetch ansible refs | 21:08 |
| *** eernst has quit IRC | 21:09 | |
| jrosser | clarkb: thankyou for looking at it, i need to !computer now, will read back tomorrow if anything turns up | 21:10 |
| clarkb | jrosser: I'm quickly out of my depth here. Reprepro's behavor seems to need an understanding of apt's essoteric options and behaviors | 21:11 |
| openstackgerrit | Merged opendev/system-config master: Restore setup-org.yaml https://review.opendev.org/682732 | 21:11 |
| clarkb | we could try changing export=always | 21:11 |
| clarkb | maybe? | 21:12 |
| clarkb | corvus: is it working then? | 21:16 |
| corvus | zm01 just handled an openstack job and didn't log an error so... yes i think? | 21:18 |
| corvus | i think i want to see it handle one more job | 21:19 |
| clarkb | I guess the executors could've been grabbing the jobs too? | 21:19 |
| clarkb | (that would explain why zuul seems to be running jobs just fine) | 21:19 |
| corvus | yeah, and merger jobs also tend to come in waves | 21:19 |
| Shrews | corvus: do you need to restart the scheduler? | 21:21 |
| corvus | Shrews: yes, that has the biggest changes | 21:21 |
| *** Goneri has quit IRC | 21:22 | |
| *** goldyfruit_ has joined #openstack-infra | 21:22 | |
| Shrews | corvus: then maybe wait until the autohold stuff all lands? guess it isn't necessary, but since it's in flight... | 21:23 |
| corvus | Shrews: well, it'll take a while to get installed; i'd likely eod before i'm able to do that, so i was thinking we'd restart for that later in the week | 21:23 |
| Shrews | corvus: yah, that's fine then | 21:23 |
| *** goldyfruit___ has quit IRC | 21:25 | |
| *** Goneri has joined #openstack-infra | 21:26 | |
| corvus | okay i give up waiting; i'm assuming that one job without errors is good enough. i'll restart the whole shebang now. | 21:27 |
| EmilienM | clarkb: I have a short update in our outage | 21:27 |
| EmilienM | clarkb: we have identified the root cause and we have a few patches what we would need to land, on different repos | 21:28 |
| clarkb | EmilienM: ya I saw your email. Won't that bring back the authentication failures though? | 21:28 |
| clarkb | (also I thought cloudnull's change had been in place for a few weeks now but the problems you were seeing are much more recent/) | 21:28 |
| EmilienM | clarkb: no we don't want to revert that one yet | 21:28 |
| EmilienM | clarkb: we want to disable something in our testing, which isn't related to the docker.io thing | 21:28 |
| clarkb | ah that is the third chagne you link | 21:28 |
| clarkb | https://review.opendev.org/#/c/682905/ that one | 21:29 |
| EmilienM | clarkb: could we exceptionally land https://review.opendev.org/#/c/683001/ now ? | 21:29 |
| clarkb | EmilienM: we can certainly enqueue and promote it to the gate | 21:29 |
| EmilienM | clarkb: I'm asking for a force merge | 21:29 |
| clarkb | EmilienM: why? | 21:29 |
| clarkb | can you explain why that is necessary in this case? | 21:29 |
| cjloader | gates down? | 21:30 |
| EmilienM | it'll probably fail in gate | 21:30 |
| EmilienM | as we need other patches in tripleoclient | 21:30 |
| clarkb | usually force merges are reserved for unwedging dependencies between changes | 21:30 |
| EmilienM | 2 actually | 21:30 |
| EmilienM | we need https://review.opendev.org/#/c/682905/ and https://review.opendev.org/#/c/682943/ to land | 21:30 |
| clarkb | EmilienM: can we enqueue them both then? | 21:30 |
| EmilienM | we need them 3 together.. | 21:31 |
| EmilienM | they passed check | 21:31 |
| clarkb | EmilienM: https://review.opendev.org/#/c/682943/ well that one sets the new behavior to what we want by default right? | 21:31 |
| EmilienM | we would save hours... | 21:31 |
| clarkb | so if that one goes in first the others can follow? | 21:31 |
| cjloader | https://zuul.opendev.org/t/openstack/status down.... | 21:32 |
| corvus | cjloader: it's restarting | 21:32 |
| cjloader | ugh | 21:32 |
| corvus | cjloader: sorry? | 21:32 |
| EmilienM | clarkb: right but for that to be effective we need https://review.opendev.org/#/c/683001/ | 21:32 |
| clarkb | EmilienM: I'm just trying to understand what the interdependencies are here. My read of the commit messages is that 682943 does what we want by default and it passes check | 21:33 |
| cjloader | it was almost done with a job | 21:33 |
| clarkb | EmilienM: we need that even if disabled is the default? | 21:33 |
| corvus | cjloader: hang around, let's discuss this when i'm done with the maintenance | 21:33 |
| EmilienM | clarkb: right | 21:34 |
| clarkb | EmilienM: also we are now up to 4 chagnes we need and 2 of them are not approved. Why would force merging a single change help? | 21:34 |
| EmilienM | weshay: ^ can you help me here please | 21:34 |
| weshay | I'm here | 21:34 |
| EmilienM | I really want to leave my laptop for today | 21:35 |
| weshay | ok.. | 21:35 |
| EmilienM | we need these patches merged, that's all I want | 21:35 |
| weshay | EmilienM, you want to force merge those? | 21:35 |
| EmilienM | yes this is what I asked | 21:35 |
| weshay | ok | 21:35 |
| clarkb | well you asked to force merge one of them | 21:35 |
| weshay | What EmilienM said | 21:35 |
| EmilienM | it happens every 6 months | 21:35 |
| clarkb | but apparently all 4 are necessary | 21:35 |
| EmilienM | I need https://review.opendev.org/#/c/683001/ | 21:35 |
| clarkb | and only 2 are approved | 21:35 |
| EmilienM | I just approved | 21:35 |
| EmilienM | 682943 682905 and 683001 | 21:36 |
| clarkb | so now I'm confused over what is actually necessary. It would be helpful if someone can explain that and why we can't sequence them and promote to the gate | 21:36 |
| weshay | ok.. | 21:37 |
| corvus | i'm seeing a problem with the re-enqueue process; we may need to revert | 21:37 |
| weshay | this one can go in w/o any deps https://review.opendev.org/#/c/683001 | 21:37 |
| weshay | totally safe | 21:37 |
| clarkb | weshay: and if we enqueue that one at the top of the gate we don't expect it will pass on its own without the others? | 21:37 |
| clarkb | probably because the others are what take action on that config change? | 21:38 |
| weshay | this one is in the gate.. https://review.opendev.org/#/c/682905/ now .. and is safe | 21:38 |
| EmilienM | in theory yes they could be enqueued | 21:38 |
| EmilienM | but will likely fail since 683001 isn't landed | 21:38 |
| clarkb | EmilienM: is the commit message wrong about the default being disabled then? or maybe these are two different flags that we are disabling? | 21:38 |
| clarkb | (reading the commit messages those two changes seem to be equivalent just in different locations) | 21:39 |
| clarkb | they both disable validations | 21:39 |
| mriedem | should i expect to be seeing this from zuul after the restart? | 21:39 |
| mriedem | "Unable to freeze job graph: 'dict_keys' object does not support indexing" | 21:39 |
| clarkb | mriedem: corvus just mentioned a problem with the requeue and that we may need to revert. I'm going to guess that is related | 21:40 |
| EmilienM | clarkb: one is to disable all validations (683001) and the other ones are to bind the new inflight validations which we now try to disable as well to the parameter set in 683001 | 21:40 |
| corvus | mriedem: ack, thx | 21:40 |
| *** panda has quit IRC | 21:41 | |
| clarkb | EmilienM: ok so they are different sets of validations? | 21:41 |
| *** mriedem is now known as mriedem_afk | 21:41 | |
| *** e0ne has quit IRC | 21:41 | |
| EmilienM | clarkb: yes, that we try to configure at once | 21:42 |
| EmilienM | and we need to disable them | 21:42 |
| clarkb | EmilienM: and we think both of them cause the gate problems? | 21:42 |
| EmilienM | they need to be reworked, they caused timeouts | 21:42 |
| clarkb | (eg disabling one or the other won't cause jobs to start passing) | 21:42 |
| *** panda has joined #openstack-infra | 21:42 | |
| corvus | restarting scheduler on 3.10.2 | 21:42 |
| corvus | i'll leave the rest of the system on HEAD | 21:43 |
| weshay | clarkb, timeouts.. lots of things can contribute to timeouts | 21:43 |
| clarkb | weshay: right but is the impact of one of these sufficient to address the timeouts? If so it can be enqueued to the head of the gate and ocne merged should allow everything behind it to pass | 21:44 |
| weshay | http://dashboard-ci.tripleo.org/d/si1tipHZk/jobs-exploration?orgId=1&from=now-90d&to=now&fullscreen&panelId=16 | 21:44 |
| weshay | no.. as EmilienM was saying.. we're trying to address it with a few patches.. that never should have been merged | 21:44 |
| weshay | sorry for the trouble | 21:45 |
| weshay | you guys are probably busy atm | 21:45 |
| EmilienM | clarkb: could you please go ahead and merge https://review.opendev.org/#/c/683001/ https://review.opendev.org/#/c/682905/ and https://review.opendev.org/#/c/682943/ | 21:45 |
| *** nhicher has quit IRC | 21:46 | |
| *** nhicher has joined #openstack-infra | 21:47 | |
| clarkb | EmilienM: weshay is this accurate for a message in gerrit for this "Force merging https://review.opendev.org/#/c/683001/ https://review.opendev.org/#/c/682905/ https://review.opendev.org/#/c/682943/ together to disable validations that cause jobs to timeout. Once disabled these can be added back in little by little as their functionality is verified" ? | 21:47 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: Store a list of held node per held build in hold request https://review.opendev.org/682466 | 21:47 |
| weshay | +1 from .. well said | 21:48 |
| EmilienM | yes, +1 | 21:48 |
| weshay | me that is | 21:48 |
| corvus | re-enqueueing | 21:48 |
| clarkb | weshay: EmilienM ok I'll do that as soon as zuul restart is sorted | 21:49 |
| cjloader | corvus: can we discuss? | 21:49 |
| *** michael-beaver has quit IRC | 21:49 | |
| weshay | aye.. bad timing | 21:50 |
| weshay | sorry | 21:50 |
| EmilienM | I need to step away for family stuff, I'll check later. Hopefully we can land that in the next hour then i'll look at the 401 errors | 21:51 |
| corvus | clarkb: re-enque is still in progress but otherwise i think we're stable; the first of EmilienM's changes is set to be re-enqueued in check, the second in gate, the third not. | 21:51 |
| clarkb | corvus: ok I think I udnersatnd the problem well enough now and if indeed these tests all contributing to timeouts then merging them to avoid them thrashing each other makes sense | 21:52 |
| EmilienM | corvus: I haven't asked a re-enqueue | 21:52 |
| clarkb | corvus: I was planning to click the submit button on those three | 21:52 |
| corvus | EmilienM: i know, you're getting one anyway since that's the restart process; nothing i can do :) | 21:52 |
| corvus | EmilienM: clarkb can dequeue afterwords if he wants | 21:52 |
| clarkb | I'll do that now that you've said the startup is happy | 21:52 |
| clarkb | corvus: ++ | 21:53 |
| corvus | cjloader: yes. our zuul is a very busy system, and it is nearly always on the verge of merging a change. when we perform a restart, we take several things into consideration, including backlog and release activity. but we can't wait until it's idle, because it never is. instead, we save the queues and restore them so that, aside from some additional delay, it shouldn't be a big impact. this | 21:55 |
| corvus | is further mitigated by zuul's dependency system which means you can continue to develop and even approve changes which depend on un-merged changes that are still in the queue. | 21:55 |
| EmilienM | corvus, clarkb : thanks for the help today | 21:55 |
| clarkb | EmilienM: weshay the merging is done I'll work on dequeuing now | 21:55 |
| EmilienM | ++ thanks again | 21:55 |
| weshay | rock on! | 21:55 |
| corvus | clarkb: gate re-enqueue is done; check in progress | 21:55 |
| EmilienM | we can now focus on 401 and bring back our jobs | 21:55 |
| cjloader | corvus: my initial +1 was seconds away from occurring on https://review.opendev.org/#/c/682446/ which occurred on a bad gateway | 21:57 |
| clarkb | and now dequeues of those three changes are complete | 21:58 |
| cjloader | I had already gotten approval from a core | 21:58 |
| cjloader | corvus: I initiated the recheck, and was in the middle of "uploading swift logs" | 21:59 |
| cjloader | on the last one | 21:59 |
| cjloader | but okay... | 22:00 |
| *** eharney has quit IRC | 22:04 | |
| *** rlandy is now known as rlandy|bbl | 22:04 | |
| *** jamesmcarthur has quit IRC | 22:06 | |
| corvus | cjloader: i agree that's bad timing. but due to our scale, it is unavoidable that it will happen to some change in the system -- on average, at any given time, there are about 7 changes that zuul is 5 minutes or less away from completing. but it needn't slow down your development process. if your work depends on that change, you can use git dependencies or the depends-on header to continue | 22:06 |
| corvus | building on it. if you would like to avoid that sort of thing entirely in the future, there is a work-in-progress spec for implementing a highly-available scheduler for zuul. if you'd like to contribute to it, that would be welcome, i'd be happy to point you at it. | 22:06 |
| *** jamesmcarthur has joined #openstack-infra | 22:09 | |
| cjloader | corvus: ok ty | 22:09 |
| cjloader | corvus: sorry i'm a bit frustrated because of all the blockers recently I fully understand your point and i'll leave it as that. | 22:11 |
| *** goldyfruit_ has quit IRC | 22:15 | |
| cjloader | corvus: that actually helped me out btw | 22:15 |
| corvus | cjloader: no worries. we all have changes that have been sitting out there a while :) | 22:15 |
| cjloader | all good now corvus -- https://review.opendev.org/#/c/680284/ | 22:20 |
| cjloader | used the Depends-On | 22:20 |
| corvus | cjloader: great! | 22:21 |
| *** markvoelker has quit IRC | 22:24 | |
| *** JorgeFranco has quit IRC | 22:24 | |
| *** jamesmcarthur has quit IRC | 22:33 | |
| *** ociuhandu has quit IRC | 22:40 | |
| *** goldyfruit_ has joined #openstack-infra | 22:40 | |
| clarkb | I've learned things from filing that gitea bug about timing stuff out | 22:41 |
| clarkb | there does already exist git command timeouts, but not all git commands use them | 22:41 |
| *** rfolco has quit IRC | 22:43 | |
| clarkb | oh hrm if you don't specify a timeout it should make it to the default timeout whcih is 60 seconds | 22:44 |
| clarkb | so how do we end up servicing requests for hours? | 22:44 |
| clarkb | possible the timeouts don't actualyl work or something else is eating the time | 22:46 |
| *** jamesmcarthur has joined #openstack-infra | 22:50 | |
| clarkb | reading the timeout code I think they should work | 22:51 |
| clarkb | they rely on stdlib context.WithTimeout contexts and exec.CommandContext which should kill -9 a process once the context is marked done which happens when the timeout is reached | 22:51 |
| clarkb | other things I've learned, we should have a 16 hour cache ttl on the objects we do cache | 22:53 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: WIP: Fix gerrit errors from production https://review.opendev.org/683006 | 22:53 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: DNM: Use http for all gerrit tests https://review.opendev.org/683007 | 22:53 |
| *** mriedem_afk has quit IRC | 22:53 | |
| corvus | clarkb: you mean that's the default cache ttl? | 22:55 |
| clarkb | corvus: yes, but I think we've found we don't cache much | 22:55 |
| clarkb | there are a number of defunct git processes on gitea01. i wonder if these are proceses that hit those git timeouts | 22:58 |
| clarkb | I suppose they could be otherwise leaked because we don't have an init system in our docker container | 22:58 |
| clarkb | ya their parent is gitea web | 22:58 |
| clarkb | I bet it needs to reap them | 22:58 |
| *** dchen has joined #openstack-infra | 22:59 | |
| *** tkajinam has joined #openstack-infra | 23:02 | |
| clarkb | https://github.com/go-gitea/gitea/issues/491 has had interesting conversation in recent days too | 23:02 |
| clarkb | there is a suggestion that adding commit graphs to repos will help then testing that shows it doesn't help at all | 23:04 |
| *** jamesmcarthur has quit IRC | 23:09 | |
| *** whoami-rajat has quit IRC | 23:10 | |
| *** rcernin has joined #openstack-infra | 23:16 | |
| *** mriedem has joined #openstack-infra | 23:17 | |
| clarkb | taking the idea of a config flag to not look for modified times, I think that is possible with a config flag that does not run the function to get that commit data and selects a different template | 23:19 |
| *** goldyfruit_ has quit IRC | 23:24 | |
| *** goldyfruit_ has joined #openstack-infra | 23:29 | |
| *** jamesmcarthur has joined #openstack-infra | 23:34 | |
| *** sthussey has quit IRC | 23:35 | |
| *** tosky has quit IRC | 23:40 | |
| clarkb | ya, pretty sure we would need to update the templates otherwise we'll get errors doing this. But the code sort of already supports it by returning nil values on error. We could return nil values when diasbled as well then handle them in the templates | 23:41 |
| clarkb | its too late in the day for me to learn golang templating though | 23:41 |
| clarkb | I actually think the algorithm they use now goes through the entire repo history? | 23:44 |
| *** efried_pto is now known as mriedem1 | 23:47 | |
| *** mriedem1 is now known as efried_pto | 23:47 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!