Friday, 2018-01-12

*** spzala has joined #openstack-infra00:02
*** andreas_s has joined #openstack-infra00:03
*** markvoelker has joined #openstack-infra00:04
*** xarses has quit IRC00:04
*** vsaienk0 has joined #openstack-infra00:06
*** andreas_s has quit IRC00:09
*** gouthamr has quit IRC00:10
openstackgerritMerged openstack-infra/puppet-openstackci master: Lay groundwork for zuulv2/v3 coexisting  https://review.openstack.org/52395100:11
openstackgerritMerged openstack-infra/storyboard-webclient master: Fix reference to angular-bootstrap in package.json  https://review.openstack.org/53281900:11
*** spzala has quit IRC00:12
*** sree has joined #openstack-infra00:14
*** sdague has quit IRC00:16
*** sree has quit IRC00:19
*** nicolasbock has joined #openstack-infra00:19
*** vsaienk0 has quit IRC00:23
*** claudiub has quit IRC00:27
*** jascott1 has quit IRC00:28
*** jascott1 has joined #openstack-infra00:29
*** jascott1 has quit IRC00:33
*** edmondsw has joined #openstack-infra00:37
*** RickWong has joined #openstack-infra00:38
*** edmondsw has quit IRC00:42
*** RickWong has left #openstack-infra00:43
*** sree has joined #openstack-infra00:45
*** sree has quit IRC00:50
*** tosky has quit IRC00:51
openstackgerritZara proposed openstack-infra/storyboard-webclient master: Update angular-ui-router to @uirouter/angularjs  https://review.openstack.org/53302200:52
*** bobh has joined #openstack-infra00:52
*** armaan has joined #openstack-infra00:53
*** xarses has joined #openstack-infra00:53
clarkbZara: hey if you are around any idea how much work https://review.openstack.org/198884 needs to be current? and if https://review.openstack.org/159406 is still an issue?00:53
*** inc0 has quit IRC00:53
*** markvoelker has quit IRC00:53
*** xarses has quit IRC00:53
*** markvoelker has joined #openstack-infra00:54
*** threestrands has quit IRC00:58
*** markvoelker has quit IRC00:58
Zarahey! :) I don't know much about those but at a glance I think we still don't have the things referred to in the first one, so the points about what we would need are likely still true. the details probably would need updating, (at a glance I see the commit message talks about 'work for bower compatibility' as something to happen in the future, and bower is deprecated.)00:59
ZaraI think that's the first time I've seen that fkeys spec00:59
* fungi wonders why Zara is awake at such an hour00:59
Zaraah I'm experimenting with montreal hours, at first I was trying to work a montreal day01:00
Zarabut it didn't work well because I didn't see other humans at all (since I was busy all evening)01:00
fungigot it. sounds fun01:00
*** Apoorva has quit IRC01:00
Zaraso now I'm trying 'work for a bit, go see people, come back to work', and while I'm waiting for builds it's a good time to update some webclient components01:00
Zarait does mean I'm half asleep though :D01:01
*** stakeda has joined #openstack-infra01:01
clarkbI did similar when living in seattle01:02
clarkbwas useful for working with apac01:02
clarkbkids ruined any idea that schedules are flexible01:02
*** threestrands has joined #openstack-infra01:02
clarkbZara: thanks, I'll have to see if anyone wants to modernize the js tooling spec01:02
*** caphrim007 has quit IRC01:03
Zarayeah, somehow I keep ending up on work that overlaps with our seattle office, so if I started working uk time I ended up working late to overlap with mtl and then staying on later to overlap with seattle..01:03
*** threestrands has quit IRC01:03
fungiworth noting, mordred's got some changes up for modernizing sb-wc's js stack anyway01:03
*** threestrands has joined #openstack-infra01:04
Zarayep, https://review.openstack.org/#/c/528436/ went in01:04
*** markvoelker has joined #openstack-infra01:04
*** caphrim007 has joined #openstack-infra01:04
*** caphrim007 has quit IRC01:09
*** markvoelker has quit IRC01:09
*** threestrands has quit IRC01:11
*** markvoelker_ has joined #openstack-infra01:13
Zaraah, though I missed that he updated the same package as part of a different commit (because it was my old one, hah)01:13
*** rickwong has joined #openstack-infra01:13
* Zara checks that one instead01:17
*** hamzy has joined #openstack-infra01:17
*** bobh has quit IRC01:17
*** afred312 has quit IRC01:22
*** bobh has joined #openstack-infra01:23
*** afred312 has joined #openstack-infra01:24
*** numans has quit IRC01:24
openstackgerritTristan Cacqueray proposed openstack-infra/nodepool feature/zuulv3: Do pep8 housekeeping according to zuul rules  https://review.openstack.org/52294501:24
*** threestrands has joined #openstack-infra01:24
openstackgerritTristan Cacqueray proposed openstack-infra/nodepool feature/zuulv3: Use same flake8 config as in zuul  https://review.openstack.org/50971501:24
*** numans has joined #openstack-infra01:25
*** threestrands has quit IRC01:25
*** threestrands has joined #openstack-infra01:26
*** afred312 has quit IRC01:26
*** gyee has quit IRC01:26
openstackgerritZara proposed openstack-infra/storyboard-webclient master: Update UI-Router  https://review.openstack.org/53302401:28
Zarafailed rebase, ignore, I need sleep :)01:29
*** cuongnv has joined #openstack-infra01:29
*** numans has quit IRC01:29
*** jascott1 has joined #openstack-infra01:30
openstackgerritGhanshyam Mann proposed openstack-infra/project-config master: Add tempest-stress project repo  https://review.openstack.org/53302501:33
*** bobh has quit IRC01:35
*** numans has joined #openstack-infra01:35
*** hongbin has joined #openstack-infra01:41
*** dhill__ has quit IRC01:41
openstackgerritGhanshyam Mann proposed openstack-infra/project-config master: Add tempest-stress project repo  https://review.openstack.org/53302501:42
*** rickwong_ has joined #openstack-infra01:42
*** rickwong has left #openstack-infra01:42
*** zhurong has joined #openstack-infra01:45
*** dave-mccowan has joined #openstack-infra01:46
*** sree has joined #openstack-infra01:48
*** liujiong has joined #openstack-infra01:53
*** sree has quit IRC01:53
*** threestrands has quit IRC01:54
*** afred312 has joined #openstack-infra01:56
*** threestrands has joined #openstack-infra01:56
*** felipemonteiro_ has joined #openstack-infra01:59
*** kukacz has quit IRC02:00
*** Apoorva has joined #openstack-infra02:00
*** afred312 has quit IRC02:01
*** kukacz_ has joined #openstack-infra02:01
*** afred312 has joined #openstack-infra02:03
*** ramishra has joined #openstack-infra02:08
*** yamahata has quit IRC02:09
*** tinwood has quit IRC02:10
*** tinwood has joined #openstack-infra02:11
*** iyamahat has quit IRC02:12
*** sree has joined #openstack-infra02:12
*** kambiz_ has quit IRC02:13
mriedemthis change adds a new job to nova https://review.openstack.org/#/c/532689/ to run in the experimental queue, i've run it through experimental several times but zuul never posts results back on the change02:16
*** kambiz has joined #openstack-infra02:16
mriedemis there possibly something not setup correctly in there? it's basically copy/paste/modify of an existing test02:16
*** sree has quit IRC02:17
dmsimardZara: ohai I'm in Montreal o/02:18
*** caphrim007 has joined #openstack-infra02:20
*** caphrim007 has quit IRC02:25
*** ramishra has quit IRC02:27
*** rickwong_ has quit IRC02:28
*** jianghuaw_ has quit IRC02:29
*** jianghuaw has joined #openstack-infra02:29
*** markvoelker_ has quit IRC02:31
*** jlk has quit IRC02:31
*** dhill__ has joined #openstack-infra02:33
*** annp has joined #openstack-infra02:39
ianwmriedem: 2018-01-12 00:09:03,808 DEBUG zuul.GerritReporter: Report change <Change 0x7f4523a52828 532689,1>, params {}, message: This change depends on a change that failed to merge.02:39
mriedemblarg02:40
mriedemok02:40
mriedemianw: thanks02:40
ianwyeah looks liek the parent failed then it dies02:41
mriedemthe dep chain on this is massive so i probably just pushed something up today and forgot about it02:41
*** mriedem has quit IRC02:42
*** ijw has quit IRC02:43
*** Apoorva has quit IRC02:46
*** rosmaita has quit IRC02:47
*** iyamahat has joined #openstack-infra02:54
*** sree has joined #openstack-infra02:59
*** andreas_s has joined #openstack-infra03:02
*** yamahata has joined #openstack-infra03:02
*** sree has quit IRC03:03
*** numans has quit IRC03:05
*** numans has joined #openstack-infra03:07
*** ricolin has joined #openstack-infra03:07
*** andreas_s has quit IRC03:08
prometheanfiredoes openstack use mailman3 for the lists?03:09
prometheanfirenope, 2.1.16, nvm :P03:10
prometheanfire(we are looking at it)03:10
*** edmondsw has joined #openstack-infra03:14
* Zara waves back at dmsimard03:15
ZaraI  am not there but am working to a mtl timetable, kinda03:15
Zarahope there's not too much snow! (I hear there's always snow)03:16
*** edmondsw has quit IRC03:18
*** felipemonteiro_ has quit IRC03:20
*** dhill__ has quit IRC03:24
*** felipemonteiro_ has joined #openstack-infra03:25
*** amotoki has joined #openstack-infra03:30
*** sree has joined #openstack-infra03:31
*** slaweq has joined #openstack-infra03:32
*** dave-mccowan has quit IRC03:33
*** sree has quit IRC03:36
*** slaweq has quit IRC03:37
openstackgerritsebastian marcet proposed openstack-infra/openstackid-resources master: Added new endpoint get attendees by summit  https://review.openstack.org/53305603:38
openstackgerritMerged openstack-infra/openstackid-resources master: Added new endpoint get attendees by summit  https://review.openstack.org/53305603:40
openstackgerritIan Wienand proposed openstack-infra/project-config master: Revert "Pause builds for dib 2.10 release"  https://review.openstack.org/53270103:44
openstackgerritIan Wienand proposed openstack-infra/project-config master: Add fedora 27 build and nodes  https://review.openstack.org/53305903:44
*** larsks has quit IRC03:56
*** larsks has joined #openstack-infra03:56
*** sree has joined #openstack-infra03:57
*** felipemonteiro_ has quit IRC04:01
*** zhurong has quit IRC04:03
*** yamamoto has joined #openstack-infra04:03
openstackgerritCao Xuan Hoang proposed openstack-infra/project-config master: Remove legacy jobs in neutron-vpnaas  https://review.openstack.org/52336904:06
*** sree has quit IRC04:06
openstackgerritsebastian marcet proposed openstack-infra/openstackid-resources master: Added new endpoint for admin get attendee by id  https://review.openstack.org/53306404:06
*** Apoorva has joined #openstack-infra04:07
openstackgerritMerged openstack-infra/openstackid-resources master: Added new endpoint for admin get attendee by id  https://review.openstack.org/53306404:09
*** markvoelker has joined #openstack-infra04:11
openstackgerritLars Kellogg-Stedman proposed openstack-infra/git-review master: Fix compare_review's use of fetch_review  https://review.openstack.org/53235904:12
*** sree has joined #openstack-infra04:12
openstackgerritTristan Cacqueray proposed openstack-infra/nodepool feature/zuulv3: Do pep8 housekeeping according to zuul rules  https://review.openstack.org/52294504:18
openstackgerritTristan Cacqueray proposed openstack-infra/nodepool feature/zuulv3: Use same flake8 config as in zuul  https://review.openstack.org/50971504:18
*** edmondsw has joined #openstack-infra04:18
openstackgerritCao Xuan Hoang proposed openstack-infra/openstack-zuul-jobs master: Remove legacy jobs in neutron-vpnaas  https://review.openstack.org/52336804:20
openstackgerritTristan Cacqueray proposed openstack-infra/nodepool feature/zuulv3: config: add statsd-server config parameter  https://review.openstack.org/52896904:20
openstackgerritTristan Cacqueray proposed openstack-infra/nodepool feature/zuulv3: handler: fix support for handler without launch_manager  https://review.openstack.org/52477304:20
openstackgerritTristan Cacqueray proposed openstack-infra/nodepool feature/zuulv3: Add a plugin interface for drivers  https://review.openstack.org/52462004:20
*** nicolasbock has quit IRC04:21
*** sree has quit IRC04:22
*** sree has joined #openstack-infra04:22
*** edmondsw has quit IRC04:22
*** felipemonteiro_ has joined #openstack-infra04:24
*** links has joined #openstack-infra04:31
*** pgadiya has joined #openstack-infra04:34
openstackgerritsebastian marcet proposed openstack-infra/openstackid-resources master: Fixed summit attendee serializer  https://review.openstack.org/53306904:40
*** ankkumar has joined #openstack-infra04:41
openstackgerritMerged openstack-infra/openstackid-resources master: Fixed summit attendee serializer  https://review.openstack.org/53306904:41
ankkumarhello all04:43
ankkumarI am trying to build openstack/ironic CI using zuul v3.. Can anyone tell me, how would it would read my zuul.yaml customized file, because openstack/ironic repo has there one?04:43
ankkumarHow would my gate will run, I mean using which zuul yaml files?04:43
ianwankkumar: sorry, i'm not understanding what you're getting at.  you're trying to modify ironic jobs that run in the openstack gate?04:49
ankkumarianw, we are trying bring up the third party ci with zuul v304:50
ankkumarand the third party ci will run the ironic jobs only04:50
*** vivsoni has quit IRC04:50
ankkumarany idea how to do that?04:51
*** namnh has joined #openstack-infra04:51
*** psachin has joined #openstack-infra04:51
*** ijw has joined #openstack-infra04:57
ianwankkumar: well, that is a rather large thing ... zuulv3 is also not actually released as such, so there will be some rough edges04:57
*** vivsoni has joined #openstack-infra04:58
ankkumarianw, ok04:59
ianwankkumar: i'm not sure how far you've got.  looking a pabelangers windmill project might help http://git.openstack.org/cgit/openstack/windmill/05:01
*** ijw has quit IRC05:01
*** Apoorva has quit IRC05:02
AJaegerianw: could you review https://review.openstack.org/#/c/523369 to avoid extra running jobs for vpnaas, please?05:11
*** armax has quit IRC05:14
*** hongbin has quit IRC05:18
*** claudiub has joined #openstack-infra05:19
*** hoangcx has joined #openstack-infra05:20
*** ykarel|away has joined #openstack-infra05:22
*** felipemonteiro_ has quit IRC05:29
openstackgerritmegan guiney proposed openstack-infra/openstack-zuul-jobs master: [WIP] docs sitemap generation automation  https://review.openstack.org/52486205:38
openstackgerritMerged openstack-infra/project-config master: Remove legacy jobs in neutron-vpnaas  https://review.openstack.org/52336905:41
*** zhurong has joined #openstack-infra05:42
*** ijw has joined #openstack-infra05:45
*** ijw has quit IRC05:49
*** ijw has joined #openstack-infra05:49
*** iyamahat has quit IRC05:56
*** andreas_s has joined #openstack-infra05:57
*** KingParrot has joined #openstack-infra05:57
KingParrothow do I invite others to my irc chat room?05:58
*** bhavik1 has joined #openstack-infra05:59
*** caphrim007 has joined #openstack-infra06:01
*** andreas_s has quit IRC06:03
*** dhajare_ has joined #openstack-infra06:04
openstackgerritTristan Cacqueray proposed openstack-infra/nodepool feature/zuulv3: builder: do not cleanup image for driver not managing image  https://review.openstack.org/51692006:04
openstackgerritTristan Cacqueray proposed openstack-infra/nodepool feature/zuulv3: Implement a static driver for Nodepool  https://review.openstack.org/46862406:04
*** caphrim007 has quit IRC06:05
*** aeng has quit IRC06:06
KingParrotyeah image06:07
*** liujiong has quit IRC06:09
*** iyamahat has joined #openstack-infra06:12
*** slaweq has joined #openstack-infra06:15
*** liujiong has joined #openstack-infra06:15
*** ykarel|away is now known as ykarel06:17
*** iyamahat has quit IRC06:19
*** slaweq has quit IRC06:19
AJaegerchandankumar: did you see my question on https://review.openstack.org/#/c/530023 ?06:28
*** dbecker has quit IRC06:31
*** liujiong has quit IRC06:34
*** harlowja has quit IRC06:35
*** dbecker has joined #openstack-infra06:44
*** ijw has quit IRC06:46
*** jbadiapa has joined #openstack-infra06:56
*** daidv has quit IRC06:57
*** annp has quit IRC06:57
*** annp has joined #openstack-infra06:58
*** daidv has joined #openstack-infra06:58
*** jascott1 has quit IRC06:58
*** jascott1 has joined #openstack-infra06:58
*** hoangcx has quit IRC06:58
*** cuongnv has quit IRC06:58
*** kiennt26 has quit IRC06:58
*** hoangcx has joined #openstack-infra06:59
*** kiennt26 has joined #openstack-infra06:59
*** cuongnv has joined #openstack-infra06:59
*** bhavik1 has quit IRC07:02
KingParrotpool07:03
*** tdasilva has quit IRC07:07
*** harlowja has joined #openstack-infra07:10
*** harlowja has quit IRC07:10
gmannAJaeger: 2nd you mean governance patch ? on this- https://review.openstack.org/#/c/528180/07:15
gmannor adding job ?07:16
*** tdasilva has joined #openstack-infra07:21
*** KingParrot has left #openstack-infra07:22
*** rcernin has quit IRC07:28
*** pcaruana has joined #openstack-infra07:29
*** vsaienk0 has joined #openstack-infra07:33
*** armaan has quit IRC07:40
*** armaan has joined #openstack-infra07:40
vsaienk0ianw: could you please review https://review.openstack.org/#/c/531375 adds relese note jobs to n-g-s project07:41
openstackgerritGhanshyam Mann proposed openstack-infra/project-config master: Adding jobs to new tempest-stress  https://review.openstack.org/53311107:42
gmannAJaeger: ^^07:42
*** makowals has quit IRC07:42
*** ijw has joined #openstack-infra07:43
*** rlandy|bbl has quit IRC07:44
*** slaweq has joined #openstack-infra07:49
*** dhajare_ has quit IRC07:49
openstackgerritTristan Cacqueray proposed openstack-infra/zuul-jobs master: Add buildset-artifacts-location and fetch roles  https://review.openstack.org/53067907:52
*** eumel8 has joined #openstack-infra07:53
eumel8good morning07:54
*** threestrands has quit IRC07:56
eumel8ianw: still online?07:56
*** tosky has joined #openstack-infra07:56
*** tesseract has joined #openstack-infra08:08
*** florianf has quit IRC08:08
*** florianf has joined #openstack-infra08:09
*** jtomasek has joined #openstack-infra08:14
*** jtomasek has quit IRC08:14
*** jtomasek has joined #openstack-infra08:14
*** pcichy has joined #openstack-infra08:15
*** makowals has joined #openstack-infra08:16
*** hashar has joined #openstack-infra08:21
AJaegergmann: adding job - as you did.08:22
*** kopecmartin has joined #openstack-infra08:22
*** shardy has joined #openstack-infra08:24
openstackgerritMarcin Juszkiewicz proposed openstack/diskimage-builder master: provide names for arm64 grub packages  https://review.openstack.org/53312608:24
*** alexchadin has joined #openstack-infra08:30
*** cshastri has joined #openstack-infra08:30
*** ramishra has joined #openstack-infra08:32
*** vsaienk0 has quit IRC08:41
*** pcichy has quit IRC08:42
*** dhajare_ has joined #openstack-infra08:43
*** yamamoto_ has joined #openstack-infra08:45
*** vsaienk0 has joined #openstack-infra08:46
*** yamamoto has quit IRC08:48
*** jpena|off is now known as jpena08:51
*** dhajare_ has quit IRC08:53
openstackgerritAndreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Adjust integrated gate for neutron changes  https://review.openstack.org/53313208:55
*** jpich has joined #openstack-infra08:55
*** alexchad_ has joined #openstack-infra08:58
*** alexchadin has quit IRC08:59
*** ralonsoh has joined #openstack-infra09:02
*** ramishra has quit IRC09:02
openstackgerritAndreas Jaeger proposed openstack-infra/project-config master: Remove legacy jobs from Trove  https://review.openstack.org/51114909:04
AJaegerfrickler, could you review ^ to remove the trove jobs, please?09:04
*** zhurong has quit IRC09:20
*** eglute has quit IRC09:21
*** e0ne has joined #openstack-infra09:23
*** lucas-afk is now known as lucasagomes09:25
*** eglute has joined #openstack-infra09:26
*** slaweq has quit IRC09:33
*** slaweq has joined #openstack-infra09:34
*** jascott1 has quit IRC09:35
*** jascott1 has joined #openstack-infra09:35
fricklerAJaeger: +209:39
AJaegerthanks09:39
*** derekh has joined #openstack-infra09:39
*** jascott1 has quit IRC09:39
*** dtantsur|afk is now known as dtantsur09:42
hoangcxfrickler: Hi. Could you please help to approve this one https://review.openstack.org/#/c/527872/09:47
*** stakeda has quit IRC09:47
*** rossella_s has quit IRC09:49
hoangcxfrickler: Or do you know who do I need to ask for help? Thanks a lot09:50
*** gcb has quit IRC09:55
*** tosky has quit IRC09:56
fricklerhoangcx: yeah, I'll take a look in a bit09:59
hoangcxfrickler: Thank you :)10:00
*** pcichy has joined #openstack-infra10:01
*** mdrabe has quit IRC10:01
*** rossella_s has joined #openstack-infra10:01
*** caphrim007 has joined #openstack-infra10:01
*** pcichy has quit IRC10:03
*** mamitchl has joined #openstack-infra10:04
*** pcichy has joined #openstack-infra10:04
*** mdrabe has joined #openstack-infra10:05
*** numans has quit IRC10:05
*** caphrim007 has quit IRC10:05
*** numans has joined #openstack-infra10:07
*** yamamoto has joined #openstack-infra10:07
*** kiennt26 has quit IRC10:07
*** tosky has joined #openstack-infra10:08
*** yamamoto_ has quit IRC10:10
*** daidv has quit IRC10:10
openstackgerritYAMAMOTO Takashi proposed openstack-infra/project-config master: neutron-vpnaas: Add neutron-lib-master periodic job  https://review.openstack.org/53314110:11
*** yamahata has quit IRC10:14
*** namnh has quit IRC10:15
*** wxy has quit IRC10:16
*** pbourke has quit IRC10:16
*** pbourke has joined #openstack-infra10:18
Adri2000hello, would appreciate a second +2 at https://review.openstack.org/#/c/531820/ as I keep getting a lot of spam on that -owner@lists.o.o adress10:20
fricklerAdri2000: done10:22
*** sambetts|afk is now known as sambetts10:23
Adri2000thanks!10:23
hrwnow time for +W? ;D10:25
ianweumel8: around for a little, something not working?10:28
*** hoangcx has quit IRC10:33
*** yamamoto_ has joined #openstack-infra10:36
openstackgerritStephen Finucane proposed openstack-infra/openstack-zuul-jobs master: Remove legacy jobs in nova  https://review.openstack.org/51431010:36
*** ldnunes has joined #openstack-infra10:38
*** jascott1 has joined #openstack-infra10:38
*** yamamot__ has joined #openstack-infra10:38
openstackgerritStephen Finucane proposed openstack-infra/project-config master: Remove legacy jobs in nova  https://review.openstack.org/51431110:38
*** yamamoto has quit IRC10:39
AJaegerfrickler, ianw, could you review https://review.openstack.org/#/c/529946/ , please? the dependency is in... Thanks10:39
openstackgerritMerged openstack-infra/project-config master: Add relase notes to n-g-s  https://review.openstack.org/53137510:40
*** dulek has quit IRC10:40
*** zoli has quit IRC10:40
AJaegerand somebody please review https://review.openstack.org/#/c/514311 - the dependency is merged as well10:41
*** yamamoto_ has quit IRC10:42
*** markvoelker has quit IRC10:44
eumel8ianw: many thanks . everything works for me as expected10:47
*** gridinv has quit IRC10:47
eumel8ianw: some problems with email, but frickler checked the logs and it caused by my company email server10:49
*** shardy has quit IRC10:50
*** gridinv has joined #openstack-infra10:51
*** shardy has joined #openstack-infra10:52
openstackgerritMerged openstack-infra/project-config master: Remove legacy jobs from Trove  https://review.openstack.org/51114910:53
*** dulek has joined #openstack-infra10:53
*** wxy has joined #openstack-infra10:54
*** zoli has joined #openstack-infra10:55
*** pabbi has joined #openstack-infra10:55
*** edmondsw has joined #openstack-infra11:03
*** edmondsw has quit IRC11:08
*** yamamot__ has quit IRC11:12
*** sdague has joined #openstack-infra11:14
*** tpsilva has joined #openstack-infra11:16
*** yamamoto has joined #openstack-infra11:17
*** cuongnv has quit IRC11:17
openstackgerritMerged openstack-infra/system-config master: Add #openstack-vpnaas to statusbot/meetbot  https://review.openstack.org/52787211:19
openstackgerritMerged openstack-infra/system-config master: Treat mail sent to openstack-fr-owner as spam  https://review.openstack.org/53182011:19
*** odyssey4me has quit IRC11:19
*** yamamoto has quit IRC11:21
*** pabbi has quit IRC11:30
*** pcichy has quit IRC11:31
*** pcichy has joined #openstack-infra11:31
*** armaan has quit IRC11:32
*** armaan has joined #openstack-infra11:32
*** armaan_ has joined #openstack-infra11:33
*** armaan has quit IRC11:33
*** nicolasbock has joined #openstack-infra11:36
ianweumel8: yeah, i dropped a comment in the original review.  it's likely the return address of noreply@openstack.org, which doesn't exist11:40
eumel8ianw: alright. You saw the sender address verification check in the log?11:42
*** odyssey4me has joined #openstack-infra11:42
*** yamamoto has joined #openstack-infra11:43
*** jkilpatr has quit IRC11:44
*** odyssey4me has quit IRC11:45
*** odyssey4me has joined #openstack-infra11:45
*** alexchad_ has quit IRC11:51
*** alexchadin has joined #openstack-infra11:52
openstackgerritMerged openstack-infra/project-config master: remove some irrelevant-files for tempest-full  https://review.openstack.org/52994611:53
openstackgerritMerged openstack-infra/project-config master: Remove legacy jobs in nova  https://review.openstack.org/51431111:53
*** odyssey4me has quit IRC11:53
*** odyssey4me has joined #openstack-infra11:54
*** yamamoto has quit IRC11:55
*** amoralej has joined #openstack-infra11:56
amoralejhi11:58
*** odyssey4me has quit IRC11:58
amoralejtripleo jobs are queued for long time11:58
*** odyssey4me has joined #openstack-infra11:58
*** dtantsur is now known as dtantsur|brb11:59
ianweumel8: saw it getting greylisted, suggesting at least @redhat.com's mail servers thought it was suspect :)11:59
ianwi think it has to be somehting that accepts a bounce.  might want to check with fungi, maybe postmaster@ or something is good12:02
*** alexchadin has quit IRC12:03
*** annp has quit IRC12:07
*** alexchadin has joined #openstack-infra12:08
*** alexchadin has quit IRC12:10
*** edmondsw has joined #openstack-infra12:11
*** yamamoto has joined #openstack-infra12:11
*** odyssey4me has quit IRC12:13
*** odyssey4me has joined #openstack-infra12:13
hrwis #openstack-dib in use?12:13
hrwor should diskimage-builder discussed here?12:13
*** yamamoto has quit IRC12:17
*** alexchadin has joined #openstack-infra12:18
*** rosmaita has joined #openstack-infra12:24
*** pblaho1 has quit IRC12:26
*** lucasagomes is now known as lucas-hungry12:26
*** jkilpatr has joined #openstack-infra12:26
*** ganso has joined #openstack-infra12:34
gansoHello folks. It looks like a recent oslo.cfg change has broken manila's driverfixes/ocata branch. We can't pass py27 and approve patches there. Does anybody know if the solution to this is proposing a patch to openstack/requirements stable/ocata branch to change the oslo.cfg version? I was surprised to see this sort of breakage in ocata branch. Here is the breakage details https://review.openstack.org/#/c/52445812:36
*** janki has joined #openstack-infra12:41
*** yamamoto has joined #openstack-infra12:42
*** pblaho1 has joined #openstack-infra12:43
*** jpich has quit IRC12:43
*** psachin has quit IRC12:45
*** markvoelker has joined #openstack-infra12:45
*** jpena is now known as jpena|lunch12:49
*** hjensas has quit IRC12:51
*** smatzek has joined #openstack-infra12:53
*** dave-mccowan has joined #openstack-infra12:53
*** hjensas has joined #openstack-infra12:55
*** hjensas has quit IRC12:55
*** hjensas has joined #openstack-infra12:55
*** pblaho1 has quit IRC12:59
*** pblaho has joined #openstack-infra12:59
*** caphrim007 has joined #openstack-infra13:01
*** Goneri has joined #openstack-infra13:05
*** caphrim007 has quit IRC13:06
*** ameade has quit IRC13:09
*** ameade has joined #openstack-infra13:10
*** hrybacki has quit IRC13:10
*** fkautz has quit IRC13:10
*** sweston has quit IRC13:11
*** ggherdov- has quit IRC13:12
*** alexchadin has quit IRC13:12
*** olaph1 has joined #openstack-infra13:13
*** fkautz has joined #openstack-infra13:13
*** hrybacki has joined #openstack-infra13:13
*** olaph has quit IRC13:14
*** sweston has joined #openstack-infra13:14
*** lucas-hungry is now known as lucasagomes13:16
*** markvoelker has quit IRC13:20
*** numans has quit IRC13:20
*** numans has joined #openstack-infra13:20
*** dprince has joined #openstack-infra13:21
*** rkukura_ has joined #openstack-infra13:22
*** rkukura has quit IRC13:22
*** rkukura_ is now known as rkukura13:22
*** alexchadin has joined #openstack-infra13:24
*** dtantsur|brb is now known as dtantsur13:30
*** slaweq has quit IRC13:30
*** alexchadin has quit IRC13:30
*** slaweq has joined #openstack-infra13:31
*** alexchadin has joined #openstack-infra13:31
panda|ruck|afkI have an openstack-tox-pep8 timing out, but it really seems to pass, not sure what is timing out on 9/1/check-tripleo/tripleo-ci-centos-7-ovb-ha-oooq-pike/0f7a222/job-output.txt.gz13:35
*** alexchadin has quit IRC13:36
panda|ruck|afksorry wrong link13:36
*** pgadiya has quit IRC13:36
*** trown|outtypewww is now known as trown13:36
panda|ruck|afkhttp://logs.openstack.org/26/526526/1/gate/openstack-tox-pep8/2d1402d/13:36
toskyit stops at [upload-logs : gzip console log and json output]13:38
toskyI've seen that since yesterday, zuul seems to be still flacky13:38
toskybut I'm just a user13:38
*** efried is now known as fried_rice13:38
panda|ruck|afktosky: I knew about that, but it actually times out before13:41
panda|ruck|afk2018-01-12 12:00:07.710925 | ubuntu-xenial |   congratulations :)13:41
*** ykarel is now known as ykarel|away13:41
panda|ruck|afk2018-01-12 12:28:30.739893 | RUN END RESULT_TIMED_OUT: [untrusted : git.openstack.org/openstack-infra/zuul-jobs/playbooks/tox/run.yaml@master]13:41
panda|ruck|afkit hangs for 28 minutes13:41
panda|ruck|afkmaybe  it wants to confratulate you thoroughly13:41
*** armaan_ has quit IRC13:42
*** armaan has joined #openstack-infra13:43
*** wolverineav has joined #openstack-infra13:43
*** stephenfin is now known as finucannot13:44
*** rlandy has joined #openstack-infra13:44
*** jpena|lunch is now known as jpena13:49
*** david-lyle has quit IRC13:49
*** ykarel|away has quit IRC13:53
*** bhagyashris has joined #openstack-infra14:00
*** jonher has quit IRC14:02
*** ankkumar has quit IRC14:04
*** esberglu has joined #openstack-infra14:04
*** ralonsoh has quit IRC14:07
*** ralonsoh has joined #openstack-infra14:10
*** hashar is now known as Camembert14:11
*** jonher has joined #openstack-infra14:11
*** eharney has joined #openstack-infra14:14
*** amoralej is now known as amoralej|lunch14:14
*** florianf has quit IRC14:16
*** jonher has quit IRC14:16
*** jonher has joined #openstack-infra14:17
*** kgiusti has joined #openstack-infra14:18
fungiganso: chances are your driverfixes/ocata branch isn't using the stable/ocata constraints file. check in tox.ini14:18
*** mriedem has joined #openstack-infra14:18
*** edmondsw has quit IRC14:19
*** edmondsw has joined #openstack-infra14:19
gansofungi: it is https://github.com/openstack/manila/blob/driverfixes/ocata/tox.ini#L914:19
gansofungi: I tried to run both driverfixes/ocata and stable/ocata locally, when cleaning up .tox both fail to create the env, but it shows driverfixes/ocata picking up the correct oslo.cfg version (3.22.0) instead of 5.2.0 as the CI is doing14:21
gansofungi: for some reason I can't create my test env in those branches14:22
gansofungi: I am not sure how CI is able to14:22
*** ethfci has joined #openstack-infra14:22
gansofungi: but whatever it is doing differently, it is causing it to not pick the upper-constraint from stable/ocata. While the stable/ocata CI py27 job is working flawlessly14:23
*** edmondsw has quit IRC14:24
*** smatzek has quit IRC14:25
*** smatzek has joined #openstack-infra14:26
fungi#status log manually started the apache2 service on ask.openstack.org since it seems to have segfaulted and died during log rotation14:26
openstackstatusfungi: finished logging14:26
*** hwoarang has quit IRC14:27
AJaegerganso: but that is ignored, it's the fallback. Check http://logs.openstack.org/58/524458/1/check/openstack-tox-py27/fb410c8/job-output.txt.gz#_2018-01-10_17_48_02_894190 - it uses master branch for requirements14:29
fungiganso: got it, failing in ci and not locally? so yeah the fallback url is correct in tox.ini but the job definition may need an override. i'm looking for an example (i think cinder fixed it)14:29
*** hwoarang has joined #openstack-infra14:29
*** smatzek has quit IRC14:30
gansoAJaeger: humm so it needs to be set to not use the master branch for requirements, it should do the same as stable/ocata branch14:30
*** odyssey4me has quit IRC14:30
AJaegerfungi, could you put https://review.openstack.org/#/c/531470/ - Zuul v3 native jobs for bindep - on your review queue, please?14:30
*** odyssey4me has joined #openstack-infra14:31
fungiganso: right, the problem is that the stable/ocata branch jobs find a stable/ocata branch in requirements. the driverfixes/ocata jobs look for driverfixes/ocata in requirements but don't find one, so they call back on its master branch14:31
*** odyssey4me has quit IRC14:32
*** numans has quit IRC14:32
*** odyssey4me has joined #openstack-infra14:33
gansofungi: humm ok I'll take a look at the job definition then, thanks!14:34
fungiaha, looks like cinder vendored in a copy of the constraints file14:34
fungihttps://review.openstack.org/50371114:34
gansofungi: yea they have a upper-constraints.txt file14:34
purpHowdy, folks. I know it's not an Infra service, but wondering if someone has noted the internal server error on Stackalytics and, if not, who I could chase down on that.14:34
gansofungi: great! I'll try to do the same!14:35
gansofungi: thank you!14:35
fungiganso: i guess that's one way to do it, but we ought to have a more general override solution to pass the desired one into the tox role14:35
fungiganso: anyway, it's for sure a quick fix for no14:36
fungiw14:36
*** esberglu has quit IRC14:36
*** janki has quit IRC14:36
*** hwoarang_ has joined #openstack-infra14:37
*** hwoarang has quit IRC14:38
fungipurp: https://review.openstack.org/#/admin/groups/stackalytics-core14:39
fungithat's all the contact info i know about14:39
*** bobh has joined #openstack-infra14:39
purpThanks, fungi ... just what I needed.14:39
purpfungi .. except it's an admin link, and I'm not that all-powerful14:40
fungipurp: you should be able to see it as long as you log into gerrit?14:41
*** hashar has joined #openstack-infra14:44
gibihi, I have a question about irrelevant-files field in the zuul job definitions14:50
gibiThere is the job openstack-tox-functional job definition https://github.com/openstack-infra/openstack-zuul-jobs/blob/21241aec13dd3d8eae2916518a5ad8933509a512/zuul.d/jobs.yaml#L36114:50
gibiIt is defined not to run on "- ^doc/.*$"14:50
gibiHere is the definition of the same job in project-config https://github.com/openstack-infra/project-config/blob/5017c2ba472bd0439392f037036dbf198e00c137/zuul.d/projects.yaml#L1079714:50
gibiHere with different irrelevant-files regexes, e.g. "- ^doc/source/.*$"14:50
gibiIs there a way to override the regex coming from the jobs.yaml?14:51
gibibecuase it seems that the current project-config definition does not override it14:51
*** edmondsw has joined #openstack-infra14:52
smcginnisgibi: Yep, if the job is added through a template, then you can't override that.14:52
smcginnisgibi: A confusing and frustrating "feature" to me.14:52
*** markvoelker has joined #openstack-infra14:52
smcginnisAnd based on the number of jobs in project-config that are attempting to override that, I think it's widely misunderstood.14:53
gibismcginnis: so how can I make sure that the openstack-tox-functional runs on ./doc changes in nova?14:53
smcginnisgibi: You mean "does not run", right?14:54
gibismcginnis: run on ./doc but does not run on ./doc/sources14:54
smcginnisHmm, doesn't it now?14:54
gibismcginnis: as now it skips the whole ./doc14:54
gibismcginnis: but that is too much for nova14:54
gibismcginnis: e.g. changes in ./doc/notification_samples should trigger the functional test14:55
*** ykarel|away has joined #openstack-infra14:55
AJaegersmcginnis: what we have right now was auto-converted - so somebody needs to take it on and fix...14:55
AJaegergibi: why do you want that?14:55
gibiAJaeger: because those sample files are used in the functional test14:55
smcginnisAJaeger, gibi: I'm assuming the actual job definition has an irrelevant-files match that is causing that to skip. Are they all rst files?14:56
gibismcginnis: the actual skip is caused by "- ^doc/.*$" I think14:56
gibismcginnis: they are .json files14:56
smcginnisgibi: It must be in the job definition though, which probably is close to what is in where you linked to. I usually have the opposite problem.14:57
smcginnisAJaeger: Would adding an inverse to include .*json get that to run, do you think?14:57
* smcginnis is blanking on the name of whatever is the opposite of irrelevant-files14:58
AJaegersmcginnis: that would be files list - and then it would be only if those are changed which you don't want either14:58
gibiAJaeger: correct14:59
*** amoralej|lunch is now known as amoralej14:59
*** smatzek has joined #openstack-infra14:59
AJaegergibi: do you have a change to look at that triggred this discussion?15:00
smcginnisI thought you do want it if json files are changed?15:00
*** makowals has quit IRC15:00
gibiAJaeger: here it is https://review.openstack.org/#/q/39f9ef719eacb2fbcf0c37680b845152fcff01cc15:00
AJaegersmcginnis: "files:" is the opposite. But files means: only if one of those are changed15:00
gibiAJaeger: so the original change https://review.openstack.org/#/c/531381/ got merged without functional test run but it actually broke the test15:01
AJaegergibi: let me push a test change...15:02
gibiAJaeger: sure, thanks for the help15:02
*** esberglu has joined #openstack-infra15:02
AJaegergibi, https://review.openstack.org/533202 - I'm curious about the debug output...15:05
*** hwoarang_ has quit IRC15:07
AJaegergibi: interesting, the tox-functional job is run...15:09
AJaegerah, might be due to the .zuul.yaml change, I might need to split into 2...15:09
*** ggherdov- has joined #openstack-infra15:11
gibiAJaeger: it seems that it now triggered everything15:12
AJaegergibi: yes - since I changed another file as well. https://review.openstack.org/533206  reproduces your problem (stacked on top).15:12
gibiAJaeger: ahh, I see15:13
fricklerinfra-root: I've seen multiple post_failures on ovh-bhs, see e.g. https://review.openstack.org/532188 and https://review.openstack.org/532189 , each recheck just seems to run into another failure for a different job. maybe someone can dig a bit more, I'm off for the weekend now15:13
*** katkapilatova1 has joined #openstack-infra15:14
AJaegerthere might be two solutions - but let's wait for 533206 to finish first - 1) change the job in openstack-zuul-jobs to use doc/source instead of doc in the irrelevant files list. 2) create a new nova-tox-fucntional job that inherits from openstack-tox-functional and has it's own irrelevant-file list.15:14
AJaegergibi: ^ do you want to try 2? Make it two changes: First to add job, second to test that it does the right thing...15:14
* AJaeger will be back later...15:15
gibiAJaeger: thanks, I will try #2)15:15
*** kopecmartin has quit IRC15:16
*** panda|ruck|afk is now known as panda|ruck15:17
*** yamahata has joined #openstack-infra15:18
*** armaan has quit IRC15:22
*** armaan has joined #openstack-infra15:22
rosmaitathe status page tells me that one of the gates has a post-failure, so i'm eventually going to get a -1 from zuul.  if i do a recheck on the review right now, will zuul kill off the other gate jobs its running, and start over?15:23
*** pcrews has quit IRC15:25
smcginnisrosmaita: I don't believe so.15:25
smcginnisI think I've tried that before. Would be nice if it did to not waste time and resources in some cases, but also good protection for the many times I've seen someone comment recheck 10 minutes after someone else already did it.15:26
pabelangerrosmaita: no, you can only use recheck after all jobs have finished running.15:27
rosmaitasmcginnis. pabelanger thanks! that's what i thought, but figured it would be worth asking15:27
fungiwell, also if a change ahead of it in a shared queue fails then that change will get all its jobs rerun anyway. zuul doesn't know how to tell whether a change is failing due to a bug in a change approved ahead of it and so has to assume it may be15:27
*** jd_ has quit IRC15:27
* smcginnis would like a "killitnow" command to just stop a patch when we now we don't need all jobs to finish running15:29
pabelangerShrews: maybe have some time today to help clear out infracloud-vanilla from zookeeper?15:30
rosmaitasmcginnis ++15:30
pabelangersmcginnis: you can abandon the patch, then restore. That is one way to dequeue it from queue.15:30
smcginnispabelanger: But then restore would just kick it off again, right? I guess that's still a useful trick in some cases.15:31
pabelangerthere have also been discussions about fast fail in the past also, so when the first job fails, the abort the rest of running jobs15:31
*** dansmith is now known as superdan15:31
pabelangersmcginnis: right, would start check jobs again15:31
Shrewspabelanger: sure15:31
smcginnispabelanger: Probably what is actually needed in most cases.15:31
pabelangerI wonder with zuulv3 changes, if something like fail strategy could be setup per project, vs global15:32
pabelangerunsure TBH15:32
Shrewspabelanger: do you know if we also need to delete vanilla instances? I guess we really can't...15:34
pabelangerShrews: yah, we cannot right now.15:34
pabelangerno controller15:34
pabelangerwe'll need to decide on the best plan on moving the compute nodes into chocolate next week15:35
*** jtomasek has quit IRC15:36
*** hashar is now known as hasharELATER15:38
*** david-lyle has joined #openstack-infra15:38
*** ijw has quit IRC15:39
*** ijw has joined #openstack-infra15:39
*** hwoarang has joined #openstack-infra15:40
Shrewspabelanger: zookeeper is cleansed15:41
Shrewspabelanger: fwiw, if vanilla ever returns (is that the plan?) we'll need to manually clean any existing instances (if any)15:42
*** Goneri has quit IRC15:44
AJaegercorvus: I'm confused again with irrelevant-files, see the discussion with gibi above. I expected that https://review.openstack.org/#/c/533211/ would run the new nova-tox-functional but it does not. What am I missing?15:46
*** bauzas is now known as bauwser15:48
corvusAJaeger: looking15:49
*** ijw_ has joined #openstack-infra15:49
*** ijw has quit IRC15:52
*** felipemonteiro_ has joined #openstack-infra15:53
corvusAJaeger, gibi: http://git.openstack.org/cgit/openstack-infra/openstack-zuul-jobs/tree/zuul.d/jobs.yaml#n352 says that any file in "doc/" is irrelevant and so will never run.15:53
gibicorvus: yes15:54
gibicorvus: but we need to override that with something like  - ^doc/source/.*$15:54
AJaegercorvus: so, inheritance (the parent) will not help?15:54
AJaegercorvus: I mean: Inheritance with a new irrelevant-files list15:55
corvusAJaeger, gibi: if you want it to run, then it seems like the irrelevant-files section for openstack-tox-functional is wrong -- things should only go there if it's safe to say that no openstack project will ever want to run the job for those files.15:55
corvusAJaeger, gibi: matchers aren't inherited, they determine whether that job or variant is used: https://docs.openstack.org/infra/zuul/feature/zuulv3/user/config.html#matchers15:56
gibicorvus: in nova the ./doc/notificaton_samples and ./doc/api_samples are used in the functional test15:56
AJaegergibi: then change the job in openstack-zuul-jobs...15:57
AJaegercorvus: thanks!15:57
corvusor you could just copy the definition and make a new one.  there isn't much to it.  :)15:57
*** cburgess has quit IRC15:57
gibicorvus, AJaeger: I would copy it as I don't want to change the irrelevant-files for every other projects using the openstack-tox-functional15:58
AJaegergibi: then copy it over...15:58
*** cburgess has joined #openstack-infra15:59
gibiAJaeger: I will do.16:00
gibicorvus, AJaeger: thanks for the help16:00
pabelangerShrews: I don't think we'll have infracloud-vanilla again, the plan it to let that hardware die off.  It is possible we'll use the name again, but unsure when16:00
corvusgibi: you're welcome!16:01
*** rpittau has quit IRC16:01
*** hongbin has joined #openstack-infra16:02
*** felipemonteiro__ has joined #openstack-infra16:02
*** Apoorva has joined #openstack-infra16:02
amotokiAJaeger: I see your change in horizon https://review.openstack.org/#/c/533215/ but I cannot find a Needed=Bu change. How can I understand the current situation?16:03
pabelangerShrews: great, nodes for vanilla look right now. However, we still have infracloud-vanilla references for images, we'll also need to clean them up16:03
AJaegeramotoki: give me a second - this is a cross-repo afford. Let me finish the parts and send an email out...16:03
AJaeger(better 5 mins)16:03
amotokiAJaeger: thanks. no need to rush :)16:04
amotokii saw your change just now and it seems a bad timing :p16:04
*** felipemonteiro_ has quit IRC16:05
pabelangerShrews: corvus: I am noticing a larger number of ready nodes in grafana for nodepool currently, is there any why to see why that is?  I'm guess it is because we are waiting for all the nodes in a nodeset to be online before moving to used?16:06
openstackgerritMerged openstack-infra/project-config master: Create stable/queens specific ACL  https://review.openstack.org/53220516:07
pabelangeryah, most ready / locked nodes appear to be in rackspace16:08
*** katkapilatova1 has quit IRC16:08
pabelangerwhich take a little longer to come online16:08
AJaegeramotoki, corvus https://review.openstack.org/533225 Rename tox_venvlist16:08
openstackgerritDoug Hellmann proposed openstack-infra/openstack-zuul-jobs master: print some progress information  https://review.openstack.org/53322616:10
openstackgerritDoug Hellmann proposed openstack-infra/openstack-zuul-jobs master: write the sitemap to the doc/build/html directory  https://review.openstack.org/53322716:10
openstackgerritDoug Hellmann proposed openstack-infra/openstack-zuul-jobs master: convert "trunk" to "latest" in URLs  https://review.openstack.org/53322816:10
openstackgerritDoug Hellmann proposed openstack-infra/openstack-zuul-jobs master: look for .rst files in git as well as .html  https://review.openstack.org/53322916:10
pabelangeralso, it seems we might have some quota calculation issues, we're launching nodes in inap but at our quota limits16:13
pabelangershade.exc.OpenStackCloudHTTPError: (403) Client Error for url: https://compute.api.mtl01.cloud.iweb.com/v2/addf341f4fa44528b1ac73e2f5c68a21/servers Quota exceeded for instances: Requested 1, but already used 200 of 200 instances16:13
pabelangergoing to dig into it and see why16:14
mgagnepabelanger: checkign16:14
pabelangerwe have max-servers: 190 in nodepool.yaml16:15
mgagneyes, we tried 200 and nodepool managed to still reach quota even when there wasn't any issue in our database related to quota16:15
AJaegeramotoki: mail sent - should I explain more?16:16
pabelanger2018-01-12 16:15:52,675 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for inap-mtl01: {'compute': {'instances': 200, 'cores': inf, 'ram': inf}}16:16
mgagnepabelanger: yes, we set 190 on purpose for the reasons mentioned above16:16
pabelangeryah, so I think our new quota logic isn't respecting max-servers now16:17
mgagnewill ask Nova to forcefully recompute quota usage16:17
pabelangerShrews: ^ is that right?16:17
mgagnecurrently 178 instances in nova list16:17
mgagne178 < 20016:17
pabelangermgagne: k16:17
*** slaweq has quit IRC16:18
*** slaweq has joined #openstack-infra16:18
Shrewspabelanger: not sure. sounds like a bug. maybe we should ask tristanC16:20
mgagneok, forcefully asked nova to recompute quota usage16:20
AJaegergibi: your new change looks fine. So, you culd change the commit message, add to gate as well - and send a followup for project-config to remove the job there.16:20
gibiAJaeger: I will do that, but only on Monday as the weekend will hit soon here16:21
pabelangerShrews: thanks16:21
pabelangermgagne: okay, let me check nodepool again16:21
corvusShrews: you mean tobiash ?16:21
Shrewsor that, yes16:21
*** ykarel|away has quit IRC16:21
pabelangerProvider quota for inap-mtl01: {'compute': {'instances': 200, 'cores': inf, 'ram': inf}}16:22
mgagnepabelanger: quota is the same, usage is not16:22
mgagnewas 200 used when in fact there was only like ~178 instances16:22
pabelangeryah, it seems max-servers is not longer honored16:22
*** slaweq has quit IRC16:22
mgagnepabelanger: ok so maybe there are 2 problems16:22
corvuspabelanger: why do you say that?16:22
corvushow many instances did nodepool think were in use?16:23
mgagneyea, that too16:23
AJaegergibi: enjoy your weekend!16:23
corvuspabelanger: (iirc, the log line you copied doesn't include max-servers, that's applied at a different state -- that's just saying what nova is telling us our quota is)16:24
pabelangercorvus: we have 190 right now16:24
mgagne149+31 (use+ready) atm according to grafana16:24
pabelangerlet me check logs again to see if we are still trying to book16:24
pabelangerboot*16:24
corvuspabelanger: so it seems like we're observing max-servers and maybe the only problem is that nova counted wrong, thinking we had more in use than we really did?16:24
mgagnepabelanger: so nodepool is ok, 190 max but our database is out of sync and thinks you are in fact using 20016:24
mgagnepabelanger: what corvus said16:25
pabelangerAh, okay.16:25
*** armaan has quit IRC16:25
*** sbra has joined #openstack-infra16:25
fungigood news! this is not a nodepool bug. bad news... openstack has bugs16:26
mgagnethe usual out of sync quota usage in nova bug16:26
gibiAJaeger: thanks, same to you!16:26
AJaegercorvus: could you check https://review.openstack.org/#/c/529944/ , please? - I'd like to confirm that adding this to the "parent" of all the devstack jobs will not harm us. The change added irrelevant-files...16:26
*** jd_ has joined #openstack-infra16:27
*** sree has quit IRC16:27
corvusAJaeger: well, i think you know what it'll do -- it will mean that no one can ever run a devstack job if only those files change.  i never think irrelevant-files is a good idea, so i'm not likely to +2 it.  best if i abstain.16:29
pabelangerhttp://status.openstack.org/zuul looks down again16:29
pabelangerconfirmed, we are swapping again on zuulv3.o.o16:30
fungiyeah, cacti says memory usage on zuul just doubled in the last few minutes16:30
pabelangerhttp://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=63979&rra_id=all16:30
corvusi'll see if i can grab the queues16:31
AJaegerOops, me just pushed 9 changes that all change .zuul ;( Hope I didn't kick it over ;/16:32
AJaegerSorry!16:33
corvus9 doesn't sound like a lot16:33
corvusthough it does seem like there may have been quite a few others16:35
corvusmight have pushed it over the edge16:35
corvusi'm not getting any data back for queues.  should we restart it, or wait an hour?16:36
*** armax has joined #openstack-infra16:37
*** jistr has quit IRC16:37
*** dtantsur has quit IRC16:37
*** akscram1 has quit IRC16:37
*** antonym has quit IRC16:37
*** berendt has quit IRC16:37
*** petems has quit IRC16:37
*** cloudnull has quit IRC16:37
*** mhayden has quit IRC16:37
*** asettle has quit IRC16:37
*** StevenK has quit IRC16:37
*** lamt has quit IRC16:37
*** ari[m] has quit IRC16:37
*** mdbooth has quit IRC16:37
*** dosaboy has quit IRC16:37
*** charz has quit IRC16:37
*** lamt has joined #openstack-infra16:37
*** charz has joined #openstack-infra16:37
*** dosaboy has joined #openstack-infra16:37
*** StevenK has joined #openstack-infra16:37
*** asettle has joined #openstack-infra16:37
*** asettle is now known as Guest3153216:37
*** cloudnull has joined #openstack-infra16:37
*** lamt is now known as Guest6566216:37
*** akscram1 has joined #openstack-infra16:38
*** berendt has joined #openstack-infra16:38
*** petems has joined #openstack-infra16:38
* AJaeger votes for restart16:38
*** mhayden has joined #openstack-infra16:38
*** antonym has joined #openstack-infra16:38
fungitough call. the longer we wait the more queued events we're missing anyway16:38
*** links has quit IRC16:38
*** mdbooth has joined #openstack-infra16:39
corvusi'll restart16:39
*** SergeyLukjanov has quit IRC16:39
fungii guess if it doesn't look like it's on its way to recovering sanity enough to return status data, then restarting sooner means that much sooner people can recheck their jobs at least16:40
*** ari[m] has joined #openstack-infra16:40
*** jistr has joined #openstack-infra16:40
funginot to mention, if we reenqueued a huge stack of config changes, we might just put it right back into the same state anyway?16:40
corvusfungi: depends on if there's a leak or not.  no one has debugged that yet.16:41
fungiahh16:41
corvus(or, to the extent that i have, i have not found a leak)16:41
amotokiAJaeger: thanks. it explains enough16:41
corvusstatus notice Zuul has been restarted and lost queue information; changes in progress will need to be rechecked.16:42
openstackgerritDmitrii Shcherbakov proposed openstack-infra/project-config master: add charm-panko project-config  https://review.openstack.org/53276916:42
corvushow's that look?16:42
*** SergeyLukjanov has joined #openstack-infra16:42
*** eumel8 has quit IRC16:43
*** links has joined #openstack-infra16:43
AJaeger++16:43
corvus#status notice Zuul has been restarted and lost queue information; changes in progress will need to be rechecked.16:43
*** corvus is now known as jeblair16:43
jeblair#status notice Zuul has been restarted and lost queue information; changes in progress will need to be rechecked.16:43
openstackstatusjeblair: sending notice16:43
*** jeblair is now known as corvus16:43
corvuszuul is running again now16:44
corvusi installed the repl, so if memory usage climbs, we can inspect16:45
-openstackstatus- NOTICE: Zuul has been restarted and lost queue information; changes in progress will need to be rechecked.16:45
fungii guess if AJaeger rechecks his 9 config changes and memory spikes again, we'll know what's up16:45
AJaegerfungi: shall I? I planned to recheck 4 of them for now only...16:46
fungii think we've done far more before with no real ill effects16:46
pabelangercorvus: thanks16:46
openstackstatusjeblair: finished sending notice16:46
AJaegerfungi, pabelanger, could you review my bindep fallback Zuul v3 jobs, please? https://review.openstack.org/#/c/531470/16:47
AJaegerfungi, ok, will recheck all of them in 2 minutes16:47
*** ijw_ has quit IRC16:47
*** ricolin has quit IRC16:51
fungiAJaeger: i was just pulling it back up from your earlier mention in scrollback, but the reminder is appreciated16:51
AJaegercorvus: is the webserver up? http://zuulv3.openstack.org/ does not return...16:52
AJaegerthanks, fungi16:52
*** slaweq has joined #openstack-infra16:52
fungialso trying to suss out what's going on with the post_failure issues frickler mentioned seeing in ovh-bhs1 earlier16:52
corvusit's up, but i guess it needs restarting16:53
corvusdone16:53
AJaegerthanks, now working...16:53
*** sree has joined #openstack-infra16:55
*** slaweq has quit IRC16:56
*** vsaienk0 has quit IRC16:56
* AJaeger rechecked the changes...16:57
*** sree has quit IRC16:59
*** sree has joined #openstack-infra17:02
fungimemory usage spiked up pretty quickly17:02
fungithough not near capping out17:03
*** pcaruana has quit IRC17:03
fungiclose though17:03
AJaegerfar less changes in pipelines as well...17:03
toskyright, less than 70 reviews now vs >>200 before17:04
fungifree is suggesting this next snmp sample should jump quite a bit higher still17:06
fungiusing around 12-13gb active17:07
*** fried_rice is now known as fried_rolls17:07
AJaegerhigher than average of 9 GB - which was used earlier today...17:09
*** Goneri has joined #openstack-infra17:10
*** sree has quit IRC17:11
*** tesseract has quit IRC17:12
*** caphrim007 has joined #openstack-infra17:13
*** felipemonteiro__ has quit IRC17:15
*** felipemonteiro__ has joined #openstack-infra17:15
*** jascott1 has quit IRC17:15
*** jascott1 has joined #openstack-infra17:15
*** jascott1 has quit IRC17:16
corvusi'll poke17:20
corvusthere are 38 layouts, all of which are in the queues17:20
*** armaan has joined #openstack-infra17:20
fungiokay, so AJaeger's 9 changes are a mere quarter of the total17:20
fungilooks like at current layout size, we can satisfy a maximum of ~40 concurrent layouts in main memory?17:21
*** sree has joined #openstack-infra17:21
fungiwithout encroaching on swap17:21
corvusyeah, seems likely.  this is probably about as good of a test of that as we'd get.17:22
fungithough that also leaves very little headroom for buffers/cache17:22
corvusmaking this more efficient is next on my list, but i have to get through my current tasks first17:22
corvusin the mean time, best we can do is not upload many config changes at once17:22
corvusor grow the server17:23
fungii guess if we consider 40 concurrent layouts insufficient, we could also look at rebuilding with more ram17:23
fungiyeah17:23
clarkbit is a 16GB instance now right?17:23
*** links has quit IRC17:24
*** Dmitrii-Sh has joined #openstack-infra17:25
fungi15gb17:25
fungibut yeah17:25
funginext step would be 3017:26
*** sree has quit IRC17:26
*** spzala has joined #openstack-infra17:28
*** shardy has quit IRC17:28
*** slaweq has joined #openstack-infra17:28
*** slaweq has quit IRC17:29
zxiiroIs there a best practice for beta releases once the official release goes out? do we keep them around? do we remove them somehow?17:29
*** slaweq has joined #openstack-infra17:30
clarkbzxiiro: I think you just leave them on pypi, the trouble iwth deleting releases is if someone has depended on it they won't be able to get it any longer17:30
*** slaweq has quit IRC17:30
zxiirotrue... ok that works for me17:31
*** slaweq has joined #openstack-infra17:31
*** jlk has joined #openstack-infra17:31
*** jlk has joined #openstack-infra17:31
*** slaweq has quit IRC17:31
*** bhavik1 has joined #openstack-infra17:32
*** slaweq has joined #openstack-infra17:33
*** slaweq has quit IRC17:35
pabelangera new layout in zuul is created echo time a patchset contains zuul.yaml, is that correct?17:36
pabelangereach*17:36
*** cshastri has quit IRC17:36
*** felipemonteiro_ has joined #openstack-infra17:37
fungipabelanger: afaik, yes17:39
*** felipemonteiro__ has quit IRC17:41
AJaegerfungi: there're more than my 9 changes in the queue that touch zuul files, a quick look around found at least another two17:42
AJaegerfungi: and the first are tested already...17:43
pabelangerAJaeger: fungi: mind a +3 on https://review.openstack.org/533012/ so we can finish up reboots of infracloud-chocolate17:43
*** esberglu has quit IRC17:43
AJaegerclarkb, pabelanger, could either of you review https://review.openstack.org/#/c/531470/ - this adds a new zuul v3 native job for bindep.17:43
AJaegerpabelanger: will review...17:43
AJaegerpabelanger: +2A17:44
*** esberglu has joined #openstack-infra17:45
pabelangerAJaeger: +317:45
*** pcichy has quit IRC17:45
*** pcichy has joined #openstack-infra17:46
AJaegerthanks, pabelanger.17:46
*** pcichy has quit IRC17:46
*** pcichy has joined #openstack-infra17:47
AJaegerpabelanger: regarding  /usr/local/jenkins/common_data/bindep-fallback.txt - this is used by *all* legacy jobs. If we change it, we need to touch many jobs, including in-repo ones. I'd love to see it changed but that would be a slow migration - so, first introduce a new place, then change lots of places...17:47
tosky+1 for non-breaking (or limited-breaking) migrations17:48
*** bhavik1 has quit IRC17:48
AJaegertosky: agreed17:48
clarkbfungi: corvus with feature freeze and then release coming up it probably is a good idea to grow the server17:49
AJaegerpabelanger: care to review the bindep followups https://review.openstack.org/#/c/531536/ and https://review.openstack.org/#/c/531537/ as well, please?17:49
* AJaeger can +A himself once dependencies are in17:50
*** esberglu has quit IRC17:50
pabelangerAJaeger: done17:50
AJaegerthanks, pabelanger !17:50
toskyAJaeger: talking about keeping the changes minimal... :) https://review.openstack.org/#/c/532533/17:51
*** katkapilatova1 has joined #openstack-infra17:51
AJaegertosky: your call ;) I have my preference but will not block it...17:52
toskyAJaeger: I'm asking because it may influence the choice of how to deal with other copies of tox_install.sh around17:53
AJaegertosky: commented in case anybody waits for my feedback17:53
toskythanks17:53
openstackgerritMerged openstack-infra/project-config master: Set infracloud chocolate to max-servers: 0  https://review.openstack.org/53301217:53
toskybut apart from that - is there a way to totally get rid of tox_install.sh?17:53
toskyor at least a possible plan?17:53
AJaegertosky: mordred was investigating that17:53
pabelangerI thought there was a plan to remove it17:53
pabelangerand patchs even up17:53
toskyuh17:54
AJaegertosky: https://review.openstack.org/#/q/topic:rm-tox_install17:54
AJaegertosky: that's the trivial part ;) ^17:54
openstackgerritClark Boylan proposed openstack-infra/nodepool feature/zuulv3: Add DIB faq to docs  https://review.openstack.org/53325617:54
fungithe reasons for having a pip wrapper used by tox are being slowly whittled away, but it won't be viable to get rid of it for some projects, or at least not for the foreseeable future17:54
clarkbinfra-root and nodepool core ^ the arm64 thread inspired me to write that change. Would be nice if you could review it for acuracy17:55
*** Apoorva_ has joined #openstack-infra17:55
fungiand even then, there may be projects which still need one for local developer use cases even if they end up ignored by ci jobs17:55
toskyso the fact that there is no patch for sahara-dashboard hints that it may not possible to remove the script for that project?17:55
*** trown is now known as trown|lunch17:56
*** e0ne has quit IRC17:56
AJaegertosky: exactly - the problem is that all these dashboard repos install horizon.17:57
*** ijw has joined #openstack-infra17:57
*** Apoorva has quit IRC17:57
AJaegertosky: so, next step: Publish horizon and neutron on pypi, then we can do some magic with tox_siblings to get rid of tox_install everywhere...17:57
AJaeger... if everything works as planned ;)17:57
toskyoh, aren't they on pypi already?17:57
toskytoo many details I ignore O.o17:58
AJaegerany config-core to review that next step? https://review.openstack.org/53182217:58
Dmitrii-ShHi, happy Friday everybody. Could anybody please take a look at https://review.openstack.org/#/c/532769/2 ?17:58
AJaegertosky: not yet17:58
AJaegerDmitrii-Sh: what's the problem?17:58
*** derekh has quit IRC17:58
AJaegerDmitrii-Sh: you need two changes as explained in http://docs.openstack.org/infra/manual/creators.html17:59
AJaegerPlease split it up.17:59
Dmitrii-ShAJaeger: ok, will do17:59
*** pcichy has quit IRC17:59
*** pcichy has joined #openstack-infra17:59
openstackgerritMerged openstack-infra/bindep master: Add native Zuul v3 bindep-fallback test  https://review.openstack.org/53147017:59
*** markvoelker has quit IRC18:00
clarkbmordred: congrats on your board seat18:01
fungioh, results are in?18:02
clarkbfungi: ya jbryce sent them out18:02
clarkbcongrats to wendar too18:02
*** ijw has quit IRC18:02
wendarclarkb: thanks!18:02
fungiaha, mutt was open editing a reply, that's why i didn't see a highlight18:02
fungivery glad we get to keep wendar on the board!18:02
fungiand it will be nice to have mordred back18:03
*** katkapilatova1 has quit IRC18:03
*** felipemonteiro_ has quit IRC18:03
*** felipemonteiro_ has joined #openstack-infra18:04
openstackgerritIhar Hrachyshka proposed openstack-infra/devstack-gate master: Switch from lib/neutron-legacy to lib/neutron  https://review.openstack.org/43679818:04
*** pcichy has quit IRC18:04
AJaegercongrats wendar, mordred, sdake, ...18:04
sdakeAJaeger cheers thanks!18:04
*** pcichy has joined #openstack-infra18:04
*** ralonsoh has quit IRC18:05
wendarAJaeger: thanks :)18:06
*** wolverin_ has joined #openstack-infra18:06
*** lucasagomes is now known as lucas-afk18:07
*** wolverineav has quit IRC18:09
*** armaan has quit IRC18:11
*** ijw has joined #openstack-infra18:12
*** david-lyle has quit IRC18:12
*** ijw has quit IRC18:13
openstackgerritMerged openstack-infra/nodepool feature/zuulv3: Log request ID on request failure  https://review.openstack.org/53285718:14
*** gouthamr has joined #openstack-infra18:16
*** vsaienk0 has joined #openstack-infra18:17
*** tobiash has quit IRC18:18
*** tobiash has joined #openstack-infra18:18
fungioh, yes congrats to you too on being reelected, sdake. it's good to have board representatives with direct experience in the technical community18:21
*** panda|ruck has quit IRC18:23
*** panda has joined #openstack-infra18:25
*** vsaienk0 has quit IRC18:27
* fungi goes out to lunch, bbiaw18:27
openstackgerritDmitrii Shcherbakov proposed openstack-infra/project-config master: add charm-panko project-config  https://review.openstack.org/53276918:32
openstackgerritDmitrii Shcherbakov proposed openstack-infra/project-config master: add gerrit permissions for charm-panko  https://review.openstack.org/53325918:32
openstackgerritDmitrii Shcherbakov proposed openstack-infra/project-config master: configure GerritBot for charm-panko  https://review.openstack.org/53326018:32
openstackgerritDmitrii Shcherbakov proposed openstack-infra/project-config master: add charm-panko to zuul  https://review.openstack.org/53326118:32
AJaegerDmitrii-Sh: two changes - not four. please read the docs again.18:33
mordredAJaeger: thanks!18:33
mordredclarkb, fungi thanks to you too :)18:34
*** sdague has quit IRC18:34
openstackgerritAndreas Jaeger proposed openstack-infra/project-config master: Remove bindep-fallback from bindep  https://review.openstack.org/53153618:34
*** petrovich has quit IRC18:34
*** d34dh0r53 has quit IRC18:34
*** d34dh0r53 has joined #openstack-infra18:34
*** pcrews has joined #openstack-infra18:34
*** petrovich has joined #openstack-infra18:34
*** esberglu has joined #openstack-infra18:35
*** jpena is now known as jpena|off18:35
*** slaweq has joined #openstack-infra18:35
*** yee379 has joined #openstack-infra18:36
*** bandini has quit IRC18:36
*** rhagarty has quit IRC18:37
*** amotoki has quit IRC18:37
*** yee37932 has quit IRC18:38
*** bandini has joined #openstack-infra18:39
*** amotoki has joined #openstack-infra18:39
*** felipemonteiro__ has joined #openstack-infra18:40
Dmitrii-ShAJaeger: can options be specified in the first change (namely, "delay-release")?18:41
*** rhagarty has joined #openstack-infra18:43
*** slaweq_ has joined #openstack-infra18:43
*** felipemonteiro_ has quit IRC18:44
*** slaweq__ has joined #openstack-infra18:46
openstackgerritMerged openstack-infra/project-config master: Remove bindep-fallback from bindep  https://review.openstack.org/53153618:46
*** slaweq has quit IRC18:47
*** slaweq_ has quit IRC18:48
*** jkilpatr has quit IRC18:49
*** dtantsur has joined #openstack-infra18:50
*** sdague has joined #openstack-infra18:51
*** gyee has joined #openstack-infra18:52
*** rmcall has joined #openstack-infra18:54
AJaegerDmitrii-Sh: yes18:55
pabelangerlooks like ansible wheel is broken18:57
pabelangerworking on it now18:57
pabelangerI think because infracloud-vanilla controller is down18:58
*** felipemonteiro__ has quit IRC18:58
*** felipemonteiro_ has joined #openstack-infra18:58
pabelangerand it still exists in clouds.yaml18:58
AJaegerany config-core to review mordred's change to publish horizon and neutron to pypi, please? https://review.openstack.org/#/c/531822/18:59
*** rmcall has quit IRC18:59
AJaegermordred: did you see my comment on https://review.openstack.org/532304 ? I fear something is broken in our jobs...19:01
*** gouthamr has quit IRC19:01
*** jkilpatr has joined #openstack-infra19:02
openstackgerritPaul Belanger proposed openstack-infra/system-config master: Remove infracloud-vanilla from clouds.yaml  https://review.openstack.org/53327219:03
pabelangerinfra-root: ^ now that infracloud-vanilla controller is no more, we should remove it from clouds.yaml. I believe that is why our ansible wheel is no longer working and dynamic inventory is failing19:03
mordredAJaeger: yah - dmsimard and I were looking at that - it's REALLY weird - but let me look in to your suggestion real quick ...19:04
mordredpabelanger: remind me next week - we need to circle back around on how our dynamic inventory stuff is working19:04
mordredcorvus: when next you have a moment where you would find a context switch or distraction from cross-source welcome, I think dmsimard and I could use your eyes/brains on https://review.openstack.org/#/c/532304/19:06
openstackgerritDmitrii Shcherbakov proposed openstack-infra/project-config master: add charm-panko project-config  https://review.openstack.org/53327319:06
openstackgerritDmitrii Shcherbakov proposed openstack-infra/project-config master: add system-req template and jobs for charm-panko  https://review.openstack.org/53327419:06
*** electrofelix has quit IRC19:07
*** sree has joined #openstack-infra19:07
Dmitrii-ShAJaeger: ok, ta. I split it into two patchsets for meta/jobs now19:07
openstackgerritMerged openstack-infra/openstack-zuul-jobs master: Remove legacy bindep-ballback jobs  https://review.openstack.org/53153719:08
pabelangermordred: https://review.openstack.org/#/c/326441/1/modules/openstack_project/templates/puppetmaster/ansible-clouds.yaml.erb appears to be one option19:08
pabelangermordred: unsure if that works however19:08
*** sambetts is now known as sambetts|afk19:09
mordredpabelanger: it should19:12
cmurphyi'm expecting to see https://review.openstack.org/#/c/515215/ making its way through the gate but i don't see it in zuulv3.o.o, do i need more patience or is it time to recheck?19:12
*** sree has quit IRC19:12
*** dhill__ has joined #openstack-infra19:13
*** gyee has quit IRC19:16
pabelangermordred: okay, next week then. Seem we already set it: http://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/templates/puppetmaster/ansible-clouds.yaml.erb#n219:16
*** amoralej is now known as amoralej|off19:16
pabelangercmurphy: you might need a recheck, we lost queues today19:17
cmurphypabelanger: okay thanks19:17
openstackgerritAndreas Jaeger proposed openstack-infra/project-config master: Co-gate tempest-plugins with main repo  https://review.openstack.org/53327819:20
openstackgerritAndreas Jaeger proposed openstack-infra/project-config master: Co-gate tempest-plugins with main repo  https://review.openstack.org/53327819:21
*** trown|lunch is now known as trown19:22
*** katkapilatova1 has joined #openstack-infra19:22
AJaegerconfig-core, what do you think of the change above to co-gate X-tempest-plugin and X for all plugins? ^19:24
*** mriedem has quit IRC19:25
*** harlowja has joined #openstack-infra19:27
*** claudiub has quit IRC19:30
AJaegerconfig-core, could you review https://review.openstack.org/#/c/533132/ Adjust integrated gate for neutron changes - we currently run on neutron both legacy and native jobs, this change avoids this.19:31
*** mriedem has joined #openstack-infra19:32
*** david-lyle has joined #openstack-infra19:33
*** ijw has joined #openstack-infra19:33
ihrachysclarkb, hey!19:34
clarkbihrachys: hello19:34
ihrachysclarkb, I have a question on how to approach lib/neutron adoption in gate19:34
*** ijw has quit IRC19:34
ihrachysclarkb, we are at a point where we have all patches lined up to get lib/neutron used in integrated gate and pass neutron gate19:35
ihrachysso no q-* services19:35
*** ijw has joined #openstack-infra19:35
ihrachysthere are some notions of q-* in different jobs across different repos19:35
ihrachysI think they should still work after devstack-gate switch19:35
ihrachysbut I can't exactly guarantee :)19:35
clarkbihrachys: do you hvae any examples?19:36
ihrachyswe have some patches for other's jobs here: https://review.openstack.org/#/q/topic:new-neutron-devstack-in-gate+(status:open+OR+status:merged)19:36
clarkbihrachys: one place you'll want to patch too likely is os-loganalyze19:36
ihrachysclarkb, yeah. f.e. see this code in dragonflow: http://git.openstack.org/cgit/openstack/dragonflow/tree/devstack/devstackgaterc-common#n2019:37
ihrachysthey override services, including picking q-svc19:37
ihrachysso I think it should still work using lib/neutron-legacy code19:37
ihrachysbecause whenever a single service with name q-* is present, lib/neutron-legacy should kick in19:37
clarkbgotcha19:37
ihrachysbut that may fall apart in unknown ways19:38
ihrachyssadly I can't even cross-check with depends-on in some cases19:38
clarkbihrachys: so mostly you just need to update the devstack zuulv3 stuff and the devstack-gate stuff? and I guess send a warning to everyone about the switch19:38
clarkbihrachys: actually19:38
ihrachysbecause some code is in infra repos that are not in pipeline19:38
*** sree has joined #openstack-infra19:38
clarkbihrachys: maybe you only want to switch it in the devstack zuulv3 native stuff as that is all new too then its a natural switch19:39
clarkb?19:39
ihrachysclarkb, at this point we are merely at: https://review.openstack.org/#/c/436798/19:39
ihrachysclarkb, not sure what native means. we are switching default service names in devstack-gate that shouldn't care about how you call it.19:40
ihrachysclarkb, another example is http://git.openstack.org/cgit/openstack/python-openstackclient/tree/.zuul.yaml#n12519:41
ihrachyssee that it has q-qos that we could switch at any time but also q-metering19:41
clarkbihrachys: https://git.openstack.org/cgit/openstack-dev/devstack/tree/.zuul.yaml#n80 is replacing devstack-gate19:41
clarkbihrachys: so just thinking maybe we leave devstack-gate as is which will prevent breaking anyone still using devstack-gate and 1-*19:41
clarkb* q-*19:41
ihrachysclarkb, are matrices in devstack-gate not used for new jobs?19:41
clarkbthen as people switch to using the devstack thing above they get the new stuff19:41
clarkbihrachys: correct its now https://git.openstack.org/cgit/openstack-dev/devstack/tree/.zuul.yaml#n7719:42
clarkbihrachys: but that requires an explicit job conversion so only one or two have been done at this point I Think19:42
clarkbactually it may still be using the feature matrix I guess19:42
ihrachysand where's the code to configure e.g. dvr or vxlan...19:43
clarkbmordred: ^ do you know if it does?19:43
clarkbihrachys: in devstack19:43
ihrachysI mean, it doesn't seem like devstack-gate goes away19:43
ihrachysclarkb, what do you mean? where do we choose that this job is dvr and that one isn't19:43
*** sree has quit IRC19:43
clarkbhttps://git.openstack.org/cgit/openstack-dev/devstack/tree/.zuul.yaml#n50 you set your localrc values in the job and then devstack does it19:43
clarkbihrachys: ^ via the localrc content19:43
ihrachysok I see19:43
clarkbso rather than having a complicated devstack-gate script generate that stuff its in the job then the job invokes devstack19:44
ihrachyshonestly it was not my plan to spend even more time to get it in (and it looks like what you propose is piece meal approach), but I can adopt I guess19:45
clarkbit seems like it would be less work?19:45
ihrachysclarkb, not if we flip it with what we already have ready19:46
clarkbyou just have to make sure devstack's base job doesn't use q-* then as people switch to devstack job they just get neutron19:46
ihrachysand then I can clean up remaining q-* and kill lib/neutron19:46
clarkbihrachys: ya I guess my concern is that feature freeze and release are coming up fast and that has the potential to break a lot of people19:46
clarkbwhereas doing it the other way is more sfae. Could also just wait for after featurefreeze and release and push out the changes19:46
ihrachysnot sure what's better. we are on this task for 3+ cycles already at least with no end in sight, pushing for later. :)19:47
clarkbihrachys: I would definitely bring it up with the qa team19:47
*** sree has joined #openstack-infra19:47
clarkband the release team. I think the infra team will support whatever is decided19:48
mordredclarkb: do I know which thing?19:48
clarkbI'm happy to modify devstack-gate if thats what we want19:48
clarkbmordred: does the new zuulv3 native devstack stuff use the feature matrix in d-g at all?19:48
clarkbI don't think it does but it may19:48
fungimmm, thankfully fewer than 100 lines of scrollback while i was at the bar19:49
mordredclarkb: there is a role, test-matrix, that's in the devstack-gate repo19:49
mordredclarkb: I believe that's the v3 version of the features matrix stuff19:50
AJaegerclarkb, fungi, could either of you review mordred's change to publish horizon and neutron to pypi, please? https://review.openstack.org/#/c/531822/19:50
corvusmordred, clarkb: devstack native v3 job is in the devstack repo, not devstack-gate19:51
mordredclarkb: the native devstack job has a required-projects on devstack-gate and uses test-matrix in its pre playbook19:51
corvusoh, wow, that's.... interesting19:51
mordredgah.  not a required-projects -a roles:19:51
clarkbihrachys: ok I was wrong ^ d-g has to be updated in this case19:51
*** sree has quit IRC19:52
corvusmordred: so... i guess the devstack-gate repo will live on -- with one file?19:52
mordredclarkb: that said - it might be nice to orchestrate a move of that role and thematrix file into the devstack repo19:52
clarkbcorvus: ya its not what I expected19:52
clarkbmordred: I actually expected more of a job def would specify what projects to run19:52
clarkband have no features matrix at all19:52
clarkbbecause with job defs being per branch you get 99% of what the features matrix does for us by putting it in the jobs19:53
corvusclarkb: ++19:53
mordredclarkb: yah. I totally agree19:53
ihrachysclarkb, I am not savvy in infra ways. what does it mean in terms of the approach to take for my work?19:53
mordredI think there was *some* reason we needed to still support it though?19:53
clarkbihrachys: I think your current appraoch is correct19:53
clarkbihrachys: so now its just a matter of getting buy in from qa and release teams I think then we can merge changes19:53
ihrachysack!19:53
ihrachysI will craft a mail on Monday. I assume email is enough?19:54
clarkbihrachys: ya should be19:54
ihrachysgotcha. ok thanks for sticking with me! ;)19:54
clarkbcorvus: mordred https://review.openstack.org/#/c/436798/51 is the change in question19:54
ihrachysclarkb, btw os-loganalyze is already aware of neutron-*, I had this patched like a year ago :)19:55
clarkbihrachys: perfect19:56
mordredclarkb: so - where I think sometihng like features matrix would be nice in the context of the new devstack job (but obviously should just become an actual devstack thing) - is that I frequently have NO IDEA which all services I need to enable/disable to runa thing19:56
clarkbmordred: thats fair, but also probably something devstack should support directly rather than only in jobs19:57
mordredclarkb: ++19:57
mordredclarkb: TOTALLY19:57
mordredI think the approach of having the new job expose devstack in a way that's easy to consume is right on and is nice to work with19:57
*** katkapilatova1 has quit IRC19:59
corvusmordred: service dependencies are in progress19:59
corvuscould help here19:59
*** pcichy has quit IRC20:02
*** hasharELATER is now known as hashar20:03
*** harlowja has quit IRC20:04
openstackgerritPaul Belanger proposed openstack-infra/project-config master: Remove zuul-launchers from nodepool dashboard  https://review.openstack.org/53328720:04
openstackgerritMerged openstack-infra/system-config master: Remove infracloud-vanilla from clouds.yaml  https://review.openstack.org/53327220:06
pabelangerclarkb: okay, going to manually apply ^ to puppetmaster and see if dynamic inventory works20:07
*** sree has joined #openstack-infra20:08
AJaegerpabelanger, mordred, clarkb, fungi, could you help reviewien https://review.openstack.org/#/c/533132/ , https://review.openstack.org/#/c/531822/ , https://review.openstack.org/533278, please?20:10
*** yamahata has quit IRC20:10
*** sree has quit IRC20:13
*** ldnunes has quit IRC20:14
pabelangerAJaeger: I left a -1 on 533132 due to adding legacy jobs into project-templates, up into now it has only contained native zuulv3 jobs20:15
AJaegerpabelanger: ok, will move back...20:16
clarkbcorvus: thank you for the nodepool dib review20:17
openstackgerritAndreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Adjust integrated gate for neutron changes  https://review.openstack.org/53313220:17
AJaegerpabelanger: updated ^20:17
pabelangerAJaeger: +2 thanks20:18
AJaegerthanks for reviewing, pabelanger !20:18
openstackgerritMerged openstack-infra/project-config master: Publish neutron and horizon to pypi  https://review.openstack.org/53182220:20
EmilienMpabelanger: when could I get some reviews on my mirror patches?20:22
*** olaph1 has quit IRC20:22
EmilienMi've been asking for review for more than a week :(20:23
openstackgerritMerged openstack-infra/project-config master: Publish bifrost to PyPI  https://review.openstack.org/53182420:23
pabelangeryah, we need to set aside time to create AFS stuff, not a good week for that20:23
pabelangerhopefully next week20:23
EmilienMpabelanger: please ping me when you have time20:23
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: WIP: Support cross-source dependencies  https://review.openstack.org/53080620:23
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: WIP: Add cross-source tests  https://review.openstack.org/53269920:23
EmilienMpabelanger: no worries for this week20:24
EmilienMI'm just making sure it happens one day20:24
pabelangerokay, puppet is running again20:24
*** sree has joined #openstack-infra20:24
*** alexchadin has joined #openstack-infra20:25
pabelangerEmilienM: one day, I'm trying to see if dmsimard and frickler are interested in learning how to do it also20:25
EmilienMcool20:26
*** alexchadin has quit IRC20:26
*** alexchadin has joined #openstack-infra20:27
*** sree has quit IRC20:28
*** alexchadin has quit IRC20:35
openstackgerritMerged openstack-infra/grafyaml master: Add legend for graphs  https://review.openstack.org/53079920:41
*** katkapilatova1 has joined #openstack-infra20:42
openstackgerritsebastian marcet proposed openstack-infra/openstackid-resources master: Fix on attendee serializer  https://review.openstack.org/53329620:44
openstackgerritMerged openstack-infra/grafyaml master: Add support for influxdb datasource  https://review.openstack.org/30605020:45
*** yamahata has joined #openstack-infra20:45
corvusclarkb: you're welcome; hope it was helpful20:45
*** openstackstatus has quit IRC20:46
*** openstack has joined #openstack-infra20:48
*** ChanServ sets mode: +o openstack20:48
*** jascott1 has joined #openstack-infra20:52
pabelangerokay, infracloud chocolate is now disabled in nodepool20:56
openstackgerritMerged openstack-infra/openstackid-resources master: Fix on attendee serializer  https://review.openstack.org/53329620:56
AJaegerI just saw a MODULE failure on "Syncing to AFS" in the post publishing job - http://logs.openstack.org/62/62095ea403b9a4d6f6380e93cd2bd75ad1c1978e/post/publish-api-ref/f53d200/job-output.txt.gz#_2018-01-12_20_50_48_754402 . Any ideas?20:57
AJaegerand again here: http://logs.openstack.org/13/135a9d709d963960d34867775fb2227774c5b6bd/post/publish-openstack-sphinx-docs/ccb31d4/job-output.txt.gz#_2018-01-12_20_48_38_78836320:58
*** sree has joined #openstack-infra20:58
clarkbAJaeger: http://logs.openstack.org/62/62095ea403b9a4d6f6380e93cd2bd75ad1c1978e/post/publish-api-ref/f53d200/ara/result/4cabf239-e7b9-4818-81db-ec8779835efc/ no suc file or dir20:58
AJaegerthanks to ara!20:59
AJaegerso, we have a race ;(20:59
*** felipemonteiro has joined #openstack-infra21:00
pabelangerAJaeger: oh, are 2 jobs both publishing?21:00
AJaegerpabelanger: yes, but different docs. both nova21:01
AJaegerwait - one to developer.o, one to docs.o. Still, we had two nova changes in the post queue21:01
*** fried_rolls is now known as fried_rice21:01
pabelangerAJaeger: is using semaphore something we can do?21:01
AJaegerpabelanger: we could.21:02
*** sree has quit IRC21:02
*** felipemonteiro_ has quit IRC21:03
*** Jeffrey4l has quit IRC21:03
AJaegerquestion is whether we need do it for all access to docs.o.o?21:03
AJaegerfungi, thanks for reviewing21:03
clarkbso it is --delete-after doing it?21:04
openstackgerritMerged openstack-infra/openstack-zuul-jobs master: Adjust integrated gate for neutron changes  https://review.openstack.org/53313221:04
AJaegeranybody else to review review https://review.openstack.org/533278 to co-gate jobs, please?21:04
AJaegerclarkb: no idea21:04
AJaegerclarkb: yes, zuul-jobs/roles/upload-afs/library/zuul_afs.py uses delete-after21:05
clarkbI'm wondering if this is an issue outside of the job status. Do you know if the docs published properly in the end?21:06
openstackgerritDoug Hellmann proposed openstack-infra/python-storyboardclient master: update documentation build for new PTI  https://review.openstack.org/53330121:08
openstackgerritDoug Hellmann proposed openstack-infra/python-storyboardclient master: fix formatting of contributing file  https://review.openstack.org/53330221:08
AJaegerclarkb: I don't. Looking at the log file, I expect it did and just couldn't rename one file since the other job deleted it.21:08
clarkbAJaeger: ya thats what I'm thinking21:08
Roamer`hm, is there a problem with Cinder creating snapshots on LVM on some node(s) running tempest-full?  My new Nova libvirt driver failed tempest-full twice in the same way, and I don't really think it's the change itself... http://logs.openstack.org/33/140733/18/check/tempest-full/7478be5/controller/logs/screen-c-sch.txt.gz#_Jan_12_17_57_10_94788121:09
AJaegerclarkb: check the publish-sphinx:  Looking at the missed file, I think the later job overrode it, so all is fine...21:10
clarkbRoamer`: the new devstack job setup may not configure the block device for cinder properly? I think devstack makes a small one by default that we've bumped up in d-g?21:10
*** sree has joined #openstack-infra21:10
clarkbRoamer`: I think you want to read the cinder lvm setup in the devstack log to check on that21:11
AJaegerbut timing could have been different and a later change run before an earlier one. A risk we took since we assume we publish often...21:11
*** Jeffrey4l has joined #openstack-infra21:11
AJaegerlet'S leave it for now...21:11
Roamer`clarkb, sounds like something like that, there's an "Insufficient free space" warning just before the error; I'll take another look21:13
corvusAJaeger: after 3.0 is out, i'll work on that new pipeline manager for post to solve that21:13
clarkbRoamer`: if that does end up being the problem then bumping up the default size in devstack itself may be the best fix that way people at home can run tempest successfully too21:14
AJaegercorvus: thanks. So, let's ignore it for now...21:14
*** sree has quit IRC21:15
AJaegerconfig-core, if anybody has some review cycles, I've categorized some reviews on https://etherpad.openstack.org/p/Nvt3ovbn5x .21:17
* AJaeger signs off now and wishes everybody a great weekend21:17
*** spzala has quit IRC21:18
fungiclarkb: recipients of the devstack home game consolation prize?21:19
*** spzala has joined #openstack-infra21:19
fungihave a most excellent weekend yourself, AJaeger!21:19
*** dhill__ has quit IRC21:19
*** wolverin_ has quit IRC21:22
*** wolverineav has joined #openstack-infra21:22
*** jascott1 has quit IRC21:22
Roamer`clarkb, it does seem that it might be related to that; the devstack log shows it creating a 10250M volume (the default for devstack), then the warning says it has -10.6G free space, and devstack-vm-gate.sh sets it to 24G which does look like 10250M + 10.6G + a safety margin21:22
Roamer`and, yeah, devstack-vm-gate.sh does have commit 841ebc30 which bumps it to 24G and mentions "the tempest parallel run"21:23
*** spzala has quit IRC21:23
Roamer`clarkb, so you think I should submit a change to devstack to bump the default?  sure, I can do that21:24
Roamer`thanks!21:24
*** Jeffrey4l has quit IRC21:24
mtreinishman, I remember that patch. It's hard to believe that was over 4.5 years ago...21:24
*** spzala has joined #openstack-infra21:24
*** smatzek has quit IRC21:25
*** smatzek has joined #openstack-infra21:25
*** harlowja has joined #openstack-infra21:26
openstackgerritMerged openstack-infra/project-config master: Co-gate tempest-plugins with main repo  https://review.openstack.org/53327821:27
Roamer`ok, so I guess my question now is how do any Nova changes get merged?  Do people hit that, shake their heads, and do five-six rechecks until the stars alig... err, I mean, until tempest reorders its tests correctly?21:27
Roamer`sounds plausible, because I only hit that at the 18th iteration of the patch...21:28
Roamer`...but then twice in a row21:28
clarkbRoamer`: my guess is it depends on test ordering21:28
clarkbRoamer`: since you probably need 4 volume tests going at once to trip it21:29
clarkbor maybe slow cleanup21:29
clarkband yes patch to bump default in devstack21:29
*** rlandy has quit IRC21:29
fried_riceIf one of y'all has a mo, could you please help me understand why my patches appear umpteen times in the http://zuulv3.openstack.org/ dashboard?  I have 518982 521686 in the filter box and am seeing six instances of the former (one of which looks like it's really running) and 11 of the latter (likewise).21:29
mtreinishclarkb: the testing order should be completely deterministic since we ripped out the subunit pre seeding21:29
*** spzala has quit IRC21:30
mtreinishexcept for when new tests are added21:30
*** smatzek has quit IRC21:30
Roamer`hm, now y'all will have me digging into the test runs of the previous iterations to see whether they were reordered or the cleanup did indeed run faster...21:30
fried_riceoho, those counts happen to correspond to how many patches are in each series.  But patches on top (later in the series) shouldn't be affecting the bottom one running, should they?21:30
clarkbfried_rice: it is showing you that those changes are being tested together21:31
clarkbbut each series in check runs indepdently other than having the previous trees in its git repo21:32
clarkbso 518982 is tested but then 521189 is tested with 518982 in its tree21:32
fried_riceOkay, it makes more sense when I add 521189 to the filter and see that one of the weird ones is "connected" to it.21:33
*** jascott1 has joined #openstack-infra21:34
*** spzala has joined #openstack-infra21:34
fried_riceclarkb Thanks for the explanation.  Though I'm still not sure I fully understand.  The presence of that box with the grey circle and no individual jobs - that guy does *not* have to go away before the actual patch will go to the gate queue, right?21:35
*** edmondsw has quit IRC21:36
clarkbfried_rice: the whole series will go away at once when the check is reported21:36
clarkbfried_rice: but you will not go to the gate until your deps do21:36
clarkbbecause you can't merge change B before its parent change A21:36
fried_riceclarkb Right, that's the point - the one I care about is the one at the *bottom* of the series, the great granddaddy of them all.21:37
fried_riceAll these grey dots make it look like all his descendants have to pass (or at least finish?) before he gets to go to the gate.21:37
fried_riceWhich wouldn't make sense.21:37
clarkbyes that is true21:37
clarkbbecause parent always has to merge first21:37
*** Jeffrey4l has joined #openstack-infra21:37
clarkbor rather it is true that the parent change must be gateable first21:37
clarkbin the case of 521189: 518982 must go to the gate first21:38
openstackgerritDoug Hellmann proposed openstack-infra/storyboard master: update documentation build for new PTI  https://review.openstack.org/53330621:39
openstackgerritDoug Hellmann proposed openstack-infra/storyboard master: fix rst formatting  https://review.openstack.org/53330721:39
*** spzala has quit IRC21:39
*** wolverineav has quit IRC21:39
fried_riceCool, that's as it should be.  The grey dots are confusing me just because I don't know how to interpret this page :(21:39
fungiwoah! ci/cd gets its own track at the summit in vancouver? i hope you're all dusting off your old conference talks ;)21:40
clarkbfried_rice: all it is doing is showing you that the code being tested for 521189 includes the code from 51898221:40
fungii also encourage as many infra contributors as possible to inquire about becoming track chairs21:40
fried_riceclarkb Dig.  I interpreted "Dependent change required for testing" as "this guy has a dependency" whereas it's actually "this guy IS a dependency".21:40
Roamer`mtreinish, it seems to be that the order of the test cases within a single class is deterministic, but what tests run in parallel should also depend on how fast they complete, right?21:41
pabelangerfungi: ++21:41
fungiwe probably know better than most others in our community what makes for a good ci/cd tall21:41
fungitalk21:41
*** wolverineav has joined #openstack-infra21:41
corvusfungi: puppets21:41
fungiall teh puppetz21:41
Roamer`and yes, it does look like there were several volume tests running at the same time, at least in the last run21:41
Roamer`(the failed on)21:41
Roamer`(the failed one)21:41
clarkbRoamer`: yes tests run in parallel depends on how fast things go21:41
fungioh, s/track chairs/programming committees/ according to lsell's announcement21:42
clarkbeach process should run the same set of tests in the same order, but the overlap between the test processes will change depending on speed21:42
clarkbI've got top gear on in the background and now I'm imagining our test runs as jaguars going around a dirt track21:43
clarkber not top gear21:43
clarkbthe new thing21:43
Roamer`clarkb, so I guess I'll just recheck and hope for the best, and submit a devstack patch... thanks again!21:44
fungibetting on actual live jaguars?21:44
corvusclarkb: things always seem to go wrong at tricky bit 221:44
fungiclarkb: reminder... ci/cd track and programming committee suggestion in announcements portion of next weekly infra team meeting21:45
Roamer`okay, and now I have to go watch something else before going off to sleep, if only just to get the image of jaguars snapping at each other while their paws skid at the turns...21:46
clarkbfungi: thanks I'll put it on the agenda now so I don't forget21:46
fungiawesome21:47
corvuswow, the current version of the csd change fails only one test, but the failure is:21:48
corvustesttools.matchers._impl.MismatchError: '1,1 2,1 5,1 6,1 3,1 4,1 7,1' != '1,1 2,1 3,1 4,1 5,1 6,1 7,1'21:48
corvusit's way too friday for me to fix that.21:48
fungibring in the infra code talkers21:48
fungiMismatchError is running its own numbers station21:49
clarkbcorvus: for after you fix that there was a big that jlvillal found in depends on loop detection that I filed in storyboard tha tyou might want to look at with all that code paged in21:49
pabelangerokay, we're back up to max capacity again at inap, lets see if we have quota issues again21:51
*** andreww has quit IRC21:51
mgagnepabelanger: hello, checking21:51
corvusclarkb: if it's not super-critical, i may leave that for another time.21:51
mgagnesays 187 in_use21:51
*** sree has joined #openstack-infra21:52
*** r-daneel has joined #openstack-infra21:52
mgagnereached 19021:52
pabelangerI see 180 in CLI21:53
clarkbnote I've single core approved https://review.openstack.org/#/c/532364/1 as it only touches CI things (bindep.txt) and I want to be able to proprely review https://review.openstack.org/#/c/532359/21:53
pabelangerwonder if we might have leaked some instances21:53
clarkbfungi: ^ fyi git-review changes21:53
clarkbcorvus: it is probably not super critical. Basically a non cycle is detected as a cycle21:53
fungiclarkb: thanks for the heads up. i haven't been paying super close attention to git-review as it's been hard to sift through the changes we're likely to reject and attempt to find useful bug fixes21:54
clarkbcorvus: A <- B, B <- C, A <- C21:54
pabelangermgagne: okay, 179 according to nodepool and openetack client21:55
mgagnesays 18721:55
mgagneI think it includes nodes in building, deleting, ready and in-use states.21:56
*** sree has quit IRC21:56
*** spzala has joined #openstack-infra21:56
pabelangermgagne: so, does nova think there is more then actually online?21:57
mgagnepabelanger: only 179 online so I think there is something leaking somewhere21:58
pabelangerI see 180 now21:58
mgagneyea, more or less21:59
pabelangerokay, lets see if it gets back up to 190 again21:59
mgagneso nova quota usages is buggy21:59
mgagnewell, 180 means there are 10 free slots (max-servers: 190) but actual quota is 200.21:59
pabelangeryah22:00
mgagneso the ~10 leaked instances won't show until more are leaked in quota usage22:00
mgagnenow I'm wondering how it's happening..., less than 8h and we have out of sync quota usage again22:00
mgagneshows 197 now22:01
pabelangerokay, we look to be back at 189 now22:01
pabelangerlet me check something22:01
pabelangeryah, just got an exception22:01
pabelangershade.exc.OpenStackCloudHTTPError: (403) Client Error for url: https://compute.api.mtl01.cloud.iweb.com/v2/addf341f4fa44528b1ac73e2f5c68a21/servers Quota exceeded for instances: Requested 1, but already used 200 of 200 instances22:01
*** dprince has quit IRC22:01
fungi189 is according to server list, or limits?22:02
pabelangerfungi: nodepool list and openstack server list22:02
fungik22:02
pabelangerso, does sound like quota is buggy22:02
pabelangerin nova22:02
clarkbso nova thinks we have ~10 more instances than we are using? I want to say OVH had a similar problem at one time?22:03
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: WIP: Support cross-source dependencies  https://review.openstack.org/53080622:04
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: WIP: Add cross-source tests  https://review.openstack.org/53269922:04
pabelangerclarkb: yah22:04
pabelanger2018-01-12 22:03:59,958 DEBUG nodepool.driver.openstack.OpenStackNodeRequestHandler[nl01.openstack.org-30110-PoolWorker.inap-mtl01-main]: Predicted remaining tenant quota: {'compute': {'cores': inf, 'instances': 10, 'ram': inf}}22:04
pabelangeris what we are currently reporting in nodepool22:04
mtreinishRoamer`: no, the tests are partitioned in order to different workers based on the list upfront: https://github.com/testing-cabal/testrepository/blob/2173ae70261029b49e99208c1d6470ef9833c7a5/testrepository/testcommand.py#L382-L44522:04
*** spzala has quit IRC22:04
mtreinishRoamer`: since there is no timing data to effect the order things are sorted in the worker split will be deterministic based on test name22:05
*** kgiusti has left #openstack-infra22:05
*** spzala has joined #openstack-infra22:05
mgagneso I'm not sure how to address this. I know there is a Nova periodic task you can enable to "refresh" quota but I'm not sure how well (or not) it plays with cellsV1...22:06
mtreinishexecution inside of a worker will differ so you'll slight variations in which tests run at the same time, but the tests and the order they run in each worker should be the same (as long as the test class list remains the same)22:06
mgagnestill in our backlog22:06
*** slaweq__ has quit IRC22:06
mgagnehttp://www.dorm.org/blog/nova-max_age-and-until_refresh-settings-for-easing-quotas-pain/22:07
*** hashar has quit IRC22:07
clarkbmgagne: pabelanger trivial fix would be to bump our quota by 10 instances or drop our max-servers value by 1022:07
*** trown is now known as trown|outtypewww22:07
clarkbprobably not a great long term solution as it may go to 11 then 12 etc22:08
*** slaweq has joined #openstack-infra22:08
mgagneclarkb: until more is leaked in quota usage :-/22:08
mgagneyea22:08
mgagneI can reset quota now22:08
pabelangerI have to run for now, family. But I want to check nodepool logs when I return. I does seem that the 190th nodes wont come online before of quota mismatch22:09
mgagnedone22:09
mgagneshould be fine until next time...22:09
pabelangerk22:09
*** puiterwijk has quit IRC22:10
*** puiterwijk has joined #openstack-infra22:11
*** Jeffrey4l has quit IRC22:11
*** Jeffrey4l has joined #openstack-infra22:14
prometheanfiremhayden: vancouver cfp is open now22:14
mhaydenprometheanfire: oh fancy -- i am still finshing my proposal for the best dad jokes22:14
Roamer`mtreinish, yes, that makes sense; but say several different sets of volume-related tests are partitioned off to different workers and it just so happens that the workers get to them at the same time... that's still unreproducible and left to chance (and I don't think that there is a way to fix that, nor that it really needs to be fixed)22:16
*** rosmaita has quit IRC22:16
fungimhayden: prometheanfire: dad jokes will probably go over well in the shiny new "ci/cd" track!22:16
*** spzala has quit IRC22:17
* fungi expects anything to pair well with ci/cd22:17
openstackgerritMerged openstack-infra/git-review master: Add bindep.txt  https://review.openstack.org/53236422:20
*** felipemonteiro has quit IRC22:23
*** rosmaita has joined #openstack-infra22:23
*** felipemonteiro has joined #openstack-infra22:23
*** weshay is now known as weshay_PTO22:29
*** katkapilatova1 has quit IRC22:32
*** jkilpatr has joined #openstack-infra22:34
*** sree has joined #openstack-infra22:36
*** spzala has joined #openstack-infra22:37
*** sbra has quit IRC22:38
*** dave-mccowan has quit IRC22:39
*** sree has quit IRC22:40
*** spzala has quit IRC22:42
*** wolverineav has quit IRC22:42
*** felipemonteiro_ has joined #openstack-infra22:44
*** felipemonteiro has quit IRC22:48
*** slaweq has quit IRC22:49
corvusclarkb, fungi, mordred: can you review https://review.openstack.org/533006 ?22:53
*** mriedem has quit IRC22:54
*** bobh has quit IRC22:54
*** bobh has joined #openstack-infra22:55
clarkbcorvus: looking22:55
*** esberglu has quit IRC22:57
*** slaweq has joined #openstack-infra22:58
mordredcorvus: +A22:58
*** wolverineav has joined #openstack-infra22:58
*** bobh has quit IRC23:00
*** harlowja has quit IRC23:01
*** spzala has joined #openstack-infra23:01
*** yamamoto has quit IRC23:01
*** slaweq has quit IRC23:02
*** yamamoto has joined #openstack-infra23:04
*** sree has joined #openstack-infra23:05
*** yamamoto has quit IRC23:09
*** sree has quit IRC23:10
*** spzala has quit IRC23:10
openstackgerritBrian Rosmaita proposed openstack-infra/openstack-zuul-jobs master: Remove glance_store legacy jobs  https://review.openstack.org/52695723:10
*** owalsh has quit IRC23:12
*** owalsh has joined #openstack-infra23:12
*** spzala has joined #openstack-infra23:13
*** felipemonteiro_ has quit IRC23:17
*** felipemonteiro_ has joined #openstack-infra23:17
*** spzala has quit IRC23:20
openstackgerritMerged openstack-infra/system-config master: Add zuul mailing lists  https://review.openstack.org/53300623:20
*** wolverineav has quit IRC23:21
*** wolverineav has joined #openstack-infra23:22
*** spzala has joined #openstack-infra23:25
*** wolverineav has quit IRC23:26
*** spzala has quit IRC23:30
*** bobh has joined #openstack-infra23:34
corvuslooks like that just missed a cycle, so should happen next one23:34
*** fried_rice is now known as efried23:35
*** r-daneel has quit IRC23:35
*** spzala has joined #openstack-infra23:37
*** abelur_ has quit IRC23:37
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: WIP: Add cross-source tests  https://review.openstack.org/53269923:39
*** bobh has quit IRC23:39
*** sdague has quit IRC23:40
*** Apoorva_ has quit IRC23:42
prometheanfiremhayden: I'm always off by one in my irc...23:42
corvuslists.o.o has been updated to create the new site and new lists23:43
*** spzala has quit IRC23:46
*** hongbin has quit IRC23:53
*** spzala has joined #openstack-infra23:57
*** sree has joined #openstack-infra23:58

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!