Wednesday, 2020-07-22

fungiyeah, we consider it a long-standing defect that we have to reenqueue things at all for a scheduler restart00:01
fungiand aim to make it essentially hitless00:01
fungi(excepting times when data structure changes for that state i suppose)00:01
dansmithcool00:01
*** Lucas_Gray has joined #openstack-infra00:12
*** ryohayakawa has joined #openstack-infra00:13
*** gyee has quit IRC00:20
*** hamalq has quit IRC00:27
*** tetsuro has joined #openstack-infra00:27
openstackgerritIan Wienand proposed openstack/project-config master: Add pypa/cryptography to zuul config  https://review.opendev.org/74232100:46
*** rlandy|ruck|bbl is now known as rlandy|ruck01:09
openstackgerritIan Wienand proposed openstack/openstack-zuul-jobs master: [dnm] pypa/cryptograhpy testing  https://review.opendev.org/74232301:16
openstackgerritIan Wienand proposed openstack/openstack-zuul-jobs master: [dnm] pypa/cryptograhpy testing  https://review.opendev.org/74232301:20
openstackgerritIan Wienand proposed openstack/openstack-zuul-jobs master: [dnm] pypa/cryptograhpy testing  https://review.opendev.org/74232301:22
*** hamalq has joined #openstack-infra01:30
*** hamalq_ has joined #openstack-infra01:31
*** hamalq has quit IRC01:36
openstackgerritIan Wienand proposed openstack/openstack-zuul-jobs master: [dnm] pypa/cryptograhpy testing  https://review.opendev.org/74232301:36
openstackgerritIan Wienand proposed openstack/project-config master: Add pyca/cryptography to zuul config  https://review.opendev.org/74232101:37
*** hamalq_ has quit IRC01:38
*** rlandy|ruck has quit IRC01:52
*** ricolin has joined #openstack-infra02:01
*** hamalq has joined #openstack-infra02:04
*** hamalq has quit IRC02:10
openstackgerritIan Wienand proposed openstack/openstack-zuul-jobs master: [dnm] pypa/cryptograhpy testing  https://review.opendev.org/74232302:17
*** tetsuro has quit IRC02:20
openstackgerritIan Wienand proposed openstack/openstack-zuul-jobs master: [dnm] pypa/cryptograhpy testing  https://review.opendev.org/74232302:31
*** dave-mccowan has quit IRC02:51
*** rfolco has quit IRC03:00
*** stevebaker has quit IRC03:16
openstackgerritIan Wienand proposed openstack/openstack-zuul-jobs master: [dnm] pypa/cryptograhpy testing  https://review.opendev.org/74232303:22
openstackgerritIan Wienand proposed openstack/openstack-zuul-jobs master: [dnm] pypa/cryptograhpy testing  https://review.opendev.org/74232303:26
*** psachin has joined #openstack-infra03:30
openstackgerritIan Wienand proposed openstack/project-config master: Add repo for pyca/cryptography Zuul jobs  https://review.opendev.org/74233703:36
openstackgerritIan Wienand proposed openstack/openstack-zuul-jobs master: [dnm] pypa/cryptograhpy testing  https://review.opendev.org/74232303:37
openstackgerritIan Wienand proposed openstack/openstack-zuul-jobs master: [dnm] pypa/cryptograhpy testing  https://review.opendev.org/74232303:42
*** redrobot has quit IRC03:42
*** stevebaker has joined #openstack-infra03:50
openstackgerritIan Wienand proposed openstack/project-config master: Add repo for pyca/cryptography Zuul jobs  https://review.opendev.org/74233703:53
*** ykarel has joined #openstack-infra04:30
*** Lucas_Gray has quit IRC04:36
*** hamalq has joined #openstack-infra04:43
*** hamalq_ has joined #openstack-infra04:44
*** hamalq has quit IRC04:48
*** stevebaker has quit IRC04:52
*** hamalq_ has quit IRC04:55
*** matt_kosut has joined #openstack-infra05:20
*** lmiccini has joined #openstack-infra05:22
openstackgerritIan Wienand proposed zuul/zuul-jobs master: Revert "Enable tls-proxy in ensure-devstack"  https://review.opendev.org/74234405:23
*** udesale has joined #openstack-infra05:27
*** ryohayakawa has quit IRC05:37
*** ryohayakawa has joined #openstack-infra05:38
*** marios has joined #openstack-infra05:39
*** vishalmanchanda has joined #openstack-infra05:55
*** ykarel_ has joined #openstack-infra05:59
*** ykarel has quit IRC06:02
*** eolivare has joined #openstack-infra06:18
*** dayou has quit IRC06:20
*** iurygregory has joined #openstack-infra06:25
*** dayou has joined #openstack-infra06:28
*** elod is now known as elod_off06:31
*** ykarel_ is now known as ykarel06:38
*** eharney has quit IRC06:40
*** eharney has joined #openstack-infra06:52
*** nightmare_unreal has joined #openstack-infra07:01
*** ociuhandu has quit IRC07:10
*** pkopec has joined #openstack-infra07:11
*** ralonsoh has joined #openstack-infra07:29
*** rcernin has quit IRC07:32
*** jcapitao has joined #openstack-infra07:35
*** xek has joined #openstack-infra07:36
*** ociuhandu has joined #openstack-infra07:36
*** tosky has joined #openstack-infra07:39
*** markvoelker has joined #openstack-infra07:47
*** marios has quit IRC07:48
openstackgerritMerged zuul/zuul-jobs master: Add upload-logs-s3  https://review.opendev.org/74180907:50
*** jpena|off is now known as jpena07:51
*** markvoelker has quit IRC07:52
*** ryohayakawa has quit IRC08:03
*** tosky has quit IRC08:04
*** tosky_ has joined #openstack-infra08:04
*** lucasagomes has joined #openstack-infra08:07
*** kaisers has joined #openstack-infra08:11
*** tosky_ is now known as tosky08:19
*** dchen has quit IRC08:20
*** tetsuro has joined #openstack-infra08:24
*** ryohayakawa has joined #openstack-infra08:28
*** ryohayakawa has quit IRC08:44
*** tetsuro has quit IRC08:48
*** marios has joined #openstack-infra08:50
*** ryohayakawa has joined #openstack-infra08:50
*** dtantsur|afk is now known as dtantsur08:52
*** rcernin has joined #openstack-infra09:10
*** xek has quit IRC09:19
*** ryohayakawa has quit IRC09:20
*** Wasaac has quit IRC09:22
*** rcernin has quit IRC09:35
*** udesale_ has joined #openstack-infra09:58
*** udesale has quit IRC10:01
*** Wasaac has joined #openstack-infra10:01
openstackgerritRico Lin proposed openstack/openstack-zuul-jobs master: Add job for openstack-python3-victoria-jobs-arm64  https://review.opendev.org/74209010:20
*** Wasaac has quit IRC10:25
*** Wasaac has joined #openstack-infra10:26
*** priteau has joined #openstack-infra10:33
*** Lucas_Gray has joined #openstack-infra10:41
*** ricolin has quit IRC10:48
*** tkajinam has quit IRC10:59
*** jcapitao is now known as jcapitao_lunch11:08
*** sshnaidm is now known as sshnaidm|afk11:14
*** Wasaac has quit IRC11:23
*** Wasaac has joined #openstack-infra11:23
*** jpena is now known as jpena|lunch11:34
*** Wasaac has quit IRC11:47
*** rcernin has joined #openstack-infra11:49
*** rfolco has joined #openstack-infra11:51
*** rlandy has joined #openstack-infra11:55
*** rlandy is now known as rlandy|ruck11:55
*** ociuhandu has quit IRC12:03
*** ociuhandu_ has joined #openstack-infra12:03
*** Wasaac has joined #openstack-infra12:03
*** dave-mccowan has joined #openstack-infra12:10
*** ykarel has quit IRC12:21
*** jcapitao_lunch is now known as jcapitao12:23
*** artom has quit IRC12:24
*** Goneri has joined #openstack-infra12:25
*** howell_ has joined #openstack-infra12:31
*** yankcrime has quit IRC12:32
*** jpena|lunch is now known as jpena12:37
*** howell has quit IRC12:38
*** rm_work has quit IRC12:38
*** howell_ is now known as howell12:38
*** ykarel has joined #openstack-infra12:40
*** mordred has quit IRC12:40
*** yankcrime has joined #openstack-infra12:41
*** rm_work has joined #openstack-infra12:42
*** mordred has joined #openstack-infra12:49
*** zxiiro has joined #openstack-infra13:21
*** artom has joined #openstack-infra13:21
*** xek has joined #openstack-infra13:26
*** __ministry1 has joined #openstack-infra13:28
openstackgerritMerged openstack/reviewstats master: Use unittest.mock instead of third party mock  https://review.opendev.org/71301913:29
*** sshnaidm|afk is now known as sshnaidm13:29
*** bnemec has joined #openstack-infra13:30
*** rcernin has quit IRC13:32
openstackgerritTristan Cacqueray proposed zuul/zuul-jobs master: dco-license: remove the empty nodeset  https://review.opendev.org/74243013:41
*** ricolin has joined #openstack-infra13:43
*** xek has quit IRC14:05
*** dave-mccowan has quit IRC14:07
*** __ministry1 has quit IRC14:07
*** yolanda has quit IRC14:10
*** yolanda has joined #openstack-infra14:12
*** dave-mccowan has joined #openstack-infra14:20
*** ramishra has quit IRC14:22
*** ramishra has joined #openstack-infra14:22
*** psachin has quit IRC14:24
*** dave-mccowan has quit IRC14:26
openstackgerritJames E. Blair proposed zuul/zuul-jobs master: Temporarily disable tag cleanup in docker promote  https://review.opendev.org/74244714:28
*** paladox has quit IRC14:32
*** Guest23136 has joined #openstack-infra14:47
openstackgerritTristan Cacqueray proposed zuul/zuul-jobs master: dco-license: remove the empty nodeset  https://review.opendev.org/74243014:47
*** xek has joined #openstack-infra14:48
*** Guest23136 is now known as redrobot14:49
*** paladox has joined #openstack-infra14:53
*** KeithMnemonic has joined #openstack-infra14:59
*** cmurphy_afk is now known as cmurphy15:07
openstackgerritMerged zuul/zuul-jobs master: Temporarily disable tag cleanup in docker promote  https://review.opendev.org/74244715:07
openstackgerritMerged zuul/zuul-jobs master: dco-license: remove the empty nodeset  https://review.opendev.org/74243015:23
*** rcernin has joined #openstack-infra15:25
*** lmiccini has quit IRC15:26
*** factor has joined #openstack-infra15:26
*** rcernin has quit IRC15:29
openstackgerritJames E. Blair proposed zuul/zuul-jobs master: Reinstate docker tag cleanup  https://review.opendev.org/74246115:40
*** gyee has joined #openstack-infra15:48
*** lucasagomes has quit IRC15:57
*** zbr|rover is now known as zbr15:58
*** factor has quit IRC16:08
*** factor has joined #openstack-infra16:09
*** ykarel is now known as ykarel|away16:11
*** marios is now known as marios|out16:12
*** marios|out has quit IRC16:13
*** __ministry1 has joined #openstack-infra16:16
*** ykarel|away has quit IRC16:17
openstackgerritSean McGinnis proposed openstack/hacking master: Add H211 to flag use of third party mock  https://review.opendev.org/74247516:17
*** xek has quit IRC16:17
*** ricolin has quit IRC16:19
*** hamalq has joined #openstack-infra16:22
*** hamalq_ has joined #openstack-infra16:23
*** Lucas_Gray has quit IRC16:24
*** hamalq has quit IRC16:27
*** Lucas_Gray has joined #openstack-infra16:31
*** dtantsur is now known as dtantsur|afk16:35
*** eolivare has quit IRC16:38
*** factor has quit IRC16:39
*** factor has joined #openstack-infra16:40
*** factor has quit IRC16:40
*** udesale_ has quit IRC16:47
*** lbragstad_ has joined #openstack-infra16:54
*** jcapitao has quit IRC16:57
*** lbragstad has quit IRC16:57
*** jpena is now known as jpena|off17:03
*** factor has joined #openstack-infra17:04
*** sshnaidm is now known as sshnaidm|afk17:09
*** ociuhandu has joined #openstack-infra17:11
*** ociuhandu_ has quit IRC17:14
*** ociuhandu has quit IRC17:16
*** ociuhandu has joined #openstack-infra17:22
*** nightmare_unreal has quit IRC17:22
*** Lucas_Gray has quit IRC17:23
*** sshnaidm|afk has quit IRC17:23
*** ociuhandu has quit IRC17:26
*** sshnaidm has joined #openstack-infra17:27
*** sshnaidm is now known as sshnaidm|afk17:27
*** lbragstad_ has quit IRC17:37
*** lbragstad_ has joined #openstack-infra17:38
*** __ministry1 has quit IRC17:42
*** lbragstad__ has joined #openstack-infra17:45
*** lbragstad_ has quit IRC17:47
*** rlandy|ruck is now known as rlandy17:56
*** _erlon_ has joined #openstack-infra18:02
*** icarusfactor has joined #openstack-infra18:07
*** icarusfactor has left #openstack-infra18:08
*** factor has quit IRC18:09
dansmithclarkb: fungi should logstash be working?18:29
dansmithI seem to get a spinny-forever thing no matter what I query for18:29
clarkbdansmith: its been sad because neutron logs were extremely large due to profiling of function calls in debug logs18:30
clarkbdansmith: I haven't had a chance to swing back around to that since reporting it to neutron, if they've fixed the issue we can restart everything and have it catch up from current state but the old state is likely not happy18:30
dansmithsad meaning slower than usable18:30
dansmith?18:30
clarkbsad meaning we shoved way too much data into it ya18:30
clarkbthings were OOMing and stuff18:31
dansmiththe default query shows results from today, but with log timestamps from Jul 1518:31
dansmithoh okay18:31
clarkbthere was a patch to address it in neutron /me looking for it now18:31
clarkbhttps://review.opendev.org/#/c/741540/ it has not merged yet18:32
clarkbbasically when that merges we can go clear out indexes to make space and restart services but until then we'll just get into the same spot18:32
clarkb(additionally we can exclude the neutron logs but since they were triyng to fix it I figured we'd wait on that)18:32
dansmithokay I thought I was doing something wrong18:34
openstackgerritSean McGinnis proposed openstack/hacking master: Fix H904 hacking_delayed_string_interpolation  https://review.opendev.org/74251118:44
*** ralonsoh has quit IRC18:45
*** xek has joined #openstack-infra18:49
dansmithclarkb: apologies if you're the wrong person, but.. if a tox job times out and zuul (or whatever) kills it.. are there any tricks for being able to examine the system state so I can figure out what is hung and where?18:51
dansmithI can't repro the hang here, and it's only intermittent in zuul, but I'm sure it's sucking enough to make things slow18:51
dansmithclarkb: like, if I C-c a tox job locally, I see all the threads die and dump stack, but we don't get that in zuul... I assume because it's kill -9ing or similar18:54
clarkbya I think the timeout may be somewhat forceful18:56
dansmithis that something baked into zuul or is there a kill command I could augment somewhere within my reach?18:57
clarkbits baked into zuul, it kills the ansible process which causes ansible to clean up19:01
dansmithokay19:01
clarkbdansmith: do you have a link to an example? there may be some clues in the logs19:01
dansmithmaybe I can try to start a thread from within the tests to watchdog it19:01
dansmithclarkb: https://4004d1b9c5e942ed3ec7-660b01bbd2fa84d0c67c5f821a133691.ssl.cf5.rackcdn.com/742314/6/check/openstack-tox-functional-py36/3813715/job-output.txt19:01
openstackgerritSean McGinnis proposed openstack/hacking master: Fix H903 hacking_no_cr  https://review.opendev.org/74251419:03
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/project-config master: Remove os_congress gating  https://review.opendev.org/74251519:03
*** bnemec has quit IRC19:03
*** stevebaker has joined #openstack-infra19:05
clarkbdansmith: in the past with testr we would copy the in flight .testrepository/tmpfile file which is where it recorded running progress. It doesn't look like we do that anymore (maybe an stestr change broke that?)19:05
dansmithoh, sneaky, but yeah I dunno19:05
clarkbdansmith: but I think that is where I'd try to look next, both stestr and testr write that content into a tmpfile in their .dir and we cna have log collection grab it. Then that should tell you what was in progress19:06
dansmithtmpfile is what, the temporary stdout or subunit stream?19:06
clarkbsubunit stream. and its not called tmpfile, its one of those randomly named tmpfiles19:06
clarkbdansmith: if you run the tests locally and do an ls on the .dir you should see it19:06
clarkbthen when it completes the testing it mv's it to the numbered iteration file19:06
clarkb0 1 2 3 etc19:06
dansmithokay19:06
* clarkb is looking at the zuul job side to see if we still try to grab that fiel19:07
clarkbya we do for tempest and neutron things19:08
clarkblet me see if I can figure out a chnage to do it for unittests19:08
dansmithclarkb: this is for glance by the way19:08
clarkbya in the functional job19:09
dansmithoh yeah have you noticed? :)19:09
dansmithokay yeah I see that temp file locally19:10
dansmithdoes that include "test started, test ended" events? I kinda assumed it would just be test completions19:10
openstackgerritmelanie witt proposed openstack/project-config master: Update ceph grafana for nova  https://review.opendev.org/74251619:11
openstackgerritSean McGinnis proposed openstack/hacking master: Fix H904 hacking_delayed_string_interpolation  https://review.opendev.org/74251119:12
clarkbdansmith: it should be both since it is hte raw subunit and subunit emits both19:15
dansmithokay19:15
clarkbI think I've found the bit of code htat needs updating and now just trying ot make sure I get things correct19:16
dansmithclarkb: thanks!19:16
dansmithclarkb: if there's any tmp files inside the testr dir, why not grab them always?19:16
dansmiththat just means we kill-9'd tester if they're there right?19:16
clarkbyup thats basically what I'm working on. A grab them if they are there step19:17
dansmithcool19:17
openstackgerritSean McGinnis proposed openstack/hacking master: Fix H903 hacking_no_cr  https://review.opendev.org/74251419:23
noonedeadpunkAJaeger: sorry I was reffering to https://docs.opendev.org/opendev/infra-manual/latest/drivers.html#retiring-a-project instead19:31
*** ociuhandu has joined #openstack-infra19:31
*** ociuhandu_ has joined #openstack-infra19:34
*** ociuhand_ has joined #openstack-infra19:35
AJaegernoonedeadpunk: the project-team-guide is a recent addition.19:35
noonedeadpunkyeah, following it isntead19:36
*** ociuhandu has quit IRC19:36
AJaegernoonedeadpunk: will review once all changes are up and linked.19:36
*** ociuhandu_ has quit IRC19:38
*** matt_kosut has quit IRC19:40
*** matt_kosut has joined #openstack-infra19:40
openstackgerritClark Boylan proposed zuul/zuul-jobs master: Collect partial subunit files  https://review.opendev.org/74252719:41
clarkbdansmith: ^ I think that will do it19:41
dansmithclarkb: okay so once that merges, I just wait for it to happen and I'll be able to snag that file?19:42
clarkbyup assuming I didn't screw it up too badly19:42
dansmithcool, thanks very much19:42
clarkbyou'll see tmpxyz_123 in the logs dir for hte job19:42
*** matt_kosut has quit IRC19:44
*** ociuhand_ has quit IRC19:45
noonedeadpunkAJaeger: oh... I just realized difference between retirement and deprecation..... right after I sand a ML...19:55
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/project-config master: Remove os_congress gating  https://review.opendev.org/74251520:01
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/project-config master: Remove os_congress gating  https://review.opendev.org/74251520:02
openstackgerritClark Boylan proposed zuul/zuul-jobs master: Collect partial subunit files  https://review.opendev.org/74252720:03
*** yolanda has quit IRC20:05
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/project-config master: Revert "Remove os_congress gating"  https://review.opendev.org/74253220:07
*** yolanda has joined #openstack-infra20:08
noonedeadpunkAJaeger: I think I posted all required changes... https://review.opendev.org/#/q/topic:osa/os_congress20:18
openstackgerritClark Boylan proposed zuul/zuul-jobs master: Collect partial subunit files  https://review.opendev.org/74252720:25
openstackgerritClark Boylan proposed zuul/zuul-jobs master: Disable workspace setup testing  https://review.opendev.org/74253720:25
*** hemna has quit IRC20:32
*** hemna has joined #openstack-infra20:39
*** matt_kosut has joined #openstack-infra20:40
openstackgerritClark Boylan proposed zuul/zuul-jobs master: Disable workspace setup testing and cleanup z-c testing  https://review.opendev.org/74253720:41
openstackgerritClark Boylan proposed zuul/zuul-jobs master: Collect partial subunit files  https://review.opendev.org/74252720:41
*** matt_kosut has quit IRC20:41
*** matt_kosut has joined #openstack-infra20:42
*** matt_kosut has quit IRC20:46
*** priteau has quit IRC20:47
smcginnisFor visibility - someone on the mailing list asking about ask.openstack.org being down.20:48
openstackgerritClark Boylan proposed zuul/zuul-jobs master: Disable base role testing that runs code on localhost  https://review.opendev.org/74253720:56
openstackgerritClark Boylan proposed zuul/zuul-jobs master: Collect partial subunit files  https://review.opendev.org/74252720:56
*** hamalq_ has quit IRC20:57
*** hamalq has joined #openstack-infra20:58
*** pkopec has quit IRC20:59
*** bnemec has joined #openstack-infra21:04
*** bnemec has quit IRC21:11
fungithanks smcginnis i'll see if i can spot the problem21:15
*** bnemec has joined #openstack-infra21:18
openstackgerritClark Boylan proposed zuul/zuul-jobs master: Disable base role testing that runs code on localhost  https://review.opendev.org/74253721:23
openstackgerritClark Boylan proposed zuul/zuul-jobs master: Collect partial subunit files  https://review.opendev.org/74252721:23
fungismcginnis: and it's fixed21:25
openstackgerritClark Boylan proposed zuul/zuul-jobs master: Disable base role testing that runs code on localhost  https://review.opendev.org/74253721:34
openstackgerritClark Boylan proposed zuul/zuul-jobs master: Collect partial subunit files  https://review.opendev.org/74252721:34
*** ociuhandu has joined #openstack-infra21:46
*** ociuhandu has quit IRC21:50
*** vishalmanchanda has quit IRC22:01
*** d34dh0r53 has quit IRC22:17
*** rlandy is now known as rlandy|bbl22:26
openstackgerritClark Boylan proposed zuul/zuul-jobs master: Disable base role testing that runs code on localhost  https://review.opendev.org/74253722:32
openstackgerritClark Boylan proposed zuul/zuul-jobs master: Collect partial subunit files  https://review.opendev.org/74252722:32
*** Lucas_Gray has joined #openstack-infra22:32
*** rcernin has joined #openstack-infra22:52
*** rcernin has quit IRC22:58
*** rcernin has joined #openstack-infra23:01
*** tosky has quit IRC23:03
*** rcernin has quit IRC23:04
*** rcernin has joined #openstack-infra23:05
*** Lucas_Gray has quit IRC23:09
*** yolanda has quit IRC23:23
*** yolanda has joined #openstack-infra23:24
*** markvoelker has joined #openstack-infra23:27
*** markvoelker has quit IRC23:32
*** dchen has joined #openstack-infra23:41
*** zxiiro has quit IRC23:45
*** ociuhandu has joined #openstack-infra23:47
*** ociuhandu has quit IRC23:52

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!