Monday, 2019-12-16

*** tosky has quit IRC00:02
ianwthe kernel install scripts must handle this?  it's been a bit since i looked at those00:03
ianwiirc it's systemd00:04
clarkbgrub-mkconfig does it iirc00:05
ianwit's not driven by https://www.freedesktop.org/software/systemd/man/kernel-install.html ?00:05
clarkbit compiles the stuff in /etc/grub then writes it to the target destinatiob00:05
clarkbthe kernel install may trigger grub-mkinstall though?00:05
clarkbsomething has to update the grub entries00:06
ianw99-grub-mkconfig.install:        grub2-mkconfig --no-grubenv-update -o /boot/grub2/grub.cfg >& /dev/null00:08
ianwi think it is in the install.d files of kernel-install00:08
ianwhttps://bugzilla.redhat.com/show_bug.cgi?id=1737355#c6 is where i spent a long time debugging something similar-ish, but not for efi00:20
openstackbugzilla.redhat.com bug 1737355 in diskimage-builder "no vmlinuz or initramfs on kernel update" [High,On_qa] - Assigned to iwienand00:20
ianwcentos/rhel7 may be different again, though00:22
*** jamesmcarthur has joined #openstack-infra00:23
*** jamesmcarthur has quit IRC00:29
*** smarcet has joined #openstack-infra00:32
*** smarcet has joined #openstack-infra00:34
*** smarcet has left #openstack-infra00:34
ianw"The grub2-efi package installs a prebaked grubx64.efi on the EFI System partition, which looks for grub.cfg on the ESP in /EFI/fedora/ whereas the grub2-install command creates a custom grubx64.efi, deletes the original installed one, and looks for grub.cfg in /boot/grub2/."00:36
fungidebian/ubuntu update grub configuration as a hook triggered by kernel package installs00:46
fungii have a /boot/efi/EFI/debian/grubx64.efi00:47
*** rkukura has quit IRC00:48
ianwyea, but no grub.conf, right?00:49
fungiuses /boot/grub/grub.cfg00:49
fungiseparate partitions00:49
*** ricolin has joined #openstack-infra00:50
fungii have /boot/efi on /dev/mmcblk0p1 and /boot on /dev/mmcblk0p200:50
fungiand then /dev/mmcblk0p3 luks-encrypted as a pv for the vg which contains my volume for /00:51
clarkbianw: on debuntu you get both if using secyreboot00:51
clarkband both are used00:51
clarkbthe one in the ESP points to the one in /boot/grub00:51
clarkbfungi: if you cat the one in the efi dir you should have ~3 lines to confirm thqt00:51
fungii ran strings on it but it doesn't include any reference to a grub.cfg00:54
clarkbfungi its ascii should be openable00:55
funginope00:55
fungi/boot/efi/EFI/debian/grubx64.efi: PE32+ executable (EFI application) x86-64 (stripped to external PDB), for MS Windows00:55
clarkboh sorry not the .efi file the .cfg file00:55
clarkbI reaf that as a .cfg on the esp00:56
fungithere isn't one in my efi partition00:56
clarkbif you dont have that then the efi file should refer to the /boot00:56
clarkbit may not refer specifically to the cfg file but you'll have the boot partition uuid or other identifier in there00:56
fungiyeah, i didn't find any hd reference other than a templated string for substitution00:58
ianwi have  (,gpt2)/grub in there00:58
fungiahh, lookin00:58
fungiindeed, (,gpt2)/grub in mine too00:59
fungii guess that means same raw block device as the efi tree, but second partition00:59
*** jamesmcarthur has joined #openstack-infra01:04
ianwhrm, i'm not sure i'm any closer to understanding how to install it all though01:05
clarkbfungi: is sda2 where your /boot/grub lives?01:07
fungiwell /dev/mmcblk0p2 but close enough01:07
clarkbianw: I tested eith grub-mkconfig on an arm testnodeand that seemed to update the files01:07
clarkbfungi: ya partition two which maps to efi gpt2 scheme01:07
fungicorrect01:07
fungigrub-mkconfig (via a wrapper script) is what gets called in the debian package trigger for kernel updates too01:08
*** jamesmcarthur has quit IRC01:09
clarkband itself is anotger script that runs scripts in /etc/grub01:09
clarkbor /etc/grub.d01:09
openstackgerritwangfaxin proposed openstack/reviewday master: Update the community page.  https://review.opendev.org/69914101:35
*** jamesmcarthur has joined #openstack-infra01:35
openstackgerritIan Wienand proposed zuul/nodepool master: Dockerfile: add DEBUG environment flag  https://review.opendev.org/69484501:37
openstackgerritIan Wienand proposed zuul/nodepool master: Also build sibling container images  https://review.opendev.org/69739301:37
openstackgerritIan Wienand proposed zuul/nodepool master: Add container-with-siblings functional test  https://review.opendev.org/69346401:37
openstackgerritIan Wienand proposed zuul/nodepool master: Dockerfile: install nodepool-builder dependencies  https://review.opendev.org/69330601:37
openstackgerritIan Wienand proposed zuul/nodepool master: Add a container-with-releases functional test  https://review.opendev.org/69881801:37
openstackgerritIan Wienand proposed zuul/nodepool master: Functional tests - use common verification script  https://review.opendev.org/69883401:37
*** jamesmcarthur has quit IRC01:40
*** yamamoto has joined #openstack-infra01:54
*** yamamoto has quit IRC02:02
*** larainema has joined #openstack-infra02:04
*** yamamoto has joined #openstack-infra02:10
*** yamamoto has quit IRC02:16
*** hwoarang_ has quit IRC02:22
*** yamamoto has joined #openstack-infra02:33
*** jamesmcarthur has joined #openstack-infra02:36
*** jamesmcarthur_ has joined #openstack-infra02:46
*** jamesmcarthur has quit IRC02:46
*** lbragstad has joined #openstack-infra03:27
*** jamesmcarthur_ has quit IRC03:28
*** jamesmcarthur has joined #openstack-infra03:32
*** psachin has joined #openstack-infra03:34
*** jamesmcarthur has quit IRC03:37
*** yamamoto has quit IRC03:39
*** yamamoto has joined #openstack-infra03:42
*** iokiwi has quit IRC04:08
*** openstackgerrit has quit IRC04:08
*** dmellado has quit IRC04:08
*** diablo_rojo has quit IRC04:08
*** irclogbot_0 has quit IRC04:08
*** osmanlicilegi has quit IRC04:08
*** dmsimard has quit IRC04:08
*** strigazi has quit IRC04:08
*** ddurst has quit IRC04:08
*** bdodd has quit IRC04:08
*** sean-k-mooney has quit IRC04:08
*** stephenfin has quit IRC04:08
*** jlvillal has quit IRC04:08
*** Jeffrey4l has quit IRC04:08
*** tbarron has quit IRC04:08
*** brwyatt has quit IRC04:08
*** edausq has quit IRC04:08
*** ianw has quit IRC04:08
*** otherwiseguy_ has quit IRC04:08
*** aspiers has quit IRC04:08
*** gagehugo has quit IRC04:08
*** zer0c00l has quit IRC04:08
*** szaher has quit IRC04:08
*** larsks has quit IRC04:08
*** gothicmindfood has quit IRC04:08
*** jmccrory has quit IRC04:08
*** openstackstatus has quit IRC04:11
*** openstackstatus has joined #openstack-infra04:13
*** ChanServ sets mode: +v openstackstatus04:13
*** hwoarang has joined #openstack-infra04:20
*** dave-mccowan has quit IRC04:23
*** goldyfruit has joined #openstack-infra04:27
*** rkukura has joined #openstack-infra04:39
*** ykarel has joined #openstack-infra04:53
*** goldyfruit has quit IRC04:58
*** raukadah is now known as chkumar|rover05:22
*** surpatil has joined #openstack-infra05:33
*** yamamoto has quit IRC05:53
*** yamamoto has joined #openstack-infra05:54
*** yamamoto has quit IRC05:59
*** gregoryo has joined #openstack-infra05:59
*** ramishra has joined #openstack-infra06:04
*** kozhukalov has joined #openstack-infra06:24
*** pgaxatte has joined #openstack-infra06:31
*** yamamoto has joined #openstack-infra06:42
*** kozhukalov has quit IRC06:51
*** slaweq has joined #openstack-infra06:54
*** rcernin has quit IRC06:58
*** zzehring has quit IRC07:04
*** gregoryo has quit IRC07:07
*** slaweq has quit IRC07:15
*** pcaruana has joined #openstack-infra07:16
*** lpetrut has joined #openstack-infra07:28
*** ociuhandu has joined #openstack-infra07:32
*** slaweq has joined #openstack-infra07:36
*** ociuhandu has quit IRC07:36
*** ykarel is now known as ykarel|lunch07:37
*** yamamoto has quit IRC07:46
*** yamamoto has joined #openstack-infra07:46
*** tesseract has joined #openstack-infra07:48
*** ralonsoh has joined #openstack-infra07:49
*** yamamoto has quit IRC07:58
*** ociuhandu has joined #openstack-infra07:58
*** yamamoto has joined #openstack-infra07:58
*** ociuhandu has quit IRC08:00
*** ociuhandu has joined #openstack-infra08:01
*** tosky has joined #openstack-infra08:02
*** dchen has quit IRC08:04
*** tkajinam has quit IRC08:06
*** factor has quit IRC08:08
*** tesseract has quit IRC08:11
*** factor has joined #openstack-infra08:17
yoctozeptomorning08:20
yoctozeptoopenstackgerrit seems permagone08:21
*** ykarel|lunch is now known as ykarel|08:30
*** ykarel| is now known as ykarel08:30
*** rpittau|afk is now known as rpittau08:34
*** pkopec has joined #openstack-infra08:35
*** hashar has joined #openstack-infra08:37
*** ociuhandu_ has joined #openstack-infra08:40
*** sshnaidm|off is now known as sshnaidm08:41
*** yamamoto has quit IRC08:42
*** ociuhandu has quit IRC08:43
*** yamamoto has joined #openstack-infra08:45
amorinclarkb, johnsom, I am reading my name from few days ago, you were asking if we changed something on our side08:51
amorincan you let me know which region? We have both GRA1 and BHS1 for OSF08:51
amorinand I dont know what TGC is :p08:51
*** jpena|off is now known as jpena08:52
*** rcernin has joined #openstack-infra08:54
*** lucasagomes has joined #openstack-infra08:55
*** tesseract has joined #openstack-infra09:03
*** tesseract has quit IRC09:04
*** tesseract has joined #openstack-infra09:04
*** apetrich has joined #openstack-infra09:06
*** mugsie has quit IRC09:19
*** mugsie has joined #openstack-infra09:21
*** iurygregory has joined #openstack-infra09:26
*** yolanda has quit IRC09:28
*** yolanda__ has joined #openstack-infra09:28
*** rcernin has quit IRC09:36
*** rcernin has joined #openstack-infra09:36
*** derekh has joined #openstack-infra09:44
*** ociuhandu_ has quit IRC09:45
*** zer0c00l has joined #openstack-infra10:05
*** strigazi has joined #openstack-infra10:05
*** Jeffrey4l has joined #openstack-infra10:05
*** ddurst has joined #openstack-infra10:05
*** jmccrory has joined #openstack-infra10:05
*** edausq has joined #openstack-infra10:06
*** larsks has joined #openstack-infra10:06
*** otherwiseguy has joined #openstack-infra10:07
*** gagehugo has joined #openstack-infra10:07
*** diablo_rojo has joined #openstack-infra10:07
*** stephenfin has joined #openstack-infra10:07
*** osmanlicilegi has joined #openstack-infra10:07
*** dmellado has joined #openstack-infra10:07
*** ianw has joined #openstack-infra10:07
*** ociuhandu has joined #openstack-infra10:07
*** szaher has joined #openstack-infra10:07
*** brwyatt has joined #openstack-infra10:07
*** irclogbot_3 has joined #openstack-infra10:09
*** bdodd has joined #openstack-infra10:11
*** factor has quit IRC10:11
*** jaosorior has joined #openstack-infra10:16
*** rcernin has quit IRC10:16
*** aspiers has joined #openstack-infra10:17
*** gfidente has joined #openstack-infra10:38
*** yamamoto has quit IRC10:53
*** yamamoto has joined #openstack-infra10:53
*** yamamoto has quit IRC11:03
toskyuhm, I see fortnebula mirror issues11:21
donnydOn it11:21
toskythanks!11:22
*** rpittau is now known as rpittau|bbl11:22
donnydThere was a power blip over the weekend and it must not have came back on11:23
donnydThe mirror is on it's own ups and is usually unaffected by power issues11:25
*** ociuhandu has quit IRC11:25
*** ociuhandu has joined #openstack-infra11:26
yoctozeptoinfra-root, any change to bring back openstackgerrit? :-)11:28
yoctozeptochance* :-)11:28
donnydinfra-root can you ssh into the fortnebula mirror and see why the web server is not coming up11:28
*** SurajPatil has joined #openstack-infra11:34
*** surpatil has quit IRC11:36
*** surpatil has joined #openstack-infra11:40
*** surpatil has quit IRC11:40
*** tbarron has joined #openstack-infra11:41
*** yamamoto has joined #openstack-infra11:41
*** yamamoto has quit IRC11:41
*** yamamoto has joined #openstack-infra11:42
*** SurajPatil has quit IRC11:43
*** ociuhandu has quit IRC11:47
*** ociuhandu has joined #openstack-infra11:48
*** ociuhandu has quit IRC11:52
*** ociuhandu has joined #openstack-infra11:52
*** ociuhandu has quit IRC11:53
*** elod has quit IRC11:55
*** elod has joined #openstack-infra11:56
*** rfolco has joined #openstack-infra12:03
donnydstill working on the FN mirror - it appears to be an l3-agent issue. Its not bringing the router back up properly12:10
*** dmsimard has joined #openstack-infra12:12
*** ociuhandu has joined #openstack-infra12:15
*** ociuhandu has quit IRC12:15
*** ociuhandu has joined #openstack-infra12:16
*** ociuhandu has quit IRC12:17
*** ociuhandu has joined #openstack-infra12:18
*** ociuhandu has quit IRC12:18
*** ociuhandu has joined #openstack-infra12:19
donnydservice is restored to the fortnebula mirror12:20
*** yamamoto has quit IRC12:20
*** yamamoto has joined #openstack-infra12:21
*** ociuhandu has quit IRC12:21
*** ociuhandu has joined #openstack-infra12:22
*** ociuhandu has quit IRC12:23
*** pcaruana has quit IRC12:27
*** pcaruana has joined #openstack-infra12:33
*** jpena is now known as jpena|lunch12:39
*** yamamoto has quit IRC12:43
*** ociuhandu has joined #openstack-infra12:46
*** tobias-urdin has quit IRC12:49
*** ociuhandu has quit IRC12:51
*** dpawlik has joined #openstack-infra12:53
*** ociuhandu has joined #openstack-infra12:56
*** rlandy has joined #openstack-infra12:58
*** ociuhandu has quit IRC13:02
*** jamesmcarthur has joined #openstack-infra13:04
*** ykarel is now known as ykarel|afk13:05
*** ociuhandu has joined #openstack-infra13:07
*** Lucas_Gray has joined #openstack-infra13:08
*** goldyfruit has joined #openstack-infra13:09
*** rpittau|bbl is now known as rpittau13:09
*** ociuhandu has quit IRC13:14
*** rosmaita has joined #openstack-infra13:14
*** rh-jelabarre has joined #openstack-infra13:15
*** ykarel|afk has quit IRC13:15
*** jamesmcarthur has quit IRC13:15
*** ociuhandu has joined #openstack-infra13:15
*** ociuhandu has quit IRC13:16
*** tobias-urdin has joined #openstack-infra13:17
*** ociuhandu has joined #openstack-infra13:17
*** ociuhandu has quit IRC13:19
*** ociuhandu has joined #openstack-infra13:19
*** jpena|lunch is now known as pjena13:31
*** pjena is now known as jpena13:32
*** ociuhandu has quit IRC13:33
*** yamamoto has joined #openstack-infra13:34
*** yamamoto has quit IRC13:34
*** yamamoto has joined #openstack-infra13:34
*** goldyfruit has quit IRC13:37
*** ociuhandu has joined #openstack-infra13:43
*** Goneri has joined #openstack-infra13:44
*** ociuhandu has quit IRC13:47
*** ociuhandu has joined #openstack-infra13:48
johnsomamorin It was ubuntu-bionic-ovh-bhs1-0013345112. We just saw a job finish much faster than it usually does using the TCG (software virtualization) qemu driver. I was just curious if you had done something to speed up your nodepool instances or not.13:48
fungilooking into openstackgerrit's disappearance now13:49
fungi2019-12-16 04:08:57     <--     openstackgerrit (~openstack@review01.openstack.org) has quit (*.net *.split)13:52
fungilooks like it never recovered after that. it's still listening to gerrit events but doesn't realize it's not on a server13:53
fungirestarting now13:53
*** ociuhandu has quit IRC13:54
fungi#status restarted gerritbot after a 04:08z netsplit left it disconnected from freenode13:54
openstackstatusfungi: unknown command13:54
fungithanks for your quick work on the fn mirror, donnyd!13:55
fungil3 agent on the hypervisor host died/crashed?13:55
*** mriedem has joined #openstack-infra13:57
*** stephenfin has quit IRC13:59
*** stephenfin has joined #openstack-infra14:00
*** openstackgerrit has joined #openstack-infra14:01
openstackgerritMonty Taylor proposed opendev/system-config master: Use explicit image paths  https://review.opendev.org/69051114:01
openstackgerritMonty Taylor proposed opendev/system-config master: Update pip3 role to install from get-pip.py  https://review.opendev.org/69076614:01
openstackgerritMonty Taylor proposed opendev/system-config master: Add service playbook and test run for prod gerrit  https://review.opendev.org/69117114:01
*** psachin has quit IRC14:01
openstackgerritMonty Taylor proposed opendev/system-config master: Add launchpadlib credentials to gerrit ansible  https://review.opendev.org/69117214:01
openstackgerritMonty Taylor proposed opendev/system-config master: Add replication config to gerrit ansible  https://review.opendev.org/69117314:01
openstackgerritMonty Taylor proposed opendev/system-config master: Plumb through storyboard hiera data  https://review.opendev.org/69177714:01
openstackgerritMonty Taylor proposed opendev/system-config master: Clean up review comments  https://review.opendev.org/69200314:01
openstackgerritMonty Taylor proposed opendev/system-config master: Plumb through secure.config contents  https://review.opendev.org/69180014:01
*** haleyb has joined #openstack-infra14:01
*** stephenfin has quit IRC14:02
*** jaosorior has quit IRC14:02
*** stephenfin has joined #openstack-infra14:02
*** stephenfin has quit IRC14:04
*** ociuhandu has joined #openstack-infra14:05
*** stephenfin has joined #openstack-infra14:05
donnydfungi: something is going on with the latest round of patches for Stein.14:05
*** ykarel|afk has joined #openstack-infra14:06
donnydI have had very very few issues with the L3 agent over the years. Honestly for me it has been very reliable14:06
donnydBut something in the last update has made it not work so well14:06
*** Lucas_Gray has quit IRC14:08
donnydI opened a bug because something isn't right14:11
*** ociuhandu_ has joined #openstack-infra14:11
*** ociuhandu has quit IRC14:13
*** stephenfin has quit IRC14:13
*** stephenfin has joined #openstack-infra14:14
*** ociuhandu_ has quit IRC14:16
*** ociuhandu has joined #openstack-infra14:17
*** ociuhandu has quit IRC14:21
*** Lucas_Gray has joined #openstack-infra14:26
*** dkehn has joined #openstack-infra14:26
*** Lucas_Gray has quit IRC14:27
*** stephenfin has quit IRC14:28
*** stephenfin has joined #openstack-infra14:29
*** smarcet has joined #openstack-infra14:29
openstackgerritMonty Taylor proposed zuul/zuul master: Add --check-config option to zuul scheduler  https://review.opendev.org/54216014:30
*** ykarel|afk is now known as ykarel|away14:34
*** beekneemech is now known as bnemec14:42
*** onovy has quit IRC14:53
*** smarcet has quit IRC14:56
*** chkumar|rover is now known as ignoreirc14:57
*** ignoreirc is now known as chkumar|rover14:58
openstackgerritMerged zuul/nodepool master: Dockerfile: create APP_DIR  https://review.opendev.org/69364614:59
*** KeithMnemonic has joined #openstack-infra15:02
*** psachin has joined #openstack-infra15:06
openstackgerritMark Goddard proposed openstack/project-config master: Add publish-to-pypi template for kolla-cli  https://review.opendev.org/69922315:11
donnydfungi: well that wasn't too hard to get fixed up. FN should be back to 100% from the last update now15:14
fungigreat, thanks again!15:15
*** smarcet has joined #openstack-infra15:17
*** jgwentworth is now known as melwitt15:21
*** smarcet has quit IRC15:21
*** lpetrut has quit IRC15:22
*** goldyfruit has joined #openstack-infra15:23
*** ykarel|away has quit IRC15:23
*** larainema has quit IRC15:33
*** eharney has joined #openstack-infra15:37
*** onovy has joined #openstack-infra15:43
*** goldyfruit has quit IRC15:45
*** ykarel|away has joined #openstack-infra15:56
*** iurygregory has quit IRC15:57
AJaegerconfig-core, one tiny and one large grafana update for review, please: https://review.opendev.org/699001 and https://review.opendev.org/69899415:59
openstackgerritSlawek Kaplonski proposed openstack/project-config master: Update Neutron's Grafana dashboard  https://review.opendev.org/69826415:59
*** efried is now known as efried_pto16:00
AJaegerconfig-core, another grafana update: https://review.opendev.org/#/c/698264 and one small review, please: https://review.opendev.org/69922316:01
*** hashar has quit IRC16:05
*** hashar has joined #openstack-infra16:07
*** jamesmcarthur has joined #openstack-infra16:09
*** jaosorior has joined #openstack-infra16:10
*** chkumar|rover is now known as raukadah16:14
*** pgaxatte has quit IRC16:15
*** goldyfruit has joined #openstack-infra16:20
*** dpawlik has quit IRC16:24
*** gyee has joined #openstack-infra16:28
*** yamamoto has quit IRC16:28
*** yamamoto has joined #openstack-infra16:30
*** yamamoto has quit IRC16:35
*** ykarel|away has quit IRC16:38
clarkbAJaeger: I can take a look in a bit16:41
*** ykarel|away has joined #openstack-infra16:47
*** lucasagomes has quit IRC16:52
*** smarcet has joined #openstack-infra16:53
clarkbslaweq: AJaeger https://review.opendev.org/#/c/698264/2 will need to be rebased after haleyb's change merges. Sorry didn't catch that until approving haleyb's change17:00
*** ykarel|away has quit IRC17:01
*** ykarel|away has joined #openstack-infra17:02
openstackgerritColleen Murphy proposed opendev/irc-meetings master: Change keystone meeting time  https://review.opendev.org/69924617:04
AJaegerclarkb: thanks17:04
*** ricolin has quit IRC17:06
haleybslaweq: should have pinged you on my dashboard change, think you had already left for the weekend17:07
openstackgerritMerged openstack/project-config master: Update Neutron failure rate dashboard  https://review.opendev.org/69900117:09
openstackgerritMerged openstack/project-config master: Add publish-to-pypi template for kolla-cli  https://review.opendev.org/69922317:10
openstackgerritMerged openstack/project-config master: Update the Octavia failure rate dashboard  https://review.opendev.org/69899417:10
*** udesale has joined #openstack-infra17:14
*** hashar has quit IRC17:14
*** ociuhandu has joined #openstack-infra17:15
*** ykarel|away has quit IRC17:16
*** goldyfruit has quit IRC17:17
*** ociuhandu has quit IRC17:20
haleybslaweq: you want me to rebase your dashboard change?17:22
*** smarcet has quit IRC17:23
*** mattw4 has joined #openstack-infra17:26
clarkbslaweq: also tobiko-devstack-faults-centos-7 is failing after three job attempts because it runs devstack with python2.7 (because centos7) and nova requires python3.6 now17:26
clarkbslaweq: I don't know what the best solution to that problem is as I don't think devstack supports centos 8 yet, but wanted to point it out as it is something I've discovered via trackign of job retries17:27
*** lpetrut has joined #openstack-infra17:31
openstackgerritTobias Henkel proposed zuul/zuul master: Add support for smart reconfigurations  https://review.opendev.org/65211417:31
amorinjohnsom ok, we did nothing that I am aware of on the osf aggregate hosts17:32
amorinso I dont know for now,17:32
*** lpetrut has quit IRC17:32
amorinI will ask the whole team to be sure17:32
johnsomamorin Ok, maybe we just got luck and an empty host. lol17:32
amorinare the results still doing very fast or was it only a one time?17:32
*** lpetrut has joined #openstack-infra17:32
johnsomamorin Not a big deal, really just for curiosity sake.17:33
johnsomI'm not sure. I haven't dug into other runs to see which hosts they ran on.17:33
fungiokay, heading out to grab lunch, back soon17:33
amorinyes, maybe you were alone on the host, but that's weird, we are not doing so much oversubscription on your aggregate AFAIR17:33
amorinso even on non empty host, the results should not be significantly different17:34
amorinanyway, being alone on a host is definitely better than sharing :p17:34
johnsomIt was ~20 minutes faster than normal. Getting close to what we see with KVM, but I confirmed it was running the TCG driver.17:35
fungialso, as far as busy hosts and oversubscription, what we tend to see on a lot of our jobs is that they're more i/o-constrained than anything else. could be that i/o starvation (or lack thereof) is at work in those cases?17:35
fungianyway, really headed to lunch now17:36
openstackgerritMerged opendev/irc-meetings master: Change keystone meeting time  https://review.opendev.org/69924617:36
amorinfungi, johnsom yes, about IO congestion, I remember we already saw that by the past17:40
clarkbzuul_attempts shows that ironic jobs are still filling disks (well checking attempts cross referenced against our cleanup playbook output for those jobs)17:40
amorinI will check tomorrow, but maybe we added a host with nvme disks instead of SSD, which could improve the results17:40
clarkbI've passed that along to the ironic irc channel17:40
openstackgerritBrian Haley proposed openstack/project-config master: Update Neutron's Grafana dashboard  https://review.opendev.org/69826417:40
johnsomamorin Well, it was *good* so... Anyway, not worth a lot of effort. I was just curious if you had deployed some magic.17:41
amorinI know we have nvme in some aggregates, not sure about the foundation one17:41
*** smarcet has joined #openstack-infra17:41
amorinjohnsom, ok, dont worry, it's also interesting me if we can improve the results :p17:41
*** goldyfruit has joined #openstack-infra17:43
*** ramishra has quit IRC17:48
clarkbjohnsom: amorin it is theoretically possible that qemu improved too?17:48
amorinthat's an other option yes, I know we upgrade some part of libvirt / qemu in the last month17:49
amorinI cant determine from here, but I may check that tomorrow17:49
johnsomPossible, however none of the other providers have improved. This is why I was inquiring.17:52
*** goldyfruit has quit IRC17:58
*** derekh has quit IRC18:02
*** ijw has joined #openstack-infra18:07
*** jpena is now known as jpena|off18:08
*** ijw has quit IRC18:08
*** ijw has joined #openstack-infra18:08
*** psachin has quit IRC18:19
*** lpetrut has quit IRC18:19
clarkbemail sent on what I've found re flaky jobs and retries18:20
clarkbhopefully that gives people a bit more context on why RETRY_LIMITS are somethign to look at if consistent18:20
*** ralonsoh has quit IRC18:25
*** sshnaidm is now known as sshnaidm|afk18:26
*** ociuhandu has joined #openstack-infra18:30
AJaegerclarkb: want to review slaweq's change since haleyb rebased and fixed it, please? https://review.opendev.org/#/c/69826418:35
clarkbdone18:36
*** lpetrut has joined #openstack-infra18:36
*** smarcet has quit IRC18:36
*** smarcet has joined #openstack-infra18:39
openstackgerritMerged openstack/project-config master: Update Neutron's Grafana dashboard  https://review.opendev.org/69826418:45
*** rpittau is now known as rpittau|afk18:55
*** gfidente is now known as gfidente|afk18:57
*** smarcet has quit IRC18:59
*** tesseract has quit IRC19:06
*** smarcet has joined #openstack-infra19:08
*** rfolco has quit IRC19:09
*** rfolco has joined #openstack-infra19:10
*** lbragstad_ has joined #openstack-infra19:12
*** lbragstad has quit IRC19:14
openstackgerritMerged zuul/zuul master: Add support for smart reconfigurations  https://review.opendev.org/65211419:21
*** rlandy is now known as rlandy|brb19:22
*** auristor has quit IRC19:24
*** jamesmcarthur has quit IRC19:27
*** jamesmcarthur has joined #openstack-infra19:28
*** smarcet has quit IRC19:29
*** kozhukalov has joined #openstack-infra19:36
*** rlandy|brb is now known as rlandy19:45
*** auristor has joined #openstack-infra19:46
*** jamesmcarthur has quit IRC19:48
*** kozhukalov has quit IRC19:51
*** kozhukalov has joined #openstack-infra19:51
*** udesale has quit IRC19:52
*** Goneri has quit IRC19:53
*** jaosorior has quit IRC19:59
*** gyee has quit IRC19:59
*** eharney has quit IRC20:00
*** ijw has quit IRC20:07
*** dtroyer has joined #openstack-infra20:08
*** smarcet has joined #openstack-infra20:13
*** smarcet has joined #openstack-infra20:15
*** pkopec has quit IRC20:26
*** gyee has joined #openstack-infra20:40
*** slaweq has quit IRC20:49
*** jamesmcarthur has joined #openstack-infra20:49
ianwinfra-root:  https://review.opendev.org/691187 and https://review.opendev.org/690798 add a mirror test job to the new arm64 zuul queue (spoiler alert: it worked, just a bit slowly :)20:53
clarkbianw: neat I'll try to take a look after lunch20:53
ianwinfra-root: also https://review.opendev.org/660541 is a quick one to stop errors on bridge now we have ansible 2.9 there20:53
*** jamesmcarthur_ has joined #openstack-infra20:58
*** jamesmcarthur has quit IRC21:01
*** Goneri has joined #openstack-infra21:06
*** smarcet has quit IRC21:10
*** factor has joined #openstack-infra21:13
*** yamamoto has joined #openstack-infra21:18
*** rcernin has joined #openstack-infra21:21
*** yamamoto has quit IRC21:23
openstackgerritMerged openstack/diskimage-builder master: Add arm64 based functional test  https://review.opendev.org/67611121:31
*** ijw has joined #openstack-infra21:34
*** slaweq has joined #openstack-infra21:39
*** goldyfruit has joined #openstack-infra21:44
*** lpetrut has quit IRC21:48
openstackgerritLucas Xu proposed opendev/grafyaml master: Add Optional Query Paramter  https://review.opendev.org/69930121:48
openstackgerritLucas Xu proposed opendev/grafyaml master: Add Optional Query Paramter  https://review.opendev.org/69930121:49
*** pcaruana has quit IRC21:55
*** slaweq has quit IRC21:57
*** rfolco is now known as rfolco|bbl21:58
openstackgerritLucas Xu proposed opendev/grafyaml master: Add Optional Query Paramter  https://review.opendev.org/69930122:02
*** mattw4 has quit IRC22:04
*** mattw4 has joined #openstack-infra22:04
*** mriedem has quit IRC22:12
openstackgerritLucas Xu proposed opendev/grafyaml master: Add Optional Query Paramter  https://review.opendev.org/69930122:13
*** kozhukalov has quit IRC22:18
*** mattw4 has quit IRC22:28
*** mattw4 has joined #openstack-infra22:28
*** smarcet has joined #openstack-infra22:35
*** rcernin has quit IRC22:39
openstackgerritIan Wienand proposed opendev/system-config master: Add roles for a basic static server  https://review.opendev.org/69758722:39
*** rcernin has joined #openstack-infra22:39
*** iokiwi has joined #openstack-infra22:41
*** gfidente|afk has quit IRC22:48
*** rlandy is now known as rlandy|bbl22:51
*** smarcet has quit IRC22:54
*** ijw has quit IRC23:05
*** tkajinam has joined #openstack-infra23:07
*** rascasoft has quit IRC23:11
*** rascasoft has joined #openstack-infra23:13
*** jamesmcarthur_ has quit IRC23:26
*** smarcet has joined #openstack-infra23:30
*** rascasoft has quit IRC23:35
*** tosky has quit IRC23:43
*** ociuhandu has quit IRC23:43
clarkbcorvus: fungi ianw's changes above should be quick reviews if you have a moent23:45
corvusclarkb, ianw: why does install-ansible need to run on aarch64?  shouldn't that just run on bridge which is ubuntu?23:47
corvuser which is x8623:47
clarkbcorvus: the test ianw adds runs an arm64 bridge too because nodesets are provisioned in the same cloud region aiui23:47
clarkbcorvus: its the second change if you want to look at it23:48
clarkb(the linaro cloud can't provision x86 nodes)23:48
corvusoooh... it was the cloud region thing i was missing23:48
ianwyep, that :)  we discussed it when i tried mixing clouds and it didn't work :)23:48
corvusseems like a reasonable workaround.  i expect if we ever have a cloud that can do both, we would do so in order to have a more realistic test.23:49
corvuswe could also add this as a use-case for nodepool to support crossing providers23:49
corvus(we just never thought of that when we designed it)23:49
*** dchen has joined #openstack-infra23:50
corvuswe have a check-arm64 pipeline?  why is that?23:50
clarkbthe linaro cloud can only provide ~8 nodes and those nodes are quite a bit slower than x86 nodes we get elsewhere. Kolla in particular didn't want to wait for normal check results but still wants arm64 jobs to run23:52
clarkbthe idea is its like experimental but it always runs23:52
corvusack23:52
corvus+3 all around23:53
ianwyeah, it's useful for kolla, this, and i put a dib job in there too23:58

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!