| sreejithp | hello, i am seeing an issue with zuul jobs vms for airship. its not have 100gb disk space. seems like partition expansion is not work. how can i check and confirm this? | 00:07 |
|---|---|---|
| clarkb | sreejithp: can you link to an example? we have noticed that happening in kna1 and added some debugging info around growroot to try and debug it further | 00:07 |
| clarkb | if you link to an example we can check that debugging info | 00:08 |
| sreejithp | yes the slave node was on kna1. this is the job link https://zuul.opendev.org/t/openstack/build/8f677202146b49fb95c0e760e150a304/log/job-output.txt | 00:08 |
| sreejithp | i think this is the hostname ubuntu-bionic-expanded-airship-kna1-0023204072 | 00:09 |
| clarkb | sreejithp: https://zuul.opendev.org/t/openstack/build/8f677202146b49fb95c0e760e150a304/log/job-output.txt#314-385 is the growroot logs for that job | 00:09 |
| clarkb | well specifically for the instance the job ran on | 00:09 |
| clarkb | 'resize2fs: Bad message While checking for on-line resizing support' is why growroot failed and the disk remained small | 00:10 |
| clarkb | fungi: ^ fyi we finally caught one | 00:10 |
| *** tosky has quit IRC | 00:12 | |
| clarkb | unfortunately that doesn't give us a ton of info. We likely need to escalate that with the citynetwork operators and have them cross check disk/filesystem stability for instance 755aefe9-6ff6-4a75-af60-bbc176f77c1d running on hypervisor 2e57327cd82629e0855870473a1f85e93148eeadd4288f239a458840 | 00:12 |
| sreejithp | sure, thanks clarkb | 00:14 |
| clarkb | sreejithp: is that something ya'll want to do from the airship side? I think historically you have done that as there is an ericsson intermediary? | 00:15 |
| clarkb | fungi: who was it that helped last time? | 00:15 |
| fungi | searching for that string turns this up from a decade ago: "https://sourceforge.net/p/e2fsprogs/bugs/293/" | 00:15 |
| fungi | clarkb: noonedeadpunk had contacts | 00:15 |
| clarkb | fungi: I don't see a bad message string in your link | 00:16 |
| fungi | interesting | 00:16 |
| fungi | yeah, i guess ddg scraped the bottom of the barrel and found "Device or resource busy While checking for on-line resizing support" | 00:17 |
| clarkb | sreejithp: when we brought this up with some nova maintainers previously the one thing they thought of was maybe the disk is encrypted and we're hitting bugs in that | 00:17 |
| clarkb | but it isn't clear to us if that is the case or not (we aren't requested encrypted disks) | 00:17 |
| fungi | noonedeadpunk said they're only just investigating/entertaining encrypted backing devices there, so definitely not in place yet | 00:18 |
| sreejithp | i am not sure about the configuration/setup for the node. i will reach out to people from my end | 00:18 |
| clarkb | fungi: it wasn't clear to me if that was vexxhost or citynetwork though | 00:19 |
| fungi | pretty sure he understood we were asking aboutcitynetwork | 00:21 |
| fungi | he's apparently involved in both though, yeah | 00:21 |
| clarkb | a manjaro forum post, https://forum.manjaro.org/t/failed-to-open-directory-bad-message/6168/5, indicates that could be the result of a corupt fs | 00:24 |
| fungi | i guess if the block device is misbehaving on boot that could happen | 00:24 |
| clarkb | I doubt it is our image's fs though since we're booting many nodes from the same image without trouble | 00:24 |
| clarkb | which makes me suspect something hypervisor side that is opaque to us | 00:25 |
| fungi | yeah, doubt it's image corruption or it would happen more, and would be likely to go away when we upload new images | 00:25 |
| clarkb | it would also affect other providers more | 00:25 |
| clarkb | whereas we've largely seen this happen in kna1 | 00:25 |
| fungi | unless it's corruption of the image copies in the nova image cache | 00:26 |
| clarkb | oh ya that is possible too I think | 00:26 |
| fungi | we've only seen it happen in kna1 afaik | 00:26 |
| clarkb | fungi: we've seen bad message happen on some osa jobs elsewhere but the growroot disk size thing seems to be kna1 specific | 00:26 |
| fungi | right | 00:27 |
| *** ianychoi__ is now known as ianychoi | 00:34 | |
| *** hamalq has quit IRC | 01:10 | |
| *** sorrison has quit IRC | 01:15 | |
| *** sorrison has joined #openstack-infra | 01:21 | |
| *** zzzeek has quit IRC | 01:26 | |
| *** zzzeek has joined #openstack-infra | 01:27 | |
| *** jhesketh_ is now known as jhesketh | 01:49 | |
| *** ramishra has joined #openstack-infra | 01:58 | |
| *** rcernin_ has joined #openstack-infra | 02:08 | |
| *** rcernin has quit IRC | 02:08 | |
| *** EmilienM has joined #openstack-infra | 02:46 | |
| *** sreejithp has quit IRC | 02:59 | |
| *** ricolin has joined #openstack-infra | 03:03 | |
| *** rcernin_ has quit IRC | 03:09 | |
| *** ricolin has quit IRC | 03:10 | |
| *** ricolin has joined #openstack-infra | 03:15 | |
| *** rcernin_ has joined #openstack-infra | 03:22 | |
| *** eharney has quit IRC | 03:28 | |
| *** rcernin_ has quit IRC | 03:32 | |
| *** rcernin has joined #openstack-infra | 03:32 | |
| *** zbr has joined #openstack-infra | 03:50 | |
| *** dmsimard has quit IRC | 04:08 | |
| *** dmsimard has joined #openstack-infra | 04:09 | |
| *** psachin has joined #openstack-infra | 04:28 | |
| *** ociuhandu has joined #openstack-infra | 04:30 | |
| *** ociuhandu has quit IRC | 04:35 | |
| *** rcernin has quit IRC | 06:58 | |
| *** rcernin_ has joined #openstack-infra | 06:58 | |
| *** lpetrut has joined #openstack-infra | 07:29 | |
| *** dklyle has quit IRC | 07:32 | |
| *** rcernin_ has quit IRC | 07:41 | |
| *** ysirndjuro has quit IRC | 07:42 | |
| *** icey_ is now known as icey | 07:43 | |
| *** elod has quit IRC | 07:45 | |
| *** elod has joined #openstack-infra | 07:49 | |
| *** slaweq_ has joined #openstack-infra | 07:50 | |
| *** dchen has quit IRC | 08:03 | |
| *** amoralej is now known as amoralej|off | 08:09 | |
| *** slaweq_ has quit IRC | 08:22 | |
| *** andrewbonney has joined #openstack-infra | 08:24 | |
| *** yoctozepto5 has joined #openstack-infra | 08:26 | |
| *** yoctozepto has quit IRC | 08:28 | |
| *** yoctozepto5 is now known as yoctozepto | 08:28 | |
| *** yoctozepto6 has joined #openstack-infra | 08:32 | |
| *** yoctozepto has quit IRC | 08:33 | |
| *** yoctozepto6 is now known as yoctozepto | 08:33 | |
| *** gfidente has joined #openstack-infra | 09:08 | |
| *** gagehugo has quit IRC | 09:37 | |
| *** coreycb has quit IRC | 09:37 | |
| *** gagehugo has joined #openstack-infra | 09:37 | |
| *** coreycb has joined #openstack-infra | 09:38 | |
| *** ricolin_ has joined #openstack-infra | 09:57 | |
| *** mugsie_ is now known as mugsie | 10:16 | |
| *** yoctozepto8 has joined #openstack-infra | 10:47 | |
| *** yoctozepto has quit IRC | 10:47 | |
| *** yoctozepto8 is now known as yoctozepto | 10:47 | |
| *** yoctozepto1 has joined #openstack-infra | 10:54 | |
| *** yoctozepto has quit IRC | 10:55 | |
| *** yoctozepto1 is now known as yoctozepto | 10:55 | |
| *** iurygregory has quit IRC | 11:13 | |
| *** gfidente has quit IRC | 11:15 | |
| *** tosky has joined #openstack-infra | 11:25 | |
| *** gfidente has joined #openstack-infra | 11:30 | |
| *** yoctozepto2 has joined #openstack-infra | 11:32 | |
| *** yoctozepto has quit IRC | 11:33 | |
| *** yoctozepto2 is now known as yoctozepto | 11:33 | |
| *** ricolin_ has quit IRC | 12:10 | |
| *** ociuhandu has joined #openstack-infra | 12:17 | |
| *** yoctozepto4 has joined #openstack-infra | 12:45 | |
| *** yoctozepto has quit IRC | 12:46 | |
| *** yoctozepto4 is now known as yoctozepto | 12:46 | |
| *** slaweq_ has joined #openstack-infra | 12:49 | |
| *** psachin has quit IRC | 12:58 | |
| *** zbr9 has joined #openstack-infra | 13:20 | |
| *** zigo has joined #openstack-infra | 13:21 | |
| *** zbr has quit IRC | 13:22 | |
| *** zbr9 is now known as zbr | 13:22 | |
| *** slaweq_ has quit IRC | 13:31 | |
| *** stephenfin has joined #openstack-infra | 13:38 | |
| *** zbr0 has joined #openstack-infra | 13:43 | |
| *** zbr has quit IRC | 13:46 | |
| *** zbr0 is now known as zbr | 13:46 | |
| *** iurygregory has joined #openstack-infra | 13:49 | |
| *** ociuhandu has quit IRC | 13:53 | |
| *** ociuhandu has joined #openstack-infra | 13:54 | |
| *** ociuhandu has quit IRC | 13:59 | |
| *** zbr3 has joined #openstack-infra | 14:03 | |
| *** ociuhandu has joined #openstack-infra | 14:04 | |
| *** zbr has quit IRC | 14:05 | |
| *** zbr3 is now known as zbr | 14:05 | |
| *** sreejithp has joined #openstack-infra | 14:06 | |
| *** zbr7 has joined #openstack-infra | 14:08 | |
| *** zbr has quit IRC | 14:10 | |
| *** zbr7 is now known as zbr | 14:10 | |
| *** zbr4 has joined #openstack-infra | 14:33 | |
| *** zbr has quit IRC | 14:35 | |
| *** zbr4 is now known as zbr | 14:35 | |
| *** ociuhandu has quit IRC | 14:49 | |
| *** zbr4 has joined #openstack-infra | 14:52 | |
| *** zbr has quit IRC | 14:53 | |
| *** zbr4 is now known as zbr | 14:53 | |
| *** zbr5 has joined #openstack-infra | 15:12 | |
| *** zbr has quit IRC | 15:14 | |
| *** zbr5 is now known as zbr | 15:14 | |
| *** lpetrut has quit IRC | 15:15 | |
| *** tosky has quit IRC | 15:19 | |
| *** tosky has joined #openstack-infra | 15:20 | |
| *** zbr8 has joined #openstack-infra | 15:30 | |
| *** zbr has quit IRC | 15:33 | |
| *** zbr8 is now known as zbr | 15:33 | |
| *** dklyle has joined #openstack-infra | 15:51 | |
| *** lpetrut has joined #openstack-infra | 16:05 | |
| *** ociuhandu has joined #openstack-infra | 16:07 | |
| *** zbr2 has joined #openstack-infra | 16:11 | |
| *** zbr has quit IRC | 16:13 | |
| *** zbr2 is now known as zbr | 16:13 | |
| *** zbr6 has joined #openstack-infra | 16:15 | |
| *** zbr has quit IRC | 16:17 | |
| *** zbr6 is now known as zbr | 16:17 | |
| *** calbers has quit IRC | 16:32 | |
| *** icey has quit IRC | 16:33 | |
| *** icey has joined #openstack-infra | 16:34 | |
| *** calbers has joined #openstack-infra | 16:35 | |
| *** zbr has quit IRC | 16:38 | |
| *** gyee has joined #openstack-infra | 16:40 | |
| *** harlowja has joined #openstack-infra | 16:41 | |
| *** hemna has quit IRC | 16:47 | |
| *** hemna has joined #openstack-infra | 16:49 | |
| *** harlowja has quit IRC | 16:59 | |
| *** slaweq_ has joined #openstack-infra | 17:08 | |
| *** ociuhandu_ has joined #openstack-infra | 17:09 | |
| *** ociuhandu has quit IRC | 17:13 | |
| *** ociuhandu_ has quit IRC | 17:14 | |
| *** hamalq has joined #openstack-infra | 17:16 | |
| *** ociuhandu has joined #openstack-infra | 17:23 | |
| *** gfidente has quit IRC | 17:23 | |
| *** ociuhandu has quit IRC | 17:28 | |
| *** __ministry1 has joined #openstack-infra | 17:34 | |
| *** arne_wiebalck has quit IRC | 17:35 | |
| *** arne_wiebalck has joined #openstack-infra | 17:36 | |
| openstackgerrit | Jeremy Stanley proposed openstack/project-config master: Move git-review to opendev tenant https://review.opendev.org/c/openstack/project-config/+/777774 | 17:36 |
| *** lpetrut has quit IRC | 17:40 | |
| *** slaweq_ has quit IRC | 17:54 | |
| *** andrewbonney has quit IRC | 18:15 | |
| *** __ministry1 has quit IRC | 18:29 | |
| *** slaweq_ has joined #openstack-infra | 18:33 | |
| *** slaweq_ has quit IRC | 18:44 | |
| openstackgerrit | Merged openstack/project-config master: Move git-review to opendev tenant https://review.opendev.org/c/openstack/project-config/+/777774 | 18:50 |
| *** sreejithp has quit IRC | 19:16 | |
| *** slaweq_ has joined #openstack-infra | 19:44 | |
| *** tjgresha has joined #openstack-infra | 19:50 | |
| *** slaweq_ has quit IRC | 20:22 | |
| *** ociuhandu has joined #openstack-infra | 20:58 | |
| *** ociuhandu has quit IRC | 21:02 | |
| tjgresha | any special permissions needed for a 3rdparty ci account to be able to comment on project - say Nova or Neutron? | 21:33 |
| tjgresha | gerrit permissions that is | 21:37 |
| clarkb | tjgresha: yes for nova your third party ci account must be in the nova-ci group | 21:45 |
| clarkb | tjgresha: for neutron it is the neutron-ci group and so on | 21:45 |
| clarkb | those groups are managed by the projects themselves | 21:45 |
| tjgresha | meaning i should hit up corresponding IRC channels for access? | 21:46 |
| clarkb | yes, it will be up to the respective projects to decide if they want you to vote +/-1 or not (all third party ci can leave comments iwthout vote values) | 21:47 |
| tjgresha | ok, Thanks clarkb | 21:48 |
| *** tjgresha has quit IRC | 22:17 | |
| *** sreejithp has joined #openstack-infra | 22:21 | |
| *** hamalq has quit IRC | 23:40 | |
| *** zzzeek has quit IRC | 23:41 | |
| *** zzzeek has joined #openstack-infra | 23:43 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!