Wednesday, 2023-08-02

« Tuesday, 2023-08-01 Index Thursday, 2023-08-03 »
opendevreviewJacob Anders proposed openstack/ironic master: [WIP] Wait for secure boot state change if it's not immediate  https://review.opendev.org/c/openstack/ironic/+/86399902:27
opendevreviewKe Niu proposed openstack/ironic-tempest-plugin master: Cleanup py27 support  https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/89028102:30
opendevreviewJacob Anders proposed openstack/ironic master: [WIP] Wait for secure boot state change if it's not immediate  https://review.opendev.org/c/openstack/ironic/+/86399904:03
opendevreviewJacob Anders proposed openstack/ironic master: [WIP] Wait for secure boot state change if it's not immediate  https://review.opendev.org/c/openstack/ironic/+/86399904:32
opendevreviewAdam Rozman proposed openstack/ironic-python-agent master: implement basic-auth support for user-image download process  https://review.opendev.org/c/openstack/ironic-python-agent/+/89027205:50
opendevreviewAdam Rozman proposed openstack/ironic-python-agent master: implement basic-auth support for user-image download process  https://review.opendev.org/c/openstack/ironic-python-agent/+/89027205:57
rpittaugood morning ironic! o/06:46
rpittauwell it looks like bifrost ci cs9 jobs are foobar08:17
dtantsurwhat a surprise!08:17
rpittau¯\_(ツ)_/¯08:18
rpittauseems something on the ipmi connection08:18
rpittauI doubt it's credentials, could be some obscure networking change in libvirt08:21
fricklerrpittau: https://bugs.launchpad.net/neutron/+bug/2029335 maybe? I seriously wonder why people still use such a proven unstable platform for CI08:34
dtantsurfrickler: you suggest alma/rocky?08:35
rpittaufrickler: thanks! 08:35
rpittauit does look similar to what we're seeing08:37
rpittaulol "Installed: systemd-container-252-16.el9.x86_64"08:38
rpittauI think that's it08:38
fricklerdtantsur: rocky at least seems much more stable and is supported by devstack08:38
rpittauI'll try with a downgrade08:39
dtantsurfrickler: the problem we used to have was: if we do hit a bug in stable RHEL, it will take weeks to months to get it to the images we use08:39
dtantsurStream fixes that (at the cost of hitting more bugs, sigh)08:39
rpittauI guess it will be fixed soon(TM), it's quite destructive08:40
opendevreviewlikui proposed openstack/ironic master: tox: Remove basepython  https://review.opendev.org/c/openstack/ironic/+/89031108:50
opendevreviewlikui proposed openstack/ironic master: tox: Remove basepython  https://review.opendev.org/c/openstack/ironic/+/89031108:57
opendevreviewlikui proposed openstack/ironic-inspector master: tox: Remove basepython  https://review.opendev.org/c/openstack/ironic-inspector/+/89031308:58
opendevreviewlikui proposed openstack/ironic-prometheus-exporter master: tox: Remove basepython  https://review.opendev.org/c/openstack/ironic-prometheus-exporter/+/89031408:59
opendevreviewlikui proposed openstack/ironic-python-agent master: tox: Remove basepython  https://review.opendev.org/c/openstack/ironic-python-agent/+/89031509:00
rpittaummm we do restart libvirtd, but maybe it's too early09:00
rpittauI'm a bit concerned about pinning systemd, let's see if I can find a different way to bypass that, maybe restarting libvirtd later 09:00
opendevreviewlikui proposed openstack/ironic-specs master: tox: Remove basepython  https://review.opendev.org/c/openstack/ironic-specs/+/89031609:00
opendevreviewlikui proposed openstack/python-ironicclient master: tox: Remove basepython  https://review.opendev.org/c/openstack/python-ironicclient/+/89031709:01
opendevreviewRiccardo Pittau proposed openstack/bifrost master: Exclude usage of bugged systemd-252-16.el9  https://review.opendev.org/c/openstack/bifrost/+/89031909:17
rpittaulet's see if this ^ works09:17
dtantsurRETRY_LIMIT \o/10:21
dtantsurrpittau: the syntax does not work10:23
dtantsurit's possible that bindep does not understand systemd versioning10:23
opendevreviewAdam Rozman proposed openstack/ironic-python-agent master: implement basic-auth support for user-image download process  https://review.opendev.org/c/openstack/ironic-python-agent/+/89027210:30
rpittau:(11:59
rpittaummmm ok12:00
rpittauah of course12:00
opendevreviewRiccardo Pittau proposed openstack/bifrost master: Exclude usage of bugged systemd-252-16.el9  https://review.opendev.org/c/openstack/bifrost/+/89031912:01
iurygregorygood morning Ironic12:15
dtantsurrpittau: bindep is still struggling: https://zuul.opendev.org/t/openstack/build/f3d87499a22b4ac28d7b8737ca6c2e4a :(12:47
rpittauok that's weird12:48
last_memberHi, maybe someone know how to fix that problem? https://storyboard.openstack.org/#!/story/2010473.   UEFI with software raid can run out of disk space when deploying ESP partition. The ESP partition size is defined in disk image builder as 550M https://opendev.org/openstack/diskimage-builder/src/branch/master/diskimage_builder/elements/block-device-efi/block-device-default.yaml#L10. The ESP partition size is also defined in IPA12:59
last_member as 550M https://github.com/openstack/ironic-python-agent/blob/a99bf274e4baec8e585bc9979e492bb8d85d17b5/ironic_python_agent/raid_utils.py#L30. Using dib to create an ubuntu 20.04 UEFI image the ESP partition is exactly 550M. The partition created by IPA when using software raid is (550MiB -  64KiB) presumably to account for an md superblock or other header in the  partition. When the ESP partition is copied from the image to th12:59
last_membere disk the following error occurs: cp /dev/md127p1 /dev/md0" fails with 'No space left on device' 12:59
dtantsurmmm, someone has complained about it, I don't remember what they did....13:00
dtantsurJayF do you by chance remember?13:00
dtantsur(the 550M - 64K part seems like a bug to me, I would expect us to create exactly 550M)13:02
last_memberI tried to create a smaller ESP partition for base image but it's corrupted grub.13:03
dtantsurI wonder if we can just bump it in IPA slightly (maybe it's better to understand where the 64K goes)13:05
dtantsurhttps://github.com/openstack/ironic-python-agent/blob/a99bf274e4baec8e585bc9979e492bb8d85d17b5/ironic_python_agent/raid_utils.py#L10513:05
dtantsurwe even add 1MiB, so it's not enough?13:05
dtantsurlast_member: could you shows us your resulting partition table?13:05
last_memberYes, one moment, need to have a one more run for deploy13:07
rpittaufantastic, locally bindep works just fine13:09
last_membercat mdstat13:11
last_memberPersonalities : [raid1] [multipath] 13:11
last_membermd1 : active raid1 nvme0n1p2[2] sdb2[1] sda2[0]13:11
last_member      563136 blocks super 1.0 [3/3] [UUU]13:11
last_member          resync=DELAYED13:11
last_member      13:11
last_membermd0 : active raid1 nvme0n1p1[2] sdb1[1] sda1[0]13:11
last_member      487689216 blocks super 1.2 [3/3] [UUU]13:11
last_member      [==>..................]  resync = 10.0% (49162496/487689216) finish=47.5min speed=153804K/sec13:11
last_member      bitmap: 4/4 pages [16KB], 65536KB chunk13:11
last_memberunused devices: <none>13:11
last_member(bifrost) root@bifrost:/opt/stack/bifrost/playbooks# cat mount 13:11
last_memberrootfs on / type rootfs (rw,size=65717024k,nr_inodes=16429256)13:11
last_memberproc on /proc type proc (rw,nosuid,nodev,noexec,relatime)13:11
last_membersysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)13:11
last_memberdevtmpfs on /dev type devtmpfs (rw,nosuid,noexec,size=4096k,nr_inodes=65536,mode=755)13:11
last_membersecurityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)13:11
last_membertmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)13:11
last_memberdevpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)13:11
last_membertmpfs on /run type tmpfs (rw,nosuid,nodev,size=26341372k,nr_inodes=819200,mode=755)13:11
last_membertmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)13:11
last_membercgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot)13:11
last_memberpstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)13:11
last_memberefivarfs on /sys/firmware/efi/efivars type efivarfs (rw,nosuid,nodev,noexec,relatime)13:11
last_membernone on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700)13:11
last_membersystemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=9709)13:11
last_memberhugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M)13:11
last_membermqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)13:11
last_memberdebugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime)13:11
last_membertracefs on /sys/kernel/tracing type tracefs (rw,nosuid,nodev,noexec,relatime)13:11
last_memberfusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime)13:11
last_memberconfigfs on /sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime)13:11
last_memberbinfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,nosuid,nodev,noexec,relatime)13:11
last_membertmpfs on /run/user/1000 type tmpfs (rw,nosuid,nodev,relatime,size=13170684k,nr_inodes=3292671,mode=700,uid=1000,gid=1000)13:11
last_member(bifrost) root@bifrost:/opt/stack/bifrost/playbooks# cat parted 13:11
rpittauspaaaam! \o/13:11
last_memberModel: ATA Samsung SSD 870 (scsi)13:11
last_memberDisk /dev/sda: 500GB13:11
last_memberSector size (logical/physical): 512B/512B13:11
last_memberPartition Table: gpt13:11
last_memberDisk Flags: 13:11
last_memberNumber  Start   End    Size   File system  Name           Flags13:11
last_member 2      1049kB  578MB  577MB  fat32        uefi-holder-0  boot, esp13:11
last_member 1      578MB   500GB  500GB               primary13:11
last_memberModel: ATA Samsung SSD 870 (scsi)13:11
last_memberDisk /dev/sdb: 500GB13:11
last_memberSector size (logical/physical): 512B/512B13:11
last_memberPartition Table: gpt13:11
last_memberDisk Flags: 13:11
last_memberNumber  Start   End    Size   File system  Name           Flags13:11
last_member 2      1049kB  578MB  577MB  fat32        uefi-holder-1  boot, esp13:11
last_member 1      578MB   500GB  500GB               primary13:11
last_memberModel: CT500P2SSD8 (nvme)13:11
last_memberDisk /dev/nvme0n1: 500GB13:11
last_memberSector size (logical/physical): 512B/512B13:11
last_memberPartition Table: gpt13:11
last_memberDisk Flags: 13:11
last_memberNumber  Start   End    Size   File system  Name           Flags13:11
last_member 2      1049kB  578MB  577MB  fat32        uefi-holder-2  boot, esp13:11
last_member 1      578MB   500GB  500GB               primary13:11
last_memberModel: Linux Software RAID Array (md)13:12
last_memberDisk /dev/md0: 499GB13:12
last_memberSector size (logical/physical): 512B/512B13:12
last_memberPartition Table: gpt13:12
last_memberDisk Flags: 13:12
last_memberNumber  Start   End     Size    File system  Name  Flags13:12
last_member 1      1049kB  578MB   577MB   fat32        ESP   boot, esp13:12
last_member 2      578MB   586MB   8389kB               BSP   bios_grub13:12
last_member 3      586MB   6375MB  5789MB  ext4         root13:12
last_member 4      499GB   499GB   68.1MB13:12
last_memberModel: Linux Software RAID Array (md)13:12
last_memberDisk /dev/md1: 577MB13:12
last_memberSector size (logical/physical): 512B/512B13:12
last_memberPartition Table: loop13:12
last_memberDisk Flags: 13:12
last_memberNumber  Start  End    Size   File system  Flags13:12
last_member 1      0.00B  577MB  577MB  fat3213:12
last_memberoh, sorry13:12
last_memberhttps://pastebin.com/MqXZ5MNV13:12
last_membersorry for that13:12
dtantsurMB-vs-MiB makes it confusing, but it looks like the partition size is actually correct?13:14
dtantsurCan you make it output in MiB somehow?13:14
dtantsur(and yes, please you paste services)13:14
dtantsurlast_member: ^^13:14
last_memberGot it, I'll try.13:21
last_memberYour idea that MiB != MB? 13:21
dtantsurwell, I'm just trying to figure out your actual layout13:22
dtantsurthe output in MB is not very helpful13:22
dtantsurbut the Start of 1049K means it's more than 1M, so maybe your idea is actually right13:22
dtantsurI wonder if we misunderstood what 'sgdisk -n X:A:+B' means13:23
last_memberOutput in Kb will be better?13:23
dtantsurKiB ideally (so powers of 2, not 10)13:23
last_memberhttps://opendev.org/openstack/diskimage-builder/src/branch/master/diskimage_builder/elements/block-device-efi/block-device-default.yaml#L10 this is my disk layot for base image (ubuntu focal)13:24
dtantsurWhat I'm trying to get is "1      1049kB  578MB   577MB   fat32        ESP   boot, esp" expressed in KiB or sectors, not in MB13:25
last_memberI'll take a break and will back with new results in KiB13:25
last_memberCurrent output from ironic deployment logs.13:25
last_memberNeed to do that manually13:26
last_memberThanks for the help. Because I've been disappointed that I cannot provision node with RAID13:28
dtantsurI'll need to leave soon, but the others may be able to help you.13:35
opendevreviewRiccardo Pittau proposed openstack/bifrost master: Pin systemd to 252-15 for CentOS Stream 9  https://review.opendev.org/c/openstack/bifrost/+/89031913:36
rpittauansible ftw ^ (hopefully)13:36
dtantsurrpittau: we cannot just exclude the wrong version, right? I'm worries, how long the repos will have the previous version.13:37
rpittauI think we can't exclude that version13:38
rpittaubuyt we can pin the precedent one13:38
rpittauin the meantime we wait patiently for a new one, that maybe is not broken13:38
rpittau252-16 was published 1 week ago13:38
rpittauno, less than a week13:39
dtantsurokay, let's hope it's kept long enough13:39
rpittauheh...13:40
opendevreviewAdam Rozman proposed openstack/ironic-python-agent master: implement basic-auth support for user-image download process  https://review.opendev.org/c/openstack/ironic-python-agent/+/89027213:48
JayFlast_member: I believe the answer we gave the last person was make something with a larger ESP13:48
JayFs/something/an image/13:48
JayFhttps://github.com/al45tair/netifaces/issues/78 :-( 13:58
JayFwe use this in inspection, too, I believe13:58
rpittaugood night! o/16:09
last_member@JayF use larger ESP for IPA image or for base image with OS?18:08
JayFlast_member: I am sorry, I don't remember specificially and can't find it right now.22:16
JayFlast_member: I will try to recall and look again in the morning22:17
« Tuesday, 2023-08-01 Index Thursday, 2023-08-03 »

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!