Disks
Harddisks
1. Firmware 2. ATA, SCSI Driver 3. Physical Block Devices 4. MBR, UEFI GPT Partitions 5. LVM2 Physcical Volume (disk or partition) 7. Volume Group, spanning physical volumes 8. Logical Volumes 9. Device Mapper 10.Filesystem
Harddisk
ls -al /dev/disk/by* blkid
/dev/xvda2: UUID="2ab116ed-fdae-4ee6-aafd-f2febf9a4a96" TYPE="ext4" PARTUUID="3278fa68-67cf-476f-88e1-fa858160c1c5" /dev/xvda3: UUID="9O6sva-rWF6-DO3I-GKIz-iuMg-qrQo-y13KV4" TYPE="LVM2_member" PARTUUID="478fca97-0ca6-4597-aaa6-e709947d1868" /dev/xvdb: UUID="0d331770-95d3-4d15-839a-cce7ddcb6741" TYPE="ext4" /dev/mapper/ubuntu--vg-ubuntu--lv: UUID="597d1543-fe70-4f53-9d82-7fb0c2249ac7" TYPE="ext4"
Disks may have partitions or raw. Partion tables can be the newer GPT UEFI style, or the old MBR masterbootrecord BIOS style, limited to 4 partitions and one can be extended to house more partions, one can be 'active' which windows can boot from.
https://en.wikipedia.org/wiki/GNU_GRUB#/media/File:GNU_GRUB_on_GPT_partitioned_hard_disk_drives.svg
Boot process, grub in the first 512bytes, menu which partition to find the kernel, the initfs kernel modules and where the rootf is.
GPT
fdisk cfdisk cgdisk sfdisk sgdisk
S for scriptable, F for BIOS/MBR layout, G for UEFI/GPT layout sfdisk -l
Temp partition
Format, Journaling
mount
parted partmagic losetup
LVM
https://en.wikipedia.org/wiki/GNU_GRUB#/media/File:GNU_GRUB_on_GPT_partitioned_hard_disk_drives.svg
Physical volume Volume group Logical volume
apk add lvm2 insmod lvm vgchange -ay
pvdisplay vgdisplay lvdisplay
vgchange -ay
lvcreate -L 32G -n ubuntest-hvm /dev/vg0 dd if=/dev/vg0/ubuntu-hvm of=/dev/vg0/ubuntest-hvm lvremove /dev/vg0/truenas-hvm
lvcreate -L 16G -n truenas-hvm /dev/vg0 lvcreate -l 100%FREE -n truenas-pool /dev/vg0
--- Logical volume --- LV Path /dev/vg0/ubuntest-hvm LV Name ubuntest-hvm VG Name vg0 LV UUID OZeTmq-fCJ3-HY87-vLkH-0G2m-bPKo-SCMpwm LV Write Access read/write LV Creation host, time nl, 2020-11-08 12:15:41 +0000 LV Status available # open 0 LV Size 32.00 GiB Current LE 8192 Segments 1 Allocation inherit Read ahead sectors auto - currently set to 256 Block device 253:4 --- Logical volume --- LV Path /dev/vg0/data LV Name data VG Name vg0 LV UUID 3wIcBP-IM69-It1X-2QGs-Bczs-KIvA-wb54hG LV Write Access read/write LV Creation host, time nl, 2020-12-30 08:14:01 +0000 LV Status available # open 1 LV Size 89.97 GiB Current LE 23033 Segments 1 Allocation inherit Read ahead sectors auto - currently set to 256 Block device 253:5
apk add device-mapper sfdisk multipath-tools dmsetup ls dmsetup info vg0-data sfdisk -l /dev/mapper/vg0-data kpartx -av /dev/mapper/vg0-data mount /dev/mapper/vg0-data1 /data
Extend VG/LV
pvcreate /dev/sdb vgextend ubuntu-vg /dev/sdb lvextend -l +100%FREE /dev/mapper/ubuntu--vg-ubuntu--lv resize2fs /dev/mapper/ubuntu--vg-ubuntu--lv
Next level
Mounting a nested volume group, which is in a GPT inside a Logical Volume, through the dev mapper using kpartx in multipath-tools
apk add multipath-tools kpartx -av /dev/mapper/vg0-data mount -t ext4 /dev/mapper/vg0-data1 /data sfdisk /dev/vg0/data dmsetup info vg0-data vgchange -ay
mount /dev/dm-4 /data/ mount -t ext4 /dev/dm-4 /data/ mount -t vfat /dev/dm-4 /data/ mount -t vfat /dev/dm-4 /data/ mount -v -t vfat /dev/dm-4 /data/ mount -vvv -t vfat /dev/dm-4 /data/ strace mount -t vfat /dev/dm-4 /data/ strace mount -t vfat /dev/dm-4 /data/ mount -vvv -t vfat /dev/dm-4 /data/
sfdisk /dev/vg0/data file -s /dev/vg0/data file -s /dev/mapper/vg0-data file /dev/vg0/data file /dev/mapper/vg0-data ls -la /dev/vg0/data mount /dev/mapper/vg0-data /data
Nexter level
inotify
auditctl
bpftrace
sleuthkit jls
debugfs
logdump
iSCSI
https://wiki.alpinelinux.org/wiki/Linux_iSCSI_Target_(TCM)
targetcli
https://askubuntu.com/questions/101801/set-up-eth0-network-interface-using-dhcp-in-initramfs#103225
https://forums.raspberrypi.com/viewtopic.php?t=134799
cd /backstores/block create block0 /dev/vg0/ubuntu-iscsi create lun=0 storage_object=/backstores/block/block0 cd /iscsi/iqn.2003-01.org.linux-iscsi.nl.x8664:sn.aeea2726a6c3/tpg1 set attribute authentication=0 set attribute generate_node_acls=1 set attribute demo_mode_write_protect=0 cd portals delete ip_address=0.0.0.0 ip_port=3260 create ip_address=10.0.0.5
/> ls o- / ............................................................ [...] o- backstores ................................................. [...] | o- block ..................................... [Storage Objects: 1] | | o- block0 [/dev/vg0/ubuntu-iscsi (48.0GiB) write-thru activated] | | o- alua ...................................... [ALUA Groups: 1] | | o- default_tg_pt_gp .......... [ALUA state: Active/optimized] | o- fileio .................................... [Storage Objects: 0] | o- pscsi ..................................... [Storage Objects: 0] | o- ramdisk ................................... [Storage Objects: 0] o- iscsi ............................................... [Targets: 1] | o- iqn.2003-01.org.linux-iscsi.nl.x8664:sn.aeea2726a6c3 . [TPGs: 1] | o- tpg1 ..................................... [gen-acls, no-auth] | o- acls ............................................. [ACLs: 0] | o- luns ............................................. [LUNs: 1] | | o- lun0 [block/block0 (/dev/vg0/ubuntu-iscsi) (default_tg_pt_gp)] | o- portals ....................................... [Portals: 1] | o- 10.0.0.5:3260 ....................................... [OK] o- loopback ............................................ [Targets: 0] o- vhost ............................................... [Targets: 0] o- xen-pvscsi .......................................... [Targets: 0]
service targetcli restart
iscsiadm -m discovery -t sendtargets -p 10.0.0.5 iscsiadm -m node -o show iscsiadm -m node --login iscsiadm -m session -o show
NOP-IN
The initiator doesn't boot IPXE, GRUB2, iSCSI because on the initiator (client) can't read from the iscsi target (server)
block device blk_update_request: I/O error, dev sdb, sector 77769088 op 0x0:(READ) flasg 0x3000 phys_seg 1 prio class 0
On the target the error message is
Did not receive response to NOPIN on CID: 0, failing connection for I_T Nexus
This requires NOP-in to be disabled, but the StackOver flow article and the IPXE page it refers to state to disable time out using lio_node, but that command doesn't exist in the implementation of iscsi, I use and the iscsi-linux.org page doesn't exist anymore. open-iscsi on github has a readme that spells out the NOP-in disabling
https://ipxe.org/appnote/iscsi_targets https://github.com/open-iscsi/open-iscsi
https://github.com/open-iscsi/open-iscsi
8.2 iSCSI settings for iSCSI root
node.conn[0].timeo.noop_out_interval = 0
node.conn[0].timeo.noop_out_timeout = 0
IPXE Boot from iSCSI SAN
Contents of /var/www/ipxe/bootstrap.ipxe
#!ipxe sanboot --no-describe --filename \EFI\ubuntu\grubx64.efi --keep iscsi:10.0.0.5::::iqn.2003-01.org.linux-iscsi.nl.x8664:sn.aeea2726a6c3 || goto fail
mount --bind /dev /mnt/dev mount --bind /dev/pts /mnt/dev/pts mount --bind /proc /mnt/proc mount --bind /sys /mnt/sys chroot /mnt
update-initramfs -u
update-initramfs -c -k all
unmkinitramfs initrd.img-5.15.0-91-generic custom
unmkinitramfs initrd.img-5.15.0-91-generic custom mkinitramfs -o initrd.img-5.15-generic -r custom/main
initramfs-tools from debian depends on iscsistart, where upstream open-iscsi recommends to start the iscsid daemon and use the iscsiadm command.
I didn't understand where networking was started, so I created an /etc/initramfs/iscsi /bin/sh script that does an "dhclient enp0s31f0" and runs iscsistart with all the parameters.
Maybe using IPXE and GRUB2 should not use iSCSI for loading the kernel, instead, use HTTP to pull the first stage and then switch to ISCSI as root.
Maybe the Ubuntu initramfs should set iscsistart parameters to disable noop values? Or may this is not a problem anymore after iscsi is only use after the boot.
node.conn[0].timeo.noop_out_interval = 0
Then I don't have to rely on initramfs scripts other than for those commands, the iscsi boot continues, because the kernel has an IP address and the root device is now mounted.