疑惑 最近在反覆搭建ceph集群過程中，總是遇到osd創建不成功的問題，疑似硬碟殘留信息，排查中引出了很多陌生的命令，比如vgremove等，於是打算重新瞭解這部分。 LVM是什麼？ 邏輯捲管理器（LVM，Logical Volume Manager）是一種把硬碟空間分配成邏輯捲的方法。 看到定義可 ...
疑惑
最近在反覆搭建ceph集群過程中，總是遇到osd創建不成功的問題，疑似硬碟殘留信息，排查中引出了很多陌生的命令，比如vgremove等，於是打算重新瞭解這部分。
LVM是什麼？
邏輯捲管理器（LVM，Logical Volume Manager）是一種把硬碟空間分配成邏輯捲的方法。
看到定義可能還比較懵，不妨結合場景：
有一塊系統盤空間隨著時間推移容量需要擴容該怎麼做？
這時候使用LVM就比較方便了，它可以彈性的調整文件系統的容量，可以整合多個物理分區在一起，讓這些分區看起來像是一個磁碟一樣。通俗理解就是它可以將很多硬碟/分區全部拿過來作為一個資源池，然後自己再隨意劃分成邏輯層面的分區，那麼這個分區後期進行擴容縮容刪除就比較方便了！
LVM相關概念
1、Physical Volume，PV，物理捲
[root@node3 ~]# fdisk /dev/sdb
Welcome to fdisk (util-linux 2.23.2).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.
Device does not contain a recognized partition table
Building a new DOS disklabel with disk identifier 0x172f2548.
Command (m for help): p ###輸入 p 查看分區情況
Disk /dev/sdb: 10.7 GB, 10737418240 bytes, 20971520 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x172f2548
Device Boot Start End Blocks Id System
Command (m for help): n ###輸入 n 創建新分區
Partition type:
p primary (0 primary, 0 extended, 4 free)
e extended
Select (default p): p ### 預設是 p ，創建主分區，視情況而定
Partition number (1-4, default 1): 1 ###這裡做第一塊主分區
First sector (2048-20971519, default 2048):
Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-20971519, default 20971519):
Using default value 20971519
Partition 1 of type Linux and of size 10 GiB is set
Command (m for help): p ###再次輸入 p 查看分區情況
Disk /dev/sdb: 10.7 GB, 10737418240 bytes, 20971520 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x172f2548
Device Boot Start End Blocks Id System
/dev/sdb1 2048 20971519 10484736 83 Linux
Command (m for help): w ###最後輸入 w 保存從sdb分出來的sdb1
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.
fdisk -l查看是否分區成功
[root@node3 ~]# fdisk -l
Disk /dev/sdb: 10.7 GB, 10737418240 bytes, 20971520 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x172f2548
Device Boot Start End Blocks Id System
/dev/sdb1 2048 20971519 10484736 83 Linux
可以看到實際分區Id欄位為83，使用時需要通過 fdisk 命令將Id欄位調整為8e(LVM 的標識符)，再經過 pvcreate 命令將它轉為 LVM 最底層的物理捲(PV)，之後這些PV才能夠被利用
[root@node3 ~]# fdisk /dev/sdb
Welcome to fdisk (util-linux 2.23.2).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.
Command (m for help): t ###輸入 t 修改分區類型
Selected partition 1
Hex code (type L to list all codes): L ###輸入 L 列出所有分區類型
0 Empty 24 NEC DOS 81 Minix / old Lin bf Solaris
1 FAT12 27 Hidden NTFS Win 82 Linux swap / So c1 DRDOS/sec (FAT-
2 XENIX root 39 Plan 9 83 Linux c4 DRDOS/sec (FAT-
3 XENIX usr 3c PartitionMagic 84 OS/2 hidden C: c6 DRDOS/sec (FAT-
4 FAT16 <32M 40 Venix 80286 85 Linux extended c7 Syrinx
5 Extended 41 PPC PReP Boot 86 NTFS volume set da Non-FS data
6 FAT16 42 SFS 87 NTFS volume set db CP/M / CTOS / .
7 HPFS/NTFS/exFAT 4d QNX4.x 88 Linux plaintext de Dell Utility
8 AIX 4e QNX4.x 2nd part 8e Linux LVM df BootIt
9 AIX bootable 4f QNX4.x 3rd part 93 Amoeba e1 DOS access
a OS/2 Boot Manag 50 OnTrack DM 94 Amoeba BBT e3 DOS R/O
b W95 FAT32 51 OnTrack DM6 Aux 9f BSD/OS e4 SpeedStor
c W95 FAT32 (LBA) 52 CP/M a0 IBM Thinkpad hi eb BeOS fs
e W95 FAT16 (LBA) 53 OnTrack DM6 Aux a5 FreeBSD ee GPT
f W95 Ext'd (LBA) 54 OnTrackDM6 a6 OpenBSD ef EFI (FAT-12/16/
10 OPUS 55 EZ-Drive a7 NeXTSTEP f0 Linux/PA-RISC b
11 Hidden FAT12 56 Golden Bow a8 Darwin UFS f1 SpeedStor
12 Compaq diagnost 5c Priam Edisk a9 NetBSD f4 SpeedStor
14 Hidden FAT16 <3 61 SpeedStor ab Darwin boot f2 DOS secondary
16 Hidden FAT16 63 GNU HURD or Sys af HFS / HFS+ fb VMware VMFS
17 Hidden HPFS/NTF 64 Novell Netware b7 BSDI fs fc VMware VMKCORE
18 AST SmartSleep 65 Novell Netware b8 BSDI swap fd Linux raid auto
1b Hidden W95 FAT3 70 DiskSecure Mult bb Boot Wizard hid fe LANstep
1c Hidden W95 FAT3 75 PC/IX be Solaris boot ff BBT
1e Hidden W95 FAT1 80 Old Minix
Hex code (type L to list all codes): 8e ###輸入 8e 修改為指定分區類型
Changed type of partition 'Linux' to 'Linux LVM'
Command (m for help): p ######輸入 p 查看分區情況
Disk /dev/sdb: 10.7 GB, 10737418240 bytes, 20971520 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x172f2548
Device Boot Start End Blocks Id System
/dev/sdb1 2048 20971519 10484736 8e Linux LVM
Command (m for help): w ###最後輸入 w 保存修改的分區類型
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.
使用pvcreate創建pv
[root@node3 ~]# pvcreate /dev/sdb1
Physical volume "/dev/sdb1" successfully created.
2、Volume Group，VG，捲組
VG可以理解為很多物理捲（PV）所組成的組
[root@node3 ~]# vgcreate storage /dev/sdb1
Volume group "storage" successfully created
[root@node3 ~]# vgdisplay
--- Volume group ---
VG Name storage
System ID
Format lvm2
Metadata Areas 1
Metadata Sequence No 1
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 0
Open LV 0
Max PV 0
Cur PV 1
Act PV 1
VG Size <10.00 GiB
PE Size 4.00 MiB
Total PE 2559
Alloc PE / Size 0 / 0
Free PE / Size 2559 / <10.00 GiB
VG UUID fK1Wcs-B1BL-TY3G-IRiJ-7JnV-wd0T-CRFx95
使用vgextend添加 pv 到 vg
[root@node3 ~]# vgextend storage /dev/sdc
sdc sdc1
[root@node3 ~]# vgextend storage /dev/sdc1
Physical volume "/dev/sdc1" successfully created.
Volume group "storage" successfully extended
[root@node3 ~]# vgdisplay
--- Volume group ---
VG Name storage
System ID
Format lvm2
Metadata Areas 2
Metadata Sequence No 2
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 0
Open LV 0
Max PV 0
Cur PV 2
Act PV 2
VG Size 19.99 GiB
PE Size 4.00 MiB
Total PE 5118
Alloc PE / Size 0 / 0
Free PE / Size 5118 / 19.99 GiB
VG UUID fK1Wcs-B1BL-TY3G-IRiJ-7JnV-wd0T-CRFx95
可以看到兩個 pv 已經成功添加到 vg 中
3、Logical Volume，LV，邏輯捲
LV 是由 VG 切分而來的，此時 LV 就可以當成一個分區格式化來使用了，
lvcreate 參數
-L 是直接制定容量，基礎單位是MB，如果要切割出1G空間，則 -L 1G，MB以此類推。
-l 是以基本單元的個數為單位，每個基本單元的大小是4MB，-l 20，則是切割出4*20=80M空間。
-n 是指定生成邏輯捲的名稱。
在捲組中切割出一個：名稱為test1，大小為5G的空間：
[root@node3 ~]# lvcreate -n test1 -L 5G storage
Logical volume "test1" created.
[root@node3 ~]# lvdisplay ###查看lv
--- Logical volume ---
LV Path /dev/storage/test1
LV Name test1
VG Name storage
LV UUID WkSHKt-2GPL-sZX0-ZZeu-KGZB-9Gao-vufFmC
LV Write Access read/write
LV Creation host, time node3, 2023-02-01 15:22:25 +0800
LV Status available
# open 0
LV Size 5.00 GiB
Current LE 1280
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 8192
Block device 253:0
格式化、掛載使用
[root@node3 ~]# mkfs.ext4 /dev/storage/test1
mke2fs 1.42.9 (28-Dec-2013)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
327680 inodes, 1310720 blocks
65536 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=1342177280
40 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736
Allocating group tables: done
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done
[root@node3 ~]# mkdir /mnt/dir1
[root@node3 ~]# mount /dev/storage/test1 /mnt/dir1/
[root@node3 ~]# df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 1.9G 0 1.9G 0% /dev
tmpfs 1.9G 0 1.9G 0% /dev/shm
tmpfs 1.9G 20M 1.9G 2% /run
tmpfs 1.9G 0 1.9G 0% /sys/fs/cgroup
/dev/sda2 19G 2.2G 17G 12% /
/dev/sda1 1014M 142M 873M 14% /boot
tmpfs 378M 0 378M 0% /run/user/0
/dev/mapper/storage-test1 4.8G 20M 4.6G 1% /mnt/dir1
4、Physical Extend，PE，物理擴展塊
PE 是整個 LVM 最小的存儲塊，LVM 的 VG 最多僅能含有 65534 個 PE，一個 PE 預設的大小時 4M。
所以可以通過調整 PE 來調整 VG最大容量。
# vgdisplay
--- Volume group ---
VG Name ceph-77651246-631b-44d8-9ece-1cc0af8e0b68
System ID
Format lvm2
Metadata Areas 1
Metadata Sequence No 17
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 1
Open LV 1
Max PV 0
Cur PV 1
Act PV 1
VG Size 9.00 GiB
PE Size 1.00 GiB
Total PE 9
Alloc PE / Size 9 / 9.00 GiB
Free PE / Size 0 / 0
VG UUID GdTfRy-au4B-DgAN-Ej2X-XgdK-hFNz-wDrLm1
LV 擴容、縮容
擴容
[root@node3 ~]# umount /mnt/dir1/ ###解除掛載
[root@node3 ~]# lvextend -L 15G /dev/storage/test1 ###vg容量不夠
Insufficient free space: 2560 extents needed, but only 2558 available
[root@node3 ~]# lvextend -L 10G /dev/storage/test1 ###擴展後的分區容量，必須大於原容量
Size of logical volume storage/test1 changed from 5.00 GiB (1280 extents) to 10.00 GiB (2560 extents).
Logical volume storage/test1 successfully resized.
[root@node3 ~]# e2fsck -f /dev/storage/test1 ###檢查硬碟的完整性
e2fsck 1.42.9 (28-Dec-2013)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/storage/test1: 11/327680 files (0.0% non-contiguous), 58462/1310720 blocks
[root@node3 ~]# resize2fs /dev/storage/test1 ###重置硬碟容量
resize2fs 1.42.9 (28-Dec-2013)
Resizing the filesystem on /dev/storage/test1 to 2621440 (4k) blocks.
The filesystem on /dev/storage/test1 is now 2621440 blocks long.
[root@node3 ~]# mount /dev/storage/test1 /mnt/dir1/ ###重新掛載使用
[root@node3 ~]# df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 1.9G 0 1.9G 0% /dev
tmpfs 1.9G 0 1.9G 0% /dev/shm
tmpfs 1.9G 20M 1.9G 2% /run
tmpfs 1.9G 0 1.9G 0% /sys/fs/cgroup
/dev/sda2 19G 2.2G 17G 12% /
/dev/sda1 1014M 142M 873M 14% /boot
tmpfs 378M 0 378M 0% /run/user/0
/dev/mapper/storage-test1 9.8G 23M 9.2G 1% /mnt/dir1
縮容
lvextend -L 3G /dev/storage/test1 ###操作參考擴容
LV刪除、卸載
[root@node3 ~]# umount /mnt/dir1/ ###解除掛載
[root@node3 ~]# lvremove /dev/storage/test1 ###刪除lv
Do you really want to remove active logical volume storage/test1? [y/n]: y
Logical volume "test1" successfully removed
[root@node3 ~]# vgremove storage ###刪除vg
Do you really want to remove volume group "storage" containing 1 logical volumes? [y/n]: y
Do you really want to remove active logical volume storage/test2? [y/n]: y
Logical volume "test2" successfully removed
Volume group "storage" successfully removed
[root@node3 ~]# pvremove /dev/sdb1 ###刪除pv
Labels on physical volume "/dev/sdb1" successfully wiped.
