July 15, 2024
#Fun

It happened after Ubuntu Auto-Remove iproute2

目录

上文配置好 Nix 管理后（还没完全发布），有些洁癖，就用 apt 卸载掉部分基础包，同时执行 apt auto-remove。这时习惯性 reboot 后发现 ssh 连接失败，控制台重启系统也还是不行。无奈，用 KVM 提供的 VNC 上去看看。但并不能复制

备份

救援模式启动！

挂载系统盘后备份 Nix 配置文件。

lsblk 或 fdisk -l 查看硬盘和分区，找出我们的系统盘

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51

root # lsblk
NAME    MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
loop0     7:0    0 79.9M  1 loop /snap/lxd/22923
loop1     7:1    0   87M  1 loop /snap/lxd/28373
loop2     7:2    0 63.9M  1 loop /snap/core20/2318
loop3     7:3    0 38.8M  1 loop /snap/snapd/21759
loop4     7:4    0 61.9M  1 loop /snap/core20/1434
sr0      11:0    1  376K  0 rom  
vda     252:0    0   80G  0 disk 
├─vda1  252:1    0 79.9G  0 part /
├─vda14 252:14   0    4M  0 part 
└─vda15 252:15   0  106M  0 part /boot/efi
root # fdisk -l
Disk /dev/loop0: 79.95 MiB, 83832832 bytes, 163736 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk /dev/loop1: 87.03 MiB, 91258880 bytes, 178240 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk /dev/loop2: 63.95 MiB, 67051520 bytes, 130960 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk /dev/loop3: 38.83 MiB, 40714240 bytes, 79520 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk /dev/loop4: 61.9 MiB, 64909312 bytes, 126776 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk /dev/vda: 80 GiB, 85899345920 bytes, 167772160 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 

Device      Start       End   Sectors  Size Type
/dev/vda1  227328 167772126 167544799 79.9G Linux filesystem
/dev/vda14   2048     10239      8192    4M BIOS boot
/dev/vda15  10240    227327    217088  106M EFI System

Partition table entries are not in disk order.

Type 与 Szie 都能发现 /dev/vda1 是我们要找的跟分区

创建挂载点并挂载分区

  1. 创建一个挂载点目录，这里选择 /mnt/rescue
1

mkdir /mnt/rescue
  1. 挂载根分区到这个挂载点：
1

mount /dev/vda1 /mnt/rescue

如果需要，可以用这个方法挂载 /boot 分区

这时就可以顺利备份，或者还可以在挂载好必要的文件系统后使用 chroot 进入挂载的环境，方便修复

退出时使用 umount 卸载所有文件系统再 reboot

修复

本来到此就可以告一段落，直接重置系统安装 Nix，恢复配置一条命令结束，但想着在 Linux 下遇到一点问题就重装还是不爽，还是想试试恢复。于是就有了下面的部分

ifconfig 发现只剩下 loopback

ifconfig -a / ip a 看网络配置，网卡还在，只是没激活

demsg 看报错，但在这个 VNC 下，less 会有 bug，用着用着就输入错乱，也不能方便的导出到本地（其实可以导出，输出写入文件即可，但是懒得再次关机进入 rescue system）

lspci 查网卡是否还在，存在 Ethernet controller: Red Hat, Inc. Virtio network device 条目

运行 Ubuntu(17.10 及之后版本使用 Netplan), 于是去 /etc/netplan/ 看配置, 长下面这个样子

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

network:
  version: 2
  ethernets:
    eth0:  # 网卡名称可能与 eth0 不同，根据实际情况(ifconfig -a)调整
      dhcp4: no
      addresses:
        - VPS IP/24  # CIDR 表示法
      gateway4: gateway IP # 也可能是 routes 字段中的 via 子项的值
      nameservers:
        addresses:
          - 8.8.8.8
          - 8.8.4.4

ifconfig 激活网卡

1

ifconfig ${网卡名} up

接着用 ifconfig 配置 ipadress 与 mask

1
2

ifconfig ${网卡名} ${ipadress}
ifconfig ${网卡名} netmask ${mask, 根据 `/etc/netplan/` 中文件的 CIDR 表示的 mask 转为 x.x.x.x 格式}

后用 route 配置网关

1

route add default gw ${`/etc/netplan/` 里查到的网关地址}

也可以用 ip 配置，随后尝试 ping 8.8.8.8，没有问题

但 ping www.google.com 失败，ip 能访问，域名访问失败，猜测是本地 nameserver 配置问题。编辑 /etc/resolv.conf，加上一个域名服务器地址即可

至此，SSH 可以链接了，下面来排查当时具体移除了什么 pkg（其实这一步该放到最开始看，但当时没多想就先想着有网连上 SSH 后好用）

回到案发现场（/var/log/apt/history.log）

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59

Start-Date: 2024-07-10  06:08:05
Commandline: /usr/bin/unattended-upgrade
Upgrade: python3.10:amd64 (3.10.12-1~22.04.3, 3.10.12-1~22.04.4), libpython3.10-minimal:amd64 (3.10.12-1~22.04.3, 3.10.12-1~22.04.4), libpython3.10-stdlib:amd64 (3.10.12-1~22.04.3, 3.10.12-1~22.04.4), libpython3.10:amd64 (3.10.12-1~22.04.3, 3.10.12-1~22.04.4), python3.10-minimal:amd64 (3.10.12-1~22.04.3, 3.10.12-1~22.04.4)
End-Date: 2024-07-10  06:08:08

Start-Date: 2024-07-13  18:21:13
Commandline: apt remove neovim
Remove: neovim:amd64 (0.6.1-3)
End-Date: 2024-07-13  18:21:14

Start-Date: 2024-07-13  18:50:24
Commandline: apt autoremove
Remove: libvterm0:amd64 (0.1.4-1), lua-luv:amd64 (1.36.0-0-1), libluajit-5.1-2:amd64 (2.1.0~beta3+dfsg-6), x11-common:amd64 (1:7.7+23ubuntu2), libtree-sitter0:amd64 (0.20.3-1), libtermkey1:amd64 (0.22-1), neovim-runtime:amd64 (0.6.1-3), libxmu6:amd64 (2:1.1.3-3), python3-pynvim:amd64 (0.4.2-1), libunibilium4:amd64 (2.1.0-1), libmsgpackc2:amd64 (3.3.0-4), libice6:amd64 (2:1.0.10-1build2), xclip:amd64 (0.13-2), python3-msgpack:amd64 (1.0.3-1build1), python3-neovim:amd64 (0.4.2-1), libsm6:amd64 (2:1.2.3-1build2), python3-greenlet:amd64 (1.1.2-3build1), libxt6:amd64 (1:1.2.1-1), libluajit-5.1-common:amd64 (2.1.0~beta3+dfsg-6)
End-Date: 2024-07-13  18:50:25

Start-Date: 2024-07-13  18:52:19
Commandline: apt remove vim
Remove: vim:amd64 (2:8.2.3995-1ubuntu2.16), ubuntu-server:amd64 (1.481)
End-Date: 2024-07-13  18:52:20

Start-Date: 2024-07-13  18:55:12
Commandline: apt remove git
Remove: git:amd64 (1:2.34.1-1ubuntu1.11)
End-Date: 2024-07-13  18:55:12

Start-Date: 2024-07-13  18:55:24
Commandline: apt remove gcc
Remove: gcc:amd64 (4:11.2.0-1ubuntu1)
End-Date: 2024-07-13  18:55:24

Start-Date: 2024-07-13  18:55:59
Commandline: apt autoremove
Remove: libgomp1:amd64 (12.3.0-1ubuntu1~22.04), manpages-dev:amd64 (5.10-1ubuntu1), gcc-11:amd64 (11.4.0-1ubuntu1~22.04), libtsan0:amd64 (11.4.0-1ubuntu1~22.04), cpp:amd64 (4:11.2.0-1ubuntu1), libjpeg-turbo8:amd64 (2.1.2-0ubuntu1), libdeflate0:amd64 (1.10-2), libjpeg8:amd64 (8c-2ubuntu10), libtiff5:amd64 (4.3.0-6ubuntu0.9), libcc1-0:amd64 (12.3.0-1ubuntu1~22.04), libmpc3:amd64 (1.2.1-2build1), libxpm4:amd64 (1:3.5.12-1ubuntu0.22.04.2), libfontconfig1:amd64 (2.13.1-4.2ubuntu5), fontconfig-config:amd64 (2.13.1-4.2ubuntu5), libsodium23:amd64 (1.0.18-1build2), libasan6:amd64 (11.4.0-1ubuntu1~22.04), libnsl-dev:amd64 (1.3.0-2build2), libpython3.10:amd64 (3.10.12-1~22.04.4), rpcsvc-proto:amd64 (1.4.2-0ubuntu6), libwebp7:amd64 (1.2.2-2ubuntu0.22.04.2), libcrypt-dev:amd64 (1:4.4.27-1), vim-runtime:amd64 (2:8.2.3995-1ubuntu2.16), cpp-11:amd64 (11.4.0-1ubuntu1~22.04), liberror-perl:amd64 (0.17029-1), libitm1:amd64 (12.3.0-1ubuntu1~22.04), libjbig0:amd64 (2.1-3.1ubuntu0.22.04.1), libc-dev-bin:amd64 (2.35-0ubuntu3.8), libc-devtools:amd64 (2.35-0ubuntu3.8), libisl23:amd64 (0.24-2build1), libc6-dev:amd64 (2.35-0ubuntu3.8), libquadmath0:amd64 (12.3.0-1ubuntu1~22.04), fonts-dejavu-core:amd64 (2.37-2build1), libubsan1:amd64 (12.3.0-1ubuntu1~22.04), liblsan0:amd64 (12.3.0-1ubuntu1~22.04), gcc-11-base:amd64 (11.4.0-1ubuntu1~22.04), libgd3:amd64 (2.3.0-2ubuntu2), git-man:amd64 (1:2.34.1-1ubuntu1.11), libtirpc-dev:amd64 (1.3.2-2ubuntu0.1), libgcc-11-dev:amd64 (11.4.0-1ubuntu1~22.04), libatomic1:amd64 (12.3.0-1ubuntu1~22.04), linux-libc-dev:amd64 (5.15.0-113.123)
End-Date: 2024-07-13  18:56:00

Start-Date: 2024-07-13  18:59:03
Commandline: apt remove htop
Remove: htop:amd64 (3.0.5-7build2)
End-Date: 2024-07-13  18:59:03

Start-Date: 2024-07-13  18:59:19
Commandline: apt remove nano less
Remove: less:amd64 (590-1ubuntu0.22.04.3), nano:amd64 (6.2-1), ubuntu-minimal:amd64 (1.481)
End-Date: 2024-07-13  18:59:20

Start-Date: 2024-07-13  19:00:11
Commandline: apt autoremove
Remove: libnl-3-200:amd64 (3.5.0-0.1), libnl-genl-3-200:amd64 (3.5.0-0.1)
End-Date: 2024-07-13  19:00:11

Start-Date: 2024-07-13  19:49:04
Commandline: apt remove iproute2
Remove: byobu:amd64 (5.133-1), cloud-init:amd64 (22.1-14-g2e17a0d6-0ubuntu1~22.04.5), isc-dhcp-client:amd64 (4.4.1-2.3ubuntu2.3), open-vm-tools:amd64 (2:12.1.5-3~ubuntu0.22.04.4), netplan.io:amd64 (0.106.1-7ubuntu0.22.04.4), iproute2:amd64 (5.15.0-1ubuntu2)
End-Date: 2024-07-13  19:49:07

Start-Date: 2024-07-13  19:49:26
Commandline: apt autoremove
Remove: pastebinit:amd64 (1.5.1-1ubuntu1), zerofree:amd64 (1.1.1-1build3), libatm1:amd64 (1:2.5.1-4build2), libmspack0:amd64 (0.10.1-2build2), libbpf0:amd64 (1:0.5.0-1ubuntu22.04.1), isc-dhcp-common:amd64 (4.4.1-2.3ubuntu2.3), libisc-export1105:amd64 (1:9.11.19+dfsg-2.1ubuntu3), libxmlsec1:amd64 (1.2.33-1build2), run-one:amd64 (1.17-0ubuntu1), libxslt1.1:amd64 (1.1.34-4ubuntu0.22.04.1), libdns-export1110:amd64 (1:9.11.19+dfsg-2.1ubuntu3), python3-newt:amd64 (0.52.21-5ubuntu2), libnetplan0:amd64 (0.106.1-7ubuntu0.22.04.4), libxmlsec1-openssl:amd64 (1.2.33-1build2)
End-Date: 2024-07-13  19:49:27

发现 13日晚上 删除 iproute2 同时 auto-remove 了 cloud-init 以及 netplan，太难蹦了。删除后 Nix 中 home.nix 也包含了 iproute2，却忘了把自动移除的包也写入 home.nix，导致没有网络。那解决方案就很简单了，把缺少的包写入 home.nix 后执行安装即可。

1
2
3
4

Start-Date: 2024-07-13  19:49:04
Commandline: apt remove iproute2
Remove: byobu:amd64 (5.133-1), cloud-init:amd64 (22.1-14-g2e17a0d6-0ubuntu1~22.04.5), isc-dhcp-client:amd64 (4.4.1-2.3ubuntu2.3), open-vm-tools:amd64 (2:12.1.5-3~ubuntu0.22.04.4), netplan.io:amd64 (0.106.1-7ubuntu0.22.04.4), iproute2:amd64 (5.15.0-1ubuntu2)
End-Date: 2024-07-13  19:49:07

Reference

#Fun

Similar articles: