Increasing OpenBSD's Resilience to Power Loss on a Powerful Laptop
Introduction
I have occasionally had kernel panics when losing power while running OpenBSD on FFS on softraid CRYPTO on cheap SSDs. The loss of power would occur either because a laptop battery ran out of power, a laptop battery fell out (from a ThinkPad X201), or a desktop power cable became unplugged.
After the initial panic, the system would continue to panic after I would try to access certain parts of the softraid disk.
I never really figured out what the problem was, just that it happened with this pattern. Rather than fixing it, I got around this issue by switching one of my computers to FreeBSD and using GELI and ZFS. (There were also many other reasons for switching the operating system.)
For my laptop, however, I wanted OpenBSD, as OpenBSD officially supports its hardware (ThinkPad X240) and OpenBSD has worked much better with this hardware than FreeBSD has, in my experience.
Related work
I took inspiration from Marko Cupać's method on How to Increase OpenBSD's Resilience to Power Outages. Increased resilience is achieved when everything is mounted either as read-only or as in-memory filesystems.
Marko mentions that he uses this technique for network appliances, presumably with SD card storage "(such as branch office routers in godforsaken places where having electricity and Internet access at all is considered a lucky circumstance)" and he documents an example of a PC Engines apu3b4.
Present application
Unlike Marko's situation, my situation is a general-purpose laptop with SSD storage that I mostly use to log in to the other computer and that I sometimes write things on temporarily when I don't have an internet connection.
It happens that I have pretty good internet connectivity where I am, so I store most data on the main computer and access it through mosh. On the other hand, I do want to be able to store some things locally as I sometimes have periods of a few minutes without internet access and I sometimes turn off the other computer. I also wanted to keep it closer to standard OpenBSD so that configuration would be easier for me.
I am mainly trying to prevent corruption of the softraid CRYPTO disk.
Differences from Marko's configuration
I changed some things to account for the differences in method of use between Marko and me.
Root filesystem
I decided to mount most things read-only or in memory but to keep / as read-write. I kept / read-write for these reasons.
- This is the most annoying one to set up as read-only. I did not want to spend the time on that, and I did not trust myself to do it properly.
- I frequently want to edit files in /etc, at least right now while I am still setting things up.
- OpenBSD has the /altroot feature.
- I am not really concerned in part about the hardware's lifespan. Marko is, because he uses SD card storage and because it is inconvenient for him to travel to wherever the computer is and replace the SD card when it goes bad. I am using an SSD rather than an SD card, and I don't turn the computer on very much, so I am not worried about lifespan of the storage. Furthermore, I have the computer with me, so it will be easy to replace the SSD when it goes bad.
Extra partitions
While Marko puts /var/syspatch in its own non-memory partition, I kept /var/syspatch in memory, as I can have up to 8 GB of memory in this laptop. If the size does become an issue, maybe I will move it later.
I did create a read-only /opt partition for large files. So far I have used it only for a database dump and a corresponding search index. I search the database frequently and update the database infrequently.
Frequency of saving
Marko runs a daily cron job to save the contents of the in-memory filesystem to the SD card. I use an hourly cron job, because
- My edits to the system are more important. More than logging and caching, they include things like the present report.
- I am not worried about SD card wear.
I also explicitly save periodically.
Method
I mostly followed Marko's "Software", "Installation", and "First single-user mode boot" steps.
I did something similar to "First standard boot", but it is different enough to warrant explanation.
- I wrote something similar to /root/syncvar.sh, but it is shorter, and it is saved directly as /etc/rc.shutdown.
- I installed /etc/rc.shutdown in several other places as well.
- I did not modify rc.local, as I did not need to mount / read-only.
Final configuration
The final configuration can be summarized by the currently-mounted partitions, the disklabels, and the contents of a few important files.
I am running OpenBSD 6.6 on a ThinkPad X240. It has a ~256GB SSD on sd0, softraid CRYPTO on sd2, and a keydisk for the softraid CRYPTO on sd1. My filesystems look like this at boot.
$ mount
/dev/sd2a on / type ffs (local)
/dev/sd2f on /mfs type ffs (local, nodev, nosuid, read-only)
/dev/sd2d on /usr type ffs (local, nodev, read-only)
/dev/sd2e on /usr/X11R6 type ffs (local, nodev, read-only)
/dev/sd2g on /usr/local type ffs (local, nodev, wxallowed, read-only)
mfs:8611 on /tmp type mfs (asynchronous, local, nodev, nosuid, size=262144 1K-blocks)
mfs:15822 on /var type mfs (asynchronous, local, nodev, nosuid, size=262144 1K-blocks)
mfs:20765 on /home type mfs (asynchronous, local, nodev, nosuid, size=1048576 1K-blocks)
/dev/sd2h on /opt type ffs (local, nodev, wxallowed, read-only)
$ df -h
Filesystem Size Used Avail Capacity Mounted on
/dev/sd2a 123M 89.4M 27.2M 77% /
/dev/sd2f 6.9G 208M 6.3G 3% /mfs
/dev/sd2d 3.0G 955M 1.9G 33% /usr
/dev/sd2e 1008M 202M 756M 21% /usr/X11R6
/dev/sd2g 11.8G 1.6G 9.6G 14% /usr/local
mfs:8611 247M 46.5M 188M 20% /tmp
mfs:15822 247M 14.4M 220M 6% /var
mfs:20765 991M 190M 751M 20% /home
/dev/sd2h 24.3G 6.2G 17.0G 27% /opt
At the time of writing I did not have the keydisk plugged in, so I report the disklabels only of the SSD and softraid disk.
$ disklabel -p G sd0
# /dev/rsd0c:
type: SCSI
disk: SCSI disk
label: SAMSUNG MZ7PD256
duid: c0fd74026484bd67
flags:
bytes/sector: 512
sectors/track: 63
tracks/cylinder: 255
sectors/cylinder: 16065
cylinders: 31130
total sectors: 500118192 # total bytes: 238.5G
boundstart: 64
boundend: 500103450
drivedata: 0
16 partitions:
# size offset fstype [fsize bsize cpg]
a: 48.0G 64 RAID
c: 238.5G 0 unused
$ disklabel -p G sd2
# /dev/rsd2c:
type: SCSI
disk: SCSI disk
label: SR CRYPTO
duid: 15ea7c8e8a859e25
flags:
bytes/sector: 512
sectors/track: 63
tracks/cylinder: 255
sectors/cylinder: 16065
cylinders: 6266
total sectors: 100678763 # total bytes: 48.0G
boundstart: 64
boundend: 100663290
drivedata: 0
16 partitions:
# size offset fstype [fsize bsize cpg]
a: 0.1G 64 4.2BSD 2048 16384 1998 # /
b: 0.1G 257024 4.2BSD 2048 16384 1999
c: 48.0G 0 unused
d: 3.0G 514080 4.2BSD 2048 16384 12958 # /usr
e: 1.0G 6811552 4.2BSD 2048 16384 12958 # /usr/X11R6
f: 7.0G 8916064 4.2BSD 2048 16384 12958 # /mfs
g: 12.0G 23599456 4.2BSD 2048 16384 12958 # /usr/local
h: 24.7G 48773312 4.2BSD 2048 16384 12958 # /opt
Note that the only read-write filesystem in the fstab is /.
$ cat /etc/fstab
15ea7c8e8a859e25.a / ffs rw 1 1
bfb4775bb8397569.b /altroot ffs xx 0 0
15ea7c8e8a859e25.f /mfs ffs ro,nodev,nosuid 1 2
15ea7c8e8a859e25.d /usr ffs ro,nodev 1 2
15ea7c8e8a859e25.e /usr/X11R6 ffs ro,nodev 1 2
15ea7c8e8a859e25.g /usr/local ffs ro,wxallowed,nodev 1 2
15ea7c8e8a859e25.h /opt ffs ro,wxallowed,nodev 1 2
swap /tmp mfs rw,nodev,nosuid,-s256m 0 0
swap /var mfs rw,nodev,nosuid,-s256m,-P=/mfs/var 0 0
swap /home mfs rw,nodev,nosuid,-s1g,-P=/mfs/home 0 0
With the above context, I can explain the /etc/rc.shutdown. Filesystems start out as read-only, with the memory filesystems mounted. I may mount them read-write for whatever reason, like when I am installing packages. In case I did not switch them back to read-only, that happens at shutdown. After than, the memory filesystems are saved to /mfs.
$ cat /etc/rc.shutdown
save() {
local dir="$1"
/usr/local/bin/rsync -ax --quiet --delete --no-specials --no-devices "${dir}/" "/mfs${dir}"
}
# In case they are mounted read-write, mount read-only.
for fs in /usr/local /usr/X11R6 /usr /opt; do
mount -ur $fs
done
# Save memory file systems.
mount -uw /mfs
save /var
save /home
mount -ur /mfs
It may seem unnecessary to mount the filesystems as read-only right before shutdown. Before shutdown this is not so important, but I use the same file in other places, and it is important in those places.
For example, I save my work periodically by typing "save" in the shell to run /etc/rc.shutdown.
$ fgrep /etc/rc.shutdown ~/.tcshrc
alias save doas sh /etc/rc.shutdown
I have frequently had computers fail to suspend when I /etc/rc.shutdown, so I have configured apmd to run it too.
$ ls -lh /etc/apm
total 0
lrwxrwxr-x 1 rmarrone wheel 16B Nov 19 11:58 hibernate -> /etc/rc.shutdown
lrwxrwxr-x 1 rmarrone wheel 13B Nov 19 11:58 resume -> /etc/rc.local
lrwxrwxr-x 1 rmarrone wheel 16B Nov 19 11:59 standby -> /etc/rc.shutdown
lrwxrwxr-x 1 rmarrone wheel 16B Nov 19 11:59 suspend -> /etc/rc.shutdown
The dmesg could be interesting as well.
$ cat /var/run/dmesg.boot
OpenBSD 6.6 (GENERIC.MP) #372: Sat Oct 12 10:56:27 MDT 2019
deraadt@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP
real mem = 3951247360 (3768MB)
avail mem = 3818782720 (3641MB)
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.7 @ 0xbcd3d000 (60 entries)
bios0: vendor LENOVO version "GIET75WW (2.25 )" date 06/24/2014
bios0: LENOVO 20AMS1KV00
acpi0 at bios0: ACPI 5.0
acpi0: sleep states S0 S3 S4 S5
acpi0: tables DSDT FACP SLIC DBGP ECDT HPET APIC MCFG SSDT SSDT SSDT SSDT SSDT SSDT SSDT SSDT PCCT SSDT TCPA UEFI POAT ASF! BATB FPDT UEFI DMAR
acpi0: wakeup devices LID_(S4) SLPB(S3) IGBE(S4) EXP2(S4) XHCI(S3) EHC1(S3)
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpiec0 at acpi0
acpihpet0 at acpi0: 14318179 Hz
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: Intel(R) Core(TM) i7-4600U CPU @ 2.10GHz, 798.26 MHz, 06-45-01
cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,PERF,ITSC,FSGSBASE,TSC_ADJUST,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
cpu0: 256KB 64b/line 8-way L2 cache
cpu0: smt 0, core 0, package 0
mtrr: Pentium Pro MTRR support, 10 var ranges, 88 fixed ranges
cpu0: apic clock running at 99MHz
cpu0: mwait min=64, max=64, C-substates=0.2.1.2.4.1.1.1, IBE
cpu1 at mainbus0: apid 1 (application processor)
cpu1: Intel(R) Core(TM) i7-4600U CPU @ 2.10GHz, 798.16 MHz, 06-45-01
cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,PERF,ITSC,FSGSBASE,TSC_ADJUST,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
cpu1: 256KB 64b/line 8-way L2 cache
cpu1: smt 1, core 0, package 0
cpu2 at mainbus0: apid 2 (application processor)
cpu2: Intel(R) Core(TM) i7-4600U CPU @ 2.10GHz, 798.17 MHz, 06-45-01
cpu2: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,PERF,ITSC,FSGSBASE,TSC_ADJUST,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
cpu2: 256KB 64b/line 8-way L2 cache
cpu2: smt 0, core 1, package 0
cpu3 at mainbus0: apid 3 (application processor)
cpu3: Intel(R) Core(TM) i7-4600U CPU @ 2.10GHz, 798.16 MHz, 06-45-01
cpu3: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,PERF,ITSC,FSGSBASE,TSC_ADJUST,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
cpu3: 256KB 64b/line 8-way L2 cache
cpu3: smt 1, core 1, package 0
ioapic0 at mainbus0: apid 2 pa 0xfec00000, version 20, 40 pins
acpimcfg0 at acpi0
acpimcfg0: addr 0xf8000000, bus 0-63
acpiprt0 at acpi0: bus 0 (PCI0)
acpiprt1 at acpi0: bus -1 (PEG_)
acpiprt2 at acpi0: bus 2 (EXP1)
acpiprt3 at acpi0: bus -1 (EXP2)
acpiprt4 at acpi0: bus -1 (EXP3)
acpicpu0 at acpi0: C3(200@506 mwait.1@0x60), C2(200@148 mwait.1@0x33), C1(1000@1 mwait.1), PSS
acpicpu1 at acpi0: C3(200@506 mwait.1@0x60), C2(200@148 mwait.1@0x33), C1(1000@1 mwait.1), PSS
acpicpu2 at acpi0: C3(200@506 mwait.1@0x60), C2(200@148 mwait.1@0x33), C1(1000@1 mwait.1), PSS
acpicpu3 at acpi0: C3(200@506 mwait.1@0x60), C2(200@148 mwait.1@0x33), C1(1000@1 mwait.1), PSS
acpipwrres0 at acpi0: PUBS, resource for XHCI, EHC1
acpitz0 at acpi0: critical temperature is 200 degC
acpibtn0 at acpi0: LID_
acpibtn1 at acpi0: SLPB
acpipci0 at acpi0 PCI0: 0x00000000 0x00000011 0x00000001
acpicmos0 at acpi0
tpm0 at acpi0: TPM_ addr 0xfed40000/0x5000, device 0x0000104a rev 0x4e
acpibat0 at acpi0: BAT0 model "45N1773" serial 15792 type LION oem "SANYO"
acpibat1 at acpi0: BAT1 model "45N1137" serial 102 type LION oem "SANYO"
acpiac0 at acpi0: AC unit offline
acpithinkpad0 at acpi0
"PNP0C14" at acpi0 not configured
"PNP0C14" at acpi0 not configured
"PNP0C14" at acpi0 not configured
"INT340F" at acpi0 not configured
acpivideo0 at acpi0: VID_
acpivout at acpivideo0 not configured
acpivideo1 at acpi0: VID_
cpu0: using VERW MDS workaround (except on vmm entry)
cpu0: Enhanced SpeedStep 798 MHz: speeds: 2701, 2700, 2600, 2400, 2300, 2100, 2000, 1800, 1700, 1600, 1400, 1300, 1100, 1000, 800, 756 MHz
pci0 at mainbus0 bus 0
pchb0 at pci0 dev 0 function 0 "Intel Core 4G Host" rev 0x0b
inteldrm0 at pci0 dev 2 function 0 "Intel HD Graphics" rev 0x0b
drm0 at inteldrm0
inteldrm0: msi
azalia0 at pci0 dev 3 function 0 "Intel Core 4G HD Audio" rev 0x0b: msi
azalia0: No codecs found
xhci0 at pci0 dev 20 function 0 "Intel 8 Series xHCI" rev 0x04: msi, xHCI 1.0
usb0 at xhci0: USB revision 3.0
uhub0 at usb0 configuration 1 interface 0 "Intel xHCI root hub" rev 3.00/1.00 addr 1
"Intel 8 Series MEI" rev 0x04 at pci0 dev 22 function 0 not configured
em0 at pci0 dev 25 function 0 "Intel I218-LM" rev 0x04: msi, address 28:d2:44:e4:f9:ac
azalia1 at pci0 dev 27 function 0 "Intel 8 Series HD Audio" rev 0x04: msi
azalia1: codecs: Realtek ALC292
audio0 at azalia1
ppb0 at pci0 dev 28 function 0 "Intel 8 Series PCIE" rev 0xe4: msi
pci1 at ppb0 bus 2
rtsx0 at pci1 dev 0 function 0 "Realtek RTS5227 Card Reader" rev 0x01: msi
sdmmc0 at rtsx0: 4-bit, dma
ehci0 at pci0 dev 29 function 0 "Intel 8 Series USB" rev 0x04: apic 2 int 23
usb1 at ehci0: USB revision 2.0
uhub1 at usb1 configuration 1 interface 0 "Intel EHCI root hub" rev 2.00/1.00 addr 1
pcib0 at pci0 dev 31 function 0 "Intel 8 Series LPC" rev 0x04
ahci0 at pci0 dev 31 function 2 "Intel 8 Series AHCI" rev 0x04: msi, AHCI 1.3
ahci0: port 0: 6.0Gb/s
scsibus1 at ahci0: 32 targets
sd0 at scsibus1 targ 0 lun 0: <ATA, SAMSUNG MZ7PD256, DXM0> naa.5002538500000000
sd0: 244198MB, 512 bytes/sector, 500118192 sectors, thin
ichiic0 at pci0 dev 31 function 3 "Intel 8 Series SMBus" rev 0x04: apic 2 int 18
iic0 at ichiic0
spdmem0 at iic0 addr 0x50: 4GB DDR3 SDRAM PC3-12800 SO-DIMM
isa0 at pcib0
isadma0 at isa0
pckbc0 at isa0 port 0x60/5 irq 1 irq 12
pckbd0 at pckbc0 (kbd slot)
wskbd0 at pckbd0: console keyboard
pms0 at pckbc0 (aux slot)
wsmouse0 at pms0 mux 0
wsmouse1 at pms0 mux 0
pms0: Synaptics clickpad, firmware 8.1, 0x1e2b1 0x940300
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
vmm0 at mainbus0: VMX/EPT
umodem0 at uhub0 port 4 configuration 1 interface 1 "Lenovo N5321 gw" rev 2.00/0.00 addr 2
umodem0: data interface 2, has CM over data, has break
umodem0: status change notification available
ucom0 at umodem0
umodem1 at uhub0 port 4 configuration 1 interface 3 "Lenovo N5321 gw" rev 2.00/0.00 addr 2
umodem1: data interface 4, has CM over data, has break
umodem1: status change notification available
ucom1 at umodem1
umb0 at uhub0 port 4 configuration 1 interface 6 "Lenovo N5321 gw" rev 2.00/0.00 addr 2
umodem2 at uhub0 port 4 configuration 1 interface 9 "Lenovo N5321 gw" rev 2.00/0.00 addr 2
umodem2: data interface 10, has CM over data, has break
umodem2: status change notification available
ucom2 at umodem2
ugen0 at uhub0 port 4 configuration 1 "Lenovo N5321 gw" rev 2.00/0.00 addr 2
uvideo0 at uhub0 port 8 configuration 1 interface 0 "J31E8IAE0 Integrated Camera" rev 2.00/10.04 addr 3
video0 at uvideo0
umass0 at uhub0 port 11 configuration 1 interface 0 "Generic USB3.0 Card Reader" rev 3.00/15.32 addr 4
umass0: using SCSI over Bulk-Only
scsibus2 at umass0: 2 targets, initiator 0
sd1 at scsibus2 targ 1 lun 0: <Generic, STORAGE DEVICE, 1532> removable serial.05e30749000000001532
sd1: 1876MB, 512 bytes/sector, 3842048 sectors
uhub2 at uhub1 port 1 configuration 1 interface 0 "Intel Rate Matching Hub" rev 2.00/0.04 addr 2
vscsi0 at root
scsibus3 at vscsi0: 256 targets
softraid0 at root
scsibus4 at softraid0: 256 targets
sd2 at scsibus4 targ 1 lun 0: <OPENBSD, SR CRYPTO, 006>
sd2: 49159MB, 512 bytes/sector, 100678763 sectors
root on sd2a (15ea7c8e8a859e25.a) swap on sd2b dump on sd2b
inteldrm0: 1366x768, 32bpp
wsdisplay0 at inteldrm0 mux 1: console (std, vt100 emulation), using wskbd0
wsdisplay0: screen 1-5 added (std, vt100 emulation)
Final notes
I did some other things to reduce the risk of power loss and consequent problems with softraid CRYPTO.
- I removed the wifi card, to save power.
- I purposely chose this laptop for its two batteries. This makes for long battery life and for a reduced risk of batteries falling out, especially because one of them is inside the case.
- I purposely got a laptop with an i7 processor, as those use less power than the other options for this laptop. I don't know if that matters so much though.
It was easy for me to set this up, and I have found it very convenient to use. I am no longer worried about what will happen if I lose power on my laptop.