File systems organize data storage on a disk. Sometimes, file systems may be corrupted hence making them inaccessible. This may cause systems to fail boot or services to fail starting.
When a file system is corrupt, sometimes it changes its mode to Read-Only mode. This means it becomes impossible to perform any action on the file system e.g copy, move or remove data. Also, services e.g apache or any other service running on the file system will fail to start. This is because these services normally need to write events on a log file as they start and a system being in read only mode means files cannot be written on..
One circumstance this can occur is after replacing a defective disk on a RAID system. The new disk may have Inode inconsistency or other issues with its new RAID disk partner, leading to the above problems.
To fix these, you need to run a file system check. The steps below show how to achieve this
1.Boot your server into Rescue Mode.
2. Check partitions names to determine which one to fix.
root@rescue ~ # lsblk
3. Run the command below to check that all partitions have a valid partition table
root@rescue ~ # parted /dev/sda print all
Change /dev/sda with your partition name
If the output is something like this, it means the partition needs to be assigned a partition table.
root@rescue ~ # parted /dev/sda print all Error: /dev/sda: unrecognised disk label Model: ATA INTEL SSDSC2KW25 (scsi) Disk /dev/sda: 256GB Sector size (logical/physical): 512B/512B Partition Table: unknown Disk Flags:
Model: ATA HGST HUS726020AL (scsi) Disk /dev/sdb: 2000GB *******more content below*******
4. Fix the error above by running the commands below
root@rescue ~ # parted /dev/sda
(parted) unit GB
(parted) mklabel msdos
(parted) mkpart primary 0 -1
(parted) quit
Now when you run the command in step three above, the output will be as follows
root@rescue ~ # parted /dev/sda print all
Model: ATA INTEL SSDSC2KW25 (scsi)
Disk /dev/sda: 256GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags:
Number Start End Size Type File system Flags
1 1049kB 255GB 255GB primary
Model: ATA HGST HUS726020AL (scsi) Disk /dev/sdb: 2000GB Sector size (logical/physical): 512B/4096B ******more details below*******
- Now, you can proceed to run a file system check on any partition that is in Read Only mode as follows
root@rescue ~ # fsck -fy /dev/md2
Replace /dev/md2 with your LVM or Partition Name
6. Once done, boot your server back to Normal Mode.
Automatic filesystem check #
To perform automatic file system check you can run the following commands to configure
root@rescue ~ # tune2fs -c 3 /dev/md2 #[where 3 is the number of boot processes in between the checks]
root@rescue ~ # tune2fs -i 1m /dev/md2
You can repeat those two commands for all your partitions