Internet forums are full of stories about SD card corruption and data loss when running Raspberry PI. It happened to me as well and obviously I wasn't very happy about that. It took me a while to recover all the data from backups and make the PI running again. I decided to find the way how to avoid this problem in the future or at least reduce the chances of card being corrupted.
The overall problem is called SD card wear out. The main point is flash memory media have erasable cells, each of which can be put through a limited number of erase cycles before becoming faulty. Usually it takes thousands of cycles which may look like a huge number, but it depends what are you doing with you Raspberry. If your app is not designed carefully or worse you're using persistent database, thousands of erase cycles can be depleted within few days.
I came across this problem in discussion about Monitoring internet connection with Smokeping. Smokeping uses RRDTool as internal database for measured data. And since the monitoring is continuous process it writes a new data to SD card every few minutes. That can contribute to SD card wear out pretty significantly.
It's crucial to reduce write operations to SD card. The obvious solution is to use RAM disk or tmpfs (i. e. file system that stores all the files in volatile memory instead of persistent storage). But problems will arise when you realize you'll lose all the data during reboot. For example if we move /var/log to tmpfs we'll always start with empty directory after reboot. That won't work very well since certain apps expect some specific directory structure within /var/log and they'll crash at startup.
More advanced solution is to combine tmpfs with persistent storage effectively creating persistent ramdisk. So the life-cycle of data will look like this:
- After boot data from chosen directory is copied from SD card to tmpfs
- When data is modified it happens at no cost in RAM
- To prevent data loss during electricity outage or random restart there is hourly back-up made to SD card
- When properly shutting down all actual data from tmpfs are copied back to SD card
Anything-sync-daemon
I found a great tool called Anything-sync-daemon written in bash that does exactly what we need – maps chosen directories to tmpfs and syncs changes back to SD card when needed.
Unfortunately Anything-sync-daemon is available only for Arch linux so I decided to compile and build my own package for Debian/Raspbian 8 Jessie.
Anything-sync-daemon package for Debian
All commands need to be run as root, thus sudo su
Install required dependencies for building a package and rsync
apt-get install git build-essential checkinstall rsync
Clone the current source code of Anything-sync-daemon from Github
git clone https://github.com/graysky2/anything-sync-daemon.git
Open the source code directory and checkout the current version
cd anything-sync-daemon/ && git checkout v5.81
Build the Debian package and install it
make && checkinstall --pkgversion 5.81 make install-systemd-all
Now it's time to setup directories that need to be managed by asd. Edit /etc/asd.conf
WHATTOSYNC=('/var/lib/smokeping' '/var/log')
VOLATILE="/dev/shm"
- WHATTOSYNC – List of directories that are mapped to RAM
- VOLATILE – Path of the currently mounted volatile memory (tmpfs)
systemctl enable asd.service && service asd restart
Debugging Anything-sync-daemon
There is an option to print the current configuration and state of monitored directories# asd p Anything-sync-daemon v5.81 on Raspbian GNU/Linux 8 (jessie) Systemd service is currently active. Systemd resync service is currently active. Overlayfs technology is currently inactive. Asd will manage the following per /run/asd.conf settings: owner/group id: smokeping/115 target to manage: /var/lib/smokeping sync target: /var/lib/.smokeping-backup_asd tmpfs target: /dev/shm/asd-smokeping/var/lib/smokeping dir size: 12M recovery dirs: none
