Applications Must Ensure Data Integrity
This is part 3 of a three part post: Part 1, Part 2. In the last 2 posts I demonstrated how file systems on Windows and Linux do little to help with data integrity in a crash. This means real data integrity for actual file contents is left up to applications.
The most robust form of data integrity is called ACID. Atomicity, Consistency, Isolation, and Durability.
Atomicity the file write or operation is all or none. If it is interrupted the file(s) is returned to its previous state.
Consistency guarantees that a write in the application never leaves your file(s) in a half-finished state.
Isolation keeps write operations separated from each other until they’re finished.
Durability guarantees that the application will keep track of pending changes in such a way that the server can recover from an abnormal termination (e.g. crash or power loss).
The most important is Atomicity, Durability and Consistency. Most applications need to support these 3 parts of the ACID test. Depending on the application Isolation may or may not be important. For a relational database system for example they are usually all important. For a mail server… probably only A, D, and C is important.
Do all applications implement their own protection to guarantee the integrity of file contents?
Some do. Many don’t. It depends on the application. Here are some examples.
Examples of applications that Do have atomic Writes
Here are examples of application that have Atomic, Durable and Consistent file writes. Be aware that all these applications depend on correctly functioning hardware and operating systems.
MS SQL Server – SQL Server uses write-ahead logging for to provide excellent protection from crashes or power loss during writes. SQL Server is also ACID compliant. For excellent details on SQL Server Atomic writes see: http://support.microsoft.com/kb/230785
R1Soft CDP 3.0 for Windows & Linux – The new Disk Safe in CDP 3.0 meets the ACID test. CDP 3 uses a write-ahead journal file to ensure Disk Safes are always recoverable from crashes or power loss. CDP 3 is fully ACID compliant and in my humble opinion as durable and consistent as any application can possibly be.A crash or power loss after a large initial replica can typically roll-back in under 2-3 minutes when the application restarts.
CDP 3 is so tough we typically never shut it down cleanly when we are doing development on it or testing it. We typically just hard kill the application right in the middle of replications… all the time… it’s just normal for CDP 3 to take that kind of abuse. Still… I warn… your hardware must be sound.
By the way we have shipped Alpha test versions of CDP 3 Standard Editions to qualified CDP 2 customers to take for a spin.
MS Exchange Server (Jet Database)– Exchange is fully ACID compliant. A checkpoint file is used to keep track of changed database pages during the Jet databases concept of transactions called “checkpoints”. Read more here: http://support.microsoft.com/kb/271987
Examples of applications that do Not support atomic writes :
FTP / SFTP / scp (Windows & Linux) – Network file copy utilities –A crash of your server in the middle of overwriting a file with FTP or SFTP can corrupt the file
Tape Backup Software (Symantec, Tivoli, GNU tar, Bacula, Zmanda) – crash your tape backup server in the middle of writing to tape and you just corrupted your tape backup. This should need no explanation.
Many Linux Email Servers – The e-mails sitting on most Linux mail servers. Postfix / Sendmail mail spool files are not protected from corruption caused by crashes or unclean shutdowns. In fact Postfix attempts to detect corrupt mail spools and moves them to /var/spool/postfix/corrupt by default. It’s worth noting Qmail style Maildir format has built-in atomic mail delivery that is typically safe from crashes and power loss.
Acronis True Image on Windows & Linux (TIB files) – Acronis True Image file data corruption usually occurs when the software application or the operating system crashes while the .TIB data file is open in memory. In most cases, headers or parts of the data file are not saved to disk causing the data corruption or the application to fail. To my knowledge there is no repair utility and no attempt for protection from crashes or power loss. This is a big issue if you depend on True Image for backups e.g. the chained incremental file backup it supports.
MySQL on Windows & Linux. If MySQL is writing to MyISAM tables during a power loss or crash there is a very good chance they will become corrupted. Corruption can go unnoticed until the corrupted portion of the database is needed. Corrupted MyISAM tables can cause MySQL to crash or hang cascading into further database corruption. MySQL provides the myisam_recover option which checks MyISAM tables that were not cleanly closed and checks may trigger a repair. The MyISAM repair process at a high level is similar to a file system being meta-data consistent. The table structure is sound but row content may still be corrupt.
MySQL DDL Corruption. Even the MySQL InnoDB storage engine which implements ACID compliant transactions and data integrity is not truly ACID compliant. If MySQL crashes or looses power while a Database Definition Language statement is issued like CREATE TABLE or ALTER TABLE even InnoDB can be corrupted.For more on MySQL crash recovery I highly recommend the book “High Performance MySQL”. Also check out the MySQL Performance Blog www.mysqlperformanceblog.com
R1Soft CDP Server 2 -The Disk Safe in CDP 2 does periodic flushes to disk at critical moments when writing (fsync() on Linux and FlushFileBuffers on Windows). There is still a good chance of corruption from crash or power failure if a Disk Safe is being written to during the failure. When a Disk Safe is opened for the first time an unclean shutdown is automatically detected triggering a Disk Safe Repair. The Disk Safe repair is very similar in concept to MySQL MyISAM repair that fixes Disk Safe structure and in many cases can completely recovery from a failure. It repairs pages of the data and index Disk Safe files that were being written to and have not been synced to Disk if there is a crash. The repair process is not perfect.
Why don’t all applications have built-in crash protection?
That’s easy. It’s very complicated and expensive to develop and maintain. I know from experience.
How do I know if application XYZ is safe from crashes and is Atomic, Durable, and Consistent?
The easiest way is to ask the application vendor or developer. You may get a confused look, thats OK, this is a complicated subject. One tactic is to ask the vendor what happens if the computer losses power in the middle of writing to the XYZ file. If the answer is “repair”, then likely it is not fully atomic. That doesn’t mean it’s a bad application, it just means you need to considered that and be aware.
Many applications use a relational database system to store data, and if that is the case your job is a little easier. Look at the relational databases the application supports. For example many applications support MySQL and SQL Server and Oracle and…. You have a choice… other times its just MS SQL Server or just MySQL.
Special Notes for MySQL Durability
If you are like me you are probably using MySQL everywhere. When using MySQL as the data store for your application make sure you consult with your database manager or consultant to ensure the InnoDB storage engine is being used. This can be tricky as for some strange reason InnoDB is often disabled and MySQL happily defaults to MyISAM when this happens. Most MySQL application also do not do a good job of warning you they are using MyISAM.
Beware of MySQL compressed tables. Many applications like popular forum apps compress tables without you knowing. Compressed MyISAM tables are extra sensitive to unclean shutdowns and crashes. Do not corrupt these. If you do and you don’t have backups you are toast.