This year marks the 20th anniversary of the release of Pixar's computer-animated, Academy-Award winning film, Toy Story. Originally released in 1995, Toy Story was succeeded by two additional films that continued to follow the adventures of Andy's toys as the little boy matured into a young man. What you may not know, however, is that Andy's journey to adulthood was almost cut short, with little hope of Woody, Buzz or a slice of Pizza Planet's finest saving the day.
Have you ever seen data vanish into thin air? For the creators of Toy Story 2, panic struck the Pixar offices when the movie was almost permanently deleted due to a rogue command and improper backup and disaster recovery (BDR) practices.
Why the Makers of Toy Story 2 Needed Backup
Someone's poisoned the water hole...and the script!
Usually, the kind of script people refer to when discussing Tinsel Town's latest production is the collection of lines actors must recite, not a single line of code. If there's an issue with a movie script, you bring it up with the screenwriter. If there's an issue with a command script, you may not have a movie to release at all...
- there was an error in the script, and the wrong command launched, unbeknownst to the crew
- the movie creators noticed that whole sections of the movie were being deleted
- the team figured they could recover the files from their back up
- ...but found out their backups had failed for the past month
- one of the employees remembered she had copied files to her home computer, which was stored offsite
- they were able to recover the data and the movie was saved
It began innocently enough. The rm* UNIX command - believed to be /bin/rm -r -f * in its entirety - was accidentally run on the root level, the drive where the movie files were stored. Shortly after the "remove" code was deployed, those working on Toy Story 2 noticed their artistic creations disappearing right before their very eyes, beginning with Woody's hat and quickly escalating into complete sequences from the film. Before any more damage could be done, the plug was pulled on the movie's master machine. At that point, 90% of the movie had been erased, sending the crew into a tailspin to salvage their project.
Forgetting about Your Backed up Data
If you've seen any of the three Toy Story movies, you know that a central theme weaved throughout each is the idea of toys being forgotten about and left behind. In the first installment, Woody is threatened by the arrival of Buzz Lightyear, a shiny new space ranger action figure vying for Andy's attention. Viewers again witness a story of heartbreaking toy neglect in the sequel, as newcomer, Jessie's backstory is told through the haunting vocals of Sarah McLachlan. During the sequence, we learn that Jessie's kid, Emily, grew out of wanting to play with her doll, instead opting for makeup and beauty products. Finally, we all feel the same anxiety as Andy's toys in Toy Story 3, when an unfortunate mix-up leads them to believe their beloved owner threw them away. The film begins with an elaborate mission to get Andy to play with his toys after they hijack and call his cell phone and ends with the same treasured characters finally getting one last play session in with their kid before he leaves for college.
More than make the inner child in you feel guilty for discarding your own toys and replacing them with fancy gadgets, these examples should illustrate what's at stake when you treat backed up data like something to just throw in a bin and never revisit.
The Cost of Failing to Regularly Test Backups
Data loss is a horrifying reality for you and the businesses you serve. Understanding this, many of you have ventured into the disaster-recovery-as-a-service (DRaaS) market and now offer backup for clients as an additional line of revenue. Still, depending on the client, it's not enough to simply back up now and then. You have to regularly take and test your backups to make sure they're working so you don't lose all of the data accumulated between recovery points.
The makers of Toy Story 2 had to learn this the hard way. For them, the cost was the equivalent of 20-30 people working for a full year, only to have their efforts deleted within 20 seconds. While they believed they only would have had to reassemble half a day's work worth of film, as they quickly realized, their backups had failed for the last month. It was good they had established a schedule of frequent backups for such a highly-anticipated project, however the whole ordeal proves that backups are useless if they're ineffective. The only way to verify you can recover data is by testing that your backups were successfully taken. But wait, don't fall into the trap of believing that if you can restore one file, you can restore all files. As a pro-tip from TechTarget, check "the directories to make sure everything that should be backed up is actually backed up." Leave no folder unturned.
Accidental Site-to-Site Disaster Recovery to the Rescue!
When we last left our frantic Pixar team, they had been scrambling to restore what they could of Toy Story 2, faced with the grim reality that they'd basically have to revisit the drawing board. How then was the movie almost finished by its deadline? Offsite replication. In order to take care of her children, one of the movie's technical directors had worked from home and had the copied film stored on her personal computer. Just when the outlook was dire and they thought the data was beyond saving, the crew was able to recover the movie from an offsite location. Sound familiar?
As hosting providers, you know the slightest mistake can compromise or wipe out your clients' data. Even if you or your end user have the best of intentions, one wrong line of script or one irreversible moment of human error can shut everything down. Game over. Because you recognize this, you understand that the only way to maintain business continuity is to offer BDR. But what happens if a backup fails locally, as it did in this Toy Story 2 example?
Always make sure you have redundant data backed up at a separate site should your on-premise storage malfunction. Look for a server backup solution that lets you configure the replication of data from one server to the next, a feature known as site-to-site disaster recovery. We actually offer this in our Server Backup Manager (SBM) product, which enables you to use replicated Disk Safes to directly restore protected servers.
The creative geniuses behind Toy Story 2 almost lost everything because they didn't have the right data loss defenses in place. If you take away anything from their data recovery struggle, review what backup technologies (if any) your clients have in place, offer your BDR solution to those who need it, test that all backups work and all necessary data can be recovered and build a policy of redundant cloud storage for all client data.
Only then, will your clients truly have a friend in you.
Have you tuned in to our backup webinar?