Originally posted on the DataMungeBlog and Data Tactics Corporation Blog by Eric Whyne.
Code versioning repositories are clingy with your data. It's almost impossible to lose data when using them. In the case of distributed version control systems like Git, the entire directory tree and all of it's files are replicated to each computer that has cloned the repository. Every single computer interacting with the repo would have to be wiped or destroyed in order for code to be lost.
Document versioning repositories are also becoming ever more popular. Sharepoint, Dropbox, Google Drive, Office 365, and pretty much any other modern document handling system now does revision control automatically. So the benefits aren't just limited to developers and their code anymore. However, if you're dealing with code or documents that are classified or proprietary this can be problematic.
A common problem arises when users submitting open source code to GitHub accidentally publish their private cryptography keys to the public repository. Now in this case, the toothpaste is out of the tube. The only responsible course of action is to change the keys and issue a revocation if they are asymmetric keys. But in cases where you have accidentally published sensitive information and want to remove that information from the history, things get complicated.
https://help.github.com/articles/remove-sensitive-data People post sensitive documents to the wrong place all the time. It happens.
If you're in a situation where this is a risk, there are a few things you can do to make life easier and to help everyone sleep better at night. Do a rock drill. Rock drills are emulating a hypothetical scenario and moving as closely as is possible through the actions you'd take in response to it if it were to actually happen. This is more than just thinking about and talking through the response, it means doing the actions on the actual systems.
When I was a USMC Captain running multi-vehicle convoys and patrols in Iraq 2007-2008, I'd do a rock drill after almost every run. As soon as we pulled onto a safe base “behind the wire”, I'd direct the convoy to an open area, park us in whatever formation we were moving in an then I'd announce over the radio that one of the vehicles was notionally disabled or that some other crisis had happened. Four years into the Iraq War, Improvised Explosive Devices were a common occurrence as were broken vehicles. During my tour as part of a small team advising the Iraqi Army for a year, we probably saw every possible real-life scenario. A common and often confusing series of actions was to remove all personnel and important equipment from a vehicle and distribute it across the remaining vehicles. This required multiple people accounting for disparate gear and finding spare room across multiple vehicles. After mobility was re-established then we'd move through the proper reactions for each scenario which usually involved shifting the vehicles into another formation depending on the situation and terrain. There were exceptions of course, in the case of a response to ambush often redistributing equipment and personnel is not the first priority; mitigating the ambush is. Often initial damage from IEDs would be minimal, but vehicle tires would catch on fire and consume everything in the vehicle within minutes as shown in the picture below. Things can get complicated, but I always kept the drills simple and limited to only one scenario each run.
Doing this rock drill immediately after a live combat convoy ensured that the training was as realistic as possible. Our gear and number of personnel was exactly what we were just running an operation with. Everybody was tired, hungry, and usually stressed. The thing I remember most about these patrols is that even after the most boring and short patrols I'd be absolutely exhausted when I got back to safety. When the adrenaline and tension left my body it would feel like getting hit by a truck. Sore and weak muscles, sleepiness, my body armor and weapon would feel like they were suddenly made of lead; everything was drained without doing much physical activity.
To get the most benefit I'd always pick the vehicle that I thought had the best chance of screwing it up. I'd pick the vehicle that was transporting the cryptography keys for the radios, or that had a type of ordnance or gear that was easily forgotten. Or... I'd do it to the vehicle with the personnel that I just plain thought would mess it up. I'd pick on both the new guys that didn't know better yet and I'd do it to the Marines that had been in combat for months and had grown complacent. This ensured weakest link would get stronger after every patrol.
|Soft dirt, heavy vehicle.|
The logistics and training of combat patrols has nothing directly to do with our versioned repository problem. But the principle of periodically running a rock drill can ensure that if you do have a spillage onto one of these platforms you can react quickly. Periodic practice will ensure that people know what to do. Once you identify the problem, if you don't take immediate action in some cases the spillage gets distributed to every user that decides to do a clone or update from the repository. The longer your reaction takes the worse the problem gets. Just like during those patrols, reaction time matters.
Now I never treated my rock drills as a test where I withheld information. I treated it like the real thing, just with more humor and levity. I was there and I was giving direction and instructing on what needed to happen just like I would be and had been in the real life situations. Where I saw weakness I'd immediately address it by patient instruction. Where I saw brilliance, I pointed it out to the others. I think if I had treated it more as a test the Marines would have considered it harassment and not a learning opportunity. The most important concept to adhere to during group training is to “praise in public and punish in private”. Even when pointing out mistakes for the benefit of the group I never blamed the individual in public. Shame is a powerful force and I think that truly hurting someones pride will only ever result in decreased performance, not increased. You could be on your way to the stage to get a Nobel Prize for cancer research, but if you trip on the stage stairs all you'll feel is shame. Shame is more powerful to us than praise every time, vastly so. It should be used proportionate to it's power and so rarely as to be avoided altogether in my opinion.
In an office environment we have a little more control of the conditions than I did practising with tired Marines on abandoned parts of Iraq bases. We can often practice with the exact equipment and exact scenarios that we are preparing to face and do everything but spill actual sensitive data. On development teams, we always seem to have one or two rock stars that can easily accomplish anything you throw at them. The first time you do a rock drill, let them be there and shine. The next time you pull one, do it when they are on vacation or out to lunch. Exercise the team, but be respectful of their time too. Even though real life situations may occur on weekends or after hours, you don't need that much realism to respond to an event. Practice during normal hours works just fine to make sure people are ready to respond after hours, and they'll be less likely to think somebody is crying wolf during a real event. And in some rare cases the volume of events is such that rock drills aren't even necessary because your responses are already regularly getting exercised anyway.
The steady advance of software brings with it wonderful new capability. Sometimes, as in the case, that capability can work against us. But realizing the risks and taking measures to be prepared can greatly reduce the consequences of human and machine error.