Thursday, June 20, 2013

Sensitive Data in Versioned Repositories

Originally posted on the DataMungeBlog and Data Tactics Corporation Blog by Eric Whyne.

Code versioning repositories are clingy with your data. It's almost impossible to lose data when using them. In the case of distributed version control systems like Git, the entire directory tree and all of it's files are replicated to each computer that has cloned the repository. Every single computer interacting with the repo would have to be wiped or destroyed in order for code to be lost.

Document versioning repositories are also becoming ever more popular. Sharepoint, Dropbox, Google Drive, Office 365, and pretty much any other modern document handling system now does revision control automatically. So the benefits aren't just limited to developers and their code anymore. However, if you're dealing with code or documents that are classified or proprietary this can be problematic.

A common problem arises when users submitting open source code to GitHub accidentally publish their private cryptography keys to the public repository. Now in this case, the toothpaste is out of the tube. The only responsible course of action is to change the keys and issue a revocation if they are asymmetric keys. But in cases where you have accidentally published sensitive information and want to remove that information from the history, things get complicated. People post sensitive documents to the wrong place all the time. It happens.

If you're in a situation where this is a risk, there are a few things you can do to make life easier and to help everyone sleep better at night. Do a rock drill. Rock drills are emulating a hypothetical scenario and moving as closely as is possible through the actions you'd take in response to it if it were to actually happen. This is more than just thinking about and talking through the response, it means doing the actions on the actual systems.

Good day.
When I was a USMC Captain running multi-vehicle convoys and patrols in Iraq 2007-2008, I'd do a rock drill after almost every run. As soon as we pulled onto a safe base “behind the wire”, I'd direct the convoy to an open area, park us in whatever formation we were moving in an then I'd announce over the radio that one of the vehicles was notionally disabled or that some other crisis had happened. Four years into the Iraq War, Improvised Explosive Devices were a common occurrence as were broken vehicles. During my tour as part of a small team advising the Iraqi Army for a year, we probably saw every possible real-life scenario. A common and often confusing series of actions was to remove all personnel and important equipment from a vehicle and distribute it across the remaining vehicles. This required multiple people accounting for disparate gear and finding spare room across multiple vehicles. After mobility was re-established then we'd move through the proper reactions for each scenario which usually involved shifting the vehicles into another formation depending on the situation and terrain. There were exceptions of course, in the case of a response to ambush often redistributing equipment and personnel is not the first priority; mitigating the ambush is. Often initial damage from IEDs would be minimal, but vehicle tires would catch on fire and consume everything in the vehicle within minutes as shown in the picture below. Things can get complicated, but I always kept the drills simple and limited to only one scenario each run.
Bad day.

Doing this rock drill immediately after a live combat convoy ensured that the training was as realistic as possible. Our gear and number of personnel was exactly what we were just running an operation with. Everybody was tired, hungry, and usually stressed. The thing I remember most about these patrols is that even after the most boring and short patrols I'd be absolutely exhausted when I got back to safety. When the adrenaline and tension left my body it would feel like getting hit by a truck. Sore and weak muscles, sleepiness, my body armor and weapon would feel like they were suddenly made of lead; everything was drained without doing much physical activity.

Predictable damage.
To get the most benefit I'd always pick the vehicle that I thought had the best chance of screwing it up. I'd pick the vehicle that was transporting the cryptography keys for the radios, or that had a type of ordnance or gear that was easily forgotten. Or... I'd do it to the vehicle with the personnel that I just plain thought would mess it up. I'd pick on both the new guys that didn't know better yet and I'd do it to the Marines that had been in combat for months and had grown complacent. This ensured weakest link would get stronger after every patrol.

Soft dirt, heavy vehicle.
The logistics and training of combat patrols has nothing directly to do with our versioned repository problem. But the principle of periodically running a rock drill can ensure that if you do have a spillage onto one of these platforms you can react quickly.  Periodic practice will ensure that people know what to do. Once you identify the problem, if you don't take immediate action in some cases the spillage gets distributed to every user that decides to do a clone or update from the repository. The longer your reaction takes the worse the problem gets. Just like during those patrols, reaction time matters.

Now I never treated my rock drills as a test where I withheld information. I treated it like the real thing, just with more humor and levity. I was there and I was giving direction and instructing on what needed to happen just like I would be and had been in the real life situations. Where I saw weakness I'd immediately address it by patient instruction. Where I saw brilliance, I pointed it out to the others. I think if I had treated it more as a test the Marines would have considered it harassment and not a learning opportunity. The most important concept to adhere to during group training is to “praise in public and punish in private”. Even when pointing out mistakes for the benefit of the group I never blamed the individual in public. Shame is a powerful force and I think that truly hurting someones pride will only ever result in decreased performance, not increased. You could be on your way to the stage to get a Nobel Prize for cancer research, but if you trip on the stage stairs all you'll feel is shame. Shame is more powerful to us than praise every time, vastly so. It should be used proportionate to it's power and so rarely as to be avoided altogether in my opinion.

In an office environment we have a little more control of the conditions than I did practising with tired Marines on abandoned parts of Iraq bases. We can often practice with the exact equipment and exact scenarios that we are preparing to face and do everything but spill actual sensitive data. On development teams, we always seem to have one or two rock stars that can easily accomplish anything you throw at them. The first time you do a rock drill, let them be there and shine. The next time you pull one, do it when they are on vacation or out to lunch. Exercise the team, but be respectful of their time too. Even though real life situations may occur on weekends or after hours, you don't need that much realism to respond to an event. Practice during normal hours works just fine to make sure people are ready to respond after hours, and they'll be less likely to think somebody is crying wolf during a real event. And in some rare cases the volume of events is such that rock drills aren't even necessary because your responses are already regularly getting exercised anyway.

The steady advance of software brings with it wonderful new capability. Sometimes, as in the case, that capability can work against us. But realizing the risks and taking measures to be prepared can greatly reduce the consequences of human and machine error.

Wednesday, June 5, 2013

Relative Self-definition is Fatal

I've gotten in a few deep conversations lately with friends about what is more important: the "why" or the "what". As with most things, I tried to write it down here to attain some sort of clarity on my thoughts about this.

People have a natural tendency to organize themselves into hierarchies. Promotions at work, social status in high school, how much influence one has in a social club. Wherever we go, these structures seem to follow us around and are in large part necessary for the smooth functioning of any organization. Groups of people are best able to move in a single direction and accomplish significant things when there is a structure to the dissemination of intent. A Marine in training or combat lives and dies by this. I spent nine and a half years as a US Marine. Inside the hierarchy of military ranks, the linear passage of time and dependable performance ensure steady progression. My experiences there left me with the feeling that it’s a bad idea to define ourselves by our station in life. I think we should "do the job" and not "be the job". I'll explain more of what I mean in a bit.

I think there is a great danger to evaluating your success by your rank in an arbitrary hierarchy (wherever you happen to be at that time). When you embark on this crusade of relative self-evaluation you'll find yourself competing against others around you. You'll seek to evaluate your station by advancing against those above you while defending yourself against those beneath you in the hierarchy. Just the way I'm phrasing this has probably already struck a nerve with you. Our intuition abhors competing in this manner. But if you self-validate by looking at your station or "rank", this is exactly what you are doing. By using your rank in a hierarchy to fuel your self-image you'll find yourself acting toward others based upon their rank in the hierarchy and ignoring their other traits. You'll be laboring for the admiration of others, and by doing so you'll be binding yourself to their world view and their expectations of you. In essence, you willingly become a slave to them.

So how should we determine success? Success, in my mind, can be determined by how much you love what you are doing and the work that you are producing. Not love of self, which would be vain, but love of the product of your labor and the path that took you there. To evaluate ourselves in this manner requires internal reflection on what we are doing and why we are doing it. I think why we do something matters more than what we do. If you are constantly seeking validation from others, you'll be cutting yourself off from this "why" and focusing on the "what". Even though when focusing on the "why" we often produce an amazing and beautiful product (a good "what"). Focusing on the "what" and how it benefits our standing in a hierarchy would be a form of prostitution. Selling our time and effort solely for the gratification of others in order to raise our status in a, relatively speaking, meaningless hierarchy.

I should speak for a moment about the dilution of local hierarchies. To state that local hierarchies are meaningless might seem a little brash. But I think they are becoming even less important as the world becomes more connected. There's the old cliché statement about "big fish in a little pond". Well, the ponds are becoming ever more connected to each other. As they do, so do the hierarchies within. A pecking order can only have so many chickens in it, when the numbers get too big hierarchies break down. Individuals in large groups become essentially anonymous; hierarchal rank becomes ephemeral and fleeting. Status becomes temporary and driven by the moment. On the Internet, just because a large group of people thought you were wonderful last week doesn't mean they care this week. Our individual brains can't handle the multitude of faces and names we interact with each day in a connected world, let alone try and assign a relative ranking to them. We are beyond the capabilities of what we have evolved over the entirety of human history to be able to handle. I'm guessing from my own experience that we can handle 200 or so ranked people in our minds and maybe maintain familiarity with 1000 personalities. These numbers are minuscule drops in the vast oceans of people available to us through the windows to the online world we carry around with us every moment of our waking lives. Surely recognition of famous people extends beyond that, but just fame does not imply any level of trust or a working relationship. Very often the characters portrayed by famous actors in my own culture are not persons I'd trust, let alone give any level of rapport to the people beneath those masks. Anonymity is a fact of life when living as part of a hyper-connected world. It doesn't matter how famous you are, you're always certain to be able to find somebody that doesn't know you and doesn't care. So what are we to do?

The answer: don't tie your happiness to where you are because where you are is temporary and doesn't matter most other places anyway. I'm not saying don't participate in established structures. A Marine needs to obey orders and is still responsible for those subordinate to him, follow that commitment you made through to completion no matter what. My bosses still determine the work I do for my pay, if I don't like that arrangement I'm free to find another job. Ensure that you work hard on or toward something you love and don't tie your happiness too much to what others think about your work. This kind of professional detachment means that you'll start thinking in innovative ways and taking the right risks to follow your gut. You need to do the work that you feel is important. To attempt to do the work you feel that others might want of you is to condescend to them. It's an attempt to game the game. To do this is to make the decision that what you love is a more pure thing than what they'd love and too good for them, so you're going to just produce whatever junk you think they want as though you can read their minds. I'm not saying this approach never works, but it does lack a purity of intent. And that cleanness of intent matters to me. I think that the truly great and transformative things can only be made this way. It's the only way we can fully commit ourselves to our work and go forth with clear conscience. When we work from and for our hearts, we work for god; an unequivocally greater thing than ourselves. Otherwise, we are just working for a pay check; is that how you want to spend the majority of your waking hours on this earth?