Be Data Influenced, Not Data Driven

Let’s get this out of the way up front.  If used correctly, metrics can help an organization stay focused and push towards new goals.  I will not dispute this point, so please don’t take the rest of this post as a rant against metrics and measuring your results.  That being said, I’ve always had an uneasiness about the way job metrics are used in many organizations.  You must close 100 tickets this month or you don’t get your bonus.  Your commission is based on gross sales so close as much as possible.  Maybe those metrics intuitively make sense, but it’s how your organization uses them that can determine if the metrics are helping or hurting your company.

You may be looking at the numbers above and wondering what could go wrong, they seem like valid things to track and measure performance.  In the first example, what happens if it is a slow month and there aren’t tickets to close?  Is it on the person in the support role to go and find more tickets?  What if one of the employees cherry picks all the easy tickets leaving the more difficult tickets to another employee.  So the person adding less value to the organization is going to be better compensated for their work?  What do you think this is going to do to your more valuable employee?  What about your company culture?  In the second situation, what happens to your company profit margin?  If the sales goal is strictly to get as much business as possible, is this sustainable business?  Can you properly execute on it?

These are questions that have long bothered me around performance metrics, but given the value I can see when the metrics are used correctly I haven’t been able to articulate how to deal with the dichotomy of my thoughts.  That’s why recently while reading The Year Without Pants by Scott Berkun (very good read fyi) I about jumped out of my seat when he made a mostly passing reference to the idea of having a data influenced culture as opposed to a data driven culture.  What’s the difference?  Why should we make the distinction?

Look at the above examples.  In a data driven culture the only thing people care about are the numbers.  Many times middle management will live and die by these numbers so they push their employees to hit them, no matter the cost to the company.  A similar example is in education right now with all of the required standardized testing.  School districts get punished if their test scores aren’t high enough, so what do they do?   They take valuable teaching time and teach students how to take the test and teach only the material they know will be on the test.  Does this seem like a valuable use of our teachers’ times?  Does it seem like the best way to prepare the next generation for the world we live in?  Of course not, but it is the way things are done simply because that’s the metric they are measured on.

A data influenced company may still have many similar metrics, but uses them as one piece of the puzzle.  They would look at that employee that is closing 5 times as many tickets as anybody else and try to figure out why.  Then if it is because of something good that employee is doing, share those findings with the rest of the group.  If the results are skewed, they address the situation.  A data influenced organization would use standardized test scores as one metric to gauge performance of a school.  But they would also understand that some students learn different than others, so maybe their progress can be measured in other ways.  A data influenced sales organization can use gross sales as a metric for commissions, but that needs to be balanced with profit metrics and making sure contracts can be executed.

Use data to help make decisions, but don’t become blind to the numbers.  Always look at your numbers within context and if a metric becomes stale or doesn’t seem to be driving performance, re-evaluate it to see why and if things can be tweaked.  At a minimum, keeping this idea in the back of your mind can help you step back and look at the metrics being used to make decisions.  Trust your organization to be data influenced as opposed to data driven and you may be surprised with the quality of decisions that come out of the group.

How Bad Data Leads to Worse Decisions

“Bad data is worse than no data at all” – lots of people

I’ve heard a number of people make this proclamation (me included), but few of them can actually explain why when pressed.

So why is bad data worse than no data?

From my perspective there are (at least) two major categories this can fall into.  The reason data makes both of these situations worse is largely psychological and can be avoided by having the wherewithal to step back from the situation and realize what is happening (easier said than done…right?).

The numbers a decision maker is looking at are telling the wrong story

This scenario can come about for many different reasons.  It’s possible the data is just bad (inconsistent, incomplete, etc), someone could be crunching the data incorrectly, and a common one in the tech world is someone doesn’t understand how data flows through an application before drawing conclusions from it.  I’m sure you can imagine many other possibilities for why the reports or data a decision is made on would be telling the wrong story.

So why would this lead to a worse decision than if the decision maker had no data at all?  It comes down to psychology.  If you have ‘evidence’ to lead you into a decision you are going to be much more bold in that decision.  And with Big data, this is even worse.  Not only do you have data to back up your decision, but you have a metric ton of data to back up your decision.  Making the decision to go all in that much easier.

For example, if I want to invest in Tesla, without any data I could ease my way into an investment.  But let’s say I have what I think is insider information about some upcoming news (ignore the illegality of this fictional scenario) I’d be more likely to make a much larger investment.  Then if it turns out my data was wrong and Tesla tanks, I’ve ended up losing much more money than I would have by easing into the decision.

In most organizations we have to assume that most decision makers got there for a reason, they have some knowledge about the decisions they are making.  So their domain knowledge would very likely drive a better decision than one based on the incorrect data.

You are measuring the wrong thing

“Everything that can be counted does not necessarily count; everything that counts cannot necessarily be counted.” – Albert Einstein

I’m just going to touch on this briefly because entire tomes have been written on the subject.  But what you choose to measure in your organization will drastically adjust how the organization behaves.  If bonuses are based on gross revenue, expect gross revenue to increase even at the expense of profit.  Brad Feld had a great article on picking three metrics to measure your startup on that gets across a similar point.

I’ve been in many situations where the things that get measured are simply the easiest things to measure.  People want to make data-driven decisions, even in absence of any context to what the measures are.  Managers will make some asinine metric their departments goal simply because they can measure it and make a pretty graph to use in a powerpoint presentation.

So what?  Should I never use data?

Of course not, the point is to be skeptical (not cynical) when new data is presented to you.  Someone should be able to explain the origins of the data and someone (or several people) should be responsible for quality checking the results.  Without this due dilligence, many times we’re better off not having the data at all.

Lies, Damned Lies, and Big Data

There are three kinds of lies: lies, damned lies, and statistics. – Benjamin Disraeli (or Mark Twain or who knows)

This quote (whatever the actual origin of it is) rings as true today as it ever has.  We are currently in the midst of a big data explosion that encompasses nearly all businesses and other organizations in existence.  This has come about in part because the cost of collecting and storing the data has dramatically decreased.  In part, because technology has started to catch up and offer ways to use these drastically large data sets.  And in part, because it’s the cool thing to do.  If you aren’t using big data you are a dinosaur.

To be fair, there are many valid use-cases for big data.  Scientific research is a key one, as is website usage, and customer demographic analysis among many others.  The issue that we run into is people want to use the big data stores to make decisions based on some algorithm without understanding any context surrounding that data.  Either context surrounding how the data was captured or context around what the algorithm is actually measuring.

One example of this is the algorithm Netflix uses to recommend other movies to watch.  Early on in the life of Netflix, they began tracking what movies you watch (and why wouldn’t they).  They would then use this data to recommend other things to watch.  What happens though for people like me who’s Netflix account is not only used by myself, but my wife and my two young children as well.  Now, not only are they recommending things for me that I have no interest in watching, but since their algorithm is primarily based on others with similar viewing habits, my data is skewing the results they show to other people.  Oh you watch The Walking Dead, I’m going to recommend you also watch Yo Yo Gabba based on Travis’s profile.  Netflix has realized this issue (Spotify, Amazon, and others have very similar issues) and has taken some steps to reduce the skew.  They’ve introduced profiles you can choose while watching shows (though very limited at this point) and asking for genres you like.  These won’t be perfect solutions by any means, but they help provide context around the data so they can provide you with better recommendations.

Another important element of analyzing big data is understanding the source.  Many items are tracked, logged, and stored because it is easy to do so.  Not necessarily because it was the right thing to store.  Look at the abundance of weblogs and social media data that is available.  It’s easy, most web servers will automatically store this for you and social media data is available throughs simple searches.  But does the fact that your website saw a 20% spike in traffic on the third Tuesday of 6 months out of 18 actually mean anything?  Not likely, there is probably another independent variable that’s pushing the spike.  Maybe on those 6 months someone on your marketing staff decided to pick fights with people on the internet and that pushed traffic up?  Maybe you had interesting job postings right around then.  Who knows?  But the fact that these spikes all happened on the third Tuesday of the month likely have nothing to do with why the spike occurred.  The question to consider is what other things could you be tracking or logging to determine the actual reason for that spike or if that spike was actually valuable.  Maybe it’s just a vanity number that ends up costing more in hosting fees.

The key to making good decisions based on data is to understand the context in which it was gathered.  Some times this can be difficult, especially if you are dealing with a complex application in which you may need to understand some under-the-cover details to really get that context.  Volume of data is good, especially in cases where you are looking for answers about niche audiences and this is the only way you’d get enough data on that niche to make a decision (this is why big data is so popular in marketing and advertising companies), but it’s not the solution to everything.

We should always strive to make decisions based on facts and while data can be an element of that, without context it’s no better than flipping a coin (and in many cases is can be much worse, but that’s a topic for another day).

Product Management – The Missing Link

Having already highlighted many of the mistakes companies make when assigning someone to be the product manager (PM) in a previous post.  It’s time to take the opposite side of the discussion and look at what the PM role should look like.  What skills are needed and how do they interact with the development team?

The PM should be the CEO of the project.  The same way the CEO has to decide what direction to send the entire company, the PM should do that for their project.  They need to know the environment, understand how changes to the product will impact the users and how it will impact the company.  In the same way a CEO has to sell his or her vision to the rest of the board of directors, investors, and the rest of the company the PM needs to be able to sell other departments, users, and the development team on their vision.

Prioritization is another major role the PM has.  This article has the best description of product prioritization I’ve read:

The question isn’t what is the best list of ideas you can come up with for the business – the question is what are the next three things the team is going to execute on and nail.

The key to this way of thinking is that the thing that is ultimately most valuable to the company or product may not be the next thing you can execute on.   A PM needs to realize that and set the priority according to what can be done right now and understand how that ultimately gets the team to the finish line.

Ultimately the PM role is very important to the execution of the project and requires a multitude of skills that may be hard to find in a single person.  The product manager is the conduit between the development team and the rest of the company.  By setting priorities and selling the team on one product vision, the right PM can be the missing link to unlocking the potential of your development team.

Product Management – The Most Misunderstood Role in Product Development?

How many times have you been working on a project where it is obvious that nobody understands the big picture?  You just keep building things in the name of progress without any vision to where things are going beyond the current sprint.  Who is responsible for understanding why particular features are more important than another or why are you building this product and not another product altogether?

Enter the most misunderstood role on the development team, the product manager/product owner(PM from here on out).  Too many times organizations try to be ‘agile’ so they use the title without really understanding the purpose.  In my experience this is especially a problem in bigger organizations that have difficulty really empowering a team to do what is needed.  Many times because of politics and control the role gets bungled and unless you have a strong individual at this position projects have a tendency to just wander and never have a clear direction.

These people are not product managers:

  • Team Managers – This is the most common example I see in large teams.  The person who is responsible for the team gets the product manager role.  While this can be done in the right circumstances, it’s not generally the best for the team or the product.  Many times team managers have multiple development products under their watch so they can’t put the focus needed on the product management role.  Beyond this, it is important that people on the development team can push back on product management if they don’t agree and this becomes more difficult if you report to the person making all the decisions.
  • Business Analysts – It seems like a lot of times companies look at business analysts and say, well you can talk to the business and you can talk to the engineering team so you must be a product manager.  It’s not enough to be able communicate with both sides, the PM really must understand the technology side of things in a way that they understand what is possible and how to move from zero to an end product and in most cases the BA doesn’t have that vision.
  • Lead Developer – In the same way the BA lacks in development knowledge, many times the lead developer lacks in understanding of the business and how to prioritize features based on business value.  Beyond this, in cases where the lead developer does have the aptitude most times this person is busy focusing on how to execute the plan that he or she can’t step back and keep the big picture in mind.
  • Project Manager – Another common misconception.  I think it’s because the initials for both are PM.  Project managers tend to be good at task tracking and making sure the right resources are on the right teams, but many times again they lack the vision to set direction for an entire product.
  • Nobody – Sadly another common occurrence on development teams is they just don’t have a PM.  The thought could be magnanimous and the management wants to ultimately empower the team, or it could be ignorance or over-simplification of the details needed to build the product (management never oversimplifies things).  Teams will many times make progress in these situations, but with no high level view of the future most projects meander and don’t provide the value they could with strong product leadership.

All of that said, anybody in the roles listed above can be a good PM.  It’s just not common for that to be the case.  On smaller teams and in startups you have to wear many hats and sometimes the PM is also the BA or sometimes the PM has to be the lead developer.  In many early stage startups the founder and CEO is the PM, which makes sense because presumably they had the idea for the company and should be the one setting product direction.  But as teams scale beyond a certain point (this is fully dependent on the complexity of the product and the team in place), it becomes necessary for a single person to be responsible for the direction of your product.

The next part of this series will discuss what a PM should look like.

Gamification at its Best

As the NCAA basketball tournament and March Madness is upon us.  It got me thinking as to why this has become such a popular event.  Sure some of it has to do with the great basketball you can watch.  And some has to do with the fact that 64 (or 68 now I guess) fanbases get to watch their teams take a shot at the national championship.  However, I think the biggest reason it’s so popular is that so many people fill out a bracket(whether for gambling purposes or just for fun).

Just the simple act of filling out a bracket gives you an excuse to care and therefore an excuse to watch.  Otherwise why would you care to watch Wichita St. vs Virginia Commonwealth?  You wouldn’t, except you want to make sure you get a better score than the loudmouth that sits next to you in the office or you don’t want to be embarrassed by your mom again in your family pool.  This is a prime example of how gamification works and why it can be so powerful.  Throw fantasy football into this discussion as well.  The NFL has been popular for a long time, but it quickly became the most popular sport in the country with the popularization of fantasy football.

There are plenty other successful examples of gamification.  The key to the strategy working is making sure your user base has that group of people that want to compete with.  This is why the gamification in Foursquare and other similar services work so well in some groups, but not in others.  Keep this in mind when looking for ways to build or expand your brand.  Gamification can work, but don’t over apply it and make sure that your particular situation can benefit from it.

Brainstorming Session – Google+

I’ve spent a lot of time recently thinking about Google products and their connections with each other so I’ll probably have several posts in the near future specifically on Google products.  I’m going to start with a discussion specifically on some ideas for Google+.  As a disclaimer, Google+ is somewhat close to my heart because I worked on a startup a few years back that had some similar ideas and it was somewhat satisfying to see a lot of them (circles specifically) implemented in a product.

For the basis of this discussion, I’m going to leave out how Google may or may not be able to convince people to spend more time in the service.  I think that is something they are deeply concerned about right now, but at some level I’m not sure it matters.  The more services they tie it into (and they will) the more people will be using the service whether they are aware of it or not.  And as long as the usage in some way drives additional advertising revenue, I’m not sure it’s all that important that people spend X number of hours actually on the Google+ site.  To get the discussion started, I’m going to review what I think Google+ excels at because I think that will help the enhancement ideas make more sense.

What Google+ excels at

First and foremost the thing I’ve seen on Google+ that other social networks can’t match is the level of interaction among users.  This could simply be the result of early adopters currently being the main users and that crowd tends to be more likely to interact with each other.  Even if that is the case, I think the design and focus of Google+ lends itself more to people having conversations with people they may not already know.  Twitter is very good about letting users express their ideas to anyone that wants to listen, and Facebook is very good at allowing users to have conversations with people you know, but I think Google+ really gives the opportunity to have conversations with people that you share interests with.

I believe the concept of allowing you to put people you want to interact with in groups is also a huge benefit.  This is a feature I’ve been wanting in Twitter for a long time.  Some times I want to mute the comedians I follow or pay special attention to the tech crowd I follow.  I can do that using circles in Google+, but have no real mechanism to do so in Twitter.  What it comes down to is allowing the user to minimize the noise of their social network at any given time.  While there are numerous additional specific features I like in Google+(hangouts are pretty cool), these are the ones that I think could really provide a competitive advantage with some additional focus.

Bring other Google services into the Google+ feed

The most obvious (and I would assume easiest to implement) thing to do would be for Google to allow you to put YouTube subscriptions and Google Reader feeds right into Google+ circles.   This would allow a user to use Google+ as an information portal and make it easier to start conversations around these items.  The next potential step would be to use Google+ as the engine for comments on the actual article, much like they talked about implementing with Google Wave.  This would allow the users to see what the overall conversation around an item is as well as have a personal conversation that is limited to their circles.

Implement a “Read me Later” button

Many times while checking Google+ when time is limited, I may not have time to actually read all of the articles I find interesting.  So it would be useful to have a “Read me Later” button so I could easily come back to either that article or maybe check up on the user comments that have gone through since I last checked.  I’ve seen some users hack together a solution by simply having a private “Read me Later” circle that they will share things to.  While this would work, I think this is a feature that enough people would use that it would be worthwhile to make it more prominent in the UI.  This would also allow less technical users to enjoy the same functionality.

Bring other Networks into Google+ Feeds

This one is probably impossible given the relationships among Google, Facebook, and Twitter.  So maybe it would be best to just classify this as a pipe dream.  Imagine having one location where you could go and check all of your social network feeds and have the level of interaction with your friends/followers that currently exists on Google+.  I’d love it, and I imagine I’m not alone.  Maybe at some point in the future we can get some “neutral” 3rd party to step in the game and provide this, but the way things look now I’m not going to hold my breath.