Data science can really change the way you look at life. It’s already hot stuff when it comes to examining those abstract subjects like business, physics, and the weather.
However, I envisage an era in the not too distant future where we will increasingly be running data science experiments on ourselves – that’s where some of the greatest insights to our own lives can be found. We don’t do enough of it right now, but we will.
This year, I’ve been using the Habitbull app to track and analyse my habits throughout my own day-to-day life (you can call it the New Year’s resolution that I somehow never managed to break). I’ve set up quite a neat algorithm for myself in R where I take the data in csv format and generate graphs, histograms, regression output, and all that jazz, on my own progress.
I’ll often track metrics such as:
- How many clients do I reach out to per day?
- How many minutes did I workout this morning?
- How much did I manage to save today?
In tracking these habits, I’ve found many of the principles of data science to be fascinating in terms of what they can reveal about the events in our own lives. Here are three of them in particular:
1. Law of Large Numbers: Computers don’t get emotional, and neither should you
Let’s suppose I asked you to do some extensive calculations by hand 1,000 times, a simulation of sorts. There’s a catch too – the first 500 results will turn out to be statistically insignificant (i.e. not useful).
Chances are, you would not do as good a job as the computer. Many of you would even refuse to do the task due to its sheer magnitude. Even if you were a rock star at doing calculations by hand, you would eventually become bored and fatigued by the process, especially taking into account the results you are getting at first. You will likely begin to say, “Hey, what a pointless exercise. I don’t see the point in this whatsoever”.
As a result, you would probably give up before reaching 1,000 calculations.
A computer, on the other hand, has no emotions. It simply runs the simulation however many times is specified until the calculations are done and the findings are in. Computers are not always faster either. Even if an operation takes several days to run, a computer will keep working at it non-stop.
Now, while we are not computers, we as humans have a tendency to make premature conclusions. If we fail at something the first few times, we conclude ourselves to be a failure and quit. However, a computer would continue the process regardless.
Also, a computer doesn’t expect what we all too often do – instant results. It takes an average of 10,000 hours to excel at any skill.
The computer has no concept of time, and doesn’t invest emotion into results like we do. Therefore, a computer could potentially run a process for 10,000 hours and master a particular skill in just over a year. While it would take us a lot longer (we still need to eat, sleep and take care of ourselves mentally), where we differ from computers is that we are too results-oriented when we need to be more process-oriented.
The takeaway from all this? Focus on the process and keep upping the numbers. The results will come by themselves.
2. Power of a test: Your life should revolve around working smarter, not harder
It’s only this year that I’ve seen for myself how true the 80/20 rule is in our lives, where 80% of our results come from 20% of our efforts.
For instance, I’ve found that:
- 80% of my best clients have come from 20% of the prospects I have reached out to
- 80% of my workout results come from 20% of the quality sessions I do rather than the other 80% of mediocre sessions
- 80% of my savings have come from eliminating 20% of my prior spending habits
These types of findings reminded me of the power of a test. The power of a test is a statistical tool that we use in order to make our large numbers not so large. For instance, let us suppose that a researcher collects 1000 samples for a study. However, the power of a test indicates that a sample size of 100 would yield the same statistical significance as collecting 1000 samples. Which begs the question – why would we spend time collecting 1000 samples when 100 will do?
The same goes in life – we spend most of our time doing things that don’t give us any additional return, i.e. sweating the small stuff. Instead, if we focused on the things that really matter and make a difference, then we would see a vast improvement in our lives. Whatever our goals are.
The beginning phase of racking up our samples (making effort everyday towards our goal), can seem tedious. However, we then have the ability to see what works and what doesn’t, and filter the things that really contribute positively to our goals. Then, success is all but guaranteed.
3. Know how to handle your error term (the uncontrollables)
When we run any regression, we have our predictors and then our error term (the “unknown unknowns”). This is a fancy way of acknowledging that many variables in a process are not in our control, and that includes life.
So, how would we handle an error term in a regression? Well, there are times where we should seek to minimize the error, while on other occasions it is better to simply leave things be. For instance, if we have a large error term due to an important variable left out of our model, then it makes sense to include it.
However, sometimes there won’t be any suitable variable but we’ll still try to include something nonsensical anyway. In much the same fashion, we have a tendency to try and seek solutions to problems where none exist.
Instead, we need to realise:
Perfect is the enemy of good.
As opposed to continuously seeking solutions to a problem where there are none, simply move on and keep your eye on the most important things.
Ultimately, I have found that the path to success lies in being process-oriented and focusing on what really matters (the 20%). By focusing on the process, the results will come by themselves.