Approaching Parallelism

When we begin to learn Parallel computing,  we forget to think why parallel computing was introduced, rather we tend to apply it anywhere and everywhere we can. Before going into the actual discussion, let's first understand what Parallel computing is.

According to Wikipedia,

Parallel computing is a type of computation in which many calculations or the execution of processes are carried out simultaneously. Large problems can often be divided into smaller ones, which can then be solved at the same time.

Why Parallel computing ?

Moore's law is a computing term which originated around 1970. The simplified version of this law states that processor speeds, or overall processing power for computers will double every two years.

To break down the law even further, it specifically stated that the number of transistors on an affordable CPU would double every two years.

From 1986 – 2002, microprocessors were speeding like a rocket, increasing in performance an average of 50% per year. Since then, it’s dropped to about 20% increase per year.

Since the speed of microprocessors is not increasing as it did before, instead of designing and building faster microprocessors, an intelligent idea would be to put multiple processors on a single integrated circuit. This is where parallel computing comes into picture.

A problem here is that adding more processors doesn’t help much if programmers aren’t aware of when to and how to use them.

Why we need ever-increasing performance?
Computational power is increasing, but so are our computation problems and needs. Problems we never dreamed of have been solved because of past increases, such as decoding the human genome. More complex problems are still waiting to be solved.

Coming back to our discussion, 

Now that we have understood why parallelism, we'll understand the right approach(strictly my opinion) towards it.

Running multiple instances of a serial program often isn’t very useful. It is not all about making parallel whatever we see, but rather thinking about what will increase performance. At the end of the day we want to make things run faster.

Approach?
First thing is to look for the possibility of optimizing the serial code. Sometimes, the solution will be in front of us and we search all over. For example, changing the accessing order of an array will have a huge impact on the performance i.e cache-friendly code. It is very important to identify small things like these to optimize the code which. If there is no more room for the serial optimization then we may go for parallelism.

Is it really worth to parallelize?
May be. If the problem is too simple that parallelizing it would not really make a difference in the performance, then it might be a bad idea. For example, if we consider a problem involving a very few computations. If we make it parallel, then the overhead of making it parallel might be more than the time it takes for computations.

When we look on the performance, dividing the input size in an appropriate fashion also seems very important. In parallel terms it is called as Granularity. Depending on the amount of work which is performed by a parallel task, parallelism is classified into three categories : fine-grained, medium-grained and coarse-grained parallelism.

There are many more things that I couldn't mention, I'll try to sum up all of them in a future article.