You might ask, computers have been around for a very long time, what’s up with the all the fuss about artificial intelligence and in particular machine learning and deep learning in the recent past? So much so, that the media has started to paint doomsday scenarios that AI may one day take over from humans as the dominant form of intelligence on Earth!

In reality we are a fair way away from such a scenario. A statement by Andrew Ng with respect to the above, provides a more realistic picture:

Fearing a rise of killer robots is like worrying about overpopulation on Mars.

Andrew Ng – CEO Landing AI, Adjust Professor at Stanford University

In other words, we haven’t even solved some of the fundamental problems in AI and that’s why focusing on a potential AI takeover is the wrong problem to focus on.

It does still beg the question, why the fuss about deep learning. After all, a quick search on Google Trends, for the term “deep learning” shows an exponential increase in interest in the past decade:

To answer why, we have to first understand some of the historical limitations of machine learning. The TLDR for why deep learning is so hot right now is that Machine Learning requires Feature Engineering while Deep Learning does not.

Machine Learning requires Feature Engineering while Deep Learning does not.

TLDR

Let’s dig deeper to understand what we mean by that statement. Let’s assume you have heard all the hype about artificial intelligence and you decide that it’s time you start your own company that sells self-driving cars. As the founder, you get started with figuring out how to build an effective algorithm for safely navigating the car on the road. Such a learning algorithm should at least be able to identify when there is another car on the road. How would we do that?

Let’s say you installed a camera on the front bumper and use it to capture a 1 minute video, similar to the one from your competitor Tesla. Because you are a thoughtful engineer, you chop the video into 10ms segments each of which you can now treat as an image. You now have 600 images from the 1 minute video and your task is to make the computer output whether there is a car in the image or not. The task seems simple enough:

Assume you know nothing about learning algorithms but you can do a bit of coding, after all you are a founder of a tech startup. You visually inspect your 600 images and manually label each as “1” if it contains a car and “0” if it does not contain a car. Now you decide to write a program to output car/no car depending on the input. So you write a simple program:

If Data == 1
     print('Car')
else
     print('No car')
end

You show the above piece of code to your co-founder, Bob, who recently read a blog about what machine learning is. Bob shakes his head.

Bob points out that what you wrote looks like a traditional program, i.e. it’s a hard-coded set of instructions. The data you gave to the computer consisted of 1s and 0s and you instructed the computer to print the correct answer and that, he argues, looks a lot like the traditional programming described in the blog:

Secondly, Bob argues that since you did the job of labeling images as car or no car and simply provided the labels (0 or 1) to the computer without the raw image data, you didn’t give the computer the chance to learn anything. You agree with Bob and ask him for help. Based on his knowledge, he draws the following:

You note that Bob has inverted the relationship between output and program and is now supplying the raw image data as input to the computer. Bob says that if the machine can learn such a program then this program can be deployed in a self-driving car to predict cars in new images that the camera collects! That’s true, you say…

But you point out to Bob that that the raw data takes the form of an RGB image with many different pixels. Some of the pixels have something to do with a car, while others do not. Any single pixel, on it’s own, doesn’t tell me anything about a car anyway, you argue. How would I teach the computer to find a car in the image by looking at a bunch of pixels?

All cars should have tires, windows and number plates, Bob says. What if we, the engineers, extract these main features of a car and ask the machine (computer) to perform classification, i.e. predict whether a car is present in the image or not using these features instead of the raw data:

You nod and realize that by providing the computer information about the high level representation of a car (i.e. tires, windows) instead of raw pixels, you make the learning algorithm’s job easier.

Mapping the raw data to the output has historically been a difficult task for machine learning algorithms. ML algorithms have therefore often depended on mapping manually extracted features to the output instead.

In the simplest case, you can see how this might work. You, as the engineer, would identify a part of an image containing a tire. You would provide this tire feature to the computer and ask the computer to check if a tire similar to this is present in another image. Presumably if a tire is present, the likelihood of a car being present is high. Now the computer’s job is basically that of a detective, it needs to look at each patch of a new image and check if it contains the tire patch:

You point out to Bob that the computer could take the pixels of the tire patch, slide it over the new image in steps and multiply it with the pixels of the image, and report back the product. If the product of the pixels leads to a large value that means the tire feature is activated in the image. The computer could then sum over the pixels and report back that the car is in the image if the product of the pixels is above a certain threshold.

Extracting tire features manually from the input image data is hard work. Bob realizes that bicycles also have tires, so you might confuse a bicycle for a car using this approach as a bike would also activate the same features. But Bob thinks you are on to something. He asks if what you are doing looks something like this:

You again nod in agreement. Bob’s drawing reminds you a lot of neurons in the brain, where dendrites provide input data, the cell body performs some function (in our case product of tire features with input image followed by applying a threshold) and an axon provides the output:

You wonder if the artificial neural network you are building can learn weights (parameters) similar to synapses in the brain:

So instead of supplying tire features, you ask if artificial neural networks can learn weights (i.e. parameters) by reducing the error between the desired output and predicted output? You argue that you already have the desired output for the 600 images, since you labeled the image as 1 if it contains a car and 0 if it doesn’t. Why not just use that desired output to adjust the weights of our Artificial Neural Network. You draw another figure to illustrate the point:

Bob likes the idea but he sees a couple of issues. He doubts if the computer can learn high level representation like tires and windows from just the raw pixels directly. He wonders if you could use many neurons, stacked one after other, instead of just one neuron to make the task easier for the machine.

You get the main idea but are not sure how stacking the neurons one after another in layers such that one neuron’s output is received as the next neuron’s input can lead to the machine’s learning task becoming easier. Bob thinks that the weights of the earlier neurons will learn lower level features (like edges) while weights of later neurons will learn object parts (e.g. tires) automatically using information provided by earlier neurons. You stare back cluelessly at Bob and he tries to help out with the following figure:

You finally get what Bob is doing. The question Bob poses is this: could the computer automatically learn higher level features (such as presence of tires) from simpler representations (e.g. edges corresponding to a sudden change of intensity in the image) which in turn are generated from the raw data? You aren’t convinced with the details just yet, but you see where Bob is going with this. Bob draws another figure:

Bob says that many neurons stacked in layers one after another can automatically learn the representation as well as the mapping from the representation to the output. He calls this Deep Learning and points out that as opposed to machine learning, which requires a human to perform the feature extraction, this learning approach is completely automatic since the machine learns the representation (i.e. features) on it’s own.

You note that the input is again the raw image. If you understand it correctly, Bob is suggesting we ask the computer to perform end-to-end learning. The Deep Learning algorithm would perform representation learning (i.e. learn salient feature like tires from simpler features like edges) as well as map the representations to the output. The human would not be required at all.

Deep Learning enables end-to-end learning.

But you also realize that to do what Bob is suggesting, you would need a lot of data and sufficient computational power, since the learning algorithm has to learn the representation itself as well:

Bob agrees and gives the analogy where the computing power (CPU, GPU, TPU) is the engine of a rocket and the input data is it’s fuel. You need a combination of both to thrust the rocket into space, i.e. to enable Deep Learning

So Bob says that training artificial neural networks would only make sense if you have a lot of data, otherwise machine learning algorithms that require features extraction may just do better. He makes another plot:

Bob thinks that large Neural Networks having many layers of neurons can continue to learn with increasing data while other learning algorithms’ performance tend to stall as the representation itself is a bottleneck.

You understand what Bob means and try to put his Deep Learning idea within context of Artificial Intelligence, Machine Learning and Representation Learning:

You realize that the first program you wrote for identifying a car could be considered as a simple AI algorithm:

The idea of using a tire as a feature and asking the computer to map the tire feature to the car would come under classical machine learning:

Finally, the idea of using multiple neurons stacked in many layers, where the multiple layers through automatic representation learning/feature extraction enable End to End Learning without human intervention, would be considered Deep Learning:

You realize that using Deep Learning, given enough data and computing power, you could tackle all kinds of problems, without requiring any human expertise for feature extraction. e.g. without needing medical doctors to identify portions of a brain MRI image to which the computer should pay attention when performing a diagnosis.

And that’s why Bob, you and the rest of the world think that Deep Learning isn’t just hype but is in fact here to stay. And that’s why you are so confident about launching your own startup into orbit using Deep Learning!

There are four distinct ways in which machines learn.

If the above is a bunch of gibberish, let’s go into detail.

1. In the case of supervised learning, the learning engine has access to data (e.g. image of a cat) and desired output (e.g. label, binary digit 1, indicating presence of a cat).
2. In self-supervised learning, the learning engine has access to data (e.g. image of a cat) but NOT to the desired output (e.g. label, binary digit 1, indicating presence of a cat). Self-supervised learning is also sometimes referred to as unsupervised learning.
3. Semi-supervised learning is a combination of supervised learning and self-supervised learning. So in this case, only some parts of the data is labeled. As an example, maybe 10% of cats/non-cat images are labeled while the rest are not.
4. Reinforcement learning is a completely different beast. It assumes that an agent (e.g. a robot) is in an environment (e.g. a maze). When the agent performs a certain action, e.g. move one step forward, it is provided a reward depending on whether that action helps it to get closer or not to completing it’s task (e.g. get out of the maze).

Below we see concrete examples for each of the four types.

On the far left, we see an example of supervised learning, where the learning algorithm is tasked with learning the decision boundary between the green circles and red triangles.

A clustering example is shown for self-supervised learning. Here the squares need to be divided into 3 distinct portions depending on their relative distance in the two-dimensional domain spanned by x1 and x2.

For semi-supervised learning, the task is the same as the one shown for supervised learning, but in this case, some of the data is not labeled (i.e. the learning engine does not have access to whether the labels is a green circle or a red triangle).

Finally, only the far right, we see an example of reinforcement learning, where an agent (e.g. a robot) needs to navigate through an environment (e.g. a maze) by performing actions and updating it’s state based on the relative reward it receives.

In industry, supervised learning is responsible for most of the economic value generated so far. But creating a dataset for supervised learning is also particularly expensive in terms of man-power, as a human being has to go through the process of labeling the data. Also, remember that while it’s easy for just about any person to label images as cat/non-cat, often datasets require expert input, e.g. labelling a dataset of radiology images requires a radiologist or sometimes even a team of radiologists and a radiologist’s time is expensive!

That’s why the machine learning research community, as a whole, is trying to leverage the power of the other three learning strategies, particularly self-supervised learning, in order to training learning algorithms.

Have fun reading!

So let’s get the formal definition out of the way.

“Machine Learning is the art of getting computers to learn without being explicitly programmed”

Arthur Samuel (1958)

If you read the definition above, the part about not explicitly programming the computer is what really distinguishes machine learning from traditional programming. In traditional programming, a computer take data and a program, i.e. an explicit set of instructions, as input and generates an output.

On the other hand, in machine learning, a combination of data and the output is used to make the machine learn the program:

Note that one thing didn’t change. The data is still an input in both cases.

But how is machine learning actually performed? For example, how can we make a computer learn to identify that an image contains a cat.

This is harder than it sounds. Human beings are very good at doing this sort of pattern recognition. But to a computer an RGB image of a cat is just a bunch of pixels. An extremely unique combination of pixels results in an image of a cat. And an effective machine learning algorithm needs to be able to automatically identify what are the combinations and patterns of pixels, that when put together, results in a cat appearing in an image.

How would you would go about teaching such a thing to a computer? Well, there are three components of learning:

Any machine/deep learning algorithm you come across will contain the above three components. This point is definitely worth repeating, any learning engine will contain the above three parts. So let’s take it step by step. What do we mean by representation:

Representation is the heart of any machine learning algorithm. A learner must be represented in some formal language the computer can handle. In practice, this formal language is basically some mathematical model that takes some input data, performs a computation on it, and returns an output. Neural Networks, Gaussian Processes, Support Vector Machines and Decision Trees are all examples of representation.

Having a representation is not enough though. All mathematical models have parameters that can change in value. Parameters are just variable values in the model that need to be adjusted for the task to be performed correctly. For example, the tuner knob on your car radio represents a parameter that can be tuned:

The value of the parameters of a model will change depending on the problem being solved. For the radio analogy above, you can imagine that you need to tune the knob different amounts depending on which channel you want to hear. Therefore, you need a mechanism or a method to identify which set of values for the parameters of the model will result in a learning algorithm that effectively performs the task at hand, e.g. learning whether an image contains a cat or not. In other words, we need an objective function that can distinguish good solutions from bad ones. This is what the evaluation component of our learning engine does:

Mean Squared Error, which is basically an average of the square of the error between the actual output and the machine learning engine prediction, is an example of such an objective function.

Finally, once you have a representation, i.e. some mathematical model, and and an evaluation method to identify which parameters might be good choices, you need to be able to search for the set of parameters that lead to the optimal result, i.e. the smallest error between actual output and the machine learning prediction. This is where the Optimization component comes in handy:

Optimization is basically a method to search for the best solution. There are many different optimization methods, some depend on computing the gradient or slope of a function, while other approaches such as Genetic Algorithms, take inspiration from how nature identifies the best solutions.

If we put all of the components together, we can write out the complete learning problem as the below optimization problem:

In other words, learning involves finding the optimal parameters, theta_star, that maximize a certain goodness function (e.g. Mean Squared Error) given a certain representation (e.g. Neural Networks) with varying parameters, theta, and a set of data (e.g. Dataset containing images with and without cats together with labels of whether the respective image contains a cat) to train the algorithm over.

Hope your learning algorithm manages to tune the parameters just right, so you can enjoy the end result!

Or you can go ahead and learn about the different types of machine learning approaches.

References

Pedro Domingos. 2012. A few useful things to know about machine learning. Commun. ACM 55, 10 (October 2012), 78–87. DOI:https://doi.org/10.1145/2347736.2347755

This then, is what our Minister of Science & Technology believes the process of scientific inquiry ought to look like: an array of beakers, centrifuges and bubbling colorful liquids being mixed together in a poorly-lit lab by a man, who looks every bit the part of a ‘mad scientist’, with a newly- acquired and abiding love for science. And what about the goal, you ask? Well it’s nothing short of cleaning Pakistan’s water. This is the man who we are meant to follow as he takes us and our science into the 21st century — and plans Pakistan’s virgin expedition to outer space. What sort of reaction is this video meant to evoke? What is the attempt here? Is the video, meant to introduce us to the creative efforts of our Minister towards solving deep-seated problems in our society through the use of science? Or is it to show us that anyone can be a renowned scientist simply by looking the part even if the inherent skill set is missing? Fake it until you make it? Or is it meant to ridicule real and serious water related problems that Pakistan faces today and in the future? Whatever the original aim, Fawad Chaudhry eventually ends up making a mockery of himself, his Ministry and the scientific endeavor itself. For me, his video and recent statements were an unpleasant déjà vu.

In 2012, the Federal Minister for Science & Technology, Changez Ahmed Jamali, went to Sukkur, Pakistan with Religious Affairs Minister Khursheed Shah in tow, to meet Engineer Agha Waqar Ahmad, of the water kit running car fame, in the hope of replacing Pakistan’s dependence on oil imports. Waqar declared that he could run a car purely on water and went about demonstrating his water kit to all and sundry. What followed were several weeks of madness in which Agha Waqar received vociferous support from all corners ranging from television anchors, politicians, all the way to the head of Pakistan Council of Scientific and Industrial Research (PCSIR), Dr. Shaukat Parvaiz. Standing incredulously and desolately on the other side trying but failing to make the nation see reason (and the laws of thermodynamics) were Dr. Pervez Hoodbhoy, Dr. Atta Ur Rehman and Dr. Shaukat Hameed Khan.

One would have expected that the water-kit saga would have led to serious, national-level introspection. One, in which, we would have asked ourselves how it was possible that a charlatan had exposed the academic elite of the country with such alacrity and ease. And how, instead of being exposed for his fraud, Agha Waqar was eulogized and celebrated in the media. What transpired betrayed a more potent malaise, a deep-rooted anti-intellectualism at the heart of our republic and national culture.

While scientific pursuit is still treated as a bit of a joke in Pakistan, there has historically been an espousal and celebration of our intellectual past, particularly our Muslim intellectual past. On Nov 4, 1969, for example, the Pakistan Postal Service issued a postage stamp commemorating and remembering the life of Ibn-al-Haitham, the medieval Arab mathematician and physics, whose legacy as the “founder of physics in the modern sense of the word” is only now being reclaimed and acknowledged.

The stamp itself draws attention to Al-Haitham’s contribution towards the field of optics. However, Al-Haitham’s lesser known, but equally significant contribution towards the scientific method, is what we could stand to benefit from even more. Rosanna Gorini describes his contribution to laying down the foundations of the scientific method in the following words: “[Al-Haitham’s] investigations [were] based not on abstract theories, but on experimental evidences and his experiments were systematic and repeatable”. Al-Haitham was a strong advocate of empiricism and took on the might of Euclid, Ptolemy and Aristotle, in his work when he went on to prove that the emission theory of light was “superfluous and useless”.

One of the most striking features of the medieval Arab scholars of that time was the acceptance and admission of their own academic and philosophical limitations and their opening up to ideas from outside. In doing so, these figures from our imagined Muslim past not only made themselves vulnerable towards different thoughts that perhaps clashed with their own, but also actively engaged in conversation with these subjects. A concerted effort and attempt was made towards translating works from all parts of the known world into Arabic. Translators of the time served not only to merely reproduce the works in Arabic, but often added corrections and commentaries to the work they were translating. It is through this rich dialogue between indigenous and transnational knowledge and the deconstruction of these ideas by Arab philosophers and scientists that people such as Ibn Al-Haitham managed to challenge established concepts.

We, as a nation, purport to follow luminaries of the past and at least claim to seek inspiration from their work. On the other hand, today, in our society, there is rampant anti-intellectualism and a certain degree of armoring up, where we shut out ideas that may attack our established dogmas and beliefs. In fact, the process of holding on to and appropriating our Islamic scientific heritage is a form of epistemological defense meant to shield us from critique about the dismal state of scientific progress within postcolonial Pakistan.

There is both a lack of admission of our own failings in terms of staying abreast with contemporary research in each discipline as well as a lack of exposure of our thoughts lest somebody catches us for the charlatans that we fear we all are. While failing to engage with these thoughts we simultaneously wrap ourselves in the comfort of the belief that if our appropriated Muslim past had not transpired, the world would not have marched towards modernity. These Muslim pasts are inextricably embedded within our national discourse on science and technology. A quick glance at the ‘About Us’ page of the Al-Khwarizmi Institute of Computer Science (KICS) in the most elite engineering school of the country, the University of Engineering and Technology, Lahore, furthers this argument. It reads: “Al-Khwarizmi’s work laid the foundations for the building of computers, and the creation of encryption in the 20th century. The modern technology industry would not exist without the contributions of Muslim mathematicians like Al-Khwarizmi.” In other words, if it were not for our Muslim predecessors, all work thereafter would have come to naught. In a sense, we are precluding ourselves of the responsibility of any further progress by stating that whatever has taken place, has done so mainly through Muslim contribution thus far. This sense of smug self-satisfaction has led to a reactionary society that fails to look inward and to introspect, that searches vainly for conspiracy theories to explain away its own self-inflicted wounds, that plays up the possibility of a world colluding to see it fail and that looks for shortcuts and miracles to extricate itself. We had begun peddling fake news in Pakistan well before the term fake news had become a commonly used term.

In fact, in Pakistan we consistently observe a suspicion of the life of the mind and of those who are considered to represent it. Being an intellectual is considered synonymous with the ability to earn degrees and with the number of years spent in the classroom alone. Learning is seen to begin and end in school, while any other intellectual pursuit, be it reading or writing, is considered extraneous and wasteful. Go to any bookstore in Pakistan, and the only books that consistently sell are school and university prescribed textbooks, i.e. material that must be bought, compulsory, as we like to say. Beyond these books, there is very little interest in reading for the sake of reading or gaining knowledge just for the sake of it. This leaves the nation holding on to the straws of intellectuals who exhibit only a skin-deep level of free and open thinking.

What is the root of this rife anti-intellectualism? Where does it stem from? Perhaps our colonial past might hold some answers. The British colonial project was primarily interested in strengthening and furthering the British hold over India. It did this partially by building a veritable civil and administrative force. The colonial project sought to ingrain within the bourgeois and the local populace the idea that prestige and honor lay in administrative and clerical duties. It developed educational institutions that mass-produced degree wielding individuals for whom the zenith of ambition meant a complete, homogeneous and servile entrenchment within the British Indian civil service. This necessarily came at the cost of emphasis on skill development, on nurturing creative intelligence and in promoting scientific disciplines. Free-thinking, idea-generating individuals would in fact have been a threat to the British stranglehold over the crown jewel. This mindset of relative prestige has persisted through to this age in the way in which a civil servant is viewed as belonging to a higher social stratum with respect to, for instance, a university professor, within our unwritten social order.

In fact, the local Pakistani intelligentsia class that succeeded the British not only eroded what little share it got of the well-oiled civil setup it received from the colonial masters, but at the same time it failed to recognize the need to realign its priorities in training the workforce by emphasizing more creative pursuits and instilling greater intellectual curiosity. It was not surprising that the brown sahibs consolidated British efforts to root out independent thought, since they saw personal gain in keeping the locals unaware and illiterate. Pakistani metropolises like Karachi, Lahore and Islamabad, as a result, have not evolved into hubs of intellectual and philosophical debate which they once showed promise of becoming.

While we can continue to analyze the roots of our anti-intellectualism at the macro level, it also pays to observe what role our domestic lives have played. It is here that children first exhibit their natural curiosity about the world around them and it is here that they depend on the environment to help them nurture their intellectual development. The home is effectively the first intellectual center of a child. According to Judith Roden, young children are natural scientists. She suggests that when children play, they explore the world in much the same way that scientists do by forming hypotheses and conducting experiments. How much they are rewarded (or punished) for these early forays into the unknown then determine how amenable they would be to exploration later in life. A conducive atmosphere in the home then remains critical in determining their natural intellectual disposition.

The causes of anti-intellectualism in Pakistan are complex and multi-variate and do not yield easy solutions. And while we may use absence of funding, resources, infrastructure as explanations for the lack of scientific progress, what we miss is more fundamental. We miss people who can exhibit critical inquiry, who can ask the right questions, identify the most important problems and go about systematically finding the means of solving them. If we don’t realize this soon, we will continue to produce ministers who want to clean Pakistan’s water in a day or want to put their bets on the same water, once miraculously purified, to run our cars.

Samee ur Rehman holds a PhD in Applied Mathematics from Delft University of Technology. He currently works as a Data Scientist in Silicon Valley, California. https://www.linkedin.com/in/surehman/