If you are reading this, I’m assuming that you’ve read the “part 1” of the series and already have the context. If not, suffices to say that the article below was written a while ago, during my tenure as the Senior Research Scientist of the Tandemlaunch startup foundry in Montreal, Canada, and that without their kind permission you would not be seeing the text below here on the Cinteraction blog. Hope you find it interesting!

On Deep Learning and The Future of Computer Vision vol. 2 (written in October 2019)

This one was hard to name.

Somewhere within the process of toying with several topics that would make for a good column contribution, I realized that in the time between the August 2018 article titled “On Deep Learning and the Future of Computer Vision” quite a few ground-breaking things happened that relate, and even validate, what I wrote there. Once that worm was in my system, it was hard to get excited about the other topics I considered, and they simply got pushed back. So here we are, going into the summer, end of spring season edition, borrowing from that great sitcom legacy of “recap” episodes.

In all seriousness though, while I was aware of the speed at which computer vision and AI research was progressing and growing, what I saw at the Neural Information Processing Systems (NeurIPS)¹, NVIDIA GPU Technology Conference (GTC)² and the International Conference on Robotics and Automation (ICRA)³ still managed to surprise me..

I should at this point say that I feel truly blessed that both NeurIPS and ICRA took place right here at the Palais de Congrès in Montreal within the last 9 months, enabling all of us locals to experience them within the metropolitan environment we call home. Add to that the fact that Montreal’s own Yoshua Bengio has been, together with Geoffrey Hinton and Yann LeCun, awarded the Turing Award “for conceptual and engineering breakthroughs that have made deep neural networks a critical component of computing” and see if you can blame me for wanting to write of nothing else than deep learning and robots, which is essentially what I wrote about in my first article for this column.

Looking back, there were only two people I mentioned by name in the text (although I mentioned FacebookAI, which is pretty much synonymous with Yann LeCun): Geoffrey Hinton and Josh Tenenbaum. While I remain a “crystal-dyslexic computer vision enthusiast”, it fills me with no small amount of joy to be able to report that in addition to the great recognition of Geoff Hinton and Yann, Josh has since been named R&D Magazine’s 2018 Innovator of the Year. The missing Turing laureate, Yoshua Bengio, is a giant in the deep learning domain. Even more so for his work on the Montreal Declaration for responsible AI development. However, Yoshua’s research focus has mainly been on natural language processing technology, so he did not find his place in “vol. 1” of this article. Just as well, more for the future.

Back to the technology, the limitations of deep learning and what the last three quarters have taught us. The basics have not changed: we are still working on creating a well-structured description of the world and developing ways to make our AI systems run on more devices (on the edge), rather than modern-day equivalents of the Harvard Mark I and IBM mainframes residing in the cloud. And boy are we moving!

On the structured-description front I’ve discussed Hinton’s capsule networks, which are inspired by both the way we handle objects in computer graphics and insights from neuroscience, as well as the efforts by FacebookAI and Josh Tenenbaum’s research team at MIT CSAIL, which have focused on representing the visual world with programs (probabilistic in the latter case).

I still think that Hinton is very much worth listening to and, if you haven’t already, take a look at his recent Google I/O talk. He muses that capsule nets might be “the thing he ends up not doing”, but gives a nod to transformer nets, which share the basic concept of controlling the flow of information through the network with the capsule nets. The bidirectional transformers (BERT)⁴ have already led to amazing improvements in natural language understanding, by improving the contextual representations used in the domain. So, the basic ideas created to help us understand the visual world better are now helping us create better representations for understanding text. This is something that I think we will see increasingly in the future.

In my August article I pointed out that the IMAGENET Large Scale Image Recognition Challenge classes, which allows us to train our first deep learning models for vision, were designed to represent the WordNet synsets (sets of cognitive synonyms), which was the result of efforts to build structured descriptions from text. Subsequent initiatives, such as the Common Objects in Context, have enabled us to do more challenging things, such as generate reasonable descriptions of images. The important thing here was not the achievement in and of itself, but to the successful integration two traditionally very much separate areas of AI – computer vision and natural language understanding. There was no well-structured description to speak of (or see with) as of yet, but it allowed us to bridge the gap caused by the different set of research skills you needed to deal with text and visual data.

So, you could generate a reasonable description of visual data without a real understanding of what is going on, but what proved more elusive is answering specific questions about the things you see. Ergo, Visual Question Answering (VQA), the pinnacle of which are the latest extensions of FacebookAI work⁵ and the MIT CSAIL (Josh Tenenbaum) stuff ⁶.

Welcome to the age of neuro-symbolic models which learn both visual concepts and words jointly and can do so in an unsupervised way, as is the case with MIT CSAIL’s fresh-of-the-press neuro-symbolic concept learner. Just in the last 9 months, significant breakthroughs have been made in the domain of structural descriptions for those who cross the now obsolete boundary between natural language processing and computer vision. I did not expect it to progress at this speed, but I stick to my original recommendation, these are the groups to pay attention to. If the last three quarters have taught us anything, it’s that I am hardly the only one out there who thinks so.

Since the last time I wrote on the subject, edge-focused deep learning accelerators have moved from the realm of press releases, and circuits integrated within mobile phones, to the palms of our hands. In my case, quite literally, as I could not resist getting my hands on a Jetson Nano right there at the Silicon Valley GTC in March. And since I got something that can achieve 64 frames per second inference with MobileNet-v2⁷ for $99, I have no regrets. Google⁸ and Intel⁹ are not lagging and have made their own edge inference accelerators available as USB sticks with competitive pricing. And all these vendors are offering software to make standard deep learning models more suitable to run on these devices with limited, albeit impressive, computational capabilities. AI has not so much edged, but leapt, towards the edge in the last few months.

The only bad news that I have to report is of our microbot of the future, relying on neuromorphic hardware. The development in the domain of neuromorphic computing seems to be following a more conventional path. But never fear, groups such as that of Arijit Raychowdhury¹⁰ at Georgia Tech are already prototyping low-power AI accelerators focused on micro bot (swarm) applications. And although International Conference on Robotics and Automation (ICRA) has been a much smaller affair than NeurIPS or GTC, with a much more “same old stuff” feel, I never tired of watching Bitcraze Crazyflie¹¹ swarm take off, fly around the enclosure and land to recharge, before doing it all again. The 5-person company has been around since 2011 and targets researchers while following a strict open source policy, allowing it to capitalize from improvements by teams around the world. The drones can follow trajectories autonomously (but they need to use a system based on anchors to determine their position) and are due to get peer-to-peer communication capability in the next few months. Again, the price of the drone is very affordable. You can get it for $200 or so. While we are waiting for the neuromorphic systems to deliver on their promise, we have all the things we need deploy AI and CV on the edge, and the edge might be moving under its power while hanging out with a swarm of its friends.

References:

¹ https://nips.cc/
² https://www.nvidia.com/en-us/gtc
³ https://www.icra2019.org
⁴ Devlin, J., Chang, M. W., Lee, K., & Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
⁵ Vedantam, R., Desai, K., Lee, S., Rohrbach, M., Batra, D., & Parikh, D. (2019). Probabilistic Neural-symbolic Models for Interpretable Visual Question Answering. arXiv preprint arXiv:1902.07864.
⁶ Mao, J., Gan, C., Kohli, P., Tenenbaum, J. B., & Wu, J. (2019). The Neuro-Symbolic concept learner: Interpreting scenes, words, and sentences from natural supervision. arXiv preprint arXiv:1904.12584.
⁷ https://developer.nvidia.com/embedded/jetson-nano-dl-inference-benchmarks
⁸ https://coral.withgoogle.com/
⁹ https://software.intel.com/en-us/movidius-ncs
¹⁰ Cao, N., Chang, M., & Raychowdhury, A. (2019, February). 14.1 A 65nm 1.1-to-9.1 TOPS/W Hybrid-Digital-Mixed- Signal Computing Platform for Accelerating Model-Based and Model-Free Swarm Robotics. In 2019 IEEE International Solid-State Circuits Conference-(ISSCC) (pp. 222-224). IEEE.
¹¹ https://www.bitcraze.io/

As an engineer, I love reuse. So, when the time came to write the first article for our blog, I went sifting through the stuff I had on my hard drive. Some of it actually did not look too bad, if you are looking for ways to bootstrap your blog-writing process and don’t mind being judged by readers who have the benefit of knowing the future.

To provide some context, while I served as the Senior Research Scientist for the Tandemlaunch startup foundry in Montreal, Canada, I had the pleasure to write a column for the Tandemlaunch Industry Report. I’ve reached out to my old friends and colleagues and they have kindly agreed to let me republish those articles as part of the Cinteraction blog. I’ve selected the two that have dealt with the future of Computer Vision as I saw it back in 2018 and 2019. Since you will probably be feeling a bit like Marty McFly while you are reading them, I’m now calling them Back of the Future series, but I’ve refrained from modifying them in any other way. So without further ado and thanks to Tandemlaunch, I present to you:

On Deep Learning and The Future of Computer Vision (written in September 2018)

When the idea of writing this contribution was first pitched to me, I reacted instinctively: “Right, let me just get the old crystal ball out!” That one needs such an implement to complete a task such as this is best illustrated by the fact that if you approached 99.9% of Computer Vision (CV) researchers in 2011 and told them that in a few years 90% of the domain will boil down to Deep Learning (DL) you would be treated to an indignant stare and something along the lines of: “Pull the other one. It's got bells on.” On second thought, I remembered some interesting discussions with amazing people at a Dagstuhl seminar a few years back and decided that there might be interesting things to say on this topic, even if one is a crystal-dyslexic CV enthusiast.

This was 2015, and CV has already been revolutionized by DL, so the discussions basically focused on the limitations of DL. When you scratch the surface, there are quite a few, the most prominent being: 1 - that they basically provided no well-structured description of the world that you could work with and 2 -you need a seriously powerful GPU to train - or even run - these things at a decent frame rate. Three years later these problems remain, although we’ve made serious headway on the latter and more and more researchers are expressing concerns and working toward resolving the first.

Don’t get me wrong, I am a huge Neural Net (NN) fan and was using them to do video segmentation back in 2005, but we should be clear on what the Convolutional Neural Nets (CNNs) actually brought to the CV table. And this is an, admittedly huge, improvement in the way we extract features that drive our machine learning. As a consequence, we were, in some cases kicking and screaming, forced to thrash the manually designed approaches used in the two preceding decades. And since the neural nets came with their own classification engine built-in, this also ended the reign of SVMs as everyone’s favourite classifier. Small price to pay for a technology that has finally enabled us to move on from the “this is a cat and this is a dog” problems¹ to actually being able to get the pixels occupied by every single cat in the image².

Now, while CV was struggling to get ahead on classifying images with any kind of decent accuracy, the text-mining people have had this sorted out a long time ago and have since moved to creating models of the world that would enable computers to understand not only the basic entities, but also the relationships between them using ontologies. It was not unreasonable to expect CV to produce a similar model of the world, or to map visual data to an already existing model. In fact, the creators of the ImageNet Large Scale Image Recognition Challenge - with which the revolution of CV is inextricably linked - have had this in mind. At the advent of DL, the CV community was competing to identify 1000 classes pertaining to synsets (sets of cognitive synonyms) of WordNet³. Unfortunately, the appearance of objects in images (and, naturally, video) exhibits far greater variance than the related concepts in text, so, while the text-mining community was trying to understand the relationships, we were trying to build models (design features) that would make CV algorithms agnostic to such transformations. In our bid to understand that a coffee mug was in the image, we lost the ability to say where and what it was resting on and how it was oriented. Things that humans learn at a very young age; understanding that you can put a different object in the same place, that it may still be there even if you don’t see it at the moment and that it looks like a fluffy toy you once had.

Most people in CV are aware of this, and if they’re not focused on solving the problem of DNN efficiency or trying to apply DNNs to the handful of CV problems where it has yet to beat the state of the art, they are chugging away towards better models to represent the visual world, with all its intricacies. There exists some variability in the approach and, among them, there are several groups worth keeping an eye out for. Joshua Tenenbaum’s group at MIT CSAIL is working towards representing the scenes as probabilistic programs⁴, while FacebookAI is representing the scene and relationships within it, with a more traditional type of program, parametrized by the output from a CNN⁵. But the guy to lookout for, in my mind, is Goeff Hinton, whose inspiration is derived from computer graphics and whose group has managed to create Capsule NNs that encode the transformations of the objects as well as their appearance⁶. By now Hinton is widely known as the person who kept the NN torch burning during the SVM era of computer vision and machine learning, and he continues to surprise. If there is a crystal ball of neural networks, it is probably broadcasting Hiton’s talks 24/7. This is the man who advocated for moving beyond the limitations of CNNs, just as the CV community was trying to come to terms with the revolutionary results on ImageNet produced by his associates. Right now it seems that CV is all about the NNs for the foreseeable future, and it might be that in a few years CNNs in CV will mean Hinton’s new Capsule and not the Convolutional NNs.

What these networks will run on and how much power will they use is another question entirely. A crucial element of the DL revolution has been the existence of powerful GPUs (next time you get the urge to tell a teenager that playing video games serves no purpose, you may want to think again). Although GPUs on cloud servers still account for the major part of computing power used to process visual data, we are witnessing a proliferation of hardware designed primarily to allow for the efficient execution of DNNs, often in embedded scenarios. DNN-based CV solutions can now be found running in cars, drones and high-end mobile phones (equipped with chips designed to support DNN applications). While the new hardware is powerful in comparison, there is a whole stream of research dealing with making CNNs, as the primary tool of CV, fit the constraints of the hardware. Pruning, quantization - often to the point of binarization - and AI algorithms designed to optimize CNNs are used increasingly. As this is an engineering problem, it is safe to predict that the applications of CV will spread to edge and fog devices and, in time, probably come to be limited by the meaningfulness of applications rather than capabilities of hardware.

Where the future of CV gets blurry in this direction is the advent of neuromorphic hardware and very recent results showing that such architectures could be coaxed in performing the CV tasks using Spiking Neural Nets, instead of our classic (generation II) NNs, while consuming much less energy and dissipating less heat. Both IBM (TrueNorth) and Intel (Loihi) already produce neuromorphic hardware, but the state-of-the-art CV approaches are just starting to be ported to such architectures. Accuracy still lags behind the classical DNNs, but the gap is rapidly closing.

The future of CV is still the same as it always was. CV is a gateway towards achieving human-level general AI, by creating ways for computers to understand the visual world to the extent that humans do. And we know that when something we’ve never seen before looks like a ball, bounces off the wall and goes under something that looks like a filing cabinet, it is probably a ball and somebody had better fetch a stick to get it out. If you happen to be a neuromorphic-sporting microbot reading this a few decades in the future, the stick is optional.

References:

¹ Krizhevsky, Alex, Ilya Sutskever, and Geoffrey E. Hinton. "Imagenet classification with deep convolutional neural networks." Advances in neural information processing systems. 2012.
² He, Kaiming, et al. "Mask r-cnn." Computer Vision (ICCV), 2017 IEEE International Conference on. IEEE, 2017.
³ Miller, George A. "WordNet: a lexical database for English." Communications of the ACM 38.11 (1995): 39-41.
⁴ Tenenbaum, Josh. "Building Machines that Learn and Think Like People." Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems. International Foundation for Autonomous Agents and Multiagent Systems, 2018.
⁵ Johnson, Justin, et al. "Inferring and Executing Programs for Visual Reasoning." ICCV. 2017.
⁶ Sabour, Sara, Nicholas Frosst, and Geoffrey E. Hinton. "Dynamic routing between capsules." Advances in Neural Information Processing Systems. 2017.