Proud Canadian here. Canada is hot right now and I thought this story of Canadian contribution to AI would be interesting to share.

A lot of the LLM technology that has hit the market in the last few years traces back through a set of breakthroughs where the key people were Canadian, or were doing their work in Canada. It isn’t the whole story of AI, but it’s a big through-line, and it starts at the University of Toronto.

The Hinton lab at University of Toronto

Geoffrey Hinton spent decades at U of T pushing on neural networks when most of the field had given up on them. In 2006, Hinton and collaborators published a paper on Deep Belief Networks that showed you could actually train deep, multi-layer neural networks effectively. That paper is often credited with reviving deep learning during the AI winter, and it’s when the rest of the field started paying attention again.

Six years later, in 2012, two of his graduate students, Ilya Sutskever (Soviet born Israeli Canadian) and Alex Krizhevsky 🇨🇦, worked with him on a convolutional neural network called AlexNet. They entered it into the ImageNet competition and beat the field by a wide margin. That result is what most people point to as the moment deep learning became the dominant paradigm in AI. The source code is being preserved at the Computer History Museum in Silicon Valley, which tells you a bit about how significant the moment was.

A lot of that work was sustained through the AI winter by CIFAR, the Canadian Institute for Advanced Research, which funded Hinton for years when most of the field thought neural nets were a dead end. I think that’s an important piece as it speaks to the need for funding of pure research and long term bets. Without CIFAR, this timeline probably doesn’t happen.

Hinton went on to win the 2024 Nobel Prize in Physics for that lineage of work. He’s often referred to as the godfather of AI, which he probably finds embarrassing, but which is broadly accurate. He’s a fascinating human.

The move to Google

Right after AlexNet, Hinton, Sutskever, and Krizhevsky formed a spinoff called DNNResearch. Google acquired it in 2013, which brought that group into Google Brain. Sutskever spent the next few years there, working on things like sequence-to-sequence learning and AlphaGo, and then in 2015 he co-founded OpenAI with Sam Altman and Greg Brockman, where he was Chief Scientist for nearly a decade. I think it’s fair to say that without Hinton’s lab in Toronto, the OpenAI we know doesn’t exist.

The attention paper

The other landmark paper in this story is “Attention Is All You Need,” published out of Google Brain in 2017. It introduced the transformer architecture that every modern LLM, including the ones we all use every day, is built on.

Aidan Gomez 🇨🇦 was one of the eight authors on that paper, and he had a very Toronto path to it. As a U of T undergrad, he cold-emailed Hinton with an idea for a new activation function. Hinton wrote back and pointed him toward Roger Grosse and the U of T machine learning group. That connection is what eventually pulled him into research at Google Brain in Mountain View, where he ended up working alongside Hinton on the paper.

Aidan has said in interviews that the paper wasn’t a grand plan. The goal at the time was narrow: make Google Translate roughly 3% better and model sequences more efficiently. The rest of the field then applied the same architecture to images, audio, and video, and it took over pretty much everything.

Aidan wasn’t the only U of T person to end up at Google Brain. Nick Frosst 🇨🇦 did his undergrad at U of T under Hinton and was Hinton’s first hire at Google Brain when the Toronto office opened, where he stayed for four years. Both of them eventually left Google.

What isn’t in this story

Canada is a big part of this story but not the whole of it. The ImageNet dataset that AlexNet won on came from Fei-Fei Li at Stanford. NVIDIA’s CUDA and the GPUs that made deep learning economically viable came from Silicon Valley. Seven of the eight authors on the transformer paper are not Canadian. And the scaling that turned these ideas into products people actually use, GPT-3, GPT-4, ChatGPT, is largely American engineering backed by American capital.

Canada seeded a lot of the ideas and a lot of the people. The value capture happened elsewhere.

Cohere and coming home

That’s part of what makes Cohere exciting to me, and hopefully other Canadians. I know Mark Carney feels the same. After the transformer paper, Aidan went to Oxford for a PhD. He’s talked about watching the language models get dramatically better from there, and in particular the GPT-2 moment, when models went from barely stringing a sentence together to producing plausible, fluent text. He called Nick Frosst and Ivan Zhang 🇨🇦, and in 2019 the three of them co-founded Cohere in Toronto. They deliberately built the company in Canada, and they’ve talked openly about data and AI sovereignty as a value the company is trying to embody. I honestly can’t think of a more Canadian set of values guiding a company. Cohere is now one of the more serious enterprise LLM providers in the world.

Incredible story arc with Canadians at the centre. The 2006 revival of neural networks at U of T. The 2012 breakthrough. The spinoff into Google. The transformer paper with a Canadian undergrad on it. Sutskever helping start OpenAI. And a set of those researchers coming back to Toronto to build a Canadian AI company on their own terms.

Why I wanted to write this

We in Canada are quietly aware of how educated the country is, but we don’t always talk about what that education has produced. This is one of those cases where the outcomes are enormous and mostly global, and Canada often doesn’t capture the credit or the economic upside.

I like being reminded that some of the best ideas come from academia, and it’s the relentless pursuit of a big idea, built on decades of other big ideas, that sometimes results in world-changing technology. It’s pride inducing for me, and I hope it is for other Canadians reading this. And the Cohere story especially, of coming home to build something in Canada with a specifically Canadian set of values, is inspiring to me.