To try out
Understanding artificial intelligence (AI)
“If you understand how a model learns, you also know where it goes wrong.”
How do models learn from data? From a language model in miniature to the oldest tool of statistics: three pieces to touch, explained simply. Everything is calculated in your browser; nothing is stored or sent.
First: AI in everyday life
What is behind the things you use every day.
Many things in everyday life run on exactly the methods you can try on this page:
- ChatGPT, ClaudeA language model predicts the next word, again and again, learned from huge amounts of text.Try it: Language model
- Alexa and other voice assistantsA neural network turns speech into text; newer assistants answer with a language model.Try it: Language model
- Driver assistance in the carLane keeping and emergency braking recognise lanes, cars and people in the camera image, today mostly with neural networks.Try it: Neural network
- Ads and suggestions while surfingFrom clicks, searches and purchases, models estimate what interests you, often many decision trees together with regressions.Try it: Decision tree
- Music and films suggestedSongs and films become numbers, like words; similar ones lie close together, as on the word map below.Try it: Words as numbers
- Loans and insuranceScores estimate from a few features how likely a loan is to be repaid, for decades with regressions, today also with trees.Try it: Regression
Simplified; providers rarely say exactly which method they use.
Old methods, new computing power
What you can try on this page, and why everyone can use it only now.
Artificial intelligence sounds new. The methods behind it are not: the line through a cloud of dots dates from 1805, the first decision tree from 1963, the first learning artificial neuron from 1958. What is new is computing power. Only when computers became fast and cheap enough could everyone use the methods: regression with the PC in the 1980s, when spreadsheets such as Lotus 1-2-3 calculated it at a command; decision trees and their forests with free software in the 2000s, such as the randomForest package for R in 2002. Neural networks were already at work in the 1990s, but small and for narrow tasks: in American banks they read several million checks a day. Deep networks with many layers could hardly be trained then, and between 1991 and 2015 computing power grew about a million-fold, mostly through graphics cards. In 2012 a deep network trained on two gaming graphics cards won the most important image-recognition contest; since 2015 at the latest, when Google released its TensorFlow software for free, anyone can train such networks. Since the mid-2010s they have been at work in speech recognition and translation; ChatGPT is only the best-known product: two months after its launch at the end of 2022 it had an estimated 100 million users.
1958 first learning neuron (Rosenblatt) · 1986 learning by working back through the errors (backpropagation) · 1990s small networks read checks in banks, deep ones can hardly be trained (Hochreiter 1991) · 2012 AlexNet, trained on two gaming graphics cards, wins the ImageNet contest (deep learning) · 2015 TensorFlow free for everyone · 2017 transformer, the design of today's language models · November 2022 ChatGPT, an estimated 100 million users two months later
1963 first tree (Morgan and Sonquist) · 1984 CART · 2001 random forests, many trees at once (Breiman) · 2002 free for everyone in the R package randomForest
1805 Legendre, 1809 Gauss · 1980s spreadsheets on the PC such as Lotus 1-2-3 calculate it at a command
inventedwaiting for computing power for everyone
Model 1 of 3
A language model in miniature
At their core, Claude or ChatGPT do the same: they predict the next word. Here with ten short texts instead of a large part of the internet.
1 Read and learn
The model splits the texts into words and learns which word follows the two before. Each word becomes a row of numbers. You can change the texts and let it learn again.
Settings for learning
More numbers and neurons mean more parameters and more computing, and the model learns the texts by heart sooner.
2 A look inside the model
Type a beginning. On the left the last words, next to them their numbers, in the middle the neurons, on the right the most likely next words. Teal means positive, plum negative; the stronger, the more. The lines show what contributes most to the most likely word. Tap a word to add it.
3 Keep writing
The model picks word after word. Creativity sets how often it departs from the most likely word: low repeats the texts, high mixes them anew, and too high becomes nonsense. In the look inside the model above you can follow how it calculates word by word.
4 Words as numbers
Each word is a row of numbers. Shown on a map, words that are used in a similar way end up close together. Nobody told the model which words belong together.
What large models do differently
They learn from trillions of words instead of ten sentences and have billions of numbers instead of a few thousand. They look not only at the last two words but at a great deal of text before, often as much as a whole book, and weigh which parts matter right now (attention). Afterwards they are refined with feedback from people, so that they answer helpfully instead of merely continuing text. The principle stays the same: predict the next word.
Model 2 of 3
Decision tree: questions, one after another
A tree asks yes-or-no questions, one after another, until the groups are as pure as possible. In research I use trees to sort shares, for example.
Each dot is a day: to the right the temperature, upwards the chance of rain. Teal: went for a walk; plum: stayed at home. The tree learns the rule from the days. More depth means more questions.
Model 3 of 3
Regression: a line through the cloud
The oldest tool of statistics, and still one of the most useful: how much does one thing change when another changes?
Each dot is a summer day at an ice-cream parlour: to the right the temperature, upwards the scoops sold. Tap to add a dot, drag a dot to move it. The line lies where the squares together are smallest: least squares (OLS).
How the models hang together
More complex, but nested inside each other: the straight line is in every one of them.
The three models are not separate worlds. A decision tree is a regression on yes-or-no questions and their combinations. A neural network is a regression on kinks that it learns itself, many layers one after another. Take the kinks away and the straight line remains. More complexity means more interactions, more non-linearity and more computing; the core stays the same: a number plus a number times a feature.
- 1Regressiony = a + b × xa straight line
- 2Decision treey = a + b1 × [x ≥ s1] + b2 × [x ≥ s1 und z ≥ s2]a regression on yes-or-no questions; the “and” is the interaction
- 3Neural networky = a + b1 × max(0, c1 + d1x) + b2 × max(0, c2 + d2x)a regression on kinks it learns itself, in many layers
- 4Language modelthe same, billions of times, with words instead of xa very large network
Each dot is a day at an ice-cream parlour over the year. When it gets too hot, people stay inside. Pick a model and how complex it may be; below it the model is written out as a regression.
scoops sold
temperature, °C
Always the same form: a number plus a number times a feature. Only the features become more refined.
In practice
These tools are not a toy for me: I have used them in research since 2013.
- 2013An AI model with decision trees for picking shares, run on the Harvard supercomputer; in 2015 the best paper at the annual meeting of the German Finance Association.
- 2018PhD on textual analysis and machine learning in capital markets research, awarded the PLEXUS prize for artificial intelligence in finance.
- TodayWith students and partners at universities, such as UC Berkeley: machine learning in choosing funds, and models for scenarios.
All projects in Career and research
Anyone who would like to know how such models can sensibly be used is welcome to write to me: mail (at) benjaminmoritz (dot) de