← Index / 2026-07-28

The Angle Between Two Sentences

Waypoint is a chatbot I built for my school. You ask it a question — where a club meets, who to email about a parking permit — and it answers from the school’s own handbooks and announcements rather than from whatever a language model happened to pick up off the internet.

It works in two steps.

First it searches the school’s documents and pulls out the paragraph most likely to contain the answer. Then it hands that paragraph to a language model and says, in effect: answer the question using only this.

Almost every mistake Waypoint made happened in the first step. The model itself was rarely at fault — it did exactly what it was told and wrote a clear answer from the paragraph it was given. But if that paragraph is the wrong one, a clear answer built from it is a confidently wrong answer. From the outside this looks like the model making things up. Usually it isn’t. It is the search step quietly failing, and nothing about the reply that comes out looks like a failure.

So this post is about the search step, and the question sitting underneath it is stranger than it first sounds: how do you get a computer to notice that two pieces of writing are about the same thing, when it does not know what a single word means?

The answer is that you turn each piece of writing into an arrow, and then check whether the two arrows point the same way. That sounds like a metaphor. It isn’t. It is literally the arithmetic, and it is simple enough that by the end of this you could do it on paper.

Everything you can drag or click below is running the real calculation, live, in your browser. There is no model and no server behind any of it.

Step one: turn a sentence into numbers

A computer cannot compare two sentences. It has no concept of a sentence. It can compare numbers, and not much else. So before anything can happen, a sentence has to become numbers.

The oldest way of doing this is also the easiest to picture. You count the words.

Suppose the only three sentences in existence are these:

Chess club meets Thursday.
Robotics club meets Friday.
The library closes Friday.

Write out every different word that appears anywhere in the three: chess, club, meets, Thursday, robotics, Friday, the, library, closes. Nine words. That list is the vocabulary, and it is fixed from here on — it is the complete set of boxes we are allowed to use.

Now any sentence can be written as nine numbers, one per box, each holding how many times that word appears.

Chess club meets Thursday becomes a 1 in the chess box, a 1 in club, a 1 in meets, a 1 in Thursday, and a 0 in the remaining five. Nine numbers: 1, 1, 1, 1, 0, 0, 0, 0, 0.

That row of numbers is the sentence now, as far as the computer is concerned. Nothing else survives.

Two bits of housekeeping happen first.

Words like the, is, and of turn up in nearly every sentence ever written, so learning that a sentence contains the tells you nothing about what it is about. Those get thrown out before counting. They are called stop words.

And meets and meet should not get separate boxes, since they are the same word wearing a different ending. So a rough rule chops endings like -s off so both land in one box. Mine is three lines long and gets a fair number of words wrong. Professional ones are longer and still get words wrong.

Here is that whole process on a real sentence. Struck-through words are the ones being thrown away:

Count the words that are left

Pick a line, or write your own. Struck-through words get thrown away before counting — they turn up in almost every sentence, so they say nothing about the subject.

your own line

chessclubmeeteverythursdayafterschoolinroom214

2141

chess1

club1

meet1

room1

school1

thursday1

Those numbers are now the entire sentence, as far as the computer is concerned. In a real system there is one box for every word in the whole collection, so nearly all of them sit at zero.

In code the entire thing is about six lines:

import re
from collections import Counter

STOP = {"the", "is", "a", "of", "in", "to", "and", "on", "at"}

def tokenize(text):
    words = re.findall(r"[a-z0-9']+", text.lower())
    return [stem(w) for w in words if w not in STOP]

def count_vector(text):
    return Counter(tokenize(text))

Two things to notice before we move on.

The first is scale. Three sentences gave us nine boxes. A real collection of documents has thousands of different words in it, so every sentence becomes a row of thousands of numbers — of which a handful are 1 or 2 and all the rest are 0. That is normal and it is fine. It just means these rows are enormous and almost entirely empty.

The second is what we gave up. This representation has a name that admits its own weakness: bag of words. Tip the sentence into a bag and shake it. The teacher approved the roster and the roster approved the teacher come out as exactly the same row of numbers, because they contain exactly the same words. Word order is gone. Keep that in your pocket — it comes back at the end.

Step two: how alike are two rows of numbers?

Put sentences aside for a moment. Suppose I hand you two shopping lists.

Yours: 2 apples, 1 loaf of bread, 3 cartons of milk. Mine: 1 apple, 0 bread, 1 carton of milk.

How similar are these lists? Here is about the simplest scheme that could work. Go item by item, multiply the two amounts together, and add up everything you get.

Apples: 2 × 1 = 2. Bread: 1 × 0 = 0. Milk: 3 × 1 = 3. Total: 5.

That number is doing something sensible, and it is worth seeing exactly what.

An item only adds to the total if it is on both lists. Bread contributed nothing, not because bread is unimportant, but because one of us bought zero of it and anything times zero is zero. And the score rewards emphasis: milk contributed the most because we both bought a lot of it.

So the total is a measure of agreement, weighted by how strongly each side agrees. Run the same scheme on two sentences instead of two shopping lists and you have the core of the whole thing. This operation has a name — the dot product — and it is written like this:

\mathbf{a}\cdot\mathbf{b}=\sum_i a_i b_i.

That symbol means nothing more than what we just did by hand: go box by box, multiply the two numbers sitting in that box, and add up the results.

Where the arrows come in

Now for the part that surprised me the first time I saw it.

Shrink the vocabulary down to just two words, so a sentence is only two numbers. Two numbers can be drawn: go that far across, then that far up, and put a dot there. Draw a line from the corner out to the dot and you have an arrow.

So every sentence is an arrow. And two sentences are two arrows setting off from the same corner, with some angle opening up between them.

Two sentences about the same thing use similar words, so their arrows lean the same way and the angle between them is small. Two sentences with nothing in common have arrows sitting at a right angle to each other.

Drag either arrow tip below and watch the bottom two rows of the readout — that angle is the thing we are chasing:

Two sentences, two arrows

Only two words in the vocabulary, so a sentence is just two numbers — and two numbers can be drawn. Drag either arrow tip, or click one and use the arrow keys.

sentence a: (1, 1)
sentence b: (4, 0)
dot product: 4.00
lengths ‖a‖ ‖b‖: 5.66
score cos θ: 0.707
angle θ: 45.0°

Word counts are never below zero, so both arrows stay in this corner and the score stays between 0 and 1. Drag a tip straight away from the corner: the arrow gets longer, the dot product grows, and the angle does not move at all.

Play with it for a second and something should become clear: the angle is not affected by how long you make an arrow. Drag a tip straight outwards, away from the corner, and the angle does not budge. Only swinging it around changes anything.

That is the property we want, and it is exactly the property the dot product on its own does not have. Make an arrow twice as long and its dot product doubles too, even though it is pointing in precisely the same direction as before.

Happily, geometry gives us the exact relationship between the two. This is a standard fact, not something I invented:

\mathbf{a}\cdot\mathbf{b}=\|\mathbf{a}\|\,\|\mathbf{b}\|\cos\theta.

Unpacked, it says: the dot product equals the length of the first arrow, times the length of the second arrow, times a number that depends only on the angle between them.

The two length symbols, $\|\mathbf{a}\|$ and $\|\mathbf{b}\|$ , are ordinary distances — how far each arrow reaches from the corner, worked out with the Pythagorean theorem you already know, just with more than two numbers to square and add.

The last piece, $\cos\theta$ , is the only new thing, and if you have never done trigonometry you do not need to start now. All you need is that $\cos\theta$ is a dial that depends on nothing but the angle:

it reads 1 when the two arrows lie right on top of each other,
about 0.7 when they are 45 degrees apart,
and 0 when they meet at a right angle.

Look at that equation again and you can see the problem and the fix at once. The dot product is the thing we want — the dial — multiplied by two lengths we do not care about. So divide the lengths back out:

\cos\theta=\frac{\mathbf{a}\cdot\mathbf{b}}{\|\mathbf{a}\|\,\|\mathbf{b}\|}.

Everything on the right is something we can work out from two rows of numbers. What comes out is a score that depends only on which way the arrows point and not at all on how long they are.

That score is called cosine similarity, and the rest of this post rests on it. Two sentences using the same words, in the same proportions, score 1. Two sentences with no words in common score 0. In code it is one line:

import numpy as np

def cosine(a, b):
    return float(a @ b / (np.linalg.norm(a) * np.linalg.norm(b)))

One loose end. Real vocabularies have thousands of boxes, not two, so the arrows live in thousands of dimensions — which nobody can picture, including me. But look at that formula: nothing in it mentions how many boxes there are. It is addition, multiplication, and a square root, and those do not care. The two-word picture you just dragged around is not a watered-down version of the real thing. It is the real thing, drawn small enough to see.

Why the direction and not the distance

There is an obvious objection to all of this. If both sentences are just dots on a page, why bother with angles? Why not measure the plain straight-line distance between the two dots and call the closer one more similar?

Here is the reason, and I think it is the clearest way to understand what the angle is actually measuring.

Think of a row of word counts as a recipe.

A smoothie made with 2 bananas and 1 cup of berries and a smoothie made with 4 bananas and 2 cups of berries taste the same. Same recipe. One is just a bigger batch.

The length of the arrow is the size of the batch. The direction of the arrow is the recipe.

Now apply that to documents. A document that says the same thing twice is a bigger batch of the same recipe. It is not less relevant to your question — it is the same thing, more of it. But its arrow is twice as long, so straight-line distance says it has drifted further away. Measure by distance and you quietly punish every document for being long.

Watch both measures at once. The document below is one sentence, repeated. Nothing new is being said, only more of it:

Saying it twice does not make it truer

The query is design lab hours. The document is one sentence, repeated. Nothing new is being said — only more of the same.

copies: 1
‖d‖: 2.45
distance: 1.73
cosine: 0.707

copies of the sentence1

Straight-line distance keeps climbing, so by that measure a longer document is always less relevant. The score by angle does not move by a thousandth. Length is the size of the batch; direction is the recipe.

Distance climbs steadily and never stops. The cosine does not move by a thousandth, and that is not luck. Doubling a row of numbers doubles the dot product on top of the fraction and doubles the lengths on the bottom, and the two cancel exactly. Dividing by the lengths is precisely the step that throws away how much was said and keeps what it was about.

Some words are worth far more than others

Counting treats every word as equally useful for identifying something. Words are nothing like equally useful.

Suppose I ask you to find one particular student in a school of two thousand, and all I give you is a description. If the description says the person is wearing shoes, you are no closer than when you started — everyone is wearing shoes. If it says the person plays chess, you might be done in a minute.

Words in documents behave the same way. In a school handbook, half the pages mention a club. One mentions chess. Matching on chess is nearly the whole answer; matching on club barely narrows anything.

So we weight each word by how rare it is. If there are $N$ documents in total and a word turns up in $n_t$ of them, that word’s weight — its inverse document frequency — is

\text{idf}(t)=\log\frac{1+N}{1+n_t}+1.

In plain terms: count how many documents contain the word, and hand out bigger weights to words that appear in fewer of them. A word in every document ends up near the bottom. A word in a single document ends up near the top. The logarithm is only there to stop the very rarest words from running away with everything.

import math

def idf(term, documents):
    appears_in = sum(term in doc for doc in documents)
    return math.log((1 + len(documents)) / (1 + appears_in)) + 1

Multiply every count by its word’s weight before taking the cosine, and rare matches start to outweigh common ones. Search engines ran on roughly this idea for decades, under the name TF-IDF.

Below are nine stand-in handbook lines and three ways of scoring them: by how many words the query and the line literally share, by the angle using raw counts, and by the angle after rare words are given more weight.

Rank nine lines

Nine stand-in handbook lines. Three ways of scoring them against the same query.

query

Chess club meets every Thursday after school in room 214.
2
Robotics club is sponsored by the engineering teacher and meets in the design lab.
1
The design lab is open to any student during club hours on Wednesday.
1
Club sponsors submit a roster to the front office at the start of each semester.
1

Top line: number 2 at 2.

Try chess club through all three. There is a decoy line in there that says club four times and answers nothing at all. Counting literal shared words, it looks harmless. Switch to the raw-count angle and it jumps to first place, 0.58 against the chess line’s 0.54 — purely on the strength of repeating one cheap word. Turn on rare-word weighting and it drops to second, because its four matches were all on the least informative word available while the chess line matched on the most informative one.

Then try the third preset — who is in charge of the build team — and watch every single score fall to zero.

The wall

That query is a wall, and it is the same wall Waypoint kept walking into.

Not one word of it appears anywhere in the collection. The handbook says sponsor; the student said in charge of. The handbook says robotics club; the student said build team. Every arrow in the collection sits at a perfect right angle to the question. The search finds nothing, the model is handed nothing, and it answers anyway.

You cannot fix this by counting more cleverly, and the reason is worth being precise about. Each box in these rows is one word, and the boxes know nothing about each other. The sponsor box and the advisor box are exactly as unrelated as the sponsor box and the hallway box — they are simply different boxes. There is nowhere in this whole system to record that two words mean nearly the same thing. No weighting scheme can put information somewhere that has no room for it.

What you need is a system where the boxes are not words at all. That is what an embedding is.

Instead of one box per word, a piece of text is turned into a few hundred numbers by a trained model. A few hundred numbers is still an arrow, so we still measure angles between them in exactly the way described above — that arithmetic does not change at all. What changes is where the arrows end up pointing.

How does a model decide where to point them? The idea behind it is old and surprisingly blunt.

Imagine joining a school halfway through the year and trying to work out who is friends with whom, with nobody telling you anything. You would not need names. You would just watch who sits together, who turns up at the same places, who leaves at the same time. People who keep the same company are connected.

Embedding models are trained on that idea applied to words. A model reads an enormous amount of text and notices that sponsor, advisor, and in charge of keep appearing surrounded by the same other words — clubs, teachers, approving things, signing forms. So training nudges them to point in the same direction. Nobody ever tells the model they are synonyms. It works it out from the company they keep.

And once they point the same way, the cosine formula from earlier finds them with no words in common whatsoever. Drag the marker and watch which phrases light up:

Neighborhoods with no words in common

Drag the marker, or focus it and use the arrow keys to swing it and +/− to push it in and out. The five closest phrases light up.

runs · sponsor · advisor · in charge of · supervises

Push the marker close to the middle and switch to distance: the neighbors scramble, because near the origin everything is nearby. The angle does not care how far out you are.

I placed those phrases by hand, and I want to be straight about that: it is a drawing, not the output of a model. A real embedding puts each phrase somewhere in a few hundred dimensions, the positions come out of training on a huge amount of text, and none of the axes mean anything a person could read. What survives the simplification is the part that matters — meaning is carried by direction, and things with no words in common can still point the same way.

Where the answer actually got cut in half

Swapping counts for embeddings fixed the synonym problem, and Waypoint got noticeably better. It also kept getting one kind of question wrong, and the cause had nothing to do with counts or embeddings.

A whole document is too long to turn into a single arrow. If you counted every word in a twelve-page handbook, you would get one arrow pointing at the average of everything the handbook discusses — parking, and clubs, and dress code, and lunch — which is to say pointing at nothing in particular. It would be a mediocre match for every question and a good match for none.

So you cut the document into pieces first and give each piece its own arrow. This is called chunking. The simplest possible version counts off a fixed number of words and cuts:

def chunks(words, size, overlap):
    step = size - overlap
    return [" ".join(words[i:i + size]) for i in range(0, len(words), step)]

Below is a short passage that does contain the answer to who sponsors robotics club. Cut it into pieces and search the pieces:

Cut the passage, then search it

The question is who sponsors robotics club. The answer is wavy-underlined. Bold words are the ones the question and the chunk share.

chunk size (words)14overlap (words)0

Every club at this school needs a staff sponsor. The sponsor approves the roster,
0.548
attends one meeting a month, and signs off on fundraising. For robotics, that job
0.204
belongs to the engineering teacher, who also keeps the design lab open on Wednesday
0.000
afternoons. Any club still missing one by the second week of the semester comes
0.192
off the activities list.
0.000

The top chunk scores 0.548 and does not contain the answer — it is about sponsors and clubs in general, which is exactly why it won. The chunk holding the answer comes 4th, at 0.000.

The sentence holding the answer never says “sponsor” or “club” — it says “that job.” On its own it looks almost unrelated to the question. It only ranks when it stays attached to the sentence that names the thing it is referring to.

Start small and look at which piece wins. It is the one explaining that clubs need sponsors in general — every important word from the question is in it, and no answer.

Meanwhile the piece that actually names the engineering teacher scores 0.000. Dead last. And here is why, because this is the whole bug in one observation: that sentence never repeats the words sponsor or club. It says “that job.”

Reading normally, you carry the subject across the sentence break without noticing you are doing it. The second sentence does not need to restate what it is about, because the first one just said it. But cut between them, and the piece holding the answer no longer contains any of the words that say what the answer is to. On its own it reads as though it is about something else entirely.

That is the failure, and it took me far longer to find than it should have, because from the outside it was indistinguishable from the model inventing things.

Now push either slider up and watch where the answer chunk ranks. It climbs — fourth, third, second — and still stays behind, because every chunk beating it is beating it for the same reason: the words that identify the club and the words that answer the question are in different sentences. Then at thirty-three words the cut finally falls past the end of that sentence, both halves land in the same piece, and the top result contains its own answer. There was nothing to tune in between. Either a reference and the thing it refers to are in the same piece of text, or they are not.

Drag it the other way instead, down to eight words, and it fails in a way I did not see coming: the cut lands inside the answer, so engineering and teacher end up in different pieces. Retrieving more results cannot save you there, because the answer has stopped existing anywhere in the index as a whole phrase.

None of this is special to this passage. Pronouns, this, that role, the above, a heading that applies to the six paragraphs beneath it, a table whose column names sit in a different piece than its rows — all the same failure. Written text is full of references pointing backwards, and every cut you make risks severing one.

What the geometry cannot see

I do not want to oversell any of this, because it has one limit that no amount of tuning removes.

One word that changes everything, and almost nothing

The same word-counting score as everywhere above, run on four pairs. This one deliberately keeps the word “not” instead of discarding it as filler — and it still barely matters.

The library is open on Friday.
0.866
The library is not open on Friday.
opposite instructions
Freshmen may park in the north lot.
0.913
Freshmen may not park in the north lot.
opposite rules
Club rosters are due in September.
0.750
Club rosters are due in October.
different deadline
The library is open on Friday.
0.000
The shuttle leaves from the main hallway.
genuinely unrelated

The three pairs that say opposite things score near the top. Only the genuinely unrelated pair scores zero. The angle can tell you two sentences are about the same subject. It cannot tell you they disagree.

Adding not to a sentence reverses its meaning completely and moves the arrow almost not at all — because one word out of a dozen is one box out of a dozen, and the other eleven still agree. Changing September to October flips which answer is correct and barely registers either.

Trained embeddings handle this somewhat better than raw counting does, but only somewhat. Negation is a well-known weak spot, and the reason is structural rather than something a better model fixes. All of these measures were built to find text on the same subject. Two sentences that flatly contradict each other are, unavoidably, on the same subject.

Which leads to the thing worth remembering if you remember nothing else here. A retrieval system finds you passages about your question. It does not find you passages that are true, or current, or the one that applies to you. Last year’s handbook and this year’s handbook point in nearly identical directions. Deciding between them has to happen somewhere else — in what you choose to put in the index and what you throw out — because the geometry will never tell you which one is stale.

What we changed

The fixes that mattered were unglamorous, and not one of them involved a better model.

We stopped cutting on word counts and started cutting on structure, so a piece is a section or a single bulleted entry rather than the next thirty words — which mostly means the cuts land where a person would have put them. We kept an overlap between neighboring pieces, not because it rescues the case above (widening the piece is what does that) but as cheap insurance against a cut landing somewhere unlucky. We retrieve several pieces instead of one, since the top result being right most of the time is not the same as being right. We kept a plain keyword search running alongside the embedding search and merged the two rankings, because exact matches on names, room numbers, and dates are the one thing simple counting is genuinely better at. And when nothing scores above a floor, Waypoint now says it does not know, instead of handing the model a bad paragraph and letting it improvise.

What I would tell myself a year ago is that the interesting part of this system was never the language model. It was the question of what “similar” can possibly mean to something that does not know what any of the words mean — and how far you can get on an answer as small as point them in the same direction and measure the angle. Remarkably far, it turns out. The places where it fails are worth knowing precisely, because the failures do not look like failures. They look like clear, confident, wrong sentences.