Tagging and parsing English sentences

In the last few posts we turned raw text into clean tokens, then into bags of words and TF-IDF vectors. Every one of those representations threw away the order of the words. This post puts the structure back. We load the Universal Dependencies English EWT treebank, train a Conditional Random Field part-of-speech tagger that reaches 91.4 percent accuracy on dev against a unigram baseline of 86.1 percent, and then walk through the other layers of linguistic structure: named entities, dependency trees, constituency trees, coreference, and topics. The through-line is the tree. A sentence is not a bag of words, it is a small tree, and each task in this post either builds one, reads one, or flattens one back into a label.

The dataset is the standard English treebank, about 254,000 labeled words spread across train, dev, and test files in CoNLL-U, a token-per-line plain-text format with ten tab-separated columns. Each token carries a gold part-of-speech tag, a lemma, a head word, and a dependency relation, which is exactly what we need to train and evaluate taggers and parsers. The raw sentences also give us free text for the unsupervised topic model at the end. We load it, clean it, and look at what we have before training anything.

What the treebank holds

The CoNLL-U format is one token per line, ten tab-separated columns, blank lines between sentences. Missing values are encoded as an underscore. That underscore matters more than it looks. When we count them, misc is empty for 160,600 tokens, feats for 66,592, and lemma for 2,750. Critically, the universal part-of-speech column (upos) is missing for 2,614 tokens and the head and dependency-relation columns for 2,651. We drop those rows and treat every underscore as missing, not as a literal feature. That leaves 204,615 training tokens and 25,152 dev tokens.

The tag distribution is the first thing to understand, because it sets the floor for everything that follows.

Nouns are 16.98 percent of training tokens, but the all-NOUN tagger scores 16.73 percent on dev, so the distribution sets a low floor. The rest of the distribution is a long tail of verbs, punctuation, determiners, and adjectives, each far smaller than nouns. A tagger that only knows the global most-frequent tag is nearly useless, so the real question is how much context buys us.

We also check for leakage, meaning the same sentence appearing in train and dev, before trusting any split. There are 810 duplicate sentences inside train and 36 sentences that appear in both train and dev. We drop the duplicates and the overlapping dev sentences, which leaves 199,235 training tokens and 25,015 dev tokens with zero sentence overlap, confirmed by an assertion. The text is Zipfian, meaning a few words are very common and most are rare.

The straight line on log-log axes is the signature of a power law: a handful of words like "the" and "of" appear tens of thousands of times, while most of the 16,654-word training vocabulary appears once or twice. Sentences average 16.98 tokens with a median of 15 and a maximum of 159. The long tail is why character-level features matter for rare words. We cap CRF training at 2,000 sentences for runtime.

Tagging with a baseline first

Part-of-speech tagging assigns a grammatical class to every token. Before reaching for a sequence model, we build the two baselines the treebank deserves. The naive one tags every word with the global most-frequent tag, NOUN, and scores 16.73 percent on dev. The unigram tagger is smarter: it learns the most frequent tag for each individual word and backs off to NOUN for words it has never seen. That jumps to 86.06 percent. Memorizing word-tag pairs is most of the game, which tells us the remaining headroom is small and comes from context.

The CRF earns its keep

A Conditional Random Field models the entire tag sequence at once. Instead of scoring each token in isolation, it learns transition weights between tags, so it knows that a determiner is likely to be followed by a noun. We give each token a feature dictionary built from the word itself, its lowercased form, its first and last three characters, and whether it is capitalized or numeric, plus the same features for the previous and next word. The feature function below turns one token and its neighbors into the CRF's view of that position. Then we fit the CRF on a reproducible 2,000-sentence subsample.