All news

AI ·

A Ten-Year-Old Case Study on AI 'Black Box' Is Overturned by Original Researcher

Nick Cammarata, an early mechanistic interpretability scientist at OpenAI and co-author of the classic Zoom In paper, has reversed his decade-old conclusion about a famous InceptionV1 neuron, arguing it detects V-shaped geometry rather than cats or cars.

A stylized diagram of a neural network node with a V-shape and a circle on opposite sides, suggesting positive and negative responses to different geometric patterns.
A stylized diagram of a neural network node with a V-shape and a circle on opposite sides, suggesting positive and negative responses to different geometric patterns.

For ten years, researchers in AI interpretability have used one iconic example to argue that neural networks are fundamentally opaque: a neuron in the InceptionV1 model, numbered 4e:55, that appeared to fire strongly for both cat faces and car fronts. This puzzling activation pattern was widely seen as evidence that the internal logic of deep learning models could not be understood, pushing the field toward 'disentangling' tools like sparse autoencoders, feature dictionaries, and feature disentanglement methods.

Nick Cammarata, an early mechanistic interpretability scientist at OpenAI and co-author of the paper Zoom In that popularized the case, now says that conclusion was wrong. According to Cammarata, the neuron does not recognize cats or cars at all. Instead, it detects a V-shaped geometric form, with positive activation triggered by things like a fox's pointed face or a jet of flame, while its negative pole responds to circular shapes.

Cammarata argues that the original misjudgment likely came from limited observation tools available at the time. Rather than proving that neural networks are hopelessly entangled black boxes, the corrected interpretation suggests that many neurons may have simpler, human-understandable roles after all.

Source: 雷峰网

Written with AI from the linked source