Tutorial
Title: The Life of a Token: From Words to Bits on the Wire
Description: Large language models (LLMs) transform vast collections of unstructured text into learned representations that support language generation and reasoning. These capabilities lie in a concrete process: words become tokens, tokens become vectors, and training moves data as streams of bits between interconnected accelerators. As models scale, understanding this journey becomes essential for characterizing the communication demands of LLM training and the network infrastructure needed to support it.
This tutorial follows the life of a token from raw text to the data exchanged during distributed LLM training. We first introduce tokenization and embeddings, trace tensor transformations through Transformer blocks, and examine the forward and backward passes on single-GPU training. Building on this foundation, we explore how scaling through data, pipeline, and tensor parallelism determines which tensors are exchanged between which accelerators, and how often. We then introduce logical payload sizes and show how collective operations such as AllReduce turn these payloads into network traffic.
Short, guided exercises accompany the tutorial, using a common example to introduce participants to tensor shapes, data sizes, and the exchanges involved in distributed training. Participants also examine how collective algorithms turn these exchanges into network traffic. The discussion links this traffic to practical considerations such as bandwidth, latency, interconnect topology, and GPU placement.
Davide Avesani
(CNAM, France )
Bio: Davide Avesani is a PhD researcher at the Conservatoire national des arts et métiers (CNAM) in Paris, working on Network Acceleration for Generative AI. His research within the Net4AI project focuses on optimizing networks to support GenAI workloads. His recent work on “The Life of a Token: From Words to Bits on the Wire” explains how text becomes network traffic, helping readers understand the network requirements of distributed LLM training. He holds an MSc in Computer Science and Engineering from Sapienza University of Rome, undertaken in collaboration with ISEP Paris, where he also developed his master’s thesis on LLM-based hate speech detection.
Sotiris Skaperas
(CNAM, France )
Bio: Sotiris Skaperas is an Associate Professor of Computer Networks and Systems at the Conservatoire national des arts et métiers (CNAM), France. He previously worked as a postdoctoral researcher at ENSEA and the Athena Research Center. He holds a Ph.D. in Informatics from the University of Macedonia, Greece, as well as M.Sc. and B.Sc. degrees in Mathematics from Aristotle University of Thessaloniki, Greece. His research focuses on 6G wireless and edge–cloud systems, with an emphasis on statistical modeling and machine learning for anomaly detection, adaptive resource management, and security.