一句话结论：listen_fd 这个整数从来不变，变的是它背后那个监听 socket 的 accept 队列：三次握手由内核独立完成，握手成功的连接排进队列，队列非空，监听 socket 就「可读」。epoll 把「登记 fd」和「等待事件」拆开，fd 就绪时内核通过回调把它挂到就绪链表，epoll_wait 只把就绪的那几个交给你；select 每次都要把整个集合拷进内核、两边各扫一遍。所以 for (i = 0; i < n; i++) 里的 n 是就绪数，不是连接总数。
这篇是我自己的复习笔记，围绕一份 epoll 回显服务器（下文称 demo.c）讲清两个问题：监听 fd 可读到底意味着什么，以及 epoll 和 select / poll 的区别到底在哪。事实以 Linux man pages（epoll(7)、select(2)、poll(2)、listen(2)）和内核源码为准；文中的命令输出来自 WSL2 Ubuntu 24.04（内核 5.10）上的真实运行。TCP 握手本身的序号细节见另一篇《从三次握手到 TLS 1.3》。
先用一家餐厅把全貌讲一遍
把服务器想成一家餐厅：
这家餐厅运转起来是这样：前台在门口接待新客人（内核做握手），客人在门口排好队；服务员不用挨桌去问「要不要点菜」，而是谁按铃就去谁那里（epoll 的就绪通知）。服务员每次回到厨房（epoll_wait 返回），手里拿着的是「这几桌按了铃」的单子：如果是前台按的铃，就去领位（accept）；如果是某张桌子按的，就去上菜（read / write）。
select 的服务员则是另一种工作方式：每一轮都把所有桌号抄一遍交给经理，经理挨桌问一遍，再把单子还回来，服务员自己还要从头到尾再看一遍是谁举了手。
**类比到此为止。**下面讲技术细节时都回到真实概念：fd、socket、队列、回调。餐厅里没有「内核态和用户态之间的拷贝」，而这恰恰是 select 和 epoll 差别最大的地方。
整个程序：一个循环，两个分支
demo.c 的结构可以压成一句话：先建好监听 socket 和 epoll，然后永远循环 epoll_wait，返回的每个就绪 fd 只有两种可能——是监听 fd 就 accept，是连接 fd 就 read。
- socket() 创建 TCP socket，得到 listen_fd（通常是 3）。
- bind() 绑定 0.0.0.0:8888。
- listen(listen_fd, 128)：socket 变成监听状态，内核开始替它完成三次握手。
- epoll_create1(0) 创建 epoll 实例，得到 epfd（通常是 4）。
- epoll_ctl(ADD, listen_fd)：把监听 fd 登记进 epoll，关心 EPOLLIN。
- 循环 epoll_wait()：阻塞到至少一个 fd 就绪，返回就绪数 n；对每个就绪事件取出 fd = events[i].data.fd。
- 分支一，fd == listen_fd：有新连接 → accept() 得到 conn_fd → epoll_ctl(ADD, conn_fd) 也登记进 epoll。
- 分支二，其他 fd：某个连接有数据 → read()；返回值大于 0 就 write() 原样写回，返回 0（对方关闭）或小于 0（出错）就 close(fd)。
- 处理完 n 个事件，回到 epoll_wait()。
▶ 本节有交互演示「demo.c 的总流程」，可在原文中动手操作：打开交互版
对应到代码，循环体就是下面这一段（摘自 demo.c，省略了打印和错误处理）：
for (;;) {
int n = epoll_wait(epfd, events, MAX_EVENTS, -1);
for (int i = 0; i < n; i++) {
int fd = events[i].data.fd;
if (fd == listen_fd) {
int conn_fd = accept(listen_fd, (struct sockaddr *)&cli, &len);
ev.events = EPOLLIN;
ev.data.fd = conn_fd;
epoll_ctl(epfd, EPOLL_CTL_ADD, conn_fd, &ev);
} else {
ssize_t r = read(fd, buf, sizeof(buf));
if (r <= 0) { close(fd); continue; }
write(fd, buf, r); // 回显
}
}
}
两个问题都藏在这段代码里：为什么 listen_fd 会出现在 events 里（下一节），以及外层这个 for 和 select 的循环有什么不同（再下一节）。
listen_fd 为什么「可读」就说明有新连接
fd 只是一个下标
listen_fd = 3 是进程 fd 表里的下标，它指向内核里的一个 socket 对象。整数本身没有状态，状态都在它指向的对象上。运行中的 demo 可以直接看到这张表：
$ ls -l /proc/$(pidof demo)/fd | awk '{print $9, $10, $11}'
0 -> /dev/pts/0
1 -> /dev/pts/0
2 -> /dev/pts/0
3 -> socket:[19642]
4 -> anon_inode:[eventpoll]
5 -> socket:[19643]
0、1、2 是标准输入输出，3 是监听 socket，4 是 epoll 实例，5 是第一个 accept 出来的连接。方括号里是 socket 的 inode 号：3 和 5 是两个不同的 socket 对象。这也是第一个连接的 conn_fd 通常是 5 的原因——Linux 总是分配当前最小的空闲 fd。
listen 之后，握手由内核独立完成
listen(listen_fd, 128) 把 socket 变成监听状态。从这一刻起，内核会替这个 socket 处理所有到达 8888 端口的 SYN，而且整个握手过程不需要用户程序执行任何一行代码：
- 客户端的 SYN 到达，内核记下这个「半连接」（概念上叫 SYN 队列），回 SYN+ACK；
- 客户端回 ACK，握手完成，内核为它创建一个新的 socket，放进监听 socket 的全连接队列（accept 队列）；
- accept() 只做一件事：从 accept 队列头部取走一个已经建好的连接，给它分配一个新 fd。
listen 的第二个参数 128 限制的是 accept 队列的长度，并且会被 /proc/sys/net/core/somaxconn 截断（Linux 5.4 起默认 4096，更早是 128）。半连接的数量由另外的参数管：tcp_max_syn_backlog，开启 syncookies 时没有逻辑上限——这一段 listen(2) 的手册写得很清楚。
「可读」对监听 socket 的定义
对普通连接 socket，可读的意思是「接收缓冲区里有数据」；对监听 socket，内核把可读定义成 accept 队列非空（内核源码 inet_csk_listen_poll()：队列不空就返回 EPOLLIN）。所以 demo.c 里那行注释是准确的：
ev.events = EPOLLIN; // 关心「可读」事件（对监听 fd 来说 = 有新连接）
握手完成、新连接入队的那一刻，内核还会唤醒在这个 socket 上等待的人——epoll 就是在这里收到通知的（下一节细讲回调）。下面把客户端、内核、用户程序三个区域放在一起，一步步走一遍：
客户端（nc）、内核、用户程序三方的时间线：
- 程序执行 socket → bind → listen：内核里出现一个监听 socket（fd 3），它带着两个队列：SYN 队列和 accept 队列，accept 队列上限 128。
- 程序执行 epoll_create1（fd 4）和 epoll_ctl(ADD, 3)，然后阻塞在 epoll_wait。
- 客户端发 SYN：连接进入 SYN 队列。用户程序仍阻塞在 epoll_wait，一行代码都没执行。
- 内核回 SYN+ACK，客户端回 ACK：连接从 SYN 队列移到 accept 队列。客户端的 connect() 已经返回，nc 认为自己连上了。用户程序还是一行代码都没执行。
- accept 队列从空变成非空：监听 socket 变成可读。listen_fd = 3 这个数字没变，变的是它背后 socket 的状态。
- epoll 的回调把 fd 3 挂上就绪链表，epoll_wait 返回 1，events[0].data.fd == 3。
- 程序调用 accept()：从 accept 队列取走这个连接，得到新的 conn_fd = 5。队列又空了，监听 socket 不再可读。
- 实验：如果程序一直不 accept，第二、第三个客户端照样能握手成功，只是排在 accept 队列里。
- 队列满了以后，新来的 SYN 会被丢弃，客户端只能重传 SYN；具体行为取决于内核参数（见下文）。
▶ 本节有交互演示「三次握手与 accept 队列」，可在原文中动手操作：打开交互版
listen_fd 这个数字从头到尾都是 3，变的是它背后那条 accept 队列是空还是不空。
accept 队列满了会怎样：取决于内核参数
这是容易被问、也容易说错的地方。以 Linux 5.x 的实现为准，分两种情况：
- 队列满时又来一个 SYN：内核直接丢掉这个 SYN，不回任何东西，计数器 ListenOverflows / ListenDrops 加一。客户端收不到 SYN+ACK，会按指数退避重传 SYN，表现为 connect() 卡住，最后可能超时。
- 握手的最后一个 ACK 到达时队列已满：默认（net.ipv4.tcp_abort_on_overflow = 0）内核丢掉这个 ACK，连接留在半连接状态，稍后重传 SYN+ACK，如果到时候队列有空位就能「自愈」；设为 1 时，内核直接回 RST，客户端会看到 Connection reset by peer。
不同内核版本在细节上有差别（比如队列的判满条件），所以描述这一点时说「取决于 tcp_abort_on_overflow，默认是丢弃而不是 RST」就够了，不要说死。下文「动手验证」里有一个把 backlog 改成 1 的实验，可以亲眼看到 SYN 被丢。
epoll_wait 之后还要写 for 循环，和 select 有什么区别
区别不在「有没有循环」，而在循环的对象是谁、循环之前内核做了多少事。先把三种机制各自在一次调用里做了什么摆出来。
select：每次都交出整张表，再拿回整张表
select(maxfd + 1, &readfds, NULL, NULL, NULL);
fd_set 是一个位图，每个 bit 代表一个 fd。每次调用：
- 把整个位图从用户态拷贝进内核；
- 内核从 0 扫到 maxfd，对每个置位的 fd 调一次它的 poll 函数，问「你现在可读吗」；都没就绪，就把自己挂到每个 fd 的等待队列上睡眠；
- 任何一个 fd 就绪都会把它唤醒，醒来后内核再扫一遍所有 fd，把没就绪的 bit 清零；
- 把位图拷回用户态，返回就绪数；
- 用户程序只知道「有几个」，不知道「是哪几个」，必须再用 FD_ISSET 把每个 fd 检查一遍；
- 位图被内核改写了，下一轮调用前要用 FD_ZERO / FD_SET 重新设置。
另外，fd_set 的大小固定为 FD_SETSIZE（1024），编号大于等于 1024 的 fd 不能用 select 监视。
poll：换了数据结构，没换工作方式
poll 用 struct pollfd 数组代替位图：events 写你关心什么，内核把结果写到另一个字段 revents。所以 poll 解决了两个问题——没有 1024 的上限，也不用每轮重建集合。但每次调用依然要把整个数组拷进内核、内核依然要逐个检查、用户依然要逐个看 revents，开销仍然和监视的 fd 总数成正比。
epoll：登记一次，就绪时回调
epoll 把 select 一个调用里的事拆成三个调用：
- epoll_create1：在内核里建一个 epoll 实例，里面有两样东西——兴趣列表（你登记了哪些 fd，内核用红黑树存）和就绪链表（哪些 fd 已经就绪）。
- epoll_ctl：登记、修改、删除一个 fd。登记时内核把一个回调函数（ep_poll_callback）挂到这个 socket 的等待队列上。只在登记时做一次，之后每次等待都不用再交出集合。
- epoll_wait：只看就绪链表。链表空就睡，非空就把就绪的事件拷给用户，返回个数。
socket 收到数据时，网络协议栈唤醒它的等待队列，回调函数就把对应的条目挂到就绪链表尾部。于是 epoll_wait 返回时，events 数组里只有就绪的 fd。
场景：已连接 10 个 fd（fd 5～14），其中只有 fd 7 和 fd 12 收到数据。
一轮下来，按「逐个检查一个 fd」计数：select 内核检查 20 次、用户检查 10 次；epoll 内核只确认就绪链表上的 2 个，用户循环 2 次。
▶ 本节有交互演示「select 与 epoll：10 个连接，2 个有数据」，可在原文中动手操作：打开交互版
那个 for 和 if 到底在做什么
回到 demo.c：
int n = epoll_wait(epfd, events, MAX_EVENTS, -1);
for (int i = 0; i < n; i++) {
int fd = events[i].data.fd;
if (fd == listen_fd) {
- n 是这一次就绪的事件数，不是连接总数。1 万个连接里只有 3 个有数据，n 就是 3，循环跑 3 次。select 的用户循环则要从 0 扫到 maxfd。
- if (fd == listen_fd) 不是在「搜索」哪个 fd 就绪了——events 里的每一项都已经是就绪的。它只是在已就绪的 fd 里分类型：这个是监听 socket（去 accept），那个是连接 socket（去 read）。select 版本里同样需要这个判断，而且是在 FD_ISSET 找到之后。
所以两者的差别是：select 的循环长度是「监视了多少个 fd」，epoll 的循环长度是「就绪了多少个 fd」；select 每次调用都要把整个集合在用户态和内核态之间来回拷贝、由内核扫一遍，epoll 只在 epoll_ctl 时付一次登记成本。
epoll 不是在所有情况下都更快
- ✔ 连接多、活跃少时优势最大 等待的开销跟就绪数走，而不是跟连接总数走，上万个空闲长连接的服务器最受益。
- ✘ 连接全都很活跃时优势变小 每次大部分 fd 都就绪，epoll 也要把它们逐个交给你，和 select / poll 的线性扫描差不了多少。
- ✘ 登记本身有成本 每个 epoll_ctl 都是一次系统调用，大量短连接频繁增删时这部分开销会显现出来。
- ✘ fd 很少时没有明显差别 只盯几个 fd 的程序（很多嵌入式应用就是这样），select / poll 足够用，而且可移植。
- 只是「等」的部分更快 epoll 不会让 read / write 本身变快，业务处理慢的瓶颈它解决不了。
ev.data.fd：你自己贴的标签，内核原样还给你
epoll_event 的定义（来自 epoll_ctl(2)）是这样的：
typedef union epoll_data {
void *ptr;
int fd;
uint32_t u32;
uint64_t u64;
} epoll_data_t;
struct epoll_event {
uint32_t events; /* Epoll events */
epoll_data_t data; /* User data variable */
};
data 是一个联合体，内核不解释它的内容，只负责存下来、在事件发生时原样还给你。真正决定「监视哪个 fd」的是 epoll_ctl 的第三个参数，不是 data.fd：
ev.events = EPOLLIN; // 关心「可读」事件（对监听 fd 来说 = 有新连接）
ev.data.fd = listen_fd; // 事件发生时，靠这个知道是哪个 fd
if (epoll_ctl(epfd, EPOLL_CTL_ADD, listen_fd, &ev) < 0) die("epoll_ctl listen");
demo.c 把 fd 本身当标签，所以 events[i].data.fd 正好就是那个 fd——这是约定，不是 epoll 的规定。更复杂的服务器常常在 data.ptr 里放一个指向连接结构体的指针，事件来了直接拿到这个连接的缓冲区和状态，省掉一次查表。
标签的一次往返：
- 用户填好 ev：events = EPOLLIN，data.fd = 3。
- epoll_ctl(epfd, EPOLL_CTL_ADD, 3, &ev)：内核按第三个参数 3 找到要监视的 socket，同时把整个 ev（包括 data）拷贝一份存进兴趣列表里的条目。
- fd 3 背后的 socket 就绪：这个条目被挂上就绪链表。
- epoll_wait 把条目里存着的 events 和 data 原样拷进用户的 events[0]。
- 用户读到 events[0].data.fd == 3——就是第 1 步自己写进去的那个值。
如果第 1 步写的是 data.u64 = 0xBEEF 或者一个指针，第 5 步拿回来的就是 0xBEEF 或那个指针；如果故意写成 data.fd = 99，epoll 也照样还给你 99，它不检查。
▶ 本节有交互演示「ev.data 这张标签怎么来回走」，可在原文中动手操作：打开交互版
这也解释了 demo.c 里两个容易让人犯嘀咕的地方：
- 同一个 ev 变量被反复使用：先给监听 fd 用，后面每 accept 一个连接又改写一次。这是安全的，因为 epoll_ctl 在调用时就把内容拷进了内核，之后你改用户态的 ev 不会影响已经登记的条目。
- 「close 之后 epoll 会自动把这个 fd 移除」：在 demo.c 里成立，但有前提。epoll 跟踪的其实是 fd 背后的「打开文件」，只有当指向它的所有 fd 都关闭了（没有被 dup 过、没有 fork 出去的子进程还拿着）才会自动移除——这是 epoll(7) 问答部分专门提醒的一条。稳妥的写法是先 epoll_ctl(EPOLL_CTL_DEL) 再 close。
水平触发和边缘触发：同一份数据，通知几次
epoll 给每个 fd 提供两种通知方式，由登记时 ev.events 里有没有 EPOLLET 决定：
- 水平触发（LT，默认）：只要 fd 处于就绪状态，比如接收缓冲区里还有数据，每次 epoll_wait 都会报告它。demo.c 用的就是 LT。
- 边缘触发（ET）：只在状态发生变化时报告。对「可读」来说，就是有新数据到达的那一刻；之后缓冲区里就算还剩数据，只要没有新数据进来，epoll_wait 就不会再报告这个 fd。
下面是实测（WSL2，内核 5.10）：对端一次发来 3000 字节，服务端每次被唤醒只 read 1024 字节。
ET 只读一次的那一行里，剩下的 1976 字节并没有丢，只是不会再触发通知。实测中对端随后再发 2 字节，epoll_wait 又返回了 1 次：新数据到达是一次新的「边缘」。如果对端正在等你的回复、不再发东西，这条连接就会一直卡着。
▶ 本节有交互演示「LT 与 ET：3000 字节，每次读 1024」，可在原文中动手操作：打开交互版
ET 必须配非阻塞 fd，读到 EAGAIN 为止
ET 只通知一次，所以收到通知后必须把数据读空。读空的标志是 read 返回 -1 且 errno == EAGAIN（或 EWOULDBLOCK），这要求 fd 是非阻塞的。accept 出来的 fd 默认是阻塞的：如果不改，读空之后的最后那次 read 会一直阻塞，整个事件循环都卡在这一个连接上。
如果把 demo.c 改成 ET，accept 之后要先把 fd 设成非阻塞，登记时加上 EPOLLET，5b 分支改成循环读。下面这段是示意，不是 demo.c 原文：
// accept 之后：
fcntl(conn_fd, F_SETFL, fcntl(conn_fd, F_GETFL) | O_NONBLOCK);
ev.events = EPOLLIN | EPOLLET;
// 5b 分支：
for (;;) {
ssize_t r = read(fd, buf, sizeof(buf));
if (r > 0) { write(fd, buf, r); continue; } // 读到数据，接着读
if (r == 0) { close(fd); break; } // 对方关闭
if (errno == EAGAIN || errno == EWOULDBLOCK) break; // 读空了，等下一次通知
if (errno == EINTR) continue; // 被信号打断，重试
close(fd); break; // 其他错误
}
监听 fd 也一样：ET 下几个连接同时到达只会通知一次，accept 也要循环调用到返回 EAGAIN，否则剩下的连接会一直排在 accept 队列里。
什么时候用哪个
- LT：默认值，容易写对。「每次通知只读一次」这种写法只在 LT 下正确，漏读了下一轮还会被提醒。不确定时用 LT。Redis 的事件库用的就是 LT。
- ET：同一份数据只通知一次，epoll_wait 的返回次数更少，代价是必须用非阻塞 fd、每次都读写到 EAGAIN，每个连接的状态要管得更严。Nginx 用的是 ET。
- 写事件的差别更明显：LT 下只要发送缓冲区还有空间，EPOLLOUT 每次都会报告；ET 下只在发送缓冲区从满变成不满时报告一次。下一节会讲 EPOLLOUT 该怎么用。
read 和 write：每种返回值意味着什么
read
两点要注意：
- **TCP 是字节流，没有消息边界。**客户端一次发的一行字，服务端可能分两次读到，也可能和下一行一起读到。回显服务器不在乎，但只要有自己的协议（比如按行、按长度头分包），就得在应用层自己拼包。
- demo.c 用 r <= 0 把 0 和 -1 一起处理、一律关闭。在阻塞 fd + LT 下基本够用：epoll 报告可读之后，那一次 read 不会返回 EAGAIN。一旦换成非阻塞 fd 或 ET，就必须把 EAGAIN 单独拿出来，否则会把正常的连接误关掉。
write
实测两个细节：
- 对方关闭之后，第一次 write 仍然返回 1（数据被拷进了发送缓冲区，对方回了 RST），第二次才返回 -1 / EPIPE。如果没有忽略 SIGPIPE，第二次 write 时进程就直接退出了。demo.c 没有处理 SIGPIPE。
- 让对方一直不读，在本机回环上用非阻塞 fd 一直写，写进约 2.5 MB 后返回 EAGAIN（自己的发送缓冲区和对方的接收缓冲区都满了，具体大小取决于内核的缓冲区自动调节）。demo.c 的 fd 是阻塞的，这时 write 会一直卡住，整个事件循环陪它一起卡住：一个不读数据的客户端，就能让所有客户端都失去响应。demo.c 也没有检查 write 的返回值。
EPOLLOUT 只在需要时登记
实测：在 LT 下登记 EPOLLOUT、什么都不写，连续 5 次 epoll_wait 都立即报告可写。发送缓冲区几乎总是有空间，所以一直挂着 EPOLLOUT 会让事件循环空转。正确的用法是：
- 平时只关注 EPOLLIN；
- write 没写完（部分写入或 EAGAIN）时，把剩余数据存进这个连接自己的发送缓冲，用 EPOLL_CTL_MOD 加上 EPOLLOUT；
- 等 EPOLLOUT 到来，把剩余数据发完，再用 EPOLL_CTL_MOD 把 EPOLLOUT 去掉。
断开时 epoll 报告什么
EPOLLHUP 和 EPOLLERR 不用登记，epoll 总会报告；EPOLLRDHUP 需要自己在 ev.events 里加上。demo.c 只登记了 EPOLLIN，靠 read 的返回值判断断开，也是可以的。
动手验证：两个终端，亲眼看内核做握手
1. 跑起来
终端 A 编译并运行：
gcc demo.c -o demo
./demo
终端 B 连上去，输入什么就会回什么：
nc 127.0.0.1 8888
终端 A 会打印 new client 127.0.0.1:xxxxx (fd=5)。再开一个终端连第二个，就是 fd=6。
2. 看监听队列：ss -lnt
ss -lnt 'sport = :8888'
State Recv-Q Send-Q Local Address:Port Peer Address:Port Process
LISTEN 0 128 0.0.0.0:8888 0.0.0.0:*
对 LISTEN 状态的 socket，这两列的含义和普通连接不一样：
- Recv-Q：accept 队列里当前排着多少个已完成握手、还没被 accept 的连接；
- Send-Q：accept 队列的上限，也就是 min(backlog, somaxconn)，这里是 128。
demo 正常运行时 accept 很及时，Recv-Q 基本一直是 0。
3. 不 accept，nc 依然能连上
把 5a 分支改成「只打印、不 accept」：在 if (fd == listen_fd) { 下面加一行，原来的代码不用删，只是执行不到了：
if (fd == listen_fd) {
puts("listen_fd 可读，但我不 accept"); fflush(stdout); sleep(1); continue;
// ── 5a：新连接 → accept，拿到一个专门和这个客户端说话的新 fd
重新编译运行，然后开三个终端分别 nc 127.0.0.1 8888，每个都输入一行字（比如 msg1）。能观察到三件事。
第一，三个 nc 都没有报错，客户端认为自己已经连上了。握手是内核做的，跟程序有没有 accept 无关。
第二，监听 socket 的 Recv-Q 变成 3：三个连接都排在 accept 队列里。
State Recv-Q Send-Q Local Address:Port Peer Address:Port Process
LISTEN 3 128 0.0.0.0:8888 0.0.0.0:*
第三，连客户端发来的数据都已经被内核收下了。看服务端这一侧的三个连接，每个的 Recv-Q 都是 5（msg1 加换行正好 5 个字节）——数据躺在内核的接收缓冲区里，等着一个永远不来的 read：
ss -tn state established '( sport = :8888 )'
Recv-Q Send-Q Local Address:Port Peer Address:Port Process
5 0 127.0.0.1:8888 127.0.0.1:34192
5 0 127.0.0.1:8888 127.0.0.1:34190
5 0 127.0.0.1:8888 127.0.0.1:34188
终端 A 还会每秒打印一次 listen_fd 可读，但我不 accept：队列一直不空，监听 socket 一直可读，epoll 默认的水平触发就会每次 epoll_wait 都报告它。这就是那行 sleep(1) 的用途——去掉它，程序会在 epoll_wait 和 continue 之间空转，把一个 CPU 核跑满。
4. （可选）把队列撑满
再把 listen(listen_fd, 128) 临时改成 listen(listen_fd, 1)，保持
本文由 AI图鉴自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
One sentence concludes: the integer never changes, and the integer behind it is the accept queue: three handshakes are done independently by the kernel, a successful handshake lined up, a line is not empty, a listening socket is "readable." "Registration fd" and "waiting events" are removed, and the kernels are ready to go back to the table, and the holll wait only gives you the few that are ready; each time, select is to copy the whole collection into the core and sweep both sides. So the n in for (i = 0; i < n; i ++) is a ready number, not a total number of connections.
This is my own review note, which talks about two questions: what listening fd can mean, and what the difference between epol and self/poll is. The facts are based on Linux man pages (epoll(7), self(2), Poll(2), listen(2)) and the kernel code; the command output in the text is derived from the true operation on WSL2 Ubuntu 24.04 (kernel 5.10). For details of the TCP handshake itself, see another edition, From Three Handshakes to TLS 1.3.
Let's use a restaurant to tell the whole story.
Think of the server as a restaurant:
The restaurant operates as follows: the front desk hosts new guests at the door (the kernel shakes hands) and the guests line up at the door; the waiter does not go to the table to ask "Do you want to order" and whoever rings the bell goes to anyone's place. Each time the waiter returns to the kitchen (epoll wait), he has a list of "the tables ring the bells": if the bell is rang from the front desk, go to the position (accept); if it is a table press, go to the kitchen (read /write).
Self-servant is another way of working: on each round, copy all the table numbers to the manager, the manager goes to the table and returns the list, and the waiter himself looks at who raised his hand from the beginning to the end.
** End of analogy. ** The following technical details return to the real concept: fd, socket, queue, echo. The restaurant does not have a copy of the kernel between the kernel and the user, which is the place where the difference between self and power is greatest.
The whole program: one cycle, two branches
The structure of demo.c can be reduced to one sentence: first set up listening socket and epol, and then circulate forever soll wait, every ready fd returned, only two possibilities -- fd listening, fd connecting, read.
- Socket() Create TCP socket, get listn fd (usually 3).
- bind() binding 0.0.0.0:888.
-listen (listen fd, 128): Socket becomes wired and the kernel begins to shake hands three times.
- Poll create1(0) create an example of a Poll with an epfd (usually 4).
- Poll ctl (ADD, listen fd): Register fd into epolll, care about EPOLLIN.
- Looping Poll wait(): Blocking to at least one fd ready and returning to the number of n; extracting fd = events[i].data.fd for each compliance event.
- Branch I, fd = listen fd: A new connection → accept() is also registered in epol.
- Branch II, other fd: a connection has data → read(); returns value greater than 0 write() as it is, returns 0 (closed) or less than 0 (in error).
- Finished n events and returned to poll wait().
▶ This section contains an interactive presentation of the "demo.c master process" that can be used in the original language: open the interactive version
Corresponds to the code, the circle is the following (from demo.c, print and error processing omitted):
for (;;) {
int n = epoll_wait(epfd, events, MAX_EVENTS, -1);
for (int i = 0; i < n; i++) {
int fd = events[i].data.fd;
if (fd == listen_fd) {
int conn_fd = accept(listen_fd, (struct sockaddr *)&cli, &len);
ev.events = EPOLLIN;
ev.data.fd = conn_fd;
epoll_ctl(epfd, EPOLL_CTL_ADD, conn_fd, &ev);
} else {
ssize_t r = read(fd, buf, sizeof(buf));
if (r <= 0) { close(fd); continue; }
write(fd, buf, r); / /backs
}
}
}
Both questions are hidden in this code: why is the list fd going to appear in events (the next section) and how this for the outer layer is different from the circle for self.
listen fd why readable means a new connection
fd is just a subscript
Listen fd = 3 is the subscript in the fd table, which points to a socket object in the kernel. The whole number itself has no state, all on the object it points to. Runn demo can see this table directly:
$ ls -l /proc/$(pidof demo)/fd | awk '{print $9, $10, $11}'
0 -> /dev/pts/0
1 -> /dev/pts/0
2 -> /dev/pts/0
3 -> socket:[19642]
4 -> anon_inode:[eventpoll]
5 -> socket:[19643]
0, 1, 2 is standard input output, 3 is listening socket, 4 is an example of apoll, 5 is the first connection out of accept. The square brackets are the inode of socket: 3 and 5 are two different socket objects. This is also why the first connected conn fd is usually 5 - Linux always allocates the minimum available fd.
After that, the handshake was done independently by the kernel.
listen(listen fd, 128) turns socket into a listening state. From this moment on, the kernel will process all SYNs arriving at the 8888 port for this socket, and the whole handshake does not require a user program to execute any line code:
- Client SYN arrives, kernel notes this semi-connection (the concept called SYN queue), back to SYN+ACK;
- Client-end ACK, handshake complete, kernel creates a new socket for it and puts it in the full connection queue (accept queue);
- Accept() does one thing: removes a built connection from the head of the accept queue and assigns it a new fd.
The second parameter for listen 128 limits the length of the accept queue and is cut off by /proc/sys/net/core/somaxconn (Linux 5.4 default 4096, earlier 128). The number of semi-connected items is covered by a separate parametric tube: tcp max syn backlog, with no logical limit for opening syncookies - this section of the list(2) manual is well written.
Definition of "readable" listening socket
For normal connection socket, readable means "receiving data in the buffer zone"; for listening socket, kernel defines readable as accept queue not empty (kernel source code inet csk listen poll(): queue is returned to EPOLLIN if empty. So demo.c's line is accurate:
ev.events = EPOLLIN;/ / Care about "readable" events (for listening fd = new connection)
The moment the handshake is complete and the new connection is in, the kernel wakes up the person who waited on this socket — the Poll — that's where the notice came from. Below are three areas of client, kernel, user program, one step at a time:
Timeline for client (nc), kernel, user program:
- Program execution socket → bind →listen: One listening socket (fd 3) with two queues: SYN and accept, accept upper limit.
- Program execution: pollell create1(fd 4) and poll ctl (ADD, 3) and then blocked at pollell wait.
- Client sent SYN: Connection into SYN queue. The user program is still blocked at Epolll wait, and no line of code is executed.
- kernel back SYN+ACK, client back ACK: connect from SYN queue to accept queue. Client conect() has returned, nc considers itself connected. The user application is still not coded.
- Accept queue from empty to non-empty: listening socket to readable. The listen fd =3 has not changed, the status of the socket behind it.
- Poll's callback hangs fd 3 on the table and Poll wait returns 1, events [0].data.fd=3.
- Program calls accept(): Removes this connection from the accept queue and gets a new conn fd = 5. The queue is empty again, listening socket is no longer readable.
- Experiment: If the program has not been successful, the second and third client will be able to shake hands, just in line with the accept.
- When the queue is full, the newly arrived SYN will be discarded and the client will be able to retransmit SYN; the behaviour depends on the kernel parameters (see below).
▶ This section has an interactive presentation "Three handshakes and accept queue" that can be used in the original language: Open interactive versions
This number is three from the beginning to the end, and the change is whether the accept line behind it is empty or not.
Accept's queue is full: depending on kernel parameters
This is an easy place to ask and to say wrong. Based on the achievement of Linux 5.x, in two cases:
- Another SYN: The kernel drops this SYN and doesn't return anything. If the client does not receive SYN+ACK, the SYN will be retransmitted on an index basis, in the form of connect() stuck and possibly timeout.
- The last ACK to shake hands is full when it arrives: the default (net.ipv4.tcp abort on overflow=0) core loses the ACK, the connection remains in semi-connected status, and later retransmits SYN+ACK, which can be "healed" if the team has an empty slot; when set to 1, the kernel returns directly to RST, and the client will see Connection Reset by Peer.
Different kernel versions differ in detail (e.g., the queue is full), so describing this point, "depending on the tcp abort on overflow, the default is abandonment rather than RST" is enough, not death. Here's an experiment to change the backlog to one, to see SYN lost.
After that, I'll write for the loop, and what's the difference?
The difference is not "if or not there's a cycle," but who is the object of the cycle and how much the core does before it. First, what each of the three mechanisms has done in a single call.
Self: Give me the whole watch every time and get it back.
select(maxfd + 1, &readfds, NULL, NULL, NULL);
fd set is a bitmap, each bit represents a fd. Every call:
- Copying the entire map from the user state into the core;
- Core sweeps from 0 to maxfd, adjusts its Poll function once for each slot fd, asking "Do you read now "; if not ready, hang yourself on each fd waiting line;
- Any fd ready will wake it up, and then the kernel will sweep all fds again, and the unready bit will be zero;
- Copy the bitmap back to the user state and return to readiness;
- The user program only knows "a few" and "a few" and has to check every fd with FD ISSET;
- The bitmap has been modified by the kernel to be reset with FD ZERO / FD SET before the next rotation.
Furthermore, fd set is fixed to FD SETSIZE (1024) and fd with a number greater than 1024 cannot be monitored with self.
Poll: Changed data structure, no change in working style
Poll uses structpolfd array instead of bitmap: events write what you care about, kernel writes the results to another field returns. So Poll solves two problems - no ceiling of 1024 and no need for a round of reconstruction. However, every call is still made to copy the entire array into the kernel, the kernel is still checked on a case-by-case basis, users are still looking at revents, spending is still proportional to the total number of fds monitored.
Poll: Once registered, back when ready
Poll splits a call from a call into three calls:
- Poll create1: An example of building an poll in the inner core, which contains two items - the list of interests (which fd you registered, which kernels are stored with) and the table of ready chains (which fd are already in place).
- Poll ctl: Register, modify, delete a fd. The kernel at the time of registration hangs a callback function (ep poll callback) on this waiting queue for the socket. Only once at the time of registration and then every waiting period does not have to be handed over again.
- Poll wait: read only the Ready Chain Table. The chain watch sleeps empty and the event is copied to the user and returned to number.
When the data is received, the web protocol stack wakes up its waiting queue, and the callback functions hang the corresponding entry to the end of the ready-link table. So when the power wait returns, only fd is available in the events array.
Scene: 10 fd (fd 5-14), of which only fd 7 and fd 12 received data.
Round down, counted as `one fd on a case-by-case basis': 20 self-checks, 10 user checks, and only 2 on the chain of readiness, 2 user loops.
▶ This section has an interactive presentation on "Setect and Poll:10, 2 with data" that can be used in the original text: open the interactive version
That for and if
Back to demo.c:
int n = epoll_wait(epfd, events, MAX_EVENTS, -1);
for (int i = 0; i < n; i++) {
int fd = events[i].data.fd;
if (fd == listen_fd) {
- n is the number of events that are ready for this, not the total number of connections. Only 3 out of 10,000 connections have data, and n is three, three run cycles. The user loop for select is to sweep from 0 to maxfd.
- If (fd = list fd) is not in "search" which fd is in place - every one of the events is already in place. It is only a sort of fd that is already in place: this is listening socket, the connection socket. This judgement is also needed in the select version, after FD ISSET was found.
So the difference between the two is that the end length of the cycle is "how many fds have been monitored" and the end length of the cycle is "how many fds have been put in place"; each call is made by the end of the cycle, which brings the whole body together between the user state and the internal nucleus, which is scanned by a kernel, and that the endol only pays the registration cost once at the end.
Poll, it's not always faster.
- There's a lot of connections and the most advantage when you're less active, and the costs of waiting to go up and down instead of following the total number of connections, with tens of thousands of free and long servers benefiting most.
Every time most fds are ready, Poll will also give them to you one by one, and the linear scans of self/poll will be much smaller.
- The registration itself has a cost, and each polll ctl is a system call, and this part of the cost will appear when a lot of short connections are increased or deleted.
- Very rarely fd with no apparent difference but a few fd programs (as is the case for many embedded applications), self / pol sufficient and portable.
- It's just that the "waiting" part is faster. Poll won't let the read /write itself get faster, the bottlenecks of slow business processing can't be solved.
ev.data.fd: Your own label, the kernel.
The definition of "poll event" (from "poll ctl(2)") is:
typedef union epoll_data {
void *ptr;
int fd;
uint32_t u32;
uint64_t u64;
} epoll_data_t;
struct epoll_event {
uint32_t events; /* Epoll events */
epoll_data_t data; /* User data variable */
};
Data is a conglomerate whose contents are not explained by the kernel. It is responsible for saving them and returning them to you as they were at the time of the incident. The third parameter that really determines which fd to spy on is poll ctl, not data.fd:
ev.events = EPOLLIN;/ / Care about "readable" events (for listening fd = new connection)
ev.data.fd =listen fd; // / at the time of the incident, which fd
if (epoll_ctl(epfd, EPOLL_CTL_ADD, listen_fd, &ev) < 0) die("epoll_ctl listen");
Demo.c uses fd as a label itself, so it's the fd -- it's an agreement, not a Poll. More complex servers often place a pointer in the data.ptr that points to the connector structure, and the event comes directly to the buffer zone and state of the connection and saves one check.
Label one round trip:
- User fills ev: events = EPOLLIN, data.fd = 3.
- Poll ctl (epfd, EPOLL CTL ADD, 3, &ev): The kernel uses the third parameter3 to find the socket to monitor, while copying the whole ev (including data) into the list of interests.
- fd 3 behind the socket readiness: this entry is hung on the index.
- Poll wait to copy the original entries of the entries and data into the user's original ones [0].
- User reads events [0].data.fd = = 3 — the value that step 1 itself has written in.
If step 1 reads data.u64 = 0xBEEF or a pointer, step 5 brings back 0xBEEF or that pointer; if deliberately written as data.fd = 99, Poll also pays you 99, it does not check.
▶ This section has an interactive presentation on "v.data: How does this label go back and forth" that can be used in the original language: Open the interactive version
This explains the fact that there are two easy places to whisper:
- The same ev variable is used repeatedly: first for listening fd, then for rewriting every connection in the back. It's safe because when you call, you copy the contents into the kernel, and then you change the user's ev without affecting the entries already registered.
- "Poll will automatically remove this fd after close": set up in demo.c, but with a premise. Epolol tracks the "open file" behind the fd, which is only automatically removed if all the fd that points to it is closed (not dup, not fork out subprocessed) -- This is a specific reminder in the question and answer section of Poll(7). It's safe to put it in the form of a poll ctl and then a close.
Horizontal trigger and edge trigger: same data, several notifications
Epoll provides two types of notice for each fd, which is determined at the time of registration by the EPOLLET.
- Horizontal Trigger (LT, default): As long as fd is in a state of readiness, for example, to receive data in the buffer zone, it is reported every time it is done. Demo.c uses the lt.
- Edge trigger (ET): reported only when status changes. For "readable" it's the moment when new data arrives; then even if there is still data in the buffer zone, as long as there is no new data coming in, it won't be reported fd.
The following is a measurement (WSL2, kernel 5.10): The opposite end is sent at once to 30000 byte, and the service end is awakened only to read 1024 byte.
In the line where ET read only once, the rest of 1976 byte didn't drop, just never trigger the notice again. Two bytes were subsequently sent to the end of the field, and a new one was returned: the new data arrived at the edge. If the other end is waiting for your reply and you don't send anything, the connection will be stuck.
▶ This section has an interactive presentation "LT and ET: 3000 byte, 1024 per reading" which can be done in the original language: open the interactive version
ET must be unblocked, fd, read until EAGAIN
ET is notified only once, so data must be emptied after notification. The sign that reads empty is read returns -1 and errno = EAGAIN (or EWOLDBLONK), which requires that fd is not blocked. The default fd from accept is blocked: if not, the last read after reading empty will be blocked, and the entire event cycle is stuck to this connection.
If demo.c is changed to ET, accessept and then fd is set to be unobstructed, the EPOLET 5b branch is added to the register and read in circular. The following is a sign, not a demo.c:
After:
fcntl(conn_fd, F_SETFL, fcntl(conn_fd, F_GETFL) | O_NONBLOCK);
ev.events = EPOLLIN | EPOLLET;
Branch:
for (;;) {
ssize_t r = read(fd, buf, sizeof(buf));
If (r > 0) {write(fd, buf, r); continue;} / / Read data and continue reading
If (r = 0) {close(fd); break;} / / each side closes
If (errno = EAGAIN || errno = EWOLDBLONK) break; / / / read empty until next notification
If (errno = EINTR) continue; / / Interrupted by signal, retry
close(fd); break; // other errors
}
The same is true for listening fd: the next ET connections arrive only once, and accept also recycles calls to return EAGAIN, otherwise the rest of the connections will remain in the accept queue.
When?
LT: Default, easy to write. The phrase "read only once for each notice" is correct only under LT, and the next round will be reminded if it is not read. Use LT when not sure. The Redis event library uses LT.
- ET: The same data are only notified once, and Poll wait returns less frequently at the expense of having to read and write EAGAIN every time it is not blocked. Nginx uses ET.
- The difference between writing incidents is even more evident: whenever there is space available to send the buffer zone under the LT, EPOLLOUT reports every time; ET reports only once when sending the buffer zone from full to disgruntled. Next section will talk about EPOLLOUT's use.
read and write: what each return value means
read
Two points:
- ** TCP is byte, no message border. ** A line of text sent by the client, which may be read by the service either twice or with the next line. Retrospective servers do not care, but as long as they have their own protocols (e.g. line-by-line, head-by-length subcontracts), they have to pack themselves at the application level.
- demo.c Use r< = 0 to process 0 with -1, all closed. Basically enough to block fd + LT: After the Poll report is readable, read will not return EAGAIN that time. Once replaced with a non-blocked fd or ET, the EAGAIN must be taken out alone or the normal connection will be turned off by mistake.
write
Two details were measured:
- After the other side closed, for the first time the write returned 1 (data was being copied into the sending buffer zone, the other party returned RST) and for the second time -1/EPPE. If SIGPIPE is not ignored, the process exits directly at the second time. No, demo.c doesn't handle SIGPIPE.
- To keep the other side from reading, to use non-blocked fd on this loop, to write about 2.5 MB and then return to EAGAIN (both its own sending buffer zone and the other side's receiving buffer zone are full, depending on the size of the inner core's buffer zone automatically regulated). The fd of demo.c is blocked, and the write will be stuck all the time, and the whole cycle of events will be stuck with it: a client who does not read the data will lose the response of all clients. Demo.c does not check the return value for write.
EPOLLOUT only registers when needed
EPOLLOUT is registered under LT, does not write anything, and reports are available five times. There's almost always room for sending buffers, so it's always hanging EPOLLOUT that makes the event go empty. The correct use is:
- Focus only on EPOLLIN;
- Write does not finish (partially written or EAGAIN), saves the remaining data in this sending buffer that connects itself with EPOLL CTL MOD plus EPOLLOUT;
- When the EPOLLOUT arrives, distribute the remaining data, then use EPOLL CTL MOD to remove the EPOLLOUT.
When it's off, Poll, report what?
EPOLLHUP and EPOLLERR do not need to be registered; EPOLLRDHUP needs to be added to the ev.events. Demo.c only registered EPOLLIN, separated by the return value of the read.
Let's do it: two terminals, see the core for yourself. Hands.
1. Run
Terminal A compiles and runs:
gcc demo.c -o demo
./demo
Terminal B, go on, enter whatever you want.
nc 127.0.0.1 8888
Terminal A prints new client 127.0.0.1:xxx (fd=5). One more terminal, the second, is fd=6.
2. Watch the listening queue: ss-int
ss -lnt 'sport = :8888'
State Recv-Q Send-Q Local Address:Port Peer Address:Port Process
LISTEN 0 128 0.0.0.0:8888 0.0.0.0:*
For LISTEN states, the meaning of these two columns is different from that of normal connections:
- Recv-Q: how many handshakes are currently in line in the Recept queue and are not connected to them;
- The upper limit of the Send-Q:accept queue, i.e. min (backlog, somaxconn), here is 128.
Demo when running properly accept is timely, Recv-Q has been basically 0.
No, it's still connected.
Replace the 5a branch with the following: if (fd=listen fd) {Add a line below, the original code is not deleted but cannot be executed:
if (fd == listen_fd) {
fflush (stdout); sleep (1); continue;
/ / : 5a: new connection → accept, get a new fd that specifically talks to this client
Recompile and then open three terminals, each of which enters a line (e.g. msg1). Three things can be observed.
First of all, three ncs didn't make a mistake and the client thinks they're connected. Handshakes are made by kernels and have nothing to do with the program.
Second, listen to socket's Recv-Q becomes 3: all three connections are in the accept queue.
State Recv-Q Send-Q Local Address:Port Peer Address:Port Process
LISTEN 3 128 0.0.0.0:8888 0.0.0.0:*
Third, even the data sent by the client had been collected by the kernel. Look at the three connections on this side of the service, each Recv-Q is five. - The data lies in the receiving buffer of the core, waiting for a never coming back.
ss -tn state established '( sport = :8888 )'
Recv-Q Send-Q Local Address:Port Peer Address:Port Process
5 0 127.0.0.1:8888 127.0.0.1:34192
5 0 127.0.0.1:8888 127.0.0.1:34190
5 0 127.0.0.1:8888 127.0.0.1:34188
Terminal A will also print listn fd readable once a second, but I don't get it: queues are always empty, listening socket is always readable, and the level of the podl is triggered by the default level that reports it every time. That's what the line sleep(1) is for -- to remove it, the program will be empty between Epolll wait and continue and run a CPU full.
4. Fill the queue (optional)
Replace the listen (listen fd, 128) with the listen (listen fd, 1), on a temporary basis.
Auto-fetched and translated by AI Atlas. All rights belong to the original author.