Goroutines — I feel dumb
Doing a Golang refresher. Realize I still do not understand how exactly a new thread is spun when a syscall happens, or what happens to M and P when we are waiting on a channel? What does it mean that “Every M must be able to execute any runnable G” — that is, what does the word “execute” mean here? This document says so below again: “When an M is willing to start executing Go code, it must pop a P form the list. When an M ends executing Go code, it pushes the P to the list.” What is “ends executing”?
Similarly here, what does it mean “M will skip the G”? How does it “skip it” if thread M is running G’s instructions now? Doesn’t it block with the blocking G? What am I missing?
OK, so let’s say in case of I/O, it’s due to netpoller magic:
Whenever you open or accept a connection in Go, the file descriptor that backs it is set to non-blocking mode. This means that if you try to do I/O on it and the file descriptor isn’t ready, it will return an error code saying so. Whenever a goroutine tries to read or write to a connection, the networking code will do the operation until it receives such an error, then call into the netpoller, telling it to notify the goroutine when it is ready to perform I/O again. The goroutine is then scheduled out of the thread it’s running on and another goroutine is run in its place.
When the netpoller receives notification from the OS that it can perform I/O on a file descriptor, it will look through its internal data structure, see if there are any goroutines that are blocked on that file and notify them if there are any. The goroutine can then retry the I/O operation that caused it to block and succeed in doing so.
But still unclear — is it netpoller itself that schedules G out of M? How does M stop running G and starts running some other G’? And what about blocking on a channel operation?
Per this post, this “scheduling out” is done by runtime:
When M executes a certain G, if a syscall or other blocking operations occur, M will block. If there are some Gs currently executing, the runtime will remove the thread M from P, and then create a new thread.
In Go: Goroutine, OS Thread and CPU Management, Vincent describes this as
Go optimizes the system calls — whatever it is blocking or not — by wrapping them up in the runtime. This wrapper will automatically dissociate the P from the thread M and allow another thread to run on it.
The “wrapper” seems to be the netpoller (see above). Ok, I suppose all of this connects, in a somewhat handwavy enough wave, that I’m almost satisfied. I feel like it’s just a couple of dots that are unconnected though, still… I suppose we can stipulate syscalls, but how are channel blocks handled? Is the same mechanism done but via channels, rather than the netpoller in that case?
Some good deeper-than-usual resources
- Go: Goroutine, OS Thread and CPU Management
- Diving deep into the Golang channels
- Illustrated tales of Go runtime schedule
- The Go netpoller and Go Scheduler (this finally explains what a “Processor” P in the GMP trio is)
- Scalable Go Scheduler Design Doc
- runtime/proc.go — scheduler source
- Go study notes
Some refresher examples for myself
As I was refreshing my Golang, I made a bunch of snippets as a kind of study cards. Nothing sophisticated here, just basics. Yesh, it’s like Go By Example, but writing them myself is better for remembering. (Kinda like lecture notes, except, well, you can’t do these with pen and paper…)
DEBEDb 