There's some big work I'm missing thats more recent too, again about allocating & scheduling IIRC. Still trying to find it. The third link is in DAMON, which is trying to do a lot to optimize; good thread to tug more on!
I have this pocket belief that eventually we might see post NUMA post coherency architectures, where even a single chip acts more like multiple independent clusters, that use something more like networking (CXL or UltraEthernet or something) to allow RDMA, but without coherency.
Even today, the title here is woefully under-describing the problem. A Epyc chip is actually multiple different compute die, each with their own NUMA zone and their own L3 and other caches. For now yes each socket's memory is all via a single IO die & semi uniform, but whether that holds is in question, and even today, the multiple NUMA zones on one socket already require careful tuning for efficient workload processing.
stego-tech 2 hours ago [-]
Solid writeup of NUMA, scheduling, and the need for pinning for folks who don’t spend a lot of time in the IT side of things (where we, unfortunately, have been wrangling with this for over a decade). The long and short of it is that if you’re building a HPC application, or are sensitive to throughput and latency on your cutting-edge/high-traffic system design, then you need to manually pin your workloads for optimal performance.
One thing the writeup didn’t seem to get into is the lack of scalability of this approach (manual pinning). As core counts and chiplets continue to explode, we still need better ways of scaling manual pinning or building more NUMA-aware OSes/applications that can auto-schedule with minimal penalties. Don’t get me wrong, it’s a lot better than ye olden days of dual core, multi-socket servers and stern warnings against fussing with NUMA schedulers from vendors if you wanted to preserve basic functionality, but it’s not a solved problem just yet.
jasonjayr 2 hours ago [-]
This strikes me as something that Kubernetes could handle if it could support it. You can use affinity to ensure workloads stay together on the same machines, if K8s was NUMA aware, you could extend that affinity/anti-affinity mechanism down to the core/socket level.
EDIT: aaaand ... I commented before reading the article, which describes this very mechanism.
ccgreg 40 minutes ago [-]
> The long and short of it is that if you’re building a HPC application, or are sensitive to throughput and latency on your cutting-edge/high-traffic system design, then you need to manually pin your workloads for optimal performance.
Last time I was architect of a network chip, 21 years ago, our library did that for the user. For workloads that use threads that consume entire cores, it's a solved problem.
I'd guess that the workload you had in mind doesn't have that property.
PerryStyle 58 minutes ago [-]
There are some solutions that try to tackle this in HPC. For example https://github.com/LLNL/mpibind is deployed on El Capitan.
Would be interesting to see if something similar appears for cloud workloads.
wmf 54 minutes ago [-]
If auto-NUMA doesn't handle your workload well and you don't want to manually pin anything, it's always possible to use single-socket servers and set NPS=1. This will make everything uniformly "slow" (which is not that slow).
ccgreg 34 minutes ago [-]
Historically, the Sparc 6400 was derided for not being NUMA, but instead being Uniformly Slow.
colechristensen 2 hours ago [-]
This is one of those way down the road optimizations for folks in fairly rare scale situations in fairly rare tight loops.
Most of us are in the realm of the lowest hanging fruit being database queries that could be 100x faster and functions being called a million times a day that only need to be called twice.
stego-tech 1 hours ago [-]
100% with you there. I can count one time in my entire 15 years where I had to pin a production workload for performance, and it was Hyperion.
In 99% of use cases, there’s other, easier optimizations to be had. You’ll know if you’re in the 1% workload pinning is advantageous to.
For everyone else, it’s an excellent explainer why most guides and documentation will sternly warn you against fussing with the NUMA scheduler.
toast0 10 minutes ago [-]
> In 99% of use cases, there’s other, easier optimizations to be had. You’ll know if you’re in the 1% workload pinning is advantageous to.
Cpu pinning can be super easy too. If you have an application that uses the whole machine, you probably already spawn one thread per cpu thread. Pinning those threads is usually pretty easy. Checking if it makes a difference might be harder... For most applications, it won't make a big difference, but some applications will see a big difference. Usually a positive difference, but it depends on the application.
If you want to cpu pin network sockets, that's not as easy, but it can also make a big difference in some circumstances; mostly if you're a load balancer/proxy kind of thing where you don't spend much time processing packets, just receive and forward. In that case, avoiding cross cpu reads and writes can provide huge speedups, but it's not easy.
frollogaston 53 minutes ago [-]
Yeah, I was once in this situation with a perf-focused software defined networking project. Pinning to the wrong NUMA node slowed it down badly.
Probably another situation is if you're working on a DBMS itself.
https://www.phoronix.com/news/Linux-6.17-NUMA-Locality-Rando... https://www.phoronix.com/news/Linux-6.13-Sched_Ext https://www.phoronix.com/news/DAMON-Self-Tuned-Memory-Tierin... https://www.phoronix.com/news/Linux-6.14-FUSE
There's some big work I'm missing thats more recent too, again about allocating & scheduling IIRC. Still trying to find it. The third link is in DAMON, which is trying to do a lot to optimize; good thread to tug more on!
I have this pocket belief that eventually we might see post NUMA post coherency architectures, where even a single chip acts more like multiple independent clusters, that use something more like networking (CXL or UltraEthernet or something) to allow RDMA, but without coherency.
Even today, the title here is woefully under-describing the problem. A Epyc chip is actually multiple different compute die, each with their own NUMA zone and their own L3 and other caches. For now yes each socket's memory is all via a single IO die & semi uniform, but whether that holds is in question, and even today, the multiple NUMA zones on one socket already require careful tuning for efficient workload processing.
One thing the writeup didn’t seem to get into is the lack of scalability of this approach (manual pinning). As core counts and chiplets continue to explode, we still need better ways of scaling manual pinning or building more NUMA-aware OSes/applications that can auto-schedule with minimal penalties. Don’t get me wrong, it’s a lot better than ye olden days of dual core, multi-socket servers and stern warnings against fussing with NUMA schedulers from vendors if you wanted to preserve basic functionality, but it’s not a solved problem just yet.
EDIT: aaaand ... I commented before reading the article, which describes this very mechanism.
Last time I was architect of a network chip, 21 years ago, our library did that for the user. For workloads that use threads that consume entire cores, it's a solved problem.
I'd guess that the workload you had in mind doesn't have that property.
Would be interesting to see if something similar appears for cloud workloads.
Most of us are in the realm of the lowest hanging fruit being database queries that could be 100x faster and functions being called a million times a day that only need to be called twice.
In 99% of use cases, there’s other, easier optimizations to be had. You’ll know if you’re in the 1% workload pinning is advantageous to.
For everyone else, it’s an excellent explainer why most guides and documentation will sternly warn you against fussing with the NUMA scheduler.
Cpu pinning can be super easy too. If you have an application that uses the whole machine, you probably already spawn one thread per cpu thread. Pinning those threads is usually pretty easy. Checking if it makes a difference might be harder... For most applications, it won't make a big difference, but some applications will see a big difference. Usually a positive difference, but it depends on the application.
If you want to cpu pin network sockets, that's not as easy, but it can also make a big difference in some circumstances; mostly if you're a load balancer/proxy kind of thing where you don't spend much time processing packets, just receive and forward. In that case, avoiding cross cpu reads and writes can provide huge speedups, but it's not easy.
Probably another situation is if you're working on a DBMS itself.