Von Neumann Bottleneck and CPU Cache

The articles written on this blog were created with the goal of teaching. Also, the information in the article may be incorrect, so please be careful.

No matter how fast the CPU can calculate, it cannot do anything if the instructions and data to calculate do not arrive. The phenomenon in which the transfer speed between the CPU and memory limits overall performance is called the Von Neumann Bottleneck.

CPU waits for memory

In the von Neumann architecture, instructions and data use the same memory and bus.

CPU requests instruction
        ↓
Retrieves instructions from memory
        ↓
CPU requests required data
        ↓
Fetch data from memory
        ↓
CPU calculates

Compared to the CPU's calculation speed, main memory access is slow. Additionally, if commands and data move together through a single channel, a situation may arise where each other waits for their turn.

Cache reduces this waiting

Modern CPUs pre-load frequently used instructions and data into Cache Memory near the CPU. The L1 cache is small but very fast, and as it goes toward L2 and L3, it gradually slows down instead of getting bigger.

CPU → L1 Cache → L2 Cache → L3 Cache → Main Memory
fast and small slow and big

Cache uses locality, which means that programs repeatedly use nearby data.

• Temporal locality: The value you just used is likely to be used again soon.
• Spatial locality: If you use an address, there is a high possibility that nearby addresses will also be used.

The reason why code that reads an array sequentially from the front is more advantageous to the cache than code that reads it by jumping around is because of spatial locality.

//Since consecutive memories are read in order, the cache is easy to utilize.
for (int i = 0; i < count; ++i)
{
    sum += values[i];
}

Instruction cache and data cache

Modern CPUs keep instructions and data together in main memory, but separate Instruction Cache and Data Cache in L1 cache. Thanks to this, fetching the next instruction and accessing the current data can be performed somewhat simultaneously.

That is why it is called the modified Harvard structure. It uses a mixture of the strengths of the Harvard structure and the von Neumann structure in the necessary layers.

Ultimately, computer performance is not determined solely by the number of calculations performed by the CPU. You should also look at how contiguously the data is placed, what the cache hit rate is, and how long the CPU waits for memory.