The hardware scheduler on NVIDIA GPUs is highly inefficient in utilizing micro-architectural hardware resources. It places blocks from the same kernel within the same GPU Streaming Multiprocessor (SM) core, resulting in a stacking co-locating problem, where identical blocks are placed within the same SM core, saturating only a subset of intra-SM hardware resources while leaving others underutilized.

The primary challenge in addressing this issue is that the NVIDIA hardware is closed-source, preventing us from directly modifying the hardware scheduler. To bridge the semantic gap between the resource demands of kernels and the scheduler, we introduce \emph{μShare}, which enables intra-SM scattered co-locating of kernels through a non-intrusive \emph{half-plus blocksize shaping} method. It shapes the blocksize of kernels to a half-plus blocksize (i.e., slightly more than half of the SM’s thread capacity), scattering identical blocks of the same kernel across different SMs. It further adopts a \emph{time-shifted launching} method to reduce intra-SM resource contention. Compared to state-of-the-art systems, \emph{μShare} does not require intrusive modifications to hardware or kernel code, yet it can still improve inference throughput by 26.90%-54.09% and increases low-level hardware utilization by 38.53%–61.15%.