Applied Science, Faculty of
Electrical and Computer Engineering, Department of
DSpace
UBCV
Zarif, Niloofar
Master of Applied Science - MASc
University of British Columbia
Recommender systems are an essential part of many industries and businesses. Generating accurate recommendations is critical for user engagement and business revenue. Currently, deep learning recommender models are commonly used, but they face challenges in processing and representing categorical data, which is a significant portion of the data used by these models. Embedding layers are often used to handle these complications by storing the numerical representation of different categories of a feature in a reduced vector space. Vectors representing all the categories of a feature in the reduced vector space will be stored in a tabular structure named embedding table. The operation of fetching the vector representation of a category from the embedding table and pooling them is called embedding lookup. However, embedding lookups have large memory footprints and require high memory bandwidth, leading to high latency and low throughput. We have developed a new system called PIM-Rec to address these challenges by using the first commercially available Processing-In-Memory (PIM) capable DRAM modules for embedding lookups. PIM-Rec is the first system to use such DRAM modules, and it has shown an 80% decrease in end-to-end inference cycle latency and an 80% increase in latency-bound throughput compared to the standard CPU-only implementation. PIM DRAM modules are a good candidate for handling embedding lookups, especially with the recent drastic size increase of embedding tables. Although PIM-Rec faced obstacles, it offers a realistic solution and analysis while discovering the obstacles and projecting them. This new system provides a promising solution for improving the efficiency of recommender systems and reducing the load they incur in data centers.
Offloading Embedding Lookups to Processing-In-Memoryfor Deep Learning Recommender ModelsbyNiloofar ZarifBSc. of Computer Engineering, Sharif University of Technology, 2019A THESIS SUBMITTED IN PARTIAL FULFILLMENTOF THE REQUIREMENTS FOR THE DEGREE OFMaster of Applied ScienceinTHE FACULTY OF APPLIED SCIENCE(Electrical and Computer Engineering)The University of British Columbia(Vancouver)August 2023© Niloofar Zarif, 2023The following individuals certify that they have read, and recommend to the Fac-ulty of Graduate and Postdoctoral Studies for acceptance, the thesis entitled:Offloading Embedding Lookups to Processing-In-Memory for Deep Learn-ing Recommender Modelssubmitted by Niloofar Zarif in partial fulfillment of the requirements for the de-gree of Master of Applied Science in Electrical and Computer Engineering.Examining Committee:Alexandra Fedorova, Professor, Electrical and Computer Engineering, UBCSupervisorPrashant Nair, Assistant Professor, Electrical and Computer Engineering, UBCSupervisory Committee MemberSathish Gopalakrishnan, Associate Professor, Electrical and Computer Engineer-ing, UBCSupervisory Committee MemberiiAbstractRecommender systems are an essential part of many industries and businesses.Generating accurate recommendations is critical for user engagement and businessrevenue. Currently, deep learning recommender models are commonly used, butthey face challenges in processing and representing categorical data, which is asignificant portion of the data used by these models. Embedding layers are oftenused to handle these complications by storing the numerical representation of dif-ferent categories of a feature in a reduced vector space. Vectors representing allthe categories of a feature in the reduced vector space will be stored in a tabularstructure named embedding table. The operation of fetching the vector representa-tion of a category from the embedding table and pooling them is called embeddinglookup. However, embedding lookups have large memory footprints and requirehigh memory bandwidth, leading to high latency and low throughput.We have developed a new system called PIM-Rec to address these challengesby using the first commercially available Processing-In-Memory (PIM) capableDRAM modules for embedding lookups. PIM-Rec is the first system to use suchDRAM modules, and it has shown an 80% decrease in end-to-end inference cyclelatency and an 80% increase in latency-bound throughput compared to the stan-dard CPU-only implementation. PIM DRAM modules are a good candidate forhandling embedding lookups, especially with the recent drastic size increase ofembedding tables. Although PIM-Rec faced obstacles, it offers a realistic solutionand analysis while discovering the obstacles and projecting them. This new sys-tem provides a promising solution for improving the efficiency of recommendersystems and reducing the load they incur in data centers.iiiLay SummaryPersonalized recommender systems are present in many platforms in different shapesand consume a considerable portion of AI cycles within datacenters. Most recom-mender systems use deep learning models to generate personalized recommenda-tions. The deep learning models used for personalized recommendation are com-monly memory-intensive. A part of such models called the embedding layer ismostly responsible for making them memory-intensive. Up to 80% of a model’sinference cycle latency can be taken up by the embedding layer operations. Bring-ing computation closer to memory helps with reducing latency by defying thememory-wall. PIM-Rec uses the first commercially available dynamic randomaccess memory (DRAM) modules that are capable of simple data processing inmemory to accelerate inference on embedding layers. A feature of modules usedby PIM-Rec is the scaling of compute power with memory size. This translatesinto the ability to keep up with the rapid pace at which embedding layer sizes aregrowing. PIM-Rec can reduce the inference cycle latency by 89% and increase thethroughput by 9.61X relative to the baseline CPU implementation.ivPrefaceThis thesis represents an original, unpublished, and independent work by the au-thor, N. Zarif. I would like to acknowledge the invaluable contributions of mycollaborators, Justin Wong and Dr. Alexandra Fedorova.Dr. Alexandra Fedorova has served as my supervisor, providing guidance andexpertise throughout the research process. Her profound knowledge of computersystems has been instrumental in shaping the direction of this study, ensuring itsrigor and integrity. I am deeply grateful for her insightful feedback and continuoussupport.I would also like to express my appreciation to Justin Wong for his remark-able assistance in implementing our experimental framework and conducting ex-periments. His technical expertise and meticulous attention to detail have greatlycontributed to the successful completion of this research.vTable of ContentsAbstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iiiLay Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ivPreface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vTable of Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viList of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viiiList of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ixGlossary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiiAcknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiv1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.1 Processing-In-Memory . . . . . . . . . . . . . . . . . . . . . . . 21.2 UPMEM PIM Solution . . . . . . . . . . . . . . . . . . . . . . . 31.3 Embedding Layers . . . . . . . . . . . . . . . . . . . . . . . . . 51.4 Embedding Lookups . . . . . . . . . . . . . . . . . . . . . . . . 81.5 Meta’s DLRM . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122.1 DLRM and its architectural Implications . . . . . . . . . . . . . . 122.2 Recommender Models and Processing In or Near Memory . . . . 14vi2.3 Meta Training and Inference Accelerator . . . . . . . . . . . . . . 163 Workload Characteristics . . . . . . . . . . . . . . . . . . . . . . . . 184 PIM-Rec Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 244.1 CPU-Centric Design . . . . . . . . . . . . . . . . . . . . . . . . 254.2 Minimizing Implementation Overhead . . . . . . . . . . . . . . . 254.3 Pre-processing Embedding Tables . . . . . . . . . . . . . . . . . 274.4 Embedding Data Layout . . . . . . . . . . . . . . . . . . . . . . 284.5 PIM Embedding Lookups . . . . . . . . . . . . . . . . . . . . . . 315 Experimental Results . . . . . . . . . . . . . . . . . . . . . . . . . . 345.1 Speedup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 355.2 Processor Metrics: IPC . . . . . . . . . . . . . . . . . . . . . . . 405.3 Cache Metrics: Hit Rate . . . . . . . . . . . . . . . . . . . . . . 445.4 Embedding Layer Latency Breakdown . . . . . . . . . . . . . . . 466 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 526.1 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55viiList of TablesTable 5.1 The favourable workload for PIM-Rec for which we observed9.61X speedup . . . . . . . . . . . . . . . . . . . . . . . . . . 36Table 5.2 Baseline workload characteristics for experiments in figure 5.2 37Table 5.3 Baseline workload characteristics for experiments in figure 5.3 38Table 5.4 Baseline workload characteristics for experiments in figure 5.4 39Table 5.5 Workload characteristics for experiments in figure 5.5 . . . . . 40Table 5.6 Workload characteristics for experiments in figure 5.6 . . . . . 42Table 5.7 Workload characteristics for experiments in figure 5.7 . . . . . 43Table 5.8 Workload characteristics for experiments in figure 5.8 . . . . . 45Table 5.9 Workload characteristics for experiments in figure 5.9 . . . . . 45Table 5.10 Workload characteristics for experiments in figure 5.10 . . . . 47Table 5.11 Workload characteristics for experiments in figure 5.11 . . . . 49Table 5.12 Workload characteristics for experiments in figure 5.12 . . . . 50viiiList of FiguresFigure 1.1 UPMEM PIM Module Architecture . . . . . . . . . . . . . . 4Figure 1.2 One-hot and multi-hot representation of categorical features . 6Figure 1.3 Example of embedding tables . . . . . . . . . . . . . . . . . 8Figure 1.4 Illustrain of the embedding lookup operation . . . . . . . . . 9Figure 1.5 DLRM general architecture . . . . . . . . . . . . . . . . . . 10Figure 2.1 Profiling of different implementations of a sample DLRM ona single device[19] . . . . . . . . . . . . . . . . . . . . . . . 13Figure 2.2 Inference cycle latency for 3 production-scale instances of DLRM(left) Breakdown of the inference cycle latency for each instance[10].[10] . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14Figure 3.1 LLC miss ratio for different batch sizes for a DLRM instancetrained on Criteo Kaggle dataset[3] and included in MLPerfbenchmark suite[18]- The percentages on top of bars indicatecache miss rate . . . . . . . . . . . . . . . . . . . . . . . . . 20Figure 3.2 L1D miss ratio for different batch sizes for a DLRM instancetrained on Criteo Kaggle dataset[3] and included in MLPerfbenchmark suite[18]- The percentages on top of bars indicatecache miss rate . . . . . . . . . . . . . . . . . . . . . . . . . 21Figure 3.3 IPC of a 4 core machine for different batch sizes for a DLRMinstance trained on Criteo Kaggle dataset[3] and included inMLPerf benchmark suite[18] . . . . . . . . . . . . . . . . . . 21ixFigure 3.4 Latency change with different number of cores for a DLRMinstance trained on Criteo Kaggle dataset[3] and included inMLPerf benchmark suite[18] . . . . . . . . . . . . . . . . . . 22Figure 3.5 Stack trace graph for a process without page faults during em-bedding lookup for a DLRM instance trained on Criteo Kaggledataset[3] and included in MLPerf benchmark suite[18] . . . . 22Figure 3.6 Stack trace graph for a process with page faults during embed-ding lookup for a DLRM instance trained on Criteo Kaggledataset[3] and included in MLPerf benchmark suite[18] . . . . 23Figure 3.7 Another Stack trace graph for a process with page faults dur-ing embedding lookup for a DLRM instance trained on CriteoKaggle dataset[3] and included in MLPerf benchmark suite[18] 23Figure 4.1 The software stack each lookup query goes through . . . . . . 26Figure 4.2 The UPMEM DIMM which includes 64 DPUs on each side ofthe DIMM (rank) and 128 DPUs in total on each DIMM thatincludes 2 ranks[24] . . . . . . . . . . . . . . . . . . . . . . 28Figure 4.3 Data Distribution Layout for Different Embedding Table Col-umn Size . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30Figure 4.4 Illustration of a lookup query on the ith table from the perspec-tive of the jth DPU, which stores the jth column of the ith table. 32Figure 4.5 Illustration of how a lookup query on the ith table is distributedamong the tasklets of the jth DPU to utilize the parallelismoffered by the DPUs. . . . . . . . . . . . . . . . . . . . . . . 33Figure 5.1 Inference cycle speedup of PIM-Rec for an favourable work-load shown in table 5.1 . . . . . . . . . . . . . . . . . . . . . 36Figure 5.2 PIM-Rec inference cycle speedup compared to default CPUimplementation when varying the size of embedding data storedon each DPU - table 5.2 shows complete setup . . . . . . . . 37Figure 5.3 PIM-Rec inference cycle speedup compared to default CPUimplementation when varying the number of embedding tablesworkload has - table 5.3 shows complete setup . . . . . . . . 38xFigure 5.4 PIM-Rec inference cycle speedup compared to default CPUimplementation when varying the the batch size - table 5.4shows complete setup . . . . . . . . . . . . . . . . . . . . . . 39Figure 5.5 CPU and DPU IPC within an inference cycle for workloadshown in table 5.5 with a variable amount of embedding dataper DPU . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41Figure 5.6 CPU IPC within an inference cycle for workload shown in ta-ble 5.6 with variable number of DPUs used for storing embed-ding data . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42Figure 5.7 CPU IPC within an inference cycle for workload shown in ta-ble 5.7 with variable batch size . . . . . . . . . . . . . . . . . 43Figure 5.8 cache hit rate within an inference cycle for workload shown intable 5.8 with variable amount of embedding data stored perDPU . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44Figure 5.9 Cache LLC and L1D hit rate within an inference cycle forworkload shown in table 5.9 with variable number of DPUsused for storing embedding data . . . . . . . . . . . . . . . . 46Figure 5.10 PIM-Rec Lookup latency breakdown for workload shown intable 5.10 with variable number of DPUs used for storing em-bedding data . . . . . . . . . . . . . . . . . . . . . . . . . . 48Figure 5.11 PIM-Rec Lookup latency breakdown for workload shown intable 5.11 with variable amount of embedding data stored perDPU . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49Figure 5.12 PIM-Rec lookup latency breakdown for workload shown in ta-ble 5.12 with variable batch size . . . . . . . . . . . . . . . . 51xiGlossaryThis glossary uses the handy acroynym package to automatically maintain theglossary. It uses the package’s printonlyused option to include only thoseacronyms explicitly referenced in the LATEX source. To change how the acronymsare rendered, change the \acsfont definition in diss.tex.AI Artificial IntelligenceAPI Application Programming InterfaceASIC Application Specific Integrated CircuitBSC Bachelor of ScienceCPU Central Processing UnitDIMM Dual In-Line memory moduleDLRM Deep Learning Recommender ModelDMA Direct Memory AccessDPU Data Processing UnitDRAM Dynamic Random Access MemoryFIM Function In MemoryFLOPS Floating Points Per SecondFPGA Field Programmable Gate ArrayxiiGPU Graphics Processing UnitHBM High Bandwidth MemoryIPC Instructions Per CycleL1D Level 1 Data (cache)LLC Last Level CacheML Machine LearningMLP Multi-Layer PerceptronMPKI Misses per 1000 InstructionsMRAM Main Random Access MemoryMTIA Meta Training and Inference AcceleratorOS Operating SystemPE Processing ElementPIM Processing In MemorySDK Software Development KitSRAM Static Random Access MemoryUBC University of British ColumbiaWRAM Working Random Access MemoryxiiiAcknowledgmentsThanks to my supervisor who guided me through the project and thanks to JustinWong for being an important part of the project. Thanks to my partner for alwaysbeing there for me and to my parents who have been always supportive.xivChapter 1IntroductionWe have witnessed astonishingly rapid progress in machine learning over the pastdecade. This growth has been driven by advancements in neural networks andour understanding of building specialized neural layers for different applications.While certain applications like computer vision and natural language processinghave received more attention and witnessed frequent changes in their popular algo-rithms, recommendation systems gained attention relatively later in this game.For a long time, classic recommendation algorithms such as collaborative fil-tering and knowledge-based recommendation dominated the field. However, morerecently, hyperscalers such as Netflix, Meta (formerly known as Facebook), andGoogle started exploring the use of neural networks to generate recommendations.Recommendations of content, ads, or products to users directly impact a company’srevenue, making it a crucial aspect of their business. Consequently, large compa-nies were hesitant to reveal the cutting-edge methods they were internally employ-ing for recommendation systems. Nevertheless, significant scientific advancementscannot be made without the contribution of a large community of scientists. Thus,less than five years ago, we witnessed the publication of papers on how deep learn-ing and neural networks can be applied in the realm of recommendation systems.In 2016, Google published a paper on their wide and deep learning[6] approachfor recommendation, establishing a foundational framework for designing neuralnetworks and employing deep learning in recommender systems. Google’s Deepand Wide model remained popular in the field of recommendation systems for1several years until 2019, when Meta (Facebook at the time) published a paperon their Deep Learning Recommender Model (DLRM)[19]. Soon after, they re-leased an implementation of the recommender system in PyTorch and Caffe2 tothe public[7]. This enabled researchers to use the available code to conduct exper-iments on various aspects of the recommender system.Once an instance of DLRM is trained, it is deployed to handle thousands oreven millions of inference queries every day. In 2021, Meta reported that 70% ofthe AI cycles in their datacenters were dedicated to recommendation inference[10,14], highlighting the significance of inference performance for these models. Rec-ognizing this problem, we saw an excellent opportunity to leverage the new hard-ware we recently acquired: the UPMEM Processing-In-Memory (PIM) DRAMmodules[24].Originating in Grenoble, France, UPMEM[23] aimed to bring computationcloser to memory by addressing the well-known memory wall problem[26]. Theirsolution involved adding general-purpose computation units to the working mem-ory of a system. The DIMM modules developed by UPMEM utilize numeroussmall processing units to perform simple operations in memory, avoiding the datamovement overhead between the main processor and memory for each operation.What sets their solution apart is the scalability of computation power with mem-ory size. This allows us to leverage a larger pool of processors for larger memoryfootprints, ensuring manageable latency.The remainder of this chapter provides an in-depth exploration of the technolo-gies and concepts employed in this project. We first delve into a comprehensiveunderstanding of the various components of PIM-Rec. Subsequently, we discussthe design decisions we made to construct PIM-Rec and elucidate the challengeswe encountered along the way.1.1 Processing-In-MemoryThe memory wall problem, also known as the Von Neumann bottleneck, has beena challenge in computing for decades[25]. It arises from the limited bandwidthchannel that connects the central processing unit to the main memory. Even withpowerful processors and fast memory, achieving high performance for memory-2intensive workloads is hindered by the significant data access latency caused bythis bottleneck.To address this issue, researchers have explored in-memory or near-memorysolutions that aim to bring processing closer to the data, thus mitigating the VonNeumann bottleneck. Various in-memory or near-memory architectures have beenproposed, some tailored to specific problems such as graph processing[5] or deeplearning recommender models[14], while others propose general-purpose solutionsthat may not be compatible with current hardware[9]. Designing a general-purposesolution that can seamlessly integrate into modern computer systems without com-patibility issues is a complex task. In recent years, two general-purpose Processing-in-Memory (PIM) solutions compatible with current systems have emerged: UP-MEM DRAM DIMMs and Samsung HBM2 Function-In-Memory (FIM). For theacceleration of PIM-Rec, we employ UPMEM PIM modules, and we will delveinto more details about UPMEM PIM in section 3.3.Samsung has recently published a paper on their HBM2 FIM solution[17].High-Bandwidth Memory (HBM) refers to memory modules with higher band-width compared to conventional memory modules like DRAM DIMMs. HBMmodules are optimized for high-performance computing and artificial intelligenceworkloads and are available in three generations. The second generation, knownas Aquabolt, provides up to 1.2 TB/s bandwidth. Unlike UPMEM PIM modules,Samsung HBM2 PIM includes floating-point units. Considering the higher band-width of the underlying memory technology in Samsung HBM2, it may offer amore promising PIM solution for this project; however, it is not currently publiclyavailable. While we are aware of recent advancements in the field of PIM, we willutilize the UPMEM PIM solution for this project, as it is accessible to us. Given thearchitectural similarities between the two solutions, it can be inferred that any per-formance enhancements achieved using UPMEM are likely to be attainable usingSamsung HBM2 as well.1.2 UPMEM PIM SolutionUPMEM [24] provides DRAM DIMMs that are equipped with general-purposein-order processors, operating at a frequency of 450 MHz. These processors are3referred to as DRAM Processing Units (DPUs). Each 64 MB section of DRAM isaccompanied by its own DPU, enabling the computational capacity to scale pro-portionally with the memory size. Consequently, as the memory size increases,the number of processors also increases. A single DPU consists of 24 hardwarethreads that can execute in parallel to mitigate the latency caused by memory stallswithin the in-order pipeline. The DIMMs are organized in ranks, with two ranksper DIMM, and each rank comprises eight memory chips, housing eight DPUs perchip. For instance, an 8GB memory DIMM would contain 128 DPUs since thereis one DPU assigned to each 64MB section. It is important to note that DPUscannot communicate directly with one another, and any inter-DPU communicationmust be facilitated through the host processor, which is typically the main CPU ofthe system. The CPU acts as the orchestrator, responsible for transferring data be-tween the conventional memory and UPMEM DIMMs. Consequently, minimizingthe need for inter-DPU communication becomes crucial to avoid imposing addi-tional overhead on the host. It is worth mentioning that the DPU pipeline does notincorporate floating-point units, making DPUs unsuitable for floating-point oper-ations. We have explored this issue in more depth in [3.5]. The architecture ofUPMEM DRAM modules is illustrated in Figure 1.1.Figure 1.1: UPMEM PIM Module Architecture4Each DPU in UPMEM has direct access to a dedicated 64 MB slice of DRAM,referred to as MRAM, as well as a 64 KB SRAM scratch pad. Additionally, eachDPU has its own SRAM Instruction Memory. The DMA (Direct Memory Access)module is responsible for handling data transfers between the MRAM and othercomponents. The control interface of UPMEM allows a DPU to operate in eitherrunning mode or copying mode. In the copying mode, data is being transferredbetween the MRAM and the host processor (CPU). It’s important to note that theseoperations cannot occur in parallel with DPU processing. The mode switches aretriggered by the DPU program and are not automatically managed by the DMA.When designing a system using UPMEM PIM modules, it’s essential to con-sider the lack of a standard Memory Management Unit (MMU) in UPMEM mod-ules. In current systems, the MMU in DRAM modules plays a crucial role inmemory management, providing functionalities to assist the operating system. Oneimportant function of the MMU is to hide data interleaving from the perspectiveof the operating system, giving the illusion that data is stored sequentially in theDRAM. Data interleaving significantly improves memory bandwidth by enablingparallel reads and writes from different memory banks. In reality, a chunk of datawritten to a DRAM module is interleaved among different memory banks, witheach consecutive byte assigned to a different bank. However, standard DRAMmodules can reverse this interleaving through the MMU, allowing the data to beread in its original meaningful form.In the case of UPMEM DRAM DIMMs, as the data is processed by DPUs, thestorage of interleaved meaningless data is not possible. Therefore, using UPMEMDIMMs as the main memory of a system would require significant modificationsto the memory management techniques of the hardware. In this project, we utilizedUPMEM modules as accelerators rather than as the main memory component.1.3 Embedding LayersAssuming you want to develop a recommender system for a streaming platform,the effectiveness of your recommendation system heavily relies on how you rep-resent the underlying data. While numerical data such as the number of movieswatched or the duration of each movie can be easily represented, there is another5type of data that poses challenges in numerical representation. This type of dataincludes categorical information like movie names, where the strings of words arenot necessarily unique, and movies can share the same name. Additionally, movienames can vary greatly in length. Consequently, categorical data cannot be straight-forwardly expressed as numbers. The question then arises: How can we efficientlyrepresent the movies on our platform?A naive approach to representing movies would be through the use of one-hotvectors. In this representation, each vector corresponds to a specific movie. Allthe vectors have the same length, which equals the total number of movies on theplatform. Each element of the vector is set to zero, except for one element, typicallyassigned a value of 1, indicating which movie the vector represents.For example, let’s consider our platform has ”n” movies including ”Fight Club”and ”Amelie”. We can choose any of the elements of the vector to represent ”FightClub” and an arbitrary but different element to represent ”Amelie”. The one-hotrepresentations of these 2 movies can be seen in figure 1.2. In order to representfeatures that involve a number of categories, such as the movies a user has watched,we need to use the multi-hot representation. In multi-hot representation, each userwill have 1 vector of size ”n” in which multiple elements of the vector are non-zero. An example of this multi-hot representation can also be seen in figure 1.2.Is this one-hot or multi-hot vector representation efficient? Can we improve uponit? The flaws of this solution become evident when considering several problemsassociated with the one-hot system, rendering it impractical for the effective repre-sentation of categorical data.Figure 1.2: One-hot and multi-hot representation of categorical featuresThe first problem arises when dealing with a large number of categories, po-tentially reaching millions. In the case of a streaming platform, the number ofcategories, such as movies, can increase rapidly. Consequently, the corresponding6one-hot vectors used for representation become excessively long, demanding sig-nificant memory resources and making them challenging to handle. Each time anew movie is added to the platform, all the movie representations must be updatedto accommodate the increased length required for representing the newly addedmovies.Another crucial problem with the one-hot vector representation is the inabilityto capture similarities between categories. For instance, movies like ”ET” and”Star Wars” share much more similarities than ”ET” and ”Pulp Fiction.” However,the one-hot encoding fails to capture such relationships. When applying distancemetrics like Euclidean distance, all pairs of vectors are equidistant from each other.This lack of meaningful information about the movies and their properties limitsthe usefulness of the one-hot representation.To address the aforementioned challenges, embeddings are utilized for repre-senting categorical data. Embeddings are fixed-length vectors, often with dimen-sions that are powers of 2, such as 16 or 32. The length of the vectors is a decisionmade by the machine learning engineer based on the requirements of the neuralnetwork. Each vector represents a category while capturing the similarities be-tween categories through metrics like Euclidean distance. For a categorical featurewith n categories, the embedding layer transforms the one-hot system, residing inan n-dimensional space, into a reduced vector space of size D (e.g., D = 32). Theelements of this new 32-dimensional vector are not restricted to zeros and ones,but can take any value between -1 and 1. Consequently, an infinite number ofcategories can be defined within this reduced vector space.An embedding table serves as the data structure that houses all the vectorsrepresenting the categories of a categorical feature. An embedding layer may com-prise multiple embedding tables, with each table dedicated to a specific categoricalfeature. The primary role of the embedding table is to map the user-friendly rep-resentation of a category, such as a string, to a vector that possesses meaning andutility within the neural network. Typically, embedding layers are placed as thefirst layer in a neural network to perform this mapping and feed the transformeddata into the rest of the network. Without the embedding layer, the neural networkwould receive incomprehensible strings or vectors as representations of categoricaldata.7In figure 1.3 we can see examples of embedding tables. We have C embeddingtables where C is equal to the number of categorical features our data has. Eachtable has D columns where D is the size of latent vector space and N columnswhere N is the number of categories, e.g. movies, of that feature. For each table,the number of categories is going to be different so the ith table will have Ni rows.Figure 1.3: Example of embedding tables1.4 Embedding LookupsThe operation of retrieving representations of multiple categories from the embed-ding table and performing a pooling operation on them is commonly referred toas an embedding lookup. Typically, a hash function is employed to determine theindex of the table row that stores the embedding vector for a specific key. Thesecorresponding rows are then combined through an element-wise operation, suchas sum or max pooling. To illustrate this process, please refer to Figure 1.4. Forinstance, let’s consider an embedding table that represents movies. The embeddinglayer receives a list of movie names, which are passed through a hash functionto determine the respective rows in the embedding table. These rows are then re-trieved, and their representations are combined, often through element-wise sum.8The resulting output of this embedding lookup is a D-dimensional vector that rep-resents the combination of these movies, which can subsequently be fed into thesubsequent layers of the neural network.Figure 1.4: Illustrain of the embedding lookup operationIt’s worth noting that embedding lookups exhibit a non-uniform and unpre-dictable memory access pattern since the fetched rows can originate from any partof the table. Additionally, it is important to consider that the number of categoriescan reach millions or even billions, with tens or hundreds of embedding tables po-tentially present. As a consequence, the embedding layer has a substantial memoryfootprint, and the lookup operations themselves are memory-intensive. These as-pects will be further explored in section 3, where we will analyze their impact onsystem metrics.91.5 Meta’s DLRMThe Deep Learning Recommender Model (DLRM), developed by Meta, has beenmade publicly accessible to researchers for utilization and study since 2019[19].DLRM provides a collection of tools that can aid system researchers in examiningthe performance aspects of the model. Gupta et al. conducted a study on the archi-tectural aspect of DLRM [10] and emphasized the significance of addressing thelatency associated with the embedding layer, which represents a memory-intensivecomponent of the inference cycle.A DLRM instance consists of both dense (numerical) features and sparse (cat-egorical) features. The dense input features are handled by a Multi-Layer Percep-tron (MLP), while the sparse features are processed by the embedding layer. Theoutputs from these components interact at the feature interaction layer. Finally, anMLP is responsible for processing the interacted features and generating a rankedlist of recommendations. The overall architecture of DLRM is depicted in Figure1.5.Figure 1.5: DLRM general architectureDLRM can be employed in different input modes. There are three modes avail-10able for the model’s input data:• Dataset: In this mode, the model’s input is derived from a real-world dataset.Several publicly available datasets can be utilized with DLRM, includingCriteo Kaggle and Terabyte.• Synthetic: By leveraging knowledge about the characteristics of a real-worlddataset and workload, DLRM can be employed in the synthetic mode togenerate a workload with identical characteristics. The publicly availableDeepRecSys [11], also provided by Meta, can be beneficial in conjunctionwith DLRM in this mode.• Random: DLRM has the capability to generate a random workload with uni-form distributions, simplifying matters for researchers primarily interestedin exploring the architectural and performance aspects of the model.11Chapter 2Related Work2.1 DLRM and its architectural ImplicationsThe initial discussion of the algorithmic aspect of DLRM can be traced back to apaper published in 2019 by Naumov et al. [19]. The authors emphasize the in-herent difficulties in effective modeling of user behavior and preferences withinrecommendation systems. To address these challenges and enhance the accuracyand performance of such systems, they propose a deep learning-based approach.The paper provides a description of the architecture and constituents comprisingthe DLRM model, encompassing key elements such as the embedding layers re-sponsible for user and item representations, as well as the multi-layer perceptron.Despite being an early-stage exploration of DLRM, the authors dedicated a sectionto the performance challenges presented by the embedding layer. By running per-formance profilings shown in figure 2.1, they illustrate the importance of embed-ding layer in inference latency. Figure 2.1 shows performance profilings for bothPytorch and Caffe2 implementations. The blue part at the bottom of the bar chartson the right chart, labeled as ”embeddingbag ∗ ”, shows the latency caused by theembedding layer. The same color of blue at the bottom of the chart on the left of fig-ure 2.1 shows the embedding layer latency. The label reads as ”SparseLengthSum”since Sparse Length Sum (SLS) is another name for the embedding layer and itsoperations in Caffe2 implementation.Later on, almost the same group of authors published a paper to provide in-12Figure 2.1: Profiling of different implementations of a sample DLRM on asingle device[19]sights into the system performance and architectural considerations behind Meta’sDLRM[10]. Their work explores the challenges faced by Meta in building a scal-able and efficient recommender system to handle the massive user base and large-scale datasets.This paper described the challenges in designing and optimizing each compo-nent of DLRM, including issues related to scalability, efficiency, and model ac-curacy. it discusses techniques such as data parallelism, model parallelism, andmodel compression to address these challenges and improve the performance ofthe recommendation system. Yet, These techniques are mostly useful for fully-connected layers within MLPs. The issue of embedding layer latency and memoryfootprint is left unsolved. In figure 2.2 we can see the inference latency for 3 dif-ferent production-scale DLRM models. All the models are using the same batchsize for better comparsion. The latency breakdown on the left side of figure 2.2shows up to almost 80% of inference latency can be caused by embeddings.By reading this paper, we realize how important meeting throughput SystemLevel Agreements (SLAs) are since DLRM is a user-facing application. We earninsights into the deployment and serving infrastructure employed by Meta to han-dle the high-throughput demands of personalized recommendations. We also seethe use of distributed systems, caching, and load balancing techniques to achievelow latency and high throughput for recommendation requests can offer consid-13Figure 2.2: Inference cycle latency for 3 production-scale instances ofDLRM (left) Breakdown of the inference cycle latency for eachinstance[10]. [10]erable performance improvements. Yet, they do not fully address the embeddinglatency problem for embedding intensive models.2.2 Recommender Models and Processing In or NearMemoryConsidering what we have learned so far about DLRM and the optimization tech-niques employed by Meta researchers, there is limited scope for further improvingthe performance of dense layers. However, sparse embedding layers present a sig-nificant opportunity for performance optimization [10]. The workload exhibits lowlocality, with only temporal locality observed due to the reuse of popular items orcategories. Therefore, caching can provide moderate enhancements [4, 10, 14].The distinctive memory access patterns and the high latency incurred withineach inference cycle have motivated researchers to develop a specialized Near-Memory-Processing solution known as RecNMP [15]. RecNMP proposes a com-modity DRAM module utilizing DDR4 DIMMs to enable the processing of em-bedding pooling operations near memory. Their approach assumes a CPU-centricsystem for inference, and several hardware-software co-optimizations have beenimplemented. Simulations demonstrate a 9.8X reduction in memory latency and a4.2X improvement in throughput. These optimizations include table-aware packetscheduling, memory-side caching, and hot entry profiling. RecNMP primarily14targets Meta datacenters, and considerable effort has been devoted to addressingco-location issues. The researchers have explored strategies for the optimal co-location of models on servers and the distribution of queries among replicas of thesame model. However, it is important to note that RecNMP has only been evalu-ated through simulations, and To the best of our knowledge, the actual hardwarehas never been built.In contrast, our solution offers the unique advantage of processing data di-rectly in memory without requiring data movement between the DRAM moduleand a specialized processor like an FPGA. PIM-Rec leverages a real system builton general-purpose hardware, making it more versatile compared to solutions de-signed specifically for this class of models. Its adaptability to various memory-intensive tasks and workloads makes it easily deployable in datacenters, harnessingthe benefits of the proposed general-purpose hardware.Other researchers have explored the use of near-memory processing for em-bedding lookups. TensorDIMM proposes a vertically integrated hardware/softwareco-design for DIMM-based DRAM modules explicitly tailored for processing em-bedding lookups [16]. Their modules function as a remote memory pool, workingin conjunction with a GPU-centric system that utilizes GPUs for inference. Theirsimulated hardware design demonstrates a performance improvement ranging from6.2X to 17X. However, as we will discuss in section 3.5, a CPU-centric approach ispreferred for recommender model inference, as a GPU-centric system may strug-gle to meet SLA latency standards while consuming more energy. TensorDIMM’ssolution is based on the assumption that embedding tables’ maximum width is 64,limiting its generality. In contrast, PIM-Rec does not make such assumptions andoffers a generalized solution validated by real experiments.The same research team from KAIST has proposed Centaur as a hardware-software co-designed solution to enhance the performance of Deep Learning Rec-ommender models, focusing on both dense and sparse features [13]. Centaur pro-poses a chiplet-based architecture that colocates the CPU and FPGA on the samechip, enabling higher bandwidth between them. This design allows direct memoryaccess for both the CPU and FPGA, resulting in reduced latency when fetchingembedding vectors from memory. By addressing the low compute bandwidth ofCPUs in CPU-centric inference for recommender models, Centaur tackles the chal-15lenge faced by DLRM instances with smaller embedding layers but high computeintensity in the fully-connected layers of the MLP. The FPGA component in thishybrid design complements the limited computing power of the CPU.However, deploying specialized chiplets dedicated solely to a specific classof applications, such as deep learning recommender models, in datacenters is notpractical. Additionally, programming and maintaining FPGAs at scale in datacen-ters pose their own challenges, which hyperscalars are often reluctant to undertake.In summary, Centaur proposes a hardware-software co-designed solution to en-hance Deep Learning Recommender models by considering both dense and sparsefeatures. Its chiplet-based architecture colocates the CPU and FPGA, facilitat-ing improved bandwidth and lower latency for memory access. While Centauraddresses the low compute bandwidth issue of CPUs in recommender models,deploying specialized chiplets in datacenters and managing FPGAs at scale posepractical challenges for hyperscalars.2.3 Meta Training and Inference AcceleratorIn 2023, Meta introduced their own specialized DLRM (Deep Learning Recom-mendation Model) inference accelerator, known as the Meta Training and Infer-ence Accelerator (MTIA)[1]. This accelerator was purposefully designed and fab-ricated as an ASIC (Application-Specific Integrated Circuit) to cater to the spe-cific requirements of recommender inference and training within Meta datacenters.MTIA was seamlessly integrated into PyTorch, resulting in an optimized full-stacksolution.[1, 8]MTIA is equipped with a grid of processing elements (PE) capable of accessingboth on-chip and off-chip memory. To efficiently manage compute and memory re-sources, a dedicated control subsystem operates the accelerator’s firmware, main-taining a dedicated interface with the system’s main processor and orchestratingtasks executed on the accelerator.The on-chip memory of MTIA comprises 128-MB of SRAM, while the off-chip memory consists of 128-GB of DRAM. The use of on-chip memory primarilyfocuses on frequently accessed data to ensure higher bandwidth and lower latency.Each chip contains 64 PEs, which can communicate with each other and effectively16process floating-point operations.[8]MTIA shares many similarities with PIM-Rec in terms of addressing sim-ilar problems by bringing compute closer to memory, which aids in resolvingbandwidth-related issues. Notably, MTIA exhibits remarkable performance withlow and medium complexity DLRM instances, offering up to 2.84X speedup com-pared to GPUs for medium complexity instances of DLRM.However, it is essential to acknowledge that PIM-Rec offers a more genericsolution that can be extended to address embedding lookups not only for recom-mender models but also for other types of deep learning models utilizing embed-dings. Additionally, the estimated average cost per unit for UPMEM DIMMs isconsidered lower compared to MTIA.Overall, MTIA represents a significant advancement in addressing recommenderinference and training needs within Meta datacenters, showing substantial speedupfor specific DLRM instances. Nonetheless, the broader versatility and cost-effectivenessof PIM-Rec’s approach make it an attractive option for various deep learning mod-els using embeddings.17Chapter 3Workload CharacteristicsFor our initial experiments, we utilized three instances of Meta’s Deep Learn-ing Recommender Model (DLRM)[19], which were pre-trained with the CriteoKaggle[3] and Terabyte[2] datasets included in the MLPerf inference benchmarksuite[22]. MLPerf provides standardized benchmarks covering various tasks, in-cluding computer vision and recommendation, for evaluating underlying systems.It is widely accepted and used by the community and offers both training and in-ference workloads.In this phase of the study, our primary focus was to understand the DLRMworkload and the characteristics of the systems it runs on (see Section 4.1 for de-tails). To evaluate the performance of PIM-Rec, we employed Meta’s DLRM withrandomly generated input data, which allowed us to have flexibility in workloadgeneration and model architecture (refer to Chapter 5 for more details).PIM-Rec leverages Meta’s DLRM as the model to be served, benefiting fromits ability to generate workloads with various architectural and runtime parameters.To gain insights into the nature of the workload handled by PIM-Rec, we conductedexperiments performing inference on the pre-trained DLRM instances provided bythe MLPerf inference benchmark. The model we utilized[18] was trained usingthe Criteo Kaggle dataset[3]. Our main interest was to observe the memory, cache,and CPU characteristics of this workload.By analyzing the cache usage of the workload, we observed high miss rates,particularly when the inference batch size was large. Larger batch sizes are gen-18erally preferred for inference tasks as they can offer higher throughput, assumingthe SLA latency criteria are met. However, due to the high cache miss rates in thisworkload, we were unable to fully leverage the potential performance improve-ments that larger batches could provide by utilizing the system’s compute resourcesmore efficiently. Figure 3.1 illustrates the LLC cache miss rates, reaching as highas 84.26% for larger batch sizes, such as 2000. A similar pattern of higher missrates was observed for the L1D cache, with even small batch sizes experiencinghigh miss rates (see Figure 3.2).Deep Learning Recommender Models are generally not compute-intensive. Tomonitor and profile how the CPU is being utilized by these models, we used thesame DLRM model to observe the Instructions Per Cycle (IPC). The experimentswere conducted on a machine with 4 CPU cores, and the observed IPC was aslow as 0.44, which is only 10% of the system’s highest potential IPC (see Figure3.3). These experiments also utilized the DLRM model trained with the CriteoKaggle dataset[2, 18]. By juxtaposing the cache performance of the model withits CPU usage, it becomes evident how memory intensity of the workload impactsperformance and highlights the significance of addressing it.We were also curious to observe how adding parallelism would affect DLRM’sInference Latency. Our primary interest was to determine if highly parallel com-puting would help decrease latency or if the memory bottleneck would dominatethe effect of parallelism. To investigate this, we employed various numbers of CPUcores to handle the same inference job and measured the latency. As depicted inFigure 3.4, increasing the number of CPU cores did lead to a reduction in latency,but it was not a linear relationship. In fact, the latency decrease was far from linear,indicating that adding more CPU cores to this workload does not provide signifi-cant benefits.Upon analyzing the results from Figure 3.4, we became eager to understand thereasons behind this phenomenon and the minimal latency change between differentnumbers of cores. We delved into the stack trace while processing the EmbeddingLayer within an inference cycle. Upon further examination of the operations per-formed by each core during embedding lookups, we observed that the work wasdistributed across multiple CPU cores. However, some cores encountered pagefaults while others did not. Cores experiencing page faults exhibited variances and19Figure 3.1: LLC miss ratio for different batch sizes for a DLRM instancetrained on Criteo Kaggle dataset[3] and included in MLPerf benchmarksuite[18]- The percentages on top of bars indicate cache miss ratespent the majority of cycles handling them, as illustrated in Figure 3.5, Figure 3.6,and Figure 3.7 for cores without page faults and with page faults.These findings indicated that the CPU cores were indeed contending for mem-ory and spent a significant amount of time stalling for memory access. As a result,neither increasing the CPU core performance nor adding more CPU cores provedto be effective in scaling up the performance of embedding lookups. The minorperformance gain shown in Figure 3.4 merely allowed other abstracted operationsbefore and after the actual embedding lookup operation in DLRM to run faster. Toachieve effective and scalable acceleration of embedding lookups, it is imperativeto address the memory-wall problem.20Figure 3.2: L1D miss ratio for different batch sizes for a DLRM instancetrained on Criteo Kaggle dataset[3] and included in MLPerf benchmarksuite[18]- The percentages on top of bars indicate cache miss rateFigure 3.3: IPC of a 4 core machine for different batch sizes for a DLRMinstance trained on Criteo Kaggle dataset[3] and included in MLPerfbenchmark suite[18]21Figure 3.4: Latency change with different number of cores for a DLRM in-stance trained on Criteo Kaggle dataset[3] and included in MLPerfbenchmark suite[18]Figure 3.5: Stack trace graph for a process without page faults during embed-ding lookup for a DLRM instance trained on Criteo Kaggle dataset[3]and included in MLPerf benchmark suite[18]22Figure 3.6: Stack trace graph for a process with page faults during embeddinglookup for a DLRM instance trained on Criteo Kaggle dataset[3] andincluded in MLPerf benchmark suite[18]Figure 3.7: Another Stack trace graph for a process with page faults dur-ing embedding lookup for a DLRM instance trained on Criteo Kaggledataset[3] and included in MLPerf benchmark suite[18]23Chapter 4PIM-Rec DesignPIM-Rec is specifically designed for a CPU system equipped with PIM-enabledDDR4 DIMM DRAM modules, which act as accelerators for the embedding layer.The workload is generated using Meta’s DLRM and is then sent to the PIM moduleby the host CPU. Once prepared, the lookup results are subsequently transferredfrom the accelerator back to the system’s main processor. PIM-Rec encompassestwo primary phases within its design:• Pre-processing phase: In this phase, the embedding tables are copied to thePIM modules and remain there for the duration of the system’s operation.Since this operation needs to be performed only once per model instance,we neither optimize nor evaluate the overhead of this operation.• Lookup phase: During each inference cycle, a query comprising a batch ofmultiple lookups is directed to the PIM module. The PIM module performsthe necessary computations, and when the results are ready, they are copiedback to the host CPU to be fed into the subsequent layers of the neural net-work.In the following chapter, we will provide a detailed explanation of each phaseof the PIM-Rec system’s design.244.1 CPU-Centric DesignGPUs have revolutionized the field of deep learning, playing a pivotal role in driv-ing Machine Learning to its current state of excellence. Their high-throughput andremarkable parallel computing capabilities have been instrumental in advancingthe field, making GPUs the go-to choice for training deep learning recommenda-tion models. However, when it comes to inference, GPUs may not always be themost optimal solution.In customer-facing applications heavily reliant on inference, minimizing la-tency is of utmost importance [12]. The objective here is not necessarily to opti-mize the experience for each individual user to its maximum potential, but rather toprovide an acceptable experience for the majority of users. Thus, in such applica-tions, the goal is to optimize for latency-bound throughput. GPUs excel in scenar-ios where highly parallel workloads need to be optimized for throughput. However,they may not be the best choice for minimizing latency or achieving latency-boundthroughput [12]. Moreover, the cost factor associated with GPUs should also betaken into consideration, particularly when evaluating their value at scale. In hyper-scale environments, the use of GPUs for handling user requests can significantly in-crease the Total Cost of Ownership (TCO). Consequently, CPU-centric systems areoften preferred for performing inference on Recommender Models, even thoughthese models heavily rely on GPUs for training[8].4.2 Minimizing Implementation OverheadDuring the initial phases of developing PIM-Rec, we quickly realized the criticalityof the implementation specifics in our solution. To comprehend the significance ofthis aspect, it is essential to examine the software stack that our lookup queriestraverse. Figure 4.1 illustrates the software stack that each lookup query undergoesupon its arrival. Similarly, the response to the query traverses the same stack inreverse order. The programming language utilized at each stage is indicated inblue at the bottom of each corresponding box in Figure 4.1.Python is renowned for its ease of use and capability to implement intricatelogic with just a few lines of code, liberating developers from delving deeply intothe underlying mechanisms. It has gained immense popularity within the Machine25Figure 4.1: The software stack each lookup query goes throughLearning (ML) community, and major ML frameworks like PyTorch and Tensor-Flow are predominantly built on Python. However, Python’s memory managementdoes not grant users fine-grained control over memory, unlike languages such asC or C++. Consequently, the memory allocated in Python cannot be seamlesslyreused in C or C++ without incurring problems and overhead.DLRM’s lookup queries in PyTorch are essentially memory segments allocatedby Python. In the context of PIM-Rec, the UPMEM SDK acts as the consumer ofthis memory, enabling control of DPUs and data transfer to or from them. Sev-eral methods and tools exist to aid in allocating memory that can be passed fromPython to C. One such tool is the Ctypes library, which allows users to allocateC-compatible memory within Python. It also facilitates the importation of C func-tions into Python code and their invocation by passing C-compatible arguments.However, a crucial point to consider is that data stored in Python structures, suchas Python lists or Tensor instances, must be copied to the C-compatible variableprovided by Ctypes. Memory copies can be expensive, particularly in our memory-intensive workload.The initial implementations of PIM-Rec, which utilized the Ctypes library, ex-hibited an average overhead of 3X when transferring queries from PyTorch to theUPMEM SDK compared to performing lookups on the CPU as DLRM does bydefault. This necessitated finding an alternative approach to query handling thateliminates the need for memory copies. Thus, we delved deeper into PyTorch’simplementation to uncover its inner workings.PyTorch offers a Python interface, but many of its core functionalities are im-plemented in C++. The Python bindings provided within the code generate thePython interface when the C++ code is compiled. This duality of Python and26C++ enables PyTorch to be user-friendly while maintaining high-performance ef-ficiency. Upon inspecting the implementation of embedding layers in PyTorch, wediscovered that the data structures and functionalities are all implemented in C++and wrapped in the Python API. This presented an opportunity for us to leveragethe available queries in C++ and pass them to the UPMEM SDK since C++ point-ers can easily be reused in C.Consequently, rather than integrating calls to the UPMEM SDK in Pythoncode, we incorporated them directly into the C++ source code of PyTorch. Thisdecision allowed us to circumvent memory copies and significantly reduce theoverhead introduced by PIM-Rec during lookup query processing. However, thisapproach came with its own trade-offs, namely the necessity to recompile PyTorchand build our custom version specifically tailored for PIM-Rec, rather than utiliz-ing the pre-built version commonly installed through package managers like pip.Nonetheless, this trade-off proved worthwhile, as it resulted in substantial perfor-mance benefits for the end-users. By adopting this strategy, we successfully min-imized query transfer overhead and harnessed the true performance advantages ofoffloading lookups to PIM modules.4.3 Pre-processing Embedding TablesAs mentioned in the background section about UPMEM PIM modules, they cannotbe used as the system’s main memory. Therefore, in PIM-Rec, the EmbeddingData needs to be copied to the PIM module from the main memory. Dependingon the scenario, we use either a pre-trained model or generate a random one onthe fly. After loading the entire model to the main memory, PIM-Rec moves theembedding tables to the MRAM memory. The embedding table data is copied onlyonce and remains in MRAM as long as the model is up and running. The number ofDPUs (Data Processing Units) to which the embedding tables are copied dependson the model architecture, i.e., the number of embedding tables and the width ofeach table. In real-world recommender models, re-training typically occurs at mostonce or twice a day. Each time the model is re-trained and the embedding data ismodified, the pre-processing phase can be executed again to load the new data toPIM memory.27The way embedding data is distributed and placed on PIM memory directlyaffects the performance we can achieve from DPUs. To optimize performance, weneed to place data on PIM memory in a manner that maximizes the utilization ofavailable parallelism. Therefore, during the pre-processing phase, the embeddingtables are broken down into columns, and each DPU will receive one or multiplecolumns of the Embedding Table until its capacity is full. Further details on thisprocess will be provided in the next section. At this phase, a mapping of Embed-ding data to DPUs will be generated and maintained by the host processor, allowingit to direct queries to the appropriate DPUs later on.4.4 Embedding Data LayoutFigure 4.2: The UPMEM DIMM which includes 64 DPUs on each side ofthe DIMM (rank) and 128 DPUs in total on each DIMM that includes 2ranks[24]Each UPMEM DIMM consists of 2 ranks. As depicted in Figure 4.2, each sideof a DIMM is called a rank and contains 64 DPUs spread over 8 chips. Each chipis equipped with 512 MB of DRAM memory. Consequently, each rank has 4GB ofmemory, organized into 64 small sections, with 64 MB per section. Each 64 MBsection has its own processor, known as a DPU.Within each DPU, there is a 64 MB share of DRAM called MRAM. However,for storing Embedding Tables, only 56 MB of MRAM is utilized, leaving somebuffer space for storing the embedding lookup results. This ensures that the CPUcan read and access the results easily. Additionally, some MRAM memory is re-quired for copying the lookup queries from the host. Considering that tables are28broken down column-wise, each DPU receives the maximum number of columnsthat require less than 56 MB of storage.The element-wise operation employed in DLRM is summation. The key char-acteristic of element-wise summation is that it can be performed independentlyon the elements of a vector. For instance, when summing two vectors, such asA = {0.89,−0.67,0.54} and B = {−0.98,0.75,0.45}, the outcome of a0 + b0 iscompletely independent of a1 + b1. Exploiting this independence, we can assigneach column of the embedding table to a different DPU. This approach enables theaccommodation of large embedding tables with minimal inter-DPU communica-tion. Otherwise, the 64 MB of MRAM memory allocated to each DPU would beinsufficient for conventional table sizes. By breaking down the tables column-wise,we ensure that no additional post-processing is required on the host processor sideto calculate the final results from the partial results generated by the DPUs.To provide a clearer illustration, consider an embedding table of size 20. Ifwe break it down row-wise, the first 10 rows will be assigned to DPU0, while thelast 10 rows will be assigned to DPU1. Now, a lookup query arrives that requireselement-wise summation of rows 2,7,12. We can easily sum up rows 2 and 7 inDPU0, but row 12 is stored in DPU1, and DPUs cannot communicate with eachother. Consequently, we need to sum up rows 2 and 7, copy the result from DPU0to the host CPU, copy row 12 from DPU1, and utilize the main CPU to performthe summation of T2 +T7 with T12. However, if we break down the tables column-wise, in most cases, all the rows of coli will be stored in DPUi. Therefore, when theelement-wise summation for coli of a query is computed in the DPU, it representsthe final result for that element of the vector. On the host side, we simply needto ensure the elements are arranged in the correct order when copying them fromDPUs to the host.The distribution of tables among DPUs and ranks varies based on the numberof columns in the embedding tables. It is known that the number of columns willbe a power of 2, ranging from 8 to 128. The most common table lengths are 32 and64. Figure 4.3 illustrates how the tables are distributed among the DPUs accordingto the table width.An idea we explored to maximize throughput when the Embedding Layer isnot extensive and occupies less than 50% of the PIM memory was to replicate the29Figure 4.3: Data Distribution Layout for Different Embedding Table ColumnSizeembedding tables. In this scenario, instead of having one DPU responsible for aportion of the Embedding Data, two or more DPUs would share the same portion.When a query arrives at the PIM memory, the replicated DPUs would receive onlya subset of the batch. Consequently, a batch of lookups would be divided amongthe replicas, leveraging a larger number of DPUs and achieving higher throughput30through increased parallelism. However, this approach introduces overhead to thehost processor. The host processor in this CPU would be responsible for sortingthe results copied from the replica DPUs. The associated overhead cost couldpotentially outweigh the speedup gained from offloading the lookups to DPUs.Therefore, we did not pursue this idea.4.5 PIM Embedding LookupsOn each inference cycle, a query comprising a batch of queries is copied to thePIM memory. Utilizing the mapping available to the host processor, each part ofthe query is copied to the corresponding DPU or DPUs responsible for storing thatportion of the embedding data. Once the entire query is correctly placed in thePIM memory, the DPUs are launched, and they initiate the processing of the query,computing the lookup results.The queries received by the DPUs consist of two 1D arrays: the indices array,which stores all the indices needed for the lookup batches, and the offsets array,which indicates the starting points of each lookup batch. Initially, the query was ina 2D array format, with each individual lookup operation represented as a 1D arrayof indices to fetch for that operation. The query itself contained a batch of these1D arrays, as each lookup query combined several independent lookup operations.For instance, a single query might consist of 64 independent lookup operations,with each operation encompassing 32 indices, resulting in the lookup of 32 rowsfrom the table. The indices and offsets arrays observed in the DPUs representthe flattened representation of the original 2D query array. Figure 4.4 provides anillustration of how the query arrives in the DPUs.To mitigate the latency caused by memory reads, each DPU employs 12 taskletsto fill its pipeline efficiently. Each tasklet is responsible for handling one batch oflookups at a time and generating one final result for that batch across the subset ofcolumns of a table assigned to its DPU. Once a batch is completed, the tasklet pro-ceeds to the next one in a round-robin fashion. Figure 4.5 illustrates how the lookupoperations are divided among the tasklets within a DPU, effectively harnessing theavailable parallelism.The computed results from the tasklets are stored in a 2048-byte buffer in31Figure 4.4: Illustration of a lookup query on the ith table from the perspectiveof the jth DPU, which stores the jth column of the ith table.WRAM. The size of each DMA (Direct Memory Access) transfer between WRAMand MRAM can range from 8 to 2048 bytes, with the highest memory transferthroughput achieved when the largest size is used. Once the results accumulatedby the tasklets reach 2048 bytes or the entire query has been processed, the com-puted results are copied from WRAM to MRAM, where they can be accessed bythe host.Each rank of DPUs corresponds to a CPU thread responsible for handling thecopying to or from that rank. These threads are initiated during the pre-processing32Figure 4.5: Illustration of how a lookup query on the ith table is distributedamong the tasklets of the jth DPU to utilize the parallelism offered bythe DPUs.phase when the DPUs are allocated. Additionally, a rank-specific host thread canhandle the post-processing of the results generated by that particular rank. Thismeans that the ranks can operate asynchronously, leveraging both the parallelismavailable at the host side and the parallelism offered by the PIM module. The post-processing performed by the rank-specific host threads involves reformatting andarranging the results to prepare them for input to the remaining layers of the neuralnetwork. Once all processing is completed, PIM-Rec sends a signal, marking theend of the forward pass through the Embedding Layer for that inference cycle. Thesignaling and orchestration of host threads are managed by an additional thread onthe host, whose role is to continuously poll the rank-specific threads.33Chapter 5Experimental ResultsAfter presenting the information about PIM-Rec, we will now examine its perfor-mance in action. The implementation of PIM-Rec can be accessed in a GitHubrepository [21]. The main repository contains the host code and DPU code, bothof which are written using UPMEM SDK. Additionally, this repository containsseveral submodules. Among these submodules, one is a fork of Meta’s DLRM im-plementation repository [20], and the other is a fork of Pytorch, which includes ourcustom Pytorch version. Users have the option to recompile and install this customPytorch before running PIM-Rec.Once we completed the end-to-end implementation of PIM-Rec, we subjectedit to various workloads with diverse characteristics to evaluate its performance.Our primary focus was on systemic aspects, such as cache and processor usage.To assess the impact of offloading embeddings and embedding lookups toDPUs, we compared the system performance using the default Pytorch imple-mentation (which utilizes the CPU) with PIM-Rec. During the experiments, weexamined the following metrics:• Speedup: We aimed to determine the speed improvement achieved by of-floading embedding lookups to DPUs. To do this, we measured the inferencecycle latency and calculated the ratio of defualt Pytorch latency to PIM-Reclatency for the same workload.• IPC: In order to assess processor efficiency, we monitored both CPU and34DPU IPC for both baseline Pytorch and PIM-Rec implementations.• Cache metrics: We analyzed the difference in cache usage between the twoimplementations by studying the cache hit rate of CPU LLC and L1D.• Latency breakdown: Understanding the distribution of latency across eachstage of the lookup pipeline in PIM-Rec provided insight into scalability andpotential bottlenecks.In the subsequent sections of this chapter, we will delve deeper into each ofthese metrics and explore the performance of PIM-Rec under various circum-stances.5.1 SpeedupThe primary concern about PIM-Rec is its speed compared to the baseline Pytorchimplementation for DLRM inference. To address this, we measured the latency ofan inference cycle for the same workload using both lookup methods. We measuredend-to-end latency which encompasses the entire process of handling one inferencequery through all the layers of the DLRM instance, including both the latency ofthe embedding layer and the MLP layers. We chose to measure end-to-end latencyrather than solely focusing on the embedding layer latency for two main reasons:1. End-to-end latency provides a more comprehensive understanding of overallperformance, which is our main concern.2. The latency of different layers is not entirely independent. How one layerutilizes system resources can impact the completion time of other parts ofthe neural network.To ensure high accuracy in our measurements, we conducted inference on thesame DLRM instance 100 times for each lookup implementation and then cal-culated the average of those 100 inference cycles as the latency for that specificlookup implementation and workload.We initially aimed to determine the extent of speedup offered by PIM-Recand accordingly selected a favourable workload for PIM-Rec based on the system35architecture. Figure 5.1 illustrates the observed speedup of up to 9.61X comparedto baseline CPU implementation for this workload, the characteristics of which aredetailed in Table 5.1.Figure 5.1: Inference cycle speedup of PIM-Rec for an favourable workloadshown in table 5.1# DPUs emb data per DPU query size # emb tables emb table size4480 400 KB 4 KB 70 6.4 MBTable 5.1: The favourable workload for PIM-Rec for which we observed9.61X speedupAlthough the workload used in these experiments did not utilize the full mem-ory capacity of each DPU, the achieved speedup was still significant. We proceededto explore additional workloads to investigate how PIM-Rec’s speedup comparesto the baseline CPU implementation under different circumstances.Next, we assessed the impact of varying the amount of embedding data storedon each DPU. We kept all other workload characteristics the same as the baseline(see Table 5.2) and only changed the size of the embedding data on each DPU.We change the size of data stored on each DPU by increasing the size of eachembedding table. Figure 5.2 presents the results of this experiment.As depicted in Figure 5.2, PIM-Rec scales well for larger embedding data sizesand can offer up to 10.5X inference speedup when 32 MB of embedding data isstored on each DPU. That would be a workload with 8 MB of embeddings pertable. In other words, the workload for which we saw 10.5X speedup consists36Figure 5.2: PIM-Rec inference cycle speedup compared to default CPU im-plementation when varying the size of embedding data stored on eachDPU - table 5.2 shows complete setup# DPUs query size # emb tables emb table size2048 30.72 KB 32 1.024 to 114.688 GBTable 5.2: Baseline workload characteristics for experiments in figure 5.2of embedding tables that have 8 MB rows each. Considering each table has 64columns, this represents a medium-sized embedding table of size 0.5 GB. Even forsmaller embedding tables, PIM-Rec still provides a promising speedup of around2X, when all other characteristics are maintained as per Table 5.2.Subsequently, we explored how PIM-Rec’s speedup changes when varying thenumber of DPUs used. The number of DPUs is determined based on our designdiscussed in chapter 4, where each DPU stores one column of one table. Therefore,the total number of DPUs used by PIM-Rec is equal to nrcolumn ∗ nrtables. Sothere are 2 ways to change the number of DPUs being used, either by changing thenumber of embedding tables or changing the number of columns per table. Herewe have decided to change the number of embedding tables since it results in a37model architecture closer to real worklaods. The baseline workload characteristicsfor this experiment are shown in Table 5.3, and the results are presented in Figure5.3.Figure 5.3: PIM-Rec inference cycle speedup compared to default CPU im-plementation when varying the number of embedding tables workloadhas - table 5.3 shows complete setup# DPUs query size emb data per DPU emb table size128 to 2048 30.72 KB 2 MB 128 MBTable 5.3: Baseline workload characteristics for experiments in figure 5.3Looking at table 5.3 we can see the workloads used for figure 5.3 are in therange of the smaller workloads seen in figure 5.2 because they are storing smallamounts of embedding data per DPU as well as fewer number of DPUs. Hencethey also offer smaller speedups. So far what we learned is that the speedup forsmaller workloads is smaller and this trend can be seen in both figures 5.2 and 5.3.Additionally, we investigated how PIM-Rec’s speedup is affected by changingthe size of the query processed within each inference cycle, which, in turn, modi-fies the amount of query data transferred from the host to DPUs and the size of thelookup results copied back from the DPUs to the host. This was achieved by vary-38ing the batch size. As we know each lookup query is a batch of lookup operations.By changing the batch size we can change both the size of query being copied toDPUs and the size of embedding lookup results. The baseline workload for thisexperiment is shown in table 5.4 and the experiment results can be seen in figure5.4.Figure 5.4: PIM-Rec inference cycle speedup compared to default CPU im-plementation when varying the the batch size - table 5.4 shows completesetup# DPUs query size emb data per DPU emb table size2048 3.84 to 48 KB 2 MB 128 MBTable 5.4: Baseline workload characteristics for experiments in figure 5.4From the results presented in Figure 5.4, we observed that PIM-Rec does notscale well for large batch sizes, and smaller batches can offer better speedup. Sev-eral factors could contribute to this behavior, such as the possibility of spendingmore time in DPUs when processing a larger batch or encountering higher latencyin memory transfers between the host and DPUs. The increased size of batchesaffects both the data processing within the DPUs and the data transfer between thehost and DPUs during an inference cycle. In Section 5.4, we will conduct a latency39breakdown analysis to gain insights into the reasons behind the observed speedupvariations for different batch sizes in PIM-Rec.5.2 Processor Metrics: IPCOur expectation is that offloading memory-intensive lookups, which have low com-pute intensity, will free up resources for more compute-intensive operations to beperformed by the CPU. As mentioned in Section 3, using the CPU for embeddinglookups resulted in a low Instructions Per Cycle (IPC) for the host CPU. Thus, weanticipate that offloading lookups to DPUs will lead to a higher IPC for the hostCPU. This increase can be attributed to two factors: First, the CPU is no longerburdened with memory-intensive, low-compute lookups, and second, the CPU willprimarily focus on processing the more compute-intensive MLP layers of the net-work.To test this hypothesis, we monitored the IPC for both the CPU and DPUand compared the measured IPC for the same workload between CPU and DPUlookups. It is worth noting that we monitored IPC for the entire inference cycle,not solely for the embedding lookup phase. To ensure greater accuracy, we con-ducted 100 measurements for each workload and considered their average values.The results of these experiments are illustrated in Figure 5.5. For a comprehensiveoverview of the workload characteristics in these experiments, please refer to Table5.5. The experiments were conducted using a system equipped with an Intel XeonSilver 4214 CPU.# DPUs query size emb data per DPU emb data size batch size2048 30.72 KB 0.5 to 8 MB 1 to 16 GB 64Table 5.5: Workload characteristics for experiments in figure 5.5Key observations from Figure 5.5 are as follows:• DPU IPC consistently remains at one. Given that the DPU has a single core,an IPC of 1 indicates that the processor is being fully utilized. PIM-Receffectively maintains a full pipeline on the DPU by utilizing multiple taskletsfor processing the lookups. In this experiment, 14 tasklets were used.40Figure 5.5: CPU and DPU IPC within an inference cycle for workload shownin table 5.5 with a variable amount of embedding data per DPU• In all cases, when lookups are offloaded to DPUs, CPU IPC shows a sig-nificant increase compared to the default CPU lookup. This confirms ourhypothesis that offloading lookups to PIM-Rec enhances CPU efficiency, en-abling the main processor to focus on more compute-intensive tasks.• The Intel Xeon Silver 4214 CPU used in these experiments has 16 cores.Despite the considerable improvement in IPC achieved by PIM-Rec, the ob-served IPC is still below the maximum potential of this CPU. This suggeststhat there may be further room for improvements in CPU utilization.• There is no clear trend in IPC as we scale the workload and store more em-bedding data on DPUs. To gain more insights into IPC scaling, we shouldconsider other workload parameters.To explore how IPC scales with varying workloads, we conducted another ex-periment by altering the number of DPUs used. This time, all workload character-istics, including the amount of embedding data stored on each DPU, remained the41same, while the number of embedding tables in the workload was changed. Thisvariation directly impacted the number of DPUs used for storing the tables. Theresults of these experiments are depicted in Figure 5.6, and the detailed workloadcharacteristics are presented in Table 5.6.Figure 5.6: CPU IPC within an inference cycle for workload shown in table5.6 with variable number of DPUs used for storing embedding data# DPUs query size emb data per DPU emb data size batch size128 to 2048 30.72 KB 2 MB 256 MB to 4 GB 64Table 5.6: Workload characteristics for experiments in figure 5.6As the DPU IPC consistently remained at 1 in our experiments, we have omit-ted showing the DPU IPC in Figure 5.6 to focus solely on the comparison of CPUIPC. Once again, we observe that CPU IPC is significantly higher for DPU imple-mentation compared to the CPU one. However, no specific trend in IPC is evidentwhile scaling the workload in this experiment as well.To further investigate scalability, we explored another aspect by varying thebatch size, thereby altering the amount of data transferred to and from the DPUs ineach inference cycle. We monitored the IPC and observed how it changed accord-ingly. The results of this experiment are displayed in Figure 5.7, and the workload42characteristics are detailed in Table 5.7.Figure 5.7: CPU IPC within an inference cycle for workload shown in table5.7 with variable batch size# DPUs query size emb data per DPU emb data size batch size2048 3.84 to 48 KB 2 MB 4 GB 8 to 100Table 5.7: Workload characteristics for experiments in figure 5.7Indeed, the results consistently demonstrate that CPU IPC is significantly higherwhen lookups are offloaded to PIM-Rec, affirming the enhanced CPU efficiencyachieved through offloading. Although no specific trend in CPU IPC emerges whenthe batch size and query size are altered, our primary objective of achieving bet-ter CPU efficiency with PIM-Rec, compared to the baseline, has been successfullymet. The offloading of lookups to PIM-Rec has effectively improved the utiliza-tion of the main processor, making it more capable of handling compute-intensivetasks.435.3 Cache Metrics: Hit RateThe unpredictable memory access pattern of embedding lookups, with data ac-cessed over a large working set and accessed locations being highly random, resultsin a low cache hit rate for DLRM inference, as observed in Section 3. We antic-ipate that PIM-Rec will mitigate this issue to some extent. By storing embeddingdata in PIM modules and removing the CPU’s involvement in lookups, we expect areduction in cache misses. In PIM-Rec, the CPU primarily handles the MLP layersof the network, which are known to cause fewer cache misses. To verify if thisexpectation holds true for PIM-Rec, we conducted experiments illustrated in Fig-ure 5.8. These experiments examine the cache hit rates and explore how PIM-Recaffects cache behavior.Figure 5.8: cache hit rate within an inference cycle for workload shown intable 5.8 with variable amount of embedding data stored per DPUFrom the experiment results shown in Figure 5.8, we can draw the followingconclusions:• For both LLC and L1D, the cache hit rate for PIM-Rec is higher than thebaseline CPU implementation in almost all cases.44# DPUs query size emb data per DPU emb data size batch size2048 30.72 KB 0.5 to 8 MB 1 to 16 GB 64Table 5.8: Workload characteristics for experiments in figure 5.8• The increase in LLC cache hit rate is particularly significant. When 4 MBof embedding data per DPU is used, PIM-Rec’s LLC cache hit rate is nearlydouble that of the CPU baseline implementation. The difference in LLC hitrate is close to zero only for 2 MB of embedding data per DPU.• The L1D hit rate for PIM-Rec is consistently higher with no exceptions.• There is no specific trend in cache hit rate as we scale the workload.Furthermore, we examined the cache hit rate when varying the number ofDPUs used. To achieve this, we changed the number of embedding tables inPIM-Rec, which impacted the number of DPUs utilized. The experiment setupis detailed in Table 5.9, and the results are shown in Figure 5.9.# DPUs query size emb data per DPU emb data size batch size128 to 2048 30.72 KB 2 MB 256 MB to 4 GB 64Table 5.9: Workload characteristics for experiments in figure 5.9In Figure 5.9, we observe that PIM-Rec consistently maintains a higher LLChit rate compared to the CPU baseline, and this gap between PIM-Rec’s LLC hitrate and the baseline increases as we use more DPUs. The reason behind thisphenomenon is that the CPU’s LLC hit rate decreases with a larger number oftables, while PIM-Rec’s LLC hit rate increases. Although the L1D hit rate for bothimplementations is relatively similar, PIM-Rec performs slightly better in all cases.In conclusion, PIM-Rec has successfully achieved the intended goal in termsof cache hit rate. Throughout all the experiments we conducted, PIM-Rec consis-tently exhibits a higher cache hit rate compared to its CPU baseline implementa-tion. This demonstrates that by offloading lookups to PIM-Rec and allowing the45Figure 5.9: Cache LLC and L1D hit rate within an inference cycle for work-load shown in table 5.9 with variable number of DPUs used for storingembedding dataCPU to primarily handle the compute-intensive MLP layers, PIM-Rec effectivelyreduces cache misses and improves cache efficiency.5.4 Embedding Layer Latency BreakdownUnderstanding the latency breakdown of PIM-Rec’s embedding lookups is crucialin identifying the stages of the execution pipeline that scale well and those thatmay present challenges in terms of scalability. This breakdown helps us determinewhich workloads can benefit the most from switching to PIM-Rec by leveragingthe stages that scale effectively while minimizing stress on less scalable stages.PIM-Rec’s lookup execution pipeline consists of four main stages:1. Query Copy to DPUs: In this stage, when a query is received from DLRM,it is copied from the host to the DPUs and stored in MRAM. The host efficientlydistributes the relevant parts of the query to each DPU, ensuring that each DPUreceives the query segments meant for the specific part of the embedding data it46stores. Memory transfers to DPUs are done in bulk, allowing for parallel transfersacross all DPUs.2. Processing in DPU: Once DPUs receive the query and understand theirrespective embedding lookup tasks, they begin executing and processing the queryto generate the lookup results. The results are then stored in MRAM for later accessby the host.3. Result Copy from DPU to Host: After DPUs complete the processing,they inform the host, which then proceeds to copy the lookup results from MRAMback to the main memory of the system. This memory transfer is asynchronous,accommodating variations in processing times among different DPUs.4. Query Post-processing: Once all the embedding results from a DPU arecopied back to the host, post-processing begins. This stage involves converting the32-bit integers back to 32-bit floating-point values. While DPUs process data as32-bit integers due to a lack of floating-point operators, the original format of thedata was floating-point. The post-processing task leverages the host’s multithread-ing capability to handle queries from different embedding tables in parallel usingdifferent threads. Similar to the previous stage, this process is also asynchronous,considering the asynchronous nature of the previous stage, the result copy, whichmay finish at different times for different DPUs.Having identified the stages of the lookup pipeline, we can now examine someexperimental results. Figure 5.10 illustrates the lookup latency breakdown as wescale the number of DPUs used, achieved by altering the number of embeddingtables. A comprehensive overview of the workload characteristics for these exper-iments can be found in Table 5.10. These results will provide insights into howeach stage of the PIM-Rec pipeline contributes to the overall latency and how thepipeline behaves with varying numbers of DPUs.# DPUs query size emb data per DPU emb data size batch size128 to 2048 30.72 KB 2 MB 256 MB to 4 GB 64Table 5.10: Workload characteristics for experiments in figure 5.10The latency breakdown depicted in Figure 5.10 provides valuable insights intothe behavior of PIM-Rec’s lookup pipeline for different workloads involving vary-47Figure 5.10: PIM-Rec Lookup latency breakdown for workload shown in ta-ble 5.10 with variable number of DPUs used for storing embeddingdataing numbers of DPUs or embedding tables:1. For workloads with a smaller number of DPUs or embedding tables, thelookup latency in PIM-Rec is more evenly distributed between the DPU process-ing, post-processing, and query copying stages. The share of query copy to DPUsremains relatively small compared to the other stages. 2. However, as the numberof embedding tables increases and more DPUs are involved, the share of querycopy to DPUs becomes more significant. This happens because the process ofcopying the correct queries to numerous DPUs requires more time, leading to alarger portion of the overall latency being attributed to this stage. 3. The shareof DPU processing remains relatively constant since the time spent on processingwithin each DPU remains consistent regardless of the number of DPUs involved.4. Similar to DPU processing, query post-processing exhibits a fairly constantlatency, and its share of the lookup latency decreases as the number of DPUs in-creases. 5. The result of copying data back to the host takes up a small portionof the overall latency in all cases. However, its share increases when more DPUs48are involved due to the additional time required for memory transfer from multipleDPUs.To further investigate the impact of the amount of embedding data stored byeach DPU on the latency breakdown, we conducted experiments illustrated in Fig-ure 5.11. In these experiments, we kept the number of DPUs constant and variedthe amount of embedding data stored by each DPU, ranging from 0.5 MB to themaximum capacity of 56 MB. For a comprehensive description of the experimen-tal setup, please refer to Table 5.11. These experiments provide insights into howchanges in the size of embedding data affect each stage of the PIM-Rec lookuppipeline and its overall latency distribution.Figure 5.11: PIM-Rec Lookup latency breakdown for workload shown in ta-ble 5.11 with variable amount of embedding data stored per DPU# DPUs query size emb data per DPU emb data size batch size2048 30.72 KB 0.5 to 56 MB 1 to 114 GB 64Table 5.11: Workload characteristics for experiments in figure 5.11In Figure 5.11, we observe that storing more data per DPU does not signifi-49cantly affect the latency caused by processing within the DPUs. Although thereare some fluctuations, there is no clear trend, and the share of latency breakdownremains relatively constant. While the absolute value of DPU processing latencyincreases as more data is stored, it does not escalate as rapidly as the latency forquery copying. The majority of PIM-Rec’s lookup latency is attributed to the pro-cess of copying queries to DPUs.As previously seen, the latency caused by copying results back from DPUs tothe host is minimal, as the size of the result data being transferred is considerablysmaller than the query data.Given the substantial impact of query copy in PIM-Rec’s lookup latency, it isintriguing to investigate how the breakdown changes when altering the size of thequeries. By modifying the batch size, we can change both the query size beingcopied to DPUs and the result size being copied back to the host. The results ofthese experiments are depicted in Figure 5.12, and the workload characteristics canbe found in Table 5.12. These experiments shed light on how changes in query sizeaffect each stage of the PIM-Rec lookup pipeline and its overall latency distribu-tion.# DPUs query size emb data per DPU emb data size batch size2048 3.84 to 48 KB 2 MB 4 GB 8 to 100Table 5.12: Workload characteristics for experiments in figure 5.12The findings in Figure 5.12 are indeed reassuring. Despite the introduction oflarger queries with larger batch sizes, the query copying latency does not exhibit asignificant increase in its share of the latency breakdown. This indicates that PIM-Rec is capable of handling larger batch sizes without suffering a massive increasein latency caused by data transfer.Overall, the results from PIM-Rec’s latency breakdown align well with our ex-pectations. Throughout various scaling scenarios, no red flags were observed inany stages of PIM-Rec’s latency breakdown. This provides us with confidence thatPIM-Rec is well-equipped to handle larger lookup workloads without compromis-ing on efficiency and latency. The scalability demonstrated by PIM-Rec in differentdirections further validates its potential as an efficient solution for handling a wide50Figure 5.12: PIM-Rec lookup latency breakdown for workload shown in ta-ble 5.12 with variable batch sizerange of workloads.51Chapter 6ConclusionThe experimentation with PIM-Rec revealed notable advantages in terms of re-ducing latency and increasing throughput for inference on Recommender Models.Employing PIM technology holds promise, but it is essential to approach the sys-tem’s design pragmatically, considering that implementation overhead could po-tentially overshadow its speedup benefits. Although finely specialized hardwaredesigns may seem attractive, they are often impractical and challenging to justifyfor deployment in rapidly changing large datacenters. A more feasible approachlies in adopting a general-purpose PIM solution, even if it may not yield mirac-ulous performance enhancements. Nonetheless, given the immense scale of in-ference workloads at hyperscalar datacenters, even slight improvements in latencyand throughput can translate to substantial cost savings, potentially amounting tomillions of dollars.The key aspect of our scalable design is the correlation between computingresources and memory size. As the memory size increases, the system accommo-dates a proportional increase in processors. This approach ensures scalability andflexibility, crucial factors for hyper scalars, both of which are addressed effectivelyby PIM-Rec.526.1 Future WorkPIM-Rec still has the potential to incorporate more complex optimizations. Cur-rently, we are actively engaged in testing PIM-Rec with DLRM instances trainedon real datasets. Collaboration with researchers from ETH Zurich is underwayto evaluate PIM-Rec’s performance on recommendation models included in theMLPerf[18] inference benchmark suite. We expect to observe similar results andspeedups to those demonstrated in our experiments with randomly generated work-loads. Furthermore, we anticipate leveraging the skewed key access pattern presentin real datasets to further optimize PIM-Rec lookups.However, the servers currently at our disposal can only accommodate one of thebenchmarks provided by MLPerf due to two limiting factors: the size of the datasetused for model training and the model size itself. The dataset’s size presents a con-straint because the MLPerf implementation provided by the organization requiresusers to load and pre-process the entire dataset, necessitating a machine with almost256 GB of memory for a 100 GB dataset. The Criteo Kaggle dataset[3] is 12 GB,which our current servers can handle, but the Terabyte dataset at 91 GB[2] exceedstheir capacity. Therefore, the only MLPerf benchmark we can currently managewith our servers is the DLRM instance trained on the Criteo Kaggle dataset[3],which has a memory footprint of 1 GB for the entire model, including both the em-bedding and MLP layers. Fortunately, the embedding data fits comfortably withinour server’s UPMEM DIMMs, and the dataset’s size permits straightforward load-ing and pre-processing on our server.Regrettably, the other two benchmarks are beyond our current server capabili-ties:• The DLRM instance trained on the Terabyte dataset with a model size of10 GB can readily fit in memory, with our server’s UPMEM DIMMs ableto handle the embedding data and the conventional memory accommodat-ing the MLP layers. However, loading and pre-processing the dataset posechallenges.• The DLRM instance trained again on the Terabyte dataset with a model sizeof 100 GB faces constraints in both storing the model and pre-processing thedataset. Loading the entire model into conventional memory at the outset53presents issues with Python’s memory management and garbage collector,as it takes time to clear out embedding data that is no longer accessed withinthe conventional memory.To overcome these limitations, we are actively pursuing access to servers equippedwith UPMEM DIMMs and larger conventional memory sizes, which would allowus to conduct experiments on all the MLPerf benchmarks. In the meantime, weare also exploring alternative approaches to dataset pre-processing, such as try-ing pre-processing on another server and then transferring the pre-processed filesto the PIM-equipped server. Additionally, we are modifying sections of the coderesponsible for dataset loading and pre-processing.Overall, we are confident that with these efforts, we will soon be able to eval-uate PIM-Rec’s capabilities with all three models, providing valuable insights intoits performance.54Bibliography[1] Meta training and inference accelerator (ai mtia).https://ai.meta.com/blog/meta-training-inference-accelerator-AI-MTIA/,2023. Accessed: July 22, 2023. → page 16[2] Download terabyte click logs dataset on criteo.https://labs.criteo.com/2013/12/download-terabyte-click-logs/, 2023.Accessed on 17/07/2023. → pages 18, 19, 53[3] Kaggle display advertising challenge dataset on criteo. https://labs.criteo.com/2014/02/kaggle-display-advertising-challenge-dataset/,2023. Accessed on 17/07/2023. → pages ix, x, 18, 20, 21, 22, 23, 53[4] M. Adnan, Y. E. Maboud, D. Mahajan, and P. J. Nair. Acceleratingrecommendation system training by leveraging popular choices. Proc.VLDB Endow., 15(1):127–140, sep 2021. ISSN 2150-8097.doi:10.14778/3485450.3485462. URLhttps://doi.org/10.14778/3485450.3485462. → page 14[5] J. Ahn, S. Hong, S. Yoo, O. Mutlu, and K. Choi. A scalableprocessing-in-memory accelerator for parallel graph processing. In 2015ACM/IEEE 42nd Annual International Symposium on ComputerArchitecture (ISCA), pages 105–117, 2015. doi:10.1145/2749469.2750386.→ page 3[6] H.-T. Cheng, L. Koc, J. Harmsen, T. Shaked, T. Chandra, H. Aradhye,G. Anderson, G. Corrado, W. Chai, M. Ispir, R. Anil, Z. Haque, L. Hong,V. Jain, X. Liu, and H. Shah. Wide deep learning for recommender systems,2016. → page 1[7] M. N. et al. An implementation of deep learning recommendation model forpersonalization and recommendation systems.https://github.com/facebookresearch/dlrm, 2019. → page 255[8] A. Firoozshahian, J. Coburn, R. Levenstein, R. Nattoji, A. Kamath, O. Wu,G. Grewal, H. Aepala, B. Jakka, B. Dreyer, A. Hutchin, U. Diril, K. Nair,E. K. Aredestani, M. Schatz, Y. Hao, R. Komuravelli, K. Ho, S. Abu Asal,J. Shajrawi, K. Quinn, N. Sreedhara, P. Kansal, W. Wei, D. Jayaraman,L. Cheng, P. Chopda, E. Wang, A. Bikumandla, A. Karthik Sengottuvel,K. Thottempudi, A. Narasimha, B. Dodds, C. Gao, J. Zhang,M. Al-Sanabani, A. Zehtabioskuie, J. Fix, H. Yu, R. Li, K. Gondkar,J. Montgomery, M. Tsai, S. Dwarakapuram, S. Desai, N. Avidan, P. Ramani,K. Narayanan, A. Mathews, S. Gopal, M. Naumov, V. Rao, K. Noru,H. Reddy, P. Venkatapuram, and A. Bjorlin. Mtia: First generation silicontargeting meta’s recommendation systems. In Proceedings of the 50thAnnual International Symposium on Computer Architecture, ISCA ’23, NewYork, NY, USA, 2023. Association for Computing Machinery. ISBN9798400700958. doi:10.1145/3579371.3589348. URLhttps://doi.org/10.1145/3579371.3589348. → pages 16, 17, 25[9] S. Ghose, A. Boroumand, J. S. Kim, J. Go´mez-Luna, and O. Mutlu.Processing-in-memory: A workload-driven perspective. IBM Journal ofResearch and Development, 63(6):3:1–3:19, 2019.doi:10.1147/JRD.2019.2934048. → page 3[10] U. Gupta, X. Wang, M. Naumov, C. Wu, B. Reagen, D. Brooks, B. Cottel,K. M. Hazelwood, B. Jia, H. S. Lee, A. Malevich, D. Mudigere,M. Smelyanskiy, L. Xiong, and X. Zhang. The architectural implications offacebook’s dnn-based personalized recommendation. CoRR,abs/1906.03109, 2019. URL https://arxiv.org/abs/1906.03109. → pagesix, 2, 10, 13, 14[11] U. Gupta, S. Hsia, V. Saraph, X. Wang, B. Reagen, G.-Y. Wei, H.-H. S. Lee,D. Brooks, and C.-J. Wu. Deeprecsys: A system for optimizing end-to-endat-scale neural recommendation inference. In 2020 ACM/IEEE 47th AnnualInternational Symposium on Computer Architecture (ISCA), pages 982–995,2020. doi:10.1109/ISCA45697.2020.00084. → page 11[12] C. Huyen. Designing Machine Learning Systems: An Iterative Process forProduction-ready Applications. O’Reilly Media, Incorporated, 2022. ISBN9781098107963. URL https://books.google.ca/books?id=BAy zgEACAAJ.→ page 25[13] R. Hwang, T. Kim, Y. Kwon, and M. Rhu. Centaur: A chiplet-based, hybridsparse-dense accelerator for personalized recommendations. In Proceedingsof the ACM/IEEE 47th Annual International Symposium on Computer56Architecture, ISCA ’20, page 968–981. IEEE Press, 2020. ISBN9781728146614. doi:10.1109/ISCA45697.2020.00083. URLhttps://doi.org/10.1109/ISCA45697.2020.00083. → page 15[14] L. Ke, U. Gupta, C.-J. Wu, B. Y. Cho, M. Hempstead, B. Reagen, X. Zhang,D. Brooks, V. Chandra, U. Diril, A. Firoozshahian, K. Hazelwood, B. Jia,H.-H. S. Lee, M. Li, B. Maher, D. Mudigere, M. Naumov, M. Schatz,M. Smelyanskiy, and X. Wang. Recnmp: Accelerating personalizedrecommendation with near-memory processing, 2019. → pages 2, 3, 14[15] L. Ke, U. Gupta, B. Y. Cho, D. Brooks, V. Chandra, U. Diril,A. Firoozshahian, K. Hazelwood, B. Jia, H.-H. S. Lee, M. Li, B. Maher,D. Mudigere, M. Naumov, M. Schatz, M. Smelyanskiy, X. Wang,B. Reagen, C.-J. Wu, M. Hempstead, and X. Zhang. Recnmp: Acceleratingpersonalized recommendation with near-memory processing. InProceedings of the ACM/IEEE 47th Annual International Symposium onComputer Architecture, ISCA ’20, page 790–803. IEEE Press, 2020. ISBN9781728146614. doi:10.1109/ISCA45697.2020.00070. URLhttps://doi.org/10.1109/ISCA45697.2020.00070. → page 14[16] Y. Kwon, Y. Lee, and M. Rhu. Tensordimm: A practical near-memoryprocessing architecture for embeddings and tensor operations in deeplearning. In Proceedings of the 52nd Annual IEEE/ACM InternationalSymposium on Microarchitecture, MICRO ’52, page 740–753, New York,NY, USA, 2019. Association for Computing Machinery. ISBN9781450369381. doi:10.1145/3352460.3358284. URLhttps://doi.org/10.1145/3352460.3358284. → page 15[17] Y.-C. Kwon, S. H. Lee, J. Lee, S.-H. Kwon, J. M. Ryu, J.-P. Son, O. Seongil,H.-S. Yu, H. Lee, S. Y. Kim, Y. Cho, J. G. Kim, J. Choi, H.-S. Shin, J. Kim,B. Phuah, H. Kim, M. J. Song, A. Choi, D. Kim, S. Kim, E.-B. Kim,D. Wang, S. Kang, Y. Ro, S. Seo, J. Song, J. Youn, K. Sohn, and N. S. Kim.25.4 a 20nm 6gb function-in-memory dram, based on hbm2 with a 1.2tflopsprogrammable computing unit using bank-level parallelism, for machinelearning applications. In 2021 IEEE International Solid- State CircuitsConference (ISSCC), volume 64, pages 350–352, 2021.doi:10.1109/ISSCC42613.2021.9365862. → page 3[18] MLCommons. MLCommons/Inference: Recommendation - DLRMPyTorch. https://github.com/mlcommons/inference/tree/master/recommendation/dlrm/pytorch, Accessed 2023-07-18. → pagesix, x, 18, 19, 20, 21, 22, 23, 5357[19] M. Naumov, D. Mudigere, H.-J. M. Shi, J. Huang, N. Sundaraman, J. Park,X. Wang, U. Gupta, C.-J. Wu, A. G. Azzolini, D. Dzhulgakov, A. Mallevich,I. Cherniavskii, Y. Lu, R. Krishnamoorthi, A. Yu, V. Kondratenko,S. Pereira, X. Chen, W. Chen, V. Rao, B. Jia, L. Xiong, and M. Smelyanskiy.Deep learning recommendation model for personalization andrecommendation systems, 2019. → pages ix, 2, 10, 12, 13, 18[20] J. W. Niloofar Zarif. Pim-rec fork of dlrm.https://github.com/UBC-ECE-Sasha/PIM-dlrm/tree/bb593a4a6d68de6d3257ac3e4fccdf26e929a1cc, 2022. Accessed: July 22,2023. → page 34[21] J. W. Niloofar Zarif. Pim-rec on github.https://github.com/UBC-ECE-Sasha/PIM-Embedding-Lookup, 2022.Accessed: July 22, 2023. → page 34[22] V. J. Reddi, C. Cheng, D. Kanter, P. Mattson, G. Schmuelling, C.-J. Wu,B. Anderson, M. Breughe, M. Charlebois, W. Chou, R. Chukka,C. Coleman, S. Davis, P. Deng, G. Diamos, J. Duke, D. Fick, J. S. Gardner,I. Hubara, S. Idgunji, T. B. Jablin, J. Jiao, T. S. John, P. Kanwar, D. Lee,J. Liao, A. Lokhmotov, F. Massa, P. Meng, P. Micikevicius, C. Osborne,G. Pekhimenko, A. T. R. Rajan, D. Sequeira, A. Sirasao, F. Sun, H. Tang,M. Thomson, F. Wei, E. Wu, L. Xu, K. Yamada, B. Yu, G. Yuan, A. Zhong,P. Zhang, and Y. Zhou. Mlperf inference benchmark. In 2020 ACM/IEEE47th Annual International Symposium on Computer Architecture (ISCA),pages 446–459, 2020. doi:10.1109/ISCA45697.2020.00045. → page 18[23] U. SAS. UPMEM Company upmem organization, 2023. URLhttps://www.upmem.com/company/. → page 2[24] U. SAS. UPMEM PIM Technology upmem website, 2023. URLhttps://www.upmem.com/technology/. → pages x, 2, 3, 28[25] A. Saulsbury, F. Pong, and A. Nowatzyk. Missing the memory wall: Thecase for processor/memory integration. SIGARCH Comput. Archit. News, 24(2):90–101, may 1996. ISSN 0163-5964. doi:10.1145/232974.232984.URL https://doi.org/10.1145/232974.232984. → page 2[26] Wikipedia contributors. Random access memory and memory wall, 2023.URL https://en.wikipedia.org/wiki/Random-access memory#Memory wall.[Online; accessed 09-July-2023]. → page 258
Thesis/Dissertation
eng
Electrical and Computer Engineering
Vancouver : University of British Columbia Library
Attribution-NonCommercial-NoDerivatives 4.0 International
Graduate
Fedorova, Alexandra
Offloading embedding lookups to processing-in-memory for deep learning recommender models
Text