Joshua Cason’s Post

Here at Arcee Ai we're growing fast 🚀🚀🚀 Among a crew of new hires starting next week, we'd like to extend a special congratulations to Lucas Atkins and Fernando Fernandes Neto for the publication of this paper on which they are co-authors: "Spectrum: Targeted Training on Signal to Noise Ratio." The paper dives into how post-training of large language models (LLMs) is challenging due to high computational demands, and introduces Spectrum – a method that accelerates LLM training by selectively targeting layer modules based on their signal-to-noise ratio (SNR), freezing the rest. Using Random Matrix Theory and the Marchenko-Pastur distribution, Spectrum identifies and trains only the most informative layers. This approach matches the performance of full fine-tuning while reducing GPU memory usage and training time. Check out the full paper below, and join me in welcoming Lucas and Fernando to the Arcee AI research team 👏👏👏 https://hubs.li/Q02BRVPD0

#LLMs Will Always Hallucinate "...Structural Hallucination as an intrinsic nature of these systems. By establishing the mathematical certainty of #hallucinations, we challenge the prevailing notion that they can be fully mitigated." https://buff.ly/3XlV0Zk #GenerativeAI

LIBMoE: A Library for comprehensive benchmarking Mixture of Experts in Large Language Models. It is a modular open-source framework to streamline the research, training, and evaluation of MoE algorithms. Built upon three core principles: (i) modular design (ii) efficient training (iii) comprehensive evaluation. The paper benchmarks five state-of-the-art MoE algorithms over three different LLMs and 11 datasets under the zero-shot setting. #MachineLearning #GenAI #LLM #MoE #DataScience

In this episode, we discuss Scaling Smart: Accelerating Large Language Model Pre-training with Small Model Initialization by Mohammad Samragh, Iman Mirzadeh, Keivan Alizadeh Vahid, Fartash Faghri, Minsik Cho, Moin Nabi, Devang Naik, Mehrdad Farajtabar. The paper presents HyperCloning, a technique for initializing large language models with smaller, pre-trained models to leverage their predictive power. This method allows large models to require less training time and fewer GPU hours by scaling up small models while preserving their functionalities. HyperCloning offers a viable solution to efficiently manage the high costs and time investments in training large language models.

Put simply, the three libraries— #LILO (library induction from language observations), #Ada (action domain acquisition), and #LGA (language-guided abstraction)—all work to provide human-like reasoning across certain functions, such as computer programming, task planning, and robotic tasks. Using the neurosymbolic method baked into LILO, MIT uses its Stitch ( get it? ) algorithm to identify abstractions. This allows #LLMs to apply commonsense knowledge with sophistication that...

I NEVER USED LLMS REASONS 1. they cripple your own ability to express yourself 2. they make you lazy, so one does not do any research anymore 3. they slowly but surely makes you complacent, as you do not analyze information & summarize your thoughts METAPHOR As a comparison, it is nice to have a car and drive everywhere, there is no need to walk, let alone run. But then again, you get fat while eating at the same rate, so you have an unhealthy lifestyle and slowly get sick. EPILOGUE "There is nothing like writing to force you to think and get your thoughts straight." Warren Buffett LINKS LLMS IMPLY DECEPTION TO PLEASE, WHICH IS THE SAME AS SEX FOR MONEY - AS LONG AS HE PAYS, GIVE THE CUSTOMER WHAT HE WANTS REGARDLESS OF THE CONSEQUENCES https://lnkd.in/gKtnjbEh WHAT DO YOU THINK CHATGPT REALLY IS - NOTHING BUT A RECREATIONAL DRUG USE https://lnkd.in/d2w5QSi6 EVERY LANGUAGE IS ACTUALLY LOGIC - DESCRIBING THE REPRESENTATION , ORGANIZATION & FLOW OF INFORMATION IN OUR MIND, SOMETIMES WITH FEEDBACK https://lnkd.in/gPrSqRUH

#ECCV2024 New paper alert: We introduce DECIDER to detect failures in vision models using priors from LLMs and VLMs. DECIDER produces SoTA results under covariate shifts, style changes, spurious correlations and class imbalance. Preprint & codes coming out soon! DECIDER leverages large language models (LLMs) to identify the essential attributes relevant to a task. It then constructs a debiased classifier by aligning its features with these attributes using a vision-language model (VLM). DECIDER detects failures by comparing the predictions from the original and debiased models. Joint work with Kowshik Thopalli Rakshith Subramanyam Vivek Sivaraman Narayanaswamy Lawrence Livermore National Laboratory

MIT researchers found a way to reduce latency in Large Language Models by utilizing the L-Mul algorithm which performs integer addition rather than the hectic process of multiplying floating point numbers. The numbers or values are instead converted into fixed point numbers or scaled by a 100. E.g. two floating point numbers like 2.5 or 1.5 are scaled by 100 to become 250 and 150. Integer addition is performed, and the numbers are scale downwards again by the square of the number they were scaled by which is 100. Hence (10000). It is said to achieve precise results almost close to that of the traditional floating-point multiplication algorithms. L-Mul with 4-bit mantissa achieves comparable precision as float8_e4m3 multiplications, and L-Mul with 3-bit mantissa outperforms float8_e5m2. This process is mostly required in scenarios were speed trumps precision. #tech #LargeLanguageModels #Transformers #MachineLearning #DeepLearning

by removing the obfuscation layer of mathematical notation and language presentation of problems and translating it into code form that is more prevalent and condusive to train models on. models are finally able to use math to solve math. https://lnkd.in/dQQc48Ni

📢 Exciting update! Learn about achieving sparse activation in Small Language Models without retraining efforts in this new blog post. The post demonstrates the challenges and solutions for applying sparse activation to SLMs and how it can significantly reduce computing costs. Check out the full details and experimental results here: https://bit.ly/4elx3sU #LanguageModels #SparseActivation #MachineLearning