Rafael Misoczki, PhD

Beauty of Imaginary numbers https://lnkd.in/drAycZ5Y #imaginarynumbers I think this is the best source to understand imaginary numbers and why they exists in depth.

3

Well, my day ended up taking a different path than I'd thought, yesterday! Instead of my primary research, I ended up down a literature review path. I'm sure by now you've heard that OpenAI announced it's recent model release called o1 so I thought I'd provide some interesting papers that I think might align with what is implemented within the reasoning module outlined in the o1 system card. If not, it's at least a foray into some newer methods being explored today. 😊 📄 o1 System Card: https://lnkd.in/gwSRs46w 📄 Large Language Monkeys: Scaling Inference Compute with Repeated Sampling: https://lnkd.in/gSTe43Bq 📄 ReFT: Reasoning with Reinforced Fine-Tuning : https://lnkd.in/gKPzfhfN 📄 Training Large Language Models for Reasoning through Reverse Curriculum Reinforcement Learning: https://lnkd.in/gqMy_dKu 📄 Reinforced Self-Training (ReST) for Language Modeling: https://lnkd.in/gTeaCqrM

21

3 Comments

Are LLMs capable of Artificial General Intelligence (AGI)? I came across this article in the NY Times: https://lnkd.in/geXXbi8B (for non-subscribers here's the paper it talks about https://lnkd.in/gP-m7sxA) The upshot is that researchers are finding strong evidence that language is a very distinct part of the human brain whose purpose is to communicate and that it is not intrinsically involved in reasoning. This gets at a very fundamental question about the use of LLMs as the basis for Artificial General Intelligence. Can a model based on language be used as the basis for AGI? Time will tell I'm sure, but when I've asked this question to researchers I know, they generally say yes. However, I'm not sure what the basis is for this belief. CAVEAT: I'm not in the AI field, nor have I delved deeply into the scholarly articles. I have a fairly rudimentary understanding of LLMs, but would be happy to learn from those who know more. One of the fundamental articles shaping my personal views on LLM based AI is Ted Chiang's excellent piece "ChatGPT is a blurry JPEG of the web" (https://lnkd.in/gmkRBe6A). For me, it captured the idea that LLMs are just summarizing what we know. To be sure, they are capable of really cool things, but are they really doing any thinking? More important are they even capable of thinking? Again, time will tell, but I'm increasingly skeptical. The kicker for me is that they have no common sense. One of the reasons we find LLMs so amusing is they crazy ways they go off the rails (aka hallucinations). I think that these are largely because there's no common sense 'boundary'. When a human's thoughts start to get carried away, we have ways of reigning them in by asking ourselves "is this actually a reasonable thought?" Indeed, it can be quite alarming to encounter people who communicate about ideas that we find to be outside the bounds of common reason. It feels like this 'framework of rationality' is very important but I don't think it's something you can just use as guard rails. It seems like you need some kind of other AI to be able to generate that framework. What do you think? Are LLMs capable of AGI? Or are they like our own language processing center, just a small piece of a much larger 'intelligence framework'? [Edited to use AGI instead of GenAI so as not to be confused with Generative AI] [Edit: Thanks to Tareq Alkhatib here is a direct link to the paper in question: https://lnkd.in/g9b7Q66c]

41

21 Comments

The latest advances in artificial intelligence (particularly large language models) continue to reverberate. Even for an “old school” AI person like myself (who cut his teeth with Prolog) it is clear that there has been a step change in our ability to create computer systems that can interact with humans using natural language. GPT-4 et al are exhibiting early signs of “common sense” and have encoded useful conceptual representations of the world. The debate rages on as to whether this is “intelligence”, but to an engineer like me, it sure seems useful! #LLM #AI #Intelligence #Programming https://lnkd.in/ebAgNapK

15

4 Comments

Improve your LLM training throughput by 20% and reduce memory usage by 60% with a single line of code change! Checkout the insightful engineering blog on Liger-Kernel developed by LinkedIn! Kudos Pin-Lun (Byron) Hsu, Yun Dai and team!

19

It is time to look forward to the summer of 2025! https://smrtr.io/npvQ- Staff Machine Learning Scientist, Intern - Summer 2025 at Visa, inc.   Unleash the world scientist within you this summer! Inspired by Richard P. Feynman's maxim, “You do any problem that you can, regardless of field,” we're inviting you to contribute your innovative skills in cross-industry advancement of science and humanity at VISA, the technology giant at the forefront of billions of global credit payments. We’re calling all physical scientists to venture into our world of ML/AI. The mission? To uncover fundamental truths hidden in the vast and largely untapped domain of Payment transactions, a realm as complex and intriguing as Physical interactions. Just as scientists construct effective models to make sense of real-world data, we strive to unearth elemental factors driving billions of transactions per day. With the rise of neural network-based deep learning in Physics, we're pioneering the use of these techniques to revolutionize global commerce.

121

1 Comment

In the era of LLMs, "sexy" might not be the term, but bag-of-words solutions are still kicking! When the position of a word matters, consider using Positional Vectorizer over TFIDF. Check it out: https://lnkd.in/dDRYFrAX

14

1 Comment

"There is another element in this scientific attitude: only that is knowable which is expressed (or, at least, can be expressed) in numbers. To get away from the so-called 'arbitrary and subjective', to escape ethical or literary judgments (which, as everyone knows, are trivial and unfounded), the scientist must get back to numbers. What, after all, can one hope to deduce from the purely qualitative statement that the worker is fatigued? But when biochemistry makes it possible to measure fatigability numerically, it is at last possible to take account of the worker’s fatigue. Then there is hope of finding a solution. However, an entire realm of effects of technique—indeed, the largest is not reducible to numbers. Yet, since what can be said about it is apparently not to be taken seriously, it is better for the scientist to shut his eyes and regard it as a realm of pseudo-problems, or simply as nonexistent. The 'scientific' position frequently consists of denying the existence of whatever does not belong to current scientific method." — Jacques Ellul, The Technological Society یعنی وہ سمجھ سکے تو آنسو، نہ سمجھ سکے تو پانی۔

2

The course did not cover how to deploy the trained model at the end, which limits its usefulness, especially given that large-scale training requires 24 hours (and substantial associated costs). https://lnkd.in/g2nS6hdP

3

Geoffrey Hinton's contributions is significant enough that I won't quibble with the category for his prize. (Nobel committee could create a special category just for him for all I care.) Widely (and rightfully) recognized as a leading expert in deep learning, his research touches everything from * Foundational concepts such as backprop https://lnkd.in/gMunbBzA * Foundational model architectures such as AlexNet https://lnkd.in/gq2JPJ6h * Optimization algorithms such as Momentum and RMSProp * Visualization techniques such as t-SNE https://lnkd.in/gjuw3vhA I don't think it's possible to overstate how profound the impact of machine learning and artificial intelligence will have on society, promising to unlock discoveries, improving productivity. Even ideas about what mastery of language or reasoning means, will need to be re-thought. Professor Hinton deserves all the credit for all of this.

31

I reached 17 years at Google two weeks ago. Google has changed quite a bit over the years, though I won't get into that now. Sometimes, though, it's not the big things you've landed, but the smaller things that unexpectedly turned into bigger things. I frequently reflect on: - Android lint testing for Google apps - Fake "Save" button in Google Docs - The algorithm for Google Calendar's event layout - A couple of random test frameworks I created along the way, e.g. one that repeatedly prevented new crashes in Android OS itself - Some novel algorithms for detecting production issues in Android - Helping land XSS-free autoescaping for one of our most common HTML templating languages Many of these were things I started on a whim because I thought they might be valuable. Some of them I landed myself, but some were just started and took on lives of their own. Of course, I tried a lot of other things that didn't work, but it's important to try and experiment even when not everything works. What small things bring _you_ disproportionate pride?

290

19 Comments

tmux <3

3

This week we are presenting our paper "LiNR: Model Based Neural Retrieval on GPUs at LinkedIn" accepted at CIKM 2024 (https://lnkd.in/gUrWqRcD). Please stop by and say hi to Aman Gupta, who will be there in person :) We discuss our experiences and challenges in creating scalable, differentiable search indexes using TensorFlow and PyTorch at production scale. In LiNR, both items and model weights are integrated into the model binary. Viewing index construction as a form of model training, we describe scaling our system for large indexes, incorporating full scans and efficient filtering. We believe LiNR represents one of the industry's first Live-updated model-based retrieval indexes at production scale. Talented co-authors include Fedor Borisyuk, Qingquan Song, Mingzhou Zhou, Ganesh Parameswaran, Madhulekha Arun, Siva P., Tugrul Bingol, Zhoutao Pei, Stanley(Kuang) Lee, Lu Z., Hugh Shao, Syed Ali Naqvi, Sen Zhou, Aman Gupta

169

4 Comments

Yep, biggest hurdle for effective use of LLMs is dealing with the hideous non-determinism. I.e., they are big fat liars! Liar liar pants on fire! Even when grounding an LLM with good examples or context, the same model with the same input and temperature 0 (assume top-k>1) can give different answers. Mixture of experts is key, or even repeated invocations by the same model as a jury! Check out Tom's work to see the cool stuff he's doing trying to get reliable answers out of these tricky beasts.

31

4 Comments

Trying to reduce costs of your LLMs by 2x? Use a small highly optimized "router" LLM to decide if you need a heavy or a light LLM based on the context. Simple questions go to the light LLM, tougher ones to the heavy LLM. Most surprising finding: the router LLM generalizes across different heavy and light LLMs -- no fine tuning required! (Also this is my first academic paper in 20 years -- it was a lot of fun working with Amjad Almahairi on this)

113

6 Comments

*Open Source AI is the Path Forward* Open source AI represents the world’s best shot at harnessing this technology to create the greatest economic opportunity and security for everyone. Meta's release of Llama 3.1 marks a significant milestone in advancing open source AI. The benchmark results are impressive! The gap between closed-source and open-source models is quickly vanishing. Check them out here: https://llama.meta.com/ If you feel like doing a bit more reading, this blog by Mark Zuckerberg is just a gem! https://lnkd.in/d48iBtiR

24

Let us relate to tensors ML folks in Physics, shall we?

2

How good are Trainium and Inference clusters for online serving of production traffic? Check this blog from Trishul Chilimbi on how Rufus responses are powered with high throughput and resiliency. #Trainium #Neuron #AWS Amazon Web Services (AWS)

6

This quote from Saint-Exupéry articulates the super-power of a good Amazon PRFAQ. Much has been written about what an Amazon PRFAQ ("Press Release / Frequently Asked Questions") is, so I won't rehash it in this post. I myself wrote https://lnkd.in/gGvxzRig. PRFAQs aren't always 100% realistic or 100% achievable. They often do not state the exact steps to get there. More often than not the final product ends up looking nothing like the original PRFAQ. But every great Amazon product started with a PRFAQ: Alexa, Kindle, Prime, your favorite AWS service. The reason this quote really resonated with me is because a good PRFAQ always gets an emotional response from your readers: 1️⃣ "I want this thing to exist!", and 2️⃣ "I personally want to be a part of bringing this thing into existence!" During my 12 years at Amazon, I've written a good amount of PRFAQs. Some have succeeded, some haven't directly succeeded but inspired others who did eventually succeed, some did not take off at all. Some were revolutionary, some were evolutionary. They all shared the fact that they articulated a vision for a more exciting future. If you want to build a ship, don't drum up the men to gather wood, divide the work and give orders. Instead, teach them to yearn for the vast and endless sea.

123

7 Comments