Dr. Mark T. Maybury’s Post

In the LLM world size matters but very good results have been achieved with a proper use of a SML such as Phi-2 to create "expert adapters" as fine tuned models. Based on the user query, the most suitable adapter is selected. A smart and very energy-efficient approach. https://lnkd.in/d_Meru3U

Game on for LLM choice. I expect the market for base LLMs to commodify further in the near future, though we will also continue to see sparks of non-incremental change. Tx again to Jasper Schelling for the tip.

Our next keynote covers The State of LLM Agents.

📘 Configuring and Using the LLM Prompt Component in SmythOS: A Complete Guide The LLM Prompt component in SmythOS is revolutionizing how we generate content through AI. Here's a comprehensive breakdown of its capabilities: 🔧 Model Configuration • Default Models: Full OpenAI suite (GPT 3.5, GPT 4) • Custom Models: Seamless integration with Together AI and Claude AI • API Integration: Bring your own keys for maximum flexibility ⚙️ Prompt Settings & Controls • Dynamic prompt configuration with input variables • Temperature control (default: 1) • Top P settings for response breadth • Maximum output tokens customization • Stop sequence definition • Frequency and presence penalties for reduced repetition 🚀 Advanced Customization Options Create custom models with: • Amazon's Bedrock • Google's Vertex AI • Full machine learning feature customization • Credential management options 💡 Practical Implementation Example: Generating Personalized Emails: 1. Configure name and email inputs 2. Set up detailed Sales department prompts 3. Utilize debug mode for JSON output review 4. Implement expressions for content sectioning 🔗 Essential Resources: Documentation: https://lnkd.in/eu2SvfNH Training: https://lnkd.in/eCehmk4K Community Support: Join our Discord at discord.gg/smythos For developers seeking robust language modeling integration, the LLM Prompt component offers unparalleled configurability and extensive customization support.

Best Practices for Deploying LLM Inference, RAG and Fine Tuning Pipelines on K8s   🎯 Key Innovations:   - Advanced management of large language models (LLMs) lifecycle on Kubernetes.   - Use of inference servers for seamless deployment and auto-scaling of models.   - Integration of retrieval-augmented generation (RAG) with embeddings and vector databases.   💡 Notable Features:   - Customized inference pipelines utilizing NVIDIA's Nim operator and KServe.   - Efficient scheduling techniques for GPU resources with dynamic resource allocation.   - Enhanced security through role-based access control (RBAC) and monitoring capabilities.   🛠️ Perfect for:   - AI/ML Engineers deploying models in production.   - Data Scientists involved in fine-tuning and inference tasks.   - DevOps teams managing cloud-native applications on Kubernetes.   ⚡️ Impact:   - Reduced inference latency via effective model caching techniques.   - Improved GPU utilization optimizing resource allocation and scheduling.   - Increased security and manageability of AI pipelines in enterprise settings.   🔍 Preview of the Talk: In this insightful session, Meenakshi Kaushik and Shiva Krishna Merla from NVIDIA share comprehensive best practices for deploying and managing LLM inference pipelines on Kubernetes. They delve into critical challenges such as minimizing inference latency, optimizing GPU usage, and enhancing security measures. Attendees gain actionable insights on building customizable pipelines and leveraging NVIDIA’s technology stack to ensure efficient model management, ultimately leading to significant performance improvements. For more details, check out the full session here: https://lnkd.in/gRK7zPTM

What would you do if LLM generation became arbitrarily cheap and fast? Groq has already made a huge leap in LLM inference speed, but a lot of other research on improving LLM speeds has come out recently leaving me to wonder how fast things could possibly get? It will take a while before we see all of these advancements make their way into a SOTA model, with the exception of Groq they are architectural changes to the model that need to be made before training starts. But just for fun, let's see how fast we would get if their benefits were multiplicative. Groq does 300 tokens per second per user on Llama-2 70B, we can use that as a baseline. The 1-bit LLM paper claims 8.9x faster throughput on 2x A100s. Jamba claims 3x throughput on longer contexts compared to similar sized models. It isn't clear on whether this will work with Groq or the 1-bit paper, but just for fun I'm going to count it. There are other improvements out there as well, but already we are at 8,010 tokens per second. That's an average length full novel (~100k tokens) in 12.5 seconds. So again, what would you do if LLM generation became arbitrarily cheap and fast? Let me know in the comments. https://www.ai21.com/jamba https://lnkd.in/eerJiDQU https://lnkd.in/e53d2XHD

👩⚕️ Do users care for LLM benchmarks? if you ever find yourself struggling to articulate 🤔 a central point - its probably because you haven't formed enough conviction around that point (yet). This uncomfortable feeling is a powerful driver to write, to clarify, to mold and shape the central point. Spending time here is high-quality #leveraged work. but once you emerge with a clear winning 🏆 thought - it pulls you, and with it others around you. The Katanemo team has been laser focused on human-centric LLM research and development, and I was struggling to capture the essence of our next experiment - but tonight's 2-page memo has offered much clarity. The scaling laws of LLMs (attention-based models at least) state that the larger the model, the better the quality. But at what point (and for what task) do user see or care for the 2% improvement in benchmark performance that these vendors are chasing. I am almost done designing the experiment and will be running some preliminary tests with a small group of people. If you are interested and want to participate in this internet-scale study - to bring a focus back on human-centric LLM research and to reduce the waste (now measured in Billions of GPUs) shoot me a message or comment below. I'll give you an early access pass and would love for early feedback as we harden out the experiment a bit more before publicly making it available!

fyi pls do ur own research as well

https://lnkd.in/eUXuRMDQ 5 Key Points to Unlock LLM Quantization

Dive into the "Context Window Paradox" 🔍 Every foundational LLM company – OpenAI, Google, Anthropic, and more – proudly touts the “size of their context.” But what does that really mean? 🤔 In our latest video, we unravel the mystery behind context windows and uncover a surprising twist: as context windows increase, performance can actually decline. 😮 Curious to know why bigger isn’t always better? Join us for a quick yet enlightening exploration into the world of LLMs and the paradox that challenges the norm. 🎥 Watch now and unlock the secrets: https://lnkd.in/gC7vGmAv