Atanu Ghosh

Hot off the press just minutes ago, Confluent just announced Freight clusters in #KafkaSummit Bangalore. They’re a new innovative cluster type that lowers prices by up to 90% (!) 🔥 What is it and how does it work? It’s all about latency. Some workloads like: 🔷telemetry 🔷metrics 🔷logs don’t care whether the latency is 50ms, 500ms or 2s. But the same workloads typically have a TON of data. And when you need to replicate that data 3x and pipe it through the network… the network costs can eat you alive 💀 So what did #CFLT do? They decided to write the data straight to S3! Instead of having Kafka replicate it across brokers, they just write it directly to S3. S3 itself does the replication under the hood (for much cheaper) 💸 The catch? S3 is much slower. But as we mentioned, those workloads don’t care. Similar to a cargo ship, you’re fine letting the cargo (data) take longer to ship as long as the price is good. For the critical use cases - you have planes. ✈️ For everything else - you have Freight. 🛳️

600

36 Comments

What if your queries required access to multiple sub-related data from different documents, for an LLM to parse and make sense of before returning a coherent and accurate output? We often ask complicated questions, right? 🔹Well to solve this problem, these researchers released Multi-Head RAG (MRAG) which leverages activations of Transformer’s multi-head attention layer, instead of the decoder layer, as keys for fetching multi-aspect documents. They are taking advantage of the fact that multi-head attention can learn to capture different data aspects. 🔹The interesting part is that MRAG is both cost-effective and energy-efficient. In other words, it does not require additional LLM queries, multiple model instances, increased storage, or multiple inference passes over the embedding model. 🔹Moreover, MRAG can be seamlessly integrated with existing RAG frameworks and benchmarking tools like RAGAS as well as different classes of data stores. 🔹The researchers provide a comprehensive evaluation methodology, including specific metrics, synthetic datasets, and real world use cases to demonstrate MRAG’s effectiveness. 🔸Do check it out! #genai #technology #artificialintelligence Click #linkedangle and follow for more content Set the 🔔 notification on my page, don’t miss a post!

181

10 Comments

Here’s the #1 mistake EB-1A aspirants in tech make. ⚠️ They go outside-in versus inside-out. What does that even mean? It means they look externally for how other people built EB-1A profiles as a guide to how they should build their profile. I get it - if you don’t know where to start, makes sense to do what other successful applicants do, right? Wrong. This approach is breeding a generation of EB-1A’s that all look the same. 🐑 The whole point of the EB-1A process is to show why YOU are UNIQUELY recognized as among the best in your field. 💯 So how does it help to get all the same judging, awards, and media that everyone else is getting? ✅ You are supposed to stand out from the crowd, and show the impact of the work YOU are doing within the EB-1A framework. Plus, in tech, everyone has a different story to tell. How does it help to use someone else’s approach to tell your story? These EB-1A’s come across as fake - and you know who hates fakers more than anyone else? 👮‍♂️ USCIS. And when they see these applications they will profile you as a faker and completely disregard the parts that are deserving. We know because we (and many of our lawyer colleagues) are now being asked to help scores of tech workers who got their EB-1A’s denied after filling up their profiles with unimpressive awards and marketing exposes masquerading as media articles in 3rd rate publications. Question: since when do awesome tech employees ever get recognition of their expertise in the press? They don’t. Their employers do. So it looks weird that a generation of employees are now suddenly getting press coverage. ✅ Instead of an outside-in approach, go inside-out. First identify your unique impact proposition. What is your narrative? How are you uniquely amazing? Because if you are a senior employee at an innovative company, chances are you doing some incredible, impactful EB-1A-worthy work! Then, build your case from there in a way that honors that narrative. This is the formula we use in our course, and then in our law firm, that leads to a ~99% approval rate. Don’t be tempted to fake it! #immigration #eb1a #globalmobility -------- Do you work in high-tech, are stuck in long green card backlogs, and are looking to level up to an EB-1A? Our next attorney-led live EB-1A profile-building cohort begins on June 8. Apply now to secure your spot! The link is in the comments below.

55

6 Comments

Why? While discussing system design for both trading apps and stock exchanges, I strongly emphasize that they are weak cases for Cloud. While Cloud is really good for scaling and availability, it comes with a tradeoff for Latency. While temporary 300 ms+ latency is OK for most of the apps, a trader can lose a lot of money in that time frame. I am still of the belief that a broker should run its app as close to the stock exchange as possible and latency should trade off for availability . If a person is not able to place order at the. first step, it's still better than seeing order in "Pending" state and then getting executed while stock price has gone down already. https://lnkd.in/d6N9Rp6G

3

Got a question on why not use llama or open source models which are free rather paying for APIs. This was in response to my pricing war post of LLM providers Nothing is free in this world llama deployment needs infra and to scale needs decent infra setup. Need infra engineers to maintain and run it Need monitoring tools and SRE for continous availability. Electricity cost, data center cost, hardware cost etc. If running on cloud then cloud resources cost are not cheap While open source models has its advantages does not mean they come for cheap. It goes on to the same old debate of Opex vs Capex apart from other factors like performance, maintainace, support etc Over time depending on ROI of use case one can break even faster but the selection also boils down to if it meets your use case needs #artificialintelligence #generativeai

47

6 Comments

Brilliant write-up of another MOVEit SFTP server bug. The essential idea behind this multi-pronged exploit: a) Upload a public key to the target SFTP server to a known location. How? Simple: Exploit server logging :-) The researchers found that the server logged copiously. So they found an unauthenticated endpoint meant for guest users and supplied the key as payload. The server dutifully logged the key contents into the log file. Now because key files MUST begin with the ----- BEGIN PUBLIC KEY ----- text, the researchers needed to find a key format that the SFTP server accepted that would accept any file and strip out the contents until it found a valid key. They found it in the PPK format (Putty keys). So they converted their key to this format, then invoked the endpoint with the PPK key as payload and voila their public key is now resident in a log file on the target server. b) Now authenticate as a user on the target server except that you provide the path to the log file containing the public key in the previous step. Sign this request with the same key.  c) This authentication will succeed despite attacker not knowing the keys of the user being impersonated. This occurs because of a bug in how the authentication is handled and the interplay with a lower level SSH library being used. The researchers seem to imply that code reviews may have caught this bug. But it's not so simple. Fuzz testing would indeed have caught this and certainly code coverage driven testing because an error in one code path was exploited to trigger a bug in another. A final point: It's hard to see why the entire payload needs to be logged, or even why sensitive data like keys (even if public keys) are not masked.  Read this excellent write-up - link in comments - and get ideas on the ways attackers think - as better ways to secure your own code.

2

1 Comment

AWS S3 Tables make me remember all the times we tried to push database functionality into the storage layer, cutting layering and ending up with not-so-great outcomes. (Yes, I would say the same if S3 Tables were also about Hudi because this is a technical opinion.) 1️⃣ Does it eliminate manual table maintenance? This sort of "storage engine" functionality is out-of-box with Apache Hudi on Amazon Web Services (AWS) #EMR (search for EMR on Hudi Github) and even Delta Lake, for that matter, with optimized writes. You can then read them as Apache Iceberg using Apache XTable (Incubating). So, s3 tables are an iceberg solution to an iceberg problem. It's getting a little boring 🥱 to see repeated new product launches centered on the same small file handling. How-to blogs right from AWS on better alternatives: ⏩ Clustering/Hudi/EMR from 2021 : https://lnkd.in/gqAtg8AV ⏩ XTable blog from last week: https://lnkd.in/gn5V2rih 2️⃣ Is it so much more cheaper? Is it? Let's do some math. Happy to be proven wrong. Based on recent benchmarks, for clustering/compacting a 100MB file with 4 2MB files, Hudi OSS took 17 secs on m5.xlarge with one thread. Let's round that up to 20 seconds (Delta/Iceberg spark may also be comparable). That means we need roughly: (20 seconds x 5 times more data)/4 times parallelism = 25 seconds of compute costs. So running the same on EMR with m5.xlarge will cost you 25 seconds x ($0.192/hour / 3600 seconds)= 0.1 cents per GB. S3 Tables compaction is 5 cents per GB. It's about 50x more expensive. Is that a reasonable cost to pay for not setting a few configs? (I am happy for all of us - AWS users - if this forces a price drop). 3️⃣ Most of all, this is a new lock-in point This is a step backward for #open #data #architecture. Tying data management and seemingly catalog functionality tightly to storage could prevent even large established vendors like #Snowflake or #Databricks from innovating faster on top of S3. For example, the table maintenance format compatibility is not prioritized to work across non-aws vendors by what's primarily a cloud storage team. These factors affect how fast we can move as an industry. Users (and we, as a small innovative vendor) have little power to make the big players here play ball with each other. For example, it took a decade of "lobbying" from very large tech companies to get Parquet adopted broadly. Instead, I would be much more delighted if the S3 team gave us all multi-range gets or http/2 (worlds already onto http/3!). If you need help running this on EMR, as the post advocates, please ping us on Hudi Slack or DM me. I'm happy to help for fun or profit. #awsreinvent is as fun as ever! Have a great time there! #reinvent2024 #aws #bigdata #analytics #AI #dataengineering

381

12 Comments

🔍 Why Chunking Strategy is Key in Building a RAG System with LLMs 🔍 Recently, a student of mine while attending the interview was asked to explain how he would chose chunking strategy to build a better RAG system. When building a Retrieval-Augmented Generation (RAG) system using LLMs, chunking is a critical step. The process of splitting data into manageable pieces not only affects how well relevant information is retrieved but also impacts the LLM’s ability to generate coherent and accurate responses. • Contextual Understanding: LLMs have limitations in processing extremely long texts. Chunking helps maintain context and ensures the model can focus on relevant information. If you are using Opensource LLM with limitation on context length, this becomes important.  • Efficiency: Smaller chunks reduce the computational burden on the LLM, leading to faster responses. • Retrieval Accuracy: Chunking can improve the accuracy of information retrieval by making it easier to find relevant segments. Essentially, Chunks should not be too large or too small. So, what’s the right answer ? Well, it depends on the problem you are solving. First you should ask below questions to yourselves. • What is the nature of the content being indexed? Long documents, Short sentences, Paragraphs ? • Which embedding model are you using, and what chunk sizes does it perform optimally on? • What are your expectations for the length and complexity of user queries? Will they be short and specific or long and complex? This may inform the way you choose to chunk • How will the retrieved results be utilized within your specific application? For example, will they be used for semantic search, question answering, summarization, or other purposes? Then chose one of the below strategies: 𝗖𝗼𝗺𝗺𝗼𝗻 𝗖𝗵𝘂𝗻𝗸𝗶𝗻𝗴 𝗦𝘁𝗿𝗮𝘁𝗲𝗴𝗶𝗲𝘀: Sentence-Based Chunking Paragraph-Based Chunking Token-Based Chunking Overlapping Chunking (Sliding Window) Topic-Based Chunking: Identify topic boundaries using techniques like topic modeling or keyword extraction. Specialized chunking : You would need specialised packages for chunking if you are using LaTex or Markdown etc. Take that into consideration. Semantic Chunking : Variable length chunking retaining the semantic coherence. Creating Sentence groups etc. Langchain has a Semantic Chunking splitter. Hybrid Chunking: Combine multiple strategies to optimize for your specific use case. Choosing the Right Strategy: The best chunking strategy depends on factors like the structure of your documents, the complexity of the information, and the desired level of granularity. Experimentation is often key to finding the optimal approach. By carefully considering chunking strategies, you can significantly enhance the performance and effectiveness of your RAG system.

7

1 Comment

Our trajectory with Google Cloud epitomizes a transformative strategic pivot. What began as a challenging market positioning has evolved into a compelling narrative of operational resilience and technological leadership. Over seven pivotal years, we've fundamentally reimagined our enterprise value proposition—transitioning from a peripheral cloud solution to a mission-critical AI innovation partner. The financial metrics tell a powerful story: we've systematically converted recurring quarterly deficits into a robust framework for sustainable profitability. Buckle up the next 7 years are going to be epic @ Google Cloud! 🚀 🚀 🚀 🚀 🚀 🚀 🚀 🚀 🚀 🚀 🚀 🚀 🚀 🚀 🚀 🚀 🚀 🚀 🚀 🚀 🚀 🚀 🚀 🚀 🚀 🚀 #google #googlecloud #GCP #gcpcloud

30

1 Comment

AWS has released Q, a generative AI coding assistant to speed the software development process. This includes Q Apps, which sound similar to Open AIs GPTs, allowing users with no coding experience to build custom apps. #aws #ai #software ##ml #developersdevelopersdevelopers

17

Pub/Sub and queue look the same when you hear it for the first time.😱 Here is a simple explanation of what they are and the differences between them. In the illustration depicted below, 🔴 Messaging Queue: People come to the mall to shop. Before they check out, they must wait in the queue to complete the payment. Here, the mall is like a producer, producing people ( who have shopped and are ready to complete payment) who are sent to the queue to complete the payment. One by one, each person completes payment at the checkout counter. The checkout counter is like a consumer. As the queue grows ( more people join the line), more than one checkout ( one consumer ) is needed, and we will need to add more checkout counters ( more consumers). We have Producers, Queues, and Consumers. 🔴 Pub / Sub - Extending on the above example, after the checkout is completed, this customer data might be required by various services. For instance, in the image below, we have a real-time service that calculates how many people have completed the checkout. Hence, the checkout service has subscribed to receive this data. Likewise, marketing and inventory services might subscribe to receive this data. We have Producers, Topics, and Subscribers. 𝗣𝘂𝗯/𝘀𝘂𝗯:⏹️ The subscribers don’t have to reveal anything to the publishers. Messages are in the topic even after multiple subscribers consume it. Messages are sent to multiple subscribers simultaneously and asynchronously Message ordering and sending to various consumers is not guaranteed Producers and consumers have loose coupling 𝗤𝘂𝗲𝘂𝗲: ⏹️ Queues need to reveal who they are sending messages to Producers would put the messages in the queue, storing them until one of the consumers consumes them Producers put the messages in the queue, typically sent using some ordering and sequentially to a single receiver Message ordering and sending to consumers is guaranteed Producers and consumers have some sort of relationship #engineering #systemdesign #interview #productdevelopment #eventdrivenarchitecture

66

16 Comments

Pub/Sub and queue look the same when you hear it for the first time.😱 Here is a simple explanation of what they are and the differences between them. In the illustration depicted below, 🔴 Messaging Queue: People come to the mall to shop. Before they check out, they must wait in the queue to complete the payment. Here, the mall is like a producer, producing people ( who have shopped and are ready to complete payment) who are sent to the queue to complete the payment. One by one, each person completes payment at the checkout counter. The checkout counter is like a consumer. As the queue grows ( more people join the line), more than one checkout ( one consumer ) is needed, and we will need to add more checkout counters ( more consumers). We have Producers, Queues, and Consumers. 🔴 Pub / Sub - Extending on the above example, after the checkout is completed, this customer data might be required by various services. For instance, in the image below, we have a real-time service that calculates how many people have completed the checkout. Hence, the checkout service has subscribed to receive this data. Likewise, marketing and inventory services might subscribe to receive this data. We have Producers, Topics, and Subscribers. 𝗣𝘂𝗯/𝘀𝘂𝗯:⏹️ The subscribers don’t have to reveal anything to the publishers. Messages are in the topic even after multiple subscribers consume it. Messages are sent to multiple subscribers simultaneously and asynchronously Message ordering and sending to various consumers is not guaranteed Producers and consumers have loose coupling 𝗤𝘂𝗲𝘂𝗲: ⏹️ Queues need to reveal who they are sending messages to Producers would put the messages in the queue, storing them until one of the consumers consumes them Producers put the messages in the queue, typically sent using some ordering and sequentially to a single receiver Message ordering and sending to consumers is guaranteed Producers and consumers have some sort of relationship #engineering #systemdesign #interview #productdevelopment #eventdrivenarchitecture

61

10 Comments

AMAZON ACCOUNTING GAMES? What’s an easy way to boost your quarterly margins if you are binging on GPU chips? Depreciate your assets over 6 years instead of 5 Here’s my take in The Information: ‘For AWS, though, expectations are sky-high.  Ahluwalia noted that of all the “Magnificent Seven” stocks – Microsoft, Apple, Nvidia, Alphabet, Amazon, Meta and Tesla – Wall Street analyst projections are highest for Amazon in terms of its ability to grow its free cash flow.‘ Remember, firms that spend a lot on CapEx lag those that do not, on average.  Now, why do I see this as an accounting game? New GPU stacks are more efficient.  They increase obsolescence rapidly.  Amazon should *reduce* its depreciation schedule! The CapEx binge is a core part of the Lumida Wealth Semiconductor CapEx Receiver Thesis.  Recall we said the best names to own in Mag 7 are: #NVDA #META #GOOGL Those are exactly the names that are leading YTD with #TSLA and #AAPL on the bottom and Consensus names like #MSFT and #AMZN in the middle

3

Last week, I read a paper by Amazon on AWS Lambda and how they spin up 15,000 containers per second with a cold-start time of 50ms; and the three things I found most amusing are - block-level demand loading - deduplication of blocks with convergent encryption - use of erasure coding in caching for optimizing tail latencies If you want to know the details, I have a quick write-up explaining the paper along with my notes, and a quick TLDR. give it a read - https://lnkd.in/gWNx3Ek8 I have a habit of taking notes while reading papers. It has always helped me build a better understanding. To date, all of them were private in my obsidian. I have now started posting them as blogs on my website. Slowly, I will do it for all the papers in my arpitbhayani.me/papershelf and the new ones that I am reading. I absolutely love to jot down my thoughts, why not just make them public :) easy for everyone who wants to follow. #AsliEngineering #SystemDesign

718

11 Comments

How to harness AI to fuel your startup’s growth—expert tips from Google's Cloud Expert - (VJ) Vijaykumar Jangamashetti ☁️ Vijay was part of the team at Google AI Accelerator program when we first met there and we had an amazing conversation there. I learned a lot. Yesterday, we had a blast with him again on our first webinar of the series “Everything AI”. Here are some key takeaways from the session I’d like to share with you all: 🔹 What is AI really?  AI isn't magic, but it’s an umbrella of technologies mimicking human intelligence. Neural networks, Machine Learning (ML), Natural Language Processing (NLP), Deep Learning (DL), and even Large Language Models (LLMs) all have specific use cases that revolutionize industries by automating tasks and improving decision-making. 🔹 AI Hype vs. Reality  The hype says AI solves everything—it doesn’t. But it can drive monumental change by boosting efficiency and simplifying complex processes. Learning what to automate is key! 🔹 Generative AI: The Game Changer With Generative AI like Google’s Vertex Garden Model and AI Agent Model, startups can now create, simplify, and even predict future trends. AI is not just about automation anymore—isn’t it? Now, with the help of AI, businesses dig deep into their data, turning it into valuable insights that drive smarter decisions. Even startups across industries are tapping into AI for automating customer interactions, improving personalization, and streamlining processes. Whether it's healthcare, retail, or manufacturing, AI offers endless possibilities. If you do not want to miss the next webinars we are planning, then join our AI community on WhatsApp with the link in comments. AI isn’t the future—it’s NOW. 🔥 #AIRevolution #GoogleCloud #Startups #Innovation #GenerativeAI

148

14 Comments

Which tools powered by LLMs do we pay for at SigNoz today ? - ChatGPT Pro - Claude API - Github CoPilot The LLM based tools which our team is "actually using" today is using LLMs to improve internal processes - to come with new ideas for content - build small tools for internal process - to push our thinking and act as a sparring partner And if I think about it now, making LLM based outputs to show to our customers has a higher bar, as it is only "approximately correct" But as an internal tooling, it works great - as we have higher tolerance for errors there. A possible outcome I see of this is orgs running more efficiently. Leaner teams achieving much more than what they used to. What are you seeing about use cases which are actually "live" in your workplaces using LLMs?

43

6 Comments