Nikesh Arora

Challenges of AI Networking for hyperscalers and enterprises Sept 27, 2024 I had the opportunity to meet with Manohar Paluri, VP of AI at Meta, this evening at the AI Tech Leaders Club, just under 24 hours before the AI Infra Summit on September 27, 2024. It’s been confirmed that the large training cluster at 500K GPUs is the next frontier for the hyperscaler industry. Meanwhile, on the enterprise side, standalone cluster sizes typically range from 20K to much smaller setups for retraining and inference. From my experience, while the challenges in these two sectors are intertwined, they can still be stated separately. 1) Checkpointing Challenges: Large GPU clusters require numerous checkpoints to maintain system reliability. The industry has recently implemented networking and checkpointing solutions for clusters with around 20K GPUs. Imagine the challenges we’ll face at the scale of 500K GPUs! 2) GPU Utilization: Typical GPU utilization hovers around 20%. While hyperscalers may achieve better numbers, overall utilization remains low. For this, a startup came up a clever solution to power down the idle GPUs dynamically to save up to 60% of power consumption from a GPU cluster. This approach could either reduce power usage from idle GPUs or, in conjunction with increased GPU utilization through further resources sharing���like memory, network, and model partitioning/parallelization—enhance overall efficiency. 3) Deployment Flexibility: The deployment of data center networks (hyperscaler cloud) or private networks (on-premises) for enterprises often lacks the necessary flexibility. For more insight, I recommend reading the justification for Google’s Optical Circuit Switching (OCS) network. Following conventional practices without innovative solutions can lead to over-investment in AI and cloud infrastructure. 4) ROI Optimization: Classic cloud infrastructure is often not optimized for ROI in AI networking, which leads to operational challenges for enterprises. To understand this, a little involved discussion is needed. Once understood, it becomes part of your instinct. Consider the competitive disadvantage faced by businesses that optimize ROI in their AI networks. Free Takeaway from me: Conversations about the challenges of AI infrastructure and networking are very welcome at the AI Infra Summit, and participants will receive the latest AI networking problem slide! 😊

12

5 Comments

As companies go multi-cloud in their deployments, we need stand alone MLOps/LLMOps solutions that can deployed across vs being dependent on any one platform. As developers use multiple code generators, we will need security solutions that go across all of them to support developers. As businesses use multiple LLMs/AI models/APIs across the enterprise, we will need stand alone solutions to help identify, implement, and secure data and use cases. These are great opportunities to build next gen independent large software companies.

48

6 Comments

Here are my key takeaways from the second panel at the ”Resilience in the Ever-Changing Environment" conference at Nasdaq in #NY moderated by Kris Pederson who is leading the EY Center for Board Matters, featured industry luminaries like Bhawna Singh, Okta CTO, Mike O'Rourke, Head of Nasdaq AI & Emerging Technology, and Usman Sheikh; Partner & Chair of the Blockchain & FinTech Practice At Baker McKenzie They discussed product differentiation and M&A strategy in relations with emerging technologies such as #GenAI, #QuantumComputing, #Blockchain, #EdgeComputing; shaping the future of business and society, highlighting the importance for boards to be well-versed in such areas and to ask timely questions to ensure management has been considering these topics and making informed decisions. Boards should consider how generative AI can enhance productivity, improve customer experiences, and drive innovation, while ensuring it is used ethically. Boards should help establish ethical guardrails for AI usage, promote transparency, and address potential biases. They should collaborate with stakeholders to ensure AI aligns with societal values and ethical standards. Additionally, risk management around AI should include the following: AI Laws: Boards need to stay informed and proactive about the evolving legal landscape surrounding AI. This includes understanding compliance requirements, anticipating regulatory changes, and engaging with policymakers. Active engagement can help shape policies that balance innovation with ethical considerations. Ethical and Secure AI Deployment: As AI becomes more integral to business operations, ensuring its ethical use and securing AI systems are paramount. Companies should extend security measures to their entire data platform and pipeline and adhere to high standards of ethical AI practices, possibly under the guidance of specially formed committees. AI Innovation in the Wrong Hands: There is a significant risk of AI technology being misused by malicious actors. Boards should implement robust ethical guidelines to ensure their AI innovations are not misappropriated or used unethically. This involves careful vetting of partners and clients, as well as ongoing monitoring of AI deployments to detect and prevent misuse. Attack Protection / AI Security: As AI systems become more integrated into business operations, they become increasingly attractive targets for cyberattacks. Boards must prioritize AI security by investing in advanced security measures and conducting regular audits. This includes safeguarding AI algorithms, ensuring data integrity, and protecting the underlying infrastructure to prevent unauthorized access and attacks. #NasdaqResilience #TechInnovation #BoardGovernance #GenerativeAI #QuantumComputing #Blockchain #EdgeComputing #AIEthics #AIRegulation #BoardLeadership #BoardOversight #BoardResponsibilities #BoardOfDirectors

52

4 Comments

Microsoft’s #AIstudio hugs Nasuni’s file storage so their customers can natively unlock business insights from their global #unstructuredfile data. This is possible as Nasuni’s #cloud-native global file system inherently consolidates customer file data on near infinite #storage. This is just one of many #filesservices you can now activate. I love #investing in #extraordinary #founders!

5

So great to speak to Mohammed Islam on The Mo Show podcast last week! Watch the full episode here!

300

12 Comments

Looking forward to meeting women entrepreneurs who are on their journey to create some pathbreaking ideas and inspiring others to do the same. #advaitawomenentrepreneurs #womeninbusiness #entrepreneurship #womenentrepreneurs #EmpowerWomen #womenleaders #businesswomen

75

6 Comments

Key Insight #3 from Pear VC GTM Conference: "Founder Mistakes" Panel by Joseph Elias Phillips (Payabli), Jon Noronha (Gamma), Samuel Adeyemo (Aurora Solar) 1) Founder-led Sales is Essential: All three founders emphasized the importance of founders being directly involved in early sales. Jo highlighted how crucial it is to personally engage with customers, as no one can sell the vision like the founder. Jon echoed this, stating that "sales is 80-90% listening," which aligns perfectly with founder-led sales, as it helps founders identify customer needs better. 2) Build vs. Sell Dilemma: The common advice was to “sell before you build,” and all had nuanced takes. Jo shared that Payabli’s initial sales were driven by using legacy tech and reselling other people's platforms, which allowed them to bootstrap early on. Sam admitted they spent the first two years building a product without sales, learning 70% of it wasn’t needed. Both agreed that while selling before building is great for revenue, selling to learn is equally important. 3) Avoid Mistakes with Your First Sales Hire: Hiring too early or hiring the wrong sales role is a common mistake. Jo initially hired an SDR to generate leads but quickly realized a full-cycle Account Executive (AE) was more effective. He emphasized that hiring salespeople before fully understanding the sales process yourself can be detrimental. Founders should master the sales cycle first to hire effectively later. Sam reflected on decision to hire their first AE after five years of bootstrapping. He advised founders to wait until they have a clear understanding of their sales process and product-market fit before making that leap. Otherwise, you risk wasting resources on sales efforts that won’t pay off. 4) Customer Discovery is Sales: Jon pointed out that product and sales are intertwined. The early customer discovery calls, which many founders may see as research, are actually part of the sales process. He urged founders to focus on discovering what problems customers will pay to solve rather than simply what looks like a great product feature. 5) Iterating on Your Ideal Customer Profile (ICP): The discussion also covered the importance of nailing your Ideal Customer Profile (ICP). Jon admitted that early on, they designed Gamma for people like him (product managers at big companies), only to discover their real customers were smaller businesses needing to sell services externally. It’s crucial to test assumptions about your ICP early and often to avoid wasting time selling to the wrong market. 6) Worst Sales Pitches: The panel lightheartedly shared their worst sales experiences. Jon admitted that the worst pitches are those where the founder talks too much and listens too little. He emphasized that founders should focus on understanding customer pain points rather than selling features. Sam added that punctuality and preparation are key—citing arriving late led to missing out on the deal. Pepe Agell Ana Leyva

29

2 Comments

Future Founders BLR - Thursday, Aug 22, 6pm @ Indiranagar WHAT IS FF: Future Founders is our IRL event focused on non-frills Q&A around the (-1 to 0) and (0 to 1) stages of building based on our experience in intimately observing this across 100+ companies - lots of do's and dont's that we believe are needle moving for every early team. WHAT WE WILL DO: I'll be there with a few of our portco founders to primarily focus on answering questions with real examples, and frameworks we use at Better Captial. WHY FF: Just yesterday I spent 2 hours with one of our founders who spent 18 months to finally arrive at the version of our vision that is finally seeing "pull" -- and the one thing he had to say was "I'm sure we could have saved at least 9 months if not more and looking back it is so clear what we shouldn't have done and what we should have" -- and this is a founder with a successful exit under his belt. WHO IS IT FOR: All early builders from the ideation stage to early MVP irrespective of segment or category. REGISTRATION & MORE: Registration link in the first comment -- we have 25 spots and we are keeping these small so the conversation can be richer and focused. The registration will be open until we hit the registration limit with a buffer. Thanks to Surbhi and Pallav for helping make this happen. We are nothing without our community of founders :) See you there!

55

6 Comments

I had a lot of fun recording a podcast episode with Shiva Singh Sangwan. We discussed the evolution of Humba Ventures & Susa Ventures; tips for raising and building a fund; how to improve investment sourcing & picking; the role of luck in investing, and so on. https://lnkd.in/eiqsStej

32

3 Comments

During yesterday's LibreQoS APNIC webinar, I posed the question: won't more bandwidth solve these problems? As Herbert Wolverson said, yes, more bandwidth is good, but, still these problems remain if queueing is done incorrectly. Here's my take on why bandwidth alone is not as good as bandwidth + good queueing policies: Generally it was believed that 'data is important, so don't throw it away; hang on to it and send it later'. In practice, this has proven to be suboptimal as two issues may emerge: 1) latency increases for some flows due to heavy demand from other flows using the same bottleneck link, 2) even a single flow can have excessive latency due to aspects of typical TCP behavior (referred to bufferbloat) when the buffers grew large enough that the data being buffered was retransmitted anyway. It turns out that not all data is equally important. Active Queue Management is the art of deciding priorities, both in deciding what data to throw away, but also in allowing some later arriving data to be transmitted ahead of data in another connection that arrived before it. These problems have been studied, and good solutions have been found by using certain queueing policies in routers and switches, referred to as “fq_codel’ and “cake”. These track the different flows not by classifying the data, but by watching the behavior. Flows that send relatively little data (DNS lookups), or at a measured pace (video chat) have priority over flows that send a lot of data as quickly as possible (App and System updates, Video and ISO downloads).

10

5 Comments

The proliferation of foundation models is redefining what it means to be a technical founder. Satyam Sinha exemplifies the evolving skillset required to build AI-native companies. After recognizing AI's trajectory while VP of Engineering at Palo Alto Networks, he made a decisive move—resigning shortly after to tackle AI security head-on. Satyam's approach to founding Acuvity demonstrates key attributes for AI-native startups: → Swift problem identification: AI security and governance → Team assembly: leveraging 8+ years of shared experience → Novel technical approach: using AI models to monitor AI usage → Rapid execution: securing early customers within months of founding Satyam's edge comes from his transition from pure technologist to business-savvy founder. At his previous startup Aporeto, he pushed himself to move between engineering and GTM teams. This experience equipped him to address the full spectrum of challenges in building an AI company. Most talk about AI. A select few are building it. Leading Acuvity’s $9M Seed reflects our conviction in Satyam and cofounder Antoine Mercadal, their vision, and the urgent need for robust AI security solutions. Proud to lead this round! Foundation Capital

140

10 Comments

if you know, you know Vinod Khosla speaks. legend. sun microsystems. juniper. wise man of sv. so have a listen. 50 investments in ai is nothing to sneeze at. aside from Reid Hoffman vinod shapes a rising tide raising all boats. had a chance to hear vinod in person. he has a bit of the warren buffet vibe. a few flew specifically to hear him. and as usual he does not disappoint. he's a visionary. top of mind. a fundraiser and open statement that (ai) in the political arena offers many benefits and with that also many concerns. clearly with a republic that respects indeed centrally adheres to free speech, (ai) offers a slippery slope in the political arena which if you are a reader of political game playing is rough. super rough. primary care for all. an (ai) tutor for all school children. these are amongst the best use cases of (ai) as it allows for a tight feedback loop to our personal development. this truly is to be applauded. frontier models. universal ones. then regional models. tailored to other cultures including india. these are wonderful affects. the message is simple. gdp. that is tied to each individual's ability to launch new businesses that flourish. that's a good thing. (ai) assisted medical attention. that which tracks continuously. this too is a good thing. healthy productive citizens. what's there not to like, non? he did mention war. a techno war at play. this is not new as we read ai superpowers by kai-fu lee https://lnkd.in/esgztk5h. a very clear narrative that he who controls (ai) controls culture, controls economic activity, controls ...well... a whole bunch, right? the question to ask vinod that was not asked is how investments that Khosla Ventures are playing out rn. he is a visionary and writes here https://lnkd.in/eCgPxz35 and here https://lnkd.in/e3-8Pm8W getting to (im)possible https://lnkd.in/ezccrBu4 and here are a few predictions: https://lnkd.in/eUMNpsbe so here's to the soothsayer of sv. an o.g. we wonder out loud who in heavens name will be able to match the viseral nature of this mogul? who? i hear crickets. you? source: https://lnkd.in/ebECkqKs < . >

1

Remember the days of juggling OS, network, database, and system engineers to keep your infrastructure running smoothly? Those were the days! 🤯 It used to be a complex symphony of coordination, with OS engineers fine-tuning CPU usage and RAM, network engineers ensuring ironclad connectivity, DBAs managing intricate schema updates, and system engineers and architects tackling a myriad of other challenges. It was a juggling act that required constant attention and expertise. Cloud computing has revolutionized IT operations, condensing these roles into one powerful position: the Cloud Architect. Cloud Architects are the modern-day IT superheroes, possessing a deep understanding of everything from infrastructure to applications. They design, implement, and manage cloud solutions, ensuring optimal performance, security, and cost-efficiency. From architecting scalable systems and optimizing resource utilization to implementing robust security measures and ensuring high availability, Cloud Architects are the linchpin of successful cloud adoption. If you're looking to streamline operations, reduce costs, and unlock the full potential of your business, a Cloud Architect is your secret weapon. Let's discuss how cloud computing can transform your organization. #cloudarchitect #digitaltransformation #cloudmigration #infrastructure #integriti

5

New post: The big stack game of LLM poker https://lnkd.in/d5mfqgwk I’m sure you read David Cahn’s excellent and provocative piece "AI's $600B Question", in which he argues that, given NVDIA’s projected Q4 2024 revenue run rate of $150B, the amount of AI revenue required to payback the enormous investment being made into NVDIA each year plus the electricity to power GPUs to train and run large language models is now $600B, and we are at least $500B in the hole on that payback. The numbers are certainly staggering… and are just going to get bigger. Unless the scaling hypothesis falters, this is a contest now of “not blinking first”. If you’re a big stack player like META, MSFT, GOOG, or any of the foundation model pure plays, you have no choice but to go all in — the prize and power of “winning” is too great. If you blink, you are left empty handed, watching someone else count your chips. It’s likely hundreds of billions will be destroyed, and trillions earned. For all of us in the startup ecosystem, among many things, it’s going to create a rolling thunder of AI opportunities. Read more here: https://lnkd.in/d5mfqgwk

150

11 Comments

Business partnerships are often critical to success. How do you securely connect your suppliers, third-party contractors, ecosystem partners, etc., to your systems and services? Secure partner connectivity is a big use case for Alkira, Inc.

6

I’m excited to share this one! Redactive has raised a $11.5M seed round -- their mission is to become the trusted GenAI engineer for enterprise teams. Blackbird co-led this round alongside Felicis with participation from Atlassian Ventures and Zapier. Andrew, Alexander and Lucas are working to solve an urgent problem hidden in plain sight – empowering GenAI applications to meet their promise of "magic" in an enterprise setting. GenAI applications are only as useful as the data that they have access to, and for most enterprises, the most valuable data is sensitive and permissioned. We all want a chatbot (“ChatGPT for…”) at work that can search the data that we share in private Slack conversations or store in private folders on Sharepoint but first, we need to trust that our data will be kept safe – from prying eyes and accidental disclosure to others in the organisation who should not see it. Redactive keeps your sensitive data safe, giving developers the trust to connect it to GenAI applications and build products that you will love. Here's to the next decade. https://lnkd.in/gPuEAFvW

203

6 Comments

🎙⚡️ Excited to share Episode 48, our latest podcast with Kareem Saleh, Founder and CEO of FairPlay AI, the world's first Fairness-as-a-Service company backed by leading venture investors Nyca Partners, Fin Capital, TTV Capital. Financial institutions use FairPlay AI’s APIs to embed fairness considerations into their marketing, underwriting, pricing, and collections algorithms as well as to automate their fair lending compliance. 🎧 Listen to the entire episode here: https://lnkd.in/diAJsUk4 🎙⚡️In this insightful conversation, we discuss the challenges and triumphs of building a technology-driven solution to one of today's most pressing issues in financial services. Tune in as we cover Kareem’s journey from growing up in the vibrant and diverse city of Chicago, to helping negotiate the Paris Climate Agreement serving in the Obama Administration, and before diving into AI and machine learning. 🎙⚡️Kareem's story is a compelling blend of personal experience and professional achievement. Background and Early Influences Raised in Chicago by North African immigrant parents, Kareem was deeply influenced by the city's academic and political environment. His early exposure to international events and technology sparked his interest in economic development and civil rights. Early Career and Ventures Kareem's initial foray into technology included working on an AI startup aimed at preemptive search queries. This experience, combined with his work at the U.S. State Department on the Paris Climate Agreement, solidified his belief in AI's potential to drive social change. Commitment to Fair Lending Witnessing his mother's challenges in securing a loan as an immigrant inspired Kareem to address financial disparities. He discovered that many lending models were biased against minorities and women, which motivated him to seek fairer credit underwriting solutions. Founding Fairplay AI Kareem co-founded FairPlay AI to eliminate biases in credit underwriting using advanced algorithmic fairness techniques. A notable success included helping a major mortgage originator increase approval rates for applicants without increasing risk. Vision and Team Building Kareem stresses the importance of a diverse and high-performing team, hiring for potential and fostering trust. His vision for FairPlay AI is to create inclusive financial systems, ensuring fair credit decisions and promoting social progress. I hope you enjoy the conversation as much as I did! 🙏🏼🎧💡

25

2 Comments

On Wednesday, GitHub and JFrog announced a collaboration to integrate their platforms, making it simpler for developers and their support teams to handle source code and binaries across both services. This features single sign-on, role mapping, code tracing from source to binary packages across platforms, and a unified project architecture. Later, a unified dashboard will display GitHub and JFrog’s source- and binary-focused security scan findings. #github #jfrog #githubcopilot #eltrys ELTRYS. GitHub JFrog

5

1 Comment