Zhengyi Liu

Adam's post is good (and I like this categorization in general), but there are a couple of things that I don't agree with. First, I'd say time-to-recovery is a strength of CfD rather than CfM. Physalia (from our paper "Millions of Tiny Databases" https://lnkd.in/gepZPA2Y) is the CfD system that provides the metadata for EBS (https://lnkd.in/g7R3MgYk) a vastly bigger CfM system. We chose CfD for Physalia specifically to optimize recovery time, because it needs to recover before the larger system can recover. Second, I think the post is making an assumption that CfM systems are f+1 on the dataplane, use the minimal number of replicas, and don't use quorums. Aurora is an instructive counter example (https://lnkd.in/gNrNTPp7).

117

2 Comments

Thats the exact explanation I gave to a group of investors last week. (… it’s just Precision/Recall, attributes vectorization as “embedding”, K-Nearest Neighbor, User Search Intent, DAG, ML algorithms, etc all over again …). LOL and they asked what’s new this time around. And I was like not much, ML is still the backbone, and AI is the inference and actions based on the model. It works better when there is a feedback loop in a closed system. But the current state of the art like ChatGPT is based on scraping the Internet (garbage in, garbage out). Kinda like early search engines. What is a better analogy is that NVIDIA/Snowflake etc are selling tools, aka pickaxes to prospectors. The gold is up to you to find. And the opportunity right now is to monetize industry specific (healthcare, travel, financial services) clean data to build models with (aka a treasure map for the prospectors). From an enterprise perspective, what’s out there today aka generative AI isn’t very useful. Too much hype. To build useful engines will require an effort similar to search revolution 2 decades ago. From

16

3 Comments

Istio Service Mesh: great to see Ambient moving up to Beta, delta xDS becoming the default, and api-gateway becoming a first class citizen in Istio. https://lnkd.in/gYPmnTuj

34

According to Gartner® “Infrastructure and Operations (I&O) teams commonly struggle to manage Kubernetes (K8s) clusters at scale due to the talent shortage ���� — especially on heterogeneous scenarios (multicluster, hybrid, edge, etc.) or supporting multiple downstream teams. Besides inherent Kubernetes complexities, K8s teams must cope with the increase in the average number of clusters per organization from a few to dozens. As the cluster count grows and spans, the stack becomes more complex and diverse across different infrastructures (cloud, on-premises, and edge). This negatively impacts practitioners’ ability to maintain the clusters and demands more attention from I&O teams." 👉 struggling with managing complex Kubernetes? 🗓 schedule free consultancy => https://lnkd.in/dyErkV_C #angelscloud, #k8s, #Kubernetes, #digitaltransformation #multicluster #cloud

3

🔧 How do you scale an engineering team to support 32,000+ apps? At RevenueCat, our engineering team is the backbone of everything we do, and is faced with the challenge of maintaining reliable and scalable infrastructure that supports over 32,000 apps globally. How do we manage this scale while continuously delivering software that developers love? In our latest blog post, Miguel Carranza Guisado pulls back the curtain on our engineering strategy — sharing what we’ve learned along the way, from tackling tough challenges to refining our approach to engineering at scale. Here’s what we share: 🔹 Mission-driven development: We focus on building software that empowers developers to succeed, with every decision centered on delivering real value. 🔹 Challenges of scale: Balancing the demands of managing a fully remote team across 14 countries, handling 1.2 billion API requests per day, ensuring the integrity of $4B in transactions, and maintaining 10 SDKs. The complexities add up quickly, but with the right strategies, they can be effectively managed. 🔹 Guiding principles: Learn about the practices we rely on, like prioritizing reliability over features, embracing transparency in incident management, and continuously optimizing our processes. 🔹 Practical actions: We’ve put these principles into practice with strategies like rotating reactive teams, hosting regular engineering all-hands, and redefining uptime to set new standards in reliability. If you’re navigating similar challenges or just curious about how we do it, check out our latest blog post: https://lnkd.in/e_zt8AYA #EngineeringLeadership #SoftwareEngineering

32

1 Comment

On April 18th, just last week, Meta unveiled Llama 3, lauded as "the most capable openly available LLM to date". This remarkable achievement by the Meta team comes hot on the heels of Llama 2's release last July. Notably, the new models surpass their predecessors and competing offerings from other providers in terms of performance. Meta's commitment to open sourcing these powerful models while prioritizing model safety and responsible usage is commendable. (https://lnkd.in/ghv3RJRz) Yesterday, Microsoft introduced Phi-3 Mini, a member of the Phi-3 family of models, promoted as "the most capable and cost-effective small language models available". According to Microsoft, the key to achieve high performance in such a small package lies in the quality of the training data. The model has been open sourced and is now available on Ollama and Hugging Face. (https://lnkd.in/gKgSmrBn)   These compact yet powerful models serve as invaluable tools for creators eager to bring their visions to life or simply experiment with cutting-edge technology, particularly for those with limited access to resources.   Shortly after the release of Llama 3, reports began surfacing of enthusiasts running the 8B model on a Raspberry Pi 5 equipped with just 8GB of RAM—a modest $80 single-board computer. (https://lnkd.in/gbeQPhQ2) While the performance understandably reflects the limitations of such a platform, the fact that the model runs and generates an output is nothing short of magical. Similarly, the Phi-3 Mini model can run on ubiquitous devices such as smartphones.   Reflecting on my own journey, I recall the pivotal role that early access to a low-cost computer played in shaping my passion for coding. The gift of an 8-bit Commodore 64 home computer ignited my curiosity at a young age. Later, with the guidance of an outstanding Computer Science teaching staff at my high school IISS Marconi-Hack Bari, I embarked on ambitious projects like a networked multiplayer version of Battleship written in Turbo Pascal and 8086 Assembly, complete with custom sprites and lo-res graphics. These formative experiences paved the way for my career, eventually leading me to various engineering and leadership roles.   It all began with a simple, low-cost, modestly capable computer used for experimentation—a gateway to discovering my life's passion.   I encourage educators and parents alike to consider setting up similar low-cost experimentation environments. By introducing younger generations to open-source AI technologies, we can nurture their creativity and help them uncover their true passions. It's not just a weekend project—it's an investment in their future and the future of innovation and discovery. #education #youth #AI #LLM #future #innovation

24

Accelerate your Oracle to PostgreSQL migrations with DMS! Google Cloud Next '24 brings Gemini AI for code/schema conversions + explanations. Simplify the process, save time, & empower your devs. #GoogleCloudNext #Oracle #PostgreSQL #DMS

1

Databricks Clean Rooms, powered by Delta Sharing, enables organizations across industries to collaborate with partners in a privacy-safe manner without relinquishing control over sensitive data. Key features: ✔️ Any language, any workload ✔️ Any cloud, no replication ✔️ Any scale, any trust level Learn how it works👇

18

Did you hear? Anthropic's upgraded Claude 3.5 Sonnet model is now on Vertex AI! When building with the upgraded Claude 3.5 Sonnet on Vertex AI, you get a platform to take your workloads from concept to production with fully managed infrastructure, robust security and privacy, and advanced developer tools. Get all the details, here →

1

#deltalake innovation velocity continues. Kernel solves a major problem plaguing all data formats: engines don't follow the format spec and customers don't get the benefits of most of the innovation happening in formats. With Delta Kernel, we aim to make it zero effort for engines to enable new delta functionality.

30

4 Comments

Meta is leveraging AI for efficient incident response! 🚀 Discover how we're using cutting-edge AI to enhance our data infrastructure and improve response times. #MetaAI

4

1 Comment

🎉 Big congratulations to our incredible team (Haoyu Chen, Nick Greenquist,Yuxing Wang, Sridevi Maharaj, Ruiyang Wu and many more) for delivering several quarters-long impactful features on the Play Store! 🚀 We’re thrilled to have launched key updates like Search Home, Sports Moment, and WearOS recommendations, all designed to enhance personalization and help billions of users worldwide search and discover content they love. 🌍✨ Check out the Play Store Q3 updates: https://lnkd.in/dn5wx8Mi #GooglePlay #Personalization #Recommendation

72

1 Comment

Glad to see the latest #Apache #Flink 1.20 now supports Materialized Tables❗ Check FLIP-435 to learn more context and usage: https://lnkd.in/gF3eBhGU TL;DR many data warehouse and stream processing players have been adopting Materialized View for real-time data pipelines. #Snowflake ❄️ has Dynamic Tables; #Databricks 🧱 has Delta Live Table; there is even a company called Materialize. Timeplus supported Materialized View since 3 years ago. No matter you prefer calling it "Materialized View" or "Materialized Table", it's indeed a nice way to build SQL-based transformation/ETL/routing/alerting logics. Good news for #Flink users. Now you can finally run SQL queries with MV on MV, no longer need to send intermediate state to a Kafka topic and read them back to Flink SQL. The Materialized Table in Flink 1.20 can persist the result/snapshot in the storage according to the catalog, and also supports different refresh strategy/interval, even pause/resume it. However the refresh interval is not guaranteed. You may still get out-of-date result in 1.20 I think. While Timeplus is a streaming-first product, you always get the latest results in our materialized views(MV). Besides the internal storage for the MV, you can also set a target for the MV, to send query/ETL results to a Kafka topic, a ClickHouse table, or the other Timeplus stream. You can also pause/resume them. Try our open source Timeplus Proton https://lnkd.in/g8aJBsPa or check our docs about MV at https://lnkd.in/g5eNU3Fk

29

1 Comment