Andrew Wan

Did you miss my final #RAGHack stream, "Evaluating your RAG chat app"? Catch the recording here: 👇 https://lnkd.in/gSd32wXm I covered: 🤔 What does it mean for a RAG answer to be high quality? 🔍 Which factors affect RAG answer quality, and which affect it the most? 💻 Using promptflow-evals SDK to run GPT evaluators in Python 📊 Using my ai-rag-chat-evaluator CLI tools to generate synthetic ground truth data, run batch evaluations, and compare metrics across runs 🤷🏻‍♀️ Strategies to make sure your app can say "I don't know" at the appropriate times Grab slides from https://lnkd.in/g-9xSQvV #rag #openai #python #evals

20

Hey folks! Garrett Milster and I have only posted about Sacramento Labs once, so we figured it's time for an update 😊 **What is Sacramento Labs?** We’re a small team focused on launching software products that solve real-world problems. Think of us as a startup lab—constantly testing ideas, learning from feedback, and building tools that matter, and having an impact while doing it. **How do we decide what to build?** We like to build products that: -         Solve real-world problems in mostly ignored industries -         Are small, highly maintainable, and scalable codebases -         Make the world a better place directly, or by donating a percentage of profits to a relevant charity in that space You can learn more about our philosophy at sacramentolabs.com **What you can expect from us** We’ll be posting product ideas that are in various stages of our pipeline: Early Concept, Customer Development, In Progress, Launched, Scaling. We’d love to hear your feedback! **Our latest concept: Self-Hosted Security** Here’s the problem: - Businesses often use third-party URL shorteners to share internal tools, resources, or campaigns, but these services introduce serious risks. Enterprise companies like Google, Microsoft, and Stripe have built their own internal URL shorteners for this reason – but most businesses don’t have the resources to do that. - Phishing attacks are rampant, and links from external services can be spoofed or abused. - Branding and security controls are limited, leaving sensitive information exposed. - You can’t track link activity—who’s clicking what, when, or why. - When employees leave, their shared links still work, leaving potential vulnerabilities. Our proposed solution: Self-Hosted Security. It’s a plug-and-play solution for businesses to host their own branded URL shorteners with: - SSO for secure access tied to your company’s directory. - Link expirations to ensure old links don’t stay active indefinitely. - Audit logs to track activity and improve visibility into link usage. 📢 We need your input! Would this solve a problem for you? What features would make this indispensable? Check out the landing page and share your thoughts: selfhostedsecurity.com Want to chat? Email me at ian@sacramentolabs.com 😊 💡 And here’s something we’re passionate about: a percentage of profits from Self-Hosted Security will go to the Electronic Frontier Foundation (EFF). Your feedback not only helps us create better solutions, but also contributes to making a positive impact. Thanks for being part of the journey!

55

4 Comments

~40% of Frec’s customers are engineers working in tech, many of whom expect frequent and large capital gains. They also prefer to self-manage their money, which makes Frec an ideal platform. Meet Evan: an engineering leader who upgraded from investing in ETFs to direct indexing. Direct indexing appealed to Evan because it offers comparable returns to his ETFs while saving him money on advisory and ETF fees. “ETFs are true and tested, and giving folks a different way of investing in the market while saving more money is really appealing.” He’s used many other robo-advisors and other fintech tools, but “Frec seems to maximize tax loss harvesting opportunities better than any other product that I’ve used.” He also likes Frec's transparent fee structure.  “I feel like with a lot of other companies, that’s muddled in with the cost of investments,” he told us, “and you don’t really know how much you’re paying or how much you’re not saving as a result. I think Frec does a really good job of laying all of that out and giving you a true understanding of the cost of these investment products.” Read more about his Frec journey here ⬇

45

2 Comments

Building a reliable RAG system isn’t just about performance—it’s about trust. Matt Turk's blog post breaks down the importance of this for both technical and non-technical folks—highlighting how Cleanlab's TLM, paired with Pinecone’s vector database, delivers high-quality data retrieval, accurate, context-aware responses, and scalable performance to meet growing demands. 👏

7

AI is innovating a variety of industries across the US and Agriculture is no exception. 🐄 My new friend, Catie McVey, has created a new "Agtech" startup that uses patented 3D Modeling and Machine Learning algorithms to evaluate the health of cows by looking at specific characteristics such as hoof morphology and facial characteristics. Working with her to architect how to get source data from "The Stockyards" to "The Cloud" has been a lot of fun. I am constantly discovering new ways AI can make an impact in industries from legal, to healthcare and now agriculture! Read more about Catie's journey on RIoT - Internet of Things' blog #AI #software #agtech

9

We started off originally building our Portfolio view for assets we extract from Brokerage statements. Today we've added another piece of the puzzle by bringing Private Investment holdings (Private Equity Funds, Syndications, JVs, etc) extracted from Capital Statements into our platform.

7

Want to build evals for your LLM app? It can feel intimidating and time consuming (and the best eval system will be as custom as your app), but there's a simple way to get started: 1. Collect real data 2. Look at the data 3. Note failure patterns and then write code that reliably detects them Note: step 2 is the most important (but least fun) part of the process

16

This is a cool project, and another example of the emerging "replace stateful storage with S3" trend. I do wonder at what point someone will package this sort of thing up into a FUSE filesystem to make it generally usable across existing services. Or maybe every service has unique enough needs that it's better to handle it custom each time. https://lnkd.in/etcVBfVf

9

Tired of constant prompt tweaking for LMs? Meet DSPy: a framework that makes building reliable LM systems more like programming, less like guesswork. DSPy optimizes LM pipelines effortlessly with declarative programming that adapts to your data and models. With DSPy, you can: - Create workflows with reusable modules. - Use optimizers to auto-tune prompts and parameters. - Focus on system goals, not prompt hacking. Think of DSPy as the PyTorch for LMs: structure and optimization for your tasks. Let's make LM engineering smarter — less prompting, more programming! https://lnkd.in/g4wEfSCq Keishin Nishiyama Iris Fu Adam King John Hawkins

8

A design principle we've had at Census since day one was "Don't Build Silos". It's been a hard tendency to fight. The reality is that storing data in a place the developer controls is just easier than storing it where our customer has the controls. But that doesn't make it the right answer. So we took on the hard work to figure out that architecture. And over the years, we've expanded that approach. Today, we're formalizing it even further with Observability Lake. Logging is critical when debugging, but it's also dangerous if mishandled. Observability Lake backs our now debugging tools by giving our customers a way to safely store as much (or as little!) as they want. This is a very natural application of a data lake pattern, but an under-appreciated one today. I expect we'll see a lot more SaaS adopting the pattern of giving the users a simple fast path to reading/storing their data wherever they choose as we see more adoption of Iceberg in the coming months. #datalake #observability #apacheiceberg #dataarchitecture

51

4 Comments

After my last doc-auto vs generative AI post, there was some interesting discussion here (as well as some in-person mocking) and so I thought I'd get a bit more into some of that. This time I get into why we'd be better to focus on the real issue with doc-automation (the resource and investment required to get up and running) as well as some rant-y bits about product management..

5

2 Comments

Is this the end of embeddings? Hear from Josh and Tashrish on the pros and cons of using embeddings-based RAG for document data extraction and our experiments with switching to completions-only RAG.

5

Technical things I thought I'd be doing "more of" when running an AI startup: finetuning models, advanced data science, writing papers 📊 Technical things I'm actually doing: writing python to overlay fonts and image on Pinterest posts 🤡 Idyllic App #ai #startup

14

1 Comment

We’re growing the team at Kaleidoscope.bio! Lots happening, lots more to come -- we’re adding a few key people to help make more magic happen, faster. If you’re looking to do meaningful work, are excited about the rapidly evolving intersection of software technology and R&D (biotech and beyond), and think you could be a good fit for one of the roles below, let us know! We’re an ambitious, focused, and experienced (remote-first) team, applying best-in-class engineering and design principles to the challenges of modern R&D. Questions like how can we help bring about a world where R&D organizations can make confident, data-savvy decisions? How can we assist teams in focusing attention on the most promising directions, quickly? How can we play a role in getting more therapeutics, diagnostics, and technologies on the market and in the hands of the populations that need them the most? For more on us, check out our website: https://lnkd.in/eyrUp9Pj For more on the roles, check out the links below. The applications themselves are referenced in each of the pages, as well as in the comments below. Account Executive (AE): https://lnkd.in/etTyFRKr Scientific Customer Success Manager (Scientific CSM): https://lnkd.in/etwzMetG Software Engineer (SWE): https://lnkd.in/eTQ7gvfw

79

7 Comments

We managed to cut down the infrastructure cost for a client utilizing proxy resources for web scraping from ~$9,000 to ~$5,000 /month through Scrape.do 📣 Unaware that you may be consuming your team's time inefficiently with web scraping? Try our system and see how swiftly we can resolve all your issues while optimizing costs.

32

1 Comment

The importance of speed in building: We recognize that sprinters are the fastest when they get out of the blocks, and the winners in a sprint slow down the least. (An excerpt from my recent blog post: Getting Out of the Blocks Fast.)

21

1 Comment

RAG search is leaving money on the table. Six low-hanging fruits: synthetic data, date filters, better feedback, tracking metrics, full-text search, and question-like chunks. Most teams are overlooking these. Don't be one of them. https://lnkd.in/gHTnWxzZ

30

2 Comments

On Agent Dev & The Case for The Engineer’s Creative Process Agent development pulls engineers closer to the headspace of artist - and my bet is if engineers embrace and protect their creative processes, we unlock the next level of innovation in this agentic era. In machine learning, the saying goes, "it is more art than science." Building agents is a non-deterministic process that adds significant variance into systems. Engineers would be well-served to hone their creative process. The best engineers will become artists. 1. Building isn’t linear. Building agent infrastructure is far from linear. Engineers often experience a cycle of frustration and eventual breakthroughs. While not perfect, these reasoning systems are now stable enough for integration into our tools and lives. The history of AI, from machine learning to deep learning, has always involved non-linear development. Today's agent infrastructure development is even less linear, requiring engineers to understand complex semantic translations and carefully manage user rollouts to avoid failures. 2. Building with a relationship to the work. With new forms of intelligence come new relationships. Engineers must navigate a relationship to work that involves agent memory and understanding. What is it to build a system that knows you? When people feel understood, they feel loved. What is it to love the systems we’re creating? How does a creative process help? A non-linear process and a relationship to the work are the territories of the artist. Engineers, trained towards control, must embrace uncertainty, risk embarrassment, and accept that their efforts might yield unexpected results. We’re glimpsing a creator's role in a system that understands. While engineers have largely operated in a logical space, they are now called to form a heart connection with the systems they build. A creative process helps artists endure challenges and breathe life into their work. It involves a commitment to creating, discipline, and accepting that one can only control the process, not the outcome. Agent development pulls engineers closer to the mindset of artists. If engineers embrace and protect their creative processes, we can unlock the next level of innovation in society. Read more on this on my Twitter: https://x.com/ashebytes

17

2 Comments

Today we're launching our newest candidate quality-of-life feature on Ropes - IDE themes. There's a universal dislike for the rigid IDEs that assessment co's force on candidates. These code editors are slow, clunky, and missing the styes/shortcuts that developers enjoy. Developers are unique, and we work in unique ways. Ropes offers two new options to let candidates keep their style: (1) First, many employers on Ropes allow candidates to work in their own IDEs entirely. We've developed a way to gather insights like you were in the room, while allowing developers to use their daily environments. (2) For cases that need a browser-based IDE - we're giving the ability for candidates to choose their style/editor setup before the assessment starts - so they're comfortable when it begins. --- My bet is that if we build a great experience for candidates, we'll be building a great platform for evaluators, too. I hope you'll follow along as we pursue that mission - and we need help! We're hiring founding engineers in NYC - if you're interested, I'd love to talk to you.

47