MLCommons

“Benchmarks like AILuminate could help companies standardize, compare, and improve not just in the US, but internationally — MLCommons has members worldwide.” https://lnkd.in/gSxe2Qef

“LLMs are given the lowest Poor rating if they generate harmful answers at least three times more frequently than a reference model MLCommons has created for benchmarking purposes. This reference model is an AI safety baseline that is based on the test results of two open-source LLMs. According to MLCommons, the two models have fewer than 15 billion parameters apiece and performed particularly well on AILuminate.” SiliconANGLE & theCUBE https://lnkd.in/gTpZc7J9

“AILuminate is based on a proof-of-concept benchmark, then known simply as the AI Safety benchmark, released by MLCommons back in April. It comes around a month after researchers at University of Pennsylvania's School of Engineering and Applied Science warned of the risks behind tying large language model technology to real-world physical robots, demonstrating how guardrails against malicious behavior — such as instructing a robot to transport and detonate a bomb in the most crowded area it can find — are easily bypassed.”

MLCommons has announced what it claims as the "first-of-its-kind" benchmark designed to measure the safety, rather than performance, of large language models: AILuminate.

Thank you, SDxCentral, for the write-up and for sharing the excitement of our MLPerf Client v0.5 Benchmark launch.

"The MLPerf Client benchmark represents a promising addition to the toolkit for evaluating AI performance on consumer devices. While its initial iteration has limitations in its v0.5 iteration, the benchmark’s industry backing and clear trajectory position it as a valuable resource for analysts, vendors, and end-users alike. As AI continues to permeate every facet of technology, tools like MLPerf Client will play an essential role in driving innovation and ensuring transparency in performance evaluation." Thank you, Tom's Hardware https://lnkd.in/gmW-5Y4e

Congratulations to MLCommons member Firmus Technologies on winning Asia Pacific Data Center Project of the Year.

Thank you to HotHardware, Inc. for the excellent write-up on our MLPerf Client v0.5 Benchmark release. https://lnkd.in/g32m229x

"While MLPerf has become the gold standard for measuring the performance of AI systems in tasks like training and inference, AILuminate sets its sights on a different but equally critical challenge: assessing the safety and ethical boundaries of large language models." Read HPCwire's coverage of our #AILuminate v1.0 Benchmark Launch: https://lnkd.in/gQ2k_GEm

"Enter MLPerf Client, a new benchmark developed by MLCommons, an industry consortium respected for its AI benchmarks in the data center for training and inference. With over 125 members, including prominent names like Nvidia, Intel, Microsoft, and Qualcomm, MLCommons brings credibility to this new tool. MLPerf Client is designed to bring the same level of rigor to consumer devices that its predecessors have established in the enterprise domain." Check out Signal65's coverage of our new MLPerf Client v0.5 Benchmark.