Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

NVIDIA

AI overview

Sign in and the AI will write an overview from our coverage.

Headlines · 1494

  1. StreamEval Benchmarks Real-Time Interactive Video Models

    36Kr AI reports that Tsinghua University, NVIDIA, CMU, UC Berkeley, Princeton, NUS and other institutions jointly launched StreamEval, an open benchmark for real-time interactive video generation. It covers five task types, 300 test cases and 38 models, evaluating instruction execution, state consistency, visual fidelity and audio-visual coordination; the code, dataset and scoring rules are open for review and replication. The reported results show that strong video quality does not necessarily mean reliable continuous instruction following or long-term consistency.

    36氪 人工智能 · 🔥 0
  2. CITIC Securities Sees Rising Interest in Physical AI and Edge AI

    36Kr reported that CITIC Securities said interest in physical AI and edge applications was rising, with industry events including OpenAI, NVIDIA GTC and the Unitree ecosystem conference cited as catalysts. The brokerage attributed a related market rally to these events, prior share-price adjustments and divergent storage-industry expectations, but the report provided no independent data isolating their contributions. CITIC identified robot capability upgrades, greater data-collection demand and edge-device interfaces as three investment themes.

    36氪快讯 · 🔥 0
  3. Firmware unlocks target FP32 accumulation on older NVIDIA GPUs

    A Reddit user’s technical report says firmware-based unlocks can restore full tensor-core speed with FP32 accumulation on NVIDIA’s Turing, Ampere and Ada GPUs, while consumer Blackwell remains restricted. The author attributes the difference to factory fuse settings and a privileged override register, but the supplied evidence contains no independent hardware verification or NVIDIA statement confirming the mechanism or alleged intent.

    r/LocalLLaMA · 🔥 0
  4. Tinybox drops Nvidia options as AMD six-GPU config rises to $22,000

    Tiny Corp's Tinybox configurator is now AMD-only after an October 24 commit titled "rip nvidia" removed the remaining Nvidia options and the Tinybox Pro, TechRadar reported. The six-GPU Radeon AI Pro R9700 configuration offers 192GB of aggregate GPU memory, and its price increased from $19,000 to $22,000; a four-card, 128GB version now starts at $17,000. A $91,000 Tinybox green v2 with four RTX Pro 6000 Blackwell cards remains listed on the Shopify storefront, although TechRadar said it was unclear whether those orders would be honored.

    TechRadar · 🔥 0
  5. SWI Compute Launches UK Nvidia AI Factory Campus

    SWI Group has launched SWI Compute to build and operate Nvidia AI factories in Europe, with about $1 billion of Nvidia systems ordered, according to The Next Web. Its first UK factory is planned to begin at 30MW with nearly 7,000 Rubin GPUs and 3,500 Vera CPUs in Vera Rubin NVL72 racks, and is due online by the end of 2027. SWI plans to expand the campus to 530MW, while the initial deployment uses Nvidia Blackwell Ultra systems.

    The Next Web · 🔥 0
  6. NVIDIA Details Minerva AIOps Platform for AI Factories

    NVIDIA describes Minerva as an AI-native AIOps platform designed around AI factory topology and workloads. It collects telemetry across GPUs, networking, storage and workloads; maps infrastructure to active jobs in a dynamic knowledge graph; and uses topology-aware correlation to group dependent symptoms into incidents. The platform includes a native AI agent, Sherlock, that lets operators query live telemetry and the knowledge graph for root-cause analysis in natural language. NVIDIA says its testing showed reductions in notified events and tickets for supported scenarios. In one internal case, Minerva linked 81 events and anomalies to an affected job as a single incident, reducing repair time and saving about 54 hours of total cluster uptime; these results are vendor-reported. Minerva is also being aligned with DSX OS, with capabilities beginning with the data collector planned to become open-source parts of DSX OS.

    NVIDIA Technical Blog · 🔥 0
  7. Marvell Unveils 2nm Optical Interconnects for AI Scale-Up

    Marvell has unveiled 2-nanometer optical interconnect technologies for AI scale-up architectures that connect computing and memory resources as part of a larger system. It also introduced the Photonic Fabric Memory Module, which combines DDR5 memory, high-bandwidth memory caching and optical connectivity to support a shared-memory tier. Marvell executive David Lazovsky said the company’s optics and SerDes technologies are intended to reduce total power consumption, but the report provides no independent benchmarks. The report also says Nvidia made a $2 billion investment in Marvell and that the company agreed to design custom AI chips for Google.

    SiliconANGLE AI · 🔥 0
  8. AI Makes Knowledge Cheaper and Mentorship More Valuable

    A commentary published by the WeChat account Budongjing and written by Budongjing Yeshu de Rust argues that AI is reducing the cost of publicly available information while increasing the value of guidance that identifies blind spots, calibrates judgment and compresses the time needed to gain expertise. Drawing on TSMC’s US expansion, mentorship in investing and examples from the AI labor market, it contends that AI-related hiring is rewarding proven capability and speed to learn. The article presents these as its interpretation rather than as a new study.

    虎嗅 AI · 🔥 0
  9. AI Agents Enter Automated Wet-Lab Workflows

    Frontier AI companies are extending AI beyond molecular design and screening by connecting research agents with automated wet laboratories. The report says Anthropic proposed the MHS hardware standard to give agents unified access to instruments, while Google introduced VERID, a framework for tracking laboratory states and operation histories. It also reports that Google’s agents completed three design-test-analysis cycles for an adapter-design experiment. OpenAI and Ginkgo Bioworks reportedly ran six cycles involving more than 36,000 reaction combinations and reduced cell-free protein-synthesis costs by about 40%. These demonstrations are described by the report as early closed-loop systems, with data interoperability and reliable laboratory-state modeling still key challenges.

    虎嗅 AI · 🔥 0
  10. Bedrock Robotics Raises $600M Series C at $4.1B Valuation

    Bedrock Robotics raised a $600 million Series C co-led by CapitalG and Emergence Capital, valuing the autonomous construction machinery startup at $4.1 billion, according to Bloomberg and The Next Web. The company says its driverless excavators have performed paid work on live job sites since mid-August, with proceeds earmarked for deploying more machines and expanding field-support teams.

    The Next Web · 🔥 0
  11. Meta raises Potential Equity Financing Plan to $97.5 Billion for AI Buildout

    Meta increased the potential size of its equity financing plan to $97.5 billion, including a $25 billion Class A forward sale, $12.5 billion of mandatory convertible preferred stock and up to $60 billion in potential Class C sales. The funds are intended to support AI infrastructure and other businesses as Meta expects 2026 capital expenditures of $135 billion to $145 billion. Nvidia, AMD and Broadcom expressed nonbinding interest in purchasing up to $5 billion of Meta Class A stock.

    科技新报 · 🔥 0
  12. Cambricon Q3 profit jumps 148% as the “Nvidia alternative” case shifts

    Cambricon reported Q3 2026 revenue of RMB 3.531 billion, up 104.47% year over year, and attributable net profit of RMB 1.404 billion, up 147.74%, according to TechNews. Titanium Media said the chip designer remains an important Chinese AI-compute supplier, but rising competition from Nvidia, Huawei and other domestic vendors is weakening the “Nvidia alternative” capital narrative. It also highlighted inventory and prepayment growth, weaker cash conversion and a Q3 revenue figure below the market forecast.

    钛媒体 · 🔥 0
  13. Counterpoint forecasts rapid growth in humanoid robot AI accelerators

    Counterpoint Research expects cumulative shipments of AI accelerators for humanoid robots to exceed one million by 2030, representing a 144% compound annual growth rate from 2026 to 2030. The report, relayed by IT之家, says GPUs should remain the leading near-term architecture as VLA and world models converge, while AI ASICs are advancing through the migration of automotive chip expertise into humanoid robotics. Nvidia reportedly leads the GPU market through its chip, model and simulation ecosystem, while Qualcomm has entered through heterogeneous computing and design partnerships.

    IT之家 AI · 🔥 0
  14. AMD, NVIDIA and Intel Linked to $5B Taiwan Substrate Investment

    Wccftech, citing a Taiwanese supply-chain report, says AMD, NVIDIA and Intel have paid more than $5 billion to Taiwanese substrate manufacturers to reserve long-term capacity. The report says Kansas, Nanya and Umicron are expanding plants and factories, but the customer identities and payment figure have not been independently confirmed. Substrates are used to package components such as memory and GPU chips, and Ajinomoto Build-up Film is described as the most common material in this area.

    Wccftech · 🔥 0
  15. Samsung and SK hynix reportedly ramp 8Hi HBM4 for NVIDIA Rubin Ultra

    Samsung Electronics and SK hynix reportedly plan to mass-produce and ship 8Hi HBM4 to NVIDIA in Q2 2027, with 8-layer supply for Rubin Ultra and standard Rubin increasing as 12-layer production remains active. The reports attribute the shift to the greater cooling and yield challenges associated with taller DRAM stacks. Wccftech assessed the claim as “Questionable,” noting that NVIDIA has not finalized whether Rubin Ultra will use 8Hi HBM4, HBM4E, or a mixed configuration.

    IT之家 AI · 🔥 0
  16. Three Employees Suspected in Theft of Two Pallets of RTX 5090 GPUs

    Brazilian police identified three employees of Guarulhos airport as suspects in the theft of two pallets of Zotac GeForce RTX 5090 graphics cards valued at about $470,000, according to the original report. The shipment was purchased in the United States by Brazilian hardware importer and distributor Oderço Distribuidora for delivery to a warehouse in Manaus, but disappeared within the airport area. A Brazilian television station later found an OLX listing for a card whose photographed serial number reportedly matched a stolen card; the listing was subsequently deleted. The report says the investigation is ongoing, the three employees remain suspects, and it does not state whether they have been charged or convicted. It also says the number of recovered cards was unconfirmed and that a truck reportedly involved had not been found. NVIDIA denied a separate rumor that it would discontinue the RTX 5090 and redirect GB202 chips to professional graphics cards.

    cnBeta · 🔥 0
  17. Survey Links Idle GPUs With Continued AI Platform Plans

    VentureBeat Intelligence’s three-round VB Pulse survey found that 79% of surveyed enterprise respondents reported GPU capacity sitting idle at least half the time. Among companies with average utilization of no more than 50%, 70% planned to introduce, expand or replace an AI computing platform within 12 months. Of 64 respondents planning a new platform, 58, or 91%, lacked finance-reviewed ROI data. Among companies planning to procure a new computing platform whose GPUs were idle, 90% favored public clouds and model API providers, while 8% considered expanding local private clusters. The report also cited an NVIDIA guide reporting that 85%–97% of context could be reused in a Claude Code scenario and a Vals AI test in which multi-agent setups cost 1.8–5.1 times as much as a single agent, with a statistically significant performance gain in only one of four configurations.

    36氪 人工智能 · 🔥 0
  18. AI Agents Shift the Focus of SaaS Valuations

    A 红杉汇 article republished by 36Kr argues that AI agents are shifting software value away from basic feature sets toward scarce data, hard-to-replace workflows, developer distribution, security, identity and compliance capabilities. It cites reported 2026 transactions involving Cursor, Hugging Face, OpenRouter and World Labs, alongside Airtable's reported sale to Bending Spoons for about $1.3 billion, compared with Airtable's peak valuation above $11 billion in late 2021. The article presents these cases as evidence of changing acquisition-market priorities.

    36氪 人工智能 · 🔥 0
  19. NVIDIA Revises Vera Rubin Bandwidth Specifications

    IT之家 and Wccftech reported that comparisons between archived and current NVIDIA pages show lower HBM4 and sixth-generation NVLink bandwidth figures for several Vera Rubin configurations, alongside higher scale-out networking bandwidth. For the Rubin GPU, the published HBM4 bandwidth changed from a maximum of 22 TB/s to 19.2 TB/s, while NVLink 6 bandwidth fell from 3.6 TB/s to 3 TB/s. The changes also cover the Vera Rubin Superchip, NVL72 and HGX Rubin NVL8 systems. NVIDIA has not publicly explained the revision, and some product and regional pages still show older specifications. These are published specification changes rather than measured benchmark results, and the earlier 22 TB/s figure was a maximum, so the revision does not necessarily represent an equivalent decline in real-world performance across every configuration.

    Wccftech · 🔥 0
  20. Nvidia case highlights Foxconn’s AI use in manufacturing

    TechNews, citing a customer case published by Nvidia, reports that Foxconn used AI to optimize injection-molding parameter settings and changes, reducing the related engineering workload by 50%. Automated production scheduling reduced the hours required for manual scheduling by 25%, while AI-assisted SMT analysis addressed more than 20 major production-line defects, with an anomaly improvement rate above 90%. Nvidia also reports that Foxconn uses DGX SuperPOD and DGX GB200 systems to develop its FoxBrain language model, while Nvidia deployment and simulation tools support the rollout of Foxconn-developed AI technologies across factories.

    科技新报 · 🔥 0
  21. Silicon Valley AI workers’ matchmaking dinner prioritizes employer and AI views

    TMTPost, citing The San Francisco Standard, reported that a San Francisco matchmaker organized a dinner attended by seven men who paid $800 each and 15 women who entered free. Nearly all attendees reportedly worked at OpenAI, Anthropic, Nvidia or Google. The report said selection emphasized AI employers and views on AI over appearance, salary or location, while the commentary portrayed the gathering as an example of workplace identity and AI debates entering private social life.

    钛媒体 · 🔥 0
  22. High-frequency traders ramp up AI compute spending

    Bloomberg reporting carried by Techmeme and The Decoder says major high-frequency trading firms are spending billions on cloud contracts, AI startups and proprietary data centers as they compete for AI computing capacity. Jane Street reportedly runs a Texas data center with more than 4,000 Nvidia GPUs and committed $6 billion to CoreWeave services, while Hudson River Trading signed a multiyear CoreWeave deal and spends $1 billion annually on AI. XTX Markets is building a facility in Finland, and Jane Street also backed chip startup Etched and became its first customer.

    The Decoder · 🔥 0
  23. FCC Shadow Raises Stakes for China’s Optical-Chip Supply Chain

    An analysis republished by 36Kr, TMTPost and Cyzone examines how potential FCC restrictions on optical interconnect could affect China’s efforts to domesticate optical chips for AI infrastructure. It says China has made progress in 70mW and 100mW CW lasers, but remains less than one to two generations behind overseas leaders in high-end EML and high-power CW products, with yield and customer-certification hurdles. The article also warns that duplicated investment could create overcapacity in lower-end products, though those warnings are analytical rather than independently verified.

    36氪 人工智能 · 🔥 0
  24. Researchers Explore GPT-6 Astra for Robot Control

    36Kr, citing a CITIC Securities research report, says general-purpose models are extending beyond high-level robot task planning toward action generation and execution control. The report says NVIDIA, Stanford and other research institutions have been exploring GPT-6 Astra for robot control, using its reported visual understanding, spatial reasoning and task-planning abilities to reduce reliance on specialized policy training. The supplied reports describe research exploration rather than a confirmed production deployment.

    36氪快讯 · 🔥 0
  25. Data centre expansion faces growing backlash

    Data centres are expanding rapidly to support AI services such as ChatGPT and Claude, while their electricity use, water consumption, noise and environmental impact are fueling public opposition. BBC Technology reports that 16 US states have proposed laws to ban or temporarily block new facilities, with opposition also reported in Mexico, France, Indonesia and the UK. The debate will feature in the US midterms on 3 November, while Brookings Center analysis finds that hyperscale facilities generate more local jobs than colocation sites.

    BBC Technology · 🔥 0
  26. China Merchants Securities sees physical AI as a computing-demand driver

    China Merchants Securities said in a research report that NVIDIA’s GTC centered on Vera Rubin compute upgrades and the expansion of physical AI applications. It described Vera Rubin, liquid cooling and modular deployment as supporting more efficient AI infrastructure, while citing robotics, intelligent driving and industrial manufacturing as application areas. The report projected that physical AI could become an important growth direction and add demand for computing infrastructure.

    36氪快讯 · 🔥 0
  27. SpaceX Secures $40B GPU Financing Commitment from Apollo

    SpaceX disclosed a binding commitment for roughly $40 billion in financing from an Apollo-managed entity to purchase AI chips and related server hardware, then lease the equipment to SpaceX. The leases are expected to run about five to five-and-a-quarter years, with deliveries beginning in the first half of 2027; Apollo said the arrangement will help deliver Nvidia Vera Rubin chips for SpaceX’s terrestrial compute buildout. Musk described the deal as closed, but Bloomberg reported that pricing negotiations were still ongoing and SpaceX said closing remained subject to conditions by year-end.

    TechCrunch AI · 🔥 0
  28. RTX 3080 20GB ReBAR and P2P Unlock Boosts Dual-GPU Prefill

    Community developer u/nathannguyennng enabled ReBAR and peer-to-peer access for RTX 3080 20GB GPUs, according to a r/LocalLLaMA report. On one dual-GPU system running vLLM 0.32.0 in tensor-parallel mode, the reporter measured 70%–115% higher prompt-processing throughput with both unlocks enabled compared with the stock configuration. Decode results generally varied within measurement noise, although the 261,900-token test showed a stable measurable improvement from gpu_unlock rather than P2P. The report cautioned that systems without full PCIe connectivity to the second GPU may see smaller gains.

    r/LocalLLaMA · 🔥 0
  29. He Weiwen links AI investment boom to faster global trade growth

    In an authorized Barron's article republished by TMTPost, China International Trade Society council member He Weiwen argues that booming AI investment and trade in related technology products are driving stronger Chinese and global trade growth, a dynamic he says cannot be explained by industrial overcapacity alone. He cites faster import growth for AI-related IT products and shifting regional balances, including large Chinese surpluses with the US and EU but an approximately balanced overall position with RCEP members. The article also cites WTO trade-outlook figures and regional export data to support its interpretation of AI-related supply chains.

    钛媒体 · 🔥 0
  30. Nvidia outlines $2,000 per-GPU annual AI agent software licensing

    Nvidia CEO Jensen Huang said licensing Nvidia software for AI agents in “AI factories” could cost about $2,000 per GPU per year, and that software licensing will become a larger part of the company’s business, according to Tom’s Hardware and TechSpot. The figure is an example rather than final pricing: Nvidia has not disclosed the effective date, eligible GPUs, covered software, or how the proposal relates to the $4,500-per-GPU annual AI Enterprise subscription. The discussion accompanied Nvidia’s OpenShell Reference Architecture, which combines Vera Rubin infrastructure, Vera CPUs, BlueField-4 DPUs and Nvidia Sentry to provide a secured environment for AI agents.

    Tom’s Hardware · 🔥 0

← PreviousNext →

Experience and discussion from the community

Share my NVIDIA experienceAsk about NVIDIA

Switchyard: An open-source library that helps an AI agent choose which model handles each request.

What problem it solves AI agents need to choose among models while balancing task success, cost, and latency for their workloads. Applications also need flexibility in where routing runs and who controls model clients, credentials, retries, and service operation. Key features Multiple routing algorithms — Routing can be configured with Auto, Task, Execution, or Composite strategies, including mode…

github.comhttps://github.com/NVIDIA-NeMo/Switchyard

vllm: A fast and easy-to-use library for LLM inference and serving.

What problem it solves Serving large language models requires efficient memory management, flexible model execution, and practical integration with existing models and APIs. vLLM addresses these needs with inference and serving capabilities spanning batching, caching, quantization, distributed execution, and multiple hardware platforms. Key features Efficient memory and serving — vLLM manages atte…

github.comhttps://github.com/vllm-project/vllm

skills: Portable instruction sets that teach AI agents how to use NVIDIA software for end-to-end workflows.

What problem it solves AI agents need product-specific instructions to use NVIDIA software reliably across physical AI, robotics, simulation, CUDA-X libraries, RAG, and platform workflows. The catalog centralizes these skills while keeping them maintained in their respective product repositories and synchronized through an automated pipeline. Key features Agent-discoverable skill catalog — The NVI…

github.comhttps://github.com/NVIDIA/skills

SkillEvaluator: An open-source tool that checks AI agent skills and measures their effect on task results.

What problem it solves AI agent skills need to be checked for quality, security, duplication, and their actual effect on agent behavior. Static checks alone cannot establish whether guidance improves task results, so evaluation must also support live comparison with and without a skill. Key features Three-tier evaluation pipeline — Independent tiers validate skill safety and structure, detect over…

github.comhttps://github.com/NVIDIA/SkillEvaluator

llama.cpp: Plain C/C++ implementation for local and cloud LLM and VLM inference.

What problem it solves Running large language and vision-language models can require substantial setup and may depend on the available hardware. The project aims to make inference accessible across a wide range of local and cloud hardware with minimal setup. Key features Minimal-setup inference — The project provides installation options for common platforms, including scripts, Docker, pre-built b…

github.comhttps://github.com/ggml-org/llama.cpp

TensorFold: A native language-model inference engine for serving checkpoints on Apple Silicon and NVIDIA GPUs.

What problem it solves Serving language models requires reading checkpoints, tokenizing requests, and running accelerated kernels while managing a local model server. TensorFold addresses these requirements with a native binary that can run on Apple Silicon and NVIDIA GPUs without requiring Python or MLX for serving. Key features Native Metal and CUDA inference — The Zig-based engine executes mode…

github.comhttps://github.com/ashhart/TensorFold

sglang-omni: A multi-stage serving runtime for omni, speech, and TTS models.

What problem it solves Interactive multimodal and speech systems often require generation to span heterogeneous stages with different compute patterns, dependency structures, and resource needs. Serving these pipelines requires coordinated stage lifecycles, inter-stage transport, model integrations, and a unified API surface. Key features Coordinated multi-stage runtime — Models generation as coor…

github.comhttps://github.com/sgl-project/sglang-omni

polars: An analytical query engine written in Rust for DataFrame and SQL queries.

What problem it solves Data analysis workflows often require expressive query composition, flexible execution models, and interoperability with columnar data. Polars addresses these needs by allowing the DataFrame API and SQL to share the same optimizer and engine, with support for large workloads and custom extensions. Key features Unified DataFrame and SQL Queries — Queries can be written with e…

github.comhttps://github.com/pola-rs/polars

FreeToken: An edge-native MoE serving engine for running frontier-scale open-weight models on personal and consumer hardware.

What problem it solves Running frontier-scale open-weight models locally requires an inference platform that can make efficient use of the heterogeneous hardware available on a personal device. The project addresses this by treating GPUs, CPUs, host memory, and interconnects as a unified, elastic inference platform. Key features Edge-native MoE runtime — The runtime provides MoE serving with bandw…

github.comhttps://github.com/FlashML-org/FreeToken

OpenShell: A safe, private runtime for fleets of autonomous AI agents.

What problem it solves Autonomous AI agents often need to read files, install packages, call APIs, and use credentials, but unrestricted access can expose host data, secrets, and networks. OpenShell provides these capabilities within an isolated runtime where declared policies constrain agent access. Key features Kernel-level runtime enforcement — Agents run in isolated sandboxes where kernel cont…

github.comhttps://github.com/NVIDIA/OpenShell

SkillSpector: Security scanner for AI agent skills.

What problem it solves AI agent skills execute with implicit trust and often receive minimal vetting before installation. SkillSpector helps users assess whether a skill contains vulnerabilities, malicious patterns, or other security risks before installing it. Key features Multi-format scanning — Scan agent skills supplied as Git repositories, URLs, zip files, directories, or individual files. So…

github.comhttps://github.com/NVIDIA/SkillSpector

jasna: Restores mosaics in videos and images through a simple app and command line that run on your GPU.

What problem it solves Mosaics obscure portions of videos and images, making it difficult to view the affected content without changing the rest of the picture. Users need a restoration workflow that can identify selected mosaic regions and process them with GPU-accelerated video handling. Key features Targeted mosaic restoration — Detects mosaics in videos and restores them while leaving the rest…

github.comhttps://github.com/Kruk2/jasna

Model-Optimizer: A library for composing model optimization techniques and exporting optimized quantized checkpoints.

What problem it solves Deep learning models often need to be adapted for efficient deployment across different inference frameworks. Model Optimizer addresses this by providing a unified set of optimization techniques and export workflows for quantized checkpoints. Key features Post-training quantization — Compress models with quantization techniques available for Hugging Face, Megatron-Bridge, Di…

github.comhttps://github.com/NVIDIA/Model-Optimizer

nvke: A self-hosting management plane for creating and operating Kubernetes clusters on NVIDIA GPU infrastructure.

What problem it solves Platform teams need to create and operate isolated Kubernetes clusters for customers on infrastructure they supply and control. They need a management plane that can provision GPU machines, manage cluster lifecycle operations, and establish node identity before issuing join credentials. Key features Independent workload clusters — Create workload clusters through the NVKE AP…

github.comhttps://github.com/dsx-ai-factory/nvke

LocalAI: An open-source AI engine for running models locally across modalities and hardware.

What problem it solves Running AI models locally can require integrating multiple inference engines, model formats, hardware accelerators, and API surfaces. LocalAI addresses this with a small core, on-demand backends, and compatible APIs that work across supported hardware without requiring a GPU. Key features Composable inference backends — Backends are packaged separately and pulled only when a…

github.comhttps://github.com/mudler/LocalAI

TensorRT-LLM: An open-source library for optimizing LLM and Visual Gen inference on NVIDIA GPUs.

What problem it solves LLM and Visual Gen inference requires specialized kernels, efficient runtime orchestration, and flexible deployment across NVIDIA GPU configurations. Developers also need a modular framework that supports customization while integrating with established inference tooling. Key features Optimized inference kernels — Custom kernels accelerate common inference operations, includ…

github.comhttps://github.com/NVIDIA/TensorRT-LLM

cuda-rust: NVIDIA's CUDA platform for Rust, providing host-side runtime crates and GPU kernel programming models in idiomatic Rust.

What problem it solves Rust developers need host-side access to the CUDA runtime and tools for expressing GPU kernels while preserving Rust's ownership and borrowing rules. The platform provides separate programming models for workloads that require direct thread-level control and workloads naturally expressed through tensor operations. Key features Shared CUDA runtime — Host-side crates provide d…

github.comhttps://github.com/NVIDIA/cuda-rust

auto-ontology: An open library that builds a context layer from enterprise metadata and exposes it to agent harnesses.

What problem it solves Agents can retrieve documents and query databases, but unreliable results undermine agentic analytics. Teams need a context layer that records where data lives, what its definitions mean, whether it is usable, how it should be queried, its provenance, and related concepts. Key features Enterprise ontology generation — Generate semantic ontologies from database metadata, infe…

github.comhttps://github.com/NVIDIA/auto-ontology