NVIDIA cuDF

Headlines · 1

1. 3d
   NVIDIA shows how AI agents rewrite Spark UDFs for GPU speedups with cuDF skills

   NVIDIA published a technical post introducing cuDF skills for Apache Spark, a set of agent-oriented skills that guide coding agents through rewriting Apache Spark user-defined functions (UDFs) into GPU-executable implementations. The post explains that UDFs break Spark's declarative abstraction, forcing otherwise GPU-accelerated pipelines to fall back to row-by-row CPU execution with data-transfer overhead, and outlines three rewrite routes: Spark SQL expressions, the RapidsUDF interface using cuDF's Java API, or custom CUDA kernels via JNI. The skills split the work into four verification-driven loops — scaffolding, conversion, benchmarking and optimization — and ship with a project template, test harnesses and benchmark scaffolding. NVIDIA says a major consumer technology company converted more than 30 production UDFs this way; in one example a UDF used by 3,000+ jobs dropped from 274.55 to 12.78 hours of operator time on a 600-billion-row workload, a roughly 22x speedup, and states rewrites can exceed 20x. The skills are available on GitHub and installable via npx skills add NVIDIA/cudf-spark.

   NVIDIA Technical Blog · 🔥 0

Experience and discussion from the community

Share my NVIDIA cuDF experienceAsk about NVIDIA cuDF