Quobyte, an innovator in high-performance storage solutions, has unveiled a File Query Engine at ISC High Performance 2024.
This innovative addition greatly enhances file system metadata management within massive data storage clusters, bolstering the work of enterprises and researchers alike.
The File Query Engine has exceptional query speeds, making it a perfect fit for data-intensive workloads like AI/ML training. Users can now streamline workflows by labelling files directly with data, eliminating the need to manage separate metadata files.
Administrators also benefit from faster insights, enabling them to quickly identify inactive files consuming storage space or locate files based on user ownership.
The File Query Engine surpasses traditional file system scans, which are often slow and cumbersome for large storage volumes. This is achieved through its intelligent integration with Quobyte’s distributed key-value store.
This store efficiently houses metadata without requiring an additional database layer, streamlining the querying process and minimizing resource consumption. Additionally, parallel execution across all metadata servers ensures rapid scans across the entire cluster or specific volumes, with data consistency guaranteed at all times.
“The File Query Engine is a game-changer for our customers,” said Bjorn Kolbeck, CEO of Quobyte. “It streamlines the process of querying file system metadata, offering fast and efficient results even for large datasets, AI, and machine-learning workloads.’
Seamlessly integrated within Quobyte release 3.22, the File Query Engine eliminates the need for additional configuration. Users can leverage the command-line tool “qmgmt” to execute file metadata queries, with outputs available in user-friendly CSV or JSON formats. The Quobyte API also facilitates query initiation, offering flexibility and ease of use.
