![]() ![]() Stocator is a generic connector, that may contain various implementations for object stores. It doesn’t depends on the Hadoop modules and interacts directly with object stores. Stocator is implicitly designed for the object stores, it has very a different architecture from the existing Hadoop connector. It’s clear that Hadoop is designed to work with file systems and not object stores. This leads to dozens of useless requests targeted at the object store. The temp files and folders it uses for every write operation are renamed, copied, and deleted. Moreover, Hadoop Map Reduce Client is designed to work with file systems and not object stores. (these are not native object store operations). This means they support many more operations, such as shell operations on directories, including move, copy, rename, etc. Hadoop connectors, however, must be compliant with the Hadoop ecosystem. Specifically, Apache Spark requires object listing, objects creation, read objects, and getting data partitions. To access an object store, Apache Spark uses Hadoop modules that contain connectors to the various object stores.Īpache Spark needs only small set of object store functionalities. Stocator - Storage Connector for Apache SparkĪpache Spark can work with multiple data sources that include various object stores like IBM Cloud Object Storage, OpenStack Swift and more. ![]()
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |