NCSA Brown Dog
We aim to build Brown Dog to serve as an internet resource which a wide variety of applications can then build on top of and leverage in order to allow users to better work with and explore difficult datasets, files, and collections. The below clients, libraries, and applications are some of the examples the Brown Dog team has been building or supporting.
BD Fiddle provides a convenient web interface for new users to try out Brown Dog's capabilities. From here users can upload files to execute selected transformations while simultaneously obtaining code within a variety of programming languages to call these capabilities within their own tools.
The "bd" command line tool. Allows one to identify conversion options for a file/dataset, convert files/datasets, index collections, and query indexed collections (see on YouTube here). Users can optionally utilize the "--big-data" flag to invert the service in the case of large files, moving the needed transformation tools and executing them locally rather than attempting to the file itself.
A plugin for the Windows operating system allowing users to navigate local folders by indexing them based on information extracted from the contents of the contained files and allowing the user to transform specific files to formats that are more readily viewable.
An extension for Google Chrome providing in page access to Brown Dog capabilities. Like the bookmarklet above the browser extension provides a search box by which to search for site contents based on extracted metadata/tags/signatures and convert files from difficult to use/view formats (see on YouTube here).
A major component of DataNet SEAD, this open source web based content management system supports data sharing and visualization, social curation, auto-curation, and data publication within matched institutional repositories. Utilized within Brown Dog to support data extractions, Clowder is able to leverage the transformations supported within Brown Dog to extract additional information and provide conversion capabilities to files and datasets uploaded to it.
A scientific workflow and data assimilation system connecting a variety of models within the Ecology community to a variety of data sources towards regional-scale high resolution estimates of plant life and carbon storage, in a manner that can leverage HPC resources, and promote reproducibility as well as usability via a user friendly web interface. Supports transformations within Brown Dog to convert across data and model formats (see on YouTube here).
iRODS (in discussion)
A major component of the DataNet Federation Consortium, the Integrated Rule-Oriented Data System (iRODS) is an open source federated data management software which virtualizes data storage resources and execution of data processing tasks across those storage resources allowing users to more easily manage/archive their data.
A native python library containing functions wrapping the Brown Dog REST interface.
A native Matlab library containing functions wrapping the Brown Dog REST interface.
A native R library containing functions wrapping the Brown Dog REST interface.
Tmux script which pulls docker images of each of the components needed to stand up a Brown Dog instances (i.e. Fence, Clowder, Polyglot, an example extractor, an example converter, and the command line interface as a minimal means of interacting with the service). Useful when building new converters and extractors in order to test if they are interfacing properly.
Python library which simplifies the process of adding an analysis tool as a an extractor (see example here).
An example converter containing: the Brown Dog wrapper script for Polyglot, and a Dockerfile for the tool.
An example extractor containing: the Brown Dog wrapper script for Clowder utilizing pyClowder, and a Dockerfile for the tool.
University of Illinois at Urbana-Champaign
University of Maryland
Southern Methodist University
This material is based upon work supported by the National Science Foundation under Grant No. ACI-1261582.
Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.