NCSA Brown Dog
We aim to build Brown Dog to serve as an Internet resource which a wide variety of applications can then build on top of and leverage in order to allow users to better work with and explore difficult datasets, files, and collections. The below clients, libraries, and applications are some of the examples the Brown Dog team has been building or supporting.
BD Fiddle provides a convenient web interface for new users to try out Brown Dog's capabilities. From here users can upload files to execute selected transformations while simultaneously obtaining code within a variety of programming languages to call these capabilities within their own tools.
> pip install bdcli
The "bd" command line tool. Allows one to identify conversion options for a file/dataset, convert files/datasets, index collections, and query indexed collections (see on YouTube here). Users can optionally utilize the "--big-data" flag to invert the service in the case of large files, moving the needed transformation tools as dockerized containers and executing them locally rather than attempting to move a large amount of data.
Client for the Windows operating system allowing users to navigate local folders in the native file manager by indexing them based on information extracted from the contents of the contained files and allowing the user to transform files to formats that are more readily viewable.
Open source web based research data management system designed to support multiple research domains and the diverse data types utilized across those domains. In addition to data sharing and organizational functionality Clowder contains major extension points for the preprocessing, processing, previewing, and publication of data. Utilized within Brown Dog to support extractions, Clowder supports the data needs across communities such as biology, geoscience, material science, social science, medicine, cultural heritage, as well as industry.
A scientific workflow and data assimilation system connecting a variety of models within the Ecology community to a variety of data sources towards regional-scale high resolution estimates of plant life and carbon storage, in a manner that can leverage HPC resources, and promote reproducibility as well as usability via a user friendly web interface. Supports transformations within Brown Dog to convert across data and model formats (see on YouTube here).
A proof of concept custom on-the-fly data assimilation portal, leveraging the Brown Dog/PEcAn collaboration, in support of the ecological modeling community allowing data from various commonly used data products to be obtained in the formats required to run a number of community constructed ecological models.
Example Jupyter notebooks for calling Brown Dog DTS transformations from within Python.
Example modules calling Brown Dog DTS transformations from within popular GIS tools such as ArcGIS and QGIS.
Example addin for Excel calling Brown Dog DTS transformations from within a spreadsheet, taking specified rows/cols as inputs and generating new derived rows/columns based on extractable data (e.g. example generating the greeness index along a city path represented by a set of lat/long coordinates).
Brown Dog Development Base
The Brown Dog base development environment with dockerized versions of each of the services required to setup and work against the Brown Dog Data Transformation Service when developing new extractors or converters (i.e. Clowder, Polyglot, Fence, RabbitMQ, MongoDB, Redist, an example extractor, an example converter, and the BD CLI). Uses the tmux utility to spawn the terminals of each service in a separate window. Also available as a Virtual Machine image.
Python utility library simplifying the process of adding an analysis tool as a an extractor, wrapping most interactions with Clowder as python functions.
Light weight web utility used to add a REST API onto arbitrary software and tools. Component within the Polyglot framework/repository used in the creation of converters.
An example extractor containing the Clowder wrapper script (utilizing pyClowder2) and a Dockerfile for an example tool.
An example converter containing a Polyglot wrapper script and a Dockerfile for an example tool.
University of Illinois at Urbana-Champaign
University of Maryland
Southern Methodist University
This material is based upon work supported by the National Science Foundation under Grant No. ACI-1261582.
Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.
Kenton McHenry, PI email@example.com
Shannon Bradley, Project Manager, firstname.lastname@example.org
How to Acknowledge
S. Padhy, G. Jansen, J. Alameda, E. Black, L. Diesendruck, M. Dietze, P. Kumar, R. Kooper, J. Lee, R. Liu, R. Marciano, L. Marini, D. Mattson, B. Minsker, C. Navarro, M. Slavenas, W. Sullivan, J. Votava, I. Zharnitsky, K. McHenry, "Brown Dog : Leveraging Everything Towards Autocuration", IEEE Big Data, 2015