NCSA Brown Dog

Brown Dog

Clients

We aim to build Brown Dog to serve as an Internet resource which a wide variety of applications can then build on top of and leverage in order to allow users to better work with and explore difficult datasets, files, and collections.  The below clients, libraries, and applications are some of the examples the Brown Dog team has been building or supporting.

BD Fiddle

https://browndog.ncsa.illinois.edu/bdfiddle/

 

BD Fiddle provides a convenient web interface for new users to try out Brown Dog's capabilities.  From here users can upload files to execute selected transformations while simultaneously obtaining code within a variety of programming languages to call these capabilities within their own tools.

BD Bookmarklet

https://browndog.ncsa.illinois.edu/bdbookmarklet/

 

This bookmarklet, javascript encapsulated in a bookmark, provides a relatively quick and simple means of accessing Brown Dog capabilities as you browse the web. To use it simply go to the above link, drag & drop the Brown Dog image to your bookmarks, then on new pages simply click on the bookmark.  The bookmarklet will add a menu to all links on the page allowing a user to download/preview the data in any format currently supported as an output (see on YouTube here).  Further if you press Ctrl+F the bookmarklet will pop up a search window allowing you to index the contents of the page and search them based on extracted metadata/tags (e.g. objects within an image).

BD Command Line Interface

> pip install bdcli

 

The "bd" command line tool.  Allows one to identify conversion options for a file/dataset, convert files/datasets, index collections, and query indexed collections (see on YouTube here).  Users can optionally utilize the "--big-data" flag to invert the service in the case of large files, moving the needed transformation tools as dockerized containers and executing them locally rather than attempting to move a large amount of data.

BD Windows Client

BDWindowsClient.msi

 

Client for the Windows operating system allowing users to navigate local folders in the native file manager by indexing them based on information extracted from the contents of the contained files and allowing the user to transform files to formats that are more readily viewable.

Libraries

Python Library

pip install browndog

 

A native python library containing functions wrapping the Brown Dog DTS REST interface.

Matlab Library

bd.m

 

A native Matlab library containing functions wrapping the Brown Dog DTS REST interface.

R Library

install_github("NCSABrownDog/bd.r")

 

A native R library containing functions wrapping the Brown Dog DTS REST interface.

Javascript Library

bd.js

 

A native Javascript library containing code wrapping the Brown Dog DTS REST interface.  Allows for on the fly conversions and extractions  to occur within HTML by adding a "dap=output format" or dts="" attribute to your link and image tags respectively (see on YouTube here).

Community Tools

Clowder

https://clowder.ncsa.illinois.edu

 

Open source web based research data management system designed to support multiple research domains and the diverse data types utilized across those domains.  In addition to data sharing and organizational functionality Clowder contains major extension points for the preprocessing, processing, previewing, and publication of data.  Utilized within Brown Dog to support extractions, Clowder supports the data needs across communities such as biology, geoscience, material science, social science, medicine, cultural heritage, as well as industry.

PEcAn

http://pecanproject.github.io/

 

A scientific workflow and data assimilation system connecting a variety of models within the Ecology community to a variety of data sources towards regional-scale high resolution estimates of plant life and carbon storage, in a manner that can leverage HPC resources, and promote reproducibility as well as usability via a user friendly web interface.  Supports transformations within Brown Dog to convert across data and model formats (see on YouTube here).

Ecological Modeling Data Portal

http://pecan.ncsa.illinois.edu/shiny/BrownDog/

 

A proof of concept custom on-the-fly data assimilation portal, leveraging the Brown Dog/PEcAn collaboration, in support of the ecological modeling community allowing data from various commonly used data products to be obtained in the formats required to run a number of community constructed ecological models.

Jupyter

https://opensource.ncsa.illinois.edu/bitbucket/projects/BD/repos/bd-jupyter/

 

Example Jupyter notebooks for calling Brown Dog DTS transformations from within Python.

GIS Tools (alpha)

https://opensource.ncsa.illinois.edu/bitbucket/projects/BD/repos/bd-gistools/

 

Example modules calling Brown Dog DTS transformations from within popular GIS tools such as ArcGIS and QGIS.

Excel (alpha)

https://opensource.ncsa.illinois.edu/bitbucket/projects/BD/repos/bd-office-addins/

 

Example addin for Excel calling Brown Dog DTS transformations from within a spreadsheet, taking specified rows/cols as inputs and generating new derived rows/columns based on extractable data (e.g. example generating the greeness index along a city path represented by a set of lat/long coordinates).

Development Tools

Brown Dog Development Base

git clone https://opensource.ncsa.illinois.edu/bitbucket/scm/bd/bd-base.git

http://browndog.ncsa.illinois.edu/downloads/bd-base.ova

 

The Brown Dog base development environment with dockerized versions of each of the services required to setup and work against the Brown Dog Data Transformation Service when developing new extractors or converters (i.e. Clowder, Polyglot, Fence, RabbitMQ, MongoDB, Redist, an example extractor, an example converter, and the BD CLI). Uses the tmux utility to spawn the terminals of each service in a separate window. Also available as a Virtual Machine image.

pyClowder2

git clone https://opensource.ncsa.illinois.edu/bitbucket/scm/cats/pyclowder2.git

 

Python utility library simplifying the process of adding an analysis tool as a an extractor, wrapping most interactions with Clowder as python functions.

Software Server

git clone https://opensource.ncsa.illinois.edu/bitbucket/scm/pol/polyglot.git

 

Light weight web utility used to add a REST API onto arbitrary software and tools. Component within the Polyglot framework/repository used in the creation of converters.

Extractor Template

 

git clone https://opensource.ncsa.illinois.edu/bitbucket/scm/bd/extractors-template.git

 

docker pull browndog/extractors-template

 

An example extractor containing the Clowder wrapper script (utilizing pyClowder2) and a Dockerfile for an example tool.

Converter Template

 

git clone https://opensource.ncsa.illinois.edu/bitbucket/scm/bd/converters-template.git

 

docker pull browndog/converters-template

 

An example converter containing a Polyglot wrapper script and a Dockerfile for an example tool.

Partners

University of Illinois at Urbana-Champaign

Boston University

University of Maryland

Southern Methodist University

 

This material is based upon work supported by the National Science Foundation under Grant No. ACI-1261582.

Nsf.gov

Learn more about this award

 

 

Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.