Main Page

From Daxtoolkit
Revision as of 15:51, 3 April 2014 by Kmorel (Talk | contribs)

(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to: navigation, search

Data Analysis at Extreme: The Dax Toolkit

The transition to exascale machines represents a fundamental change in computing architecture. Efficient computation on exascale machines requires a massive amount of concurrent threads, at least 1000× more concurrency than existing systems. Current visualization solutions cannot support this extreme level of concurrency. Exascale systems require a new programming model and a fundamental change in how we design fundamental algorithms. To address these issues, our project builds the Data Analysis at Extreme (Dax) Toolkit.

A Toolkit for Exascale Data Analysis

Conceptual Operation

The Dax Toolkit supports the fine-grained concurrency for data analysis and visualization algorithms required to drive exascale computing. The basic computational unit of the Dax Toolkit is a worklet, a function that implements the algorithm’s behavior on an element of a mesh (that is, a point, edge, face, or cell) or a small local neighborhood. The worklet is constrained to be serial and stateless; it can access only the element passed to and from the invocation. With this constraint, the serial worklet function can be concurrently executed on an unlimited number of threads without the complications of memory clashes or other race conditions.

The Dax Toolkit provides dispatchers that apply worklets to all elements in an input mesh, the results of which are collected into a resulting mesh. Although worklets are not allowed communication, many visualization algorithms require operations such as variable array packing and coincident topology resolution that intrinsically require significant coordination among threads. Dax enables such algorithms by classifying and implementing the most common and versatile communicative operations, which, when used in conjunction with the appropriate worklets, complete the visualization algorithms.

Simplified Parallel Programming

int vtkCellDerivatives::RequestData(...)
  ...[allocate output arrays]...
  ...[validate inputs]...
  for (cellId=0; cellId < numCells; cellId++)
    ...[update progress]...
    input->GetCell(cellId, cell);
    subId = cell->GetParametricCenter(pcoords);
    scalars = cellScalars->GetPointer(0);
      subId, pcoords, scalars, 1, derivs);
    outGradients->SetTuple(cellId, derivs);
struct CellGradient : public dax::exec::WorkletMapCell
  typedef void ControlSignature(
      Topology, Field(Point), Field(Point), Field(Out));
  typedef _4 ExecutionSignature(_1,_2,_3);
  template<class CellTag> DAX_EXEC_EXPORT
  dax::Vector3 operator()(...)
    dax::Vector3 parametricCellCenter =
    return dax::exec::CellDerivative(parametricCellCenter,
VTK Code Dax Code

The Dax Toolkit simplifies the development of parallel visualization algorithms. Consider the code samples above that come from the Visualization Toolkit (VTK) on the left and our Dax Toolkit on the right. Both implementations perform the same operation; they estimate gradients using finite differences. Both toolkits provide similar classes and functions, and consequently the code looks remarkably similar.

However, because the Dax Toolkit is structured such that it can execute on a GPU, we measure that it performs this operation over 100 times faster than the VTK code running on a single CPU. Furthermore, the Dax API can be switched to a different device by changing only a single line of code. Dax currently provides support for CUDA (GPU), OpenMP (multicore CPU), and serial execution.

Using Dax


Kenneth Moreland
Sandia National Laboratories

Kwan-Liu Ma
University of California at Davis

Berk Geveci
Kitware, Inc.

Utkarsh Ayachit
Kitware, Inc.

Sandia National Laboratories is a multi-program laboratory managed and operated by Sandia Corporation, a wholly owned subsidiary of Lockheed Martin Corporation, for the U.S. Department of Energy's National Nuclear Security Administration under contract DE-AC04-94AL85000. SAND 2012-2507P


"A Pervasive Parallel Framework for Visualization: Final Report for FWP 10-014707." Kenneth Moreland. Tech Report SAND 2014-0047, Sandia National Laboratories, January 2014.

"A Classification of Scientific Visualization Algorithms for Massive Threading." Kenneth Moreland, Berk Geveci, Kwan-Liu Ma, and Robert Maynard. In Proceedings of the Ultrascale Visualization Workshop, November 2012. DOI 10.1145/2535571.2535591.

"Research Challenges for Visualization Software." Hank Childs, Berk Geveci, Will Schroeder, Jeremy Meredith, Kenneth Moreland, Christopher Sewell, Torsten Kuhlen, and E. Wes Bethel. IEEE Computer, 46(5), May 2013, pg. 34-42. DOI 10.1109/MC.2013.179.

"A Survey of Visualization Pipelines." Kenneth Moreland. IEEE Transactions on Visualization and Computer Graphics, 19(3), March 2013. DOI 10.1109/TVCG.2012.133.

"Optimizing Threshold for Extreme Scale Analysis." Robert Maynard, Kenneth Moreland, Utkarsh Ayachit, Berk Geveci, and Kwan-Liu Ma. In Proceedings of SPIE Visualization and Data Analysis, February 2013.

"Flexible Analysis Software for Emerging Architectures." Kenneth Moreland, Brad King, Robert Maynard, and Kwan-Liu Ma. In 2012 SC Companion (Proceedings of Petascale Data Analytics: Challenges and Opportunities), November 2012. DOI 10.1109/SC.Companion.2012.115.

"The SDAV Software Frameworks for Visualization and Analysis on Next-Generation Multi-Core and Many-Core Architectures." Christopher Sewell, Jeremy Meredith, Kenneth Moreland, Tom Peterka, Dave DeMarle, La-ta Lo, James Ahrens, Robert Maynard, and Berk Geveci. In 2012 SC Companion (Proceedings of the Ultrascale Visualization Workshop), November 2012, pg. 206-214. DOI 10.1109/SC.Companion.2012.36.

"Oh, $#*@! Exascale! The Effect of Emerging Architectures on Scientific Discovery." Kenneth Moreland. In 2012 SC Companion (Proceedings of the Ultrascale Visualization Workshop), November 2012, pg. 224-231. DOI 10.1109/SC.Companion.2012.38.

"Dax Toolkit: A Proposed Framework for Data Analysis and Visualization at Extreme Scale." Kenneth Moreland, Utkarsh Ayachit, Berk Geveci, and Kwan-Liu Ma. In IEEE Symposium on Large-Scale Data Analysis and Visualization (LDAV), October 2011.

"Dax: Data Analysis at Extreme." Kenneth Moreland, Utkarsh Ayachit, Berk Geveci, and Kwan-Liu Ma. In Proceedings of SciDAC 2011, July 2011.

For Developers