Uintah and Related C-SAFE Publications


M. Berzins, J. Luitjens, Q. Meng, T. Harman, C.A. Wight, J.R. Peterson. “Uintah: A Scalable Framework for Hazard Analysis,” In Proceedings of the Teragrid 2010 Conference, TG 10, Note: Awarded Best Paper in the Science Track!, pp. (published online). July, 2010.
ISBN: 978-1-60558-818-6
DOI: 10.1145/1838574.1838577


The Uintah Software system was developed to provide an environment for solving a fluid-structure interaction problems on structured adaptive grids on large-scale, long-running, data-intensive problems. Uintah uses a novel asynchronous task-based approach with fully automated load balancing. The application of Uintah to a petascale problem in hazard analysis arising from "sympathetic" explosions in which the collective interactions of a large ensemble of explosives results in dramatically increased explosion violence, is considered. The advances in scalability and combustion modeling needed to begin to solve this problem are discussed and illustrated by prototypical computational results.

Keywords: Uintah, csafe

R.M. Brannon, S. Leelavanichkul. “A Multi-Stage Return Algorithm for Solving the Classical Damage Component of Constitutive Models for Rocks, Ceramics, and Other Rock-Like Media,” In International Journal of Fracture, Vol. 163, No. 1-2, pp. 133--149. 2010.
DOI: 10.1007/s10704-009-9398-4


Classical plasticity and damage models for porous quasi-brittle media usually suffer from mathematical defects such as non-convergence and non-uniqueness. Yield or damage functions for porous quasi-brittle media often have yield functions with contours so distorted that following those contours to the yield surface in a return algorithm can take the solution to a false elastic domain. A steepest-descent return algorithm must include iterative corrections; otherwise, the solution is non-unique because contours of any yield function are non-unique. A multi-stage algorithm has been developed to address both spurious convergence and non-uniqueness, as well as to improve efficiency. The region of pathological isosurfaces is masked by first returning the stress state to the Drucker–Prager surface circumscribing the actual yield surface. From there, steepest-descent is used to locate a point on the yield surface. This first-stage solution, which is extremely efficient because it is applied in a 2D subspace, is generally not the correct solution, but it is used to estimate the correct return direction. The first-stage solution is projected onto the estimated correct return direction in 6D stress space. Third invariant dependence and anisotropy are accommodated in this second-stage correction. The projection operation introduces errors associated with yield surface curvature, so the two-stage iteration is applied repeatedly to converge. Regions of extremely high curvature are detected and handled separately using an approximation to vertex theory. The multi-stage return is applied holding internal variables constant to produce a non-hardening solution. To account for hardening from pore collapse (or softening from damage), geometrical arguments are used to clearly illustrate the appropriate scaling of the non-hardening solution needed to obtain the hardening (or softening) solution.

J.A. Burghardt, B. Leavy, J. Guilkey, Z. Xue, R.M. Brannon. “Application of Uintah-MPM to Shaped Charge Jet Penetration of Aluminum,” In IOP Conference Series: Materials Science and Engineering, Vol. 10, No. 1, pp. 012223. 2010.


The capability of the generalized interpolation material point (GIMP) method in simulation of penetration events is investigated. A series of experiments was performed wherein a shaped charge jet penetrates into a stack of aluminum plates. Electronic switches were used to measure the penetration time history. Flash x-ray techniques were used to measure the density, length, radius and velocity of the shaped charge jet. Simulations of the penetration event were performed using the Uintah MPM/GIMP code with several different models of the shaped charge jet being used. The predicted penetration time history for each jet model is compared with the experimentally observed penetration history. It was found that the characteristics of the predicted penetration were dependent on the way that the jet data are translated to a discrete description. The discrete jet descriptions were modified such that the predicted penetration histories fell very close to the range of the experimental data. In comparing the various discrete jet descriptions it was found that the cumulative kinetic energy flux curve represents an important way of characterizing the penetration characteristics of the jet. The GIMP method was found to be well suited for simulation of high rate penetration events.

B. Leavy, R.M. Brannon, O.E. Strack. “The Use of Sphere Indentation Experiments to Characterize Ceramic Damage Models,” In International Journal of Applied Ceramic Technology, Vol. 7, No. 5, pp. 606--615. September/October, 2010.
DOI: 10.1111/j.1744-7402.2010.02487.x


Sphere impact experiments are used to calibrate and validate ceramic models that include statistical variability and/or scale effects in strength and toughness parameters. These dynamic experiments supplement traditional characterization experiments such as tension, triaxial compression, Brazilian, and plate impact, which are commonly used for ceramic model calibration. The fractured ceramic specimens are analyzed using sectioning, X-ray computed tomography, microscopy, and other techniques. These experimental observations indicate that a predictive material model must incorporate a standard deviation in strength that varies with the nature of the loading. Methods of using the spherical indentation data to calibrate a statistical damage model are presented in which it is assumed that variability in strength is tied to microscale stress concentrations associated with microscale heterogeneity.

J. Luitjens, M. Berzins. “Improving the Performance of Uintah: A Large-Scale Adaptive Meshing Computational Framework,” In Proceedings of the 24th IEEE International Parallel and Distributed Processing Symposium (IPDPS10), Atlanta, GA, pp. 1--10. 2010.
DOI: 10.1109/IPDPS.2010.5470437


Uintah is a highly parallel and adaptive multi-physics framework created by the Center for Simulation of Accidental Fires and Explosions in Utah. Uintah, which is built upon the Common Component Architecture, has facilitated the simulation of a wide variety of fluid-structure interaction problems using both adaptive structured meshes for the fluid and particles to model solids. Uintah was originally designed for, and has performed well on, about a thousand processors. The evolution of Uintah to use tens of thousands processors has required improvements in memory usage, data structure design, load balancing algorithms and cost estimation in order to improve strong and weak scalability up to 98,304 cores for situations in which the mesh used varies adaptively and also cases in which particles that represent the solids move from mesh cell to mesh cell.

Keywords: csafe, c-safe, scirun, uintah, fires, explosions, simulation

J. Luitjens, J. Guilkey, T. Harman, B. Worthen, S.G. Parker. “Adaptive Computations in the Uintah Framework,” In Advanced Computational Infastructures for Parallel/Distributed Adapative Applications, Ch. 1, Wiley Press, 2010.

J. Luitjens. “The Scalability of Parallel Adaptive Mesh Refinement Within Uintah,” School of Computing, University of Utah, 2010.


Solutions to Partial Differential Equations (PDEs) are often computed by discretizing the domain into a collection of computational elements referred to as a mesh. This solution is an approximation with an error that decreases as the mesh spacing decreases. However, decreasing the mesh spacing also increases the computational requirements. Adaptive mesh refinement (AMR) attempts to reduce the error while limiting the increase in computational requirements by refining the mesh locally in regions of the domain that have large error while maintaining a coarse mesh in other portions of the domain. This approach often provides a solution that is as accurate as that obtained from a much larger fixed mesh simulation, thus saving on both computational time and memory. However, historically, these AMR operations often limit the overall scalability of the application.

Adapting the mesh at runtime necessitates scalable regridding and load balancing algorithms. This dissertation analyzes the performance bottlenecks for a widely used regridding algorithm and presents two new algorithms which exhibit ideal scalability. In addition, a scalable space-filling curve generation algorithm for dynamic load balancing is also presented. The performance of these algorithms is analyzed by determining their theoretical complexity, deriving performance models, and comparing the observed performance to those performance models. The models are then used to predict performance on larger numbers of processors. This analysis demonstrates the necessity of these algorithms at larger numbers of processors. This dissertation also investigates methods to more accurately predict workloads based on measurements taken at runtime. While the methods used are not new, the application of these methods to the load balancing process is. These methods are shown to be highly accurate and able to predict the workload within 3% error. By improving the accuracy of these estimations, the load imbalance of the simulation can be reduced, thereby increasing the overall performance.

Finally, the scalability of AMR simulations as a whole using these algorithms is tested within the Uintah computational framework. Scalability tests are performed using up to 98,304 processors and nearly ideal scalability is demonstrated.

Q. Meng, J. Luitjens, M. Berzins. “Dynamic Task Scheduling for Scalable Parallel AMR in the Uintah Framework,” SCI Technical Report, No. UUSCI-2010-001, SCI Institute, University of Utah, 2010.

Q. Meng, J. Luitjens, M. Berzins. “Dynamic Task Scheduling for the Uintah Framework,” In Proceedings of the 3rd IEEE Workshop on Many-Task Computing on Grids and Supercomputers (MTAGS10), pp. 1--10. 2010.
DOI: 10.1109/MTAGS.2010.5699431


Uintah is a computational framework for fluid-structure interaction problems using a combination of the ICE fluid flow algorithm, adaptive mesh refinement (AMR) and MPM particle methods. Uintah uses domain decomposition with a task-graph approach for asynchronous communication and automatic message generation. The Uintah software has been used for a decade with its original task scheduler that ran computational tasks in a predefined static order. In order to improve the performance of Uintah for petascale architecture, a new dynamic task scheduler allowing better overlapping of the communication and computation is designed and evaluated in this study. The new scheduler supports asynchronous, out-of-order scheduling of computational tasks by putting them in a distributed directed acyclic graph (DAG) and by isolating task memory and keeping multiple copies of task variables in a data warehouse when necessary. A new runtime system has been implemented with a two-stage priority queuing architecture to improve the scheduling efficiency. The effectiveness of this new approach is shown through an analysis of the performance of the software on large scale fluid-structure examples.

J. Schmidt, M. Berzins. “Development of the Uintah Gateway for Fluid-Structure-Interaction Problems,” In Proceedings of the Teragrid 2010 Conference, ACM, 2010.
DOI: 10.1145/1838574.1838591

P.C. Wallstedt, J.E. Guilkey. “A Weighted Least Squares Particle-In-Cell Method for Solid Mechanics,” In International Journal for Numerical Methods in Engineering, Vol. 85, No. 13, pp. 1687--1704. April, 2010.
DOI: 10.1002/nme.3041


A novel meshfree method is proposed that incorporates features of the material point (MPM) and generalized interpolation material point (GIMP) methods and can be used within an existing MPM/GIMP implementation. Weighted least squares kernel functions are centered at stationary grid nodes and used to approximate field values and gradients. Integration is performed over cells of the background grid and material boundaries are approximated with an implicit surface. The proposed method avoids nearest-neighbor searches while significantly improving accuracy over MPM and GIMP. Implementation is discussed in detail and several example problems are solved, including one manufactured solution which allows measurement of dynamic, non-linear, large deformation performance. Advantages and disadvantages of the method are discussed.

Keywords: generalized interpolation material point method, meshfree, marching cubes, weighted least squares, implicit surface, MPM, GIMP, PIC


J. Guilkey, T. Harman, J. Luitjens, J. Schmidt, J. Thornock, J.D. de St. Germain, S. Shankar, J. Peterson, C. Brownlee. “Uintah User Guide Version 1.1,” SCI Technical Report, No. UUSCI-2009-007, SCI Institute, University of Utah, 2009.

T.L. Henriksen, G.J. Nathan, Z.T. Alwahabi, N. Qamar, T.A. Ring, E.G. Eddings. “Planar Measurements of Soot Volume Fraction and OH in a JP-8 Pool Fire,” In Combustion and Flame, Vol. 156, No. 7, pp. 1480--1492. 2009.
DOI: 10.1016/j.combustflame.2009.03.002


The simultaneous measurement of soot volume fraction by laser induced incandescence (LII) and qualitative imaging of OH by laser induced fluorescence (LIF) was performed in a JP-8 pool fire contained in a 152 mm diameter pan. Line of sight extinction was used to calibrate the LII system in a laminar flame, and to provide an independent method of measuring average soot volume fraction in the turbulent flame. The presence of soot in the turbulent flame was found to be approximately 50% probable, resulting in high levels of optical extinction, which increased slightly through the flame from approximately 30% near the base, to approximately 50% at the tip. This high soot loading pushes both techniques toward their detection limit. Nevertheless, useful accuracy was obtained, with the LII measurement of apparent extinction in the turbulent flame being approximately 21% lower than a direct measurement, consistent with the influence of signal trapping. The axial and radial distributions of soot volume fraction are presented, along with PDFs of volume fraction, and new insight into the behavior of soot sheets in pool fires are sought from the simultaneous measurements of OH and LII.

Keywords: Incandescence, Fluorescence, Pool fire, JP-8

J.B. Hooper, D. Bedrov, G.D. Smith, B. Hanson, O. Borodin, D.M. Dattelbaum, E.M. Kober. “A molecular dynamics simulation study of the pressure-volume-temperature behavior of polymers under high pressure,” In Journal of Chemical Physics, 2009.
DOI: 10.1063/1.3077868


Isothermal compression of poly (dimethylsiloxane), 1,4-poly(butadiene), and a model Estane® (in both pure form and a nitroplasticized composition similar to PBX-9501 binder) at pressures up to 100kbars has been studied using atomistic molecular dynamics (MD) simulations. Comparison of predicted compression, bulk modulus, and Us--up behavior with experimental static and dynamic compression data available in the literature reveals good agreement between experiment and simulation, indicating that MD simulations utilizing simple quantum-chemistry-based potentials can be used to accurately predict the behavior of polymers at relatively high pressure. Despite their very different zero-pressure bulk moduli, the compression, modulus, and Us--up behavior (including low-pressure curvature) for the three polymers could be reasonably described by the Tait equation of state(EOS) utilizing the universal C parameter. The Tait EOS was found to provide an excellent description of simulation PVT data when the C parameter was optimized for each polymer. The Tait EOS parameters, namely, the zero-pressure bulk modulus and the C parameter, were found to correlate well with free volume for these polymers as measured in simulations by a simple probe insertion algorithm. Of the polymers studied, PDMS was found to have the most free volume at low pressure, consistent with its lower ambient pressurebulk modulus and greater increase in modulus with increasing pressure (i.e., crush-up behavior).

L.K. Huynh, H.R. Zhang, S. Zhang, E.G. Eddings, A.F. Sarofim, M.E. Law, P.R. Westmoreland, T.N. Truong. “Kinetics of Enol Formation from Reaction of OH with Propene,” In Journal of Physical Chemistry, Vol. 113, No. 13, pp. 3177--3185. 2009.
DOI: 10.1021/jp808050j


Kinetics of enol generation from propene has been predicted in an effort to understand the presence of enols in flames. A potential energy surface for reaction of OH with propene was computed by CCSD(T)/cc-pVDZ//B3LYP/cc-pVTZ calculations. Rate constants of different product channels and branching ratios were then calculated using the Master Equation formulation (J. Phys. Chem. A 2006, 110, 10528). Of the two enol products, ethenol is dominant over propenol, and its pathway is also the dominant pathway for the OH + propene addition reactions to form bimolecular products. In the temperature range considered, hydrogen abstraction dominated propene + OH consumption by a branching ratio of more than 90%. Calculated rate constants of enol formation were included in the Utah Surrogate Mechanism to model the enol profile in a cyclohexane premixed flame. The extended model shows consistency with experimental data and gives 5% contribution of ethenol formation from OH + propene reaction, the rest coming from ethene + OH.

J. Luitjens, M. Berzins. “Uintah: A Scalable Adaptive Framework for Emerging Petascale Platforms,” SCI Technical Report, No. UUSCI-2009-002, SCI Institute, University of Utah, 2009.

J. Van Rij, T. Ameel, T. Harman. “An evaluation of secondary effects on microchannel frictional and convective heat transfer characteristics,” In International Journal of Heat and Mass Transfer, Vol. 52, No. 11-12, pp. 2792--2801. 2009.
DOI: 10.1016/j.ijheatmasstransfer.2009.01.001


The frictional and convective heat transfer characteristics of rarified flows in rectangular microchannels, with either isoflux or isothermal boundary conditions, are evaluated subject to second-order slip boundary conditions, creep flow, viscous dissipation, and axial conduction effects. Numerical results are obtained using a continuum based, three-dimensional, compressible, unsteady computational fluid dynamics algorithm with first- and second-order slip velocity and temperature jump boundary conditions applied to the momentum and energy equations, respectively. The results, reported in the form of Poiseuille and Nusselt numbers, are found to be significant functions of aspect ratio, Knudsen number, slip model parameters, Brinkman number, and Peclet number.

Keywords: Rarified flow, Viscous dissipation, Axial conduction

J. Van Rij, T. Ameel, T. Harman. “The Effect of Viscous Dissipation and Rarefaction on Rectangular Microchannel Convective Heat Transfer,” In International Journal of Thermal Sciences, Vol. 48, No. 2, pp. 271--281. February, 2009.
DOI: 10.1016/j.ijthermalsci.2008.07.010


The effect of viscous dissipation and rarefaction on rectangular microchannel convective heat transfer rates, as given by the Nusselt number, is numerically evaluated subject to constant wall heat flux (H2) and constant wall temperature (T) thermal boundary conditions. Numerical results are obtained using a continuum based, three-dimensional, compressible, unsteady computational fluid dynamics algorithm with slip velocity and temperature jump boundary conditions applied to the momentum and energy equations, respectively. For the limiting case of parallel plate channels, analytic solutions for the thermally and hydrodynamically fully developed momentum and energy equations are derived, subject to both first- and second-order slip velocity and temperature jump boundary conditions, from which analytic Nusselt number solutions are then obtained. Excellent agreement between the analytical and numerical results verifies the accuracy of the numerical algorithm, which is then employed to obtain three-dimensional rectangular channel and thermally/hydrodynamically developing Nusselt numbers. Nusselt number data are presented as functions of Knudsen number, Brinkman number, Peclet number, momentum and thermal accommodation coefficients, and aspect ratio. Rarefaction and viscous dissipation effects are shown to significantly affect the convective heat transfer rate in the slip flow regime.

Keywords: Microchannel, Nusselt number, Slip flow, Brinkman number, Viscous dissipation

M. Steffen. “Analysis-Guided Improvements of the Material Point Method,” School of Computing, University of Utah, 2009.


The Material Point Method (MPM) has shown itself to be a powerful tool in the simulation of large deformation problems, especially those involving complex geometries and contact where typical finite element type methods frequently fail. While these large complex problems lead to some impressive simulations and solutions, there has been a lack of basic analysis characterizing the errors present in the method, even on the simplest of problems. However, like most methods which employ mixed Lagrangian (particle) and Eulerian strategies, analysis of the method is not straight forward. The lack of an analysis framework for MPM, as is found in finite element methods, makes it challenging to explain anomalies found in its employment and makes it difficult to propose methodology improvements with predictable outcomes. In this dissertation, we provide a formal analysis of the errors in MPM and use this analysis to direct proposed improvements. In particular, we will focus on how the lack of regularity in the grid functions used for representing the solution can hamper both spatial and temporal convergence of the method. We will show how the use of smoother basis functions, such as B-splines, can improve the accuracy of the method. An in-depth analysis of the current time stepping methods will help to explain behavior currently demonstrated numerically in the literature and will allow users of the method to understand the balance of spatial and temporal errors in MPM. Lastly, extrapolation techniques will be proposed to improve quadrature errors in the method.

L.T. Tran, J. Kim, M. Berzins. “Solving Time-Dependent PDEs using the Material Point Method, A Case Study from Gas Dynamics,” In International Journal for Numerical Methods in Fluids, Vol. 62, No. 7, pp. 709--732. 2009.