Jamil Gafur, Lintao Bu, Michael Crowley Renewable Resources and Enabling Science Center at the National Renewable Energy Lab (NREL)

1. Non-equilibrium Molecular Simulations of Polymers
under Flow: Saving Energy through Process
Optimization
Jamil Gafur, Lintao Bu, Michael Crowley
Renewable Resources and Enabling Science Center at
the National Renewable Energy Lab (NREL)
NREL is a national laboratory of the U.S. Department of Energy Office of Energy Efficiency and Renewable Energy operated
by the Alliance for Sustainable Energy, LLC

2. Background
• We are studying the process of
plastic extrusion to minimize the
polymer build up around the
extrusion pipe
• When build up starts to occur the
overall structure and stability of the
product decreases significantly https://en.wikipedia.org/wiki/Plastic_extrusion

3. Background
• We are studying the process of
plastic extrusion to minimize the
polymer build up around the
extrusion pipe
• When build up starts to occur the
overall structure and stability of the
product decreases significantly https://en.wikipedia.org/wiki/Plastic_extrusion

4. Background
• We are studying the process of
plastic extrusion to minimize the
polymer build up around the
extrusion pipe
• When build up starts to occur the
overall structure and stability of the
product decreases significantly https://en.wikipedia.org/wiki/Plastic_extrusion
Feed pipe
Flow velocity (Normal)
Laminar flow (Slow)

5. Background
• We are studying the process of
plastic extrusion to minimize the
polymer build up around the
extrusion pipe
• When build up starts to occur the
overall structure and stability of the
product decreases significantly https://en.wikipedia.org/wiki/Plastic_extrusion
Molecular Weight Distribution

6. Background
• We are studying the process of
plastic extrusion to minimize the
polymer build up around the
extrusion pipe
• When build up starts to occur the
overall structure and stability of the
product decreases significantly https://en.wikipedia.org/wiki/Plastic_extrusion
Molecular Weight Distribution

7. Impact
• This project combines DOE High Performance Computing capabilities
with established and emerging molecular simulation techniques to
address industrially relevant issues of practical interest to
US Manufacturing competitiveness.
• Our work this summer is focused on setting up the codebase for future
development, and optimizing the code to allow for optimal use of
allocated nodes

8. Initial work
• Given a zip file of Fortran 90 codes and an associated simulation script; this zip file was sent the
main way of distributing the code
• There was no documentation on how to run the simulations, or how the code worked
• We have existing codes that simulate the extrusion process; however, it does not run efficiently
(some runs taking more than 6 months to run)
• This code has not been optimized in any way, nor has the codebase been under any form of RSE
practices
• CI
• Unit testing
• Documentation

9. First steps
• Set up version control on GitHub
• Set up timelines, milestones and issues with
short term, and long-term goals in mind
• Initiated some form of unit testing.
• Currently in the process of having better unit
tests
• Ran the simulation, saved the output files
and compared future runs to this "golden"
set of output files
• "I have the same level of confidence that this
code works now, as when I got it.”
• “I have no way of knowing if there are any errors
associated with the algorithms"
- Jamil Gafur
https://github.com/crowleymf/hpc4eidow

10. File Structure and Initial profiling
• Initial results showed that the code mostly
focused on computation
• There was no MPI, or open MP added to the
codebase, thus everything was running on one
core
• The rest of the time was spent on I/O
• Takes about 11 min for the "test case" to run

11. Optimization
Using the intel profiler (VTune) we identified parts of the code that
can be optimized, our results are shown below

12. Vel.f90 utilization

13. Next step, Optimization of code
• Most of the issues we found were due to memory allocation,
something that can't easily be fixed

14. Next step, Optimization of code
• Most of the issues we found were due to memory allocation,
something that can't easily be fixed
BUT WHO SAID LIFE WAS EASY?

15. Next step, Optimization of code
• Most of the issues we found were due to memory allocation,
something that can't easily be fixed
BUT WHO SAID LIFE WAS EASY?

16. Next step, Optimization of code
• Most of the issues we found were due to memory allocation,
something that can't easily be fixed
BUT WHO SAID LIFE WAS EASY?
~20%

17. 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0
15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 0
Index
Bits
Integer
(dmask)

18. Integer
(dmask)
Use the d value as binary operators for our integer
15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 0
Integer (16) Index
Bits
0 0 0 1 0 0 1 0 1 0 1 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0
15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 0
Index
Bits

19. Use the d value as binary operators for our integer
15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 0
Integer (16) Index
Bits
0 0 0 1 0 0 1 0 1 0 1 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0
0 0 0 1 0 0 1 0 1 0 1 0 0 0 0 0
15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 0
Index
Bits
0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0
iand
Integer
(dmask)

20. • We have condensed an if-statement (>20 clock cycles) into 2 clock
cycles.
• iand
• ishft
0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0
Left 4
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1

22. Original Code we were given
Updated Code we updated
Optimized Using -0fast flag when compiling
Non-optimized !^
Output Write output logs every 10 iterations (default of
original code)
No-Output Writes output at last iteration

23. Original Code we were given
Updated Code we updated
Optimized Using -0fast flag when compiling
Non-optimized !^
Output Write output logs every 10 iterations (default of
original code)
No-Output Writes output at last iteration

24. Conclusion
• It is important to use good
programming practices and
algorithm design before
implementing parallelization
• Profiling a code gives you a
good idea on where you can
update code to get the most
speed up
• Version control, V&V,
documentation

25. Moving Forward
• Moving forward our group would like to implement MPI
to allow for multi-node support.
• Using MPI we believe we can offload the I/O to a
separate process and gain an additional 11% speedup

26. Thank you for your time
Special thanks to:
• Mentors/Advisors:
• Dr. Lintao Bu
• Dr. Michael Crowley
• Dr. Suely Oliveira
• Dr. Gretta Kellogg
• Energy Efficiency and Renewable Energy (EERE) Advanced
Manufacturing Office (AMO)
• Oak Ridge Institute for Science and Education (ORISE)
• University of Iowa Computer Science Department
About me:
https://jamilggafur.gitlab.io/
jamil-gafur@uiowa.edu