Developed for the Denver Art Museum by Ashley Blewer, this slide-deck covers some of the basics of diagnosing issues with Archivematica. Ashley covers everything from the software components involved with Archivematica, to monitoring logs, system monitoring, and upgrading your system. The presentation concludes with some useful links for tech-savvy preservationists, and Archivematica-unfamiliar system's administrators!

1. Archivematica
Technical Training:
Diagnostics Guide
Ashley Blewer @ Artefactual Systems, Inc.
ablewer@artefactual.com

2. Outline
● Technical introduction
● Microservices we use
● Software components
● Logging in
● Reading logs
● System monitoring
● Upgrading/Security
● Advanced: APIs
● Getting (free) help

3. High-level goals

4. Technical diagram

5. Supporting technology
● Python: programming language
● Django: web application framework
● Gearman: job scheduler
● MySQL: relational database
● Elasticsearch: search index
● Nginx: web server (can be apache)
● Gunicorn: interface between Python and Nginx
● git: version control system
● Ansible/Docker: deployment/configuration management

6. All on Linux
● Ubuntu 16.04 or 18.04
● CentOS 7 or Red Hat

7. Format Policy Registry
● Tools we use to perform preservation actions
● Rules we use to determine when to use the Tools
● Commands are applied to files based on the Rules

8. Tools
● Identification: FIDO and Siegfried
● Characterization: FITS, FFprobe, MediaInfo, ExifTool
● Event Detail: echo
● Extraction: 7zip, SleuthKit
● Normalization: FFmpeg, ImageMagick
● Transcription: Tesseract
● Validation: MediaConch
● Verification: File exists? Filesize is >0?

9. Production tools
Automation Tools https://github.com/artefactual/automation-tools/
Fixity https://github.com/artefactual/fixity
Deployment tools
Deploy-pub https://github.com/artefactual/deploy-pub
Ansible roles
https://github.com/artefactual-labs/ansible-archivematica-src
Am.git https://github.com/artefactual-labs/am
DevOps Tools
Ops-helpers https://github.com/artefactual-labs/ops-helpers
Dev-tools https://github.com/artefactual/archivematica-devtools

10. Technical stack
● Lots of tools = lots of potential points of failure
● Archivematica strives to relay as much information as
possible to the user -- especially about what the tools are
doing and what they are producing

11. Components
● Dashboard: for the user
● MCPClient: does the work
● MCPServer: manages the work
● Storage Service: manages storage

12. Logging in
● Logging in (ssh)
● Moving files (scp)
● What’s running (ps -sf | grep py)
● How much space? (du)
● How much free space? (df -h)
● Load average time? (top)
● Read end of logs (tail)
● Read logs (less)

13. Logging in
ssh username@url

14. Moving files
Download a file to your computer
scp
your_username@remotehost.url:your-file.txt
/your/local/directory
Send a file to your machine
scp path/to/your-file.txt
your_username@remotehost.url:/some/remote/di
rectory

15. What’s running?
ps -ef | grep py
These services should all be running:
● Dashboard (apache)
● Database (mysql)
● Elasticsearch (elastic)
● Storage Service (uwsgi or nginx)
● FITS
● Server (MCP) -- Should show MCP server and MCP client

16. What’s running?
ps -ef | grep py
Also, these dependent services should all be running:
● MySQL
● Elasticsearch
● Gearman
● Nginx
● Nailgun
● Clamav

17. du
To get the file size of each subdirectory of the directory you
are in, you can run this command:
du -h --max-depth=1
This command can take a long time if you have very large
mounted drives.
See amount of space on machine

18. Check free space on disk
df -h
● Up to 3x of free space required for processing
● cron job can auto-clear deleted/rejected files

19. Load averages
top

20. Restarting services
service archivematica-dashboard restart
service archivematica-mcp-client.service restart
service archivematica-mcp-server.service restart
service archivematica-storage-service restart
service gearmand restart

21. Reading logs
less /var/log/archivematica/dashboard/dashboard.log
less /var/log/archivematica/dashboard/dashboard.debug.log
less /var/log/archivematica/MCPClient/MCPClient.log
less /var/log/archivematica/MCPClient/MCPClient.debug.log
less /var/log/archivematica/MCPServer/MCPServer.log
less /var/log/archivematica/MCPServer/MCPServer.debug.log
less /var/log/archivematica/storage-service/storage-service.log
less /var/log/archivematica/storage-service/storage-service.debug.log

22. Finding errors
grep -rn ERROR
/var/log/archivematica/

23. System monitoring
Zappix/Grafana

24. Upgrading
● Need to decide on a new release whether you want it or not, how much
time to put aside.
● Tradeoff to not upgrading is not keeping pace with community and having
a harder time getting support from community for an older version.
● Good idea to test the upgrade- make a backup of your production
environment and test upgrade there. If that is not possible, plan for
downtime.
○ If you want to be able to do this, you might want to explore
virtualization of your Archivematica environment so you can run a
development (testing) environment in addition to the production
environment.

25. Security upgrades
● Make sure that Ubuntu is set-up to do Unattended Upgrades, which will
apply security patches (like equivalent of Windows updates).
● Sometimes these upgrades require the system to be restarted- you might
need to plan for 30 minutes of downtime (not in the middle of processing,
make sure your current Transfer/AIPs are done).

26. APIs
https://wiki.archivematica.org/Storage_Service_API
https://wiki.archivematica.org/Archivematica_API

27. Getting Help
● Participating in the community forum
○ Archivematica
https://groups.google.com/forum/#!forum/archivematica
● Documentation
○ Main docs https://www.archivematica.org/en/
○ Wiki https://wiki.archivematica.org/Main_Page
● Github issues
○ Main repo https://github.com/archivematica/Issues/issues

28. See also
This presentation in document form
● For tech-savvy preservationists:
https://docs.google.com/document/d/1GybyH7X_gpZ7wpYVo5d9__LeG
NuXYCky0oairJGJAmo/edit#heading=h.y1nyq0vlcvsl
● For Archivematica-unfamiliar systems administrators:
https://docs.google.com/document/d/1NDzGHBGuPFa7GTHCMEl3D2n
vvdZRxG2FpdsGAYoG31I/edit#

29. exit
:q
q
Quit
.quit
q
:q!
Ashley Blewer @ Artefactual Systems, Inc.
ablewer@artefactual.com