My talk on working with the CloudStack Database for data recovery and unintended manipulation. From CloudStack Collaboration Conference North America 2014

1. Don’t Break the Glass!
In case of emergency…

2. Intro
• Apache CloudStack committer, PMC member
• Security Guy
• Founder of Stratosec
• …hacker, reverse-engineer
• @johnlkinsella, #CCCNA14

3. Goal
Provide bread crumbs for you to gain expertise
on CloudStack’s data model so you can react
quickly in an emergency situation.

4. TOC
• What Should Production Look Like?
• My Nightmare
• My response
• How to prepare
• How we’re trying to make things better
• Homework

5. What Should Production Look Like?
• Briefly:
– Redundant management servers
– Redundant databases
– Redundant storage (primary and secondary)
– Redundant infrastructure
– Data backups – encrypted
– Pro tip: store backups outside CloudStack.

6. One Painful February Day…
• Made a change in control panel prior to giving customer access
• Control panel interpreted this as “Delete the Customer.”
• More precisely: “Delete the Customer, VMs and all storage. Now.”

7. Wait, What??
• Took about 15 seconds to realize what happened
• Looking in CloudStack Management UI, saw VM shutting down,
then being expunged.

8. Periodic Table…

9. Emergency Steps
• Shut down ACS management server immediately
• Fire up your favorite SQL tool (I recommend Squirrel SQL)

10. With your Data Recovery Shrine of Choice
Immediately Establish Contact
Establish Contact With Data Shrine

11. Know how CloudStack Works
• Steps to delete a VM:
– Shut down
– Destroy
– Expunge volumes
– Expunge VM

12. The “SQL API”
• DB Tables:
– vm_instance
– volumes
– nics
– user
– account
• Common Fields:
• account_id
• instance_id
• created/updated/removed
• state
• removed
BACK UP DB BEFORE MAKING CHANGES!

13. Actual SQL

14. Make change in SQL.

15. Bonus: changing VM’s IPUPDATE user_ip_address SET allocated=(SELECT allocated FROM (SELECT allocated,
public_ip_address FROM user_ip_address) AS oldip WHERE public_ip_address='184.172.14.194')
WHERE public_ip_address='184.172.14.193';
UPDATE user_ip_address SET account_id=(SELECT account_id FROM (SELECT account_id,
public_ip_address FROM user_ip_address) AS oldip WHERE public_ip_address='184.172.14.194')
WHERE public_ip_address='184.172.14.193';
UPDATE user_ip_address SET domain_id=(SELECT domain_id FROM (SELECT allocated,
public_ip_address FROM user_ip_address) AS oldip WHERE public_ip_address='184.172.14.194')
WHERE public_ip_address='184.172.14.193';
UPDATE user_ip_address SET state='Allocated' WHERE public_ip_address='184.172.14.193';
UPDATE user_ip_address SET network_id=(SELECT network_id FROM (SELECT network_id,
public_ip_address FROM user_ip_address) AS oldip WHERE public_ip_address='184.172.14.194')
WHERE public_ip_address='184.172.14.193';
UPDATE nics SET ip4_address='184.172.14.193' WHERE instance_id=1797;
UPDATE vm_instance SET private_ip_address='184.172.14.193' WHERE instance_name='i-2-1797-
VM';

16. Move VM between accounts
UPDATE user_ip_address SET account_id=10 WHERE id=29;
UPDATE user_ip_address SET domain_id=8 WHERE id=29;
UPDATE user_ip_address SET network_id=217 WHERE id=29;
UPDATE vm_network_map SET network_id=217 WHERE vm_id=144;
UPDATE vm_instance SET domain_id=8 WHERE id=29;
UPDATE op_networks SET nics_count=nics_count+1 WHERE id=217;
UPDATE op_networks SET mac_address_seq=mac_address_seq+1 WHERE id=217;
UPDATE nics SET broadcast_uri='vlan://230' WHERE id=179;
UPDATE nics SET network_id=217 WHERE id=179;
UPDATE nics SET isolation_uri='vlan://230' WHERE id=179;
UPDATE firewall_rules SET account_id=10 WHERE ip_address_id=29;
UPDATE firewall_rules SET domain_id=8 WHERE ip_address_id=29;
UPDATE firewall_rules SET network_id=217 WHERE ip_address_id=29;

17. How you can prepare
• Create written cloud policy, audit production to match
• Test your backups
• Become familiar with CloudStack internals
• Do you really need frequent data expunge processes?

18. Next: Production Lock
• Feature: Allow our customers to ensure that no automations will
modify or delete their compute or storage instances
– Working for VM
– Adding for storage, networks
– Need to add to UI

19. Homework
• Pick a user VM in your Cloud. Via the database:
– Find the VM instance record
– Find the VM’s nics in the database
– Find the VM’s storage in the database
– Try deleting a VM via normal means, then before expunge,
restore via SQL

20. kthxbye!
• http://cloudstack.apache.org
jlk@stratosec.co
@johnlkinsella
http://theresnomon.co