Drupal content-migration


Published on

Learn how to use the migrate module to migrate content from various sources into Drupal.

Published in: Technology
1 Comment
  • Hi! Thanks for this presentation, but for me as a blogger it's hard to comprehend. I've found a video by CMS2CMS which describes a way of migration to Drupal with no coding: http://www.youtube.com/watch?v=cCkW-vwzlEg

    Hope it helps someone.
    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Drupal content-migration

  1. 1. Ashok Modi (BTMash) – Drupal LA – March 2011<br />Migrating content to drupal – Migrate Module<br />
  2. 2. Agenda<br />Different Methods<br />Steps to Work with Migrate<br />Hooks<br />Build a Class<br />Description<br />Source Adding<br />Mapping<br />Additional Data<br />drush commands<br />Q & A<br />
  3. 3. Disclaimer<br />Code Heavy!<br />You may fall asleep.<br />New territory!<br />Talk about one *small* aspect of migration (migrating nodes)<br />Not talking about creating own DestinationHandlers<br />Possibly not talking about creating own FieldHandlers (depends on time)<br />Can walk through an example migration that I did if preferred.<br />Ask questions?<br />It will make the presentation less terrible <br />
  4. 4. Possible method – by hand<br />Should be accurate<br />Get all files<br />Mapping / everything works<br />Time consuming<br />Not feasible if you have a lot of content.<br />Good way to test interns / punish coworkers (?)<br />
  5. 5. Possible methods – Node Export<br />Node Export (http://drupal.org/project/node_export)<br />Has 7.x branch<br />But no way to update content from 6.x -> 7.x<br />No way to go back<br />*easy* to set up (requires exact setup between source and destination in field names, etc)<br />
  6. 6. Possible Methods - Feeds<br />Really good method<br />Map fields from source to destination<br />Can import RSS / Atom / Various types of feeds<br />Also flat files such as CSV<br />Well documented<br />Other than flat files, requires a feed source<br />Might run into issues if content is updated in source<br />*might be tricky in another cms*<br />
  7. 7. Method demonstrated - Migrate<br />Already defined many possible import sources<br />XML, JSON, CSV, Databases (any currently supported by Drupal!)<br />Can import many different types of content<br />Users, Nodes, Comments, Taxonomy, Files … all core entities<br />Can define your own import handler (not covered in presentation)<br />Can define own method for importing custom fields<br />Current already defined for all core field types<br />Has support for Media Module importing<br />Patch underway for getting date import<br />Can define your own field handler (possibly not covered in presentation)<br />Drush integration<br />Rollback, Status Updates, Limited import.<br />Caveat – Confusing documentation<br />Only a status UI – all mapping must be done in code.<br />
  8. 8. Assumptions made for presentation<br />Migrating from a database<br />Files directory for source are on same machine as the destination site directory<br />
  9. 9. Steps to work with Migrate<br />Let Migrate know about your module (1 hook!)<br />Build a Migration Class<br />Give it a description<br />Let Migrate know what you’re getting content from.<br />Let Migrate know about the type of content.<br />Map the fields the migrate class it going to fill.<br />(Optional) Massage / Add any fields you couldn’t get in the initial mapping (query).<br />
  10. 10. Step 1: Hook<br />Implement one hook – hook_migrate_api<br />Provide the api version number (currently at version 2)<br />That’s it!<br />function mymodule_migrate_api() {<br /> return array(<br /> ‘api’ => 2,<br /> );<br />}<br />
  11. 11. Step 2: Build a Class<br />Implement classes<br />Class defines type of content that will be imported<br />class NodeContentTypeMigration extends Migration {<br />public function __construct() {<br />parent::__construct();<br /> …<br />}<br />public function prepareRow($current_row) {<br /> …<br />}<br />}<br />
  12. 12. Step 2: Build a Class (functions inside)<br />public function __construct() {…}<br />Constructor for the class<br />Allows migrate to know content type (user, node, tags)<br />Where content is mapped from (db, csv, xml, etc)<br />All the mappings coming in (fields)<br />(optional)public function prepareRow($current_row) {…}<br />Any extra data (things that cannot be pulled in a single query(?))<br />Massage any of the data that was pulled in (clean up text, clean up links, etc)<br />
  13. 13. Step 2a: Create a description<br />Create a description<br />Class Description<br />Any additional source fields (not found in initial query)<br />Initial source -> destination mapping (what is the key in the source db?)<br />$this->description = t(“Import all nodes of type PAGE”);<br />Define Source Fields<br />Fields that may not be getting pulled in via your query or in the flat file data but will be getting migrated somehow<br />$source_fields = array(<br /> 'nid' => t('The node ID of the page'),<br /> ’my_files' => t(’The set of files in a field for this node'),<br />);<br />
  14. 14. Off course: query source database<br />Set up query (if need be, switch DBs using Database::getConnection)<br />$query = Database::getConnection('for_migration', 'default');<br />Then write out rest of the query<br />Alternatively, if source db is on same machine as destination db, use mysql db shortcut<br />db_select(MY_MIGRATION_DATABASE_NAME .’.table_name’, ‘t’) <br />
  15. 15. Step 2b: Call to grab data<br />NOTE: This is only for migrations from databases<br />Set up query (if need be, switch DBs using Database::getConnection)<br />$query = db_select(MY_MIGRATION_DATABASE_NAME .'.node', 'n’)<br /> ->fields('n', array('nid', 'vid', 'type', 'language', 'title', 'uid', 'status', 'created', 'changed', 'comment', 'promote', 'moderate', 'sticky', 'tnid', 'translate'))<br /> ->condition('n.type', 'page', '='); <br />$query->join(MY_MIGRATION_DATABASE_NAME .'.node_revisions', 'nr', 'n.vid = nr.vid'); <br />$query->addField('nr', 'body');<br />$query->addField('nr', 'teaser');<br />$query->join(MY_MIGRATION_DATABASE_NAME .'.users', 'u', 'n.uid = u.uid'); <br />$query->addField('u', 'name'); <br />$query->orderBy('n.changed');<br />
  16. 16. Step 2b: Why the orderby?<br />Migrate module has a feature called ‘highwater’<br />It is a key to designate and figure out if a piece of content needs to be updated rather than inserted.<br />Means content can be updated!<br />$this->highwaterField = array(<br /> 'name' => 'changed', <br /> 'alias' => 'n’,<br />);<br />
  17. 17. Step 2c: Mappings<br />Add a ‘mapping’ (this is for tracking relationships between the rows from the source db and the rows that will come in the destination site) – essentially key of source DB.<br /> $this->map = new MigrateSQLMap(<br /> $this->machineName, <br /> array( <br /> 'nid' => array(<br /> 'type' => 'int’,<br /> 'unsigned' => TRUE, <br /> 'not null' => TRUE,<br /> 'description' => 'D6 Unique Node ID’,<br /> 'alias' => 'n', <br /> )<br /> ),<br />MigrateDestinationNode::getKeySchema());<br />
  18. 18. Step 2c: Mappings (cont’d)<br />Now let the migrate module know what kind of mapping is being performed.<br />$this->source = new MigrateSourceSQL($query, $source_fields);<br />Along with the type of content<br />$this->destination = new MigrateDestinationNode('page');<br />.<br />
  19. 19. Step 2d: Map Fields<br />Usually follows the form<br />$this->addFieldMapping(‘destination_field_name’, ‘source_field_name’);<br /> $this->addFieldMapping('revision_uid', 'uid');<br />Can provide default values<br /> $this->addFieldMapping('pathauto_perform_alias')->defaultValue('1');<br />Can provide no value<br />$this->addFieldMapping('path')->issueGroup(t('DNM'));<br />Can provide arguments and separators for certain field types (body, file, etc require this methodology)<br />
  20. 20. Step 3: Additional data / cleanup<br />Optional<br />public function prepareRow($current_row)<br />Use it to add any additional data / cleanup any fields that were mapped in.<br />// Get the correct uid based on username & setauthor id for node touid<br />$user_query = db_select('users', 'u’)<br /> ->fields('u', array('uid')) <br /> ->condition('u.name', $username, '='); <br />$results = $user_query->execute(); <br />foreach ($results as $row) { <br /> $current_row->uid = $current_row->revision_uid = $row->uid;<br /> break;<br />}<br />
  21. 21. DrushCommands (the important ones)<br />drush ms – List various migration import classes<br />drush mi <importclass> - Import content<br />drushmr <importclass> - Rollback content<br />Options<br />--idlist=id1,id2,… - Import content with specific IDs<br />--itemlimit=n – Only import up to ‘n’ items<br />--feedback=“n seconds” – Show status report every ‘n’ seconds<br />--feedback=“n items” – Show status report every ‘n’ items<br />
  22. 22. Resources<br />http://drupal.org/project/migrate<br />http://drupal.org/node/415260<br />Look at the example modules <br />http://drupal.org/project/migrate_extras<br />http://drupal.org/project/wordpress_migrate<br />http://cyrve.com/import (drush documentation) <br />http://goo.gl/3e1Jm (additional documentation to be added to core project documentation)<br />http://goo.gl/2qDLh (another example module)<br />
  23. 23. Questions?<br />Thank you <br />