Daniel Harris of WebDrips and Herón Ordóñez Guillén of Nvidia discuss Drupal migration and discuss their 'on the fly' upgrade of an Nvidia website from D6 to D7.
Http4s, Doobie and Circe: The Functional Web StackGaryCoady
Http4s, Doobie and Circe together form a nice platform for building web services. This presentations provides an introduction to using them to build your own service.
Talk about add proxy user in Spark Task execution time given in Spark Summit East 2017 by Jorge López-Malla and Abel Ricon
full video:
https://www.youtube.com/watch?v=VaU1xC0Rixo&feature=youtu.be
A Groovy Kind of Java (San Francisco Java User Group)Nati Shalom
Today's application stack is built out many popular OSS frameworks such as Cassandra, MongoDB, Scala, Play, Memcache, RabitMQ alongside the more traditional JEE stack which includes app servers such as Tomcat and JBoss. In this environment the same practices that we used to have in JEE centric world for managing and deploying our app are not relevant anymore. In this session we'll introduce a new open source framework based on Groovy for packaging your application, automating the scaling, failover, and more.
May 2012 HUG: Oozie: Towards a scalable Workflow Management System for HadoopYahoo Developer Network
During the past three years Oozie has become the de-facto workflow scheduling system for Hadoop. Oozie has proven itself as a scalable, secure and multi-tenant service. Oozie stably processes more than 45% of the jobs run across more than 25 Hadoop clusters in Yahoo. At the same time adoption
in other enterprises has increased substantially since Oozie was contributed to the Apache community. We attribute these achievements to design decisions
that was selected to be presented at a workshop during the ACM/SIGMOD conference. This presentation covers the key architectural design choices described in the paper. Operational metrics will be used to illustrate production experience at Yahoo, and we will also include a quick tutorial.
Http4s, Doobie and Circe: The Functional Web StackGaryCoady
Http4s, Doobie and Circe together form a nice platform for building web services. This presentations provides an introduction to using them to build your own service.
Talk about add proxy user in Spark Task execution time given in Spark Summit East 2017 by Jorge López-Malla and Abel Ricon
full video:
https://www.youtube.com/watch?v=VaU1xC0Rixo&feature=youtu.be
A Groovy Kind of Java (San Francisco Java User Group)Nati Shalom
Today's application stack is built out many popular OSS frameworks such as Cassandra, MongoDB, Scala, Play, Memcache, RabitMQ alongside the more traditional JEE stack which includes app servers such as Tomcat and JBoss. In this environment the same practices that we used to have in JEE centric world for managing and deploying our app are not relevant anymore. In this session we'll introduce a new open source framework based on Groovy for packaging your application, automating the scaling, failover, and more.
May 2012 HUG: Oozie: Towards a scalable Workflow Management System for HadoopYahoo Developer Network
During the past three years Oozie has become the de-facto workflow scheduling system for Hadoop. Oozie has proven itself as a scalable, secure and multi-tenant service. Oozie stably processes more than 45% of the jobs run across more than 25 Hadoop clusters in Yahoo. At the same time adoption
in other enterprises has increased substantially since Oozie was contributed to the Apache community. We attribute these achievements to design decisions
that was selected to be presented at a workshop during the ACM/SIGMOD conference. This presentation covers the key architectural design choices described in the paper. Operational metrics will be used to illustrate production experience at Yahoo, and we will also include a quick tutorial.
Slides for presentation on Cloudera Impala I gave at the DC/NOVA Java Users Group on 7/9/2013. It is a slightly updated set of slides from the ones I uploaded a few months ago on 4/19/2013. It covers version 1.0.1 and also includes some new slides on HortonWorks' Stinger Initiative.
This is a presentation given on October 24 by Michael Uzquiano of Cloud CMS (http://www.cloudcms.com) at the MongoDB Boston conference.
In this presentation, we cover Hazelcast - an in-memory data grid that provides distributed object persistence across multiple nodes in a cluster. When backed by MongoDB, objects are naturally written to Mongo by Hazelcast. The integration points are clean and easy to implement.
We cover a few simple cases along with code samples to provide the MongoDB community with some ideas of how to integrate Hazelcast into their own MongoDB Java applications.
This summer, coming to a server near you, Cassandra 3.0! Contributors and committers have been working hard on what is the most ambitious release to date. It’s almost too much to talk about, but we will dig into some of the most important, ground breaking features that you’ll want to use. Indexing changes that will make your applications faster and spark jobs more efficient. Storage engine changes to get even more density and efficiency from your nodes. Developer focused features like full JSON support and User Defined Functions. And finally, one of the most requested features, Windows support, has made it’s arrival. There is more, but you’ll just have to some see for yourself. Get your front row seat and don’t miss it!
Deployment in Oracle SOA Suite and in Oracle BPM SuiteLonneke Dikmans
Deployment in a Service oriented environment is slightly more complicated because the endpoints and other settings need to be changed when promoting the components through QA, Staging and production systems. The Oracle SOA suite offers different mechanisms for different components: Configuration plans for SCA components and customization files for the OSB. You can either deploy an artifact manually, or script the deployment using ant or WLST.
This paper will discuss the different settings you can adjust in configuration plans and customization files, a way of organizing these files in different situations and the use of scripting versus manual deployment.
Ralph Schindler (of Zend Framework) and Jon Wage (of Doctrine) presented these slides for a webinar hosted by zend.com (webinar available online).
Links are contained within the slides to the demo application that was also used during the webinar.
Andrzej Ludwikowski - Event Sourcing - co może pójść nie tak?SegFaultConf
Serio? Kolejna prezentacja o Event Sourcingu? I tak, i nie. Zamiast na gloryfikacji tej jakże zacnej koncepcji, skoncentrujemy się na możliwych problemach przy jej implementacji. Co może wybuchnąć (często z opóźnionym zapłonem)? Gdzie i jak przechowywać eventy? Jak osiągnąć najwyższy poziom wtajemniczenia i skalowalności? Jak żyć z eventual consistency? I wiele innych pytań, z którymi spróbujemy się zmierzyć. Postaram się unikać jednoznacznych odpowiedzi - każde rozwiązanie jest dobre, dopóki “działa”. Przerywniki z doświadczeniami słuchaczy będą bardzo mile widziane.
Spring Data Requery is alternatives of Spring Data JPA
Requery is lightweight ORM for DBMS (MySQL, PostgreSQL, H2, SQLite, Oracle, SQL Server)
Spring Data Requery provide Query By Native Query, Query By Example and Query By Property like Spring Data JPA
Spring Data Requery is better performance than JPA
Alternatives of JPA
Requery provide simple Object Mapping & Generate SQL to execute without reflection and session, so fast than JPA, simple and easy to learn.
Slides for presentation on Cloudera Impala I gave at the DC/NOVA Java Users Group on 7/9/2013. It is a slightly updated set of slides from the ones I uploaded a few months ago on 4/19/2013. It covers version 1.0.1 and also includes some new slides on HortonWorks' Stinger Initiative.
This is a presentation given on October 24 by Michael Uzquiano of Cloud CMS (http://www.cloudcms.com) at the MongoDB Boston conference.
In this presentation, we cover Hazelcast - an in-memory data grid that provides distributed object persistence across multiple nodes in a cluster. When backed by MongoDB, objects are naturally written to Mongo by Hazelcast. The integration points are clean and easy to implement.
We cover a few simple cases along with code samples to provide the MongoDB community with some ideas of how to integrate Hazelcast into their own MongoDB Java applications.
This summer, coming to a server near you, Cassandra 3.0! Contributors and committers have been working hard on what is the most ambitious release to date. It’s almost too much to talk about, but we will dig into some of the most important, ground breaking features that you’ll want to use. Indexing changes that will make your applications faster and spark jobs more efficient. Storage engine changes to get even more density and efficiency from your nodes. Developer focused features like full JSON support and User Defined Functions. And finally, one of the most requested features, Windows support, has made it’s arrival. There is more, but you’ll just have to some see for yourself. Get your front row seat and don’t miss it!
Deployment in Oracle SOA Suite and in Oracle BPM SuiteLonneke Dikmans
Deployment in a Service oriented environment is slightly more complicated because the endpoints and other settings need to be changed when promoting the components through QA, Staging and production systems. The Oracle SOA suite offers different mechanisms for different components: Configuration plans for SCA components and customization files for the OSB. You can either deploy an artifact manually, or script the deployment using ant or WLST.
This paper will discuss the different settings you can adjust in configuration plans and customization files, a way of organizing these files in different situations and the use of scripting versus manual deployment.
Ralph Schindler (of Zend Framework) and Jon Wage (of Doctrine) presented these slides for a webinar hosted by zend.com (webinar available online).
Links are contained within the slides to the demo application that was also used during the webinar.
Andrzej Ludwikowski - Event Sourcing - co może pójść nie tak?SegFaultConf
Serio? Kolejna prezentacja o Event Sourcingu? I tak, i nie. Zamiast na gloryfikacji tej jakże zacnej koncepcji, skoncentrujemy się na możliwych problemach przy jej implementacji. Co może wybuchnąć (często z opóźnionym zapłonem)? Gdzie i jak przechowywać eventy? Jak osiągnąć najwyższy poziom wtajemniczenia i skalowalności? Jak żyć z eventual consistency? I wiele innych pytań, z którymi spróbujemy się zmierzyć. Postaram się unikać jednoznacznych odpowiedzi - każde rozwiązanie jest dobre, dopóki “działa”. Przerywniki z doświadczeniami słuchaczy będą bardzo mile widziane.
Spring Data Requery is alternatives of Spring Data JPA
Requery is lightweight ORM for DBMS (MySQL, PostgreSQL, H2, SQLite, Oracle, SQL Server)
Spring Data Requery provide Query By Native Query, Query By Example and Query By Property like Spring Data JPA
Spring Data Requery is better performance than JPA
Alternatives of JPA
Requery provide simple Object Mapping & Generate SQL to execute without reflection and session, so fast than JPA, simple and easy to learn.
Migrating to Drupal 8: How to Migrate Your Content and Minimize the RisksAcquia
Migrating to Drupal 8 might seem like a daunting task, but it doesn’t have to be. Sometimes, it is possible to migrate simply. Other times, the needs of the migration require customization of the migration, which can introduce significant challenges. However, with the right planning and processes in place, effectively migrating your content can be easy.
In this webinar, learn about the processes needed to effectively migrate your content to Drupal 8 from Drupal 6 or 7. We’ll cover the steps in the 5 phases of an effective migration: auditing your source site, planning a data model, prepping for and planning your migration strategy, and inevitably, launching the migration. Attendees will walk away with a deeper understanding of:
-How to audit your bundles, fields, and entities
-The steps to determine what needs to stay, go, and be modified
-How to work with the Migrate API and extend functionality with contrib migration modules
-The process to set up near-continuous migration of content in order to minimize downtime
-Ways to effectively create and implement a risk mitigation plan
Drupal 8 has made significant improvements towards the ability to stage configuration. But what about content staging? Has it gotten easier in Drupal 8?
This session is targeted towards site builders where we will continue to explore the content staging solution that is being built for Drupal 8 and that was initially presented in Austin. It's a solution that brings vast improvements to sites owners that need to stage or replicate content across sites.
Further, site builders will learn how this solution also applies to broader and sometimes more exciting use cases - content sharing and filtered replication across networks of sites and applications.
The recorded video is available here: https://amsterdam2014.drupal.org/session/content-staging-drupal-8-continued
Drupal isn't a content management system. It's an application development platform that happens to ship with a great CMS as it's default implementation. Look at any chart comparing features of development frameworks and you'll see user management and authentication, forms management and validation, data storage, database migrations, internationalization and translation, MVC model with flexible templating, unit testing, and caching. Sounds a lot like Drupal. We'll look at why Drupal is an ideal application development platform for apps beyond content management, talk about where Drupal can improve in these areas, and look at an example application.
My presentation from Drupal Camp Lutsk 2017 where I was describing Migration process at Drupal 8.
Goal of this presentation is to understand what to do if you get Migration task and make it simple and stable for developer.
This describes most of the critical items dealing with the installation of Drupal 8 beta 12 on GoDaddy shared hosting. See the Silicon Valley Drupal Users Group MeetUp for details and other links - http://www.meetup.com/DrupalGroup/events/223480391/ .
Review of DrupalCon LA 2015, held in Los Angeles on May 11-15th. Includes summits, sessions, and exhibits. Includes pictures of events and some technical information.
Vivek Pandya presents his iOS RESTful APIs for Drupal 8. His development work was originally for Google's Summer of Code. He presented this at the Silicon Valley Drupal Meetup on February 18th.
Drupal is the premiere CMS, used by thousands of businesses, public agencies, and schools. Drupal is changing to accommodate new web technologies such as mobile. This touches upon how Drupal works for mobile.
This 7-second Brain Wave Ritual Attracts Money To You.!nirahealhty
Discover the power of a simple 7-second brain wave ritual that can attract wealth and abundance into your life. By tapping into specific brain frequencies, this technique helps you manifest financial success effortlessly. Ready to transform your financial future? Try this powerful ritual and start attracting money today!
Italy Agriculture Equipment Market Outlook to 2027harveenkaur52
Agriculture and Animal Care
Ken Research has an expertise in Agriculture and Animal Care sector and offer vast collection of information related to all major aspects such as Agriculture equipment, Crop Protection, Seed, Agriculture Chemical, Fertilizers, Protected Cultivators, Palm Oil, Hybrid Seed, Animal Feed additives and many more.
Our continuous study and findings in agriculture sector provide better insights to companies dealing with related product and services, government and agriculture associations, researchers and students to well understand the present and expected scenario.
Our Animal care category provides solutions on Animal Healthcare and related products and services, including, animal feed additives, vaccination
APNIC Foundation, presented by Ellisha Heppner at the PNG DNS Forum 2024APNIC
Ellisha Heppner, Grant Management Lead, presented an update on APNIC Foundation to the PNG DNS Forum held from 6 to 10 May, 2024 in Port Moresby, Papua New Guinea.
1.Wireless Communication System_Wireless communication is a broad term that i...JeyaPerumal1
Wireless communication involves the transmission of information over a distance without the help of wires, cables or any other forms of electrical conductors.
Wireless communication is a broad term that incorporates all procedures and forms of connecting and communicating between two or more devices using a wireless signal through wireless communication technologies and devices.
Features of Wireless Communication
The evolution of wireless technology has brought many advancements with its effective features.
The transmitted distance can be anywhere between a few meters (for example, a television's remote control) and thousands of kilometers (for example, radio communication).
Wireless communication can be used for cellular telephony, wireless access to the internet, wireless home networking, and so on.
Bridging the Digital Gap Brad Spiegel Macon, GA Initiative.pptxBrad Spiegel Macon GA
Brad Spiegel Macon GA’s journey exemplifies the profound impact that one individual can have on their community. Through his unwavering dedication to digital inclusion, he’s not only bridging the gap in Macon but also setting an example for others to follow.
Meet up Milano 14 _ Axpo Italia_ Migration from Mule3 (On-prem) to.pdfFlorence Consulting
Quattordicesimo Meetup di Milano, tenutosi a Milano il 23 Maggio 2024 dalle ore 17:00 alle ore 18:30 in presenza e da remoto.
Abbiamo parlato di come Axpo Italia S.p.A. ha ridotto il technical debt migrando le proprie APIs da Mule 3.9 a Mule 4.4 passando anche da on-premises a CloudHub 1.0.
2. Dan Harris
Founder of Webdrips.com
Nine years Drupal Experience
Front End
Back End
etc.
Porting Node Hierarchy Module to
Drupal 8
About the Speakers
Heron Ordoñez
Senior Developer at NVIDIA
9 years of Drupal Experience
Newspapers
Magazines
Radio and TV
3. Note About the Migration Process
Although we’re covering a Drupal 6 to 7
migration in this presentation, most if not all of
these ideas presented here should work for any
Drupal to Drupal migration.
4. Overview: Initial Plan/Estimates
Initial estimate: one week of downtime
SQL queries would be used to export/import
when coverage was limited with Drupal
Migrate
Only automation provided by Migrate Modules
Existing Drupal 7 Architecture
5. Overview: Updated Plan
Virtually zero downtime
Complete migration in one business day
Over 99% automated
D7 site to be built during migration from
scratch
6. About the Drupal 6 Site
Architecturally, was a mess (Frankensite)
Migration provided chance to clean up architecture and
code
Six custom themes (1 custom/5 subthemes)
35 custom modules
151 contributed modules
7. About the Drupal 6 Site
1000 privileged users
About 400k non-privileged users
25 Content Types, including Webforms
Over 2,500 pages
8. About the Drupal 7 Site
106 Modules
Bootstrap Primary Theme
One Bootstrap subtheme, Four sub-
subthemes
Six content types only
11 Features provided architecture
9. Automated Migration Process
Requirements
Migrate modules: migrate, migrate_extras, migrate_d2d,
migrate_webform
Import modules: menu_import, path_redirect_import
Four custom modules
Scripts migration and deployment
Fast server with SSD
10. Migration Script Overview
Requirements:
Create new Drupal D7 site
Build out site architecture with features
Enable Modules
Migrate D6 to D7
Import items that couldn’t be migrated
This provided for a repeatable/reliable process
12. Custom Migration Modules
1.Disable “edits” to the D6 site
a.Basically re-direct webform pages, admin pages,
and paths like node/add, node/edit, etc.
2.Views (implemented with features) only for
migration status and post-processing
3.Migrate_d2d module
4.CSV-based Migration
13. Drupal Migrate/D2D/Extras
Handled most of the heavy lifting
Everything except menu links, path redirects, and
slide shows
Extensive drush support
Plenty of methods available to massage data
D2D: simplifies migration code
14. Migrating Users
Challenges
Nearly 400K unprivileged users
Needed to assign users to organic groups
Based on how webform questions answered
Had to fix user passwords
Fixed by writing directly to the user table inside the
migration
15. Migrate Users Code
Unprivileged vs. Privileged was a simple query:
class NvidiaPrivilegedUserMigration extends NvidiaUserMigration {
protected function query() {
$query = parent::query();
$query->condition('u.mail', '%nvidia.com', 'LIKE/NOT LIKE');
return $query;
}
}
16. Migrate Users Code
Fix the password:
public function complete($account, $row) {
parent::complete($account, $row);
$account->pass = $row->pass;
db_update('users')
->fields(array('pass' => $account->pass))
->condition('uid', $account->uid)
->execute();
$this->nvidia_memberships($row);
}
17. Assign Users to Groups (Review)
public function nvidia_memberships($row) {
$membership_query = Database::getConnection('default', 'd6source')-
>select('webform_submissions', 'ws');
$membership_query->join('webform_submitted_data', 'wd', 'wd.sid =
ws.sid');
$membership_query->fields('wd', array('cid'));
$membership_query->fields('ws', array('nid'));
$membership_query->addExpression('group_concat(data)', 'data');
$membership_query->groupBy('ws.sid');
$membership_query->groupBy('cid');
$membership_query->condition('ws.uid', $row->uid);
$membership_query->condition('ws.nid',
array(1234567,2345678,3456789,4567890,5678901), 'IN');
$membership_id = nvidia_og_membership_associate_user_with_program();
18. Node Migration Challenges
Body images & links with absolute paths
Empty fields sometimes caused display issues
Had to deal with “interesting” architecture
decisions on the D6 site
Moved larger files to the cloud
Reduced the number of content types
20. How a Strange Dev. Decision can Affect a
Migration
D6 product page and dB variables table (review) led to the following code
$variable_name = 'nvidia_product_disable_product_image_'.$row->nid;
// drush_print_r($variable_name);
$query = Database::getConnection('default', 'd6source')
->select('variable', 'v')
->fields('v', array('name', 'value'))
->condition('v.name', $variable_name, '=')
->execute()
->fetchAll();
$product_image_disabled = $query[0]->value;
if ($product_image_disabled == 'i:1;') {
$row->field_inline_image = NULL;
}
21. Remove Empty Textarea Fields
public function prepare($entity, stdClass $row) {
foreach ($row as $key => $value) {
if (!isset($row->$key) || $row->$key === null) {
$entity->$key = NULL;
}
}
}
22. “Non-Standard” Entity Migrations (Review)
D2D handles well-established Drupal entities
well
nodes, users, taxonomy, etc.
But what if you want to migrate block content
to an entity?
CSV Migration to the rescue
23. Challenges
Biggest challenge was reducing the migration
time
The original estimate just for migrating users was over
40h
Eventually that time was reduced to ~ 3 hours
We tweaked my.cnf, php.ini, drush.ini
Got a really fast server with Intel Xeon processors,
fast RAM, and a SSD
24. Challenges
Installation of modules in order
circular dependencies
features that add fields need to be installed before
migration
Relationships between content
Both nodes need to exist before creating a
relationship
“Parent” content that did not exist in original site
25. Migration timeline
-7days to release: Content freeze
-2days: Automated rebuild, content migration
and editorial approval.
-8h: Registration lockdown and migration start
-2h: Batch processing of content by editors
and final tests
26. Accelerating migration
Use Drush
Single pass for each item
Migration objects are big and slow
Don’t load an object from DB twice
Multithreading
https://www.deeson.co.uk/labs/multi-processing-part-2-how-make-migrate-move
27. Add multithreading to a working
migration class
Not very portable
needs a Drush extension
needs to run on the ‘fast’ server
Very effective
28. Add multithreading to a working
migration class
Sub-class the migration
Make all the sub-migrations use the same
index
Make the sub-migration work on a small
‘chunk’ of the index
Break the migration in parts and send chunks
of it to multiple threads
29. Add multithreading to a working
migration class
<?php
class NVMultiThread extends NvidiaUnprivilegedUserMigration {
public function __construct($args) {
$args += array(
'source_connection' => NVIDIA_MIGRATE_SOURCE_DATABASE,
'source_version' => 6,
'format_mappings' => array(
'1' => 'filtered_html',
'2' => 'full_html',
'3' => 'plain_text',
'4' => 'full_html',
),
'description' => t('Multithreaded Migration of users from Drupal 6'),
'role_migration' => 'Role',
);
This is
boilerplate
needed by
D2D
30. Add multithreading to a working
migration class
parent::__construct($args);
$this->limit = empty($args['limit']) ? 100 : $args['limit'];
$this->offset = empty($args['offset']) ? 0 : $args['offset'];
$this->map = new MigrateSQLMap('nvidiaunprivilegeduser',
array(
'uid' => array(
'type' => 'int',
'unsigned' => TRUE,
'not null' => TRUE,
'description' => 'User migration reference',
),
),
MigrateDestinationUser::getKeySchema()
);
}
map/index
table
index definition
31. Add multithreading to a working
migration class
protected function query() {
$query = parent::query();
$query->range($this->arguments['offset'], $this->arguments['limit']);
return $query;
}
}
Modify original
query to limit the
number of items
to work on
32. Measuring the improvement
Same server
Restore destination DB from backup after
each run
Same source DB
Both DBs in the same server
MySQL optimizations for concurrency issues
34. Measuring the improvement
10,000 items, 1000 per thread
Threads Time Speed
1 707s 848/min
2 303s 1980/min
3 300s 2000/min
4 291s 2061/min
5 351s 1709/min
35. Measuring the improvement
50,000 items, 5000 per thread
Threads Time Speed
3 1990s 1507/min
4 1562s 1920/min
5 1303s 2302/min
6 1637s 1832/min
36. Conclusion
Drop DNS TTL to 1 minute days before
launch
Repeatability is key
Migration is very powerful but can be slow
Automation helps drop downtime close to zero
37. Conclusion
Ask for help
There’s many ways to use Migration, if one
way is not working drop it and use it
differently
CSV vs direct read from DB
Weird things happen with orphaned fields