S03 hybrid app_and_gae_datastore_v1.0

  • 5,334 views
Uploaded on

Google BigTable and GAE Datastore

Google BigTable and GAE Datastore

More in: Technology , Business
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads

Views

Total Views
5,334
On Slideshare
0
From Embeds
0
Number of Embeds
6

Actions

Shares
Downloads
29
Comments
0
Likes
0

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Hybrid Appand GAE DataStore
    대표이사 장선진
    @jangsunjin
    • Mobile Phone: (+82) 010-4585-1770
    • 2. E-Mail: jangsunjin@softwareinlife.com
  • MapReduce
    MapReduce is a patented software framework introduced by Google in 2004 to support distributed computing on large data sets on clusters of computers.
    http://www.cs.berkeley.edu/~ballard/cs267.sp11/hw0/results/htmls/Muzaffar.html
  • 3. CAP and ACID
    CAP 
    정합성(Consistency). The client perceives that a set of operations has occurred all at once.
    가용성(Availability). Every operation must terminate in an intended response.
    단절내성(Partition tolerance). Operations will complete, even if individual components are unavailable.
    ACID
    원자성(Atomicity). All of the operations in the transaction will complete, or none will.
    일관성(Consistency). The database will be in a consistent state when the transaction begins and ends.
    고립성(Isolation). The transaction will behave as if it is the only operation being performed upon the database.
    지속성(Durability). Upon completion of the transaction, the operation will not be reversed.
    http://queue.acm.org/detail.cfm?id=1394128
  • 4. Google BigTable System
    BigTable Master Server
    Chubby Server
    Tablet Server #2
    Tablet Server #1
    MapReduce Worker #2
    MapReduce Worker #1
    GFS #2
    GFS #1
  • 5. Google BigTable Architecture
    GAE Application
    DataNucleus ORM
    (JDO or JPA Implementation)
    Low Level API
    Datastore
    Meta 0 tablet
    Big Table
    Meta 1 tablet
    Meta 1 tablet
    Meta 1 tablet
    Tab 1
    Tab 2
    Tab 3
    Tab 4
    Tab n
    GFS(Google File System)
  • 6. Google BigTable
    Dynamic control over Data Layout and Format
    Not support a full Relational Model
    Dynamic control over serving data from memory or disk
    Column oriented
    The map is indexed by
    A row key
    A column name
    A timestamp
    BigTable
    Key
    TimeStamp / Version
    Columns
    Index
  • 7. LDAP(Lightweight Directory Access Protocol)
    The Lightweight Directory Access Protocol (LDAP) is an application protocol for accessing and maintaining distributed directory information services over an Internet Protocol (IP) network.
    Directory services may provide any organized set of records, often with a hierarchical structure, such as a corporate electronic mail directory. Similarly, a telephone directory is a list of subscribers with an address and a phone number.
  • 8. Google App Engine Product Roadmap
    Features on Deck
    App Engine out of Preview
    SSL access on custom domains
    Full-text search over Datastore
    Support for Python 2.7
    Support for running MapReduce jobs across App Engine datasets
    Bulk Datastore Import and Export tool
    Improved monitoring and alerting of application serving
    Logging system improvements to remove limits on size and storage
    Integration with Google Storage for Developer
  • 9. DataStore의 제약사항
    Write가 느림
    검색 가능한 Text 사이즈: 500 자
    Entity 최대 사이즈: 1Mb
    최대 Batch (save/delete) : 500 Entity
    SQL을 사용할 수 없음
    데이터베이스의 제약조건을 적용할 수 없음
    Aggregation 함수를 지원하지 않음 (count, avg)
    쿼리가 최대 조회 레코드: 2000
    쿼리에서 Equal 연산 이외의 조건은 1개만 추가 가능
    트랜잭션의 단위는 Entity Group에 한정 됨
  • 10. Datastore API and ORM API
  • 11. Datastore API(Low Level API)
    모든 Entity는 Kind가 지정
    Entity는 하나이상의 Property를 포함
    특정 Kind에포함된 Entity는 동일한 Property를 갖지 않음
    Entity의 Property는 이름은 동일하지만 데이터 유형이 다를수 있음
    Schemaless의 특성
    ID는 지정하지 않을 경우 자동 할당 됨
  • 12. Entity
    App Engine Datastore에서 관리하는 객체는 Entity
    Entity는 하나의 Key를 포함
    모든 Entity 중 유일한 구분자
    Key 의 구성
    Path
    Parent Entity의 key
    Entity의 Kind
    Entity에 할당된 이름
    App이 할당한 값
    Datastore가 지정한 numeric ID
    지원 데이터 타임
    integers, floating point values, strings, dates, binary data 등
    각 엔티티는 하나이상의 Property를 포함
    하나의 Property에 복수의 데이터 타입 저장 가능
    하나의 Property에는 복수의 값이 저장 가능
    복수의 값들의 데이터 유형은 다를 수 있음
  • 13. Entity Model
    Settings
    Kind : Table
    Key : Primary Key
    Entity Group : Partitioning
    Property : column
  • 14. Entity Example(Create)
    import java.util.Date;import com.google.appengine.api.datastore.DatastoreService;import com.google.appengine.api.datastore.DatastoreServiceFactory;import com.google.appengine.api.datastore.Entity;// ...DatastoreServicedatastore = DatastoreServiceFactory.getDatastoreService();Entity employee = new Entity("Employee");employee.setProperty("firstName", "Antonio");employee.setProperty("lastName", "Salieri");Date hireDate = new Date();employee.setProperty("hireDate", hireDate);employee.setProperty("attendedHrTraining", true);datastore.put(employee);
  • 15. Entity Example(Scheme Free)
    import java.util.Date;import com.google.appengine.api.datastore.DatastoreService;import com.google.appengine.api.datastore.DatastoreServiceFactory;import com.google.appengine.api.datastore.Entity;// ...DatastoreServicedatastore = DatastoreServiceFactory.getDatastoreService();Entity employee = new Entity("Employee");employee.setProperty("firstName", “Sun-Jin");datastore.put(employee);
  • 16. Entity Group
    Entity Group는 부모 Entity를 갖는 Entity의 Group
    하나의 트랜잭션에서 여러 Entity를 조회, 생성, 수정, 삭제 할 수 가능하지만 그 범위는 단일 Entity Group에 국한
    부모 Entity 를 가지 않은 Entity 를 Root Entity라고 함
    Entity Group은 하나의 Root Entity와 Child Entity로 구성
    하나의 EntityGroup은 동일한 분산 네트웍 서버에 저장됨 (Tablet 서버)
  • 17. Entity Group
    Entity Group
    Root Entity
    (xxxx)
    Child Entity
    (xxxxyyyy)
    Child Entity
    (xxxxyyyyzzzz)
  • 18. Entity Example(Read)
    import java.util.Date;import com.google.appengine.api.datastore.DatastoreService;import com.google.appengine.api.datastore.DatastoreServiceFactory;import com.google.appengine.api.datastore.Entity;// ...DatastoreServicedatastore = DatastoreServiceFactory.getDatastoreService();
    Key rootKey = KeyFactory.createKey("Employee", “CEO");
    Entity employee = datastore.get(rootKey);
  • 19. Entity Example(Update)
    import java.util.Date;import com.google.appengine.api.datastore.DatastoreService;import com.google.appengine.api.datastore.DatastoreServiceFactory;import com.google.appengine.api.datastore.Entity;// ...DatastoreServicedatastore = DatastoreServiceFactory.getDatastoreService();
    Key rootKey = KeyFactory.createKey("Employee", “CEO");
    Entity employee = datastore.get(rootKey);
    employee.setProperty("firstName", “Sun-Jin");employee.setProperty("lastName", “Jang");datastore.put(employee);
  • 20. Entity Example(Delete)
    import java.util.Date;import com.google.appengine.api.datastore.DatastoreService;import com.google.appengine.api.datastore.DatastoreServiceFactory;import com.google.appengine.api.datastore.Entity;// ...DatastoreServicedatastore = DatastoreServiceFactory.getDatastoreService();
    Key rootKey = KeyFactory.createKey("Employee", “CEO");
    Entity employee = datastore.get(rootKey);datastore.delete(employee);
  • 21. Entity Example(Query)
    // ...DatastoreServicedatastore = DatastoreServiceFactory.getDatastoreService();
    Query q = new Query(”employee")
    q.addFilter("lastName", Query.FilterOperator.EQUAL, lastNameParam);
    q.addFilter("height", Query.FilterOperator.LESS_THAN, maxHeightParam);
    PreparedQuerypq = datastore.prepare(q);
    for (Entity result : pq.asIterable()) {
    String firstName = (String) result.getProperty("firstName");
    String lastName = (String) result.getProperty("lastName");
    Long height = (Long) result.getProperty("height");
    }
  • 22. Java ORM API
  • 23. DataNucleus
    DataNucleus attempts to make the whole process of persisting data a transparent process.
    The idea revolves around the developer having a series of Java classes that need persisting.
    http://www.datanucleus.org/products/accessplatform/development_process.html
  • 24. JDO Model Define(Annotation)
    import javax.jdo.annotations.PersistenceCapable;@PersistenceCapablepublic class Employee {    @PrimaryKey    @Persistent(valueStrategy= IdGeneratorStrategy.IDENTITY)    private Key key;
        @Persistent    private String firstName;    @Persistent    private String lastName;    @Persistent    private Date hireDate; }
  • 25. JDO Model Define(Annotation)
    import javax.jdo.annotations.PersistenceCapable;
    // …@PersistenceCapablepublic class Employee {    @PrimaryKey    @Persistent(valueStrategy= IdGeneratorStrategy.IDENTITY)    private Key key;
        @Persistent    private String firstName;    @Persistent    private String lastName;    @Persistent    private Date hireDate; }
  • 26. JDO Model(Create)
            PersistenceManager pm = PMF.get().getPersistenceManager();        Employee e = new Employee("Alfred", "Smith", new Date());        try {            pm.makePersistent(e);        } finally {            pm.close();        }
  • 27. JDO Model(Update)
    public void updateEmployeeTitle(User user, String newTitle) {    PersistenceManager pm = PMF.get().getPersistenceManager();    try {        Employee e = pm.getObjectById(Employee.class, “Sun-Jin”);        if (titleChangeIsAuthorized(e, newTitle) {            e.setTitle(newTitle);        } else {            throw new UnauthorizedTitleChangeException(e, newTitle);        }    } finally {        pm.close();    }}
  • 28. JDO Model(Delete)
    public void updateEmployeeTitle(User user, String newTitle) {    PersistenceManager pm = PMF.get().getPersistenceManager();    try {        Employee e = pm.getObjectById(Employee.class, “Sun-Jin”);
            pm.deletePersistent(e);     } finally {        pm.close();    }}
  • 29. JDO Model(Query)
    import java.util.List;import javax.jdo.Query;// ...    Query query = pm.newQuery(Employee.class);    query.setFilter("lastName == lastNameParam");    query.setOrdering("hireDatedesc");    query.declareParameters("String lastNameParam");    try {        List<Employee> results = (List<Employee>) query.execute("Smith");        if (!results.isEmpty()) {            for (Employee e : results) {                // ...            }        } else {            // ... no results ...        }    } finally {        query.closeAll();    }
  • 30. Demo
  • 31. 감사합니다.
    http://code.google.com/p/devmento-gae-seminar