5 Key Considerations for Data Modeling
•Download as PPTX, PDF•
1 like•2,078 views
See the companion webinar at: http://embt.co/1sc5YZl According to a recent report from SINTEF, an independent research organization in Scandinavia, 90% of all the data in the world has been generated over the last two years. Where does it all go? And how do we make sense of it all? To get control of the data, you have to know more about that data and how it relates to other data. Data modeling provides the infrastructure needed to capture the right level of information about the data and its associated metadata. In this webinar, Torquil Harkness will discuss five key considerations for effective data modeling, including: + Aspects of model design + Attributes such as naming standards + The importance of planning for future growth
Recommended
More Related Content
Viewers also liked
Viewers also liked (17)
Similar to 5 Key Considerations for Data Modeling
Similar to 5 Key Considerations for Data Modeling (20)
More from Embarcadero Technologies
More from Embarcadero Technologies (20)
Recently uploaded
Recently uploaded (20)
5 Key Considerations for Data Modeling
- 1. EMBARCADERO EMBARCADERO TTEECCHHNNOOLLOOGGIIEESS 5 Key considerations for data modeling Presenter: Torquil Harkness Technical Writer torquil.harkness@embarcadero.com
- 2. EMBARCADERO TECHNOLOGIES Data…. • Every Minute of every day we create: - More than 204 million email messages - Over 2 million Google search queries - 48 hours of new YouTube videos - More than 100,000 tweets • Megabyte, gigabyte, terabyte, petabyte, exabyte, zettabyte, yottabytes The new NSA facility in Utah can hold 5 zettabytes of data. To store only 1Zb of data, it would take 62.5 billion iPhones!
- 3. EMBARCADERO TECHNOLOGIES Island Clinic - Ebola Treatment Centre Data… used to save lives WHO /C. Black http://www.who.int/features/2014/liberia-ebola-island-clinic/en/
- 4. EMBARCADERO TECHNOLOGIES Data… used to save lives Mobile phone location data Integrating data sets from anonymised mobile phone usage and demographic indicators. mage Credit: PLOS Currents.
- 5. EMBARCADERO TECHNOLOGIES Topics • Model Design • Planning for Growth • Naming Standards • Data Lineage • Big Data
- 6. EMBARCADERO TECHNOLOGIES Model Design Data Sales Logic
- 7. EMBARCADERO TECHNOLOGIES Model Design Visualizing can be useful to see the results 75% ………………….. 25% ………………….. 25% …………………. 12.50% ……………... 12.50% ……………… All purpose flour Cake flour Granulated sugar Butter Eggs
- 8. EMBARCADERO TECHNOLOGIES Model Design
- 9. EMBARCADERO TECHNOLOGIES Model Design • Logical Model - The organisation of your data. Basically, the Blueprint. • Physical Model - The ‘physical structure’ of the data in the database.
- 10. EMBARCADERO TECHNOLOGIES Model Design • Logical Model - The organisation of your data. Basically, the Blueprint. • Physical Model - The ‘physical structure’ of the data in the database. • Normalisation - Eliminating redundancy and mitigating corruption. - 1NF:the key, 2NF: the whole key, 3NF:nothing but the key. So help me Codd, Edgar F.
- 11. EMBARCADERO TECHNOLOGIES Model Design Customer Name Customer Address Customer Tel No. Product Cost Holmes, S 221B Baker St, London +44 1632 960957 Hat 44.99 Holmes, S 221B Baker St, London +44 1632 960957 Pipe 22.99 Fletcher, J 698 Candlewood Lane, Cabot Cove, Maine +001 1632 960428 Typewriter 129.99 Fletcher, J 698 Candlewood Lane, Cabot Cove, Maine +001 1632 960428 Hat 44.99
- 12. EMBARCADERO TECHNOLOGIES Model Design Customer Customer ID Customer Name Customer Address Customer Tel No. 20 Holmes, S 221B Baker St, London +44 1632 960957 20 Holmes, S 221B Baker St, London +44 1632 960957 30 Fletcher, J 698 Candlewood Lane, Cabot Cove, Maine +001 1632 960428 30 Fletcher, J 698 Candlewood Lane, Cabot Cove, Maine +001 1632 960428 Orders Order ID Customer ID ORD001 20 ORD002 30 ORD003 Order ID Product ID Quantity ORD001 001 1 ORD001 002 1 ORD002 003 1 ORD002 001 1 Product ID Product Cost 001 Hat 44.99 002 Pipe 22.99 003 Typewriter 129.99 Order details Products
- 13. EMBARCADERO TECHNOLOGIES Naming Standards • An example of a very short naming standard. tNYEZC - table of NY Employees Zip Code.
- 14. EMBARCADERO TECHNOLOGIES Naming Standards • Be clear and understandable to everyone. • Add a detail of description – tbl for a table etc. • Use a ‘naming standards template’ to ensure consistency.
- 15. EMBARCADERO TECHNOLOGIES Planning for Growth • Each engine of a jet on a flight from London to New York generates 10TB of data every 30 minutes. Source: Pratt and Whitney. • 90% of the World’s data generated over the last two years. Source: Science Daily.
- 16. EMBARCADERO TECHNOLOGIES Planning for Growth • Planning for Storage • Predicting Growth
- 17. EMBARCADERO TECHNOLOGIES Planning for Growth
- 18. EMBARCADERO TECHNOLOGIES Data Lineage • FACT: 73.8 percent of facts are made up!
- 19. EMBARCADERO TECHNOLOGIES Data Lineage • The Data Trail
- 20. EMBARCADERO TECHNOLOGIES Data Lineage • The Data Trail
- 21. EMBARCADERO TECHNOLOGIES Big Data
- 22. EMBARCADERO TECHNOLOGIES Big Data
- 23. EMBARCADERO TECHNOLOGIES Big Data
- 24. EMBARCADERO TECHNOLOGIES Summary • Model Design • Planning for Growth • Naming Standards • Data Lineage • Big Data
- 25. EMBARCADERO TECHNOLOGIES Concluding Remarks 25
- 26. EMBARCADERO TECHNOLOGIES Thank you! • Product Videos: http://www.embarcadero.com/products/er-studio/ product-videos • Wiki and Documentation: http://docs.embarcadero.com/ • Learn more about the ER/Studio product family: http://www.embarcadero.com/data-modeling • Trial Downloads: http://www.embarcadero.com/downloads • To arrange a demo, please contact Embarcadero Sales: sales@embarcadero.com, (888) 233-2224 26
Editor's Notes
- 90 percent of the worlds data has been created in the last two years? Yes I can believe that, when i look at my iphoto library, the growth is definitely exponential. In fact I have read a few times that in total, the worlds data is doubling every two years, so at least it looks like there won't be any slowdown. It has grabbed most business headlines over the last 5 years, it sometimes it can feel like we are surrounded by an ocean of data. It is the 1000lb elephant in the room, except everyone IS actually talking about it! So with all of this new data created, how useful is it.
- Right now with the ebola epidemic affecting so many lives, data is being used to create an effective response. This is a father and daughter in Liberia, Mr Nyenati Kaffia. They have just been checked out of the Island treatment center in Monrovia. A new treatment center that was set up to fight the epidemic. He said that it was bittersweet that day as he tragically lost his son, but he and his daughter survived ebola. It is an example of how large amounts of fairly simple data, can make a crucial difference. In such countries with limited resources, information can make a crucial difference.
- The Information is gathered from mobile phone location data. Using the data correctly it was noted that when the epidemic hits certain areas it could be seen that the movement of the population changes. They are now using this data to find the most effective areas to set up emergency treatment centers. Saving lives like Mr Nyenati Kaffia and his daughter, and making those limited resources as effective as possible. And so just simple mobile phone location data, used effectively can really make a difference. Certainly great to see..
- Today we are going to look at some of the benefits of data modelling. I shall be using Embarcadero's Data Architect. There are many aspects of data modelling and it is a subject that can be very in depth. I shall be touching on a few important aspects, so let's take a short look at 5 of these. Model Design - We will look at some of the basic aspects of model design and how, high quality, large designs can be achieved in a short period with advanced data modeling. Planning for Growth - Future-proofing your design and being able to scale up for growth without having any detrimental effects is key to building a good foundation for your project. One of my previous positions I was able to see what can happen if things go wrong and you have not prepared for successful growth. I hasten to add, it was not one of my projects and mercifully there were a couple of tenacious and flexible developers that were able to put in a lot of extra work and pull things around. valuable lessons were learned and what surprised me was how quickly, other companies quickly take up the slack and filled the gaps in the niche that was created by us in the first place. As a company they are still on top and doing great business. However this is not always the case and they were very lucky. Naming Standards - Whether you are importing your design using our reverse engineering wizard or building your model from the bottom up, naming standards are useful in mitigating misinterpretation. Data Lineage - How important is it to know the origins of your data? Well very, if you are covered by any sort of data governance. Simply to know where your data comes from will help you to prove your results. Big Data - No webinar would be complete without the obligatory 'tip of the hat' to Big Data. With headlines like 'a single airliner jet engine producing 10Tb of data in just 30 mins' and, '90 percent of data on the planet being generated in the last 2 years', it continues to grab headlines and appear in our inboxes, so we shall touch on it today too. ER/Studio has recently had some exciting updates that allow you to work with big data platforms so we shall have a look at those too.
- Organisations and individuals are tight for time. While years ago it was ok for large projects to take many years of planning, having the same approach now would mean that the technology may have moved on before you have finished and made your work obsolete. Growth can be exponential and capitalizing on this can be the difference in being a success or not.
- It is easy to understand the importance of a data model. We often reach for a scrap of paper, back of a napkin, to visualise our ideas. We are creatures in a visual world. That is why we are always told to write down our goals, we accomplish more when we see it written visually, it is just who we are. We understand more when we see it.
- And unless you are a Terminator, R2D2 or another kind of robot, or cyborg for those who are picky, you most likely benefit from breaking down that visual data. Staring at a spreadsheet or looking at the piechart of the same data is an easy example of explaining the difference of this. So this data of total data, would be impossible to understand in a pie chart, but is easily recognisable in a bar graph. Design is key. No amount of smoothly written code can account for design errors. Normalizing too many times can cause performance problems. It might be possible to normalize down to the 6th normal form but not a great idea if your hardware has a seizure. So when we look at model design and future proofing there is an importance to getting it right.
- Let us just touch on the different areas of the model design. Logical Model: So the logical model is normally developed before the physical model. It is the business and functional requirements of your system and allows you to determine the organisation of the data in the database. You can think of it as your blueprint. Physical Model: The physical Model shows how the data is actually stored in the database. Specifying the data type and how the tables are stored. Keeping the two separate will keep things from getting too complex. Although it is no guarantee that things will not get complex. Starting with a good physical design will ensure that your logical design is strong. Adding business rules such as phone number and address to the customer entity, can also ensure less problems later on. For instance when the customer orders are being delivered and there is no phone number for the courier to find the house, resulting in a missed delivery and an angry customer, can be avoided, because the system will not allow an order without a telephone number. Domains are very useful in creating entity attributes or columns in your models as they are basically attribute templates that can be applied repeatedly. You build them like an attribute, specifying names, datatype properties etc. This is great as you can then ensure that anything that references it, will comply. Let me show you this is action. .....
- There are so many things we can go in to with model design and these are only a few. while we are talking about model design I also wanted to touch on normalisation. Normalization is the process of eliminating redundancy and streamlining your design. Separating out and relating your tables. There are plenty of videos out there that explain this much better, but for a point of reference, a quick and rough example is the following:
- So you want to keep a record of your customer orders on your database. You have their name, their address and what they ordered. The records would show the following: Customer ID, Name, Address, what they ordered. The table could show the following line of "Customer ID, Name, Address, product 1, product 2, product 3 etc.
- Normalisation sounds complicated, but it is common sense. normalisation is the process of removing any redundancy and duplicates. Mitigating the possibility of updating duplicate data in only one location, and therefore causing a data corruption. and of course removing redundant data, cuts down on storage and database size. Of course splitting the data in to many different related tables is possibly going to impact the IO of your database and in some cases a denormalisation may be needed for performance Once you have built or imported your model, you can use the a validation tool to check that all is well, before you move forward and generate a physical database for example. Staying with ERStudio DA, i can show that in action. Here we have a basic logical database that i have made. I could also reverse engineer a database from an existing database. That is a very exciting tool indeed. It allows you to model from any major platform. Here is a Teradata database that I have previously imported with the reverse engineer wizard. Ok so moving back to my much smaller database, I can go here and validate the model. Let's see if we find any errors.
- So to touch on naming standards, they are nomally in place already in your organisation, however because of the history of the space allowed for names, historically very small, what can be in place is some sort of naming code. This is great if you are hacked, while the hacker spends time trying to work out that tNYEZC is the table of NY Employees Zip Code. you can cut his unauthorised access maybe :-)
- It used to be that naming conventions were restricted by the length of characters. however this is generally not the case anymore and although Oracle allows for 30 bytes and SQL server allows for 128, we recommend keeping clear and concise names. What might make sense to you, needs to be clear for people after you leave or get promoted. Some people get a little upset at redundancy, so adding rpt before the name if it is a report, however I would say that there is no harm in it and it can certainly help in reverse engineering, to have everything labelled nicely. The naming standards template can allow you to enforce naming standards and also apply them later. We can have a quick look at that.
- These headlines were going to be in my Big Data slide... but really we should remember that you cannot have Big Data, without first having little data. There are a couple of aspects that i would like to touch on here. Planning for storage and predicting growth.
- Two main issues where database performance is concerned, is fragmentation and space. Data fragmentation can occur if not planned for in the creation phase and well, we all know what happens when we run out of space. Maintaining good IO speeds, you can plan for this in the physical design phase. Separating out tables from their indexes, and separating out columns in tables that are often referenced together. If you are doing this from the ground up, then hopefully you will get a chance to separate out the databases on to the drives you have available. Going as granular as splitting the contents of tables on to separate physical disks or arrays of disks. Most people have disk arrays, raid 5 or 10. Raid 5 having slower write speeds, but if you are lucky enough to be using some good hardware SSD drive arrays or a big flash memory array, then you have it better than most. hoever you may have database sizes to warrant that kind of expense, so what the left gives, the right takes away. :-) Not getting it right can result in extreme slowness or even downtime. In a previous job we were given separate drives for our logs and various numbered data drives. it was only when we have serious lag highlighted on a stress test, that we found out that the separate drives were actually virtual drives on a single sata archive array. A lack of communication and missed steps from our team because we had originally started off building the server as a development demo, but had not requested from the systems team to move it to pre-production.
- In ERStudio, as a starting point in planning for growth you can use the projected row count. We have a useful utility called the Capacity Planner which accurately predicts growth. The tool can be used from within the table editor and can predict table growth from the capacity planning tab of the table editor. covering things like growth rate, growth type and max size etc Using the capacity planner utility outside the table editor, you can forecast your storage requirements, helping you budget and plan your engineering. We have a nice section in the wiki for explaining the facets of the Planner so i shall not cover it here, but I shall put some links at the end and you can have a look at a later date. For databases that let you specify initial extent sizes, such as Oracle, set the initial extent size to the estimated size in order to avoid data fragmentation as the table grows.
- If you cast your mind back to your maths teacher at school. They told you that you should always show your workings. Because simply, if you do not show your working… we do not know what you have done… or more importantly, what you have not done. How do you get to your facts and figures?
- Data Lineage can be thought of as a data trail. The Hansel and Gretel trail of breadcrumbs of your data. Critically important for data integrity and trust by your employees and customers. Used to verify your end results, forecasting results, data governance or tracing your ETL transformations. After all, it is common for your systems to be in many places at once and on differing systems. Acquisitions, mergers, outdated systems and newly implemented systems give a common diverse topography to most company systems. Dealing with this diversity can be a challenge and keeping track of where the data originates is what data lineage though an ETL process is all about. What-if scenarios are possible. If we picture a medium sized organisation. Something as simple as changing a supplier, running a marketing drive to increase sales can all utilise your data flow, and in turn your data lineage to project any number of what-if scenarios.
- In the Data Lineage tab within Data Architect, there is the option to add data lineage information. We do have the option to ‘import’ data lineage information as well. Which can be done from SSIS and Oracle etc. This can be very useful if you have disparate systems containing data lineage information, and can be a great way to bring those disparate systems together using the reverse engineer wizard and importing data lineage. allowing you to model your complete system. You can even generate a physical database with the built in wizards, but that can be for another time. If you would like to see that in action, i am sure we have some great videos on it in action. I shall add some links at the end.
- Big data is so often in the news. There are so many headline grabbing statistics from the increase in the collection and use of data that it is easy to understand why. The rapid expansion in the volume of data and the speed at which it is used and collected. Adding to this an increasing diversity to the data, it is affecting all of us in many ways.
- There is an old story that i heard back in the military of a new military base that was being renovated. I have heard it repeated about a college campus too. It goes like this, the pedestrian footpaths had not been laid yet and there were arguments about if the paths should go around the grass at the edges, or link buildings diagonally. So the adjutant of the barracks allowed people to walk anywhere they liked, for 6 months. When the grass was finally worn down, they could match the paths to the places where the grass was worn away. I liked the military version as walking on the grass was severely punishable. The point here is that only those who walk the paths, know the best place to lay the foundations. Your customer habits, your employee processes, the supply and demand of your business are all critical factors in how your success can be developed. You know your business better than anyone, you walk across that grass every day.
- A talent shortage is also reported on frequently as the exponential growth and use of data is skyrocketing, there is a lack of people to take up the slack.... ER Studio can make your users into experts with the simple yet intuitive interface. and now it supports Hadoop Hive, MongoDB, Teradata - show link for supported platforms... Supporting the big data platforms means that reverse engineering any disparate systems, and enabling you to handle your data in one conjoined data flow.