Getting Started with MongoDB Using the Microsoft Stack
•Download as PPTX, PDF•
1 like•873 views
Speaker: John Randolph, Sr. Software Developer, Gexa Energy Level: 100 (Beginner) Track: Developer Gexa has implemented several applications using MongoDB as a document repository storing multiple types of files (PDF, XLS, CSV, etc.). This entry level session is intended to share what we’ve learned in developing and deploying our first applications in an on premise, Microsoft environment. We’ll provide architectural and development information about what we’ve done. The focus is to help get your projects up-to-speed more quickly. This will be useful to teams moving from pilot to production and for developers getting started with the .Net MongoDB drivers. Plenty of code samples will be shown. We’ll discuss our successful engagement with MongoDB Consulting to help us design and deploy a high-quality production environment. What You Will Learn: - Ideas how to store and retrieve documents of different sizes, types, and volumes. We’ll describe the storage, partitioning and indexing techniques used that provide sub-second retrieval from collections with over 100 million records. - The issues addressed moving to production, including: backup, disaster recovery, SSL, using replica sets, implementing authorization and authentication, changing default setting, and creating a full path-to-production set of environments. - A successful pattern for building applications with .Net, providing teams some ideas to jump-start their development along with tips and tricks for using the .Net drivers.
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Getting Started with MongoDB Using the Microsoft Stack
Similar to Getting Started with MongoDB Using the Microsoft Stack (20)
More from MongoDB
More from MongoDB (20)
Recently uploaded
Recently uploaded (20)
Getting Started with MongoDB Using the Microsoft Stack
- 1. Getting Started with MongoDB Using the Microsoft Stack johnerandolph@gmail.com John Randolph Gexa Energy
- 2. Overview • Background • From pilot to production • Database access and organization ideas • C# tips & tricks for MongoDB • Conclusions & Questions 2
- 3. Background 3
- 4. Gexa Energy • Retail electric provider for commercial and residential customers. • Subsidiary of NextEra Energy. • Headquarters in Houston Texas. • Development is primarily • On-Premise • Oracle • .Net • A lot of EDI 4
- 5. Why MongoDB? • We needed to create several document storage and management applications • Not JSON; PDFs, XML, Excel, Word and text documents • Other alternatives were not attractive • Oracle and high end EMC storage • Reading original files directly from disk • Specialized document management systems 5
- 6. Document Management using MongoDB First project was EDI Document management • No method to access original source documents • Wanted to provide one Second project was archiving legacy documents • Retiring systems • Multiple types of files per business transaction (XML, PDF, XLS) • Needed to store and provide access 6
- 7. Pilot M101N: MongoDB for .NET Developers M102: MongoDB for DBAs Deployed to Test Server Happy Developers & Business Users New Requirements Management Approval • Hardware • Commercial agreement with MongoDB • MongoDB for consulting engagement • Found SQL Server DBA who wanted to take on MongoDB
- 8. From Pilot to Production • Engagement with MongoDB Consulting • 4 day working session • Prerequisites • Hardware was in house and built • Availability of team members from Infrastructure, Applications, Database • Goals for engagement • Install MongoDB • Have operational, functioning production and test systems • Have an application review 8
- 9. Mongo Environment • CISCO C240 (5) • 2U height • Up to 24 SFF 12GB SAS • Only bought 7 • 2x300GB R1 for OS • 5x600GB R5 (2.1TB) • 64GB Memory • Windows Server 2012 • MongoDB 3.2 9
- 10. Database Access and Organization Ideas There is no such thing as code that can’t be improved. 10
- 11. MongoDBContext • Only one database connection object needed per application. • Responsibility • Obtain connection string • Hold the connection • Support SSL • Access collections 11
- 12. Data Access Layer Interface Implementation 12
- 13. EDI Database Organization Requirements • Large numbers of TSV text documents ~ 80M, 1M per month • Loaded in batch, never updated • Less than 1000 of documents are > 16MB. • Require GridFS to store 13
- 14. EDI Database Organization • @DocInfo collection • Indexes, key fields, id’s of actual documents • Data is partitioned by date & size • Small index collection • Data collections are partitioned • Transparently store large docs • Old collections not updated 14
- 15. Extend DA Layer for GridFS & Dynamic Names Interface Implementation 15 Added GridFS collection Added collectionName to Documents Collection
- 17. Access Data Transparently from Application 17
- 18. Extending the ObjectId Subsequent projects needed to store multiple files per document 18
- 19. ObjectId • Designed to support simultaneous updates from many clients • If that is not your use case, you can do a lot with 12 bytes 19
- 20. Leveraging the ObjectId • Leveraging a portion of the ObjectId 20
- 21. Using the Extended ObjectId • Multiple documents per transaction 21
- 22. Saving and Accessing Data Interface Use id to find 22
- 23. C# Tips & Tricks for MongoDB Ideas to save you time getting started 23
- 24. C# Tips & Tricks for MongoDB • Using the database context • Saving and retrieving files • Paging • Serialization and data annotation • Builders vs Linq • Two other tips 24
- 25. Using the DbContext in an MVC Application 25 Add ControllerFactory in App_Start directory Update Global.asax.cs to use the new ControllerFactory Inject dbContext when creating controller
- 26. Using the DbContext in a DotNet Core MVC Application 26 Add to Startup.cs
- 27. C# Tips & Tricks for MongoDB • Using the database context • Saving and retrieving files • Paging • Serialization and data annotation • Builders vs Linq • Two other tips 27
- 28. Saving and Retrieving Files Console Application 28 Web Application
- 29. Importing a file in an MVC application - View 29 enctype = “multipart/form-data” <input type=“file”
- 30. Importing a file in an MVC application - Action 30
- 31. Displaying Files (PDFs) Display in Browser Query Download 31
- 32. Returning Other Document Types Search for “MIME Types” or “Media Types” File Extension Word (doc) application/msword Word (docx) application/vnd.openxmlformats-officedocument.wordprocessingml.document Excel (xls) application/vnd.ms-excel Excel (xlsx) application/vnd.openxmlformats-officedocument.spreadsheetml.sheet Powerpoint (.ppt) application/vnd.ms-powerpoint Powerpoint (.pptx) application/vnd.openxmlformats-officedocument.presentationml.presentation Text application/text PDF application/pdf XML application/xml 32
- 33. C# Tips & Tricks for MongoDB • Using the database context • Saving and retrieving files • Paging • Serialization and data annotation • Builders vs Linq • Two other tips 33
- 35. Paging in a MVC application • PagedList.MVC NuGet Package • Return IEnumerable • Works Great! • Sub second response time on 100M records db.getCollection('Customer') .find({Name:’bob’}) .skip(0) .limit(25) Action Query 35
- 36. C# Tips & Tricks for MongoDB • Using the database context • Saving and retrieving files • Paging • Serialization and data annotation • Builders vs Linq • Two other tips 36
- 37. Serialization / Data Annotation 37
- 38. Altering Field Names in Database • Good meaningful variable names • Smaller, more database friendly field names 38
- 39. Unexpected Fields in Database 39 New field Exception thrown
- 40. Unexpected Fields in Database [BsonIgnoreExtraElements] 40 [BsonIgnoreExtraElements] No Exception
- 41. Prevent Fields From Being Written [BsonIgnore] 41 Not serialized
- 42. Prevent Null Field From Being Written [BsonIgnoreIfNull] 42 [BsonIgnoreIfNull] elements not written
- 43. Dates [BsonDateTimeOption] public DateTime startDate {get;set;} 3/4/2017 8:06:41 PM 3/5/2017 2:06:41 AM [BsonDateTimeOptions(Kind = DateTimeKind.Local)] public DateTime startDate {get;set;} Console.WriteLine(dt) 3/4/2017 8:06:41 PM 3/4/2017 8:06:41 PM 43 Insert DateTime.Now into database Read into dt Console.WriteLine(DateTime.Now); Console.WriteLine(dt)
- 44. C# Tips & Tricks for MongoDB • Using the database context • Saving and retrieving files • Paging • Serialization and data annotation • Builders vs Linq • Two other tips 44
- 45. Retrieving Select Fields • Linq doesn’t provide you to exclude certain fields 45
- 46. Dynamic Queries • Builders allow you to build dynamic queries 46
- 47. C# Tips & Tricks for MongoDB • Using the database context • Saving and retrieving files • Paging • Serialization and data annotation • Builders vs Linq • Two other tips 47
- 49. Finding Documentation Hint • .Net drivers have evolved, searches often return out of date information • Start here: http://mongodb.github.io/mongo-csharp-driver/ 49
- 51. In Conclusion • Getting started with MongoDB is incredibly easy. • Treat it as you’d treat a traditional database • MongoDB consulting was a great way to get operationally ready. • As you’re developing: • Understand the reading and updating requirements for your data • Index what you need to search for • Hide information in index key value • Reduce fields, indexes • GridFS is not all or nothing 51
Editor's Notes
- Introduce self Ask about audience
- I’m going to discuss the 1st two projects I developed at Gexa. What the problems we were trying to solve. How we went from idea to production. The rest is very code based. The two projects are similar, but evolutionary. Talk about db org & show how we do data access and how that evolved I’ll show c# things I learned that would have saved me time if I had known them, Wrap & Questions
- 3 parts to deliverying electiicity Use Service providers Utilities We make money pricing fairly and accurately so we can buy EDI is how we comuicate changes of service historical usage demand Then IT good at processing relational data Eventually put it back together in a data warehouse The problem we have is this is complex data and often the anylists need to see the original
- First project was EDI Documents We do what all good IT orgs do, we take the transactions and obliterate them, loading the pieces into relational database. MongoDB allows us to store complete documents Multiple indexes can be created to quickly access For example, with 100 million documents, we can present a list of all of a customers documents with sub-second response time View the document in a number of ways Make documents available to other applications via a WebApi Another use case is archiving legacy systems data Legacy systems generate documents on the fly (PDFs, supporting XLS & XML documents) Solution: Generate all historical documents and load into MongoDB Like EDI, index in a variety of ways to provide quick access to PDF and supporting documents Make available to other applications via a WebApi Finally, we’ve implemented print management Similar to #2, but with new, not legacy data
- DBA Classes M102 mongodb for dba’s M201 mongodb performance M122 Getting started with mongodb cluster management
- Should I talk the results on the next slide
- Results Mongo deployed on 5 servers, 4 prod, 1 test One node was non-voting standby member, used for Disaster Recover. SSL everywhere, both to the clients & between replica set members UAT env that mirrors production. Gives the DBA a place to test his stuff Dev, QA, & STG single node systems SSL everywhere, between servers & out to clinets Ops manager installed Application review LDAP was a no go Up for a year. Only one outage that was our fault.
- Backup – don’t‘ have to backup collections that aren’t changing as often If you have high cost storage, you could locate other collections on lower cost medai
- You don’t have to decide/check. Handle rare exception.
- 2nd project has multiple documents per transaction. If I used the method I just showed, I’d have to store both the Id & collection name. I’m going to show how to extend the object id to fix that.
- Put file date in ObjectId so I don’t have to maintain another field & index
- In later slide
- I’m going to show ideas which are basically building up parts of my applications
- You could just use a static class. Advantage over static is for testing
- Advantage over static is for testing
- Couldn’t figure out how to get explain
- Need to confirm linq actually brings back full object
- Need to confirm linq actually brings back full object
- Mongo drivers have evolved Often pull back out of data doc Start here. You can find the area. Use the key works to search futher
- Address fault tolerance and disaster recover Develop operational plans Change passwords & ports Secure connections Scanning a database is fine for analytics, not for speedy retrieval of documents