Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Big data in the energy sector
By John Tkaczewski
Agenda
| © Copyright FileCatalyst, 20142
The Big Data in Oil & GasThe Big Data in Oil & Gas
explorationexploration datadat...
© Copyright FileCatalyst, 2014
Data-driven Oilfields and the data we gather during production.
© Copyright FileCatalyst, 2014
Digital Data gathered during exploration
© Copyright FileCatalyst, 2014
Big data during exploration…
• Logging While Drilling or Measurements While Drilling (LWD M...
© Copyright FileCatalyst, 2014
... So we collected big data now what?
Need send it for further processing
• Analysis
• Sim...
© Copyright FileCatalyst, 2014
Are we still shipping hard drives???
© Copyright FileCatalyst, 2014
Or should we use the Internet!
- FTP, Email, File Sharing
- Cloud
- CIFS
© Copyright FileCatalyst, 2014
• Last mile. Slower links between the operations and main data centers.
Satellite, 3G/4G/LT...
© Copyright FileCatalyst, 2014
Traditional file transfers will remain slow, regardless of the size of the link.
For exampl...
TCP Overview
What is TCP? Transmission Control Protocol
• Works well for most internet traffic, email, web browsing
small ...
File Transfer Issues with TCP
• Flow control limits transmission window, causes “dead air” on
the line when high latency p...
File transfer acceleration via UDP
• Ideal for bulk file transfer
• Predictable - Can send at a perfect rate
• Not affecte...
© Copyright FileCatalyst, 2014
*NA = North America
Bandwidth Improvement example
Accelerating File Transfers
© Copyright FileCatalyst, 2014
• WAN optimization hardware
• Commercial off-the shelf (COTS) file transfer acceleration
so...
© Copyright FileCatalyst, 2014
WAN optimizers
• Pros:
• Almost Transparent
• Optimizes all the network traffic including F...
© Copyright FileCatalyst, 2014
File transfer COTS
• Pros:
• Flexible software only deployment
• Enterprise tools for trans...
© Copyright FileCatalyst, 2014
Open Source file transfer applications
• Pros:
• Free
• Flexible software only deployment
•...
Considerations for FTP Acceleration
• Plays fair with other traffic on network
• Top speed at 10 Gbps file transfer, singl...
© Copyright FileCatalyst, 2014
Case study: marine exploration
Project Requirements:
• Multiple ships delivering data in re...
© Copyright FileCatalyst, 2014
Case study: marine exploration
© Copyright FileCatalyst, 2014
Case study: marine exploration
Results:
• More control over file transfers and a dramatic s...
© Copyright FileCatalyst, 2014
Forecast for tomorrow
• Big data will only get bigger
• Global connectivity will get faster...
Questions?
Upcoming SlideShare
Loading in …5
×

Big data in the energy sector

615 views

Published on

A look at the effects of big data on the energy sector

Published in: Software
  • Be the first to comment

  • Be the first to like this

Big data in the energy sector

  1. 1. Big data in the energy sector By John Tkaczewski
  2. 2. Agenda | © Copyright FileCatalyst, 20142 The Big Data in Oil & GasThe Big Data in Oil & Gas explorationexploration datadata drillingdrilling datadata productionproduction datadata Data Supply ChainData Supply Chain source to data centre forsource to data centre for analysisanalysis toto stakeholdersstakeholders Case StudyCase Study Big data movement from vesselsBig data movement from vessels to London data center toto London data center to global stakeholdersglobal stakeholders
  3. 3. © Copyright FileCatalyst, 2014 Data-driven Oilfields and the data we gather during production.
  4. 4. © Copyright FileCatalyst, 2014 Digital Data gathered during exploration
  5. 5. © Copyright FileCatalyst, 2014 Big data during exploration… • Logging While Drilling or Measurements While Drilling (LWD MWD) • Hole size and shape • structural analysis: Rock density and porosity • Acoustic scanning for rock faults • reservoir simulation • seismic analysis • fluid dynamics • Engineering and construction
  6. 6. © Copyright FileCatalyst, 2014 ... So we collected big data now what? Need send it for further processing • Analysis • Simulation • Interpretation • High Performance Computing (HPC) Centers • Data Distribution to stake holders
  7. 7. © Copyright FileCatalyst, 2014 Are we still shipping hard drives???
  8. 8. © Copyright FileCatalyst, 2014 Or should we use the Internet! - FTP, Email, File Sharing - Cloud - CIFS
  9. 9. © Copyright FileCatalyst, 2014 • Last mile. Slower links between the operations and main data centers. Satellite, 3G/4G/LTE, Microwave – slower unreliable links • High speed fat pipes between data centers. 100 Mbps fiber connections. The connections are fast but very far apart. (example Houston – London) Sending big data – 2 fold problem
  10. 10. © Copyright FileCatalyst, 2014 Traditional file transfers will remain slow, regardless of the size of the link. For example: Transferring a 10GB file from London to Houston can take several hours. The slow file transfer in turn will affect production and exploration schedules. Why traditional file transfers are so slow on the Internet? • Latency (or delay on the network) • Packet loss (percentage of data lost during transmission)
  11. 11. TCP Overview What is TCP? Transmission Control Protocol • Works well for most internet traffic, email, web browsing small ad-hoc transfers (HTTP, FTP, SFTP, SMTP etc) • Provides reliability, error checking, ordered packets in a network stream • Congestion control built in • Internet could not survive without it © Copyright FileCatalyst, 2014
  12. 12. File Transfer Issues with TCP • Flow control limits transmission window, causes “dead air” on the line when high latency present • Very aggressive in response to network congestion • Result is less than ideal performance on wireless, satellite, or long haul links • TCP Can be tuned but still not ideal © Copyright FileCatalyst, 2014 TCP is a very serial process. Each packet of data must be received before a new packet is sent = Decreased transfer speed Source File Destination File Acknowledgments Data Packet
  13. 13. File transfer acceleration via UDP • Ideal for bulk file transfer • Predictable - Can send at a perfect rate • Not affected by latency or packet loss • Congestion Control implemented in application layer • Tunable congestion control aggression • Instantly detect link capacity © Copyright FileCatalyst, 2014 Source File Destination File Acknowledgments FileCatalyst completely saturates the pipe by sending multiple blocks of data = Increased transfer speed
  14. 14. © Copyright FileCatalyst, 2014 *NA = North America Bandwidth Improvement example Accelerating File Transfers
  15. 15. © Copyright FileCatalyst, 2014 • WAN optimization hardware • Commercial off-the shelf (COTS) file transfer acceleration software • Open source UDP based file transfer accelerations applications File transfer acceleration options:
  16. 16. © Copyright FileCatalyst, 2014 WAN optimizers • Pros: • Almost Transparent • Optimizes all the network traffic including FTP • Hardware solution • Cons: • Hardware solution • Cost • Lack of file transfer tools such as automation, monitoring and QoS
  17. 17. © Copyright FileCatalyst, 2014 File transfer COTS • Pros: • Flexible software only deployment • Enterprise tools for transferring files are included • Web applications that include accelerated file transfer • Cons: • Less expensive then hardware but not free • Limited to file transfers only • Some initial configuration required
  18. 18. © Copyright FileCatalyst, 2014 Open Source file transfer applications • Pros: • Free • Flexible software only deployment • Cons: • Not adaptable to changing network conditions • no tools advanced tools (command line or API’s only) • Complex initial configuration required
  19. 19. Considerations for FTP Acceleration • Plays fair with other traffic on network • Top speed at 10 Gbps file transfer, single stream. • Maintain high speed even with high latency and packet loss. • High performance for many small files • Security, reliability, automatic re-try and resume • Automation tools, SDK, central management and configuration • Rsync and Compression built-in © Copyright FileCatalyst, 2014
  20. 20. © Copyright FileCatalyst, 2014 Case study: marine exploration Project Requirements: • Multiple ships delivering data in real time to London • London HPC center provides data analysis • Final data to be delivered to major markets in Houston and Malaysia Using FTP: • Data from ships was not deliverable • Final data deliveries taking hours or not arriving at all
  21. 21. © Copyright FileCatalyst, 2014 Case study: marine exploration
  22. 22. © Copyright FileCatalyst, 2014 Case study: marine exploration Results: • More control over file transfers and a dramatic speed improvement . • 10 Gbps line was finally used to its full extent • All file transfers are a simple and straightforward experience, logging all file transfers and exceptions • End user experience for both clients and back office processes dramatically improved
  23. 23. © Copyright FileCatalyst, 2014 Forecast for tomorrow • Big data will only get bigger • Global connectivity will get faster and more reliable • Latency will remain as we can’t increase the speed of light • Progressive tech will shorten the exploration cycle and ensure continuous success of oil and gas extraction
  24. 24. Questions?

×