This document discusses options for integrating external data into SharePoint, including Business Connectivity Services (BCS). BCS allows SharePoint to connect to external data sources and make that data accessible via external lists. However, BCS has limitations and its future is uncertain. New options like Power BI and Logic Apps provide more flexibility for building applications that integrate external data without relying on BCS. Hybrid BCS enables accessing on-premises data from SharePoint Online by publishing data through an on-premises gateway.

1. Is BCS Dead? What's Next for Bringing
External Content into SharePoint?
@jefffried
Jeff Fried
CTO, BA Insight

2. Is it Dead?

3. Today’s Session

4. Focused on Search and
SharePoint since 2004
Longtime
Search Nerd
• CTO, BA Insight
• Senior PM, Microsoft
• VP, FAST
• SVP, LingoMotors
About Jeff Fried
Passionate About
• Search
• SharePoint
• Search-driven
applications
• Information Strategy
Blog:
BAinsight.com/blog
Technet Column
“A View from the
Crawlspace”
jeff.fried@bainsight.com

5. About BA Insight


– Connectivity
– Applications -
– Classification -
– Analytics


6. 6

7. Some Reasons for Connecting to External Data





8. We live in Silos
of knowledge workers regularly access 4
or more systems to get the information
they need to do their jobs
61%
regularly access 11 or more systems
of a typical knowledge worker’s day is spent
looking for and consolidation information
spread across a variety of systems
36%
15%

9. The average $1 billion company maintains 48 disparate
financial systems and uses 2.7 ERP systems
Integration Gaps Impact Performance
Source: The Hackett Group

10. The problem is getting worse

–

–

11. Why is integration more important now than ever?

12. Read
Cross-Reference
Index
Write
On-prem Online Hybrid
Web Parts, List View
Lookup, Report
Search, CBS, Delve
Update, Create
Structured Unstructured
Developer
Site Admin
End User
Create Deploy Compose
What do we want?

14. BCS Overview

15. SharePoint Presentation + APIs
External Lists External Columns Web Parts
Business Connectivity Services
External Content Types
(ECTs) Connectors Secure Store
BCS – Zooming in a bit

16. 16
BCS Trajectory (on-prem)
Business Data Catalog
Read Only
No Tooling
BCS
SPD Tooling
VS Tooling
External List
Foundations
Alerts & Events
OData
Excel
Apps

17. 17

18. Creating a BCS Service Application

19. … and Secure Store Service

20. BCS Authentication Choices
Authentication Type Kerberos or
NTLM
Description
Passthrough Kerberos Uses credentials of current user
Passthrough NTLM Uses credentials of app pool account or anonymous user account
RevertToSelf Uses credentials of the app pool
WindowsCredentials BCS uses SSS credentials as Windows credentials. Must pass in Target
Application name.
RdbCredentials Passes in username and password to a DB, appended to the connection
string (such as when using SQL Authentication). Must pass in Target
Application name
Credentials Used to access web services that don’t support Windows authentication,
but use Basic or Digest authentication. Must pass in Target Application
name
Application-Level
Authentication
Passes credentials along with parameters when carrying out a Read
operation on external data source.

21. Authentication Options

22. Creating an External Content Type (ECT)

23. Metadata Model











24. Creating CRUD Operations

25. Business Data Web Parts
http://office.microsoft.com/en-gb/sharepoint-server-help/use-the-business-data-web-parts-HA102770986.aspx
Web Part Description
Business Data List Displays a list of items from an external list.
Business Data Item Displays an item from an external list.
Business Data Actions Displays an action of an external list.
Business Data Connectivity Filter Filters the contents of a connected Business Data Web Part by
using a list of values from an external list.
Business Data Item Builder Creates a Business Data item from parameters in a URL query
string and provides it to other Business Data Web Parts.
Business Data Related List Displays a list of child items related to a parent item from an
external list. Requires an association.

26. If you can work with a list,
you can work with an external list

27. –
–
–
–
–
–
–
–
BCS support
27
–
–
–
–
–
–
Hybrid

28. Resources
Business Connectivity Services in SharePoint 2013
http://office.microsoft.com/en-gb/sharepoint-server-help/introduction-to-external-data-HA102891586.aspx
http://technet.microsoft.com/en-us/library/ee661740.aspx
http://msdn.microsoft.com/en-us/library/jj163251.aspx
BCS & Secure Store PowerShell Cmdlets
http://technet.microsoft.com/en-us/library/ff793361.aspx
http://technet.microsoft.com/en-us/library/ee906549.aspx
Compare Feature Availability
http://technet.microsoft.com/en-us/library/jj819267.aspx
Business Data Web Parts
http://office.microsoft.com/en-gb/sharepoint-server-help/use-the-business-data-web-parts-HA102770986.aspx
BCS Meta Man by Lightning Tools:
http://lightningtools.com/bcs/bcs-meta-man.aspx

29. Hybrid Business Connectivity Services
Enables users to publish on-premises data to a list or application external
to SharePoint Online
Enables federated users to gain access to on-premises data from
SharePoint Online
Requires a two-way authentication topology using an external URL
published by reverse proxy
Connects only through OData source

30. External Content Types in O365

31. Secure store in O365

32. Prerequisites for Hybrid BCS
Two-way
authentication
topology must be
configured
Business Connectivity
Services must be
installed on-premises
On-premises instance
must have
connectivity to the
external data source
External URL
to SharePoint
on-premises must
be configured

33. Using federated credentials, a user in need of on-premises data logs on to the online app or
external list
The app or external list creates a request for data and sends it to Business Connectivity Services
Business Connectivity Services gains access to the external content type to determine how to
access the external data and what credentials to use
Business Connectivity Services retrieves a secure-channel certificate from the secure store and an
OAuth token from Windows Azure Active Directory for user authentication
Business Connectivity Services sends an HTTPS request to the published endpoint for the data
source with the certificate and token
The reverse proxy authenticates the request and forwards it to SharePoint on-premises
SharePoint on-premises retrieves the identity from the token and maps it to the on-premises
identity that has access to the data
On-premises Business Connectivity Services forwards the request to the OData service endpoint
The OData endpoint authenticates the request viaInternet Information Services and returns the
data
❶
SharePoint
Online
tenancy
External list Business Connectivity Services
Secure
store and
Access
Control
Service
Perimeter
network
Reverse proxy
Internal
network
On-premises SharePoint farm External data source
Authentication flow
Data flow
❷ ❸
❼
❻
❹
❾
❺
❽
Hybrid BCS Auth and Data Flow

34. Steps to configure Hybrid BCS
Create a connection
to your on premises
service
Make your ECT file
“tenant ready”
Create an external
content type based
on your OData
source
❶ ❷ ❸
❹ ❻
Use an existing or
create a new OData
data source
Create an external list
and validate the
hybrid Business
Connectivity Services
solution
❺ Upload your model
(.ECT) to o365

35. Read
Cross-Reference
Index
Write
On-prem Online Hybrid Structured Unstructured
Developer
Site Admin
End User
Create Deploy Compose
Where are we now?

36. 36
BCS Trajectory (my opinion)
Business Data Catalog
Read Only
No Tooling
BCS
SPD Tooling
VS Tooling
External List
Foundations
Alerts & Events
OData
Excel
Apps
Online Limitations
Odata only
No Search
No User Profiles
Discontinue SPD

37. 37

38. 38

39. 39

40. 




Integrating External Data in SP





42. PowerBI & Data Management Gateway
42
• With the on-premises gateways, you
can keep your data fresh by
connecting to your on-premises data
sources without the need to move the
data.
• Query large datasets and benefit from
your existing investments.
• The gateways provide the flexibility you
need to meet individual needs, and the
needs of your organization.

43. Excel Access via PowerBI
Keep reports up to date
with scheduled data
refresh
PowerBI for Office
365
Data Management Gateway
Installed on-premisesOn-premise database
Workbook
On cloud
On premise
DATA REFRESH

44. Data Management Gateway
Enabling Corporate
OData Feeds
Enabling Excel Workbook Data
Refresh using
SharePoint Online
Enabling Discovery
in Power Query
capabilities
Power BI Admin CenterData Management Gateway

45. Data Management Gateway - Conceptual
Power BI Admin Center
Allows IT to configure, manage
and monitor access to corporate
data sources.
Data Management Gateway
Connects to corporate data sources
and sends data to Microsoft cloud
services through a secure channel
(Service Bus).
Corporate Data Sources
The Gateway can connect to
a variety of data sources.
Secure Credential Store
All credentials used by the
gateway are stored on-premises.
Encrypted backup of credentials
can also be stored in the cloud
to enable business continuity
scenarios.

46. What can I build with PowerApps?
PowerApps is flexible, but it has specific strengths
 Generate apps based on data (Excel, SQL etc.)
 Form-based apps for capturing and updating data
 Line of Business apps with modern capabilities (microphone, camera, GPS
coordinates)
 Apps for specific needs, signups, events – for power users and role-based needs
PowerApps-based apps can contain:
Multiple data sources Multiple screens Ink & pen support Custom APIs

47. Flow

48. Extending beyond PowerApps & Flow basic features
Custom APIs extend PowerApps &
Flow OOB functionality
Azure Functions provide effortless
building blocks
Logic Apps provide true
integration logic
Serverless computing
Trigger via HTTP call, polling or
webhooks
Can run C# scripts and .BAT files
Ad-hoc modifications, real-time
logging
Azure API Apps as a platform
Swagger-based metadata finding
Can do *anything* 
Best suited for integrating
external datasources
API Management Portal
Enterprise integration features
Full integration capabilities
Not dependent on single user
Fully Azure-based
Hybrid integration solution

49. Azure Logic Apps

50. • Box
• Chatter
• Dropbox
• Azure HD Insight
• OneDrive
• SharePoint Server
• SharePoint Online
• SQL Server
• Office 365
• Oracle DB
• QuickBooks
• Salesforce
• SugarCRM
• SAP
• Azure Service Bus
• Azure Storage Blob
• Twilio
• Twitter
• IBM DB2
• Informix
• Websphere MQ
• Azure Web Jobs
• Yammer
• Facebook
• HTTP, HTTPS
• File
• FTP
• SFTP
• POP3
• SMTP
• Flat File Encoder
• Validate
• Extract (XPath)
• Transform
• X12
• EDIFACT
• AS2
• TPM
• Rules
Connectors
Protocols BizTalk Services
Built-in API Apps
And more on the way…

52. On-Premises Data Gateway
•
•
•
• Logic Apps
• Power Apps
• Microsoft Flow
• PowerBI
• Azure Analysis Services
On-prem resources:
• BizTalk Server
• DB2
• File System
• Informix
• MQ
• Oracle Database
• SAP Application Server
• SAP Message Server
• SQL Server

53. On-Premises Data Gateway
Supports…
 SharePoint 2016
 SQL Server 2016
Machine should always be accessible
Can’t install on a domain controller

54. Read
Cross-Reference
Index
Write
On-prem Online Hybrid Structured Unstructured
Developer
Site Admin
End User
Create Deploy Compose
Flow, PowerApps, Azure Apps, OPDG

55. Resources – PowerBI, PowerApps, ....

 https://powerbi.microsoft.com/
 https://powerbi.microsoft.com/developers
 https://support.powerbi.com/

 https://msdn.microsoft.com/en-US/library/mt186542.aspx

 https://msdn.microsoft.com/en-us/library/mt147898.aspx

 https://powerapps.microsoft.com/en-us/tutorials/formula-reference/

 https://blogs.msdn.microsoft.com/carlosag/2016/05/09/using-azure-functions-
in-powerapps/

 https://powerapps.microsoft.com/en-us/blog/using-azure-functions-in-
powerapps/

56. Office 365 Connectors
3rd party app
Office 365
Group
message
“Webhook“

57. Connector Card

59. Office 365 Connectors Architecture
HTTP POST
(JSON Card)
Office 365 Group1
Provider Hosted
App
3
2

60. Building an O365 Connector


POST
https://outlook.office365.com/webhook/f1a6033d...
HTTP/1.1
Content-Type: application/json
Host: outlook.office365.com
Content-Length: 43
{"Text":"Hello World! Buzz...Buzz...Buzz!"}

61. 




O365 Connectors

62. O365 Connectors











63. Read
Cross-Reference
Index
Write
On-prem Online Hybrid Structured Unstructured
Secure
Developer
Site Admin
End User
Create Deploy Compose
O365 Connectors

64.  http://aka.ms/o365-connectors
 https://dev.outlook.com/Connectors/GetStarted
 http://aka.ms/connectorplayground
 https://dev.outlook.com/connectors/reference
 https://outlook.office.com/connectors/publish
 http://winampplugins.co.uk/curl/
 http://officespdev.uservoice.com/connectors
 https://dev.outlook.com/Connectors
Resources – O365 Connectors

65. 65

66. 




Integrating External Data in SP





67. External Content Trajectory (my opinion)
BCS
Custom
Code

68. Business Connectivity Services Feature Alternative(s)
External List Flow?, Client-Side Code (SPFx)
External Data column Client-Side Code (SPFx)
Connectors Connectors, 3rd
party add-ins
Secure Store Service Secure Credential Store
Business Data Web Parts (custom client-side web parts)
Rich Client Integration PowerBI (Excel), 3rd
party add-ins
BCS connectors for Search (protocol handlers w Hybrid Search)
InfoPath forms connected to BCS PowerApps
Notifications from External Systems Flow, O365 Connectors
Hybrid BCS On-Prem Data Gateway
SharePoint Designer Visual Studio, Flow, 3rd
party tools

69. External Content
(on-premises and/or
in the cloud)
SharePoint Server
(On-premises or Hosted)
Office 365
SharePoint Online Content
Onedrive for Business Content
Connectors
SharePoint Content
Cloud Hybrid Search - Adding External Content
Also drives:
• Office Graph (Delve,..)
• Compliance (DLP, …)

70. Connectors to Many Enterprise Systems
• Aderant
• Amazon S3
• Alfresco
• Box
• Confluence
• CuadraSTAR
• Elite / 3E
• EMC Documentum
• EMC eRoom
• Google Drive
• HP Consolidated Archive
• (EAS, aka Zantaz)
• HPE Records Manager/HP TRIM
• IBM Connections
• IBM Content Manager
• IBM DB2
• IBM FileNet P8
• IBM Lotus Notes
• IBM WebSphere
• iManage Work
• Jive
• LegalKEY
• LexisNexis Interaction
• Lotus Notes Databases
• Microsoft Dynamics CRM
• Microsoft Exchange
• Microsoft Exchange Public Folders
• Microsoft SQL Server
• MySQL
• NetDocuments
• Neudesic The Firm Directory
• Objective
• OpenText LiveLink/RM
• OpenText eDOCS DM
• Oracle Database
• Oracle WebCenter
• Oracle WebCenter Content (UCM/Stellent)
• PLC/Practical Law
• ProLaw
• Salesforce.com
• SAP ERP
• ServiceNow
• SharePoint Online
• SharePoint 2016
• SharePoint 2013
• SharePoint 2010
• SharePoint 2007
• Sitecore
• Any SQL-based CRM system
• Veeva Vault
• Veritas Enterprise Vault
(Symantec eVault)
• West km
• Xerox DocuShare
• Yammer
Plus a proven architecture and process for creating new connectors to complex systems

71. 
–
–

–
–
What to do?

73. Contact:
Jeff.Fried@BAinsight.com
www.BAinsight.com
Questions