Jen Perlee

1. #TargetXSummit
Automated Data Loading /w
Little to No Duplicates!
Jen Perlee

2. #TargetXSummit
Agenda
Things I’m going to talk about
● Introduction
● Our Problem
● Our Thought Process
● Our Solution
● Demo
● What We Still Need to Figure Out
● Q&A

3. #TargetXSummit
Hello!!
● I am Jen Perlee
● Certified Advanced Salesforce Administrator (just passed my exam last month)
● Champlain College (more about that on the next slide)
● 24 years in IT @ Champlain
● 3 years as a Salesforce Admin

4. A Little About Champlain College
● Founded in 1878, Champlain College® is a small, not-for-profit, private college
overlooking Lake Champlain and Burlington, Vermont, with additional
campuses in Montreal, Canada, and Dublin, Ireland.
● 29 Traditional on-campus Undergraduate Degrees
● 66 100% Online Certificate Programs, Associates, Bachelors and Masters
Degrees.

5. Our Salesforce Target -X Story
● Started out with our own custom Salesforce Implementation
● Quickly out grew our resources
● Have been live on Target-X since April 2017
● Started w/ our online division
● Traditional Undergrad went live w/ Inquiry in January
● Go Live w/ Applicants 8/1 (using Target-X, CommonApp & Fire Engine Red)
● Online Retention will soft-lauch on Thursday

6. #TargetXSummit
Our Problem
● Receive List from ~10 different sources (SAT, ACT, Raiseme, Royall etc)
● This doesn’t even include web inquires, phone call, e-mails
● We knew we were going to import duplicates lots of duplicates
● Duplicates are more than annoying
● They cost money (data storage, e-mail usage, manpower)
● Lead to inaccurate reporting
● Poor customer experience
● They are a time suck

7. #TargetXSummit
Thought Process
● Dupe clean-up: Allow duplicates to be created and clean-up after
● VS.
● Dupe Prevention: Append data to existing records so duplicate is never created
● We wanted to avoid as many duplicates BEFORE they came into Target-X
● Nothing will ever 100% stop duplicates but we wanted to strive for as close to 100%
● Brainstormed ideas
● Looked into buying tools
● Needed to be mostly automated
● Needed to include solutions for all data entry points

8. #TargetXSummit
Solution
● Tools we ended up using:
● Demand tools Find/Report IDs
● Demand tools Job Builder
● Basic Powershell and Batch scripts
● Informatica
● Special shout-out to the Target-X Forums and Especially Fiona Kelly from St. Martin University
for getting me started using tools we already have.
● Why?
● ‘Free’
● Low Overhead
● Relatively easy to implement

9. #TargetXSummit
High-Level Steps
● Get data to our Informatica server (secure FTP and Encrypt the files)
● Use JobBuilder to automate…..
● Run a script that checks the header against a known good header and send an e-mail if they
do not match
● Use D/T Find/Report IDs to find contacts base on a tiered approach
● Use the result of that file to import data with the found IDs using Informatica Uniquie IDs
● Run a Script that e-mails if there are multiple matches found
● Fix any multiple matches and re-import

10. #TargetXSummit
Demo
● DemandTools FindIDs
● JobBuilder
● Scripts

16. #TargetXSummit
Resources
● DemandTools Help Console –
● http://www.helpconsole.com/demandtools
● DemandTools Find/Report ID’s Help -
http://www.helpconsole.com/DemandTools/default.aspx#pageid=find_report_ids
● JobBuilder Help Console
● http://www.helpconsole.com/DemandToolsJobBuilder/
● JobBuilder Scenario Syntax
● http://www.helpconsole.com/DemandToolsJobBuilder/#pageid=demandtools_job___sce
nario_syntax

17. #TargetXSummit
Still to figure out/improve
● Duplicates in the same file
● Better way to automate/fix current duplicates
● Validty – What does this mean

18. Any Questions?

19. #TargetXSummit
Thank You
Jen Perlee
Perlee@Champlain.Edu

20. #Powershell Script to compare header rows
if(Compare-Object -ReferenceObject $(Get-Content "PATH TO CSV TO CHECK" -totalCount 1) -
DifferenceObject $(Get-Content "PATH TO KNOWN GOOD HEADER ROW"))
{$From = "FROM ADDRESS"
$To = "SEND TO EMAIL ADDRESS"
$Subject = "EMAIL SUBJECT"
$Body = "BOSY OF EMAIL"
$SMTPServer = "SMTP SERVER ADDRESS"
Send-MailMessage -From $From -to $To -Subject $Subject -Body $Body -SmtpServer $SMTPServer
$filename = "Cappex_"
$datetime = date -format MM-dd-yy-hh-mm-ss
Rename-Item -Path "ORIGINAL PATH and FILENAME SHOULD BE SAME AS ABOVE" -NewName ($filename
+ $datetime + ".csv") }

21. # Powershell Script to Look for multiple matches in csv's saved to a specific directory #
# BEG CONFIG #
$root_folder = "PATH OF THE FILES YOU ARE CHECKING"
$email = "EMAIL TO SEND TO - SEPERATE BY A COMMA"
#Select field of CSV to evaluate
$search_field = "CRMfusionField_UniqueMatchCount"
#Set search threshold, logic used is ge (greater than or equal to)
$search_threshold = 2
# END CONFIG #
#Logging function
function logit ($code, $message)
{
#Change these values to match what you want, this function assumes it will log
#to the directory the script originated from with a file called logit.txt
#If you put this function in "test.ps1" for instance and called it with these examples
#you would get indicated results.
#Example call: logit 1 "hello"
#Example Result: Date/Time - test.ps1 - ERROR - hello
#Example call: logit 0 "test"
#Example Result: Date/Time - test.ps1 - INFO - test
#Example call: logit 2 "warning test"
#Example Result: Date/Time - test.ps1 - WARN - warning test
#Code Mappings
#By default, 0 is info, 1 is error and 2 is warn when using logit. Add more to the array if needed.
#Array starts at 0, so don't change first 3, if you add a fourth element it will be
#code number 3, 5th would be 4, etc.
$codeArray = @("INFO","ERROR","WARN","ERROR","UPDATE","DEBUG")
$messagetype = $codeArray[$code]

22. #Determine what this script is called and the path.
$Invocation = (Get-Variable MyInvocation -Scope 1).Value
#$scriptpath = Split-Path $Invocation.MyCommand.Definition
#Two options for scriptname, if you want full path and extension use the second one.
#If you just want the script name without path or extension when logging use the first.
#For best operation make sure one of these lines is commented out.
$scriptname = $MyInvocation.Scriptname.Split('') | Select-Object -Last 1
#$scriptname = $MyInvocation.ScriptName
#Determine date and time
$date = Get-Date
#Create Logfile if it doesn't exist and open for appending always.
#If you want the logfile somwhere else simply replace the path/filename below.
<#
If(Test-Path $logfile)
{
#Logfile exists, continue.
}
Else
{
#Logfile doesn't exist, create it and continue.
new-item $logfile -type file
}
#>
$logstring = "$date - $scriptname - $messagetype - $message"
Write-Host $null

23. if( $code -eq "1" )
{
Write-Host "$logstring" -ForeGroundColor Red
}
if( $code -eq "2" )
{
Write-Host "$logstring" -ForegroundColor Yellow
}
if( $code -eq "0" )
{
Write-Host "$logstring"
}
if( $code -eq "3" )
{
#Adding this for errors that don't require intervention and should just be revewied in the logs.
Write-Host "$logstring" -ForeGroundColor Red
}
if( $code -eq "4" )
{
#Adding this so that AD changes show up green visually as script is run
Write-Host "$logstring" -ForeGroundColor Green
}
if ($code -eq "5" )
{
Write-Host "$logstring" -ForegroundColor Cyan
if ($debugPausing -eq "1")
{
pause
}

24. }
Write-Host $null
Add-Content $logfile "$logstring"
if ( $code -eq "1" )
{
pause
}
}
function sendEmail ($ef_to, $ef_from, $ef_subject, $ef_body, $ef_attachment)
{
#Function to send email
#Configure custom header values if you wish
#CONFIG
$custom_headers = "yes"
$smtpServer = "SMTP SERVER NAME"
$msg = new-object Net.Mail.MailMessage
$smtp = new-object Net.Mail.SmtpClient($smtpServer)
$msg.From = "$ef_from"
$msg.ReplyTo = "$ef_from"
$msg.To.Add($ef_to)
$msg.subject = "$ef_subject"
$msg.body = $ef_body
$msg.IsBodyHtml = $true
if ($custom_headers = "yes")
{
#Example:
#$msg.Headers.Add("CCAccntCreate", 1)

25. #You can add more than one in sequence
$msg.Headers.Add("CCAccntCreate", 1)
}
$msg.IsBodyHtml = $true
$msg.Body = "$ef_body"
#Add the attachment if it is there
if($ef_attachment -ne $null)
{
$msg.Attachments.Add($ef_attachment)
}
$smtp.Send($msg)
}
function successCheck()
{
#Reports success of the operation before the one that calls it.
if($?)
{
logit 0 "Operation was Successful!"
}
else
{
logit 1 "Operation Failed!"
$error_text = $error[0]
logit 2 "Error Received by SuccessCheck: $error_text"
}
}
#Set up logging file

26. $logs_folder = "$root_folderlogs"
if (!(Test-Path $logs_folder)) {New-Item -ItemType Directory $logs_folder}
$logstamp = date -format MM-dd-yy-hh-mm-ss
$logfile = "$logs_folderrecord_check_log-$logstamp.txt"
logit 0 "Script running..."
$csv_files = Get-ChildItem -Path "$root_folder" -Filter "*.csv"
foreach ($file in $csv_files)
{
$file_fullname = $file.FullName
$file_basename = $file.BaseName
$file_processed_name = "$root_folderprocessed$file_basename-processed-$logstamp.csv"
logit 0 "Processing file: $file_fullname"
$csv_data = Import-CSV $file_fullname
$count = 0
$new_csv_file = "$root_folderoutput$file_basename-filtered-$logstamp.csv"
foreach ($line in $csv_data)
{
$search_value = $line.$search_field
if ($search_value -ge $search_threshold)
{
#Write the line to a new CSV file
logit 0 "Found matching line, adding to $new_csv_file"
$count++
logit 0 "Match count for file: $file_fullname is now $count"
$line | Export-CSV $new_csv_file -Append -NoTypeInformation
}

27. }
if (Test-Path $new_csv_file)
{
#Send email to notify that file was made
logit 0 "Sending email notification..."
sendEmail $email "FROM ADDRESS" "Informatica Duplicate Records Found" "Record check found
<b>$count</b> lines where <b>$search_field</b> was greater than or equal to $search_threshold in
original file: <b>$file_fullname</b><br><br>Matching lines saved to new file: <b>$new_csv_file</b>"
}
logit 0 "Finished with file: $file_fullname"
#Move CSV file to processed directory when done with it.
logit 0 "Moving original file from $file_fullname to $file_processed_name"
Move-Item $file_fullname $file_processed_name
}