IT6701-Information Management Unit 5

IT6701 _ Information
Management
IV Year / VII Semester

UNIT V INFORMATION
LIFECYCLE MANAGEMENT
Data retention policies; Confidential and
Sensitive data handling, lifecycle management
costs. Archive data using Hadoop; Testing and
delivering big data applications for performance
and functionality; Challenges with data
administration

Introduction
 Data – for analyzing and for drawing
important conclusions and observations about
the business.
 there are constant updates and older data
might become obsolete.
 Decide which data is no longer required.

Data Retention Policies
A data retention policy, or records retention
policy, is an organization's established
protocol for retaining information for
operational or regulatory compliance needs.

 deal with the complex issues of protecting
business information for a pre-determined
length of time on a pre-determined storage
system.
 These policies define different retention
periods, depending on the type of data.

 describe procedures for archiving the
information, guidelines for destroying the
information when the time limit exceeds and
special mechanisms for handling the
information.

 Purpose:
 To maintain important records and documents for
future use or reference
 To dispose of records or documents that are no longer
needed
 To organize records so that they can be searched and
accessed easily at a later date.

 Purpose:
 Email messaging had a large impact on those who
develop and enforce data retention policies.
 This stored business information can be expensive
for the business organizations.

 Requirements:
 Legal or legitimate requirements: information related
to legal information
 Business or commercial requirements: for operational
perspective
 Personal or private requirements: for personal
perspective

 Scope: what kind of data are covered under data
retention policies
 Legal record: contracts, trademark, power of attorney,
press release.
 Final records: records of completed activities.
 Permanent records: financial registers, patents,
proposals.

retention policies
 Accounting and corporate tax records: investments, audits,
purchase, sales records
 workplace records: day to day activities of employees,
agreement, minutes of meetings
 Employment, employee and payroll records: job posting,
job advertisements, recruitment procedures, performance
reviews.

retention policies
 Bank records: bank transactions, deposits, cheque details,
cheque bouncing
 Historic records: records that are no longer required by the
organization.
Temporary records: documents that are not complete or
finalized

Data Retention Policies - Content
 to focus on the reason behind data retention.
 identification of criteria on which data needs to
be retained.
 Usually the decision is based on the creation of
date, but it is also important to examine other
criteria such as last accessed time, type of data,
time till which the data is valid, data value

Data Retention Policies - Content
 The policy document should include details of
the data that need to be archived or retained.
 the division of data would help in deciding the
duration of retention and destruction
procedures.

Managing Data Retention Policies
 to identify and managing authority
 combined effort, involving storage administrators and
application owners, along with executive support.
 The policy document should be validated by the company’s
legal counsel and should also be fully supported by the
management to be presented as a company policy and not
restricting it as an IT best practice document.

Data Retention in Telecommunication
Industry
 the storage of call detail records of telephony and
internet traffic and transaction data by
government and commercial organizations.
 Retention requirements for service providers are
found in the ISP and UASL licenses, which are
documented in the Indian Telegraph Act, 1885.

Industry
 Internet Service Provider (ISP) License:
 According to ISP license, each ISP must
maintain:
 Customers and services: A log of all
customers registered and all the services used
by them.

Industry
 Outward logins/ connections or telnet:
 Every outward login or telnet through an ISP’s
computer must be logged.
 Data Packets:
 Copies of all packets originating from the
Customer/User Premises Equipment of the ISP must
be available.

Industry
 Subscribers:
 available on the ISP website with authenticated access and must
be available to authorized Intelligence Agencies at any time.
 Internet-leased line customers:
 the complete list of customers and sub customers.
 Details like name, address of installation, IP address allotted,
bandwidth provided and contact person with phone
number/email.

Industry
 Network records and purpose
 A record of complete network topology of the set-
up each of the internet –leased line customer
premises along with details of connectivity must
be made available at the site of the service
provider.

Industry
 Commercial records:
 communications exchanged on the network must be
maintained for a year.
 Site:
 maintain the geographic location of all its subscriber
and should be able to provide it at a given point of
time.

Industry
 Remote activities:
 the network should have a complete audit trail and
must be retained for a period of 6 months.
 This information must be provided on request to
the licenser or any other agency authorized by the
licensor.

Industry
 Unified Access Service License (UASL)
 introduced by DoT through which an access
service provider can offer a fixed and/or mobile
services using any technology under the same
license.
 to retain pertaining to customer information or
transactions for security purposes.

Industry
 Mobile numbers:
 Called / Calling party mobile numbers when
required.
 Capture / Interception records:
 Time, date and duration of interception when
required.

Industry
 Site / location:
 Location of target subscribers.
 All call records:
 All call data records handled by the system when
required.
 Failed call records
 Roaming subscriber records

Industry
 Commercial records:
 the communication exchanged on the network must be
retained for 1 year.
 Outgoing call records:
 a record of checks made on outgoing calls completed
by customers making a large number of outgoing calls
day and night to various customers.

Industry
 Calling line identification
 A list of subscribers including address and details using line
identification should be kept in a password-protected website
accessible to authorized government agencies.
 Location
 The service provider must be able to provide the geographical location
of any subscriber at any point of time.
 Remote access activities:
 The complete audit trails of the remote access activities pertaining to
the network operated in India for period of 6 months.

Industry
 Sample retention Records:
 Accounting and finance:
Record type Retention Period
Accounts payable/receivable ledgers
and schedules
8 Years
Financial statements and Annual audit
reports
Permanent
Annual audit records 8 years after the completion of audit
Annual plans and budgets 3 Years

Industry
 Electronic documents:
 Email:
 All the emails are not retained
 All emails are deleted after a period of 12 months.
 All emails will be archived for 6 months after it has been deleted
by a staff, after which the mails will be permanently deleted.
 Staff will not send confidential / proprietary data to outsiders.

Industry
 Electronic documents:
 Documents:
 The maximum period of retention of documents is 6
years, depending on the content of the file.
 Web pages:
 Browsers should be scheduled to delete internet
cookies once per month.

Industry
Insurance records:
Insurance policies and certificates Permanent
Claims files Permanent
Group insurance plans ( current
employee)
Till the plan is active or employee is
terminated
Group insurance plans (Retires)
Permanent or until 5 years after the
death of last eligible andidate.

Industry
Legal files and papers:
Legal memoranda and ideas 6 years after close of matter
Court Orders Permanent

Industry
Payroll documents:
Payroll registers 6 years
Time sheets 2 years
Payroll deductions Termination + 6 years

Industry
Personnel records:
Employee service book Permanent
Employee medical records Termination/Retirement + 6 years
Bio-data of applicants not selected 1 year

Industry
Property records:
Property deeds, licenses Permanent
Purchase/ Sale/ Lease agreement Permanent
Property insurance policy Permanent

Industry
Tax records:
Tax returns – income, property Permanent
Sales/ use tax records 7 years
Tax exemption documents Permanent

Industry
 Laws related to Data retention policy in India:
 License Agreement for Provision of Internet
Services.
 maintain all commercial records with regard to the
communications exchanged on the network.
 ISPs are responsible for maintaining history or a log
of all users connected through ISP and the services
they are using.

Industry
Information Technology Act (ITA) - 2008.
 Section 67C (“Preservation and Retention of
Information by Intermediaries”)
 online service providers and at least some access point
providers – retain a specified amount of information
for a specified period of time.

Industry
Information Technology Act (ITA) - 2008.
 Section 79(2), under which intermediaries are
protected from liability for third party content
provided that they follow due carefulness while
discharging notice-and-takedown requirements of
the law

Industry
The Indian Department of Information Technology,
Ministry of Communications and Information
Technology - 2011.
 store the traffic data and “history of websites
accessed” for each user for 1 year.
 Users must be identified by their government issued
Id number and photograph.

Confidential and Sensitive Data
Handling
 Personal Data: information about an individual, and
through which an individual can easily identified ,
either directly or indirectly.
 Confidential Data: the personal data that is private and
should be disclosed to others
 Sensitive Data: secured from unauthorized access to
protect the privacy or security of an individual or
organization.

Handling
 Types of Sensitive information:
 Personal information:
 Sensitive personally identifiable information is data
that can be traced back to an individual, thus revealing
one’s identity.
 Information: biometric data, medical information and
history, bank and credit card information.

Handling
 Business Information:
 poses a risk to the company in question if
discovered by a competitor or the general public.
 Trade secrets, contract details, financial data and
supplier and customer identification.

Handling
 Classified Information:
 pertains to a government body and it restricted
according to the level of sensitivity.
 to protect security.

Handling
 Handling of Sensitive data:
 Access policy
 Access Decisions: Availability of data,
acceptability of the access data and non-sensitive
information derived from sensitive data.

Handling
Types of Disclosures:
 Displaying exact data
 Displaying bounds – between high and low value.
Range of salary
 Displaying negative results
 Displaying probable values

Handling
 Handling Data:
 Create a security risk-aware culture – Risk Management
 Define data types – classify it as confidential or sensitive.
 Clarify responsibilities and accountability for protection of
confidential or sensitive data.
 Limit the access confidential or sensitive data
 Provide training to properly use the resources and follow the
guidelines and rules.
 Authenticate compliance regularly with policies and procedures.

Handling
 Law provision in India defining Sensitive
Data and its handling:
 Information Technology Act:
 reasonable security practices and procedures while
handling sensitive personal data or information.

Handling
 Law provision in India defining Sensitive
Data and its handling:
 Information Technology Act:
 Criminal punishment for a person
 discloses sensitive personal information
 violation of the relevant contract
 with an intention of, or knowing that the disclosure
would cause wrongful loss or gain.

Handling
 Information Technology Act – Feature
 Sensitive personal information: Sensitive Personal
Data (SPD) includes passwords, financial and
credit card details, physical, physiological and
mental health condition, medical records and
history.
 SPD deals only with information of individuals
and not information of businesses.

Handling
 Privacy Policy: describe what information is
collected, what is the purpose of using the
information, to whom or how the information
might be disclosed and the sound security practices
followed to safeguard the information.

Handling
 Privacy Policy:
describe what information is collected,
what is the purpose of using the information,
to whom or how the information might be disclosed
the sound security practices followed to safeguard the
information.

Handling
 Consent for collection:
 approval has to be provided by letter, fax or email.
 prior to collecting the information, provide an
option to the information provider to not provide
such information.

Handling
 Notification:
 the business ensure that the information provider is
aware of the information being collected
 purpose of using the information
 recipients of the information and name and address of
the agency collecting the information.

Handling
 Use and retention:
 restricted to the purpose for which it was
collected.
 The business should not maintain the SPD for
longer than it is specified.

Handling
Right of access, correction and withdrawal
 permit the information provider the right to review
the information, and should ensure that any
information found to be inaccurate or deficient be
corrected.

Handling
 Transactional transfer
 transferred if it is necessary for the performance
of a lawful contract between the body corporate
and information provider or where the information
provider has provided his/her consent to such
transfer

Handling
 Security Procedures
 procedure has to be audited on a regular basis by
an independent auditor

Lifecycle Management Costs
 Data lifecycle management:
 the process of handling the flow of business
information throughout its lifespan, from
requirements through maintenance.
 automating the process involved in organizing
data into separate tiers according to specified
policies, and automating data migration from one
tier to another.

 Data lifecycle management – Stages:
 Data creation powers the enterprise:
 When an employee creates and saves a file, that
information becomes part of the organization's
daily operations
 store this active data locally and on a network
server

 Backups guard against data loss:
 enterprise can move it from primary storage into less
costly off-site tape vaults or to the cloud.
 A well-rounded data backup and recovery strategy
combines off-site tape storage with cloud backup and
data restoration capabilities

 Archiving helps contain storage costs:
 to retain older inactive data in case of a legal,
regulatory or audit event.
to hold on to data for as long as seven years
 Off-site tape archives offer high security, quick access
and lower storage costs for such long-term data storage
demands.

 Ensuring secure data destruction:
 The final stage of the data lifecycles requires secure
destruction, which is typically governed by a schedule
that defines when you must destroy unneeded data.
 Once data reaches its expiration date, secure media
destruction can ensure its environmentally friendly
disposal.

 Put secure IT asset disposition to work:
 before discarding the storage media it needs to be
completely destroyed.

 Efficient Data lifecycle management:
 the storage needs to be scalable to accommodate
 Analytics applications in some cases require us to
access archived and unstructured data.
 the storage can be optimized for maintenance and
licensing costs by migrating rarely used data into
framework like Hadoop.

 Objectives of Data lifecycle management:
 Data trustworthiness
 Both structured and unstructured data must be
managed effectively.
 Data privacy and security must be protected at all
times.

Archive Data using Hadoop
 Hadoop – to store any type of data
 ability to query Hadoop data with SQL makes
Hadoop the prime destination for archival data.
 to perform archiving is Sqoop, which can
move the data to be archived from the data
warehouse into Hadoop.

 Features:
 Schema preservation:
 to ensure that data values will be archived without loss of
precision.
 Changes to the source schema, for example adding new columns
or changing data types, should also be captured by the archive.
 allows the archive to grow organically over a long period of
time while maintaining a continuous historical record of the
changes to the schema and the data in the source EDW.

Features:
 Control and Security:
 Archived data generally inherits the same governance
requirements as the EDW.
 The archive must provide access to data on a ‘need to
know’ basis; it must guarantee that sensitive data is
encrypted or masked, and that access is audited.
 An archive must also integrate with the same
enterprise security infrastructure as the EDW.

 Features:
 SQL support
 Support for SQL access to the archived data is a must.
 Applications would require us to make use of the archived
data to generate reports or to perform analysis.
 to execute run-time interactive queries along with batch
processing.
 SLA can be relaxed or ignored

Testing and Delivering Big Data
Applications for Performance and
Functionality
 A huge set of complicated structured and
unstructured data is called as Big Data.
 testing of Big Data, a lot of processes and
techniques are involved.
 Big Data testing is a proof of the perfect
data dealing, instead of testing the tool.
 In testing of data, Performance and functional
testing are the keys.

Functionality
Signs That Show We Should Go For Testing
Are:
 Presentation Testing:
Big Data applications work together with existing
statistics for genuine occasion analytics
 makes the procedure keep going.

Functionality
Are:
 Problems With Expansion Capacity:
starts with lesser sets of statistics and ends up with
an overweight quantity of statistics.
 a number of data increases, the performance of
analytics may reduce.

Functionality
Signs That Show We Should Go For Testing Are:
Towering Quantity Of Downtime:
due to a large number of problems, the data faces
certain issues resulting in a reduction of downtime.
So if a continuous amount of downtimes occur, then
users should be a concern and be sure that it is time for
testing the Big Data Analytics

Functionality
Poor Improvement:
Failure in handling data efficiently for longer time
span would result in improper development.
 Hence for running the business appropriately, proper
testing of data is required, because the delivery of the
proper result to clients.

Functionality
Are:
No Proper Control:
require proper control of the information the
business work with.
 And this proper data can be obtained only by
frequently checking the data.

Functionality
 Signs That Show We Should Go For Testing Are:
 Poor Safety Measures:
big data stores the organization’s complete data from
credential sets to all the confidential reports so safety and
protection in Big data is a must and the management have
to make sure that the data stored in HDFS of big data is
secured to the fullest.
 to steal confidential data from the company’s storage.

Functionality
 Signs That Show We Should Go For Testing Are:
 Problems With The Proper Running Of The
Applications:
Before applying data to be used in different applications
they should undergo a testing procedure to find out if they
are fit for the analysis.
 in order to assure proper, running the applications,
performing proper testing should be a must.

Functionality
Are:
Proper Output:
In order to get the best output in any project proper
input is necessary and correction and testing of
input must be made sure to determine the best
output ever

Functionality
Unpredictable Performances:
 When the right data is used in the right way, then the
potential of any organization finds no limit.
 Only through correct testing on time will help to
decide inconsistency and removes insecurity.

Functionality
insufficient Value:
a lot of other factors need to be taken cared of like the
strength, precision, traditional values, replication,
stability, etc.
 gaining the proper data, all factors need to be checked
which led to the requirement of performing testing on
Big Data.

Functionality
 A High-Level Overview Of Phases In Testing Big Data
Applications:
 The Testing Procedure Is Filled With
 Data Phase Proofing:
 The data collected from different places need to be proved
to be correct.
The supply data and the input data needs to be similar
Make sure true and valid data is put into the HDFS.

Functionality
 A High-Level Overview Of Phases In Testing
Big Data Applications:
Proofing of MapReduce:
 data accumulation regulations are applied on data.
 proof the processed output data.

Functionality
 A High-Level Overview Of Phases In Testing Big Data
Applications:
 Proofing of the Output:
 the transformation rules are implemented accurately.
 Fill the information in the target system.
 make sure that the data in the output and in the HDFS has
no fraud.

Functionality
 Testing of the Architecture
 Hadoop is the data storage of an immense set of data with
high standard arrangement and security.
 the testing should always occur in the Hadoop atmosphere
only.
 Testing of the concert includes the clear output completion,
use of proper storage, throughput, and system commodities
 Data processing is flawless and it needs to be proved.

Functionality
 ActionFlow Testing
 Information Intake And Right Through:
 the speed of the data from different sources is
determined.
 Categorizing messages from different data frame
in different time is classified

Functionality
 Dealing With Data:
 Here determination of how fast the data is
executed is done.
 when the datasets are busy, testing of the data
processing is done in separated forum.

Functionality
 Check The Working Of All The Ingredients:
 test of each and every commodity is a must.
 The speed of message indexes, utilization of those
messages, Phases of the MapReduce procedure,
support search, all comes under this phase.

Functionality
 Performance Testing Approach:
involves testing of huge volumes of planned
and shapeless data, and it requires a specific
testing approach to test such massive data.
Hadoop is involved with storage and
maintenance of a large set of data including
both structured as well as unstructured data .

Functionality

Functionality
the set up of the application prior to the testing
procedure begins.
 Find out the required workloads and make the design
accordingly.
 Make ready each and every client separately.
 Perform the testing procedure and also check the
output carefully.
 Do the best possible organization

Functionality
 Factors For Concert Testing: Various parameters to be
verified for performance testing are
How the information will be stored
Till what extend the commit logs can enlarge
Finding out the concurrency of the read and write
procedures
Find all the standards of the start, and stop timeouts.

Functionality
 Factors For Concert Testing: Various parameters to be
verified for performance testing are
Arrange the key and row cache properly
Do consider the ingredients of the Java Virtual Machine
also
Filter and sort the working of the processing part, the
MapReduce.
Check the messaging rate and its sizes too.

Functionality
 Test Atmosphere Requirements
As always Hadoop structure should be
more spacious since it has to process a large set of
data.
The cluster should contain a large set of nodes to
handle the stored information.
The CPU should be utilized properly.

Functionality
 Challenges In Big Data Testing
 Mechanization:
High technical expert is involved with
mechanical testing.
They do not solve those unexpected problems.

Functionality
 Virtualization:
Latency in this system produces time problems in real
time testing.
Image management is also done here.

Functionality
 Large Dataset:
Proofing of large amount of data and increase of its
speed.
Need to increase the tests.
 Testing has to be done in several fields.

Functionality
 Performance Testing Challenges:
 Varieties In Technologies:
The different ingredients of Hadoop belong to
different technology
each one of them needs separate kinds of testing

Functionality
 Unavailability Of Precise Equipment:
A lot number of testing components are required
for the complete testing procedure
 Test Scripting:
 High-quality scripting is thus important and very
essential for the state of affairs.

Functionality
 Test Environment:
The perfect test atmosphere is must, and in most of the
cases not possible to obtain.
 Controlling Resolutions:
 For controlling the complete atmosphere large number
of resolutions is required which is not always present.

Challenges with Data Administration
 responsible for designing and maintaining data
stores.
 data is monitored, maintained and managed by a
person and/or organization.
 allows an organization to check its data
resources, along with their processing and
communications with different applications and
business processes.

 data usage and handling is working towards
the enterprise’s objective.
 to integrate data from multiple resources and
provide it to various applications.

 Data Administrator deals with designing of
the logical and conceptual models treating the
data at an organizational level.
 Database Administrators deal with the
implementation of databases required and in
use.

 Responsibilities of data administrators:
 Data policies, procedures and standards:
 Data policy: can interact with which data, how
that data can be changed and what is the effect of
the change.

 Data Procedures: documented plan of actions to
be taken to perform a certain activity like backup
and recovery procedures.

 Data standards: conventions and behaviors that
need to be followed so that the maintenance
becomes easy.

 Planning:
 to plan for an effective administration of data and
also provide support for future needs.

 Data conflict resolution:
 establish procedures for resolving any conflicts in
ownership.
 authority to mediate and enforce the resolution of
the conflict, they may be very effective in this
capacity.

 Managing the data repository:
 contain metadata that holds data description of the
data stored in data stores.
 describe an organization’s data and data
processing resources.

 Internal marketing of DA concepts:
 established policies and procedures must be made
known to the internal staff.

 designing the database:
 defining and creating the logical data model, physical
database model and prototyping.
 Security and authorization:
 ensures that there is no unauthorized access to the data.
 users may be granted permission to access only certain
views and relations.

Data availability and recovery from failures:
 ensure that the data is made available to its user in
such way that the users are unaware of the failure.
 Database tuning:
 modifying the database, the conceptual and logical
design.

 Creating the data repository: integrating it to
create a common data repository is
challenging.
 Emphasize the capability to build a database
quickly, tune it for maximum performance and
restore it to production quickly when problems
develop.

 Enforcing the data policies and standards,
especially those related to security.
 support should be provided to incorporate the
changes and make provision for future scope.
 with the social media, should define the
ownership of data.

 The administrator is always expected to keep
side by side with new technologies, and is
usually involved in mission-critical
applications.
 to have a comprehensive understanding of a
wide variety of topics and improve business
processes in their organization.

IT6701-Information Management Unit 5

More Related Content

What's hot

Similar to IT6701-Information Management Unit 5

More from SIMONTHOMAS S

Recently uploaded

IT6701-Information Management Unit 5