Big Insurance Data: Drawing Lessons from Amazon, Google, and Facebook

by Craig Beattie, December 19, 2011
Industry Trends
Global, Asia-Pacific, EMEA, Latin America, North America

Abstract

There has been much written about the emergence of Big Data, both as a phenomenon and as a set of practices, infrastructure, and algorithms designed to allow modern computing to analyse ever-increasing data in fast and efficient ways. Celent uses the open source Hadoop solution to provide insight into what Big Data is, what it means to the insurance industry, and the kinds of solutions it provides to new and old insurance problems.

In a new report, Big Insurance Data: Drawing Lessons from Amazon, Google, and Facebook, Celent reviews how Big Data is now relevant and accessible to the insurance industry. In this report, Celent offers a discussion on the following topics:

  • Grid computing or the use of many distributed, connected machines to undertake large-scale work.
  • Distributed file systems, which store large amounts of data across many machines.
  • MapReduce, which is an algorithm to split workloads over grid computing.
  • Hadoop, which is a specific implementation of the MapReduce algorithm.

“The simple fact is that, with little more than a credit card, today you can rent an infrastructure from Amazon similar to that used by Yahoo to index and search the entire Internet,” says Craig Beattie, Analyst with Celent’s Insurance group and author of the report. “It is important that insurers understand just how much these frameworks do on their behalf and how easy it is to leverage this infrastructure to ask big questions of our data.”

The report draws together recent developments in cloud infrastructures, grid computing, and distributed computing to address issues regarding large volumes of structured and unstructured data. While there are many approaches to leveraging grid infrastructures, this report details one of the most popular approaches and perhaps one of the simpler approaches to implement. While the topic is necessarily technical in nature, the report addresses it in an accessible manner.

Celent is a research and advisory firm dedicated to helping financial institutions formulate comprehensive business and technology strategies. Celent publishes reports identifying trends and best practices in financial services technology and conducts consulting engagements for financial institutions looking to use technology to enhance existing business processes or launch new business strategies. With a team of internationally based analysts, Celent is uniquely positioned to offer strategic advice and market insights on a global basis. Celent is a member of the Oliver Wyman Group, which is a wholly-owned subsidiary of Marsh & McLennan Companies [NYSE: MMC].

Media Contacts

North America
Michele Pace
mpace@celent.com
Tel: +1 212 345 1366

Europe (London)
Chris Williams
cwilliams@celent.com
Tel: +44 (0)782 448 3336

Asia (Tokyo)
Yumi Nagaoka
ynagaoka@celent.com
Tel.: +81 3 3500 3023

Table of Contents

Executive Summary

3

Introduction

4

 

An Overview of the Solution

5

An Overview of Grid Computing

6

Storing Data in Distributed Storage

8

 

Distributed Storage Isn’t Simple

8

 

Distributed File Systems

9

 

Alternative Solutions

11

The MapReduce Approach

13

 

Build: The Map Then Reduce Approach

13

 

Distribution and Sharing

14

 

Execution

14

Example Applications of Hadoop

16

 

Sorting Data

16

 

Finding Recommendations

16

 

Analysing Social Data

17

 

Analysing Telematics Data

18

 

Programming Hadoop

19

 

Considerations

20

Adopting Hadoop

21

 

Hadoop Grids as a Data Analysis Resource

21

 

Hadoop Grids for Real Time Log Capture

21

 

Hadoop Offerings

21

Conclusion

23

Leveraging Celent’s Expertise

24

 

Support for Financial Institutions

24

 

Support for Vendors

24

Related Celent Research

25

Sign in to download reports and access personalized information