How to Plan a Big Data project

how big data affects businesses and how big data can improve manufacturing and how big data mysql can handle and how big data analytics is implemented
LexiWills Profile Pic
LexiWills,United Kingdom,Professional
Published Date:01-08-2017
Your Website URL(Optional)
Comment
Planning for Big Data A CIO’s Handbook to the Changing Data Landscape O’Reilly Radar TeamCHAPTER 1 The Feedback Economy By Alistair Croll Military strategist John Boyd spent a lot of time understanding how to win battles. Building on his experience as a fighter pilot, he broke down the process of observing and reacting into something called an Observe, Orient, Decide, and Act (OODA) loop. Combat, he realized, consisted of observing your cir- cumstances, orienting yourself to your enemy’s way of thinking and your en- vironment, deciding on a course of action, and then acting on it. The Observe, Orient, Decide, and Act (OODA) loop. Larger version available here.. The most important part of this loop isn’t included in the OODA acronym, however. It’s the fact that it’s a loop. The results of earlier actions feed back into later, hopefully wiser, ones. Over time, the fighter “gets inside” their op- ponent’s loop, outsmarting and outmaneuvering them. The system learns. Boyd’s genius was to realize that winning requires two things: being able to collect and analyze information better, and being able to act on that informa- 1tion faster, incorporating what’s learned into the next iteration. Today, what Boyd learned in a cockpit applies to nearly everything we do. Data-Obese, Digital-Fast In our always-on lives we’re flooded with cheap, abundant information. We need to capture and analyze it well, separating digital wheat from digital chaff, identifying meaningful undercurrents while ignoring meaningless social flot- sam. Clay Johnson argues that we need to go on an information diet, and makes a good case for conscious consumption. In an era of information obesity, we need to eat better. There’s a reason they call it a feed, after all. It’s not just an overabundance of data that makes Boyd’s insights vital. In the last 20 years, much of human interaction has shifted from atoms to bits. When interactions become digital, they become instantaneous, interactive, and easily copied. It’s as easy to tell the world as to tell a friend, and a day’s shopping is reduced to a few clicks. The move from atoms to bits reduces the coefficient of friction of entire in- dustries to zero. Teenagers shun e-mail as too slow, opting for instant mes- sages. The digitization of our world means that trips around the OODA loop happen faster than ever, and continue to accelerate. We’re drowning in data. Bits are faster than atoms. Our jungle-surplus wet- ware can’t keep up. At least, not without Boyd’s help. In a society where every person, tethered to their smartphone, is both a sensor and an end node, we need better ways to observe and orient, whether we’re at home or at work, solving the world’s problems or planning a play date. And we need to be con- stantly deciding, acting, and experimenting, feeding what we learn back into future behavior. We’re entering a feedback economy. The Big Data Supply Chain Consider how a company collects, analyzes, and acts on data. 2 Chapter 1: The Feedback EconomyThe big data supply chain. Larger version available here.. Let’s look at these components in order. Data collection The first step in a data supply chain is to get the data in the first place. Information comes in from a variety of sources, both public and private. We’re a promiscuous society online, and with the advent of low-cost data market- places, it’s possible to get nearly any nugget of data relatively affordably. From social network sentiment, to weather reports, to economic indicators, public information is grist for the big data mill. Alongside this, we have organization- specific data such as retail traffic, call center volumes, product recalls, or cus- tomer loyalty indicators. The legality of collection is perhaps more restrictive than getting the data in the first place. Some data is heavily regulated — HIPAA governs healthcare, while PCI restricts financial transactions. In other cases, the act of combining data may be illegal because it generates personally identifiable information (PII). For example, courts have ruled differently on whether IP addresses aren’t PII, and the California Supreme Court ruled that zip codes are. Navigating these regulations imposes some serious constraints on what can be collected and how it can be combined. The era of ubiquitous computing means that everyone is a potential source of data, too. A modern smartphone can sense light, sound, motion, location, nearby networks and devices, and more, making it a perfect data collector. As consumers opt into loyalty programs and install applications, they become sensors that can feed the data supply chain. The Big Data Supply Chain 3In big data, the collection is often challenging because of the sheer volume of information, or the speed with which it arrives, both of which demand new approaches and architectures. Ingesting and cleaning Once the data is collected, it must be ingested. In traditional business intelli- gence (BI) parlance, this is known as Extract, Transform, and Load (ETL): the act of putting the right information into the correct tables of a database schema and manipulating certain fields to make them easier to work with. One of the distinguishing characteristics of big data, however, is that the data is often unstructured. That means we don’t know the inherent schema of the information before we start to analyze it. We may still transform the informa- tion — replacing an IP address with the name of a city, for example, or ano- nymizing certain fields with a one-way hash function — but we may hold onto the original data and only define its structure as we analyze it. Hardware The information we’ve ingested needs to be analyzed by people and machines. That means hardware, in the form of computing, storage, and networks. Big data doesn’t change this, but it does change how it’s used. Virtualization, for example, allows operators to spin up many machines temporarily, then destroy them once the processing is over. Cloud computing is also a boon to big data. Paying by consumption destroys the barriers to entry that would prohibit many organizations from playing with large datasets, because there’s no up-front investment. In many ways, big data gives clouds something to do. Platforms Where big data is new is in the platforms and frameworks we create to crunch large amounts of information quickly. One way to speed up data analysis is to break the data into chunks that can be analyzed in parallel. Another is to build a pipeline of processing steps, each optimized for a particular task. Big data is often about fast results, rather than simply crunching a large amount of information. That’s important for two reasons: 1. Much of the big data work going on today is related to user interfaces and the web. Suggesting what books someone will enjoy, or delivering search results, or finding the best flight, requires an answer in the time it takes a 4 Chapter 1: The Feedback Economypage to load. The only way to accomplish this is to spread out the task, which is one of the reasons why Google has nearly a million servers. 2. We analyze unstructured data iteratively. As we first explore a dataset, we don’t know which dimensions matter. What if we segment by age? Filter by country? Sort by purchase price? Split the results by gender? This kind of “what if” analysis is exploratory in nature, and analysts are only as productive as their ability to explore freely. Big data may be big. But if it’s not fast, it’s unintelligible. Much of the hype around big data companies today is a result of the retooling of enterprise BI. For decades, companies have relied on structured relational databases and data warehouses — many of them can’t handle the exploration, lack of structure, speed, and massive sizes of big data applications. Machine learning One way to think about big data is that it’s “more data than you can go through by hand.” For much of the data we want to analyze today, we need a machine’s help. Part of that help happens at ingestion. For example, natural language process- ing tries to read unstructured text and deduce what it means: Was this Twitter user happy or sad? Is this call center recording good, or was the customer angry? Machine learning is important elsewhere in the data supply chain. When we analyze information, we’re trying to find signal within the noise, to discern patterns. Humans can’t find signal well by themselves. Just as astronomers use algorithms to scan the night’s sky for signals, then verify any promising anomalies themselves, so too can data analysts use machines to find interesting dimensions, groupings, or patterns within the data. Machines can work at a lower signal-to-noise ratio than people. Human exploration While machine learning is an important tool to the data analyst, there’s no substitute for human eyes and ears. Displaying the data in human-readable form is hard work, stretching the limits of multi-dimensional visualization. While most analysts work with spreadsheets or simple query languages today, that’s changing. Creve Maples, an early advocate of better computer interaction, designs sys- tems that take dozens of independent, data sources and displays them in nav- igable 3D environments, complete with sound and other cues. Maples’ studies The Big Data Supply Chain 5show that when we feed an analyst data in this way, they can often find answers in minutes instead of months. This kind of interactivity requires the speed and parallelism explained above, as well as new interfaces and multi-sensory environments that allow an analyst to work alongside the machine, immersed in the data. Storage Big data takes a lot of storage. In addition to the actual information in its raw form, there’s the transformed information; the virtual machines used to crunch it; the schemas and tables resulting from analysis; and the many formats that legacy tools require so they can work alongside new technology. Often, storage is a combination of cloud and on-premise storage, using traditional flat-file and relational databases alongside more recent, post-SQL storage systems. During and after analysis, the big data supply chain needs a warehouse. Com- paring year-on-year progress or changes over time means we have to keep copies of everything, along with the algorithms and queries with which we analyzed it. Sharing and acting All of this analysis isn’t much good if we can’t act on it. As with collection, this isn’t simply a technical matter — it involves legislation, organizational politics, and a willingness to experiment. The data might be shared openly with the world, or closely guarded. The best companies tie big data results into everything from hiring and firing decisions, to strategic planning, to market positioning. While it’s easy to buy into big data technology, it’s far harder to shift an organization’s culture. In many ways, big data adoption isn’t a hardware retirement issue, it’s an em- ployee retirement one. We’ve seen similar resistance to change each time there’s a big change in in- formation technology. Mainframes, client-server computing, packet-based networks, and the web all had their detractors. A NASA study into the failure of Ada, the first object-oriented language, concluded that proponents had over-promised, and there was a lack of a supporting ecosystem to help the new language flourish. Big data, and its close cousin, cloud computing, are likely to encounter similar obstacles. A big data mindset is one of experimentation, of taking measured risks and assessing their impact quickly. It’s similar to the Lean Startup movement, which advocates fast, iterative learning and tight links to customers. But while 6 Chapter 1: The Feedback Economya small startup can be lean because it’s nascent and close to its market, a big organization needs big data and an OODA loop to react well and iterate fast. The big data supply chain is the organizational OODA loop. It’s the big busi- ness answer to the lean startup. Measuring and collecting feedback Just as John Boyd’s OODA loop is mostly about the loop, so big data is mostly about feedback. Simply analyzing information isn’t particularly useful. To work, the organization has to choose a course of action from the results, then observe what happens and use that information to collect new data or analyze things in a different way. It’s a process of continuous optimization that affects every facet of a business. Replacing Everything with Data Software is eating the world. Verticals like publishing, music, real estate and banking once had strong barriers to entry. Now they’ve been entirely disrupted by the elimination of middlemen. The last film projector rolled off the line in 2011: movies are now digital from camera to projector. The Post Office stum- bles because nobody writes letters, even as Federal Express becomes the planet’s supply chain. Companies that get themselves on a feedback footing will dominate their in- dustries, building better things faster for less money. Those that don’t are al- ready the walking dead, and will soon be little more than case studies and colorful anecdotes. Big data, new interfaces, and ubiquitous computing are tectonic shifts in the way we live and work. A Feedback Economy Big data, continuous optimization, and replacing everything with data pave the way for something far larger, and far more important, than simple business efficiency. They usher in a new era for humanity, with all its warts and glory. They herald the arrival of the feedback economy. The efficiencies and optimizations that come from constant, iterative feedback will soon become the norm for businesses and governments. We’re moving beyond an information economy. Information on its own isn’t an advantage, anyway. Instead, this is the era of the feedback economy, and Boyd is, in many ways, the first feedback economist. A Feedback Economy 7Alistair Croll is the founder of Bitcurrent, a research firm focused on emerging technologies. He’s founded a variety of startups, and technology accelerators, including Year One Labs, CloudOps, Rednod, Coradiant (acquired by BMC in 2011) and Networkshop. He’s a frequent speaker and writer on subjects such as entrepreneurship, cloud computing, Big Data, Internet performance and web technology, and has helped launch a number of major conferences on these topics. 8 Chapter 1: The Feedback EconomyCHAPTER 2 What Is Big Data? By Edd Dumbill Big data is data that exceeds the processing capacity of conventional database systems. The data is too big, moves too fast, or doesn’t fit the strictures of your database architectures. To gain value from this data, you must choose an al- ternative way to process it. The hot IT buzzword of 2012, big data has become viable as cost-effective approaches have emerged to tame the volume, velocity and variability of mas- sive data. Within this data lie valuable patterns and information, previously hidden because of the amount of work required to extract them. To leading corporations, such as Walmart or Google, this power has been in reach for some time, but at fantastic cost. Today’s commodity hardware, cloud archi- tectures and open source software bring big data processing into the reach of the less well-resourced. Big data processing is eminently feasible for even the small garage startups, who can cheaply rent server time in the cloud. The value of big data to an organization falls into two categories: analytical use, and enabling new products. Big data analytics can reveal insights hidden previously by data too costly to process, such as peer influence among cus- tomers, revealed by analyzing shoppers’ transactions, social and geographical data. Being able to process every item of data in reasonable time removes the troublesome need for sampling and promotes an investigative approach to data, in contrast to the somewhat static nature of running predetermined re- ports. The past decade’s successful web startups are prime examples of big data used as an enabler of new products and services. For example, by combining a large number of signals from a user’s actions and those of their friends, Facebook has been able to craft a highly personalized user experience and create a new kind of advertising business. It’s no coincidence that the lion’s share of ideas 9and tools underpinning big data have emerged from Google, Yahoo, Amazon and Facebook. The emergence of big data into the enterprise brings with it a necessary coun- terpart: agility. Successfully exploiting the value in big data requires experi- mentation and exploration. Whether creating new products or looking for ways to gain competitive advantage, the job calls for curiosity and an entre- preneurial outlook. What Does Big Data Look Like? As a catch-all term, “big data” can be pretty nebulous, in the same way that the term “cloud” covers diverse technologies. Input data to big data systems could be chatter from social networks, web server logs, traffic flow sensors, satellite imagery, broadcast audio streams, banking transactions, MP3s of rock music, the content of web pages, scans of government documents, GPS trails, telemetry from automobiles, financial market data, the list goes on. Are these all really the same thing? To clarify matters, the three Vs of volume, velocity and variety are commonly used to characterize different aspects of big data. They’re a helpful lens through which to view and understand the nature of the data and the software plat- forms available to exploit them. Most probably you will contend with each of the Vs to one degree or another. 10 Chapter 2: What Is Big Data?Volume The benefit gained from the ability to process large amounts of information is the main attraction of big data analytics. Having more data beats out having better models: simple bits of math can be unreasonably effective given large amounts of data. If you could run that forecast taking into account 300 factors rather than 6, could you predict demand better? This volume presents the most immediate challenge to conventional IT struc- tures. It calls for scalable storage, and a distributed approach to querying. Many companies already have large amounts of archived data, perhaps in the form of logs, but not the capacity to process it. Assuming that the volumes of data are larger than those conventional relational database infrastructures can cope with, processing options break down broadly into a choice between massively parallel processing architectures — data warehouses or databases such as Greenplum — and Apache Hadoop- based solutions. This choice is often informed by the degree to which the one of the other “Vs” — variety — comes into play. Typically, data warehousing approaches involve predetermined schemas, suiting a regular and slowly evolving dataset. Apache Hadoop, on the other hand, places no conditions on the structure of the data it can process. At its core, Hadoop is a platform for distributing computing problems across a number of servers. First developed and released as open source by Yahoo, it implements the MapReduce approach pioneered by Google in compiling its search indexes. Hadoop’s MapReduce involves distributing a dataset among multiple servers and operating on the data: the “map” stage. The partial results are then recombined: the “reduce” stage. To store data, Hadoop utilizes its own distributed filesystem, HDFS, which makes data available to multiple computing nodes. A typical Hadoop usage pattern involves three stages: • loading data into HDFS, • MapReduce operations, and • retrieving results from HDFS. This process is by nature a batch operation, suited for analytical or non-inter- active computing tasks. Because of this, Hadoop is not itself a database or data warehouse solution, but can act as an analytical adjunct to one. One of the most well-known Hadoop users is Facebook, whose model follows this pattern. A MySQL database stores the core data. This is then reflected into Hadoop, where computations occur, such as creating recommendations for What Does Big Data Look Like? 11you based on your friends’ interests. Facebook then transfers the results back into MySQL, for use in pages served to users. Velocity The importance of data’s velocity — the increasing rate at which data flows into an organization — has followed a similar pattern to that of volume. Prob- lems previously restricted to segments of industry are now presenting them- selves in a much broader setting. Specialized companies such as financial trad- ers have long turned systems that cope with fast moving data to their advan- tage. Now it’s our turn. Why is that so? The Internet and mobile era means that the way we deliver and consume products and services is increasingly instrumented, generating a data flow back to the provider. Online retailers are able to compile large his- tories of customers’ every click and interaction: not just the final sales. Those who are able to quickly utilize that information, by recommending additional purchases, for instance, gain competitive advantage. The smartphone era in- creases again the rate of data inflow, as consumers carry with them a streaming source of geolocated imagery and audio data. It’s not just the velocity of the incoming data that’s the issue: it’s possible to stream fast-moving data into bulk storage for later batch processing, for ex- ample. The importance lies in the speed of the feedback loop, taking data from input through to decision. A commercial from IBM makes the point that you wouldn’t cross the road if all you had was a five-minute old snapshot of traffic location. There are times when you simply won’t be able to wait for a report to run or a Hadoop job to complete. Industry terminology for such fast-moving data tends to be either “streaming data,” or “complex event processing.” This latter term was more established in product categories before streaming processing data gained more wide- spread relevance, and seems likely to diminish in favor of streaming. There are two main reasons to consider streaming processing. The first is when the input data are too fast to store in their entirety: in order to keep storage requirements practical some level of analysis must occur as the data streams in. At the extreme end of the scale, the Large Hadron Collider at CERN gen- erates so much data that scientists must discard the overwhelming majority of it — hoping hard they’ve not thrown away anything useful. The second reason to consider streaming is where the application mandates immediate response to the data. Thanks to the rise of mobile applications and online gaming this is an increasingly common situation. 12 Chapter 2: What Is Big Data?Product categories for handling streaming data divide into established propri- etary products such as IBM’s InfoSphere Streams, and the less-polished and still emergent open source frameworks originating in the web industry: Twit- ter’s Storm, and Yahoo S4. As mentioned above, it’s not just about input data. The velocity of a system’s outputs can matter too. The tighter the feedback loop, the greater the com- petitive advantage. The results might go directly into a product, such as Face- book’s recommendations, or into dashboards used to drive decision-making. It’s this need for speed, particularly on the web, that has driven the develop- ment of key-value stores and columnar databases, optimized for the fast re- trieval of precomputed information. These databases form part of an umbrella category known as NoSQL, used when relational models aren’t the right fit. Variety Rarely does data present itself in a form perfectly ordered and ready for pro- cessing. A common theme in big data systems is that the source data is diverse, and doesn’t fall into neat relational structures. It could be text from social networks, image data, a raw feed directly from a sensor source. None of these things come ready for integration into an application. Even on the web, where computer-to-computer communication ought to bring some guarantees, the reality of data is messy. Different browsers send different data, users withhold information, they may be using differing software ver- sions or vendors to communicate with you. And you can bet that if part of the process involves a human, there will be error and inconsistency. A common use of big data processing is to take unstructured data and extract ordered meaning, for consumption either by humans or as a structured input to an application. One such example is entity resolution, the process of deter- mining exactly what a name refers to. Is this city London, England, or London, Texas? By the time your business logic gets to it, you don’t want to be guessing. The process of moving from source data to processed application data involves the loss of information. When you tidy up, you end up throwing stuff away. This underlines a principle of big data: when you can, keep everything. There may well be useful signals in the bits you throw away. If you lose the source data, there’s no going back. Despite the popularity and well understood nature of relational databases, it is not the case that they should always be the destination for data, even when tidied up. Certain data types suit certain classes of database better. For in- stance, documents encoded as XML are most versatile when stored in a dedi- cated XML store such as MarkLogic. Social network relations are graphs by What Does Big Data Look Like? 13nature, and graph databases such as Neo4J make operations on them simpler and more efficient. Even where there’s not a radical data type mismatch, a disadvantage of the relational database is the static nature of its schemas. In an agile, exploratory environment, the results of computations will evolve with the detection and extraction of more signals. Semi-structured NoSQL databases meet this need for flexibility: they provide enough structure to organize data, but do not re- quire the exact schema of the data before storing it. In Practice We have explored the nature of big data, and surveyed the landscape of big data from a high level. As usual, when it comes to deployment there are di- mensions to consider over and above tool selection. Cloud or in-house? The majority of big data solutions are now provided in three forms: software- only, as an appliance or cloud-based. Decisions between which route to take will depend, among other things, on issues of data locality, privacy and regu- lation, human resources and project requirements. Many organizations opt for a hybrid solution: using on-demand cloud resources to supplement in-house deployments. Big data is big It is a fundamental fact that data that is too big to process conventionally is also too big to transport anywhere. IT is undergoing an inversion of priorities: it’s the program that needs to move, not the data. If you want to analyze data from the U.S. Census, it’s a lot easier to run your code on Amazon’s web services platform, which hosts such data locally, and won’t cost you time or money to transfer it. Even if the data isn’t too big to move, locality can still be an issue, especially with rapidly updating data. Financial trading systems crowd into data centers to get the fastest connection to source data, because that millisecond difference in processing time equates to competitive advantage. Big data is messy It’s not all about infrastructure. Big data practitioners consistently report that 80% of the effort involved in dealing with data is cleaning it up in the first 14 Chapter 2: What Is Big Data?place, as Pete Warden observes in his Big Data Glossary: “I probably spend more time turning messy source data into something usable than I do on the rest of the data analysis process combined.” Because of the high cost of data acquisition and cleaning, it’s worth considering what you actually need to source yourself. Data marketplaces are a means of obtaining common data, and you are often able to contribute improvements back. Quality can of course be variable, but will increasingly be a benchmark on which data marketplaces compete. Culture The phenomenon of big data is closely tied to the emergence of data science, a discipline that combines math, programming and scientific instinct. Bene- fiting from big data means investing in teams with this skillset, and surround- ing them with an organizational willingness to understand and use data for advantage. In his report, “Building Data Science Teams,” D.J. Patil characterizes data scientists as having the following qualities: • Technical expertise: the best data scientists typically have deep expertise in some scientific discipline. • Curiosity: a desire to go beneath the surface and discover and distill a problem down into a very clear set of hypotheses that can be tested. • Storytelling: the ability to use data to tell a story and to be able to com- municate it effectively. • Cleverness: the ability to look at a problem in different, creative ways. The far-reaching nature of big data analytics projects can have uncomfortable aspects: data must be broken out of silos in order to be mined, and the orga- nization must learn how to communicate and interpet the results of analysis. Those skills of storytelling and cleverness are the gateway factors that ulti- mately dictate whether the benefits of analytical labors are absorbed by an organization. The art and practice of visualizing data is becoming ever more important in bridging the human-computer gap to mediate analytical insight in a meaningful way. Know where you want to go Finally, remember that big data is no panacea. You can find patterns and clues in your data, but then what? Christer Johnson, IBM’s leader for advanced In Practice 15analytics in North America, gives this advice to businesses starting out with big data: first, decide what problem you want to solve. If you pick a real business problem, such as how you can change your adver- tising strategy to increase spend per customer, it will guide your implementa- tion. While big data work benefits from an enterprising spirit, it also benefits strongly from a concrete goal. Edd Dumbill is a technologist, writer and programmer based in California. He is the program chair for the O’Reilly Strata and Open Source Convention Con- ferences. 16 Chapter 2: What Is Big Data?CHAPTER 3 Apache Hadoop By Edd Dumbill Apache Hadoop has been the driving force behind the growth of the big data industry. You’ll hear it mentioned often, along with associated technologies such as Hive and Pig. But what does it do, and why do you need all its strangely- named friends such as Oozie, Zookeeper and Flume? Hadoop brings the ability to cheaply process large amounts of data, regardless of its structure. By large, we mean from 10-100 gigabytes and above. How is this different from what went before? Existing enterprise data warehouses and relational databases excel at process- ing structured data, and can store massive amounts of data, though at cost. However, this requirement for structure restricts the kinds of data that can be processed, and it imposes an inertia that makes data warehouses unsuited for agile exploration of massive heterogenous data. The amount of effort required to warehouse data often means that valuable data sources in organizations are never mined. This is where Hadoop can make a big difference. This article examines the components of the Hadoop ecosystem and explains the functions of each. 17The Core of Hadoop: MapReduce Created at Google in response to the problem of creating web search indexes, the MapReduce framework is the powerhouse behind most of today’s big data processing. In addition to Hadoop, you’ll find MapReduce inside MPP and NoSQL databases such as Vertica or MongoDB. The important innovation of MapReduce is the ability to take a query over a dataset, divide it, and run it in parallel over multiple nodes. Distributing the computation solves the issue of data too large to fit onto a single machine. Combine this technique with commodity Linux servers and you have a cost- effective alternative to massive computing arrays. At its core, Hadoop is an open source MapReduce implementation. Funded by Yahoo, it emerged in 2006 and, according to its creator Doug Cutting, reached “web scale” capability in early 2008. As the Hadoop project matured, it acquired further components to enhance its usability and functionality. The name “Hadoop” has come to represent this entire ecosystem. There are parallels with the emergence of Linux: the name refers strictly to the Linux kernel, but it has gained acceptance as referring to a complete operating system. Hadoop’s Lower Levels: HDFS and MapReduce We discussed above the ability of MapReduce to distribute computation over multiple servers. For that computation to take place, each server must have access to the data. This is the role of HDFS, the Hadoop Distributed File Sys- tem. HDFS and MapReduce are robust. Servers in a Hadoop cluster can fail, and not abort the computation process. HDFS ensures data is replicated with re- dundancy across the cluster. On completion of a calculation, a node will write its results back into HDFS. There are no restrictions on the data that HDFS stores. Data may be unstruc- tured and schemaless. By contrast, relational databases require that data be structured and schemas defined before storing the data. With HDFS, making sense of the data is the responsibility of the developer’s code. Programming Hadoop at the MapReduce level is a case of working with the Java APIs, and manually loading data files into HDFS. 18 Chapter 3: Apache HadoopImproving Programmability: Pig and Hive Working directly with Java APIs can be tedious and error prone. It also restricts usage of Hadoop to Java programmers. Hadoop offers two solutions for mak- ing Hadoop programming easier. • Pig is a programming language that simplifies the common tasks of work- ing with Hadoop: loading data, expressing transformations on the data, and storing the final results. Pig’s built-in operations can make sense of semi-structured data, such as log files, and the language is extensible using Java to add support for custom data types and transformations. • Hive enables Hadoop to operate as a data warehouse. It superimposes structure on data in HDFS, and then permits queries over the data using a familiar SQL-like syntax. As with Pig, Hive’s core capabilities are exten- sible. Choosing between Hive and Pig can be confusing. Hive is more suitable for data warehousing tasks, with predominantly static structure and the need for frequent analysis. Hive’s closeness to SQL makes it an ideal point of integration between Hadoop and other business intelligence tools. Pig gives the developer more agility for the exploration of large datasets, al- lowing the development of succinct scripts for transforming data flows for incorporation into larger applications. Pig is a thinner layer over Hadoop than Hive, and its main advantage is to drastically cut the amount of code needed compared to direct use of Hadoop’s Java APIs. As such, Pig’s intended audi- ence remains primarily the software developer. Improving Data Access: HBase, Sqoop, and Flume At its heart, Hadoop is a batch-oriented system. Data are loaded into HDFS, processed, and then retrieved. This is somewhat of a computing throwback, and often interactive and random access to data is required. Enter HBase, a column-oriented database that runs on top of HDFS. Modeled after Google’s BigTable, the project’s goal is to host billions of rows of data for rapid access. MapReduce can use HBase as both a source and a destination for its computations, and Hive and Pig can be used in combination with HBase. In order to grant random access to the data, HBase does impose a few restric- tions: performance with Hive is 4-5 times slower than plain HDFS, and the maximum amount of data you can store is approximately a petabyte, versus HDFS’ limit of over 30PB. Improving Data Access: HBase, Sqoop, and Flume 19

Advise: Why You Wasting Money in Costly SEO Tools, Use World's Best Free SEO Tool Ubersuggest.