In this book readers will find technological discussions on the existing and emerging technologies across the different stages of the big data value chain. They will learn about legal aspects of big data, the social impact, and about education needs and requirements. And they will discover the business perspective and how big data technology can be exploited to deliver value within different sectors of the economy.
The book is structured in four parts: Part I “The Big Data Opportunity” explores the value potential of big data with a particular focus on the European context. It also describes the legal, business and social dimensions that need to be addressed, and briefly introduces the European Commission’s BIG project. Part II “The Big Data Value Chain” details the complete big data lifecycle from a technical point of view, ranging from data acquisition, analysis, curation and storage, to data usage and exploitation. Next, Part III “Usage and Exploitation of Big Data” illustrates the value creation possibilities of big data applications in various sectors, including industry, healthcare, finance, energy, media and public services. Finally, Part IV “A Roadmap for Big Data Research” identifies and prioritizes the cross-sectorial requirements for big data research, and outlines the most urgent and challenging technological, economic, political and societal issues for big data in Europe.
This compendium summarizes more than two years of work performed by a leading group of major European research centers and industries in the context of the BIG project. It brings together research findings, forecasts and estimates related to this challenging technological context that is becoming the major axis of the new digitally transformed business environment.
“This book reports on preparatory work toward an important policy objective of the European Commission: turning Europe into a safe and privacy-respecting society that thrives by extracting maximum value from the data it produces and reuses, be it in support of important societal goals or as fuel for innovation in productive activities. Our plans for Europe are described in our July 2014 Communication on a data driven economy, where we spell out a three-pronged approach addressing regulatory issues (such as personal data protection and data ownership), framework conditions (such as data standards and infrastructures), and community building. The first visible step of our community building efforts is a massive commitment (534 million Euros by 2020), which we signed in October 2014, to enter in a Public Private Partnership with the Big Data Value Association (BDVA): with the help from industrial parties and groups that represent relevant societal concerns (such as privacy), we intend to identify and solve technical problems and framework conditions (such as skill development) that stand in the way of European companies increasing their productivity and innovativeness by making efficient use of data technologies. By shouldering some of the financial risk of these activities, we plan to leverage even more massive European investment: for every public Euro invested by the Commission, our industry partners have committed to investing four private Euros. Naturally, this requires some well-informed and clear thinking on which domains of data-related activities hold the greatest promise for a safe and prosperous Europe and on how we can avoid wasteful duplication in the development of data infrastructures, formats, and technologies. The book you are holding in your hands gives you a first lay of the land: it results from more than two years of work(also funded by the European Union) aimed at identifying issues and opportunities that are specifically European in character. We fully expect that many of these results will be included and further elaborated over the years in the strategic planning of the BDVA, and we are happy to share them in this book with the broader public. We hope that you will find them informative and that they will help you shape your own thinking on what your expectations and active role might be in a better Europe that has taught itself to run on data.”
Directorate G Media & Data, European Commission DG CONNECT
“Data has become a factor just as important to production as labor, capital, and land. For the new value creators in today’s technology start-ups, little capital and office space is required. Both can be almost free when a firm is growing 1 % per day, on any metric. But without talent, and without the right kind of data, such a takeoff is highly improbable. We see the same forces at play in SAP’s Innovation Center Network. Attracting the right talent was critical to establish the first Innovation Center in Potsdam. And having large, real-world datasets from customers and co-innovation partners is critical to many of our innovations. To make a difference in cancer treatment and research with our Medical Research Insights app, we critically depended on data-driven collaboration with the National Center for Tumor Diseases. The same holds for incubating SAP’s new sports line of business by co-innovating with the German national soccer team based on real-time sensor feeds from their players. And it holds true for SAP’s many initiatives in the Internet of Things, like the predictive maintenance apps with John Deere and Kaeser. The Big Data Value Association (BDVA) is poised to make a difference both for data availability and for talent. By bringing together businesses with leading researchers, software and hardware partners, and enabling co-innovation around large, real-world datasets, BDVA can help lower the data barrier. And helping educate the next generation of thought leaders, especially in data science, computer science, and related fields, BDVA can help increase the supply of talent. Both are critical so Europe can begin to lead, not follow, in creating value from big data. By clearly defining the opportunity in big data, by examining the big data value chain, and by deep-diving into industry sector applications, this book charts a way forward to new value creation and new opportunities from big data. Decision makers, policy advisors, researchers, and practitioners on all levels can benefit from this.”
Vice President, SAP Innovation Center Network. President, Big Data Value Association
Big Data Value Ecosystem
Big data is expected to impact all sectors, from healthcare to media, from energy to retail. The ability to effectively manage information and extract knowledge is now seen as a key competitive advantage for organizations. This chapter explores the value potential of big data with a particular focus on the European context. The chapter identifies the positive transformational potential of big data within a number of key sectors and highlights the need for a clear strategy to increase the competitiveness of European industries in order to drive innovation and competitiveness. Europe needs to foster the development and wide adoption of big data technologies, value adding use cases, and sustainable business models through a Big Data Ecosystem. Finally the chapter describes the key dimensions, including skills, legal, business, and social, that need to be addressed in a European Big Data Ecosystem.
Big Data Supply and Demand
The Big Data Public Private Forum (BIG) Project (http://www.big-project.eu/) was an EU coordination and support action to provide a roadmap for big data within Europe. The BIG project worked towards the definition and implementation of a clear big data strategy that tackled the necessary activities needed in research and innovation, technology adoption, and the required support from the European Commission necessary for the successful implementation of the big data economy. As part of this strategy, the outcomes of the project were used as input for Horizon 2020. This chapter provides an overview of the BIG project detailing the project’s mission and strategic objectives. The chapter describes the partners within the consortium and the overall structure of the project work. The three-phase methodology used in the project is described, including details on the techniques used within the technical working groups, sectorial forms, and road mapping activity. Finally, the project’s role in setting up the big data contractual Public Private Partnership (cPPP) and Big Data Value Association is discussed.
Data Ecosystem Stakeholders
Big data is the emerging field where innovative technology offers new ways to extract value from the tsunami of available information. As with any emerging area, terms and concepts can be open to different interpretations. The Big Data domain is no different. This chapter examines the different definitions of “Big Data” which have emerged over the last number of years to label data with different attributes. The Big Data Value Chain is introduced to describe the information flow within a big data system as a series of steps needed to generate value and useful insights from data. The value chain enables the analysis of big data technologies for each step within the chain. The chapter explores the concept of a Big Data Ecosystem. It examines the use of the ecosystem metaphor within the business community to describe the business environment and how it can be extended to the big data context. Key stakeholders of a big data ecosystem are identified together with the challenges that need to be overcome to enable a big data ecosystem in Europe.
Shift from Storage to Analytics
This chapter provides an overview of big data storage technologies. It is the result of a survey of the current state of the art in data storage technologies in order to create a cross-sectorial technology roadmap. This chapter provides a concise overview of big data storage systems that are capable of dealing with high velocity, high volumes, and high varieties of data. It describes distributed file systems, NoSQL databases, graph databases, and NewSQL databases. The chapter investigates the challenge of storing data in a secure and privacy-preserving way. The social and economic impact of big data storage technologies is described, open research challenges highlighted, and three selected case studies are provided from the health, finance, and energy sector. Some of the key insights on big data storage are (1) in-memory databases and columnar databases typically outperform traditional relational database systems, (2) the major technical barrier to widespread up-take of big data storage solutions are missing standards, and (3) there is a need to address open research challenges related to the scalability and performance of graph databases.
Big Data in Service Infrastructure
Big data usage covers the business goals that need access to data, its analyses, and integration into business decision-making. This chapter gives an overview of the applications of big data, focusing on decision support through big data in different sectors. Big data usage is a wide field that is addressed in this chapter by viewing data usage from various perspectives, including the underlying technology stacks, trends in various sectors, the impact on business models, and requirements on human–computer interaction. The chapter explores data usage tools, query and scripting languages, execution engines, APIs, programming models, different technology stacks, and some of the trade-offs involved are discussed. The chapter presents general aspects of decision support, followed by a discussion of specific access to analysis results through visualization and new explorative interfaces. Emerging trends and future requirements are presented with special emphasis on Industry 4.0 and the emerging need for smart data and smart services.
European Big Data Roadmaps
This chapter describes big data roadmaps for Europe in the areas of technology, business, policy, and society. The roadmaps outline the most urgent and challenging issues for big data in Europe. They are the result of over 2 years of extensive research and input from a wide range of stakeholders from the European big data ecosystem. The roadmaps will foster the creation of a more stable big data environment by enabling enterprises, business, entrepreneurs, SMEs, and society to gain from the benefits of big data in Europe. The chapter introduces the Big Data Value Association (BDVA) and the Big Data Value contractual Public Private Partnership (BDV cPPP) and describes the role played by the BIG project in their establishment. The BDVA and the BDV cPPP will provide the necessary framework for industrial leadership, investment, and commitment of both the private and public side to build a data-driven economy across Europe.