Efficient processing of complex XSD using Hive and Spark

Please use this identifier to cite or link to this item: http://hdl.handle.net/10045/117205
Información del item - Informació de l'item - Item information
Title: Efficient processing of complex XSD using Hive and Spark
Authors: Martinez-Mosquera, Diana | Navarrete, Rosa | Luján-Mora, Sergio
Research Group/s: Advanced deveLopment and empIrical research on Software (ALISoft)
Center, Department or Service: Universidad de Alicante. Departamento de Lenguajes y Sistemas Informáticos
Keywords: Hive | Spark | Performance management | Mobile network | Complex XSD | XML
Knowledge Area: Lenguajes y Sistemas Informáticos
Date Created: 23-Apr-2021
Issue Date: 17-Aug-2021
Publisher: PeerJ
Citation: Martinez-Mosquera D, Navarrete R, Luján-Mora S. 2021. Efficient processing of complex XSD using Hive and Spark. PeerJ Computer Science 7:e652 https://doi.org/10.7717/peerj-cs.652
Abstract: The eXtensible Markup Language (XML) files are widely used by the industry due to their flexibility in representing numerous kinds of data. Multiple applications such as financial records, social networks, and mobile networks use complex XML schemas with nested types, contents, and/or extension bases on existing complex elements or large real-world files. A great number of these files are generated each day and this has influenced the development of Big Data tools for their parsing and reporting, such as Apache Hive and Apache Spark. For these reasons, multiple studies have proposed new techniques and evaluated the processing of XML files with Big Data systems. However, a more usual approach in such works involves the simplest XML schemas, even though, real data sets are composed of complex schemas. Therefore, to shed light on complex XML schema processing for real-life applications with Big Data tools, we present an approach that combines three techniques. This comprises three main methods for parsing XML files: cataloging, deserialization, and positional explode. For cataloging, the elements of the XML schema are mapped into root, arrays, structures, values, and attributes. Based on these elements, the deserialization and positional explode are straightforwardly implemented. To demonstrate the validity of our proposal, we develop a case study by implementing a test environment to illustrate the methods using real data sets provided from performance management of two mobile network vendors. Our main results state the validity of the proposed method for different versions of Apache Hive and Apache Spark, obtain the query execution times for Apache Hive internal and external tables and Apache Spark data frames, and compare the query performance in Apache Hive with that of Apache Spark. Another contribution made is a case study in which a novel solution is proposed for data analysis in the performance management systems of mobile networks.
Sponsor: Unidad de Gestión de Investigación y Proyección Social from the Escuela Politécnica Nacional
URI: http://hdl.handle.net/10045/117205
ISSN: 2376-5992
DOI: 10.7717/peerj-cs.652
Language: eng
Type: info:eu-repo/semantics/article
Rights: © 2021 Martinez-Mosquera et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, reproduction and adaptation in any medium and for any purpose provided that it is properly attributed. For attribution, the original author(s), title, publication source (PeerJ Computer Science) and either DOI or URL of the article must be cited.
Peer Review: si
Publisher version: https://doi.org/10.7717/peerj-cs.652
Appears in Collections:INV - ALISoft - Artículos de Revistas

Files in This Item:
Files in This Item:
File Description SizeFormat 
ThumbnailMartinez-Mosquera_etal_2021_PeerJCompSci.pdf2,25 MBAdobe PDFOpen Preview

This item is licensed under a Creative Commons License Creative Commons