Architecture for big data implementation
What is it about?
Big data is often characterized by 3Vs which is high-volume, high-velocity, and high-variety. It is widely known that big data use can be challenging, since there are issues regarding data access, quality, and methodology, as well as the development of required skillsets. This paper aims to explain a framework specifically designed to support the use of big data in official statistics, along with how existing technology in BPS-Statistics Indonesia will support it.
Why is it important?
Based on the implementation of the proposed framework in the use of MPD for the delineation of a metropolitan area, it has become clear that the implementation of quality assurance processes to ensure the validity of the analysis of big data is an important aspect. This aspect even gains in importance as the Build and Collect phases were not conducted by BPS-Statistics Indonesia, but by Telkomsel. Statistical business processes are not always performed sequentially where there is an iteration between the Design and Process phases in order to maintain the quality of MPD results before an in-depth analysis is carried out. In addition, data scientists at BPS-Statistics Indonesia are able to implement the proposed framework, where supported by an adequate big data infrastructure. Moreover, it is important to establish a sustainable collaboration with MNOs to ensure data availability.