WARCProcessor: An Integrative Tool for Building and Management of Web Spam Corpora

WARCProcessor: An Integrative Tool for Building and Management of Web Spam Corpora

Miguel Callón;Jorge Fdez-Glez;David Ruano-Ordás;Rosalía Laza;Reyes Pavón;Florentino Fdez-Riverola;Jose Ramón Méndez;Callón, Miguel;Fdez-Glez, Jorge;Ruano-Ordás, David;Laza, Rosalía;Pavón, Reyes;Fdez-Riverola, Florentino;Méndez, Jose Ramón;
sensors 2017 Vol. 18 pp. 16-
123
callón2017sensorswarcprocessor:

Abstract

In this work we present the design and implementation of WARCProcessor, a novel multiplatform integrative tool aimed to build scientific datasets to facilitate experimentation in web spam research. The developed application allows the user to specify multiple criteria that change the way in which new corpora are generated whilst reducing the number of repetitive and error prone tasks related with existing corpus maintenance. For this goal, WARCProcessor supports up to six commonly used data sources for web spam research, being able to store output corpus in standard WARC format together with complementary metadata files. Additionally, the application facilitates the automatic and concurrent download of web sites from Internet, giving the possibility of configuring the deep of the links to be followed as well as the behaviour when redirected URLs appear. WARCProcessor supports both an interactive GUI interface and a command line utility for being executed in background.

Citation

ID: 273389
Ref Key: callón2017sensorswarcprocessor:
Use this key to autocite in SciMatic or Thesis Manager

References

Blockchain Verification

Account:
NFT Contract Address:
0x95644003c57E6F55A65596E3D9Eac6813e3566dA
Article ID:
273389
Unique Identifier:
10.3390/s18010016
Network:
Scimatic Chain (ID: 481)
Loading...
Blockchain Readiness Checklist
Authors
Abstract
Journal Name
Year
Title
5/5
Creates 1,000,000 NFT tokens for this article
Token Features:
  • ERC-1155 Standard NFT
  • 1 Million Supply per Article
  • Transferable via MetaMask
  • Permanent Blockchain Record
Blockchain QR Code
Scan with Saymatik Web3.0 Wallet

Saymatik Web3.0 Wallet