<  Back to the Polytechnique Montréal portal

Detecting very large sets of referenced files at 40/100 GbE, especially MP4 files

Adrien Larbanet, Jonas Lerebours and Jean Pierre David

Paper (2015)

Open Acess document in PolyPublie and at official publisher
Open Access to the full text of this document
Published Version
Terms of Use: Creative Commons Attribution Non-commercial No Derivatives
Download (714kB)
Show abstract
Hide abstract


Internet traffic monitoring is an increasingly challenging task because of the high bandwidths, especially at Internet Service Provider routers and/or Internet backbones. We propose a parallel implementation of the max-hashing algorithm that enables the detection of millions of referenced files by deep packet inspection over high bandwidth connections. We also propose a method to extract high-entropy signatures from MP4 files compatible with the max-hashing algorithm in order to have low false positive rates. The system first computes a set of fingerprints, which are small subsets of the referenced files a priori unique and easily identifiable. At detection time, the max-hashing algorithm eliminates the need to reconstruct the flows. A Graphics Processing Unit (GPU) card computes the fingerprints of all the IP packets in parallel and searches for hits in the onboard collection of fingerprints. Our application, dedicated to the detection of known MP4 video files, enables the detection of millions of fingerprints and demonstrates a sustained processing rate of 50 Gbps per card. Furthermore, a null false positive rate was observed for our 28.25 GB transfer test. The proposed implementation also features the detection of suspect flows based on IP addresses and ports in order to carry out deeper investigations off line.

Uncontrolled Keywords

Video fingerprinting; Network monitoring; Deep packet inspection; Content-based detection; GPU computing

Department: Department of Electrical Engineering
Research Center: GR2M - Microelectronics and Microsystems Research Group
PolyPublie URL: https://publications.polymtl.ca/34302/
Conference Title: 15th Annual DFRWS Conference (DFRWS USA 2015)
Conference Location: Philadelphia, PA, USA
Conference Date(s): 2015-08-09 - 2015-08-12
Journal Title: Digital Investigation (vol. 14, no. suppl. 1)
Publisher: Elsevier
DOI: 10.1016/j.diin.2015.05.011
Official URL: https://doi.org/10.1016/j.diin.2015.05.011
Date Deposited: 18 Apr 2023 15:07
Last Modified: 06 Apr 2024 09:21
Cite in APA 7: Larbanet, A., Lerebours, J., & David, J. P. (2015, August). Detecting very large sets of referenced files at 40/100 GbE, especially MP4 files [Paper]. 15th Annual DFRWS Conference (DFRWS USA 2015), Philadelphia, PA, USA. Published in Digital Investigation, 14(suppl. 1). https://doi.org/10.1016/j.diin.2015.05.011


Total downloads

Downloads per month in the last year

Origin of downloads


Repository Staff Only

View Item View Item