【検証環境】
CPU： Xeon X5675 (6コア) x 2 (3.06 GHz)
メモリ： 96GB
ディスク： 300GB 10000RPM SAS x 4 (RAID 5)
【事前準備】
1行1024バイトのテキストデータをランダムに生成します。
100万行で約1GB、1000万行で約10GBとなります。
生成のソースは以下の通りです。
List 1: createData.pl
#!/usr/bin/perl
use strict;
use warnings;
foreach my $i ( 1 .. 1000000 ){
print sprintf( "%010d,", $i); # row number
print "x" x 979 . ","; # fixed text (dummy)
print sprintf( "id%02d,", int( rand(100) )); # Key-1: eg) country id
print sprintf( "id%05d,", int( rand(1000) )); # Key-2: eg) branch id
print sprintf( "id%010d,", int( rand(100000000) )); # Key-3: eg) customer id
print sprintf( "%7d\n", int( rand(1000000) )); # value
}
データの内容のイメージは以下の通りです。
List 2: Data
$ perl createData.pl > sampledata.txt
$ tail -1 sampledata.txt 0001000000,xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx,id56,id00603,id0060743162, 211271
$ tail -1 sampledata.txt 0001000000,xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx,id56,id00603,id0060743162, 211271
このデータに対し、以下のコマンドで集計をかけます。
List 3: command
cat sampledata_1.txt | awk -F "," '{ s[$3 $4] += $6; c[$3 $4] += 1} END { for( x in s) print x,s[x],c[x] }' | sort > result.txt
List 3: command
cat sampledata_1.txt | awk -F "," '{ s[$3 $4] += $6; c[$3 $4] += 1} END { for( x in s) print x,s[x],c[x] }' | sort > result.txt
【検証結果】
各サイズのファイルに対し、検証を5回行った平均は以下の通りです。
Table 1: Result
File size
|
Records (rows)
|
Elapsed time (average, mm:ss)
|
1GB
|
1,000,000
|
0:31
|
10GB
|
10,000,000
|
4:49
|
20GB
|
20,000,000
|
9:20
|
 |
| Figure 1: Result |
集計対象となるログファイルがさらに巨大である場合はHadoop等のBigData系技術の採用が有用ですが、数十GB程度のテキストであれば、AWKによる処理も検討できます。
AWKによる処理ではデータロード等の前処理の必要がないことも考慮すると、優位性はさらに高まると考えられます。
なお、多くの場合において、全ての処理・ビジネスロジックをAWKで実装する必要はありません。あくまでも「高速な前処理・中間集計」として割り切って使用する事がシステム全体の最適化につながります。
[Summary]
It may sounds old fashioned, but AWK is still good solution when you summarise text file.
In this post, I list the result of simple performance test.
[Hardware]
CPU： Xeon X5675 (6 cores) x 2 (3.06 GHz)
Memory: 96GB
Disk: 300GB 10000RPM SAS x 4 (RAID 5)
[Source code]
List 1 produces the sample data. The size of each text record is 1024 bytes. List 2 shows the data.
List 3 is the main part. It summarize the total and count by two key colums (3rd and 4th column).
[Result]
The test cases were performed 5 times each. Table 1 and Figure 1 show the target file size and the average of process time.
When you handle mid-size text files, you should consider about AWK first. It may provide you good cost effectiveness.
In most of the cases, you should NOT try to process whole business logic with AWK. Use AWK simply to get preliminary results, and to reduce the file size.
