php로 하둡 다루기 : Using Hadoop And PHP

Using Hadoop And PHP

42 Replies

Getting Started

So first things first.  If you haven’t used Hadoop before you’ll first need to download a Hadoop release and make sure you have Java and PHP installed.  To download Hadoop head over to:

http://hadoop.apache.org/common/releases.html

Click on download a release and choose a mirror.  I suggest choosing the most recent stable release.  Once you’ve downloaded Hadoop, unzip it.

user@computer:$ tar xpf hadoop-0.20.2.tar.gz

I like to create a symlink to the hadoop-<release> directory to make things easier to manage.

user@computer:$ link -s hadoop-0.20.2 hadoop

Now you should have everything you need to start creating a Hadoop PHP job.

Creating The Job

For this example I’m going to create a simple Map/Reduce job for Hadoop.  Let’s start by understanding what we want to happen.

경축! 아무것도 안하여 에스천사게임즈가 새로운 모습으로 재오픈 하였습니다.
어린이용이며, 설치가 필요없는 브라우저 게임입니다.
https://s1004games.com

  1. We want to read from an input system – this is our mapper
  2. We want to do something with what we’ve mapped – this is our reducer

At the root of your development directory, let’s create another directory called script.  This is where we’ll store our PHP mapper and reducer files.

user@computer:$ ls

.
..
hadoop-0.20.2
hadoop-0.20.2.tar.gz
hadoop

user@computer:$ mkdir script

Now let’s being creating our mapper script in PHP.  Go ahead and create a PHP file called mapper.php under the script directory.

user@computer:$ touch script/mapper.php

Now let’s look at the basic structure of a PHP mapper.

Code  
#!/usr/bin/php
<?php
//this can be anything from reading input from files, to retrieving database content, soap calls, etc.
//for this example I'm going to create a simple php associative array.
$a = array(
'first_name' => 'Hello',
'last_name' => 'World'
);
//it's important to note that anything you send to STDOUT will be written to the output specified by the mapper.
//it's also important to note, do not forget to end all output to STDOUT with a PHP_EOL, this will save you a lot of pain.
echo serialize($a), PHP_EOL;
?>

So this example is extremely simple.  Create a simple associative array and serialize it.  Now onto the reducer.  Create a PHP file in the script directory called reducer.php.

user@computer:$ touch script/reducer.php

Now let’s take a look at the layout of a reducer.

Code  
#!/usr/bin/php
 
<?php
 
//Remember when I said anything put out through STDOUT in our mapper would go to the reducer.
//Well, now we read from the STDIN to get the result of our mapper.
//iterate all lines of output from our mapper
while (($line = fgets(STDIN)) !== false) {
    //remove leading and trailing whitespace, just in case 
본 웹사이트는 광고를 포함하고 있습니다.
광고 클릭에서 발생하는 수익금은 모두 웹사이트 서버의 유지 및 관리, 그리고 기술 콘텐츠 향상을 위해 쓰여집니다.
번호 제목 글쓴이 날짜 조회 수
88 [PHP] PhpStorm IDE 사용하기 file 졸리운_곰 2018.07.19 119
87 phpStorm 원격 서버 연결 및 배포 (Deployment) file 졸리운_곰 2018.07.18 152
86 php 세션 저장소를 redis 로 바꾸어 본 후기. file 졸리운_곰 2018.05.05 413
85 아이오닉과 php 그리고 mysql 연동 : Using PHP and MySQL with Ionic file 졸리운_곰 2018.01.14 340
84 CKEditor 4 설치와 PHP 연동 하기 file 졸리운_곰 2018.01.14 234
83 5가지 조미료같은 워드프레스 플러그인 file 졸리운_곰 2018.01.14 52
82 wordpress 한시간만에 만들기 졸리운_곰 2017.10.26 63
81 PHP 프레임 새로고침할 때 페이지 유지하기 입니다. 졸리운_곰 2017.09.17 365
80 How to install composer on Wamp file 졸리운_곰 2017.09.10 104
79 Install Composer on Windows and WAMP Server file 졸리운_곰 2017.09.10 46
» php로 하둡 다루기 : Using Hadoop And PHP 졸리운_곰 2017.08.03 213
77 php로 빅데이터 다루기 : BIG DATA IN PHP file 졸리운_곰 2017.08.03 47
76 PHP-ML - Machine Learning library for PHP 머신러닝 file 가을의곰 2017.06.18 267
75 xampp+php+mssql file 가을의곰 2017.06.10 142
74 Build An Automated PHP Gallery System In Minutes file 가을의곰 2017.06.10 181
73 xampp+php+mssql file 가을의곰 2017.06.04 97
72 PHP - AJAX and PHP file 졸리운_곰 2017.05.06 47
71 PhpStorm 으로 라라벨 개발 하기 졸리운_곰 2017.05.05 408
70 WordPress Development using PhpStorm 졸리운_곰 2017.05.05 456
69 [함수] SQLite 와 php 의 연동 졸리운_곰 2017.04.26 57
대표 김성준 주소 : 경기 용인 분당수지 U타워 등록번호 : 142-07-27414
통신판매업 신고 : 제2012-용인수지-0185호 출판업 신고 : 수지구청 제 123호 개인정보보호최고책임자 : 김성준 sjkim70@stechstar.com
대표전화 : 010-4589-2193 [fax] 02-6280-1294 COPYRIGHT(C) stechstar.com ALL RIGHTS RESERVED