Flume or Kafka? Try both!

In the last few month, I’ve spent a lot of time answering one question: Flume or Kafka?

Its a pretty tough choice.
Flume has been around for a while, there are many users and happy production systems, its well supported by major Hadoop vendors, there are well known best practices on how to configure and tune it and its integration with both Hadoop and common data sources is unparalleled. If you are not a developer, its easy to configure a good data pipeline with Flume without writing a single line of code.

Kafka is slightly newer, but it offers impressive latency and throughput, scalability and high availability. Its a very well designed system. In addition, Kafka makes it easy pull data out of it into variety of systems – batch, streaming, applications, dashboards and even the command line.

In many cases the decision comes down to specific priorities:
Do you prefer possibility of integration with many data targets? Or an easy way to send data to Hadoop?
Do you prefer the flexibility of writing your own data producers or consumers? Or do you prefer a configuration-only solution without any programming involved?
Is it important for you to control every aspect of performance, availability and scalability? Or do you want a system that is mostly well-tuned out of the box, but may not squeeze every bit of possible performance?

After few of those discussions, we realized that this is silly. Why do we need to choose between a system that is great for developers and a system that is great for administrators? Why choose between fantastic scalability and flexibility and a great collection of data integration sources and sinks?

Why can’t we have it all?

This is why we helped write Flafka (unofficial code name) – a set of Kafka source and sink for Flume.

Flafka serves two important use-cases:

1. You have Kafka. You have apps pushing data into Kafka. But you need to get the data to Hadoop. Not just HDFS. You need HBase and Solr too. Maybe you even need to tweak the data a bit on the way – mask sensitive data for example or flag suspicious events.  Use Kafka-source with HDFS, HBase or Solr sinks. And add an interceptor to massage the data for you.

2. You need to land data from HTTP, log files, syslog or vmstat to Kafka. You don’t want to write your own producer and rather use existing tried-and-true solution. Use a Flume source with a Kafka sink to land data in Kafka without having to figure out how to write a producer.

경축! 아무것도 안하여 에스천사게임즈가 새로운 모습으로 재오픈 하였습니다.
어린이용이며, 설치가 필요없는 브라우저 게임입니다.
https://s1004games.com

Since Flume is configuration-only, Flafka allows non-developers to easily use Kafka and to integrate Kafka with Hadoop.
But we didn’t want to sacrifice flexibility for simplicity, so the Kafka source will support any configuration that a Kafka consumer will accept, and Kafka sink can be configured just like any producer. Out of the box, we tuned Flafka for reliability rather than pure speed, reasoning that preventing data loss is our most important mission. If you have different trade-offs and preferences, it is easy to test them with Flafka.

Like all other Flume sources and sink, Flafka is designed to work in batches – you can configure batch sizes and the Kafka source will write data in batches to the Flume channel, and the Kafka sink will read data in batches from the channel and send them to Kafka. This design will increase throughput and improve resource utilization, but will also increase latency. You can tune the balance between throughput and latency to match your requirements by experimenting with different batch sizes.

You can even configure multiple Kafka sources to read from the same topic. If you configure all of the with the same groupId, each source will read a different set of partitions, which will lead to improved throughput. This is true even if the sources are configured on different agents. If one of the agents crashes, the remaining agents will automatically re-balance the partitions between them, which reduces delays and data loss.

At the moment, there is no Flume release with Flafka, so you will need to build Flafka from Flume’s source repository and build it:

git clone https://github.com/apache/flume.git flume-local
cd flume-local
git checkout trunk
mvn package -DskipTests

After building, simply copy the jars from flume-ng-dist/target/apache-flume-1.6.0-SNAPSHOT-bin/apache-flume-1.6.0-SNAPSHOT-bin/lib  to /usr/lib/flume-ng/lib and restart your Flume agent.

To get you started, here is an example of how I configure Flume to read data from a Kafka topic and write it to an HDFS directory. The directory will have the same name as the topic, with subdirectories for each day of data:

 tier1.sources  = source1
 tier1.channels = channel1
 tier1.sinks    = sink1
 
 tier1.sources.source1.type = org.apache.flume.source.kafka.KafkaSource
 tier1.sources.source1.zookeeperConnect = shapira-1:2181
 tier1.sources.source1.topic = shapira
 tier1.sources.source1.groupId = flume
 tier1.sources.source1.channels = channel1
 tier1.sources.source1.interceptors = i1
 tier1.sources.source1.interceptors.i1.type = timestamp
 tier1.sources.source1.kafka.consumer.timeout.ms = 100
 
 tier1.channels.channel1.type   = memory
 tier1.channels.channel1.capacity = 10000
 tier1.channels.channel1.transactionCapacity = 1000
 
 
 tier1.sinks.sink1.type         = hdfs
 tier1.sinks.sink1.hdfs.path    = /tmp/shapira/kafka/%{topic}/%y-%m-%d
 tier1.sinks.sink1.hdfs.rollInterval = 5
 tier1.sinks.sink1.hdfs.rollSize = 0
 tier1.sinks.sink1.hdfs.rollCount = 0
 tier1.sinks.sink1.hdfs.fileType = DataStream
 tier1.sinks.sink1.channel      = channel1 

And here’s a Flume configuration for getting data from vmstat to Kafka:

 tier1.sources  = source1
 tier1.channels = channel1
 tier1.sinks    = sink1
 
 tier1.sources.source1.type = exec
 tier1.sources.source1.command = /usr/bin/vmstat 1
 tier1.sources.source1.channels = channel1
 
 tier1.channels.channel1.type   = memory
 tier1.channels.channel1.capacity = 10000
 tier1.channels.channel1.transactionCapacity = 1000
 
 
 tier1.sinks.sink1.type         = org.apache.flume.sink.kafka.KafkaSink
 tier1.sinks.sink1.topic = sink1
 tier1.sinks.sink1.brokerList = kafkagames-1:9092,kafkagames-2:9092
 tier1.sinks.sink1.channel = channel1
 tier1.sinks.sink1.batchSize = 20

I hope you’ll enjoy the ease of use that Flume brings to Kafka. Let us know your experiences in the comments, and open Jira tickets if you run into issues, have ideas for improvements or want to contribute a patch.

[출처] http://ingest.tips/2014/09/26/trying-to-decide-between-flume-and-kafka-try-both/