[大数据]Flume-自定义Sink

自定义Sink

Sink不断地轮询Channel中的事件切批量地移除他们，并将这些事件批量写入到存储或索引系统、或被发送到另一个Flume Agent；

Sink是完全事务性的，从Channel批量删除数据之前，每个Sink用Channel启动一个事务，批量事件一旦成功写出到Sink的目的地，Sink就利用Channle提交事务，事务一旦被提交，该Channel就从自己内部的缓冲区讲相应的事件删除。

本次自定义组合为：netcat source + 自定义Sink

java代码如下：

public class MySink extends AbstractSink implements Configurable {

    // 定义两个属性，前后缀

    private Logger log = LoggerFactory.getLogger(MySink.class);

    private String prefix;
    private String suffix;

    public void configure (Context context) {

        prefix = context.getString("prefix");
        suffix = context.getString("suffix", "hello-word");
    }

    /**
     * 1、获取Channel
     * 2、从Channel获取事务以及数据
     * 3、发送数据
     */
    public Status process () throws EventDeliveryException {

        Status status = null;

        // 1、获取Channel
        Channel channel = getChannel();

        // 2、从Channel获取事务并开启
        Transaction transaction = channel.getTransaction();
        transaction.begin();

        try {
            // 3、从Channel获取数据
            Event event = channel.take();

            if (event != null) {
                // 4、处理事件
                String body = new String(event.getBody());
                log.info("{}--{}--{}", prefix, body, suffix);
            }


            // 5、提交事务
            transaction.commit();

            status = Status.READY;
        } catch (ChannelException e) {
            e.printStackTrace();
            // 提交事务失败
            transaction.rollback();

            status = Status.BACKOFF;
        } finally {
            transaction.close();
        }

        return status;
    }
}

配置文件如下：

# Name the components on this agent
a1.sources = r1
a1.sinks = k1
a1.channels = c1

# Describe/configure the source
a1.sources.r1.type = netcat
a1.sources.r1.bind = localhost
a1.sources.r1.port = 44444

# Describe the sink
a1.sinks.k1.type = com.starnet.sink.MySink
a1.sinks.k1.prefix = nihao
a1.sinks.k1.suffix = hello

# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100

# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1

运行起来之后，依次在44444端口写入nihao、wohao、dajiahao，结果如下：