1 .背景

flume是由cloudera软件公司产出的可分布式日志收集系统,后与2009年被捐赠了apache软件基金会,为hadoop相关组件之一。尤其近几年随着flume的不断被完善以及升级版本的逐一推出,特别是flume-ng;同时flume内部的各种组件不断丰富,用户在开发的过程中使用的便利性得到很大的改善,现已成为apache top项目之一.

 

2 .概述

1.  什么是flume?

apache Flume 是一个从可以收集例如日志,事件等数据资源,并将这些数量庞大的数据从各项数据资源中集中起来存储的工具/服务,或者数集中机制。flume具有高可用,分布式,配置工具,其设计的原理也是基于将数据流,如日志数据从各种网站服务器上汇集起来存储到HDFS,HBase等集中存储器中。其结构如下图所示:

2.应用场景

比如我们在做一个电子商务网站,然后我们想从消费用户中访问点特定的节点区域来分析消费者的行为或者购买意图. 这样我们就可以更加快速的将他想要的推送到界面上,实现这一点,我们需要将获取到的她访问的页面以及点击的产品数据等日志数据信息收集并移交给Hadoop平台上去分析.而Flume正是帮我们做到这一点。现在流行的内容推送,比如广告定点投放以及新闻私人定制也是基于次,不过不一定是使用FLume,毕竟优秀的产品很多,比如facebook的Scribe,还有Apache新出的另一个明星项目chukwa,还有淘宝Time Tunnel。
3.Flume的优势

1.  Flume可以将应用产生的数据存储到任何集中存储器中,比如HDFS,HBase

2.  当收集数据的速度超过将写入数据的时候,也就是当收集信息遇到峰值时,这时候收集的信息非常大,甚至超过了系统的写入数据能力,这时候,Flume会在数据生产者和数据收容器间做出调整,保证其能够在两者之间提供一共平稳的数据.

3.   提供上下文路由特征

4.   Flume的管道是基于事务,保证了数据在传送和接收时的一致性.

5.   Flume是可靠的,容错性高的,可升级的,易管理的,并且可定制的。

4. Flume具有的特征:

1. Flume可以高效率的将多个网站服务器中收集的日志信息存入HDFS/HBase中

2. 使用Flume,我们可以将从多个服务器中获取的数据迅速的移交给Hadoop中

3. 除了日志信息,Flume同时也可以用来接入收集规模宏大的社交网络节点事件数据,比如facebook,twitter,电商网站如亚马逊,flipkart等

4. 支持各种接入资源数据的类型以及接出数据类型

5. 支持多路径流量,多管道接入流量,多管道接出流量,上下文路由等

6. 可以被水平扩展

 

 3. Flume的结构

1. flume的外部结构:

 

如上图所示,数据发生器(如:facebook,twitter)产生的数据被被单个的运行在数据发生器所在服务器上的agent所收集,之后数据收容器从各个agent上汇集数据并将采集到的数据存入到HDFS或者HBase中

2. Flume 事件

事件作为Flume内部数据传输的最基本单元.它是由一个转载数据的字节数组(该数据组是从数据源接入点传入,并传输给传输器,也就是HDFS/HBase)和一个可选头部构成.

典型的Flume 事件如下面结构所示:

我们在将event在私人定制插件时比如:flume-hbase-sink插件是,获取的就是event然后对其解析,并依据情况做过滤等,然后在传输给HBase或者HDFS.

3.Flume Agent

我们在了解了Flume的外部结构之后,知道了Flume内部有一个或者多个Agent,然而对于每一个Agent来说,它就是一共独立的守护进程(JVM),它从客户端哪儿接收收集,或者从其他的 Agent哪儿接收,然后迅速的将获取的数据传给下一个目的节点sink,或者agent. 如下图所示flume的基本模型

Agent主要由:source,channel,sink三个组件组成.

 

Source:

从数据发生器接收数据,并将接收的数据以Flume的event格式传递给一个或者多个通道channal,Flume提供多种数据接收的方式,比如Avro,Thrift,twitter1%等

Channel:

channal是一种短暂的存储容器,它将从source处接收到的event格式的数据缓存起来,直到它们被sinks消费掉,它在source和sink间起着一共桥梁的作用,channal是一个完整的事务,这一点保证了数据在收发的时候的一致性. 并且它可以和任意数量的source和sink链接. 支持的类型有: JDBC channel , File System channel , Memort channel等.

sink:

sink将数据存储到集中存储器比如Hbase和HDFS,它从channals消费数据(events)并将其传递给目标地. 目标地可能是另一个sink,也可能HDFS,HBase.

 

它的组合形式举例:

 

以上介绍的flume的主要组件,下面介绍一下Flume插件:

1. Interceptors拦截器

用于source和channel之间,用来更改或者检查Flume的events数据

2. 管道选择器 channels Selectors

在多管道是被用来选择使用那一条管道来传递数据(events). 管道选择器又分为如下两种:

默认管道选择器:  每一个管道传递的都是相同的events

多路复用通道选择器:  依据每一个event的头部header的地址选择管道.

3.sink线程

   用于激活被选择的sinks群中特定的sink,用于负载均衡.

 

—————————————————-<完>——————————————————-

参考资料:
http://www.tutorialspoint.com/apache_flume/apache_flume_data_flow.htm
 作者: gongxijun
 请注明转载自 http://www.cnblogs.com/gongxijun/p/5656778.html

57条回应:“flume介绍及原理”

  1. bape bape camo down vest size us m eu 48 50

    top mens bape hoodie bronzing monkey warm down coat a bathing ape jacket ebaypuma women pink running shorts multicolor 1429270 for women online india on myntranike district 72 fz hoody 115 liked on polyvore featuring tops hoodies jackets outerwear shir…

  2. pinkqueen new menwomen 3d sweatshirts with hat print spit tongue dog hooded hoodies

    classic for men lacoste blue track pants guppy appealingbelstaff kids junior roadmaster in red signature 6oz waxed cottongown collection wine red elegant happiness off shoulder gown womens clothes dresses on carousellmens lingerie existshere are the ns…

  3. 猫 possibile avere vestiti per bambini originali

    scarpe uomo 1121578 red silver street scarpe bordeaux eleganti traforatescarpe di tendenza trussardi jeans decollete a punta nero donnadettagli su donna vans sk8 alte camoscio nero mono scarpe da ginnasticadolce e gabbana collezione autunno inverno 201…

  4. bicycle bell wikipedia

    mens minnesota timberwolves 23 jimmy butler navy blue 2017 2018 nike swingman stitched nbajason kidd signed nets jerseynhl hockey jerseys 28 john scott 2016 all star jersey mens hockey jersey black white with c patch top quality free shippingesa tikkan…

  5. patagonia switchback bra review

    freya fancies bras underwire balcony k cup ladies 1012 black womens various newjibri off shoulder sheer slouch poolside dress jibrishopolica womens padded tube bra design striped strechable seamlessdelta burke pink splice city plus size high neck tank…

  6. snakes on a plane 2006 crew cap

    original penguin caps original penguin flat cap alaskan bluenew era densuede washington redskins fitted cap for men blue planet sportsblack prolific embroidered baseball capwholesale custom embroidered snapbacks for sale logo custom embroidered hats ba…

  7. rel贸gio masculino champion anal贸gico cn20042p couro

    generic women push up padded bra beach bikini set swimsuit swimwearsale sorbet collection one piece seamless high cut swimsuit plain peach melbaanimal ring tankini topbowknot coral bustier bra high waist cut out bikini swimsuit 3

  8. appealing air jordan 12 nike air jordan retro xii 12 gym

    air jordan 22 5 8 air jordan 316381 141 goatcheap nike air max online air jordan horizon wolf grey dynamicnike react element 87. release date august 14 18usrelease date august 16 18global160. color desert sand cool grey smokey mauveparis chaussure de b…

  9. professionale hogan rosa ballerine e mocassini taglia 37 donna mocassini 36.5

    abito uomo celestetimberland barca da uomo scarpe uomo barcanero scarpe stonefly stivaletto 103099 calzature fantasiasconto quechua scarpe bambino arpenaz 50 gialle giallo paglierino montagna taglia s

  10. winter fleece 2018 pro team yellow lotto jumbo cycling jersey warmer bike cloth jacket mtb ropa ciclismo bicycle maillot ony

    winter ready kit from hoy vulpine unveiled total woakland raiders nfl pro line by fanatics branded timeless collection vintage arch tri blend raglannew era dallas cowboys nfl 2012 kids classics 59fifty cap navy hats fittedred blue polo t shirt sport de…

  11. scarpe le coq sportif uomo saint ferdinand blu vendita online qeruhrhgs uomo sneakers e sportive

    italia fantasia bellissime adidas superstar donna fiori prezzi bassi venditablack friday 2016 negozio nike uomo air max 2015 nere bianche astaacquista asics gel nimbus 19 donna grigio off37 scontiitalia criminal damage scarpe da ginnastica alte pitonat…

  12. ost茅opathe paris ost茅o charenton porter des talons sans se faire mal

    mulberry shoulder bag bordeaux leatherquechua kids outdoor travel backpack for hikingruggard outrigger 65 dslr backpack blackarmani handbags sale uk map

  13. zebra converse high tops

    beautiful sandals heels designsbuy john lewis danelle sparkle occasion sandalsvans old skool pro shoes black black white modern skate surf9 composite safety toe waterproof insulated logger

  14. giacca da uomo in tweed con giacca sportiva da giacca vintage retr貌 avoro s im fit da

    giubbotto is and am antinfortunistica s.a.s di agostoni dauroe eganti bianco b u scarpe diadora zj19001166 game ow waxed uomo on ine mnabbig iamento donna on ine acquista vestiti su mec shoppingacquista 2 off qua siasi adidas fe pe femmini i case e ott…

  15. nike womens air zoom pegasus 35 running shoe

    nike shox rivalry black black white 316317 017 purchazecap and gown 11 prom night platinum tint 11s xi gym red bred prmaustralia converse chuck taylor all star fulton leather white mens shoesnike air max 90 originales. original

  16. dames schoenen sale online zwarte laarzen sportschoenen outlet

    nike air max thea dames ren schoen rosa wit online kopen in dedr martens dr martens dames veterlaarsjes 1460 vintage bordeaux dr martens kopen in de aanbiedingadidas sandalen adidas schoenen online kopen tot 50 kortingpuma suede classic grijs heren 363…

  17. carros supra carros

    skechers s ights itebeamsan莽amento adidas busenitz kev ar artwa kt锚nis dafiti shoes cano curto z铆per urex preto compre agora kanui brasibo sa chane cocoon origina . carregando

  18. converse all star fuzzy felt fabric..satin liningsize 7 wos

    new deals on plus size marled balloon sleeve topgirls performance sheer twinkle mesh halter bra top style no tw300. loading zoommarks spencer trellis lace non wired braever pretty double v neck black evening party long dress 09983 size

  19. 2018 hippie metal lennon oval round sunglasses women metal frame circle oval tinted lens sunglasses hippie chic style cheap prescription sunglasses

    sleeveless turtleneck thong bodysuit by commandowomens ladies girls urban beach berry aqua shoes water socks sandals fw585 beach swim sea surf size uk 3 8 pinksexy backless low cut print boho chiffon dressesaibiner camo vintage mens camouflage t shirt…

  20. havaianas slim swarovski crystal white havaianas for the family from 10 onceit

    clovia 12 piece maroon nightwear set2018 new fshion spring autumn baby girls boys clothes suits cotton zipper jacket pants children casualred green sweater dress infant toddlerwomen lycra long sleeve leotard for girls gymnastics dance leotard turtlenec…

  21. 3 of 8 zara striped denim jumpsuit playsuit romper strapless sz s l nwt

    its all good life is . . motorcyclearzberg kitchen friends white freshness box gift boxed lid rednew england patriots lil fan bannerevh 5150 iii el34 micro stack guitar amp sound 4 less guitar outlet

  22. zwiewna sukienka z fa banami pretty gir moda damska

    reserved sk贸rzane botki na szpi ce czarny botkidzianinowa b uzka z bia艂ym ko艂nierzykiem czarna cudmodabuciki buty nike niemow ece 19 superstar orgina ne 7299305566buty damskie zimowe adidas hyperhiker s80827 7086885784 a egro p

  23. camisetas mascu inas emporio armani encontre no ounge a

    armband silber 925 cord weiss damen schmuck shop goldherren camouflage sport shortsworldclassca damen camouflage sweatshirt tarn kapuzen mit seitlichen streifen army t shirt leichte sommer hoodie langarmshirt oberteil pulli fitness bloggerneue trendy t…

  24. prada flip flops metallic logo flipflops

    vans mens off the wall logo pullover hooded sweatshirt dress bluevans womens shoes famous footwear nz discountwillow embellished gladiator sandals in silversorel grey wedge sandal from british columbia by big boot inn

  25. du milde kjole wauv viola . bordeaux r酶d kjole med flot m酶nster

    hurtigf忙rgen express 3 her har sammen med molslinjens andre f忙rge store problemer med sejlere i aarhusbugtensams酶e sams酶e rullekrave str. m bl氓lingeri og undert酶j online dixie pyjamas skjorte nat skjorte fraeksempler p氓 projekter i projektet virksomhed…

  26. t锚nis fila sense feminino ofertas promobit

    sapatilhas vans baratas online comprar sapatilhas online portugalagasalho adidas young knit graf femininolote de 4 playeras tipo polo varias marcas envi贸 gratis 1sapato ferracini 4360 223r preto comprar na loja online

  27. touca e luva azul marinho pronta entrega gorro la unissex

    luckyjewelry urn necklaces for ashes cremation keepsake memorial men pendant necklacediamond wedding bands 1 4 ct tw round cut 14k rose gold 80526128 kayknit sheath dresscheap michael michael kors brooklyn large leather shoulder bag black for women onl…

  28. cazal sunglasses 873 col. 722 havana gold frame brown

    fendi tops for women farfetchdita accessories dita newchums fish tip rope retainer chumsamazon uk reading sunglasses david simchi levi

  29. nouveau style moncler gui gilet sans manche homme vert store

    gucci short en jean d茅lav茅 effiloch茅 femme denim shortsceinture tissu hommetc accessories blanc noir paisley tube snood 茅charpe motard de ski sports de plein airfemme dior arc en ciel soie foulard 5185376 5185376

  30. no 56703 maior loja online de r茅plicas de malas sapatos

    odense h氓ndbold ans忙tter to assistenttr忙nerecrocs sko crocs crocband flip navyfred perry leather sko vit k枚p i dag hos stylepittommy hilfiger m忙nd str酶mper sokker undert酶j og str酶mper 2 pack t sock

  31. new balance sneaker junior 500 scarpe bambini bordeaux blu

    negozio di scarpe puma nuovo puma fenty di rihanna creepers suede di farina davena scarpe uomo tuconverse chuck taylor all star shoes men white monochromecaricamento dellimmagine in corso reebok classic leather lux tela olympic creme goldadidas scarpe…

  32. valentina silvia lago comfortable designer shoes

    zapatos para mujer hogan sandalias de marca ofertamuy barato blancolas mejores nike air force 1 07 zapatillas blanco negromai pi霉 senza sandalias mujer zapatos con hebilla rose baratas madrid

  33. 贸culos de sol ray ban preto lente verde polarizada haste de tit芒nio ray ban

    anel de noivado solit谩rio tiffany co 0.87 quilateschilli beans traz cole莽茫o em parceria com a swarovski juliana arizacomprar. rel贸gio mormaii monitor card铆aco贸culos de grau ray ban round fleck rb2447 tartaruga moda arma莽玫es

  34. Jack & Jones PREMIUM JPRNORSON SHAWL Cardigan olive night asphat Monocromo Rinforzo Bottoni Collo a scialle JAM22Q00D N11 PAPPFZU

    BOSS ATHLEISURE ZUREX Cardigan charcoal Melange Maglia Bottoni BG122Q020 Q11 UOWATJV

  35. caleche dhermes pas cher

    ceinture hermes prix femme fleurschapeau b茅ret bonnet casquette carreaux noir pour enfant b茅b茅 fille gar莽ongucci by gucci pour homme eau de toilette de gucci sephoraecharpe babylonia tricot slen organic mild taupe

  36. lyngby er et indk酶bsmekka for folk

    tre danske m氓lm忙nd holdt nullet bt fodbold http://www.bt.dksort chiffon boleo med dyp bolero festt酶j i store st酶rrelsers氓dan v忙lger du det bedste booking site til flybilletter alt ompuma b酶rne sko odense m puma b酶rne sko

  37. sandali con tacco alto fratelli rossetti

    lo scrigno sposi ha anche allestito un reparto dedicato agli abiti da sposo che offre abiti da cerimonia uomo ed accessori in vari stili ed adatti agliarticolo sneakers geox in pelle camoscio multicolore beige con zeppa internaliu jo man fantasia scozz…

  38. goedkoop adidas originals superstar grijs roze casual schoenen meisjes lagere school verkooppunt

    t锚nis couro bull terrier aton low branco compre agora dafiti brasilsand谩lia feminino rasteirinha via scarpa dourado zattini moda eall star converse branco original r 73n贸s estivemos presentes para registrar os melhores momentos da reinaugura莽茫o da loja…

  39. grey converse shoes sneakers for men

    nike foamposite one olympic release dates july 2016jeremy scott adidas wings 2.0 pixels. worn a handful of no depop100 original authentic converse chuck taylor all star high street sneakers mens style 92532 black continuously whiteconverse chuck taylor…

  40. lunettes de vue vuillet vega prestige 1471 taille lunettes de vue vuillet vega prestige 1471 taille 52

    lunettes de soleil oakley homme tincan carbon oo6017 601707 montureray ban aviator large m茅tal 003 32 55014 2n vintedlunettes enfant julbo reach l j466 noir nouvelle collectiona loccasion du 30e anniversaire de la mort de lacteur

  41. the scent by hugo boss for men eau de toilette

    under armour hi vis hooded sweatshirtfor love lemons vienna mini dress nordstrom rackthe feel good wedding blog polka dot bridediadora team racer mountain bike shoe mens

  42. portable travel home use compact ceramic hair blower 1100 w dyson supersonic hair dryer styling tools hot air brush for salon

    one blowout drybar naperville24 piece makeup brush set with bag buy online at best prices in pakistanaloe vera gel 99 aloe vera fresca di fuerteventura 2bas you may know i recently wrote a cleanser roundup of my favourite cleansing products and facial…

  43. ga hier naar de new balance webshop. plaatsen met new balance verkooppunten. 1 new balance in s gravendeel. 7 new balance in alkmaar

    populair heren schoenen asics gel kinsei 6 hardloop blauw maat eu 45 us 11trendy schoenen shoecolate 65261252 lage sneakers goud dames goede ondersteuningfootjoy dames brushed chill out vest zwartnike air zoom pegasus 35 running shoes women grey

  44. adidas pale pink trefoil tee dress

    football boots for sale in the maldives capitalinc international concepts i.n.c. mens dark floral print shirtjust my size womens plus size cooldri tank at amazon women snike air force 1 low men gray spot white blue

  45. printed blouse and border yellow ruffle bridesmaid saree

    vs pink victorias secret pink anorak windbreaker jacket full zip black white xsmall small at amazon womens clothing storegal meets glam dresses skirts gal meets glam dress marion blue satin twist sleevvictorias secret bra set garter stockings and 50 si…

  46. jimmychoo 69 on sunglasses i love pinterest sunglasses

    nike academy dri fit training pantsnwt antisocial social club salmon foreshadow embroidered hoodiefirewall fr csa hi viz 9 oz insulated overallsadidas originals three stripe track top

  47. top 10 best hair dryer in india updated 2019

    used instyler ionic styler pro ionic hot brush and ceramic flat ironwahl lithium ion clipper 79600 2101 review battery spot welding2 wahl body groomer 95028 shaver foil heads covers lithium ion 9884wahl professional hair dryer

  48. torba gumowa shopper jelly bag

    nerka czy saszetkatorebka kopert贸wka be偶 wieczorowa listowa kremowa zdj臋cie na imgedtommy hilfiger crossbody sports saszetka nerka blueeleganckie p贸艂buty jazz贸wki 12 1 czarny

  49. pour la victoire womens yvette dress sandal

    adidas copa 17.1 kith ultraboost sneakers price in saudi arabiamens reebok club c85 trainers sand stone gum sg temp forcenike wmns air foamposite one shine women lifestyle new dark stuccostylish reebok classic slide sandals mens in black outlet canada

  50. t锚nis fila vertex sl t锚nis esportivo no

    t锚nis nike star runner infantil azul marinho preto nogris para falda escocesa highland falda tradicional e para hacer vestidosadidas originals sport bh adidas originalsasics marrom gel lyte creme

  51. sal贸n negro tac贸n medio de pitillos

    tamaris textil velours pumps leder blau hell klassische pumps damenvon der sonne bis zum pluto in 3designer socken kleider jumpsuits g眉nstig overalls amp kaufen damenadidas schuhe air max 97

  52. ugg ansley slippers nightfall with blue shearling lining womens size us 7 nice fashion

    custom painted shoes vans suicide squad joker harley etsydaniel van buy bayern munich v chelsea fc allianz arena munich germany 19 may 2012vans boom boom sk8 hi slim trainers with zip back in burgandy burgundygolfilicious golf wang snapback hat vintage…

  53. gucci marmont matelasse leather belt bag

    gucci bengal slip onsspecial prices on gucci eyewear oversized sunglasses blackgucci dionysus embroidered leather shoulder bag lystgucci ace dragon sneakers

  54. tommy hilfiger sac seau color block multi

    chaussure homme boutique en ligne boss green by hugo boss arkansas formateurs noirgendarme concours sous officier de gendarmerie pr茅pa concours gendarmeadidas marque hot brun pour femme magasin en ligneveste napapijri rainforest hiver 2016 mainapps v锚t…

  55. oem glow in the dark underwear thin cup light bra and g string set 2016 factory new design

    gucci heart frame sunglassesrayban rb2132 grand optique du bassinversace sunglasses ve2175 metal mesh 100287drivewear sunglasses arrive at company news

  56. uv sun visor hat golf hat hiking helmet cap transparent color sun

    2014 new hot sell maxi multiway wrap convertible sheen shiny champagne infinity dress plus size party dresses bridesmaid dresses in bridesmaid dresses frombeautiful lace appliqued sexy high neck long sleeve princess bride dresses long vestidos novia br…

  57. womens light pink nikes glitterkick

    womens buy leather bags onlinewomens waterproof winter hiking bootsnike free run menswomens bright nike leggings