Using Pig with Scientific Applications and IndexedHBase for ...

Using Pig with Scientific Applications and IndexedHBase for ...

Using Pig with Scientific Applications and IndexedHBase for Social Media Data Queires Tak-Lon (Stephen) Wu Motivation Provide Matlab-like UDF libraries in Pig for data scientists, save reimplementation time for same algorithm, e.g. Kmeans Clustering Investigate the benefits of using Pig Performance, Lines of code, Reusability, DataFlow control Integrate Pig with different execution engines to meet different needs, such as iterative applications and streaming queries. working on Harp Integration to support iterative application in a single pig script. Pig with Apache Storm is one of the proposed Pig projects in open source community. Pig Kmeans (Single Iteration) 1 REGISTER pig-kmeans-udf-yarn.jar; 2 raw = LOAD hdfs://KmeansInput using PigKmeans('$centroids', 50000) AS (datapoints); 3 datapointsbag = FOREACH raw GENERATE FLATTEN(datapoints) AS datapointInString:chararray; 4 datapoints = FOREACH datapointsbag GENERATE STRSPLIT(datapointInString, ',', 5) AS splitedDP; 5 grouped = GROUP datapoints by splitedDP.$0; 6 newCentroids = FOREACH grouped GENERATE CalculateNewCentroids($1); 7 STORE newCentroids INTO output'; Total execution time (s)

Execution time (s) Performance Pig with Yarn Stage execution time vs. mapper number 100m, 500 10m, 5000 1m, 50000 Different Kmeans Implementation Total execution time vs. mapper number 2000 1800 1600 2000 1400 1200 1500 1000 1000 800 600 500

400 0 200 24 48 96 24 48 96 24 48 0 24 number of mappers Job startup Coll. Comm. + Reduce Centroids broadcast Job cleanup

Data points loading Centroids merge 96 Map computation 48 96 number of mappers Hadoop 100m,500 Harp 100m,500 Pig HD1 100m,500 Pig Yarn 100m,500 Hadoop 10m,5000 Harp 10m,5000 Pig HD1 10m,5000 Pig Yarn 10m,5000 Hadoop 1m,50000 Harp 1m,50000 Pig HD1 1m,50000 Pig Yarn 1m,50000 Lines of Code Java Pig Python / Bash

Total Lines Pig Kmeans Hadoop Kmeans ~345 10 ~40 395 780 0 0 780 Pig IndexedHBase meme-cooccurcount 152 10 0 162 IndexedHBase meme-cooccurcount ~434 0 28 462

Recently Viewed Presentations

  • 7 DEC 2016 BELL ACTIVITY: Read the article

    7 DEC 2016 BELL ACTIVITY: Read the article

    Character vs. self. Character vs. character. Character vs. society. Character vs. nature. Character vs. Fate. Theme …. the central idea about life or human behavior that the drama reveals. CHARACTERS. CHARACTERS ARE THE INDIVIDUALS OR FORCES THAT ARE IN THE...
  • Industrial Regulation and Quality CHMG 751

    Industrial Regulation and Quality CHMG 751

    Inventory time: In-Sort Clerk Inventory = 80 pc's LT = WIP/Production Rate Therefore, 80/3 = 26.7 min LT for inventory assumes FIFO Piece can be batch, piece, part, drum, bin, etc… All pieces should be in same unit through VSM....
  • Group A - AHD Dr. Gary Greenberg

    Group A - AHD Dr. Gary Greenberg

    Group A - AHD Dr. Gary Greenberg ... Name important risk factors for infection in the C-spine. Cervical Radiculopathy Describe the incidence , Reflex, Sensory and motor loss for the following levels involved: C5 radiculopathy. C6 radiculopathy C7 radiculopathy. C8...
  • 1. Be sure you are sitting with the

    1. Be sure you are sitting with the

    There were times when the Powhatan Indian trade revived the colony with food in exchange for glass beads, copper, and iron tools. The Algonquians (1622) attacked the plantations killing over 300 of the settlers. ... The charter was revoked in...
  • Accepted Student Checklist  Activate your NetID  Emergency Contact

    Accepted Student Checklist Activate your NetID Emergency Contact

    Parking Permit Applications are available through SU's MySlice student services. Some P22 Lot passes are available for students who are not on funding- check with UPD. No regular ESF "on-campus" parking available for students on weekdays before 5 p.m. or...
  • Climate of Costa Rica Eugene S. Takle Professor

    Climate of Costa Rica Eugene S. Takle Professor

    Climate of Costa Rica Eugene S. Takle ... Times MS Pゴシック Arial Calibri Verdana Times New Roman Blank Presentation Climate of Costa Rica Overview PowerPoint Presentation PowerPoint Presentation PowerPoint Presentation PowerPoint Presentation PowerPoint Presentation PowerPoint Presentation ...
  • Welcome to Handsworth Course Selection 2016/2017 Students entering

    Welcome to Handsworth Course Selection 2016/2017 Students entering

    Welcome to Handsworth Course Selection 2016/2017 Students entering Grade 12
  • Bild 1 - jonaskemi.weebly.com

    Bild 1 - jonaskemi.weebly.com

    2. Orsaker Växthuseffekten Växthuseffekten Naturliga och onaturliga klimatvariationer Människans påverkan Människans påverkan Koldioxidutsläppen ökar Sveriges koldioxidutsläpp Olika växthusgaser Växthusgaserna isolerar jorden Utan växthuseffekt - minus 18 grader kallt En ökning av växthusgaser skapar obalans Bild 2.1.1 - Växthuseffekten Bild: Animerad...