sql - Mapreduce Table Diff -

-

i have 2 versions (old/new) of database table 100,000,000 records. in files:

trx-old trx-new

the structure is:

id date amount memo 1  5/1     100 slacks 2  5/1      50 wine

id simple primary key, other fields non-key. want generate 3 files:

trx-removed (ids of records present in trx-old not in trx-new) trx-added   (records trx-new ids not present in trx-old) trx-changed (records trx-new non-key values have changed since trx-old)

i need operation every day in short batch window. , actually, need multiple tables , across multiple schemas (generating 3 files each) actual app bit more involved. think example captures crux of problem.

this feels obvious application mapreduce. having never written mapreduce application questions are:

  1. is there emr application this?
  2. is there obvious pig or maybe cascading solution lying about?
  3. is there other open source example close this?

ps saw diff between tables question solutions on there didn't scalable.

pps here little ruby toy demonstrates algorithm: ruby dbdiff

i think easiest write own job, because you'll want use multipleoutputs write 3 separate files single reduce step when typical reducer writes 1 file. you'd need use multipleinputs specify mapper each table.


Comments

Post a Comment

Popular posts from this blog

Perl - how to grep a block of text from a file -

-
it can in xml or text format. how in general grep block of text in perl? <track type="ws"> <range> <rangestart>0</rangestart> <rangeend>146.912</rangeend> <locationindex>0</locationindex> <propertyindex>0</propertyindex> </range> </track> <track type="ps" id="1"> <range> <rangestart>0</rangestart> <rangeend>146.912</rangeend> <locationindex>1</locationindex> <propertyindex>1</propertyindex> </range> </track> i want grep type="ps" , till </range> . one solution open file, read line line , match block. open(fh, "file.txt"); foreach $line (<fh>) { if ($line =~ m/type="cc"(.*?)<\/range>/) { print $1; } } but there more optimal solution without reading file line l
Read more

delphi - How to remove all the grips on a coolbar if I have several coolbands? -

-
Image
if set mycoolbar.fixedorder true,only grip on first band hidden. well,if use delphi 7 create vcl forms application,then put coolbar on , create 3 coolbans hold other controls,only grip , top coolbands can hidden setting mycoolbar.fixedorder:=true. i've uploaded picture make things clear. you got fixedorder property wrong. fixed property not allow bands rearranged if set true . setting property of coolbar true keep user changing bands order @ runtime, user can still move , resize bands. can give advice can do, actual solution problem, well, have wait answer. my advice use 3 coolbars in row , setting "fixedorder" property true , bandborderstyle bsnone . way grip hidden on of them. about property, it's not bug of ide, actual preference of property.
Read more

javascript - Animating array of divs; only the final element is modified -

-
this baffling me @ moment. i'm experienced programmer looking throw bit of surprise fiancee on countdown page our wedding. desired effect simulated confetti falling behind main page. setup follows: i have array of divs in javascript. each div absolutely positioned on page. have interval set updates every 50 or milliseconds. on each tick, run loop on array of divs, , calculations , update positions. problem problem having, however, though can see divs created, stored array, , modified (in case, each has slight randomized rotation) during initialization phase, once tick update position starts, div stored in whatever last slot of array gets position updated. tried hard-coding each div index testing purposes, strictly last element update position. upon printing current "left" , "top" style values each div before , after calculations shows value has been changed, no visible change noticeable. here's basic code: javascript var container;
Read more