python - Scraping a web page as you manually navigate -

-

is there way, using library or method, scrape webpage in real time user navigates manually? scrapers know of such python mechanize create browser object emulates browser - of course not looking since if have browser open, different 1 mechanize creates.

if there no solution, problem want scrape elements html5 game make intelligent agent of sorts. won't go more detail, suspect if others trying same in future (or real time scraping real user), solution useful them well.

thanks in advance!

depending on use-case is, set socks proxy or other form of proxy , configure log traffic, instruct browser use it. you'd scrape log somehow.

similarly, if have control on router, configure capture , logging there, e.g. using tcpdump. wouldn't decrypt encrypted traffic, of course.

if working 1 browser, there may way instruct @ each action via custom browser plugin, i'd have guess you'd running security model issues lot.

the problem html5 game typically of "navigation" done using lot of javascript. javascript typically doing lot -- manipulating dom, triggering requests new content fit dom, etc...

because of might better off looking os-level or browser-level scripting services can "drive" keyboard , mouse events, take screenshots, or possibly take snapshot of current page dom , query it.

you might investigate browser automation , testing frameworks selenium this.


Comments

Post a Comment

Popular posts from this blog

Perl - how to grep a block of text from a file -

-
it can in xml or text format. how in general grep block of text in perl? <track type="ws"> <range> <rangestart>0</rangestart> <rangeend>146.912</rangeend> <locationindex>0</locationindex> <propertyindex>0</propertyindex> </range> </track> <track type="ps" id="1"> <range> <rangestart>0</rangestart> <rangeend>146.912</rangeend> <locationindex>1</locationindex> <propertyindex>1</propertyindex> </range> </track> i want grep type="ps" , till </range> . one solution open file, read line line , match block. open(fh, "file.txt"); foreach $line (<fh>) { if ($line =~ m/type="cc"(.*?)<\/range>/) { print $1; } } but there more optimal solution without reading file line l
Read more

delphi - How to remove all the grips on a coolbar if I have several coolbands? -

-
Image
if set mycoolbar.fixedorder true,only grip on first band hidden. well,if use delphi 7 create vcl forms application,then put coolbar on , create 3 coolbans hold other controls,only grip , top coolbands can hidden setting mycoolbar.fixedorder:=true. i've uploaded picture make things clear. you got fixedorder property wrong. fixed property not allow bands rearranged if set true . setting property of coolbar true keep user changing bands order @ runtime, user can still move , resize bands. can give advice can do, actual solution problem, well, have wait answer. my advice use 3 coolbars in row , setting "fixedorder" property true , bandborderstyle bsnone . way grip hidden on of them. about property, it's not bug of ide, actual preference of property.
Read more

javascript - Animating array of divs; only the final element is modified -

-
this baffling me @ moment. i'm experienced programmer looking throw bit of surprise fiancee on countdown page our wedding. desired effect simulated confetti falling behind main page. setup follows: i have array of divs in javascript. each div absolutely positioned on page. have interval set updates every 50 or milliseconds. on each tick, run loop on array of divs, , calculations , update positions. problem problem having, however, though can see divs created, stored array, , modified (in case, each has slight randomized rotation) during initialization phase, once tick update position starts, div stored in whatever last slot of array gets position updated. tried hard-coding each div index testing purposes, strictly last element update position. upon printing current "left" , "top" style values each div before , after calculations shows value has been changed, no visible change noticeable. here's basic code: javascript var container;
Read more