wget - Robot names for robots.txt -

-

suppose have website uses wget crawl other websites. provide website owners chance of not being crawled website. should use robot name wget in robots.txt file, or have create other name?

common practice disallow , allow popular uas this:

user-agent: google disallow:   user-agent: * disallow: /

so think don't have problems using wget way


Comments

Post a Comment

Popular posts from this blog

objective c - Can't build GCM with Protobuf in Xcode -

-
recently i've decided integrate gcm project uses google protocol buffers. worked fine, sample ok. when i've tried integrate gcm via cocoapods project - started receiving build errors: undefined symbols architecture x86_64: "_gsdk_pblogicalrightshift32", referenced from: _serializedsize in libgcmlib.a(gcmsecuresocket.o) "_objc_class_$_gsdk_pbcodedoutputstream", referenced from: objc-class-ref in libgcmlib.a(gcmsecuresocket.o) "_objc_class_$_gsdk_pbdescriptor", referenced from: objc-class-ref in libgcmlib.a(gtalkcore.pb.o) objc-class-ref in libgcmlib.a(gtalkextensions.pb.o) "_objc_class_$_gsdk_pbgeneratedmessage", referenced from: _objc_class_$_gtalkheartbeatping in libgcmlib.a(gtalkcore.pb.o) _objc_class_$_gtalkheartbeatack in libgcmlib.a(gtalkcore.pb.o) _objc_class_$_gtalkerrorinfo in libgcmlib.a(gtalkcore.pb.o) _objc_class_$_gtalksetting in libgcmlib.a(gtalkcore.pb.o) _objc_class_$_gtalkloginrequest in libgcmlib.a...
Read more

ember.js - EmberJS Model find not up-to-date with underlying store -

-
for simple overview screen have developed route sets controller app.location.find(). app.locationsindexroute = ember.route.extend({ setupcontroller: function(controller) { controller.set('content', app.location.find()); }, rendertemplate: function() { this.render('locations.index',{into:'application'}); } }); i naively assumed go store , fetch me records, giving me up-to-date view of records. apparently not.... when external process starts removing records database, app.location.find() keeps on returning these deleted records. (although rest call doesn't show them anymore) if external process starts adding records database, app.location.find() picks them up. if delete records form within ember app model correctly updated. how should deal in ember app ? i'd have up-to-date view on whatever in database. right need refresh page (f5) date view. using linkto helpers shows me stale data. this seems yet trivial thing misse...
Read more

javascript - jQuery iScroll clickable list elements while retaining scroll? -

-
is possible have clickable elements? such list elements? while using iscroll? seems short of button, or maybe link can't click on element make something. had similar issue inputs well. enough searching able find solution helped me place work inputs. have need have scrollable ui while having ability click on thats not button or link, such li contained within specific uls. so wondering if there way of having iscroll tell difference between click or tap rather vs scroll, , such have action desired.
Read more