python - Is there a better way to do csv/namedtuple with urlopen? -

-

using namedtuple documentation example template in python 3.3, have following code download csv , turn series of namedtuple subclass instances:

from collections import namedtuple csv import reader urllib.request import urlopen      securitytype = namedtuple('securitytype', 'sector, name')  url = 'http://bsym.bloomberg.com/sym/pages/security_type.csv' sec in map(securitytype._make, reader(urlopen(url))):     print(sec)

this raises following exception:

traceback (most recent call last):   file "scrap.py", line 9, in <module>     sec in map(securitytype._make, reader(urlopen(url))): _csv.error: iterator should return strings, not bytes (did open file in text mode?)

i know issue urlopen returning bytes , not strings , need decode output @ point. here's how i'm doing now, using stringio:

from collections import namedtuple csv import reader urllib.request import urlopen import io  securitytype = namedtuple('securitytype', 'sector, name')  url = 'http://bsym.bloomberg.com/sym/pages/security_type.csv' reader_input = io.stringio(urlopen(url).read().decode('utf-8'))  sec in map(securitytype._make, reader(reader_input)):     print(sec)

this smells funny because i'm iterating on bytes buffer, decoding, rebuffering, iterating on new string buffer. there more pythonic way without 2 iterations?

use io.textiowrapper() decode urllib response:

reader_input = io.textiowrapper(urlopen(url), encoding='utf8', newline='')

now csv.reader passed exact same interface when opening regular file on filesystem in text mode.

with change example url works me on python 3.3.1:

>>> sec in map(securitytype._make, reader(reader_input)): ...     print(sec) ...  securitytype(sector='market sector', name='security type') securitytype(sector='comdty', name='calendar spread option') securitytype(sector='comdty', name='financial commodity future.') securitytype(sector='comdty', name='financial commodity generic.') securitytype(sector='comdty', name='financial commodity option.') ... securitytype(sector='muni', name='zero coupon, oid') securitytype(sector='pfd', name='private') securitytype(sector='pfd', name='public') securitytype(sector='', name='') securitytype(sector='', name='') securitytype(sector='', name='') securitytype(sector='', name='') securitytype(sector='', name='') securitytype(sector='', name='') securitytype(sector='', name='') securitytype(sector='', name='') securitytype(sector='', name='')

the last lines appear yield empty tuples; original indeed has lines nothing more comma on them.


Comments

Post a Comment

Popular posts from this blog

c++ - Function signature as a function template parameter -

-
i'd use function signature template argument. works great classes, when try same trick function templates, msvc throws error: error c2768: 'func' : illegal use of explicit template arguments here's code: template <typename signature> void func(); template <typename r, typename a1> void func<r(a1)>(); what should make work? you cannot partially specialize function template, not supported language. can create partially specialized class template static member function, , possibly trampoline function instantiate class template , invoke static function. something this: namespace detail { template<typename signature> struct helper; template<typename r, typename a1> struct helper<r(a1)> { static void call() { // stuff r , a1... } }; } template<typename signature> void func() { detail::helper<signature>::call(); }
Read more

algorithm - What are some ways to combine a number of (potentially incompatible) sorted sub-sets of a total set into a (partial) ordering of the total set? -

-
i have sorting problem i'm pretty sure doesn't have canonical "answer" - has number of different approaches, each pros/cons. i'm interested in hearing different approaches. suppose bunch of people {p_i | i=1,...,m} ranking favorite ice cream flavors. in total there n different flavors, single person p_i ranks n_i << n flavors or familiar with. i'm interested in combining these sub-rankings plausible overall ranking of n flavors. for concrete situation: m , n both 1000 (by coincidence), , each n_i 20 . can assume there's enough overlap in flavors people rank such no flavor "isolated". again, i'm interested in hearing different ways approach this, if there isn't single clear answer. thanks! one way approach problem construct directed cyclic graph representing of constraints on ordering of elements. nodes in graph represent different elements compared , have edge first node second node if had ranked first e...
Read more

How to call a javascript function after the page loads with a chrome extension? -

-
there's function on page want call after loads. not own page, can't insert html it. can please provide specific examples how chrome extension or greasemonkey? (function submitaction() ). both chrome extensions , greasemonkey offer ways specify when script inject if executed. in chrome extension, you set in manifest : "content_scripts": [ { "run_at" : "document_idle", in greasemonkey compatible userscripts, set in metadata // @run-at document-end in both cases, have add script if want ensure resources loaded when function executed : document.addeventlistener('load', submitaction);
Read more