ruby on rails - Heroku Europe: hung requests in Unicorn workers -

-

i have app on heroku forked try out europe region support. after initial hurdles (heroku europe region: "application error" when trying fork) seems working fine.

i ran simple load test ab (apache bench) tool , expected see improvements in requests per second. however, not so:

timings 1000 requests 10 concurrent users (ab -n 1000 -c 10 <url>)

us:

connection times (ms)               min  mean[+/-sd] median   max connect:      305  349  44.4    337     719 processing:   128  356 282.8    244    2213 waiting:      127  350 283.0    238    2213 total:        442  705 280.5    610    2521

eu:

connection times (ms)               min  mean[+/-sd] median   max connect:      125  188  47.6    175     451 processing:    67 2595 3537.9   3309   30171 waiting:       66 2591 3538.9   3309   30170 total:        207 2783 3536.2   3472   30321

some things jump out:

  • latency (see "connect:") lower eu us, expected. good.
  • some requests in eu taking 30s ? what?

after investigation, found:

  • heroku has default request timeout of 30s: request taking longer killed. explains max 30s.
  • but why happening? looked in logs , found requests fast, began take longer until eventually, seemed "hang" in unicorn worker. workers have timeout of 15s, after worker killed , restarted master unicorn process (shows in heroku logs h13 error). assume request not retried though, resulting in final time of 30s.

i have looked @ new relic, these curious slow requests show 99% time spent in unspecified "application code (root)" (which shows separately actual db access etc.) no possibility drill down (note: i'm on free plan). there seems nothing wrong app code (it running fine on after all).

my question: how go debugging this? there configuration change solve this? missing?

update:

i tried out suggestions in comments, , in end disabled memcachier addon entirely eu (plus changed app no caching @ anymore, test).

this did not resolve unicorn timeouts (although seem occur less (!)).

i suspect 1 of add-ons provisioned in different region. retrieve memcachier url config:get:

$ heroku config:get memcachier_url

and paste domain http://ip-lookup.net/ see where, geographically, add-on hosted.

if it's in us, need reprovision in eu:

$ heroku addons:remove memcachier $ heroku addons:add memcachier

and should provision in right region.

if add-on provisioned in wrong region, it's bug. if so, comment here app name , can it.


Comments

Post a Comment

Popular posts from this blog

c++ - Function signature as a function template parameter -

-
i'd use function signature template argument. works great classes, when try same trick function templates, msvc throws error: error c2768: 'func' : illegal use of explicit template arguments here's code: template <typename signature> void func(); template <typename r, typename a1> void func<r(a1)>(); what should make work? you cannot partially specialize function template, not supported language. can create partially specialized class template static member function, , possibly trampoline function instantiate class template , invoke static function. something this: namespace detail { template<typename signature> struct helper; template<typename r, typename a1> struct helper<r(a1)> { static void call() { // stuff r , a1... } }; } template<typename signature> void func() { detail::helper<signature>::call(); }
Read more

algorithm - What are some ways to combine a number of (potentially incompatible) sorted sub-sets of a total set into a (partial) ordering of the total set? -

-
i have sorting problem i'm pretty sure doesn't have canonical "answer" - has number of different approaches, each pros/cons. i'm interested in hearing different approaches. suppose bunch of people {p_i | i=1,...,m} ranking favorite ice cream flavors. in total there n different flavors, single person p_i ranks n_i << n flavors or familiar with. i'm interested in combining these sub-rankings plausible overall ranking of n flavors. for concrete situation: m , n both 1000 (by coincidence), , each n_i 20 . can assume there's enough overlap in flavors people rank such no flavor "isolated". again, i'm interested in hearing different ways approach this, if there isn't single clear answer. thanks! one way approach problem construct directed cyclic graph representing of constraints on ordering of elements. nodes in graph represent different elements compared , have edge first node second node if had ranked first e...
Read more

How to call a javascript function after the page loads with a chrome extension? -

-
there's function on page want call after loads. not own page, can't insert html it. can please provide specific examples how chrome extension or greasemonkey? (function submitaction() ). both chrome extensions , greasemonkey offer ways specify when script inject if executed. in chrome extension, you set in manifest : "content_scripts": [ { "run_at" : "document_idle", in greasemonkey compatible userscripts, set in metadata // @run-at document-end in both cases, have add script if want ensure resources loaded when function executed : document.addeventlistener('load', submitaction);
Read more