How to page REST calls in a future with Dispatch and Scala -
i use scala , dispatch json paged rest api. reason use futures dispatch here because want execute calls fetchissuesbyfile
in parallel, because function result in many rest calls every lookupid
(1x findcomponentkeybyrest
, n x fetchissuepage
, n number of pages yielded rest api).
here's code far:
def fetchissuepage(componentkey: string, pageindex: int): future[json4s.jvalue] def extractpaginginfo(json: org.json4s.jvalue): paginginfo def extractissues(json: org.json4s.jvalue): seq[issue] def findallissuesbyrest(componentkey: string): future[seq[issue]] = { future { var paging = paginginfo(pageindex = 0, pages = 0) val allissues = mutable.listbuffer[issue]() { fetchissuepage(componentkey, paging.pageindex) oncomplete { case success(json) => allissues ++= extractissues(json) paging = extractpaginginfo(json) case _ => //todo error handling } } while (paging.pageindex < paging.pages) allissues // (1) } } def findcomponentkeybyrest(lookupid: string): future[option[string]] def fetchissuesbyfile(lookupid: string): future[(string, option[seq[issue]])] = findcomponentkeybyrest(lookupid).flatmap { case some(key) => findallissuesbyrest(key).map( issues => // (2) process issues ) case none => //todo error handling }
the actual problem never collected issues findallissuesbyrest
(1) (i.e., sequence of issues empty) when try process them @ (2). ideas? also, pagination code isn't functional, i'm open ideas on how improve scala.
any appreciated.
thanks, michael
i think like:
def findallissuesbyrest(componentkey: string): future[seq[issue]] = // fetch first page fetchissuepage(componentkey, 0).flatmap { json => val nbpages = extractpaginginfo(json).pages // total number of pages val firstissues = extractissues(json) // first issues // other pages future.sequence((1 nbpages).map(page => fetchissuepage(componentkey, page))) // issues other pages .map(pages => pages.map(extractissues)) // combine first issues other issues .flatmap(issues => (firstissues +: issues).flatten) }
Comments
Post a Comment