Transfer of large data
Maria A. Nieto-Santisteban
nieto at skysrv.pha.jhu.edu
Thu Jul 21 06:53:47 PDT 2005
At JHU we are planning on including and evaluating the meaga-streamimg
protocols UDT and SOAP* developed by Grossman et al. at the National
center for Data Mining (NCDM).
UDT (UDP-based Data Transfer) for files and SOAP* for Web services.
here are some references:
 A. Chien, T. Faber, A. Falk, J. Bannister, R. Grossman, and J. Leigh.
Transport protocols for high performance: Whither TCP? Communications of
the ACM, 46(11):42. 49, 2003.
 R. L. Grossman, M. Mazzucco, H. Sivakumar, Y. Pan, and Q. Zhang.
SABUL, Simple Available Bandwidth Utilization Library for high-speed wide
area networks. Journal of Supercomputing, to appear.
On Thu, 21 Jul 2005, Masahiro TANAKA wrote:
> Dear all,
> Have you experienced large data transfer (e.g. >100MB) between
> distributed VO machine? Would you tell your practical solution?
> Our JVO team has built a SkyNode server which receives a query and
> returns a result VOTable through Web Services, using Apache Axis.
> Then we encounter a performance problem in large data transfer
> using SOAP messages; Large amount of memory is allocated to hold
> the entire data as Java objects. It slows processing speed, too.
> Sending large data through SOAP messages seems unrealistic.
> So we are considering SkyNodes pass result VOTable using:
> 1. Staging (Result data are separately transferred through HTTP or
> FTP, like SIAP)
> 2. Attachment
> Both will need some extension to SkyNode protocol.
> Masahiro Tanaka
Maria A. Nieto-Santisteban (nieto at pha.jhu.edu)
Johns Hopkins University
3400 N. Charles St.
Physics & Astronomy Department
Baltimore, MD 21218 (USA)
Tel: 1 410 516-7679 Fax: 1 410 516-5096
More information about the dal