From bourges.laurent at gmail.com Fri Oct 7 01:45:40 2011 From: bourges.laurent at gmail.com (=?ISO-8859-1?Q?Laurent_Bourg=E8s?=) Date: Fri, 7 Oct 2011 10:45:40 +0200 Subject: URL handling in SAMP message parameters Message-ID: Dear Samp users, In several standard MTypes, arguments can be url (string) giving the URL of the document to load : - table.load.votable - table.load.fits - table.highlight.row - table.select.rowList - image.load.fits - spectrum.load.ssa-generic - voresource.loadlist Although SAMP is dedicated to desktop interoperability, such URL can use any protocol (file://, http://, ftp://). To be concrete, I am currently using JSAMP 1.3 and I must handle both 'file://' and 'http://' URL when I process one table.load.votable message: - 'file://' are provided by our JMMC applications: data are stored into local files before sending their URL via SAMP - 'http://' are provided by topcat because it embeds its own HTTP server Two questions : - Is it possible to describe which URL procotols must be supported by SAMP clients (file, http at least) ? I do not want to support other protocols (ftp ...) : the specification / MType wiki page should give explanations / details - Is it possible to provide in SAMP implementations (JSAMP, Sampy ...) an easy API to deal with such URL like getDocument(URL url) ? I think it is boring and error prone that every application developer must handle all URL protocols on its own although he only needs to get the document itself. I know it is not so difficult but error handling can become tricky (http errors ...) Best regards, Laurent From m.b.taylor at bristol.ac.uk Fri Oct 7 02:04:03 2011 From: m.b.taylor at bristol.ac.uk (Mark Taylor) Date: Fri, 7 Oct 2011 10:04:03 +0100 (BST) Subject: URL handling in SAMP message parameters In-Reply-To: References: Message-ID: Laurent, On Fri, 7 Oct 2011, Laurent Bourg?s wrote: > Dear Samp users, > > In several standard MTypes, arguments can be url (string) giving the > URL of the document to load : > - table.load.votable > - table.load.fits > - table.highlight.row > - table.select.rowList > - image.load.fits > - spectrum.load.ssa-generic > - voresource.loadlist > > Although SAMP is dedicated to desktop interoperability, such URL can > use any protocol (file://, http://, ftp://). > To be concrete, I am currently using JSAMP 1.3 and I must handle both > 'file://' and 'http://' URL when I process one table.load.votable > message: > - 'file://' are provided by our JMMC applications: data are stored > into local files before sending their URL via SAMP > - 'http://' are provided by topcat because it embeds its own HTTP server > > Two questions : > - Is it possible to describe which URL procotols must be supported by > SAMP clients (file, http at least) ? I do not want to support other > protocols (ftp ...) : the specification / MType wiki page should give > explanations / details Although in principle this is a legitimate concern, it hasn't come up before, I guess because most languages/environments in which people are receiving SAMP messages provide some ready-made machinery for retrieving data from URLs with a minimum of programmer effort. So in practice the list of legitimate URL types is the common set of protocols handled by common SAMP language platforms - for Java it is (I think) http, https, ftp and file; I expect it's similar for Python. I expect that nearly all URLs actually sent by SAMP tools are http or file; there may be the odd https or ftp. > - Is it possible to provide in SAMP implementations (JSAMP, Sampy ...) > an easy API to deal with such URL like getDocument(URL url) ? It's not provided as part of JSAMP, because it comes for free with Java, in the java.net.URL class: InputStream in = new URL(urlString).openStream(); allows you to read the resource of a URL without worrying about whether it's http, https, ftp or file protcol. I'm pretty sure python, C# and pretty much any other modern language or language+library platform can provide something similar. > I think it is boring and error prone that every application developer > must handle all URL protocols on its own although he only needs to get > the document itself. I know it is not so difficult but error handling > can become tricky (http errors ...) it would be! But the same thing comes up in many situations other than SAMP, which is why library support to address it is widespread. Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor at bris.ac.uk +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From m.b.taylor at bristol.ac.uk Fri Oct 7 06:46:01 2011 From: m.b.taylor at bristol.ac.uk (Mark Taylor) Date: Fri, 7 Oct 2011 14:46:01 +0100 (BST) Subject: URL handling in SAMP message parameters In-Reply-To: References: Message-ID: On Fri, 7 Oct 2011, Laurent Bourg?s wrote: > Dear Samp users, > > In several standard MTypes, arguments can be url (string) giving the > URL of the document to load : > - table.load.votable > - table.load.fits > - table.highlight.row > - table.select.rowList > - image.load.fits > - spectrum.load.ssa-generic > - voresource.loadlist > > Although SAMP is dedicated to desktop interoperability, such URL can > use any protocol (file://, http://, ftp://). > To be concrete, I am currently using JSAMP 1.3 and I must handle both > 'file://' and 'http://' URL when I process one table.load.votable > message: > - 'file://' are provided by our JMMC applications: data are stored > into local files before sending their URL via SAMP > - 'http://' are provided by topcat because it embeds its own HTTP server To address this particular question of Laurent's: > - Is it possible to describe which URL procotols must be supported by > SAMP clients (file, http at least) ? I do not want to support other > protocols (ftp ...) : the specification / MType wiki page should give > explanations / details Since this is a semantic issue it's really in the realm of MTypes, but since it is a common consideration for many MTypes in use, I think it's reasonable to discuss it in the standard itself. I'm reluctant to be too prescriptive (e.g. a MANDATORY list of what senders are permitted to send or recipients are required to understand); for one thing, it will probably be ignored - we can't stop people using weird URL protocols here, and if cooperation between sender and recipients means they work then we shouldn't do - and for another thing, the list of what are 'common' protocols may change over time. Section 3.3 of the standard (SAMP Data Types) already contains the following text: Although SAMP imposes no maximum on the length of a string, particular transport protocols or implementation considerations may effectively do so; in general, hub and client implementations are not expected to deal with data items of unlimited size. General purpose MTypes SHOULD therefore be specified so that bulk data is not sent within the message. In general it is preferred to define a message parameter as the URL or filename of a potentially large file rather than as the inline text of the file itself. I suggest adding the following: SAMP defines no formal list of which URL schemes are permitted in such cases, but clients which need to dereference such URLs SHOULD be capable of dealing with at least the ``http'' and ``file'' schemes. ``https'', ``ftp'' and perhaps other schemes may also be encountered, though clients are discouraged from sending them unless there is some good reason to do so. Laurent, do you think this is OK? Anyone else have any comments? Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor at bris.ac.uk +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From dtody at nrao.edu Sat Oct 8 16:32:41 2011 From: dtody at nrao.edu (Douglas Tody) Date: Sat, 8 Oct 2011 17:32:41 -0600 (MDT) Subject: URL handling in SAMP message parameters In-Reply-To: References: Message-ID: On Fri, 7 Oct 2011, Mark Taylor wrote: > I suggest adding the following: > > SAMP defines no formal list of which URL schemes are permitted > in such cases, but clients which need to dereference such URLs > SHOULD be capable of dealing with at least the ``http'' and ``file'' > schemes. ``https'', ``ftp'' and perhaps other schemes may also > be encountered, though clients are discouraged from sending them > unless there is some good reason to do so. > > Laurent, do you think this is OK? Anyone else have any comments? I don't think we should discourage use of more specialized URL schemes (certainly not common ones such as HTTPS or FTP). Rather just suggest which ones *should* be supported, e.g. HTTP (for the Web) and FILE (for local references). This will be sufficient to suggest what to use in most cases. - Doug From m.b.taylor at bristol.ac.uk Mon Oct 10 02:01:49 2011 From: m.b.taylor at bristol.ac.uk (Mark Taylor) Date: Mon, 10 Oct 2011 10:01:49 +0100 (BST) Subject: URL handling in SAMP message parameters In-Reply-To: References: Message-ID: On Sat, 8 Oct 2011, Douglas Tody wrote: > On Fri, 7 Oct 2011, Mark Taylor wrote: > > > I suggest adding the following: > > > > SAMP defines no formal list of which URL schemes are permitted > > in such cases, but clients which need to dereference such URLs > > SHOULD be capable of dealing with at least the ``http'' and ``file'' > > schemes. ``https'', ``ftp'' and perhaps other schemes may also > > be encountered, though clients are discouraged from sending them > > unless there is some good reason to do so. > > > > Laurent, do you think this is OK? Anyone else have any comments? > > I don't think we should discourage use of more specialized URL schemes > (certainly not common ones such as HTTPS or FTP). Rather just suggest > which ones *should* be supported, e.g. HTTP (for the Web) and FILE > (for local references). This will be sufficient to suggest what to use > in most cases. How about something more neutral like this: ``https'', ``ftp'' and other schemes are also permitted, but when sending such URLs consideration should be given to whether receiving clients are likely to be able to dereference them. This doesn't really discourage use, but just makes it explicit that such schemes are not disallowed, and makes it clear what the grounds are for deciding on what is a suitable scheme to use. Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor at bris.ac.uk +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From m.b.taylor at bristol.ac.uk Sun Oct 16 06:24:18 2011 From: m.b.taylor at bristol.ac.uk (Mark Taylor) Date: Sun, 16 Oct 2011 14:24:18 +0100 (BST) Subject: Hub resource hosting service? Message-ID: Hi all. Here is an issue, and a possible solution, which was raised by some Web Profile implementation work that Tom is doing. We will discuss it in the Apps SAMP session on Thursday, but I wanted to air it here first to give people time to think about it, and also to solicit comments from people who are not in Pune. Tom said: On Fri, 14 Oct 2011, Tom McGlynn wrote: > A final largely unrelated issue... SAMP's reliance on passing VOTables only > via URLs makes it difficult to adapt to JavaScript clients. They have to > create relay URLs, stage results, or duplicate functionality on the server > side. E.g., if we were to want to send a VOTable that had been filtered > using VOView, there is no easy way to do this that I know of. I'm sure this > was considered early on, but it's possible to do quite extensive processing > in the web browser now and we may wish to consider supporting that, > especially with WebSAMP. E.g., we might allow either 'url' or 'content' > tags in the message (where the later would be the actual VOTable content) or > defining equivalent mtypes for messages with table content. This hadn't occurred to me before. I had assumed that if web apps needed URLs to pass around they would come from the origin HTTP server. While an app could pass locally generated/manipulated data back to its origin server and send the resulting URL around in SAMP messages it's clearly not an efficient solution, since the data starts and ends on the client so doesn't want to have to go via a remote server. Tom's suggestion of adding content parameters as an alternative to url parameters in MTypes is possible, but a bit unpalatable for a couple of reasons. First it falls foul of the SHOULD (though not MUST) in SAMP section 3.3 that says: "General purpose MTypes SHOULD therefore be specified so that bulk data is not sent within the message. In general it is preferred to define a message parameter as the URL or filename of a potentially large file rather than as the inline text of the file itself." largely to prevent clients having to deal with very large chunks of XML, which is especially a problem if they DOMise them as a matter of routine. Second it means that a lot of new messages (or at least new versions of messages, using content rather than url parameters to specify content) would be flying around and old clients wouldn't be able to handle them, until they were upgraded accordingly. Another way to look at the problem is that Web Profile SAMP clients lack an efficient resource hosting mechanism (since, unlike Standard Profile clients, they are unable to write temporary files or host HTTP servers). This could be addressed by allowing the hub to provide such a mechanism. The most straightforward thing would be for the hub to provide a URL to which documents could be POSTed in return for a (201 Created - see RFC2616 sec 9.5) Location URL. Web clients which want to pass data to another client could therefore POST the docuement to the hub, get a reference URL in return, and pass that reference URL in messages. The hub arranges to redeliver the content to anyone who asks for it via that reference URL later. This also isn't perfect, since it means the hub has to manage some client resources rather than clients, but to me it seems cleaner. Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor at bris.ac.uk +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From dtody at nrao.edu Sun Oct 16 07:01:51 2011 From: dtody at nrao.edu (Douglas Tody) Date: Sun, 16 Oct 2011 19:31:51 +0530 (IST) Subject: Hub resource hosting service? In-Reply-To: References: Message-ID: One thing discussed way back in the original SAMP discussions was separating the message header from the message content, which would allow the content to be anything and defined by the messaging protocol. This would allow real data to be passed via the messaging system, including even binary data. To keep things simple this was left out of SAMP 1.0 but it was always considered a possibility for future versions. Essentially what we have now is only the message header, with the message type specifying what type of message it is, and parameters providing a limited way to pass some data. Keeping messages small and lightweight is good, but what is best really depends upon the use case and should not be constrained by the messaging system. In real time applications for example (including desktop applications such as displaying an image in real time) data can be streamed in chunks and there is no possibility to build files which are stored some place and then referenced by a URL or other pointer. While limiting messages to pointing at bulk data is a good model in many cases it is not the only one, and ultimately it is the application which should decide what is the best model. I don't much like the idea of the hub being turned into a general file store - lots of complication there with persistence issues and such. This has nothing to do with messaging. If such a capability is needed it ought to be a different service, with messaging limited to only the communication aspect. - Doug On Sun, 16 Oct 2011, Mark Taylor wrote: > Hi all. > > Here is an issue, and a possible solution, which was raised by some > Web Profile implementation work that Tom is doing. We will discuss > it in the Apps SAMP session on Thursday, but I wanted to air it here > first to give people time to think about it, and also to solicit > comments from people who are not in Pune. > > Tom said: > > On Fri, 14 Oct 2011, Tom McGlynn wrote: > >> A final largely unrelated issue... SAMP's reliance on passing VOTables only >> via URLs makes it difficult to adapt to JavaScript clients. They have to >> create relay URLs, stage results, or duplicate functionality on the server >> side. E.g., if we were to want to send a VOTable that had been filtered >> using VOView, there is no easy way to do this that I know of. I'm sure this >> was considered early on, but it's possible to do quite extensive processing >> in the web browser now and we may wish to consider supporting that, >> especially with WebSAMP. E.g., we might allow either 'url' or 'content' >> tags in the message (where the later would be the actual VOTable content) or >> defining equivalent mtypes for messages with table content. > > This hadn't occurred to me before. I had assumed that if web apps > needed URLs to pass around they would come from the origin HTTP server. > While an app could pass locally generated/manipulated data back to its > origin server and send the resulting URL around in SAMP messages it's > clearly not an efficient solution, since the data starts and ends on > the client so doesn't want to have to go via a remote server. > > Tom's suggestion of adding content parameters as an alternative to url > parameters in MTypes is possible, but a bit unpalatable for a couple > of reasons. First it falls foul of the SHOULD (though not MUST) in > SAMP section 3.3 that says: > > "General purpose MTypes SHOULD therefore be specified so that bulk > data is not sent within the message. In general it is preferred to > define a message parameter as the URL or filename of a potentially > large file rather than as the inline text of the file itself." > > largely to prevent clients having to deal with very large chunks of XML, > which is especially a problem if they DOMise them as a matter of routine. > Second it means that a lot of new messages (or at least new versions > of messages, using content rather than url parameters to specify > content) would be flying around and old clients wouldn't be able to > handle them, until they were upgraded accordingly. > > Another way to look at the problem is that Web Profile SAMP clients lack an > efficient resource hosting mechanism (since, unlike Standard Profile > clients, they are unable to write temporary files or host HTTP servers). > This could be addressed by allowing the hub to provide such a mechanism. > The most straightforward thing would be for the hub to provide a URL > to which documents could be POSTed in return for a (201 Created - > see RFC2616 sec 9.5) Location URL. Web clients which want > to pass data to another client could therefore POST the docuement to > the hub, get a reference URL in return, and pass that reference URL > in messages. The hub arranges to redeliver the content to anyone > who asks for it via that reference URL later. > This also isn't perfect, since it means the hub has to manage some > client resources rather than clients, but to me it seems cleaner. > > Mark > > -- > Mark Taylor Astronomical Programmer Physics, Bristol University, UK > m.b.taylor at bris.ac.uk +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ > From m.b.taylor at bristol.ac.uk Mon Oct 17 06:26:49 2011 From: m.b.taylor at bristol.ac.uk (Mark Taylor) Date: Mon, 17 Oct 2011 14:26:49 +0100 (BST) Subject: Hub resource hosting service? In-Reply-To: References: Message-ID: Doug, On Sun, 16 Oct 2011, Douglas Tody wrote: > One thing discussed way back in the original SAMP discussions was > separating the message header from the message content, which would > allow the content to be anything and defined by the messaging protocol. > This would allow real data to be passed via the messaging system, > including even binary data. To keep things simple this was left out of > SAMP 1.0 but it was always considered a possibility for future versions. > Essentially what we have now is only the message header, with the > message type specifying what type of message it is, and parameters > providing a limited way to pass some data. > > Keeping messages small and lightweight is good, but what is best really > depends upon the use case and should not be constrained by the messaging > system. In real time applications for example (including desktop > applications such as displaying an image in real time) data can be > streamed in chunks and there is no possibility to build files which are > stored some place and then referenced by a URL or other pointer. While > limiting messages to pointing at bulk data is a good model in many cases > it is not the only one, and ultimately it is the application which > should decide what is the best model. Your analysis is quite right. SAMP as it stands is not a good candidate for a real time messaging system, and there are various other messaging scenarios that it is not targetted at. Certainly the application has to decide what's the best model, and having decided, it may decide that SAMP (as it stands) is not the system it should use to deliver that. Some future evolution of SAMP (AMP?) that is capable of addressing those issues is possible, and as you say was something we deliberately left open when we started to think about it. On the whole I don't feel that we are currently in a position where we have the need to take that further. My reluctance is mainly that the ways I can think of to do bulk data transfer within SAMP itself would complicate implementations quite a bit, and SAMP seems to be working reasonably well now for the kinds of things it's good at, with good takeup because it keeps things simple. But if you or others think different, we can re-open that discussion (though probably not in this Thursday's session, which is full already). > I don't much like the idea of the hub being turned into a general file > store - lots of complication there with persistence issues and such. > This has nothing to do with messaging. If such a capability is needed > it ought to be a different service, with messaging limited to only the > communication aspect. The Web Profile hub already provides some non-messaging services, in particular dereferencing of URLs, since the Web Profile is in the business of providing web applications with the things they need to be useful SAMP clients and can't get in any other way. However, I take your point. I don't want to push the resource hosting idea if other people are not keen. Mark > On Sun, 16 Oct 2011, Mark Taylor wrote: > > > Hi all. > > > > Here is an issue, and a possible solution, which was raised by some > > Web Profile implementation work that Tom is doing. We will discuss > > it in the Apps SAMP session on Thursday, but I wanted to air it here > > first to give people time to think about it, and also to solicit > > comments from people who are not in Pune. > > > > Tom said: > > > > On Fri, 14 Oct 2011, Tom McGlynn wrote: > > > > > A final largely unrelated issue... SAMP's reliance on passing VOTables > > > only > > > via URLs makes it difficult to adapt to JavaScript clients. They have to > > > create relay URLs, stage results, or duplicate functionality on the server > > > side. E.g., if we were to want to send a VOTable that had been filtered > > > using VOView, there is no easy way to do this that I know of. I'm sure > > > this > > > was considered early on, but it's possible to do quite extensive > > > processing > > > in the web browser now and we may wish to consider supporting that, > > > especially with WebSAMP. E.g., we might allow either 'url' or 'content' > > > tags in the message (where the later would be the actual VOTable content) > > > or > > > defining equivalent mtypes for messages with table content. > > > > This hadn't occurred to me before. I had assumed that if web apps > > needed URLs to pass around they would come from the origin HTTP server. > > While an app could pass locally generated/manipulated data back to its > > origin server and send the resulting URL around in SAMP messages it's > > clearly not an efficient solution, since the data starts and ends on > > the client so doesn't want to have to go via a remote server. > > > > Tom's suggestion of adding content parameters as an alternative to url > > parameters in MTypes is possible, but a bit unpalatable for a couple > > of reasons. First it falls foul of the SHOULD (though not MUST) in > > SAMP section 3.3 that says: > > > > "General purpose MTypes SHOULD therefore be specified so that bulk > > data is not sent within the message. In general it is preferred to > > define a message parameter as the URL or filename of a potentially > > large file rather than as the inline text of the file itself." > > > > largely to prevent clients having to deal with very large chunks of XML, > > which is especially a problem if they DOMise them as a matter of routine. > > Second it means that a lot of new messages (or at least new versions > > of messages, using content rather than url parameters to specify > > content) would be flying around and old clients wouldn't be able to > > handle them, until they were upgraded accordingly. > > > > Another way to look at the problem is that Web Profile SAMP clients lack an > > efficient resource hosting mechanism (since, unlike Standard Profile > > clients, they are unable to write temporary files or host HTTP servers). > > This could be addressed by allowing the hub to provide such a mechanism. > > The most straightforward thing would be for the hub to provide a URL > > to which documents could be POSTed in return for a (201 Created - > > see RFC2616 sec 9.5) Location URL. Web clients which want > > to pass data to another client could therefore POST the docuement to > > the hub, get a reference URL in return, and pass that reference URL > > in messages. The hub arranges to redeliver the content to anyone > > who asks for it via that reference URL later. > > This also isn't perfect, since it means the hub has to manage some > > client resources rather than clients, but to me it seems cleaner. > > > > Mark > > > > -- > > Mark Taylor Astronomical Programmer Physics, Bristol University, UK > > m.b.taylor at bris.ac.uk +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ > > > -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor at bris.ac.uk +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From Thomas.A.McGlynn at nasa.gov Mon Oct 17 12:44:47 2011 From: Thomas.A.McGlynn at nasa.gov (Tom McGlynn) Date: Mon, 17 Oct 2011 15:44:47 -0400 Subject: Hub resource hosting service? In-Reply-To: References: Message-ID: <4E9C85AF.3090303@nasa.gov> Hi Mark, I'm not particularly attached to any solution to the issue, but I do think it would be desirable to enable work in a sandbox environment (both JavaScript and untrusted applets) to use SAMP effectively. Your proposed solution is to provide a standard caching mechanism through the WebSamp hub. I've some comments on that... - The simplicity of the SAMP interface is very nice. Adding a whole new layer to the interface to enable this kind of caching seems a bit painful, but maybe that isn't necessary. We could allow the hub to transform the message: E.g., the VOTable generating client would just send the message: table.load.votable: content=... to the hub and the hub would automatically save the content and send table.load.votable: url=http://localhost:xxxx?... to the consuming clients. The hub would send a response to the generating client giving the URL. The rule could be that a hub is free to cache any 'content' element and transform them into references to that content. Given that we are implementing caching, I don't think this makes the hub any more complex, we're just using the mtype syntax for the caching messages. Presumably generated FITS data or images could be sent this way too. So there's no new protocol for clients to learn. That would be a nice feature of a solution. - I'd be worried about security with a hub that is going to write into the filesystem. I'd be more comfortable with a hub that provided say 100 MB of cache space using only memory. - In the simple, but probably very common, case of one generating client and one consuming client, using a cache may be more expensive than just sending the data, especially if we cache using a file. We have two transactions, generator->hub->write-to-file read-from-file->hub->consumer in the cached case versus generator->hub->consumer in the direct case. This would be mitigated by an in-memory cache. - Similarly in the one generator to many consumers case, using URLs means that the hub may need to read the cache many times independently to respond to each of the clients as they request the reference URL. The direct transfer may be able send data more efficiently in parallel. However, a direct transfer doesn't give a consumer a chance to ignore a request, and thus will be less efficient if there are clients that are are registered to receive certain events, but in practice ignore them. Hope you can work something out in Pune. Tom Mark Taylor wrote: > Doug, > > On Sun, 16 Oct 2011, Douglas Tody wrote: > >> One thing discussed way back in the original SAMP discussions was >> separating the message header from the message content, which would >> allow the content to be anything and defined by the messaging protocol. >> This would allow real data to be passed via the messaging system, >> including even binary data. To keep things simple this was left out of >> SAMP 1.0 but it was always considered a possibility for future versions. >> Essentially what we have now is only the message header, with the >> message type specifying what type of message it is, and parameters >> providing a limited way to pass some data. >> >> Keeping messages small and lightweight is good, but what is best really >> depends upon the use case and should not be constrained by the messaging >> system. In real time applications for example (including desktop >> applications such as displaying an image in real time) data can be >> streamed in chunks and there is no possibility to build files which are >> stored some place and then referenced by a URL or other pointer. While >> limiting messages to pointing at bulk data is a good model in many cases >> it is not the only one, and ultimately it is the application which >> should decide what is the best model. > > Your analysis is quite right. SAMP as it stands is not a good candidate > for a real time messaging system, and there are various other messaging > scenarios that it is not targetted at. Certainly the application has > to decide what's the best model, and having decided, it may > decide that SAMP (as it stands) is not the system it should use to > deliver that. Some future evolution of SAMP (AMP?) that is capable of > addressing those issues is possible, and as you say was something > we deliberately left open when we started to think about it. > On the whole I don't feel that we are currently in a position where we > have the need to take that further. My reluctance is mainly that the > ways I can think of to do bulk data transfer within SAMP itself > would complicate implementations quite a bit, and SAMP seems to > be working reasonably well now for the kinds of things it's good at, > with good takeup because it keeps things simple. But if you or > others think different, we can re-open that discussion > (though probably not in this Thursday's session, which is full > already). > >> I don't much like the idea of the hub being turned into a general file >> store - lots of complication there with persistence issues and such. >> This has nothing to do with messaging. If such a capability is needed >> it ought to be a different service, with messaging limited to only the >> communication aspect. > > The Web Profile hub already provides some non-messaging services, in > particular dereferencing of URLs, since the Web Profile is in the > business of providing web applications with the things they need > to be useful SAMP clients and can't get in any other way. > However, I take your point. I don't want to push the resource > hosting idea if other people are not keen. > > Mark > >> On Sun, 16 Oct 2011, Mark Taylor wrote: >> >>> Hi all. >>> >>> Here is an issue, and a possible solution, which was raised by some >>> Web Profile implementation work that Tom is doing. We will discuss >>> it in the Apps SAMP session on Thursday, but I wanted to air it here >>> first to give people time to think about it, and also to solicit >>> comments from people who are not in Pune. >>> >>> Tom said: >>> >>> On Fri, 14 Oct 2011, Tom McGlynn wrote: >>> >>>> A final largely unrelated issue... SAMP's reliance on passing VOTables >>>> only >>>> via URLs makes it difficult to adapt to JavaScript clients. They have to >>>> create relay URLs, stage results, or duplicate functionality on the server >>>> side. E.g., if we were to want to send a VOTable that had been filtered >>>> using VOView, there is no easy way to do this that I know of. I'm sure >>>> this >>>> was considered early on, but it's possible to do quite extensive >>>> processing >>>> in the web browser now and we may wish to consider supporting that, >>>> especially with WebSAMP. E.g., we might allow either 'url' or 'content' >>>> tags in the message (where the later would be the actual VOTable content) >>>> or >>>> defining equivalent mtypes for messages with table content. >>> >>> This hadn't occurred to me before. I had assumed that if web apps >>> needed URLs to pass around they would come from the origin HTTP server. >>> While an app could pass locally generated/manipulated data back to its >>> origin server and send the resulting URL around in SAMP messages it's >>> clearly not an efficient solution, since the data starts and ends on >>> the client so doesn't want to have to go via a remote server. >>> >>> Tom's suggestion of adding content parameters as an alternative to url >>> parameters in MTypes is possible, but a bit unpalatable for a couple >>> of reasons. First it falls foul of the SHOULD (though not MUST) in >>> SAMP section 3.3 that says: >>> >>> "General purpose MTypes SHOULD therefore be specified so that bulk >>> data is not sent within the message. In general it is preferred to >>> define a message parameter as the URL or filename of a potentially >>> large file rather than as the inline text of the file itself." >>> >>> largely to prevent clients having to deal with very large chunks of XML, >>> which is especially a problem if they DOMise them as a matter of routine. >>> Second it means that a lot of new messages (or at least new versions >>> of messages, using content rather than url parameters to specify >>> content) would be flying around and old clients wouldn't be able to >>> handle them, until they were upgraded accordingly. >>> >>> Another way to look at the problem is that Web Profile SAMP clients lack an >>> efficient resource hosting mechanism (since, unlike Standard Profile >>> clients, they are unable to write temporary files or host HTTP servers). >>> This could be addressed by allowing the hub to provide such a mechanism. >>> The most straightforward thing would be for the hub to provide a URL >>> to which documents could be POSTed in return for a (201 Created - >>> see RFC2616 sec 9.5) Location URL. Web clients which want >>> to pass data to another client could therefore POST the docuement to >>> the hub, get a reference URL in return, and pass that reference URL >>> in messages. The hub arranges to redeliver the content to anyone >>> who asks for it via that reference URL later. >>> This also isn't perfect, since it means the hub has to manage some >>> client resources rather than clients, but to me it seems cleaner. >>> >>> Mark >>> From bourges.laurent at gmail.com Tue Oct 18 02:09:36 2011 From: bourges.laurent at gmail.com (=?ISO-8859-1?Q?Laurent_Bourg=E8s?=) Date: Tue, 18 Oct 2011 11:09:36 +0200 Subject: Hub resource hosting service? In-Reply-To: <4E9C85AF.3090303@nasa.gov> References: <4E9C85AF.3090303@nasa.gov> Message-ID: Dear all, I think this topic is quite related to my previous post related to URL handling: I was and I am still convinced that the SAMP implementation (hub) could provide an easier API to deal with documents given by URL location (file, http ...). My principal concern was to avoid developers to handle on their own the download mechanism (error prone) because it can be tricky: - timeouts / proxy / protocol errors - resource management (http / ftp connection, https security ...) - security i.e. privileges (tcp ports / file permissions ...) I agree tom's proposal that the SAMP hub could handle document retrieval / caching (in-memory or file) and provide an easy lookup method: - SampDocument GetDocument(URL) where the SampDocument class could then provide both string or file representation ... Comments are always welcome as this topic seems more active than mine, Laurent 2011/10/17 Tom McGlynn : > Hi Mark, > > I'm not particularly attached to any solution to the issue, but I do think > it would be desirable to enable work in a sandbox environment (both > JavaScript and untrusted applets) to use SAMP effectively. ?Your proposed > solution is to provide a standard caching mechanism through the WebSamp hub. > ?I've some comments on that... > > ?- The simplicity of the SAMP interface is very nice. ?Adding a whole new > layer to the interface to enable this kind of caching seems a bit painful, > but maybe that isn't necessary. ?We could allow the hub to transform the > message: ?E.g., the VOTable generating client would just send the message: > ? table.load.votable: content=... > to the hub and the hub would automatically save the content and send > ? table.load.votable: url=http://localhost:xxxx?... > to the consuming clients. ?The hub would send a response to the generating > client giving the URL. > > The rule could be that a hub is free to cache any 'content' element and > transform them into references to that content. ?Given that we are > implementing caching, I don't think this makes the hub any more complex, > we're just using the mtype syntax for the caching messages. Presumably > generated FITS data or images could be sent this way too. > > So there's no new protocol for clients to learn. ?That would be a nice > feature of a solution. > > ?- I'd be worried about security with a hub that is going to write into the > filesystem. ?I'd be more comfortable with a hub that provided say 100 MB of > cache space using only memory. > > > ?- In the simple, but probably very common, case of one generating client > and one consuming client, using a cache may be more expensive than just > sending the data, especially if we cache using a file. We have two > transactions, > ?generator->hub->write-to-file ?read-from-file->hub->consumer > in the cached case versus > ?generator->hub->consumer > in the direct case. ?This would be mitigated by an in-memory cache. > > ?- Similarly in the one generator to many consumers case, using URLs means > that the hub may need to read the cache many times independently to respond > to each of the clients as they request the reference URL. The direct > transfer may be able send data more efficiently in parallel. ?However, a > direct transfer doesn't give a consumer a chance to ignore a request, and > thus will be less efficient if there are clients that are are registered to > receive certain events, but in practice ignore them. > > > Hope you can work something out in Pune. > > ? ? ? ?Tom > > > > Mark Taylor wrote: >> >> Doug, >> >> On Sun, 16 Oct 2011, Douglas Tody wrote: >> >>> One thing discussed way back in the original SAMP discussions was >>> separating the message header from the message content, which would >>> allow the content to be anything and defined by the messaging protocol. >>> This would allow real data to be passed via the messaging system, >>> including even binary data. ?To keep things simple this was left out of >>> SAMP 1.0 but it was always considered a possibility for future versions. >>> Essentially what we have now is only the message header, with the >>> message type specifying what type of message it is, and parameters >>> providing a limited way to pass some data. >>> >>> Keeping messages small and lightweight is good, but what is best really >>> depends upon the use case and should not be constrained by the messaging >>> system. ?In real time applications for example (including desktop >>> applications such as displaying an image in real time) data can be >>> streamed in chunks and there is no possibility to build files which are >>> stored some place and then referenced by a URL or other pointer. ?While >>> limiting messages to pointing at bulk data is a good model in many cases >>> it is not the only one, and ultimately it is the application which >>> should decide what is the best model. >> >> Your analysis is quite right. ?SAMP as it stands is not a good candidate >> for a real time messaging system, and there are various other messaging >> scenarios that it is not targetted at. ?Certainly the application has >> to decide what's the best model, and having decided, it may >> decide that SAMP (as it stands) is not the system it should use to >> deliver that. ?Some future evolution of SAMP (AMP?) that is capable of >> addressing those issues is possible, and as you say was something >> we deliberately left open when we started to think about it. >> On the whole I don't feel that we are currently in a position where we >> have the need to take that further. ?My reluctance is mainly that the >> ways I can think of to do bulk data transfer within SAMP itself >> would complicate implementations quite a bit, and SAMP seems to >> be working reasonably well now for the kinds of things it's good at, >> with good takeup because it keeps things simple. ?But if you or >> others think different, we can re-open that discussion >> (though probably not in this Thursday's session, which is full >> already). >> >>> I don't much like the idea of the hub being turned into a general file >>> store - lots of complication there with persistence issues and such. >>> This has nothing to do with messaging. ?If such a capability is needed >>> it ought to be a different service, with messaging limited to only the >>> communication aspect. >> >> The Web Profile hub already provides some non-messaging services, in >> particular dereferencing of URLs, since the Web Profile is in the >> business of providing web applications with the things they need >> to be useful SAMP clients and can't get in any other way. >> However, I take your point. ?I don't want to push the resource >> hosting idea if other people are not keen. >> >> Mark >> >>> On Sun, 16 Oct 2011, Mark Taylor wrote: >>> >>>> Hi all. >>>> >>>> Here is an issue, and a possible solution, which was raised by some >>>> Web Profile implementation work that Tom is doing. ?We will discuss >>>> it in the Apps SAMP session on Thursday, but I wanted to air it here >>>> first to give people time to think about it, and also to solicit >>>> comments from people who are not in Pune. >>>> >>>> Tom said: >>>> >>>> On Fri, 14 Oct 2011, Tom McGlynn wrote: >>>> >>>>> A final largely unrelated issue... ?SAMP's reliance on passing VOTables >>>>> only >>>>> via URLs makes it difficult to adapt to JavaScript clients. ?They have >>>>> to >>>>> create relay URLs, stage results, or duplicate functionality on the >>>>> server >>>>> side. ?E.g., if we were to want to send a VOTable that had been >>>>> filtered >>>>> using VOView, there is no easy way to do this that I know of. ?I'm sure >>>>> this >>>>> was considered early on, but it's possible to do quite extensive >>>>> processing >>>>> in the web browser now and we may wish to consider supporting that, >>>>> especially with WebSAMP. ?E.g., we might allow either 'url' or >>>>> 'content' >>>>> tags in the message (where the later would be the actual VOTable >>>>> content) >>>>> or >>>>> defining equivalent mtypes for messages with table content. >>>> >>>> This hadn't occurred to me before. ?I had assumed that if web apps >>>> needed URLs to pass around they would come from the origin HTTP server. >>>> While an app could pass locally generated/manipulated data back to its >>>> origin server and send the resulting URL around in SAMP messages it's >>>> clearly not an efficient solution, since the data starts and ends on >>>> the client so doesn't want to have to go via a remote server. >>>> >>>> Tom's suggestion of adding content parameters as an alternative to url >>>> parameters in MTypes is possible, but a bit unpalatable for a couple >>>> of reasons. ?First it falls foul of the SHOULD (though not MUST) in >>>> SAMP section 3.3 that says: >>>> >>>> ? "General purpose MTypes SHOULD therefore be specified so that bulk >>>> ? ?data is not sent within the message. In general it is preferred to >>>> ? ?define a message parameter as the URL or filename of a potentially >>>> ? ?large file rather than as the inline text of the file itself." >>>> >>>> largely to prevent clients having to deal with very large chunks of XML, >>>> which is especially a problem if they DOMise them as a matter of >>>> routine. >>>> Second it means that a lot of new messages (or at least new versions >>>> of messages, using content rather than url parameters to specify >>>> content) would be flying around and old clients wouldn't be able to >>>> handle them, until they were upgraded accordingly. >>>> >>>> Another way to look at the problem is that Web Profile SAMP clients lack >>>> an >>>> efficient resource hosting mechanism (since, unlike Standard Profile >>>> clients, they are unable to write temporary files or host HTTP servers). >>>> This could be addressed by allowing the hub to provide such a mechanism. >>>> The most straightforward thing would be for the hub to provide a URL >>>> to which documents could be POSTed in return for a (201 Created - >>>> see RFC2616 sec 9.5) Location URL. ?Web clients which want >>>> to pass data to another client could therefore POST the docuement to >>>> the hub, get a reference URL in return, and pass that reference URL >>>> in messages. ?The hub arranges to redeliver the content to anyone >>>> who asks for it via that reference URL later. >>>> This also isn't perfect, since it means the hub has to manage some >>>> client resources rather than clients, but to me it seems cleaner. >>>> >>>> Mark >>>> > From m.b.taylor at bristol.ac.uk Wed Oct 26 06:27:57 2011 From: m.b.taylor at bristol.ac.uk (Mark Taylor) Date: Wed, 26 Oct 2011 14:27:57 +0100 (BST) Subject: JSAMP v1.3-1 released; please consider upgrading Message-ID: [posted to apps & apps-samp lists - apologies for duplication] Hi all, I have just released JSAMP v1.3-1. You can find it in the usual place: http://software.astrogrid.org/doc/jsamp/ The most important change is that when the Hub is started, the Web Profile now runs by default. The other significant thing is that the Web Profile now has a new security measure: by default only certain named MTypes (currently [samp.app.*, samp.msg.progress, bibcode.*, coord.*, image.*, spectrum.*, table.*, voresource.*]) can be sent from a web client; others are blocked by the hub. The intention of this is that by restricting messages sent to ones with known, harmless, semantics, hostile web clients are less likely to be able to do bad things. This restriction can be switched off by explicit user action. This measure is experimental and may be modified or withdrawn at some point in the future (opinions/feedback welcome). The full story on security measures can be found in the "Profiles" section of the docs. (There are a few other minor changes at this version which can be seen in the "Change Log" section). The conclusion from discussions at Pune was that web application deployers would like science users to be able to use Web SAMP capabilities now, which means they need to be running Web Profile capable hubs. The most effective way to make this happen is for those desktop applications which bundle a hub to configure it to run the Web Profile by default. For this reason, developers of applications which bundle a JSAMP hub are encouraged to upgrade to the current version in the near future. I will be starting the ball rolling by releasing a TOPCAT which runs JSAMP 1.3-1, and hence the Web Profile by default, shortly. To get an idea of how widely the Web Profile is going to be available to users, I would be interested to hear from other developers of VO apps, either directly or on list, if/when they plan to do the same. Any feedback, bug reports, requests for help, etc related to the new JSAMP version are welcome as always, Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor at bris.ac.uk +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/