From seaman at noao.edu Tue Dec 13 21:54:59 2011 From: seaman at noao.edu (Rob Seaman) Date: Tue, 13 Dec 2011 22:54:59 -0700 Subject: Abstract deadline for SPIE 2012 Message-ID: <9E3978E7-E7BB-4302-B7A6-0F5BE35AF74E@noao.edu> Howdy, The deadline for SPIE Astronomical Instrumentation 2012 is less than one week away (19 December). Please consider submitting an abstract to "Observatory Operations: Strategies, Processes, and Systems IV", the hottest ticket next Summer in Amsterdam: http://spie.org/as107 The Observatory Ops conference at the 2010 SPIE in San Diego featured a strong focus on time domain astronomy. This has if anything been strengthened for the 2012 meeting and other focus areas of interest to the VOEvent community have been added, including operations topics for the Virtual Observatory and for systems of networked telescopes. Several names from the Hotwired community should be familiar from the Program Committee. The future of astronomy over the next several decades will pivot on time domain issues. It is unnecessary to dwell on the connection of transient response observing modes to this year's Nobel prize in Physics. Accomplishing the challenging science goals in the time domain will require new infrastructure, new procedures, new systems, new networks. To be successful, all of these will require a new commitment to coherent operational strategies. Autonomous technologies such as robotics and semantic technologies such as astroinformatics - and in particular VOEvent and related VO and exo-VO standards such as RTML and SimpleTimeSeries - need to move from the ivory towers to the telescope domes and downtown computer labs. SPIE in general and the Observatory Operations conference in particular are where creative ideas are brought into the real world. Please forward far and wide. Rob Seaman National Optical Astronomy Observatory Tucson, AZ -------------- next part -------------- An HTML attachment was scrubbed... URL: From roy at caltech.edu Tue Dec 20 08:11:35 2011 From: roy at caltech.edu (Roy Williams) Date: Tue, 20 Dec 2011 08:11:35 -0800 Subject: 0MQ: The Intelligent Transport Layer Message-ID: <4EF0B3B7.2050109@caltech.edu> As we continue to wrestle with VOEvent transport, new (and simpler) protocols arrive. This looks promising .... http://www.zeromq.org/ Anyone have any experience with this? Roy --- Caltech LIGO roy at caltech.edu 626 395 3670 From swinbank at transientskp.org Tue Dec 20 11:34:05 2011 From: swinbank at transientskp.org (John Swinbank) Date: Tue, 20 Dec 2011 19:34:05 +0000 Subject: 0MQ: The Intelligent Transport Layer In-Reply-To: <4EF0B3B7.2050109@caltech.edu> References: <4EF0B3B7.2050109@caltech.edu> Message-ID: <3B27C7BE-E972-4CAB-9C41-F34006BBBB9C@transientskp.org> Hi Roy, On 20 Dec 2011, at 16:11, Roy Williams wrote: > As we continue to wrestle with VOEvent transport, new (and simpler) protocols arrive. This looks promising .... > > http://www.zeromq.org/ > > Anyone have any experience with this? I've used ZeroMQ (by way of its Python bindings) for shifting VOEvents around within our local network. It worked well enough, and was trivially easy to set up: literally just a couple of lines of code. I haven't yet investigated any of the advanced capabilities (fan out, pubsub, etc), but I have mused to myself about their applicability within our LOFAR systems and perhaps the wider community. I'd certainly be interested in further discussion. Cheers, John From aa at astro.ex.ac.uk Tue Dec 20 15:31:09 2011 From: aa at astro.ex.ac.uk (Alasdair Allan) Date: Tue, 20 Dec 2011 23:31:09 +0000 Subject: 0MQ: The Intelligent Transport Layer In-Reply-To: <4EF0B3B7.2050109@caltech.edu> References: <4EF0B3B7.2050109@caltech.edu> Message-ID: I actually quite like the look of ?MQ, although I've yet to come across a situation where I've had the opportunity to use it in anger, I've had a play around with it. Suffice to say some of the language bindings aren't quite as feature full and bug free as others... Definitely interesting. Al. -- Sent from my iPad. On 20 Dec 2011, at 16:11, Roy Williams wrote: > As we continue to wrestle with VOEvent transport, new (and simpler) protocols arrive. This looks promising .... > > http://www.zeromq.org/ > > Anyone have any experience with this? > Roy > > --- > Caltech LIGO > roy at caltech.edu > 626 395 3670 From mjg at cacr.caltech.edu Tue Dec 20 15:37:56 2011 From: mjg at cacr.caltech.edu (Matthew Graham) Date: Tue, 20 Dec 2011 15:37:56 -0800 Subject: 0MQ: The Intelligent Transport Layer In-Reply-To: <4EF0B3B7.2050109@caltech.edu> References: <4EF0B3B7.2050109@caltech.edu> Message-ID: Hi, I guess my reaction is what's wrong with XMPP/Jabber that this solves? Cheers, Matthew On Dec 20, 2011, at 8:11 AM, Roy Williams wrote: > As we continue to wrestle with VOEvent transport, new (and simpler) protocols arrive. This looks promising .... > > http://www.zeromq.org/ > > Anyone have any experience with this? > Roy > > --- > Caltech LIGO > roy at caltech.edu > 626 395 3670 > From aa at astro.ex.ac.uk Tue Dec 20 15:46:22 2011 From: aa at astro.ex.ac.uk (Alasdair Allan) Date: Tue, 20 Dec 2011 23:46:22 +0000 Subject: 0MQ: The Intelligent Transport Layer In-Reply-To: References: <4EF0B3B7.2050109@caltech.edu> Message-ID: XMPP/Jabber is pubsub, that's not necessarily the only architecture you want to think about for event transport. Al. -- Sent from my iPad. On 20 Dec 2011, at 23:37, Matthew Graham wrote: > Hi, > > I guess my reaction is what's wrong with XMPP/Jabber that this solves? > > Cheers, > > Matthew > > On Dec 20, 2011, at 8:11 AM, Roy Williams wrote: > >> As we continue to wrestle with VOEvent transport, new (and simpler) protocols arrive. This looks promising .... >> >> http://www.zeromq.org/ >> >> Anyone have any experience with this? >> Roy >> >> --- >> Caltech LIGO >> roy at caltech.edu >> 626 395 3670 >> > From mjg at cacr.caltech.edu Tue Dec 20 15:51:02 2011 From: mjg at cacr.caltech.edu (Matthew Graham) Date: Tue, 20 Dec 2011 15:51:02 -0800 Subject: 0MQ: The Intelligent Transport Layer In-Reply-To: References: <4EF0B3B7.2050109@caltech.edu> Message-ID: Hi, Agreed but considerations should include: - Is there conceptually anything wrong with the existing infrastructure? - Is it just a case that the current tools are not good enough and no-one has been bothered enough to dedicate some time to sorting this out? - What exactly are our use cases for event transport? Cheers, Matthew On Dec 20, 2011, at 3:46 PM, Alasdair Allan wrote: > XMPP/Jabber is pubsub, that's not necessarily the only architecture you want to think about for event transport. > > Al. > -- > Sent from my iPad. > > On 20 Dec 2011, at 23:37, Matthew Graham wrote: > >> Hi, >> >> I guess my reaction is what's wrong with XMPP/Jabber that this solves? >> >> Cheers, >> >> Matthew >> >> On Dec 20, 2011, at 8:11 AM, Roy Williams wrote: >> >>> As we continue to wrestle with VOEvent transport, new (and simpler) protocols arrive. This looks promising .... >>> >>> http://www.zeromq.org/ >>> >>> Anyone have any experience with this? >>> Roy >>> >>> --- >>> Caltech LIGO >>> roy at caltech.edu >>> 626 395 3670 >>> >> > From swinbank at transientskp.org Tue Dec 20 17:14:14 2011 From: swinbank at transientskp.org (John Swinbank) Date: Wed, 21 Dec 2011 01:14:14 +0000 Subject: 0MQ: The Intelligent Transport Layer In-Reply-To: References: <4EF0B3B7.2050109@caltech.edu> Message-ID: <98C75CD2-30B0-4BD7-BCBD-5538D7443272@transientskp.org> Hello, Naive question: what existing Jabber infrastructure is there? As of now, there's nothing at which refers to Jabber. Readers who want to "get started with? subscribing to streams" are directed to SkyAlert (which ? correct me if I'm wrong ? is all web based) or to DC3 Dakota (which only covers the simple TCP transport). Googling finds me , but I'm not sure how seriously to take that: my understanding was that VOEventNet is defunct. At any rate, I tried the Java client and didn't get much sense out of it (specifying "jabber.server=moriori.cacr.caltech.edu" as per the help causes it to return "Could not connect to null:5222.: (504)"). Any other pointers appreciated. Cheers, John On 20 Dec 2011, at 23:51, Matthew Graham wrote: > Hi, > > Agreed but considerations should include: > > - Is there conceptually anything wrong with the existing infrastructure? > - Is it just a case that the current tools are not good enough and no-one has been bothered enough to dedicate some time to sorting this out? > - What exactly are our use cases for event transport? > > Cheers, > > Matthew > > On Dec 20, 2011, at 3:46 PM, Alasdair Allan wrote: > >> XMPP/Jabber is pubsub, that's not necessarily the only architecture you want to think about for event transport. >> >> Al. >> -- >> Sent from my iPad. >> >> On 20 Dec 2011, at 23:37, Matthew Graham wrote: >> >>> Hi, >>> >>> I guess my reaction is what's wrong with XMPP/Jabber that this solves? >>> >>> Cheers, >>> >>> Matthew >>> >>> On Dec 20, 2011, at 8:11 AM, Roy Williams wrote: >>> >>>> As we continue to wrestle with VOEvent transport, new (and simpler) protocols arrive. This looks promising .... >>>> >>>> http://www.zeromq.org/ >>>> >>>> Anyone have any experience with this? >>>> Roy >>>> >>>> --- >>>> Caltech LIGO >>>> roy at caltech.edu >>>> 626 395 3670 >>>> >>> >> > From mjg at cacr.caltech.edu Tue Dec 20 17:48:32 2011 From: mjg at cacr.caltech.edu (Matthew Graham) Date: Tue, 20 Dec 2011 17:48:32 -0800 Subject: 0MQ: The Intelligent Transport Layer In-Reply-To: <98C75CD2-30B0-4BD7-BCBD-5538D7443272@transientskp.org> References: <4EF0B3B7.2050109@caltech.edu> <98C75CD2-30B0-4BD7-BCBD-5538D7443272@transientskp.org> Message-ID: <3EA0D207-2DA1-4818-A09F-459D0C1F7DC8@cacr.caltech.edu> Hi John, Several projects are using Jabber in-house so there is whatever infrastructure they have. However, in terms of general user stuff, it pretty much is what we wrote for VOEventNet which should just need updating. Cheers, Matthew On Dec 20, 2011, at 5:14 PM, John Swinbank wrote: > Hello, > > Naive question: what existing Jabber infrastructure is there? > > As of now, there's nothing at which refers to Jabber. Readers who want to "get started with? subscribing to streams" are directed to SkyAlert (which ? correct me if I'm wrong ? is all web based) or to DC3 Dakota (which only covers the simple TCP transport). > > Googling finds me , but I'm not sure how seriously to take that: my understanding was that VOEventNet is defunct. At any rate, I tried the Java client and didn't get much sense out of it (specifying "jabber.server=moriori.cacr.caltech.edu" as per the help causes it to return "Could not connect to null:5222.: (504)"). > > Any other pointers appreciated. > > Cheers, > > John > > On 20 Dec 2011, at 23:51, Matthew Graham wrote: > >> Hi, >> >> Agreed but considerations should include: >> >> - Is there conceptually anything wrong with the existing infrastructure? >> - Is it just a case that the current tools are not good enough and no-one has been bothered enough to dedicate some time to sorting this out? >> - What exactly are our use cases for event transport? >> >> Cheers, >> >> Matthew >> >> On Dec 20, 2011, at 3:46 PM, Alasdair Allan wrote: >> >>> XMPP/Jabber is pubsub, that's not necessarily the only architecture you want to think about for event transport. >>> >>> Al. >>> -- >>> Sent from my iPad. >>> >>> On 20 Dec 2011, at 23:37, Matthew Graham wrote: >>> >>>> Hi, >>>> >>>> I guess my reaction is what's wrong with XMPP/Jabber that this solves? >>>> >>>> Cheers, >>>> >>>> Matthew >>>> >>>> On Dec 20, 2011, at 8:11 AM, Roy Williams wrote: >>>> >>>>> As we continue to wrestle with VOEvent transport, new (and simpler) protocols arrive. This looks promising .... >>>>> >>>>> http://www.zeromq.org/ >>>>> >>>>> Anyone have any experience with this? >>>>> Roy >>>>> >>>>> --- >>>>> Caltech LIGO >>>>> roy at caltech.edu >>>>> 626 395 3670 >>>>> >>>> >>> >> > From roy at caltech.edu Tue Dec 20 19:45:47 2011 From: roy at caltech.edu (Roy Williams) Date: Tue, 20 Dec 2011 19:45:47 -0800 Subject: 0MQ: The Intelligent Transport Layer In-Reply-To: References: <4EF0B3B7.2050109@caltech.edu> Message-ID: <4EF1566B.60907@caltech.edu> On 12/20/11 3:51 PM, Matthew Graham wrote: > Hi, > > Agreed but considerations should include: > > - Is there conceptually anything wrong with the existing > infrastructure? There are protocols in use for transporting VOEvents or other formats. -- Some are point to point: http POST, email a la MPC, GCN circulars. -- Some are publish-subscribe (pubsub): Jabber/XMPP, original GCN socket, VO-GCN broker, Dakota broker. The 0MQ people say they can do both kinds of message. > - Is it just a case that the current tools are not good enough and > no-one has been bothered enough to dedicate some time to sorting this > out? I've spent enough time trying to get Jabber/pubsub behaving itself. Sending an XML packet (VOEVent) within another XML protocol (Jabber) is delicate at best, especially when XML compliance is required. - What exactly are our use cases for event transport? Combination of push and pull transport Push Point-to-point: -- event goes direct to recipient -- personal selection, personal feed, handling emails -- driving remote annotators -- sender handles identity mgmt Push Publish-subscribe: -- following all events from a given stream or substream -- event travels to server, broadcast to subscribers -- server handles identity mgmt Pull: -- a query is sent to an event repository -- a set of events is returned Authoring: with identity management Query services result is a set of events Replication: cloning a repository elsewhere Roy --- Caltech LIGO roy at caltech.edu 626 395 3670 From roy at caltech.edu Tue Dec 20 19:56:28 2011 From: roy at caltech.edu (Roy Williams) Date: Tue, 20 Dec 2011 19:56:28 -0800 Subject: 0MQ: The Intelligent Transport Layer In-Reply-To: <3EA0D207-2DA1-4818-A09F-459D0C1F7DC8@cacr.caltech.edu> References: <4EF0B3B7.2050109@caltech.edu> <98C75CD2-30B0-4BD7-BCBD-5538D7443272@transientskp.org> <3EA0D207-2DA1-4818-A09F-459D0C1F7DC8@cacr.caltech.edu> Message-ID: <4EF158EC.1000705@caltech.edu> LIGO supports a open-source Jabber API called LVAlert: https://www.lsc-group.phys.uwm.edu/daswg/projects/lvalert.html The software is python, and can utilize any Jabber server. There is lvalert_send, lvalert_listen; then lvalert_admin for setting up nodes and subscribing etc. The same people (University of Wisconsin, Milwaukee) run a server at lvalert.phys.uwm.edu. Roy --- Caltech LIGO roy at caltech.edu 626 395 3670 On 12/20/11 5:48 PM, Matthew Graham wrote: > Hi John, > > Several projects are using Jabber in-house so there is whatever infrastructure they have. However, in terms of general user stuff, it pretty much is what we wrote for VOEventNet which should just need updating. > > Cheers, > > Matthew > > > On Dec 20, 2011, at 5:14 PM, John Swinbank wrote: > >> Hello, >> >> Naive question: what existing Jabber infrastructure is there? >> >> As of now, there's nothing at which refers to Jabber. Readers who want to "get started with? subscribing to streams" are directed to SkyAlert (which ? correct me if I'm wrong ? is all web based) or to DC3 Dakota (which only covers the simple TCP transport). >> >> Googling finds me, but I'm not sure how seriously to take that: my understanding was that VOEventNet is defunct. At any rate, I tried the Java client and didn't get much sense out of it (specifying "jabber.server=moriori.cacr.caltech.edu" as per the help causes it to return "Could not connect to null:5222.: (504)"). >> >> Any other pointers appreciated. >> >> Cheers, >> >> John >> >> On 20 Dec 2011, at 23:51, Matthew Graham wrote: >> >>> Hi, >>> >>> Agreed but considerations should include: >>> >>> - Is there conceptually anything wrong with the existing infrastructure? >>> - Is it just a case that the current tools are not good enough and no-one has been bothered enough to dedicate some time to sorting this out? >>> - What exactly are our use cases for event transport? >>> >>> Cheers, >>> >>> Matthew >>> >>> On Dec 20, 2011, at 3:46 PM, Alasdair Allan wrote: >>> >>>> XMPP/Jabber is pubsub, that's not necessarily the only architecture you want to think about for event transport. >>>> >>>> Al. >>>> -- >>>> Sent from my iPad. >>>> >>>> On 20 Dec 2011, at 23:37, Matthew Graham wrote: >>>> >>>>> Hi, >>>>> >>>>> I guess my reaction is what's wrong with XMPP/Jabber that this solves? >>>>> >>>>> Cheers, >>>>> >>>>> Matthew >>>>> >>>>> On Dec 20, 2011, at 8:11 AM, Roy Williams wrote: >>>>> >>>>>> As we continue to wrestle with VOEvent transport, new (and simpler) protocols arrive. This looks promising .... >>>>>> >>>>>> http://www.zeromq.org/ >>>>>> >>>>>> Anyone have any experience with this? >>>>>> Roy >>>>>> >>>>>> --- >>>>>> Caltech LIGO >>>>>> roy at caltech.edu >>>>>> 626 395 3670 >>>>>> >>>>> >>>> >>> >> > From mjg at cacr.caltech.edu Tue Dec 20 19:57:29 2011 From: mjg at cacr.caltech.edu (Matthew Graham) Date: Tue, 20 Dec 2011 19:57:29 -0800 Subject: 0MQ: The Intelligent Transport Layer In-Reply-To: <4EF1566B.60907@caltech.edu> References: <4EF0B3B7.2050109@caltech.edu> <4EF1566B.60907@caltech.edu> Message-ID: <2E36A2D5-22DB-46A6-911A-547A2C407756@cacr.caltech.edu> Hi Roy, There's a definite conflation in some of this between what is wanted from an event transport protocol and functionality that you layer on top of that. Matthew > - What exactly are our use cases for event transport? > > Combination of push and pull transport > > Push Point-to-point: > -- event goes direct to recipient > -- personal selection, personal feed, handling emails > -- driving remote annotators > -- sender handles identity mgmt > > Push Publish-subscribe: > -- following all events from a given stream or substream > -- event travels to server, broadcast to subscribers > -- server handles identity mgmt > > Pull: > -- a query is sent to an event repository > -- a set of events is returned > > > Authoring: > with identity management > > Query services > result is a set of events > > Replication: > cloning a repository elsewhere > > > Roy > > --- > Caltech LIGO > roy at caltech.edu > 626 395 3670 > From seaman at noao.edu Tue Dec 20 20:19:50 2011 From: seaman at noao.edu (Rob Seaman) Date: Tue, 20 Dec 2011 21:19:50 -0700 Subject: 0MQ: The Intelligent Transport Layer In-Reply-To: References: <4EF0B3B7.2050109@caltech.edu> Message-ID: <1B8E3C9F-A4CD-4631-B57B-D26A4EF93F4E@noao.edu> Questions, we got questions... On Dec 20, 2011, at 4:51 PM, Matthew Graham wrote: > - Is there conceptually anything wrong with the existing infrastructure? > - Is it just a case that the current tools are not good enough and no-one has been bothered enough to dedicate some time to sorting this out? > - What exactly are our use cases for event transport? On Dec 20, 2011, at 6:14 PM, John Swinbank wrote: > Naive question: what existing Jabber infrastructure is there? > ...my understanding was that VOEventNet is defunct [?] (implied question mark at the end, there) VOEventNet is neither "existing" nor "defunct". Infrastructure has an evolutionary lifecycle. The first VOEvent transport prototype was a variation of vanilla TCP. The second (at the 2005 NVO summer school) was something called Elvin (an orphan technology since around 2007). In addition to Jabber there was a prototype using Java messaging. RSS as well as the web. Others like Roy lists. VOEvent appears to be doing pretty well as a brand name as well as a technology. One might therefore expect VOEventNet to survive as a generic notion of the deployed infrastructure as it evolves and ramifies, independently of grants and projects. At any given epoch we will have one or more deployed functional systems and/or prototypes. A key requirement will be interoperability between systems at a single epoch and between one epoch and the next. The network will have bridges in other words. Two of the current VOEvent-compatible transport systems are SkyAlert and GCN. These are responsive to different sets of requirements and overlapping but differing stakeholders. We know that future transport must scale upwards dramatically. Personally I suspect none of the current transport layers will be up to that scalability. More to the point we will need dedicated and deployed infrastructure - permanently funded and maintained servers and network links and operations staff. On the other hand I suspect that we already know the future players out of whom this enhanced commitment will grow. Which is to say that the people issues are at least as challenging as the technology issues. Rob From jean-paul.lefevre at cea.fr Wed Dec 21 01:09:11 2011 From: jean-paul.lefevre at cea.fr (=?utf-8?Q?Jean-Paul_Le_F=C3=A8vre?=) Date: Wed, 21 Dec 2011 10:09:11 +0100 Subject: 0MQ: The Intelligent Transport Layer In-Reply-To: <4EF0B3B7.2050109@caltech.edu> References: <4EF0B3B7.2050109@caltech.edu> Message-ID: <000901ccbfc0$3438de70$9caa9b50$@cea.fr> As far as I'm concerned I started playing with html5 websockets using the Autobahn python API (http://www.tavendo.de/autobahn/) At this point most browsers can handle websockets communications so it's worth experimenting. -- Jean-Paul Le F?vre - jean-paul.lefevre at cea.fr Svom FSC project manager - CEA Saclay Irfu T?l : +33 1 69 08 34 58 -----Message d'origine----- De : voevent-bounces at ivoa.net [mailto:voevent-bounces at ivoa.net] De la part de Roy Williams Envoy? : mardi 20 d?cembre 2011 17:12 ? : IVOA List VOEvent Objet : 0MQ: The Intelligent Transport Layer As we continue to wrestle with VOEvent transport, new (and simpler) protocols arrive. This looks promising .... http://www.zeromq.org/ Anyone have any experience with this? Roy --- Caltech LIGO roy at caltech.edu 626 395 3670 From swinbank at transientskp.org Thu Dec 22 15:45:22 2011 From: swinbank at transientskp.org (John Swinbank) Date: Thu, 22 Dec 2011 23:45:22 +0000 Subject: 0MQ: The Intelligent Transport Layer In-Reply-To: <4EF0B3B7.2050109@caltech.edu> References: <4EF0B3B7.2050109@caltech.edu> Message-ID: <794E10D7-688C-4A81-B21F-7219BBB966FB@transientskp.org> Since I had some free time earlier, I wrote a simple gateway to rebroadcast VOEvents from the voevent.dc3.com broker using ZeroMQ pubsub. Should you be sufficiently enthusiastic, you can (for the time being, at least) subscribe to the ZeroMQ system at voevent.transientskp.org port 8089. The code for the gateway, along with a trivial Python script which will connect to the ZeroMQ feed and dump events received to stdout, is available from . It's very much at the proof-of-concept level. I claim no expertise on ZeroMQ, but my experience with the above indicates that its pubsub system is probably too simplistic for deployment as a public VOEvent feed, but I could certainly imagine it being useful within projects. ZeroMQ as a whole is pretty easy to work with and flexible; potentially, I could imagine it being of wider interest to the VOEvent community. Cheers, John On 20 Dec 2011, at 16:11, Roy Williams wrote: > As we continue to wrestle with VOEvent transport, new (and simpler) protocols arrive. This looks promising .... > > http://www.zeromq.org/ > > Anyone have any experience with this? > Roy > > --- > Caltech LIGO > roy at caltech.edu > 626 395 3670 From swinbank at transientskp.org Thu Dec 22 16:20:00 2011 From: swinbank at transientskp.org (John Swinbank) Date: Fri, 23 Dec 2011 00:20:00 +0000 Subject: 0MQ: The Intelligent Transport Layer In-Reply-To: <3EA0D207-2DA1-4818-A09F-459D0C1F7DC8@cacr.caltech.edu> References: <4EF0B3B7.2050109@caltech.edu> <98C75CD2-30B0-4BD7-BCBD-5538D7443272@transientskp.org> <3EA0D207-2DA1-4818-A09F-459D0C1F7DC8@cacr.caltech.edu> Message-ID: <04306694-E9F5-4ED3-971D-426FB0DF8556@transientskp.org> Hello, It seems to me there's a large gap between whatever internal infrastructure individual projects are using and a common set of tools and infrastructure the community at large has access to. Indeed, setting aside the specifics of the transport mechanism, it strikes me that there's very little infrastructure available to the end user who washes up at : SkyAlert is great, but anybody who wants to access a stream of events programmatically looks to be pretty much on their own barring the fantastic efforts of Bob Denny. I'm not clear if there are any Jabber feeds actually available (does the server at Caltech still work?) or any TCP servers available to the general public other than the one at dc3.com. That seems to me to be an unfortunate state of affairs. Am I missing something? Is there other public infrastructure out there? Is it listed somewhere? If the answers to the last two questions are "yes" and "no" respectively, please enlighten me and I'll be happy to update the wiki with the good news. Cheers, John On 21 Dec 2011, at 01:48, Matthew Graham wrote: > Hi John, > > Several projects are using Jabber in-house so there is whatever infrastructure they have. However, in terms of general user stuff, it pretty much is what we wrote for VOEventNet which should just need updating. > > Cheers, > > Matthew > > > On Dec 20, 2011, at 5:14 PM, John Swinbank wrote: > >> Hello, >> >> Naive question: what existing Jabber infrastructure is there? >> >> As of now, there's nothing at which refers to Jabber. Readers who want to "get started with? subscribing to streams" are directed to SkyAlert (which ? correct me if I'm wrong ? is all web based) or to DC3 Dakota (which only covers the simple TCP transport). >> >> Googling finds me , but I'm not sure how seriously to take that: my understanding was that VOEventNet is defunct. At any rate, I tried the Java client and didn't get much sense out of it (specifying "jabber.server=moriori.cacr.caltech.edu" as per the help causes it to return "Could not connect to null:5222.: (504)"). >> >> Any other pointers appreciated. >> >> Cheers, >> >> John >> >> On 20 Dec 2011, at 23:51, Matthew Graham wrote: >> >>> Hi, >>> >>> Agreed but considerations should include: >>> >>> - Is there conceptually anything wrong with the existing infrastructure? >>> - Is it just a case that the current tools are not good enough and no-one has been bothered enough to dedicate some time to sorting this out? >>> - What exactly are our use cases for event transport? >>> >>> Cheers, >>> >>> Matthew >>> >>> On Dec 20, 2011, at 3:46 PM, Alasdair Allan wrote: >>> >>>> XMPP/Jabber is pubsub, that's not necessarily the only architecture you want to think about for event transport. >>>> >>>> Al. >>>> -- >>>> Sent from my iPad. >>>> >>>> On 20 Dec 2011, at 23:37, Matthew Graham wrote: >>>> >>>>> Hi, >>>>> >>>>> I guess my reaction is what's wrong with XMPP/Jabber that this solves? >>>>> >>>>> Cheers, >>>>> >>>>> Matthew >>>>> >>>>> On Dec 20, 2011, at 8:11 AM, Roy Williams wrote: >>>>> >>>>>> As we continue to wrestle with VOEvent transport, new (and simpler) protocols arrive. This looks promising .... >>>>>> >>>>>> http://www.zeromq.org/ >>>>>> >>>>>> Anyone have any experience with this? >>>>>> Roy >>>>>> >>>>>> --- >>>>>> Caltech LIGO >>>>>> roy at caltech.edu >>>>>> 626 395 3670 >>>>>> >>>>> >>>> >>> >> > From roy at caltech.edu Thu Dec 22 17:30:02 2011 From: roy at caltech.edu (Roy Williams) Date: Thu, 22 Dec 2011 17:30:02 -0800 Subject: 0MQ: The Intelligent Transport Layer In-Reply-To: <04306694-E9F5-4ED3-971D-426FB0DF8556@transientskp.org> References: <4EF0B3B7.2050109@caltech.edu> <98C75CD2-30B0-4BD7-BCBD-5538D7443272@transientskp.org> <3EA0D207-2DA1-4818-A09F-459D0C1F7DC8@cacr.caltech.edu> <04306694-E9F5-4ED3-971D-426FB0DF8556@transientskp.org> Message-ID: <4EF3D99A.4090705@caltech.edu> John You ask the perfect questions! > anybody who wants to access a stream of events > programmatically looks to be pretty much on their own barring the > fantastic efforts of Bob Denny. There is a python library for VOEvent, at http://lib.skyalert.org/VOEventLib/ There is a VOEvent validation service, at http://www.skyalert.org/submit/ There are VOEvent feeds, at http://www.skyalert.org/feeds/ Which can be used to fetch the event as either XML or JSON. You can also build your own custom feed, or get the results as JSON or XML. There is also a GCN broker that puts out VOEvents, it is in test mode (talk to Scott Barthelmy) I'm not clear if there are any Jabber > feeds actually available (does the server at Caltech still work?) or > any TCP servers available to the general public other than the one at > dc3.com. The jabber server at moriori.cacr.caltech.edu has been running very reliably for five years, for more info see http://voeventnet.caltech.edu/software/index.html Hope this helps Roy --- Caltech LIGO roy at caltech.edu 626 395 3670 From seaman at noao.edu Thu Dec 22 18:06:35 2011 From: seaman at noao.edu (Rob Seaman) Date: Thu, 22 Dec 2011 19:06:35 -0700 Subject: 0MQ: The Intelligent Transport Layer In-Reply-To: <04306694-E9F5-4ED3-971D-426FB0DF8556@transientskp.org> References: <4EF0B3B7.2050109@caltech.edu> <98C75CD2-30B0-4BD7-BCBD-5538D7443272@transientskp.org> <3EA0D207-2DA1-4818-A09F-459D0C1F7DC8@cacr.caltech.edu> <04306694-E9F5-4ED3-971D-426FB0DF8556@transientskp.org> Message-ID: > Am I missing something? Is there other public infrastructure out there? Is it listed somewhere? If the answers to the last two questions are "yes" and "no" respectively, please enlighten me and I'll be happy to update the wiki with the good news. The wiki is mostly an internal site for the WG. Voevent.org currently points to the wiki, but could be easily redirected to a different front page. This would be a good time for a concise page providing answers to questions to questions such as above. In the mean time this was one of the motivations for the Hotwired book: http://hotwireduniverse.org Regarding jump-starting the deployment of VOEvent-compliant infrastructure, to a large extent this is an issue of reaching critical mass. Several major players (LSST, LOFAR, LIGO, GAIA...) are in the mix and we each tend to be aware of the context for those projects we're involved with. One thread has been from SkyAlert to the VAO Transient Facility via the VAO/LSST MOU. We could accommodate other strings on our bow. Roy mentioned our long-time partner GCN which has been alpha testing an update to their server for about a month. Related topics will likely also be a key focus of Hotwired III, sometime in 2012/13. Rob From rdenny at dc3.com Mon Dec 26 20:08:11 2011 From: rdenny at dc3.com (Bob Denny) Date: Mon, 26 Dec 2011 21:08:11 -0700 Subject: 0MQ: The Intelligent Transport Layer In-Reply-To: <22772383.57033.1324397598523.JavaMail.root@m11> References: <22772383.57033.1324397598523.JavaMail.root@m11> Message-ID: <4EF944AB.3000600@dc3.com> An HTML attachment was scrubbed... URL: From seaman at noao.edu Mon Dec 26 21:02:29 2011 From: seaman at noao.edu (Rob Seaman) Date: Mon, 26 Dec 2011 22:02:29 -0700 Subject: 0MQ: The Intelligent Transport Layer In-Reply-To: <4EF944AB.3000600@dc3.com> References: <22772383.57033.1324397598523.JavaMail.root@m11> <4EF944AB.3000600@dc3.com> Message-ID: <010ECC39-5661-4EA9-9E25-3BD619FF3C37@noao.edu> Hi Bob, I can't comment on either your head or wall :-) but nothing is wrong with either the transport note or Dakota. And however simple zeromg is, it certainly can't be simpler than vanilla TCP. My suspicion is that vTCP and thus Dakota will be supported indefinitely, but that more boutique protocols like zeromg, elvin, Java messaging - and perhaps jabber - will come and go. As has been said before, this implies the creation and maintenance of bridges in the deployed VOEventNet for any future network. VOEvent itself is transport neutral by design. Will vTCP scale to the current LSST baseline of two million alerts per night? Will other technologies? In the near term, projects now nearing completion or in commissioning will deliver many thousands of events nightly in the next few years. These several surveys are each likely to produce numerous streams of events, filtered or correlated in scientifically interesting ways. Registries - either explicitly through the VO or implicitly through the diverse stakeholders for each survey - will be used to discover streams and associated metadata and per-project schemata (again, either explicit or implicit). All of these issues and the related science requirements from the diverse event-producers and consumers will have a hand in evolving the deployed infrastructure needed to scale to the future event rates and large numbers of future subscribers of different sorts. There are a lot of different options for how that infrastructure will be funded and operated. It is quite likely that different parties will provide access to different classes of subscribers as well as classes of event authors (both humans and autonomous agents). For instance, I would think it an almost foregone conclusion that your customers would most robustly and efficiently subscribe through your software, whatever the network topology of future event brokers. Whether one or more of those brokers includes bridging capabilities from some yet-to-be-specified transport protocol to vTCP seems (at this end) to be an implementation detail. Or alternately, perhaps the native protocol of the event backbone will remain vTCP and no bridging will be needed by Dakota clients, but other subscribers will choose some trendy boutique notification service using either a push or pull paradigm. Whichever it is, I would be flabbergasted if the astronomical community in general or IVOA in particular sought to forbid boundary components of the ramifying VOEventNet from including bridging technology to zeromg or any other transport protocol. Rob -- On Dec 26, 2011, at 9:08 PM, Bob Denny wrote: > Magic Question: What is wrong with > > http://www.ivoa.net/Documents/Notes/VOEventTransport/ > > and > > http://www.ivoa.net/Documents/Notes/DakotaBroker/ > > (which is also a sender and a receiver that is cross-platform) > > ? > > Did I bang my head against the wall for the THIRD time since 2006??? > > -- Bob > > Roy W: >> >> As we continue to wrestle with VOEvent transport, new (and simpler) >> protocols arrive. This looks promising .... >> >> http://www.zeromq.org/ >> >> Anyone have any experience with this? >> Roy >> >> --- >> Caltech LIGO >> roy at caltech.edu >> 626 395 3670 -------------- next part -------------- An HTML attachment was scrubbed... URL: From rdenny at dc3.com Mon Dec 26 22:17:30 2011 From: rdenny at dc3.com (Bob) Date: Tue, 27 Dec 2011 00:17:30 -0600 (CST) Subject: 0MQ: The Intelligent Transport Layer Message-ID: <16827241.1348.1324966650909.JavaMail.root@web04> May write more soon (or maybe I'm done with this), but: (1) Is "transport neutrality" like "gauge neutrality" for a railroad? What is the benefit? The meat is in the messages not the plumbing. (2) Is "undefined and ever expanding incompatible transport" standing in the way of adoption? I suspect the uncertainty is a huge barrier to adoption. (3) Ah, the LSST spectre... I say again "Who is going to USE (or even LOOK AT) 2 million alerts per night? What's the point? It seems like the stick that the sandboxers use to beat the realists over the head with. That head again ;-) And a sentence ending in a preposition. Sorry. (4) Is transport research and development for VOEvent a vehicle for garnering additional grant (and therefore grocery/housepayment) money? If so, stability and adoption are hopeless. -- Bob On Mon Dec 26 23:02:29 CST 2011, Rob Seaman wrote: > Hi Bob, > > I can't comment on either your head or wall :-) but nothing is > wrong with either the transport note or Dakota. And however > simple zeromg is, it certainly can't be simpler than vanilla TCP. > My suspicion is that vTCP and thus Dakota will be supported > indefinitely, but that more boutique protocols like zeromg, > elvin, Java messaging - and perhaps jabber - will come and go. > As has been said before, this implies the creation and > maintenance of bridges in the deployed VOEventNet for any future > network. VOEvent itself is transport neutral by design. > > Will vTCP scale to the current LSST baseline of two million > alerts per night? Will other technologies? In the near term, > projects now nearing completion or in commissioning will deliver > many thousands of events nightly in the next few years. These > several surveys are each likely to produce numerous streams of > events, filtered or correlated in scientifically interesting > ways. Registries - either explicitly through the VO or > implicitly through the diverse stakeholders for each survey - > will be used to discover streams and associated metadata and > per-project schemata (again, either explicit or implicit). > > All of these issues and the related science requirements from the > diverse event-producers and consumers will have a hand in > evolving the deployed infrastructure needed to scale to the > future event rates and large numbers of future subscribers of > different sorts. There are a lot of different options for how > that infrastructure will be funded and operated. It is quite > likely that different parties will provide access to different > classes of subscribers as well as classes of event authors (both > humans and autonomous agents). For instance, I would think it an > almost foregone conclusion that your customers would most > robustly and efficiently subscribe through your software, > whatever the network topology of future event brokers. Whether > one or more of those brokers includes bridging capabilities from > some yet-to-be-specified transport protocol to vTCP seems (at > this end) to be an implementation detail. > > Or alternately, perhaps the native protocol of the event backbone > will remain vTCP and no bridging will be needed by Dakota > clients, but other subscribers will choose some trendy boutique > notification service using either a push or pull paradigm. > Whichever it is, I would be flabbergasted if the astronomical > community in general or IVOA in particular sought to forbid > boundary components of the ramifying VOEventNet from including > bridging technology to zeromg or any other transport protocol. > > Rob From aa at astro.ex.ac.uk Tue Dec 27 02:26:56 2011 From: aa at astro.ex.ac.uk (Alasdair Allan) Date: Tue, 27 Dec 2011 10:26:56 +0000 Subject: 0MQ: The Intelligent Transport Layer In-Reply-To: <010ECC39-5661-4EA9-9E25-3BD619FF3C37@noao.edu> References: <22772383.57033.1324397598523.JavaMail.root@m11> <4EF944AB.3000600@dc3.com> <010ECC39-5661-4EA9-9E25-3BD619FF3C37@noao.edu> Message-ID: > Will vTCP scale to the current LSST baseline of two million alerts per night? Yes, because I've run simulations to prove that it will. I also feel like I've been banging my head against a wall sine 2006 at this point! Al. -- Sent from my iPhone. From aa at astro.ex.ac.uk Tue Dec 27 03:06:44 2011 From: aa at astro.ex.ac.uk (Alasdair Allan) Date: Tue, 27 Dec 2011 11:06:44 +0000 Subject: 0MQ: The Intelligent Transport Layer In-Reply-To: <010ECC39-5661-4EA9-9E25-3BD619FF3C37@noao.edu> References: <22772383.57033.1324397598523.JavaMail.root@m11> <4EF944AB.3000600@dc3.com> <010ECC39-5661-4EA9-9E25-3BD619FF3C37@noao.edu> Message-ID: > My suspicion is that vTCP and thus Dakota will be supported indefinitely, but that more boutique protocols like zeromg, elvin, Java messaging - and perhaps jabber - will come and go. Internally, back in the day, I used JSON-RPC to move messages around. Which means, yes, I used my own JSON representation of VOEvent. I only translated to XML and vTCP for external users. Al. PS. Going down to Southampton to talk to the 4PI guys there early in January. -- Sent from my iPhone. On 27 Dec 2011, at 05:02, Rob Seaman wrote: > My suspicion is that vTCP and thus Dakota will be supported indefinitely, but that more boutique protocols like zeromg, elvin, Java messaging - and perhaps jabber - will come and go. From Hessman at Astro.physik.Uni-Goettingen.DE Tue Dec 27 05:07:20 2011 From: Hessman at Astro.physik.Uni-Goettingen.DE (Frederic V. Hessman) Date: Tue, 27 Dec 2011 14:07:20 +0100 Subject: 0MQ: The Intelligent Transport Layer In-Reply-To: <16827241.1348.1324966650909.JavaMail.root@web04> References: <16827241.1348.1324966650909.JavaMail.root@web04> Message-ID: Here, here! We should provide a vanilla solution and let people who want to use something else simply do it - if it turns out to be so much better in the mid-term, then there's more than enough time to switch. The only thing we could do as a "value-added" feature is maintain a loose collection of solutions, particularly those whose fancy features are only open-ended (i.e. can immediately connect to, e.g., vTCP). Things like 0MQ and ICE are wonderful in a closed solution space, but it's nicer to have things easy on your end even if the other end isn't. Rick On 27 Dec 2011, at 07:17, Bob wrote: > May write more soon (or maybe I'm done with this), but: > > (1) Is "transport neutrality" like "gauge neutrality" for a > railroad? What is the benefit? The meat is in the messages not the > plumbing. > > (2) Is "undefined and ever expanding incompatible transport" > standing in the way of adoption? I suspect the uncertainty is a > huge barrier to adoption. > > (3) Ah, the LSST spectre... I say again "Who is going to USE (or > even LOOK AT) 2 million alerts per night? What's the point? It > seems like the stick that the sandboxers use to beat the realists > over the head with. That head again ;-) And a sentence ending in a > preposition. Sorry. > > (4) Is transport research and development for VOEvent a vehicle > for garnering additional grant (and therefore > grocery/housepayment) money? If so, stability and adoption are > hopeless. > > -- Bob > > On Mon Dec 26 23:02:29 CST 2011, Rob Seaman > wrote: > >> Hi Bob, >> >> I can't comment on either your head or wall :-) but nothing is >> wrong with either the transport note or Dakota. And however >> simple zeromg is, it certainly can't be simpler than vanilla TCP. >> My suspicion is that vTCP and thus Dakota will be supported >> indefinitely, but that more boutique protocols like zeromg, >> elvin, Java messaging - and perhaps jabber - will come and go. >> As has been said before, this implies the creation and >> maintenance of bridges in the deployed VOEventNet for any future >> network. VOEvent itself is transport neutral by design. >> >> Will vTCP scale to the current LSST baseline of two million >> alerts per night? Will other technologies? In the near term, >> projects now nearing completion or in commissioning will deliver >> many thousands of events nightly in the next few years. These >> several surveys are each likely to produce numerous streams of >> events, filtered or correlated in scientifically interesting >> ways. Registries - either explicitly through the VO or >> implicitly through the diverse stakeholders for each survey - >> will be used to discover streams and associated metadata and >> per-project schemata (again, either explicit or implicit). >> >> All of these issues and the related science requirements from the >> diverse event-producers and consumers will have a hand in >> evolving the deployed infrastructure needed to scale to the >> future event rates and large numbers of future subscribers of >> different sorts. There are a lot of different options for how >> that infrastructure will be funded and operated. It is quite >> likely that different parties will provide access to different >> classes of subscribers as well as classes of event authors (both >> humans and autonomous agents). For instance, I would think it an >> almost foregone conclusion that your customers would most >> robustly and efficiently subscribe through your software, >> whatever the network topology of future event brokers. Whether >> one or more of those brokers includes bridging capabilities from >> some yet-to-be-specified transport protocol to vTCP seems (at >> this end) to be an implementation detail. >> >> Or alternately, perhaps the native protocol of the event backbone >> will remain vTCP and no bridging will be needed by Dakota >> clients, but other subscribers will choose some trendy boutique >> notification service using either a push or pull paradigm. >> Whichever it is, I would be flabbergasted if the astronomical >> community in general or IVOA in particular sought to forbid >> boundary components of the ramifying VOEventNet from including >> bridging technology to zeromg or any other transport protocol. >> >> Rob From swinbank at transientskp.org Tue Dec 27 08:44:21 2011 From: swinbank at transientskp.org (John Swinbank) Date: Tue, 27 Dec 2011 16:44:21 +0000 Subject: 0MQ: The Intelligent Transport Layer In-Reply-To: References: <16827241.1348.1324966650909.JavaMail.root@web04> Message-ID: <1E2D3C9D-5BC7-4517-88F8-6C362C1BD9BA@transientskp.org> Hello, Completely agree with this: given that the TCP-based protocol is already documented & in use, and I'm not hearing anybody pointing out major deficiencies with it, I don't think there's any serious basis for suggesting that it would be replaced as the "backbone" protocol. I'd certainly be surprised if scalability were an issue: 2e6 events at ? say ? 10 kB event is only 20 GB, or about 0.5 MB/s over 12 hours. That sort of rate does't sound worrying at all. The above said: I can circumstances in which bespoke protocols might be more convenient than vanilla TCP ? either for use within some particular project infrastructure, or just because surfing to SkyAlert and downloading events by HTTP in a browser is pretty user friendly! I, for one, would be interested in discussion within this WG on the systems people are developing, at least insofar as they might be relevant to the wider community ? and would not infer from such discussion an attempt to undermine or replace existing infrastructure. I don't really buy the theory that uncertainty over the transport mechanism is a barrier to adoption: returning to my theme of the other day, I suspect that a lack of documented/visible/extant infrastructure is a much bigger issue. I don't really care what protocol I use to collect events, but when I'm left scrabbling around looking at semi-abandoned web pages and vague, non-specific mentions of backbones and XMPP and pubsub and suchlike, without a definitive, up-to-date summary of the options for actually subscribing to a stream of events, it doesn't really inspire confidence. (The exception being the DC3 Dakota stuff, which is great ? much credit to Bob for this!) Per my recent mail, in the new year I'll attempt to partially address the problem by documenting as much infrastructure as I can discover on the web somewhere. [As an aside, 2e6 events is certainly scientifically interesting: obviously, nobody is going to follow up on all of them "by hand", but, for example, automatically cross-correlating them with events from LOFAR (or SKA, or whatever) could well be an illuminating exercise. At any rate, I'd rather throw many events away than have something interesting go unreported.] Cheers, John On 27 Dec 2011, at 13:07, Frederic V. Hessman wrote: > Here, here! We should provide a vanilla solution and let people who want to use something else simply do it - if it turns out to be so much better in the mid-term, then there's more than enough time to switch. The only thing we could do as a "value-added" feature is maintain a loose collection of solutions, particularly those whose fancy features are only open-ended (i.e. can immediately connect to, e.g., vTCP). Things like 0MQ and ICE are wonderful in a closed solution space, but it's nicer to have things easy on your end even if the other end isn't. > > Rick > > On 27 Dec 2011, at 07:17, Bob wrote: > >> May write more soon (or maybe I'm done with this), but: >> >> (1) Is "transport neutrality" like "gauge neutrality" for a >> railroad? What is the benefit? The meat is in the messages not the >> plumbing. >> >> (2) Is "undefined and ever expanding incompatible transport" >> standing in the way of adoption? I suspect the uncertainty is a >> huge barrier to adoption. >> >> (3) Ah, the LSST spectre... I say again "Who is going to USE (or >> even LOOK AT) 2 million alerts per night? What's the point? It >> seems like the stick that the sandboxers use to beat the realists >> over the head with. That head again ;-) And a sentence ending in a >> preposition. Sorry. >> >> (4) Is transport research and development for VOEvent a vehicle >> for garnering additional grant (and therefore >> grocery/housepayment) money? If so, stability and adoption are >> hopeless. >> >> -- Bob >> >> On Mon Dec 26 23:02:29 CST 2011, Rob Seaman >> wrote: >> >>> Hi Bob, >>> >>> I can't comment on either your head or wall :-) but nothing is >>> wrong with either the transport note or Dakota. And however >>> simple zeromg is, it certainly can't be simpler than vanilla TCP. >>> My suspicion is that vTCP and thus Dakota will be supported >>> indefinitely, but that more boutique protocols like zeromg, >>> elvin, Java messaging - and perhaps jabber - will come and go. >>> As has been said before, this implies the creation and >>> maintenance of bridges in the deployed VOEventNet for any future >>> network. VOEvent itself is transport neutral by design. >>> >>> Will vTCP scale to the current LSST baseline of two million >>> alerts per night? Will other technologies? In the near term, >>> projects now nearing completion or in commissioning will deliver >>> many thousands of events nightly in the next few years. These >>> several surveys are each likely to produce numerous streams of >>> events, filtered or correlated in scientifically interesting >>> ways. Registries - either explicitly through the VO or >>> implicitly through the diverse stakeholders for each survey - >>> will be used to discover streams and associated metadata and >>> per-project schemata (again, either explicit or implicit). >>> >>> All of these issues and the related science requirements from the >>> diverse event-producers and consumers will have a hand in >>> evolving the deployed infrastructure needed to scale to the >>> future event rates and large numbers of future subscribers of >>> different sorts. There are a lot of different options for how >>> that infrastructure will be funded and operated. It is quite >>> likely that different parties will provide access to different >>> classes of subscribers as well as classes of event authors (both >>> humans and autonomous agents). For instance, I would think it an >>> almost foregone conclusion that your customers would most >>> robustly and efficiently subscribe through your software, >>> whatever the network topology of future event brokers. Whether >>> one or more of those brokers includes bridging capabilities from >>> some yet-to-be-specified transport protocol to vTCP seems (at >>> this end) to be an implementation detail. >>> >>> Or alternately, perhaps the native protocol of the event backbone >>> will remain vTCP and no bridging will be needed by Dakota >>> clients, but other subscribers will choose some trendy boutique >>> notification service using either a push or pull paradigm. >>> Whichever it is, I would be flabbergasted if the astronomical >>> community in general or IVOA in particular sought to forbid >>> boundary components of the ramifying VOEventNet from including >>> bridging technology to zeromg or any other transport protocol. >>> >>> Rob > From seaman at noao.edu Tue Dec 27 09:24:56 2011 From: seaman at noao.edu (Rob Seaman) Date: Tue, 27 Dec 2011 10:24:56 -0700 Subject: VOEventNet is more than a transport protocol In-Reply-To: <16827241.1348.1324966650909.JavaMail.root@web04> References: <16827241.1348.1324966650909.JavaMail.root@web04> Message-ID: <5F6B7B4C-E727-4C5E-A645-27E0110197CF@noao.edu> On Dec 26, 2011, at 11:17 PM, Bob wrote: > (1) Is "transport neutrality" like "gauge neutrality" for a railroad? Henry Petroski touches on railroad gauges in his "Success through Failure", but also in other books. The two mentioned in #1 are similar in that they are a product of engineering evolution. They are dissimilar in that railroad gauge was ultimately inherited from the ruts left by ancient Roman wagons. Breaking out of the weakly analogous transport paradigm was, for instance, the challenge faced by the IAU telegrams, not VOEvent. > (2) Is "undefined and ever expanding incompatible transport" standing in the way of adoption? I suspect the uncertainty is a huge barrier to adoption. This is a "complex question" ("plurium interrogationum" if any of those Romans are listening in). The transport note provides a definition. "Ever expanding" assumes that things like Elvin don't naturally die out. "Incompatible" presupposes that bridges don't exist, whereas a precondition of deploying a new transport option on the VOEventNet is precisely that a compatible bridge exists. Uncertainty is indeed to be avoided, which is why we've handed out 700 copies of the Hotwired book (and more for the upcoming AAS). A book BTW containing Bob's chapter that describes connecting to the VOEvent backbone using Dakota and vTCP. > (3) Ah, the LSST spectre... I say again "Who is going to USE (or even LOOK AT) 2 million alerts per night? What's the point? This is backwards. Using the full firehose requires precisely that nobody be required to look at them. This naturally leads back to all the functions of the VO and astroinformatics. Few projects will subscribe to the full feed. Many users will subscribe to filtered streams that provide only the brightest events, for instance. Such requirements will have contingent implications for future transport technologies - or perhaps all the smarts should be built into the clients and servers? Should we really limit our design conversations a priori? We provided vTCP as the baseline for the backbone precisely with the understanding that no filtering occurs on the backbone. At any rate, growing the backbone is a separate activity from the tier-two delivery of alerts (and related content) to endusers. This also is discussed in Bob's chapter of the Hotwired book, where I believe the word "Jabber" is mentioned. Might zeromg replace jabber for this use? Are such discussions not within the remit of this group? > It seems like the stick that the sandboxers use to beat the realists over the head with. That head again ;-) I'm not entirely clear what is meant by "sandboxer", but presumably it is intended to imply "non-realist". Also not clear how: 1) discussing future transport options is a stick, and 2) how the heads of those who have very realistically written code and deployed services against the transport note are threatened. > (4) Is transport research and development for VOEvent a vehicle for garnering additional grant (and therefore grocery/housepayment) money? If so, stability and adoption are hopeless. Stability and adoption have been happening steadily since 2005. VOEvent has been remarkably successful at the rather daunting challenge of providing an evolutionary path for a community activity that predates the IAU itself. I am unaware of any prior celestial event report project that we have not engaged with - and sought to breathe new life into. Similarly we are engaged with future time domain projects, both surveys and follow-up telescopes. We are lucky to have a bit of breathing room between the successes of the past and those of the future to plan and build. Not surprisingly, especially in this economy, bootstrapping the deployed network of brokers is proceeding through stages. The short answer to question #4 is "no". The longer answer would be off-topic for this mailing list. Rob From seaman at noao.edu Tue Dec 27 09:34:07 2011 From: seaman at noao.edu (Rob Seaman) Date: Tue, 27 Dec 2011 10:34:07 -0700 Subject: 0MQ: The Intelligent Transport Layer In-Reply-To: References: <16827241.1348.1324966650909.JavaMail.root@web04> Message-ID: Hi Rick, > Here, here! We should provide a vanilla solution Which we long since have done, hence the follow-on transport meeting at VOEvent II leading to the IVOA transport note, and the Hotwired book pointing to same. > and let people who want to use something else simply do it - if it turns out to be so much better in the mid-term, then there's more than enough time to switch. The only thing we could do as a "value-added" feature is maintain a loose collection of solutions, particularly those whose fancy features are only open-ended (i.e. can immediately connect to, e.g., vTCP). Things like 0MQ and ICE are wonderful in a closed solution space, but it's nicer to have things easy on your end even if the other end isn't. If you put together an abstract along these lines there is still time to submit it to the SPIE Observatory Operations conference: http://spie.org/as107 Your participation and a contribution on a related topic would add to an already strong program. Rob From seaman at noao.edu Tue Dec 27 09:49:57 2011 From: seaman at noao.edu (Rob Seaman) Date: Tue, 27 Dec 2011 10:49:57 -0700 Subject: 0MQ: The Intelligent Transport Layer In-Reply-To: <1E2D3C9D-5BC7-4517-88F8-6C362C1BD9BA@transientskp.org> References: <16827241.1348.1324966650909.JavaMail.root@web04> <1E2D3C9D-5BC7-4517-88F8-6C362C1BD9BA@transientskp.org> Message-ID: Hi John, > I'd certainly be surprised if scalability were an issue: 2e6 events at ? say ? 10 kB event is only 20 GB, or about 0.5 MB/s over 12 hours. That sort of rate does't sound worrying at all. End-to-end latency is also a key issue. We should bear this in mind when any benchmarks or data challenges are performed. > I suspect that a lack of documented/visible/extant infrastructure is a much bigger issue. I don't really care what protocol I use to collect events, but when I'm left scrabbling around looking at semi-abandoned web pages and vague, non-specific mentions of backbones and XMPP and pubsub and suchlike, without a definitive, up-to-date summary of the options for actually subscribing to a stream of events, it doesn't really inspire confidence. (The exception being the DC3 Dakota stuff, which is great ? much credit to Bob for this!) Good suggestions for the 2nd edition of Hotwired. And by all means update the wiki and we can point voevent.org to an introductory page - sounds like a good job for the new chair. The larger challenge remains the evolutionary path for the bricks-and-morter network of VOEvent brokers. SkyAlert and GCN provide the main vertabrae on the backbone currently. What comes next and how do we manage the transition? Rob From fitz at noao.edu Tue Dec 27 10:15:41 2011 From: fitz at noao.edu (Mike Fitzpatrick) Date: Tue, 27 Dec 2011 11:15:41 -0700 Subject: 0MQ: The Intelligent Transport Layer In-Reply-To: References: <16827241.1348.1324966650909.JavaMail.root@web04> <1E2D3C9D-5BC7-4517-88F8-6C362C1BD9BA@transientskp.org> Message-ID: On Tue, Dec 27, 2011 at 10:49 AM, Rob Seaman wrote: > Hi John, > > > I'd certainly be surprised if scalability were an issue: 2e6 events at ? > say ? 10 kB event is only 20 GB, or about 0.5 MB/s over 12 hours. That sort > of rate does't sound worrying at all. > > End-to-end latency is also a key issue. We should bear this in mind when > any benchmarks or data challenges are performed. Indeed, the math isn't that simple and much depends on how the LSST events are deployed. Forgetting even the inefficiencies of sending many small packets the above calculation only applies to streaming the whole 20GB. Assuming these events are distributed from Chile, the current network latency is ~250ms, and the vTCP requires at least one round-trip to message to complete delivery. For 2e6 events sent one at a time the latency alone makes this transfer on the scale of 1-2 weeks (i.e. at best you could send 3-4 events / second). Although the network capacity to/from Chile will improve dramatically by the time LSST starts running, I'm not sure the latency will improve as dramatically. Secondly, vTCP couldn't be used as-is for bulk transport since there is a 32-bit limit on the byte-count that couldn't accommodate 20GB of events (but could presumably for ~10K events per LSST 'visit' sent every few minutes). Distributing 2e6 events from someplace other than Chile still begs the question of how to do the bulk transport to a better-connected location and what tweaks to vTCP would be required. Lastly, no comment has been made about how an event packet is sometimes the start of a cascade of other packets (requests for references from repositories, follow-on packets, etc), so the actual numbers flowing through the "system" may be more than 2e6 / day. vTCP is fine for authors and end-delivery of single packets, but my own opinion is that something more structured is required for bulk transport, repository replication, SEAP service requests, etc to operate a real backbone network and federated repositories. Sorting that out is the job of the VOEventStream and VOEventServer docs. -Mike -------------- next part -------------- An HTML attachment was scrubbed... URL: From swinbank at transientskp.org Tue Dec 27 11:21:17 2011 From: swinbank at transientskp.org (John Swinbank) Date: Tue, 27 Dec 2011 19:21:17 +0000 Subject: 0MQ: The Intelligent Transport Layer In-Reply-To: References: <16827241.1348.1324966650909.JavaMail.root@web04> <1E2D3C9D-5BC7-4517-88F8-6C362C1BD9BA@transientskp.org> Message-ID: <8F9FC8BB-26DF-491C-987B-156A9948932A@transientskp.org> Hello, On 27 Dec 2011, at 18:15, Mike Fitzpatrick wrote: > Assuming these events are distributed from Chile, the current network latency is > ~250ms, and the vTCP requires at least one round-trip to message to complete > delivery. For 2e6 events sent one at a time the latency alone makes this transfer > on the scale of 1-2 weeks Presumably sending in series isn't required? But fair enough: point taken! I revise my earlier statement ? perhaps people *are* pointing out deficiencies in the TCP protocol. That makes discussion of other systems even more relevant, but still doesn't mean existing infrastructure is being replaced or deprecated any time soon. Cheers, John From fitz at noao.edu Tue Dec 27 12:22:19 2011 From: fitz at noao.edu (Mike Fitzpatrick) Date: Tue, 27 Dec 2011 13:22:19 -0700 Subject: 0MQ: The Intelligent Transport Layer In-Reply-To: <8F9FC8BB-26DF-491C-987B-156A9948932A@transientskp.org> References: <16827241.1348.1324966650909.JavaMail.root@web04> <1E2D3C9D-5BC7-4517-88F8-6C362C1BD9BA@transientskp.org> <8F9FC8BB-26DF-491C-987B-156A9948932A@transientskp.org> Message-ID: On Tue, Dec 27, 2011 at 12:21 PM, John Swinbank wrote: > Hello, > > On 27 Dec 2011, at 18:15, Mike Fitzpatrick wrote: > > > Assuming these events are distributed from Chile, the current network > latency is > > ~250ms, and the vTCP requires at least one round-trip to message to > complete > > delivery. For 2e6 events sent one at a time the latency alone makes > this transfer > > on the scale of 1-2 weeks > > Presumably sending in series isn't required? > The prospect of many thousands of parallel vTCP connections is very appealing either ... Remember too, for some events to be useful (e.g. GRB followup) they cannot be delivered 12 hours later, so the problem (at least a major use-case) has to also solve the question of how to deliver 10K (or is it 50K, or 100K?) events in a matter of *minutes*. I don't think the bar napkin on which vTCP was designed was big enough to consider that case thoroughly 8-) Cheers, -Mike -------------- next part -------------- An HTML attachment was scrubbed... URL: From seaman at noao.edu Tue Dec 27 13:42:40 2011 From: seaman at noao.edu (Rob Seaman) Date: Tue, 27 Dec 2011 14:42:40 -0700 Subject: 0MQ: The Intelligent Transport Layer In-Reply-To: References: <16827241.1348.1324966650909.JavaMail.root@web04> <1E2D3C9D-5BC7-4517-88F8-6C362C1BD9BA@transientskp.org> <8F9FC8BB-26DF-491C-987B-156A9948932A@transientskp.org> Message-ID: <8294DC1A-D30E-48A1-8D5C-446CA126C0F6@noao.edu> On Dec 27, 2011, at 1:22 PM, Mike Fitzpatrick wrote: > Remember too, for some events to be useful (e.g. GRB followup) they cannot be delivered 12 hours later, so the problem (at least a major use-case) has to also solve the question of how to deliver 10K (or is it 50K, or 100K?) events in a matter of *minutes*. Fractions of a second, actually. The challenge for VOEvent is to provide a single system suitable for diverse use cases. Not all phenomena or their contingent science requires such short latency, but the alternative is to have separate networks for separate purposes. This was historically the case and we've seen how far the community infrastructure can splinter as a result. It will likely always be the case that purpose-specific publishers will prosper, but that doesn't mean they should duplicate efforts toward formats, transport, tools, brokers, repositories, etc. The LSST baseline is something like 1000 visits (paired exposures) per night, each averaging 2000 events. (By adjusting thresholds that can grow by an order of magnitude, but this is the baseline as I last heard it.) The survey's latency requirement is to publish each visit's events within 60s of the shutter closing. The corresponding requirement on the transport will be shorter than this for the simple reason that network services will have to keep up. The science requirements will vary between solar system and stellar and extragalactic projects. > I don't think the bar napkin on which vTCP was designed was big enough to consider that case thoroughly 8-) vTCP was elaborated from GCN at a session at VOEvent II in 2005. It was sketched out on the whiteboard in NOAO's La Quinta conference room and Alasdair took a picture on his cell phone. The napkin in question rather contained the outline of SimpleTimeSeries and was sketched out at the Arizona Inn during the banquet for Hotwired I (aka VOEvent III) in 2007. Both should be regarded as successful design efforts that have seeded subsequent work. That both whiteboard and napkin are finite in size provided encouragement to keep the designs pragmatic in scope. I would hope that the IVOA time series efforts remain so. From the joie de vivre with which the transport discussion has been revisited recently and in the past, it seems ripe for a renewed effort. Part of that effort should be directed toward enhancing the robustness and usability of current protocols, libraries, servers and clients - and part toward characterizing the efforts needed for future work. One realization of the latter has been planned as a US VAO activity (http://arxiv.org/pdf/1111.2356 and related efforts). Given funding challenges this is likely to remain pending for the near term, but other efforts can continue depending on their own project priorities and resources. Rob From roy at caltech.edu Tue Dec 27 14:05:40 2011 From: roy at caltech.edu (Roy Williams) Date: Tue, 27 Dec 2011 14:05:40 -0800 Subject: 0MQ: The Intelligent Transport Layer In-Reply-To: <4EF944AB.3000600@dc3.com> References: <22772383.57033.1324397598523.JavaMail.root@m11> <4EF944AB.3000600@dc3.com> Message-ID: <4EFA4134.8000000@caltech.edu> On 12/26/11 8:08 PM, Bob Denny wrote: > Magic Question: What is wrong with > http://www.ivoa.net/Documents/Notes/VOEventTransport/ Nothing wrong with vTCP/Dakota. I am happy to report that Skyalert supports it. However, I have only heard of one instance of an observatory actually making use of events distributed this way (NMSU), so I am also looking elsewhere. There are other protocols. The users of the original GCN socket will ask why not that one? Or the fixed-format emails that MPC uses, they say what is wrong with that transport. Both the CRTS and LIGO Lab have invested in Jabber/pubsub. I like http as a protocol too. We use both XML and JSON for events. So a profusion.... I would say the most important thing is interoperability with VOEvent. All of the above protocols and formats are happy to carry something semantically representable as a VOEvent, and in many cases actual VOEvent syntax. In other words, VOEvent uptake is improved by providing what people want, and by translating what they give us to VOEvent. We cannot just cry about being in the same corner since 2006 and why is nobody coming! As for scaling, I would say there are various avenues. Sheer bulk of message is one dimension, burst rate and average rate, also number of streams, number of selections on streams, number of users, size of linked data, compute requirements for annotation, required reliability, degree of replication and distribution, and several more. Lets us ask which of these dimensions are crucial to scale, and which not. Roy --- Caltech LIGO roy at caltech.edu 626 395 3670 From roy at caltech.edu Tue Dec 27 14:05:45 2011 From: roy at caltech.edu (Roy Williams) Date: Tue, 27 Dec 2011 14:05:45 -0800 Subject: 0MQ: The Intelligent Transport Layer In-Reply-To: <1E2D3C9D-5BC7-4517-88F8-6C362C1BD9BA@transientskp.org> References: <16827241.1348.1324966650909.JavaMail.root@web04> <1E2D3C9D-5BC7-4517-88F8-6C362C1BD9BA@transientskp.org> Message-ID: <4EFA4139.8000206@caltech.edu> On 12/27/11 8:44 AM, John Swinbank wrote: > without a definitive, up-to-date summary of the options for actually > subscribing to a stream of events, it doesn't really inspire > confidence. (The exception being the DC3 Dakota stuff, which is great > ? much credit to Bob for this!) John Please also take a look at the Skyalert page. It has a lot of ways to subscribe to a stream of events Atom feed, Dakota, email, Jabber, with about a dozen active streams. You can also query a repository of 20000 past events. Please alert me to any failures of advertising or documentation that fail to inspire confidence :-) Roy --- Caltech LIGO roy at caltech.edu 626 395 3670