Well, hello everybody. Here we are at IMPACT 2011, the IBM WebSphere Conference. We're actually in a sub-conference. We're at the Un-Conference, the IMPACT Un-Conference. Is that the official title of it, the IMPACT Un-Conference? Yeah, absolutely, Un-Conference. And as always, this is Michael Coté of RedMonk, and I've got a guest with myself. Want to introduce yourself? Sure. I'm Andy Piper. I work in our Hursley Laboratory, which is in the UK. I think my job title at the moment, because it kind of moves around a bit, well, it's WebSphere Messaging Community Lead is what I do. Ah, right. Basically, that's MQ. It's Q stuff, right? Yeah, absolutely. What falls under WebSphere Messaging? So the stuff, and that stuff was originally invented and developed from Hursley, which is where I sit. So yeah, so what have we got? We've got WebSphere MQ, which is Q stuff. We've got a family of products around that. So we've got the WebSphere MQ File Transfer Edition, which enables you to move files over MQ. We've got WebSphere MQ Low Latency Messaging for very high-speed messaging. We've got WebSphere MQ Telemetry, which is about mobile and sensor networks. I'm going to miss something. We've got WebSphere MQ Advanced Message Security, which is for encrypting messages. Right. We've got a message broker, which is our enterprise service bar. There's this resurgence of queues in architectures. I'm not even going to say enterprise architectures. No. Because what's interesting is that all sorts of architectures for software, they're sort of rediscovering, I guess, using a queue, or to put it another way, having an event-driven architecture, a bus or something like that. And the fact that by doing so, you can actually start to parallelize workload, you know? Isn't that an amazing thought? Right. Yeah, absolutely. The idea of feeding some work to a process over a period of time and not having to have it instantly respond, but knowing that it will be done. Have it be asynchronous, I guess. Yeah. And that word, again, we've been talking about the last couple of days. I've been a consultant, you know, and often people hear the word asynchronous and they go, "Uh, that sounds slow."
2:00Right? And it's like, no, it's just the difference between what we're doing, you know, having an immediate conversation and an immediate response. Right. Or may get a very false response, but it's not time-dependent, right? Yeah, yeah. That's what it means. Asynchronous is not something that's not time-dependent. I feel like at some point asynchronous programming fell out of use. I don't know if it fell out of favor, but it kind of, because I guess maybe it all came down to web applications, which is a very synchronous sort of process. A request comes in and there's, I'm drawing a line here, a request completes. Like, you expect the client to be waiting. Well, it does. I mean, that's right. I mean, HTTP times out, right? An HTTP request will time out. Right. And then Ajax came along and A being asynchronous, if I remember. And I guess that kind of started to change things around a little bit. So, what exactly is a queue? Like, how does a queue operate? So, I think of a queue as a bucket into which I can throw something and then take it back or something else. Somebody else can take it out every now and then when they need it. So, what queuing and message queue, MQ, lets us do is to say, here's a bunch of data I'm creating. And I'm just going to put it over there. And I may come and get it back later myself as a process or actually another process might take it from there. Right. It's just a place to put pieces of work or pieces of data. The important thing to note is it's not a database, right? It's not somewhere you're shoving data to store. Right. You're putting it there temporarily until somebody comes along and takes it away and does something else with it. And so, does that imply that it's sort of not searchable, if you will? It's not searchable in the same way as a database. What we can do is we can do things like assign a load of unique IDs. And actually, that's exactly how message queuing works is that every message has a unique ID. Right. And we can go and find that ID and get it from the pile. That's the extent to which it's searchable. But you're not expecting to do like a full text. A bunch of queries. Yeah, a full text index on a bunch of data. So, why don't you just use like a shared directory?
4:00I mean, that would kind of be, that's a bucket, right? It's a bucket which is very secure and stable and assured, right? So, the goal is to say, let's take away the need to give stuff to a file system which could crash or become corrupt. Let's give it to a thing which will put it into this queue for us and give it back to us or give it to something else. And by the way, that queue may not be local to me. It may be over a network somewhere. So, yes, you could put it in a shared directory in the sense of a network drive. But then, of course, you need the same file system technology or, you know, everybody needs to implement SAM or NFS or whatever you're going to share. Right. I mean, what you're getting at, correct me if I'm wrong, is that if you use a directory, then you're depending on the file system and the storage. Yeah. As part of your architecture rather than depending on a higher up abstraction, depending on a service. That's right. Exactly. And one of the, and I'll try not to do the, you know, the chess beating sales pitch and that's not what I'm here for at all. But one of the things about MQ and one of the reasons why it has been very widely adopted over the last 10, 15 years is because it does run on a large variety of platforms. It has, provides the same API. You're not having to think, okay, I'm on Windows now, so I need to use the .NET API to get to my file system. I'm on a, on a Mac, I need to use C to get, objective C to get to my, to my file system, right? Right. It's a totally different way. It's the same API on any platform that MQ exists. You've got the ability, as I mentioned, to choose to send data over a network to some other queue. It's a, it's a passing mechanism. And, and then inherent in networking. I mean, once, once you bring some network thing, there's security gets involved. Security. Giving access. Channels and things like that. So we, we have an MQ is a concept of a channel between two queue managers that own all of the queues on that system. And then between them, we have a channel and we can encrypt that tunnel. Right. Using SSL. So yeah, you have security. You have the ability. And this is another important aspect of MQ is assurance, right? You want to just rely on the fact that you have something reliable moving the data for you.
6:04And, and, and, and in the case of MQ and maybe queues in general. Yeah. Is that assurance the same thing as sort of transactional integrity in, in a database? Or is it more assurance that your stuff won't disappear? Or is it both? Yes. Yes. And, and, and what I mean by that is, you know, in, in, in database terms, like a transactional thing is that, uh, there will never be two states of the data. Yeah. That, that you're accessing. So absolutely. So, and, and the, and there's another aspect of course, which is when you're talking about databases, you talk about, um, two phase commits and all that kind of stuff. And the ability to say, I want to do these operations and either the whole set of operations commits or fails. And actually I might want to do that with two databases or, and a file system. Right. So transferring money between two accounts is a classic example. Exactly. So MQ is very commonly used in those kinds of scenarios to say, I need to get this message from this queue, update those two database tables belonging to two different systems and put this message to this queue and do all of that within a transaction. Right. So we, that, that's what our message broker and our MQ technology does for you. Um, but yeah, you're absolutely right. The, the assurance is I'm giving my stuff to some other process to look after for me and I want to make sure that it arrives. So when I get told locally by the queue manager that it's got my data and put it on the put has succeeded, then I can rely on the assurance that my MQ network will get that data somewhere else once and once only. Right, right, right. Okay. That's another important concept here. You're not getting duplication. Yeah, yeah, yeah. Otherwise you might end up doing the same operation twice. Uh, and, and, and then yeah, uh, if, if something crashes, if my local queue manager or remote queue manager crashes and we've put that message in what we call a persistent state because we can do things in memory as well and be non persistent. Right. Um, that we could recover to a known state. Yeah, yeah, yeah, yeah. Right. So yeah, you don't want to lose work, especially if you're updating bank accounts. Right. You must know better than I do how old MQ is. Right. But it's definitely been, it's one of the older IBM software.
8:05Yeah. It used to be called MQ series. It's about 15 years, I think now. Right, right. And so to give you the chance to not so much be a salesman, but, but, but kind of say what that gets you, like the differentiation that MQ has. I mean, what, what is it about MQ that makes it something beyond like an ES, an open source ESB that you might get or an open source queue. But first of all, it gives you stability of function, right? So we've been around for a long time. Um, and, and, and actually the upgrade ability has been really good as well. Yeah. So you can take a, you know, different versions of MQ and they can talk to one another. And that's always been. So, so the sort of cross platform interoperability. Exactly. You mentioned that earlier. And upgrade ability. Yeah, yeah, yeah. So, so that's the first thing. Longevity of the. Yeah. So in terms of feature, you've got, you know, we've also constantly working on, working to improve performance. You've got a high performing system if you, if you need one, if you need to tune it that way. You've got things like clustering, which let you, do let you distribute work over a number of different systems. And actually it's one of the things that we would come back to around this rediscovery of the idea of queuing. Right. Did you hear that news about Twitter rewriting their search engine? Oh, right, right, right. And, and I was reading that they, they've actually now got a thing which dispatches your query to one of several engines. So they can suddenly work like balance all these queries that are coming into their search engine. That's what clustering does for you, right? Right, right. You can start to move, move work around really easily. You've got, oh gosh, you've got loads of things. Security. You've got this emergence of this new family of products in the last two or three years. The file transfer, the advanced message security, the telemetry. These have all been added onto MQ because the vision for MQ from, from an IBM perspective is universal messaging. So the idea is that data, your data are anytime, anywhere, if you like. Sure. Over whatever is the appropriate quality of service to give you at the point that you're at. Are you in your transactional data center or are you on a mobile device? Right, right, right. How fast do you need to go? How reliable does it need to be? How secure does it need to be? So you sort of customize the footprint or the profile down to the end point?
10:05In a way. So that's particularly what telemetry is actually. Telemetry is absolutely for smaller devices customized to the network capability that's available, that kind of thing. One thing I'm always curious about with middleware, and I use that phrase tentatively here, is, would you say when someone's, some organization is using MQ, is it more of a middleware component in their software stack or is it a system that they're running? Oh. Does that distinction make sense? Yeah, okay. I would say it's a middleware component, really. Right. So it's kind of, it's not a service. It's a capability. It's not a universal sort of service. No, no. I mean, services would usually themselves use MQ or run on MQ. So you might, your program, your service. Right. Your application may be, may use MQ to do its interface to the world. Right. If you think about an application, we're talking about this in the on conference this morning, actually. And a guy was asking about files and how do I get, move away from file driven batch to SOA services. And actually, you know, people are always going to be sending files in some form or another. So you might be able to reduce the number of files coming in, but you're probably not going to eliminate them. So get over them. Files are just so comfortable. Well, they are. They are. And, you know, everybody has notepad and we can just hack away. Right. But more to the point. That's how I submit my batch jobs. Notepad. Notepad. Nice. Nice. So, but it's just another interface. So if you think of a program that takes input, you mentioned shared directories maybe. It could be reading files. It could be reading from queues. It could be receiving HTTP calls. It could be coming in over TCP. You know, you've got a range of different ways. So, you can see files and queues as different interfaces to your services. So, coming back to your question, therefore, I would see queuing as a component. You know, a middleware component. Right, right, right. A feature, a capability of your system. Right. So it's not something bigger than the applications that use it that you kind of integrate with. It's more you pull it in as part of your application. Yeah, I mean, one of the ways that people often talk about it is as plumbing, right? Yeah.
12:05You know, it's the pipe work that goes underneath your house. Right, right. Having taken a kind of technology-first approach. So, you know, I think a lot of people, including myself until I was beaten over the head with it a lot, like they hear things like batch job and jobs and queues. And frankly, it's sort of like I have no idea how that stuff is used. Like what the business applications of that are, right? Right. And then like you were saying, you know, people ask like, oh, we get these files transferred to us. And what do we do with that? And it would be great to hear sort of like what, I'm always hesitating because I want to come up with something that doesn't sound like a stupid business term, but like what business services are these queues being used for? I mean, everyone. I mean, I've been a consultant and I've worked with, you know, retail and finance and government and every system can conceptually use them. Right. And many companies and organizations and industries do use them. Yeah. I mean, the need to transfer data between point A and point B or point A and points B, C, D, E and F or whatever is every application needs that. But, but it seems like it seems like there's a certain class of applications that use queues a lot more than others. And like, like a lot of financial applications use queues to basically to synchronize payments or transfers between accounts. I mean, things like that. There's things that I think if you were writing this kind of application, you'd probably use a queue just without thinking about it. Whereas as you were saying, someone like Twitter is not going to be using a queue. Yeah. But I mean, the other thing is, so web services. Web services, great. I know you guys are really big fans of the web service standards and all that XML stuff. Yeah. So, so HTTP is great as a, as a transport for web services, except it's synchronous. Right. So if the other end isn't listening or isn't available or is too busy, then your call will time out and you will have to handle that. So, hey, what a great idea. If, if that guy's too busy because it's, it's got loads of incoming requests, then why not, instead of having an HTTP call, put a queue there.
14:06Yeah. So you can do things like web services over JMS. So, you know, your, your, your calling application that's invoking the service can just, just do that via, via sending a message somewhere. It doesn't have to be synchronous. Right. And then it can wait and come back later. So you've got things like the WS addressing standard, which let you say, call me back on this address later when, when you're done. All right. Okay. And, and, and, and I imagine, and correct me if I'm wrong, but situations like, like retail data might be nice for that. If you have like hundreds of stores that need to, for whatever reason, send their current inventory needs to a central place. Well, that kind of thing is also trickle feed, right? The idea of trickle feed is, is kind of very powerful in retail that you just want to, as, as people, you know, buy stuff through, come through Best Buy and going through transactions of the, of the, of the, of the, of the register, then those just maybe get fired off up. Uh, so it's a lot more real time than like 24 hour batches or something. It can be real time. It can be real time. Absolutely. And then all of those transactions batched up in the store and then overnight FTP them to, to home. You could just start to literally feed them. If you go in the Apple store, you know, and they'll say, uh, do you want me to receive email to you? Right. And, you know, magically they email it to you. Yeah. Yeah. You can also go online and see what you've purchased. So there's probably a queue involved in there somewhere. Who knows? I guess kind of the, the last question is, so, I mean, MQ is like 15 years old. Right. And what, I mean, what, what are you guys thinking about doing in the next five years or whatever? Right. Like something that's as old as MQ at some point. Let's not calling it old. Let's just, you know, it's not old. Something that says, is mature. It's rich and mature. Right. There you go. Very capable. Well said. Like, you know, there's always this question of, let me grab my blue, blue hat and blue, blue collar. Right. There's always this question of, is there anything left to do? Right. And there is. And there is. Whatever things that are left to do in the queuing space. Right. So there's emerging models and emerging programming patterns and ways of accessing data. So I mentioned our telemetry protocol, which is very optimized for what, for IBM smarter planet story. Right. The whole smarter planet. Well, let's, let's make use of this instrumented planet we've built and start to get, receive
16:10data from those sensors onto our data backbone, which is MQ. To trickle in perhaps, as you were saying. Yeah. A trickle in maybe. Or maybe you need to actually do kind of two way communications. So, so that's a nice and interesting model. There are also things like some of the new, newer web push models. And I can't talk too much about specific plans, but, but you know, we're very aware of things like the ability to push data to web browsers through new technologies. Right. There's sort of new endpoints that emerge that you want to integrate with. Yeah, absolutely. You know, mobile is obviously huge. And, and, and it's about not abandoning the existing data backbone customers, because that's, you know, because they're absolutely cool. But to extend the ability to interact with the queuing fabric from new, new programming models, new languages, and new protocols potentially. So, yeah, you can see that telemetry is as an example of one of those. The ability to plug into your existing MQ infrastructure. Right. With a specialized protocol for, for lightweight networks. Right, right, right. Yeah, no. That's, that's where we, that's, that's, that's where all of the cool stuff's happening. Yeah, yeah. And also some of the data encryption around the compliance situations that people find themselves in, we've, we've been looking at that. I mean, you've just got to look at the releases over the last few years that we've been doing. And that's where they're geared. Usability is always important as well. You know, the ability to more effectively manage things in a nice usable way. I think it's a great product from usability perspective anyway. Right. But there's always nicer things you can tweak. Well, great. Well, thanks for taking all this time to go over MQ stuff. No worries. Thanks for taking all this time to go over MQ stuff. Thank you.