1 BizTalk Overview BizTalk is a Business Process Management; Service Oriented Architecture (SOA), or an Enterprise Services Bus (ESB) platform. You can think of BizTalk as a group of tools and application services that facilitate the rapid creation of integration solutions. Most enterprises are looking for BizTalk to enable unrelated and disconnected systems to exchange data in a standard, consistent, and reliable way. The tools that BizTalk provides allow you to design reliable and robust solutions faster than is often achievable by custom coding a solution from scratch using standard development languages and tools.

With this in mind, it is important to understand what BizTalk achieves well and what is does not in order to fully realize any efficiency what a BizTalk solution can bring. If you were to look at two BizTalk projects that address identical problems, designed by two different groups, you would see two completely different solutions. Both solutions would solve the problem they were designed to address; however, you will find that one solution utilizes more BizTalk tools. This solution is the better one because it will be more robust and easier to maintain.

Pro Mapping in BizTalk Server 2009 (Expert's Voice in BizTalk) [Jim Dawson, John Wainwright] on Amazon.com. *FREE* shipping on qualifying offers. Unfortunately it does not cover the new unit testing features in BizTalk 2009 and some of the screenshots are from BizTalk 2006. But don't be fooled - the book has great.

Think of BizTalk as a toolkit, and within this toolkit, you find tools to help you build your application. 1.1 Why BizTalk? For those of you who are completely new to Enterprise Application Integration (EAI) and Business Process Management (BPM), perhaps some context can clarify the ideas behind messaging and orchestration. In addition, this section will attempt to clarify the concept of “messaging”. If you’re familiar with some basic EAI principles like loose coupling, publish /subscribe and orchestrations, you may safely skip to the following section. If two applications need to establish some form of communication between them, this can be done in many ways.

For example, application “A” needs information from application “B”, which was developed prior to “A”. This means “B” isn’t even aware of “A”’s existence. Luckily, the developers of “B” have exposed an API to integrate with. Application “A” could thus call “B”’s API to gather the information it needs. No problem, works fine, life’s easy isn’t it? Not really, integrating applications is usually about making assumptions.

And in the case of our example: we’ve made many assumptions, such as: • “B” is running while “A” makes the call. • Communication links between “B” box and “A” box are stable, secure, well established, etc. • “B” runs on a similar platform as “A” does, i.e. Same data representation. • “A” is programmed in a language that can consume “B”’s API. • “A” will never need another application to provide the information. Now, imagine that you have four or six applications that you need to integrate.

Even if you only have two applications as above imagine following scenarios for a moment: • What if we wanted to replace “B” by another, better application? • What if we wanted to phase-out “B” and move it to another box or cluster it? • What if application “B” is down while “A” needs it?

Because “A” makes a few assumptions about “B”, your integration solution might run into serious trouble. Loose coupling of applications is exactly about preventing these kinds of problems.

The fewer assumptions an application makes when integrating with another application, the more loosely coupled they are. Messaging is a very popular way to reduce coupling between applications. Messaging relies on middle-ware to transmit chunks of data (“messages”) from one application to another. Usually this is done using a kind of “channel” called “message queues”. The nice thing about queuing is that it works asynchronously.

This means that the sending application can put messages in the queue while the receiving application can pull the messages out of the queue as soon as it has time for it. For example, when it has been down and brought up again. Obviously, not all applications know how to integrate with queues. In most scenarios there will be some “adapter” converting the application’s output or input, to or from messages. Messaging reduces coupling and makes solutions more scalable.

Because of the queue acts as a load buffer, each application can work at its own speed. When you integrate application “A” with Application “B” directly (even through queues) this is referred to Point to Point integration. Now if you have 5 or 6 applications that you need to integrate. Point to point would lead to many complications and intertwining between how the applications communicate (or cooperate together) and leads to a case of spaghetti, shown in figure 1.1. In real life, most organizations rely on many software applications and they need to integrate them together. This “spaghetti” kind of integration is likely to become unmanageable.

Having a lot of point to point connections acting separately from each other is not only unmanageable; it also makes “business overviews” impossible. What do I mean with “business overviews” is, if you succeeded in connecting each and every system in an insurance organization claim processing systems, you’d expect to be able to make queries like: “How many claims were approved today?”. Figure 1.1: Spaghetti Application Integration. This requires some “business overview”, exceeding point to point communications. To overcome the problems related to having point to point connections, Hub and Spoke Integration services was invented, shown in figure 1.2. Architecturally, a Hub and Spoke Integration services are centralized processing machinery “the hub”, that accept messages from multiple applications, the Spokes. Applications interact with the hub through adapters deployed on the hub, so the spokes (applications) do not require any modifications.

The hub manages routing, mapping and tracking messages between applications. The hub and spoke mode provide strong total cost of ownership by reducing the cost to add and remove connections and through centralized management. Figure 1.2: Hub and Spoke.

The hub and spoke integration services lead to the “publish/subscribe” pattern. In the publish/subscribe pattern, two parties communicate without any shared knowledge other than the agreement on the message format. These two parties are often referred to as the producer and the consumer or – more appropriately in the context of publish subscribe – the “publisher” and the “subscriber”.

The publisher publishes its output using messaging to the pub/sub engine while the subscriber subscribes itself with the engine using a filter. The filter, based upon message content criteria, specifies which messages are of interest to the subscriber. BizTalk is Microsoft implementation of the Hub and Spoke architecture. BizTalk is the Hub that receives messages from the spokes and routes these messages to their destinations. 1.2 How BizTalk Works? BizTalk is designed to receive inbound messages, pass them through some form of logical processing, and then deliver the result of that processing to an outbound location or subsystem. BizTalk views all data and events as “Messages”.

A message is a finite entity within the BizTalk MessageBox. Messages are immutable once they are published to the MessageBox. Messages have context properties and zero-to-many message parts. BizTalk extends the idea of TCP/IP Port to include other protocols such as file, SQL, etc. A Port is either a “receive location” (i.e. BizTalk would listen on this port for incoming messages) or a send port that BizTalk will send through messages.

BizTalk groups receive locations into what it calls a “receive port” and groups send ports into send port group (I know the naming is a little bit confusing). A receive location receives messages using a component called “BizTalk Adapter”. A BizTalk Adapter is a component that understands the Transport Protocol like HTTP, FTP etc. The received message is processed by a pipeline associated with the receive location. A pipeline is a component that performs transformations on messages to prepare them to enter or leave BizTalk Server MessageBox database. In the receive locations, the main task of a receive pipeline is to convert input data into proper XML. Likewise, in send ports, the main task of a send pipeline is to convert input data into the proper data format for the transport protocol (Adapter).

Figure 1.3: BizTalk Functional view. A receive or send port can have one or more maps associated with it. A Map is a component that transforms a message from one XML schema to another XML Schema.

The resulting message is then published to the MessageBox database. The MessageBox evaluates active subscriptions and routes the message to those orchestrations, and send ports with matching subscriptions. A subscription is a collection of comparison statements, known as predicates, involving message “context” properties and the values specific to the subscription. Subscriptions match particular context properties for a message and determine endpoints (orchestrations, or Send Ports), which are interested in processing it. Orchestrations may process the message and publish messages through the MessageBox to a send port where it is pushed out to its final destination. An Orchestration is an executable business process (workflow) that can subscribe to (receive) and publish (send) messages through the MessageBox database. The design of a BizTalk solution begins with selecting which of the BizTalk tools to use for each of these three simple yet all-important tasks (receive, process, and send a message).

I'll introduce these tools in this chapter and continue to explore them further throughout the rest of the book. First, let’s start with a simpler overview:• Messaging is a component that provides the ability to communicate with a range of other software. BizTalk introduces the concept of Ports and adapters. Ports and adapters provide the logical abstraction for sending and receiving messages to and from BizTalk. They allow you to code your application in a generic fashion and not worry about the implementation details of how these messages will be consumed and delivered. A port is a logical construct that can receive and send messages to/from the BizTalk MessageBox.

A receive port must be associated with a specific receive location to accept the message. The receive location is tied to a specific adapter, which provides the implementations of the Transport protocol, and a pipeline.

A Pipeline provides a way to examine, verify, and potentially modify messages as they are received and sent out from BizTalk. I will discuss Messaging in more details in chapters 2, 3, and 4. • Orchestration engine supports the creation and running of graphically-defined processes. Bmw Engine Serial Numbers. Orchestration is the most powerful tool within the BizTalk Server toolbox. Think of orchestrations as a special form of workflows similar to the Windows Workflows Foundations. Orchestrations are used to graphically model process workflow and provide the primary mechanism to implement business process automation within BizTalk based project.

Orchestrations are created within Visual Studio and compiled into.NET assemblies that are registered in the BizTalk Management Database. Assemblies deployed to this database must also be installed into the Global Assembly Cache (GAC).

I will discuss Orchestrations in more detail in Chapter 6. • A Business Rule Engine (BRE) that evaluates complex sets of rules. The BRE allows you to model business rules using a simple GUI. It is designed to allow for versioning and modification of implemented business rules without the need to change the processes within BizTalk that uses them. You use the BRE to implement logic that requires frequent updates such as a discount policy percentage or calculations that are updated frequently as a result of legal or government regulations.

I will discuss the Business Rule Engine in more detail in Chapter 7. • The Health and Activity Tracking (HAT) tool lets you monitor and manage the engine services. HAT will be presented in more details in section 1.2.3 later in this chapter. • Enterprise Single Sign-On (SSO) facility which provides the ability to map authentication information between Windows and non-Windows systems. • Business Activity Monitoring (BAM) framework you use to build tracking and monitoring into you BizTalk based solutions.

Information workers will use it to monitor the state of the running business processes. Business Activity Monitoring (BAM) and Business Activity Services (BAS) provide the infrastructure to perform application instrumentation and metric reporting. BAM provides the ability to instrument your application to provide business-level data that is much more relevant than the default system-level information which is available in the base product.

I will discuss BAM in chapter 9. • Business Activity Services (BAS), which is used by information workers to set up and manage interactions with transacting partners. BAS provides a simple yet powerful way to display metric data from BAM and other system-level subservices using Microsoft SharePoint Portal Services. Most organizations will integrate the BAS portal into an existing SharePoint infrastructure rather than build an entire SharePoint site around BAS itself. 1.2.1 How the Publish/Subscribe mechanism Works? Understanding the Publish/Subscribe mechanism is central to understanding BizTalk. The idea of subscription can be: • Messages are received (via receive handlers or from orchestration Send ports) and handed to the BizTalk engine.

• The engine stores data about the message called contextual properties. • The engine queries a rule store to determine a set of matching subscriptions. • For each matched subscription, a record is entered into an application-specific queue, and is associated with a specific instance of a service. • Queued messages are then de-queued using a number of separate worker threads, and routed to the designated instances of services.

• The service (either an orchestration or a send port) handles the message. In essence, the subscription mechanism acts as a rules engine that infers, from a set of predicate-based rules, which messages should be handed to which service instances. Subscriptions are defined primarily in terms of message context property predicates.

Each predicate describes an expression used in subscription matching. For example, a predicate might define an expression that tests for messages with an orderQty property value that is greater than 3500. BizTalk supports the following predicate types: • Bitwise AND • Equals • Exists • GreaterThanOrEquals. • GreaterThan. • LessThanOrEquals.

Predicates are collected into ‘And’ and ‘Or’ groups. BizTalk does not allow these groups to be nested, and therefore is somewhat constrained in terms of the subscription rules which potentially could be created. Subscriptions are created by services, and results in messages being routed to orchestrations or send ports. 1.2.2 BizTalk Databases When you install and configure BizTalk it creates several databases in the assigned SQL Server. You do not need to have all these databases it depends on the features that you configure through the BizTalk Configuration that I will discuss in section.

The following are mandatory BizTalk databases: • BizTalk Management database: This database is the central meta-information store for all instances of BizTalk Server. • BizTalk MessageBox database: This database is used by the BizTalk Server engine for routing, queuing, instance management, and a variety of other tasks. • BizTalk Tracking database: This database stores health monitoring data tracked by the BizTalk Server tracking engine. • Rule Engine database: This database is a repository for: • Policies, which are sets of related rules.

• Vocabularies, which are collections of user-friendly, domain-specific names for data references in rules. • SSO database: The Enterprise Single Sign-On database securely stores the configuration information for receive locations.

The following are optional databases that based on your configuration BizTalk would create: • BAM Analysis: This database contains Business Activity Monitoring (BAM) OLAP cubes for both online and offline analysis. • BAM Archive: This database archives old business activity data. Create a BAM Archive database to minimize the accumulation of business activity data in the BAM Primary Import database. • BAM Notification Services Application database: This database contains alert information for BAM notifications.

For example, when you create an alert using the BAM portal, entries are inserted in the database specifying the conditions and events to which the alert pertains, as well as other supporting data items for the alert. • BAM Notification Services Instance database: This database contains instance information specifying how the notification services connect to the system that BAM is monitoring • BAM Primary Import database: This is the database where BAM collects raw tracking data. • BAM Star Schema: This database contains the staging table, and the measure and dimension tables.

• Tracking Analysis Server: This database stores health monitoring online analytical processing (OLAP) cubes. • Windows SharePoint Services configuration database: This database contains all of the global settings for the server. • Windows SharePoint Services content database: This database contains all of the site content, such as list items and documents. BizTalk Server runtime operations typically use BizTalk Server Management database, MessageBox databases, tracking database, and SSO database. Depending on the BizTalk Server functionality that you use, you may have some or all of the other databases in the table. Program Laporan Keuangan Sederhana.

1.2.3 BizTalk SQL Jobs BizTalk uses the following SQL Jobs and you should schedule them to run on the SQL server in order to clean the BizTalk MessageBox[1] database: • Backup BizTalk Server (BizTalkMgmtDb): This job performs full database and log backups of the BizTalk Server databases. • CleanupBTFExpiredEntriesJob_BizTalkMgmtDb: This job cleans up expired BizTalk Framework (BTF) entries in the BizTalk Management (BizTalkMgmtDb) database. • MessageBox_DeadProcesses_Cleanup_BizTalkMsgBoxDb: This job detects when a BizTalk Server host instance (NT service) has stopped and releases all work that was being done by that host instance so that it can be worked on by another host instance. • MessageBox_Message_Cleanup_BizTalkMsgBoxDb: This job removes all messages that are no longer being referenced by any subscribers in the BizTalk MessageBox (BizTalkMsgBoxDb) database tables. • MessageBox_Message_ManageRefCountLog_BizTalkMsgBoxDb: This job manages the reference count logs for messages and determines when a message is no longer referenced by any subscriber.

• MessageBox_Parts_Cleanup_BizTalkMsgBoxDb: This job removes all message parts that are no longer being referenced by any messages in the BizTalk MessageBox (BizTalkMsgBoxDb) database tables. All messages are made up of one or more message parts, which contain the actual message data. • MessageBox_UpdateStats_BizTalkMsgBoxDb: This job manually updates the statistics for the BizTalk MessageBox (BizTalkMsgBoxDb) database. • PurgeSubscriptionsJob_BizTalkMsgBoxDb: This job purges unused subscription predicates from the BizTalk Server MessageBox (BizTalkMsgBoxDb) database. 1.2.3.1 How to activate and schedule these jobs? By default these Jobs are enabled and scheduled to run at Midnight.

You might want to change these schedules depending on the schedules you setup for the data conversion processing discussed earlier. Figure 1.4: SQL Management Studio Start Microsoft SQL Management Studio select SQL Server Agent and expand Jobs as shown in figure 1.4. You will find the jobs defined there. To change a job schedule right click on the job and select properties, shown in figure 1.5. Select schedules and the select edit to edit the schedule.

1.3 About BizTalk 2009 The new BizTalk 2009 have many improvements over the BizTalk 2006 R2, most importantly is the full support for Windows 2008, Visual Studio 2008 and Hyper-V technologies. Updated with.Net 3.5 framework SP1 and improved Failover Clustering that can be deployed in multi-site cluster scenarios, where cluster nodes could reside on separate IP subnets and avoid complicated VLANs. Business Activity Monitoring has been enhanced to support scalable real-time aggregations. EDI and AS2 Protocols now support multiple message attachments, configurable auto message resend, end-to-end filename preservation, improved reporting to address new features, and Drummond re-certification for AS2. Complete support for Application Lifecycle Management (ALM) with Microsoft Team Foundation Server (TFS).

Figure 1.9: BizTalk Project Properties in Visual Studio 2008. A few details you should be concerned with: • Signing the Assembly: you should always make sure that any BizTalk project assemblies are signed and any assemblies they reference are signed and deployed to Global Assembly Cache ( GAC). • The BizTalk Group settings in Visual Studio 2008 or deployment in Visual Studio 2005 are set to your BizTalk development environment. It is a good practice to get into the habit of setting an application name for the project that you develop and not deploy to the default “BizTalk Application 1”. Figure 1.10: BizTalk Project Properties in Visual Studio 2005.

Now you are ready to add artifacts to your BizTalk Project. The artifacts can be Schema, Pipeline, Map, or Orchestration. You just right click on the project and select add new item and select one of the items in the dialog as shown in figure 1.11.

After you are finished adding the artifacts you can build or deploy the project with a single click. If you right clicked on the project and selected “Deploy”, Visual Studio will check the build status of the project and if the project is not built it will try to build it. If the project is successfully built Visual Studio will “Deploy” it to the BizTalk Group you set in figure 1.7 and 1.8. Deploy means that visual studio would GAC the assembly to the local computer GAC and add an entry into the BizTalk Group Management database. Figure 1.11: Add New Item for BizTalk Project in Visual Studio 2008. If the build and deploy goes fine you will see in the output something similar to Listing 1.1. Note that I have edited the listing to keep it short.

Figure 1.13: Platform settings. A host defines which adapters, orchestrations, and ports are going to run in its context and under its security context. A host instance is just that, an instance of the host. The instance is actually just a service that runs on the machine called BTSNTSvc.exe.

This process provides the BizTalk engine a place to execute, and allows for instances of different hosts to be running on one machine at a given time. Each host will end up being a separate instance of the BTSNTSvc.exe service from within the Windows Task Manager. 1.6.2 Health and Activity Tracking (HAT) The Health and Activity Tracking (HAT) (shown in figure 1.14) is a GUI tool that provides utilities for reporting, analyzing, and debugging data and messages which are archived in the BizTalk Tracking databases. For live data, use the BizTalk Administration Console.

You can either use the included queries that appear on the Group Hub page, or select the New Query tab to create custom queries and reports. Figure 1.14: Health and Activity Tracking (HAT). You use HAT to debug orchestration by running an instance of an orchestration then right click on it and select orchestration debugger shown in figure 1.15. You can also use HAT to see the message flow of a message by right clicking on the message and selecting message flow shown in figure 1.16. 1.7 BizTalk Project Architecture How would you organize a BizTalk Based solution? In general, you should always strive to design the BizTalk solution to be layered as Figure 1.17 shows, designing the solution in a layered manner simplifies the project and makes it easier to maintain.

Figure 1.16: Message Flow. You can design the solution to consist of at least the following three layers: • External Interface Layer: this layer contains all different maps, pipelines, schemas that would consume external data feeds and convert them to a unified internal schema that is internal to your enterprise.

This can be implemented in a DLL or several DLLS, one DLL per feed, or several DLLs per feed. It depends on your solution, future changes (new third parties, external systems etc). • The Business Logic Layer: this is where all the business logic gets implemented in orchestrations, maps, business rule polices, etc. • Enterprise System Interface: This layer is similar to the External Interface layer except it is for internal systems in the enterprise. You should note that we are always changing to a unified or a simple schema internal to the system.

I have seen many BizTalk solutions where they had so many orchestrations or maps performing the same logic only because the third party supplied different schemas. On any BizTalk solution you should strive to: • Maximize the use of the BizTalk out-of-box features, and utilities, • Minimize the amount of code (this includes, custom pipeline components, custom adapters, custom libraries, etc), and • Prefer to buy components that are not out of the box from BizTalk than developing them. As for structuring the code, there is no single answer to fit every solution. There is a tendency by many BizTalk developers to put the schemas, maps, orchestrations, and pipelines in separate projects. This is not suitable for large, medium and sometimes not even for small solutions. When you are structuring the solution code, you should consider deployment of the DLLs and other artifacts, as well as the modularity of the solution. Figure 1.17: General BizTalk Based Solution.

For example, adding new partners and additional functionality should be made easy. You should also consider minimizing the dependency between developers. For example, when all they developers working on the same project, they may hinder each other’s progress. Also if one developer is specialized in schemas, then other developers become dependent on her and waiting for her to finish the schemas so that they could start.

Note: The names of the jobs change depending on the database names given during configuration. If you have deployed multiple MessageBox databases in your environment, there will be several jobs for each MessageBox. Over 20 years of experience developing software solutions, leading the Architecture, Design, and Refactoring of many large projects.

At FCT led the architecture for several project in parallel, Motion (Releases 7, 8 and 9), Integrating with 3rd party providers (PPSA/Chattel Project) and the identity management initiative among others. At Bell/ Sympatico MSN web modernization project led the BizTalk development team, delivered the solution architecture, developed the core framework, mentored and supported the team members on the Solution and BizTalk technical issues. At Siemens led the architecture, design and implementation of the Real time server and Newscaster for the HiPath Pro (A CRM System). Holds IEEE/CSDP, MCSD (6.0, and.Net) and Oracle certifications.

Act as a referee for the IEEE Software magazine. With International and US experience in Financial, Insurance, Retail, Marketing and Telecommunications services automation.

I agree with everything written this far. All solid info. I have a few addons, coming from a fellow freelancer working with BizTalk since 2002: Unit testing. It's not easy to do, but check out BizUnit. A Codeplex based toolset written and maintained by Kevin Smith. One of the early BizTalk heroes:-) Deployment / getting things into production But also keep in mind that none of the day to day development stuff will prepare you for the part of the project where you have to deploy the app and make sure that it is 'manageable' by operations. This can be quite complex, and is a topic in it's own right.

Check out Apress Pro BizTalk 2009, it's got a decent (IMO) chapter on this. The entire development process around BizTalk. The first two chapters of the same book will give you a good impression on what a BizTalk project is about. Where to use it, and where to not use it, how to organize projects, and name your stuff. Really a good collection of info that you would only get by reading 5-6 years of blogs back in time:-) And one last thing. Depending on the roles on the project, you might be asked to optimize and tune BizTalk.

And if they don't ask you. Make sure that you ask if others have done that, because you have to do it. BizTalk should always be tuned towards what it is supposed to do.

Low latency vs high throughput, tuned according to hardware, correct setup and config of network around the SQL boxes, etc etc etc. This can be hairy stuff, and you should be careful not to jump into it before reading up on it all. But it's a subject we as freelancers are often expected to be able to deal with. So thought I might bring it up. BizTalk x64 processes on an x64 box runs really bad out of the box, actually worse than on the x86 processes. The 64 bit processes need to be tuned to really use all the MEM that are availble to them.

A bag of mixed tips and I hope you can use some of them! And good luck! It can be a tough start, but if used right, BizTalk can be a great product/toolset. And remember. If it is ugly, or hard, or both. You are doing it wrong.

And don't be afraid to dive into.net code, and bolt it onto the BizTalk box. We all do it.

Some just won't admit to it:-D. As per Thomas and Shiraz - set up an environment and get your hands dirty. If you haven't done so already, download and install But just to temper your expectation, IMHO expertise in BizTalk (or any other EAI / BPM / ESB product) can take years to accumulate.

It isn't clear whether you are developing for a client with an established BizTalk installation, or if this is the client's first BizTalk deployment. If so, one thing not to be underestimated is that the operational considerations of running a production BizTalk environment (performance, redundancy, reliability, auditing, tracking, monitoring with SCOM etc) are as complex as the development and testing - but understanding of this will be important to 'walk the talk'.

Dev, start with some a simple EAI type mapping project, and then work your way through the progress to some common messaging patterns (e.g. Batching with aggregator), and then move into the BPM type orchestrations. You can probably leave BAM and the BRE for later.

+1 to tomasr for mentioning the virtual labs. Getting hands-on is definitely the way to go, as Shiraz Bhaiji also mentions. Hopefully you're not starting with BizTalk 2006, and can go with the latest: 2010.

If that's the case, you can get the Developer Ed. Of BizTalk 2010 for free now (see link from nonnb). I'd also recommend Richard Seroter's book: 'SOA Patterns with BizTalk Server 2009' (). There are many ways to do the 'wrong' things with BizTalk, and this book does an excellent job of walking through both the how and the why of building BizTalk solutions (with the code samples available from ).

And yes, it pretty much takes a whole book to go through it all. It's a good (more readable) companion to the Pro BizTalk 20xx series (which is generally better for very specific questions/tasks).