SOA Suite 12c: In-depth look into Managed File Transfer (1/3) SOASuite12c mft blog

SOA Suite 12c: In-depth look into Managed File Transfer (1/3)

With the release of SOA Suite 12c a new product was revealed. The first sighting was at Oracle OpenWorld 2013. On the Fusion Middleware Demo Grounds there was a demo shown where B2B was integrated with a product called Managed File Transfer (shortened as MFT). At that same conference I gave a presentation about this product together with the product manager of MFT, Dave Berry. This blog gives an in-depth look into Managed File Transfer.

There is a growing problem with FTP in the enterprise where there is a lack of control, visibility, security and reliability. The lack of control is due to the uncontrolled proliferation of FTP servers & clients. Departments are creating stand-alone FTP servers and configuring users where needed. There is no central FTP server. Because of this there is no global visibility of the exchange of crucial data files – including customer data. It is highly possible that these FTP servers are not integrated with enterprise security standards where as FTP servers are rarely integrated with directories. Because these FTP servers run stand-alone they are a single point of failure and rarely offer HA capabilities, which effects the reliability. This can be tackled using Managed File Transfer.

Lack of control, visibility, security & reliability

Lack of control, visibility, security & reliability

What is Managed File Transfer?

MFT is a simple and secure End-to-End Managed File Gateway. At the base MFT uses an “Embedded” (S)FTP / SSH server which support HA clustering. MFT has a scalable architecture, which mean it can easily be expanded by adding another Weblogic node to the cluster.  It also includes an extensible framwork for pre/post processing of files. MFT integrates with Standards Based Middleware like (S)FTP, SOA, B2B, Service Bus and Web Services.

Managed File Transfer has support file delivery of very large files ~ 500GB+ which can be ZIP compressed/decompressed and encrypted and decrypted using PGP encrypting. One main feature is the possibility to send files via Web Services using Pass-by-Reference (Claim/Check pattern). It can be a reference to a FTP or File location, but there is also inline (base64) support. File transfers can be scheduled and delivery to target endpoints can be paused, resumed and resubmitted. If delivery fails then files can automatically be retried.  MFT can send notifications when files are delivered or when transfers fail. Deliveries can be done through HTTP, JCA, FTP or in-memory. Managed File Transfer support (custom) callouts to archive, move and delete files.

Transactions can be audited and monitored using the included Web UI for designing, monitoring and administration of transfers. With this lightweight Web based Design-Time Interface you can easily build, edit and deploy end-to-end transfers. I will go into all the details a bit later on.

Use cases for Managed File Transfer

Internal Business Integration like Application to Application File sharing. Managed File Transfer can integrate with SOA / SB.
External Partner Integrations like (s)FTP File gateway outside firewall and Managed File Transfer can integrate with B2B / EDI.
Commercial off-the-shelf/Cloud like OOTB End-to-end delivery channels and Application to Application solutions.
Other use cases include File Management beyond business integration, delivery of media to mobile phones.

Functional Use Case Patterns

Using Managed File Transfer as standalone application:

Managed File Transfer - Use Cases

MFT can transfer files on its own. This can be done using the embedded FTP and sFTP servers and the (local) file systems.

Using Managed File Transfer to integrate with SOA (same pattern goes for SB, SOAP):

Managed File Transfer - Use Cases

MFT can integrate with a SOA application. SOA can be the source or target of a transfer.
It is also possible to configure a SOA target endpoint in one transfer that is the source of another transfer.

Using Managed File Transfer to integrate with B2B (same pattern goes for Healthcare):

Managed File Transfer - Use Cases

MFT can integrate with a B2B application. B2B can be the source or target of a transfer.
It is also possible to configure a B2B target endpoint in one transfer that is the  source of another transfer.

Using Manager File Transfer for Hybrid Integrations:

Managed File Transfer - Use Cases

MFT as a key participant in a web of data transfers, which can include multiple application types like B2B, SOA and file shares.

All Web-based UI Functionality

Most of the functionality of Managed File Transfer can be used, using a Web-based UI. MFT, just like Service Bus, has a console. With this console you have design, monitor and administer functionality in your possession.

MFT UI Functionality

As Designer you can set up (create, read, update and delete) sources, targets and transfers. A transfer combines one source and one or more targets. A designer can also deploy and test transfers.

As Monitor you have application-level functionality like starting and stopping embedded FTP and sFTP servers; pause, resume and resubmit transfers; Enable, disable and undeploy sources, targets and transfers.  A monitor can also use the dashboard to look at metrics including artifact deployment details and dependence relationships, find specific file transfers, view (active) deliveries and look at recent errors and flow trace reposts.

As Administator you can import and export metadata and make backup or restore MFT configurations. An administator can add users and their roles; configure user directory permissions, the MFT server, B2B and Healtcare domains and the embedded FTP and sFTP servers.  It is also possible to archive, restore and purge instance data and payloads (transfered files).

Besides the MFT console it is also integrated in the Enterprise Manager (EM) and there are JDeveloper Bindings to easily integrate with SCA Composites. In the EM you can configure the MFT app & associated MBeans and look at/change global defaults.

Closer look at Designer functionality

Click the Design menu item to navigate to the designer. Here a designer can set up (create, read, update and delete) sources, targets and transfers. The startpage of the designer shows the artifact search page which by default shows a list with created transfers, but it is also possible to search for sources, targets or all artifacts.

MFT 12c: Startpage for Designers

MFT 12c: Startpage for Designers

At the left side there is a secundary menu to create, edit, rename and delete transfers, sources and targets. To create a new transfer click on the Transfers item or click on the green plus icon if the Transfers item is already selected. A modal is displayed where you can set a name for the transfer and set an optional description.

MFT 12c: Create a new transfer

MFT 12c: Create a new transfer

After creating the initial transfer a new page opens where you can create/add a source, create/add one or more targets and setup content filters, payload access and (custom) callouts. At first the page is a blank slate.

MFT 12c: Complete transfer with source and target

MFT 12c: Complete transfer with source and target

Create or Add a Source

To create or add a new source you can drag and drop an existing source to the sources section, create a new source from the navigation item and then drag and drop it on the sources section or use the links to create/add a source. The same applies to transfers, but let start with creating a new FTP source.

MFT 12c: Create FTP Source for new Transfer

MFT 12c: Create FTP Source for new Transfer

Give the Source a name and an optional Description. Then select the Type of source you want to create i.g. FTP Embedded when you want to use the FTP server provided by Managed File Transfer. Every type had it’s own init parameters you need to fill in.

At the time of writing there are eleven types of sources:

Type Parameters SOA Suite 12c: In-depth look into Managed File Transfer (1/3) Designer 05 select source
(s)FTP Embedded Relative folder path, the folder path needs to have access to the FTP server first. Access can be created under Administration -> Embedded Servers -> User Access.
FTP Remote Host, relative folder, user, password (2x), control port and optional SSL switch (checkbox)
File Absolute path on file system, the user where MFT is running on (i.g. oracle) needs to have access to file system.
SOAP Relative URL, the value is added after http://mfthost:port/mftapp/services/transfer/.
SOA, Service Bus and ODI Relative URL, the value is added after http://mfthost:port/mftapp/services/transfer/ and optional domain, when domain is selected the Source is only active for that domain.
B2B Tranding Partner Name and optional domain, when domain is selected the Source is only active for that domain.
Healthcare Endpoint Name and optional domain, when domain is selected Source is only active for this domain.

 

After creating a new Source and adding it to the “Drop Source” area the settings and type information of the source are displayed.

MFT 12c: Add FTP Source to new Transfer

MFT 12c: Add FTP Source to new Transfer

Control which files to process and who had access

When configuring a source you can control which files to process by using Content Filters. If you only want to allow/process PDF files add a Content Filter for *.pdf. It is also possible to only give certain people access to the payload by adding users, group and roles defined in weblogic. This configuration is applicable only to the targets that have a delivery method as FTP reference.

MFT 12c: Control which files to process

MFT 12c: Control which files to process

Create or Add one or more targets

A transfer can have one or more targets (i.e. a Fan Out). Create or adding Targets works the same as with Sources. There are some subtle changes like less types and different parameters. Instead of eleven types there are only nine types. You can’t use the embedded FTP as a target.

With most types you don’t have any extra options then to specify a subfolder. FTP and File targets for instance can be used in multiple transfers but can still route file to a different subfolder. The other types (non file targets) like ODI, B2B, Healthcare have the extra delivery option to pass the content of the file as reference or als inline binary. The one(s) that stand out are the SOA, Service Bus and SOAP types. These types have a special delivery option where you can pass the content as reference or as inline binary. When you choose reference you can select the type of reference, (s)FTP or File. The SOA Composite will receive a message with the reference to the file as a FTP url or a File path to the (local) file system.

MFT 12c: Add SOA Target to new Transfer

MFT 12c: Add SOA Target to new Transfer

My preference is the FTP reference, because then it does not matter if the SOA Composite or SB Service runs on a different machine/node. You can work with shares, but that needs some extra configuration.

Add a schedule and processing actions

It is possible to schedule the retrieval (source) or delivery (target) of files. Looking at the schedule functionality you have the following options:

Name: give the schedule an appropriate nameFrequency: once, hourly/minute, daily, weekly, monthly, yealy and custom. The below can change when switching frequency.Time Zone: change on which timezone schedule is active.Start Date: the datum/time when schedule becomes active.
End Date (optional): the datum/time when schedule becomes in-active.Duration (optional): the duration per cycle it should stay active
MFT 12c: Add schedule

Example 1: The tranfer can only be active between working hours. Schedule starts at 09:00 and ends 8 hours later (duration).
Example 2: A custom schedule which allows transfer on three specific times per day.

Besides adding a schedule it is possible to add pre/post processing actions. Looking at the pre-processing actions functionality you have the following options:

Actions list: add action – compress, decompress, PGP encryption and decryption – to list.* Compress file(s) into ZIP:
– Select level: Best compression, Default, Best Speed* Decompress file(s) from ZIP* Encrypt file with PGP
– Select encryption alias and type of armory (binary or ascii)* Decrypt file with PGP
– Select decryption alias
MFT 12c: Add Action

There are no post-processing actions default available at the time of writing.

Deploy the Transfer

After you configured the transfer it is time to deploy the transfer and it dependencies. First save all the changed that were made and then click on the deploy button on the top-right of the transfer page. If the transfer has dependencies to sources or targets that are not yet deployed it will automatically deploy these artifacts.

MFT 12c: Deploy Transfer and it's dependencies

MFT 12c: Deploy Transfer and it’s dependencies

Recap

In this part the purpose and basic concepts of Managed File Transfer and the creation of a FTP 2 SOA Transfer are explained.
In part two of this blog series the monitor functionality of Managed File Transfer will be dissected.


4 Comments

  1. Sam Smith June 21, 2016
  2. Pradosh June 1, 2016
  3. Raghav September 10, 2015
  4. David Gaskell March 6, 2015