With the release of SOA Suite 12c a new product was revealed. The first sighting was at Oracle OpenWorld 2013. On the Fusion Middleware Demo Grounds there was a demo shown where B2B was integrated with a product called Managed File Transfer (shortened as MFT). At that same conference I gave a presentation about this product together with the product manager of MFT, Dave Berry. This blog gives an in-depth look into Managed File Transfer.
There is a growing problem with FTP in the enterprise where there is a lack of control, visibility, security and reliability. The lack of control is due to the uncontrolled proliferation of FTP servers & clients. Departments are creating stand-alone FTP servers and configuring users where needed. There is no central FTP server. Because of this there is no global visibility of the exchange of crucial data files – including customer data. It is highly possible that these FTP servers are not integrated with enterprise security standards where as FTP servers are rarely integrated with directories. Because these FTP servers run stand-alone they are a single point of failure and rarely offer HA capabilities, which effects the reliability. This can be tackled using Managed File Transfer.
What is Managed File Transfer?
MFT is a simple and secure End-to-End Managed File Gateway. At the base MFT uses an “Embedded” (S)FTP / SSH server which support HA clustering. MFT has a scalable architecture, which mean it can easily be expanded by adding another Weblogic node to the cluster. It also includes an extensible framwork for pre/post processing of files. MFT integrates with Standards Based Middleware like (S)FTP, SOA, B2B, Service Bus and Web Services.
Managed File Transfer has support file delivery of very large files ~ 500GB+ which can be ZIP compressed/decompressed and encrypted and decrypted using PGP encrypting. One main feature is the possibility to send files via Web Services using Pass-by-Reference (Claim/Check pattern). It can be a reference to a FTP or File location, but there is also inline (base64) support. File transfers can be scheduled and delivery to target endpoints can be paused, resumed and resubmitted. If delivery fails then files can automatically be retried. MFT can send notifications when files are delivered or when transfers fail. Deliveries can be done through HTTP, JCA, FTP or in-memory. Managed File Transfer support (custom) callouts to archive, move and delete files.
Transactions can be audited and monitored using the included Web UI for designing, monitoring and administration of transfers. With this lightweight Web based Design-Time Interface you can easily build, edit and deploy end-to-end transfers. I will go into all the details a bit later on.
Use cases for Managed File Transfer
Internal Business Integration like Application to Application File sharing. Managed File Transfer can integrate with SOA / SB.
External Partner Integrations like (s)FTP File gateway outside firewall and Managed File Transfer can integrate with B2B / EDI.
Commercial off-the-shelf/Cloud like OOTB End-to-end delivery channels and Application to Application solutions.
Other use cases include File Management beyond business integration, delivery of media to mobile phones.
Functional Use Case Patterns
Using Managed File Transfer as standalone application:
Using Managed File Transfer to integrate with SOA (same pattern goes for SB, SOAP):
Using Managed File Transfer to integrate with B2B (same pattern goes for Healthcare):
Using Manager File Transfer for Hybrid Integrations:
All Web-based UI Functionality
Most of the functionality of Managed File Transfer can be used, using a Web-based UI. MFT, just like Service Bus, has a console. With this console you have design, monitor and administer functionality in your possession.
As Designer you can set up (create, read, update and delete) sources, targets and transfers. A transfer combines one source and one or more targets. A designer can also deploy and test transfers.
As Monitor you have application-level functionality like starting and stopping embedded FTP and sFTP servers; pause, resume and resubmit transfers; Enable, disable and undeploy sources, targets and transfers. A monitor can also use the dashboard to look at metrics including artifact deployment details and dependence relationships, find specific file transfers, view (active) deliveries and look at recent errors and flow trace reposts.
As Administator you can import and export metadata and make backup or restore MFT configurations. An administator can add users and their roles; configure user directory permissions, the MFT server, B2B and Healtcare domains and the embedded FTP and sFTP servers. It is also possible to archive, restore and purge instance data and payloads (transfered files).
Besides the MFT console it is also integrated in the Enterprise Manager (EM) and there are JDeveloper Bindings to easily integrate with SCA Composites. In the EM you can configure the MFT app & associated MBeans and look at/change global defaults.
Closer look at Designer functionality
Click the Design menu item to navigate to the designer. Here a designer can set up (create, read, update and delete) sources, targets and transfers. The startpage of the designer shows the artifact search page which by default shows a list with created transfers, but it is also possible to search for sources, targets or all artifacts.
At the left side there is a secundary menu to create, edit, rename and delete transfers, sources and targets. To create a new transfer click on the Transfers item or click on the green plus icon if the Transfers item is already selected. A modal is displayed where you can set a name for the transfer and set an optional description.
After creating the initial transfer a new page opens where you can create/add a source, create/add one or more targets and setup content filters, payload access and (custom) callouts. At first the page is a blank slate.
Create or Add a Source
To create or add a new source you can drag and drop an existing source to the sources section, create a new source from the navigation item and then drag and drop it on the sources section or use the links to create/add a source. The same applies to transfers, but let start with creating a new FTP source.
Give the Source a name and an optional Description. Then select the Type of source you want to create i.g. FTP Embedded when you want to use the FTP server provided by Managed File Transfer. Every type had it’s own init parameters you need to fill in.
At the time of writing there are eleven types of sources:
|(s)FTP Embedded||Relative folder path, the folder path needs to have access to the FTP server first. Access can be created under Administration -> Embedded Servers -> User Access.|
|FTP Remote||Host, relative folder, user, password (2x), control port and optional SSL switch (checkbox)|
|File||Absolute path on file system, the user where MFT is running on (i.g. oracle) needs to have access to file system.|
|SOAP||Relative URL, the value is added after http://mfthost:port/mftapp/services/transfer/.|
|SOA, Service Bus and ODI||Relative URL, the value is added after http://mfthost:port/mftapp/services/transfer/ and optional domain, when domain is selected the Source is only active for that domain.|
|B2B||Tranding Partner Name and optional domain, when domain is selected the Source is only active for that domain.|
|Healthcare||Endpoint Name and optional domain, when domain is selected Source is only active for this domain.|
After creating a new Source and adding it to the “Drop Source” area the settings and type information of the source are displayed.
Control which files to process and who had access
When configuring a source you can control which files to process by using Content Filters. If you only want to allow/process PDF files add a Content Filter for *.pdf. It is also possible to only give certain people access to the payload by adding users, group and roles defined in weblogic. This configuration is applicable only to the targets that have a delivery method as FTP reference.
Create or Add one or more targets
A transfer can have one or more targets (i.e. a Fan Out). Create or adding Targets works the same as with Sources. There are some subtle changes like less types and different parameters. Instead of eleven types there are only nine types. You can’t use the embedded FTP as a target.
With most types you don’t have any extra options then to specify a subfolder. FTP and File targets for instance can be used in multiple transfers but can still route file to a different subfolder. The other types (non file targets) like ODI, B2B, Healthcare have the extra delivery option to pass the content of the file as reference or als inline binary. The one(s) that stand out are the SOA, Service Bus and SOAP types. These types have a special delivery option where you can pass the content as reference or as inline binary. When you choose reference you can select the type of reference, (s)FTP or File. The SOA Composite will receive a message with the reference to the file as a FTP url or a File path to the (local) file system.
My preference is the FTP reference, because then it does not matter if the SOA Composite or SB Service runs on a different machine/node. You can work with shares, but that needs some extra configuration.
Add a schedule and processing actions
It is possible to schedule the retrieval (source) or delivery (target) of files. Looking at the schedule functionality you have the following options:
Example 1: The tranfer can only be active between working hours. Schedule starts at 09:00 and ends 8 hours later (duration).
Example 2: A custom schedule which allows transfer on three specific times per day.
Besides adding a schedule it is possible to add pre/post processing actions. Looking at the pre-processing actions functionality you have the following options:
There are no post-processing actions default available at the time of writing.
Deploy the Transfer
After you configured the transfer it is time to deploy the transfer and it dependencies. First save all the changed that were made and then click on the deploy button on the top-right of the transfer page. If the transfer has dependencies to sources or targets that are not yet deployed it will automatically deploy these artifacts.
In this part the purpose and basic concepts of Managed File Transfer and the creation of a FTP 2 SOA Transfer are explained.
In part two of this blog series the monitor functionality of Managed File Transfer will be dissected.