Oracle’s Application Container Cloud allows you to run Java SE, Node.js and PHP applications (and more is coming) in a Docker container hosted in the Oracle Public Cloud (OPC). Node.js can crash when applications do strange things. You can think of incorrect error handling, blocking calls or strange memory usage. In order to host Node.js in a manageable, stable and robust way in an enterprise application landscape, certain measures need to be taken. Application Container Cloud provides many of those measures and makes hosting Node.js applications easy. In this blog article I’ll describe why you would want to use Oracle Application Container Cloud. I’ll illustrate this with examples of my experience with the product.
Forking (not a real cluster)
Node.js without specifically coded forking / clustering, runs in a single OS thread.This single thread uses a single CPU. You can fork processes/workers to use multiple CPU’s. Node.js provides (among other things) the core module cluster to do this. It depends on IPC between master and workers (which can be cumbersome to manually code). Also there is no easy way to graceful shutdown workers and restart them without downtime. StrongLoop (IBM) has developed modules such as strong-cluster-control and strong-store-cluster to make this more easy. If the master process fails however, you still have a problem.
Multiple Node.js instances (a real cluster)
If you want to provide true clustering over Node.js instances and not just for forks / child processes you need additional tooling; process managers. On the express site is a short list of the most popular ones. StrongLoop Process Manager, PM2 and Forever. For example StrongLoop Process Manager encapsulates features such as nginx load balancing, supervision as well as devops functions of build, deploy, monitor and scale on remote servers and Docker containers. I have not tried this though but can imagine this requires some setting up.
Application Container Cloud
Oracle Application Container Cloud provides out of the box with very little configuration, a set of clustering and stability features to allow Node.js to run in an enterprise landscape. If you want to get up and running quickly without thinking about many of these things, you should definitely look at Application Container Cloud.
- Application Container Cloud provides an extensive user interface and API to allow you to do most required tasks easily. You do not need an expert to configure or operate Application Container Cloud (if you like, you can still hire me though ;).
- Node.js instances (inside Docker containers) can be added / deleted on the fly. The load-balancer (also hosted in the OPC) is configured for you. Thus you get cluster functionality (important for scaling and high availability) over Node.js instances with almost no effort.
- If one of the instances does not respond anymore, the load-balancer notices and sends requests to other instances. It does this by checking the root of your application.
- When a Node.js instance crashes (by means of the Node.js process exiting), it is automagically restarted (you can confirm this by looking at the log files)
Creating and deploying an application
Creating an application is easy. You can use the provided API to easily create and manage an application or use the web interface. The format of the application to upload is also very straightforward. You upload a ZIP file with your complete application and a manifest.json file which can contain as little as for example:
{
"runtime":{
"majorVersion":"0.12"
},
"command": "node app.js",
"release": {},
"notes": ""
}
You can also add a deployment.json file which can contain variables you can use within your application. These variables can be updated with the web interface or the API and the application will be restarted to apply the new settings.
After multiple uploads and downloads, I noticed from the log files that my archive did not update. I changed the filename of the zip-file I uploaded and the problem did not occur anymore. Apparently, sometimes the uploaded file cannot be overwritten by a new upload (when doing certain actions at the same time probably). This can be dangerous. For a Continuous Delivery scenario I suggest using unique filenames to avoid this issue.
You can easily use a Maven build with the maven-assembly-plugin to create the archive to be deployed in case you want to embed this in a Continuous Delivery pipeline. You can of course also use the provided Developer Cloud Service. Read more about that here. There is a nice demo here.
Management
Application Container applications can be restarted, stopped, started, deleted. You can add and remove instances. You can not restart or start/stop a single instance. When a single instance does not respond anymore (on the application root URL), the load-balancer notices and does not send requests to that instance anymore. If you want to make it available again, you have to restart your entire application (or remove and add instances and hope the faulty one is removed and recreated). When adding instances, existing instances will keep running and for the new instances, when they are done creating, the load-balancer is updated. When removing instances, the load-balancer is also updated.
If the application ends the Node.js process, a new process will to be started immediately. This new process used the same IP and hostname during my tests. If I start/stop or restart the application, the hostname and IP get new values.
Log files
Log files can be downloaded from running applications.
If an application creation fails, you can also download the log files. See an example at ‘Accessing the API’ below. This can greatly help in debugging why the application cannot be created or started. Mostly the errors have been application errors. For example:
Aug 12 14:14:42: Listening on port: 8080
Aug 12 14:14:47: Request on url: /
Aug 12 14:14:47: /u01/app/app.js:49
Aug 12 14:14:47: if (<strong>req.url.startsWith</strong>("/oraclecloud")) {
Aug 12 14:14:47: ^
Aug 12 14:14:47: TypeError: undefined is not a function
Aug 12 14:14:47: at Server. (/u01/app/app.js:49:15)
Aug 12 14:14:47: at Server.emit (events.js:110:17)
Aug 12 14:14:47: at HTTPParser.parserOnIncoming [as onIncoming] (_http_server.js:492:12)
Aug 12 14:14:47: at HTTPParser.parserOnHeadersComplete (_http_common.js:111:23)
Aug 12 14:14:47: at Socket.socketOnData (_http_server.js:343:22)
Aug 12 14:14:47: at Socket.emit (events.js:107:17)
Aug 12 14:14:47: at readableAddChunk (_stream_readable.js:163:16)
Aug 12 14:14:47: at Socket.Readable.push (_stream_readable.js:126:10)
Aug 12 14:14:47: at TCP.onread (net.js:540:20)
Do mind that the Node.js version running in the Application Container Cloud currently is 0.12.14. In this version for example you have to start Node.js with the –harmony switch if you want to use startsWith on the String object. My bad obviously.
Managing access
Even though there is an extensive user interface for management and monitoring of the Node.js application and also an extensive API (see more about about that below), the load-balancer configuration is not directly accessible. You can for example not configure a specific load-balancer probe or procedure to check whether the instance is up again. Also I did not find a way to easily secure my Node.js services to avoid them from being called by just anyone on the internet (I might have missed it though). It would be nice if for example the Node.js instance was behind an API platform or even just a configurable firewall.
Out of memory issue
I have found one situation which is not dealt with accordingly. I tried the following piece of code here.
This exposes 3 endpoints. /crashmenot, /crashmebykillingprocess and /crashmebyoutofmemory. I had created two instances of my application. By the hostname in the response message I could see which instance I was accessing.
Killing the server by out of memory however (GET request to /crashmebyoutofmemory) gave me locally the following exception in a local console (among some other things):
FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - process out of memory
When running this in Application Container Cloud I did not see the out of memory error. I did specify though that the instance should have 1Gb of RAM (the minimum allowed). An instance can currently be assigned as much as 20Gb of RAM.
Killing the process by causing out of memory, did not cause the instance to be auto restarted. The load-balancer however send me to the instance which was still working though. When I killed all instances with out of memory, it did not automatically recover. After the manual restart, the instances worked again. When the instances were running, a manual restart did not cause noticeable downtime.
Accessing the API
Application Container Cloud has an extensive and well documented API. In order to debug issues it is nice you can access the Application Container Cloud API by first determining the URL to call.
In this example my application is called CrashDummy7 and my Identity domain is accstrial. You can use an API call like (of course replace user and pass with actual values):
curl -k -i -X GET -u user:pass -H "X-ID-TENANT-NAME:accstrial" https://apaas.europe.oraclecloud.com/paas/service/apaas/api/v1.1/apps/accstrial/CrashDummy7
HTTP/1.1 200 OK
Date: Fri, 12 Aug 2016 18:10:57 GMT
Server: Oracle-Application-Server-11g
Content-Length: 1335
X-ORACLE-DMS-ECID: 005EUFDqWVE3z015Rvl3id00068i000ZBB
X-ORACLE-DMS-ECID: 005EUFDqWVE3z015Rvl3id00068i000ZBB
X-Frame-Options: DENY
Content-Language: en
Content-Type: application/json
{"identityDomain":"accstrial","appId":"a2c7998b-c905-466a-b89f-bb9b8c4044dd","name":"CrashDummy7","status":"RUNNING","createdBy":"maarten.smeets@amis.nl","creationTime":"2016-08-12T17:43:17.245+0000","lastModifiedTime":"2016-08-12T17:43:17.202+0000","subscriptionType":"MONTHLY","instances":[{"name":"web.1","status":"RUNNING","memory":"1G","instanceURL":"https://apaas.europe.oraclecloud.com/paas/service/apaas/api/v1.1/apps/accstrial/CrashDummy7/instances/web.1"},{"name":"web.2","status":"RUNNING","memory":"1G","instanceURL":"https://apaas.europe.oraclecloud.com/paas/service/apaas/api/v1.1/apps/accstrial/CrashDummy7/instances/web.2"}],"runningDeployment":{"deploymentId":"5a9aef17-8408-494c-855e-bc618dfb69e9","deploymentStatus":"READY","deploymentURL":"https://apaas.europe.oraclecloud.com/paas/service/apaas/api/v1.1/apps/accstrial/CrashDummy7/deployments/5a9aef17-8408-494c-855e-bc618dfb69e9"},"lastestDeployment":{"deploymentId":"5a9aef17-8408-494c-855e-bc618dfb69e9","deploymentStatus":"READY","deploymentURL":"https://apaas.europe.oraclecloud.com/paas/service/apaas/api/v1.1/apps/accstrial/CrashDummy7/deployments/5a9aef17-8408-494c-855e-bc618dfb69e9"},"appURL":"https://apaas.europe.oraclecloud.com/paas/service/apaas/api/v1.1/apps/accstrial/CrashDummy7","webURL":"https://CrashDummy7-accstrial.apaas.em2.oraclecloud.com"}
You can download logfiles such as:
By doing:
curl -u user:pass -k -i -X GET -H "X-ID-TENANT-NAME:accstrial" https://accstrial.storage.oraclecloud.com/v1/Storage-accstrial/_apaas/forwarder8/d177756e-399e-414f-bc9b-d2bb6e698ae6/logs/web.1/f0f62da1-b240-4251-aabe-9c1244069ed6/server.out.zip -o file.zip --raw
Here my application was called forwarder8.
Finally
I have explained when and why it is worthwhile to check out Oracle Application Container Cloud. Also I’ve shown some samples on how easy it it to use. I’ve described some of my experiences and have given some tips on how to efficiently work with it.
I have not touched on for example the easy integration with Oracle database instances running inside your cloud. The Oracle DB driver is supplied and configured for you in your Node.js instances and can be used directly with only simple configuration from the UI or API.
If you want to know more I suggest watching the following presentation from Oracle here. If you want to know more about the Oracle Db driver for Node.js, you can watch this presentation. And using the driver inside Application Container Cloud service here.
Great stuff!
