Rest API and Big data

The scenario is given below.

Large number of data  in mongo DB and the data growing  per day is 10 ,000.

Requirements and Tips

1. Needed  REST API which should perform in a proper manner.

2.REST API should be able to access frequently updating data as well as not frequently updating data. .

Ex: Latest released films of the day -frequently updating

Ex:User’s interested films  -not frequently updating data

3.No limitation to scale up(horizontal or vertical) the system

-response time ,support to large user base,load balancing (DB Replication is ignored for demonstration purpose)

4.Multiple platform supports ,WEB and Mobile

5.Same user  account can be used with different devices(Android ,IOS,Web)

-data sync

 

Aproach:

Mysql-setup - New Page (2)
1.Suggested AWS cloud architecture

Scale up the the database and make a distributed architecture

  • Increase the performance of the server-ABC
  • Indexing to most important fields
  • Server XYZ was added to store  region B related data,
  • Maintaining history collection to move old data .
  • A reference collection which has references for the data where it is stored (ex:ABC or XYZ)

Setup to build  the API environment;

  • Setup nginx on 80 port
  • Setup Node API  on 8080 port
  • Install Redis cache and mongoDB

 

Implemeting API with expressJS

Topics:

  1. Authentication with web token
  2. Signup and login
  3. Token usage for resources

please visit for the source code ,Source code.

NodeJS API (ExpressJS) for beginners with AWS setup

(

Node.js is a very powerful JavaScript-based framework/platform built on Google Chrome’s JavaScript V8 Engine. It is used to develop I/O intensive web applications like video streaming sites, single-page applications, and other web applications. Node.js is open source, completely free.

Prerequisites

Before proceeding with this tutorial, you should have a basic understanding of JavaScript.

What is Node.js?

Node.js is a server-side platform built on Google Chrome’s JavaScript Engine .

Features of Node.js

Following are some of the important features that make Node.js the first choice of software architects.

  • Asynchronous and Event Driven
  • Very Fast− Being built on Google Chrome’s V8 JavaScript Engine, Node.js library is very fast in code execution.
  • Single Threaded but Highly Scalable− Node.js uses a single threaded model with event looping.
  • No Buffering− Node.js applications never buffer any data. These applications simply output the data in chunks.
  • License− Node.js is released under the MIT license.

Node.js better results

Following are the areas where Node.js is proving itself as a perfect technology partner.

  • I/O bound Applications
  • Data Streaming Applications
  • Data Intensive Real-time Applications (DIRT)
  • JSON APIs based Applications
  • Single Page Applications

Environment setup:

Use the MSI file and follow the prompts to install the Node.js. By default, the installer uses the Node.js distribution in C:\Program Files\nodejs. The installer should set the C:\Program Files\nodejs\bin directory in window’s PATH environment variable. Restart any open command prompts for the change to take effect.

https://nodejs.org/en/

Linux :

$ cd /tmp

$ wget http://nodejs.org/dist/v6.3.1/node-v6.3.1-linux-x64.tar.gz

$ tar xvfz node-v6.3.1-linux-x64.tar.gz

$ mkdir -p /usr/local/nodejs

$ mv node-v6.3.1-linux-x64/* /usr/local/nodejs

Add /usr/local/nodejs/bin to the PATH environment variable.

OS Output
Linux export PATH=$PATH:/usr/local/nodejs/bin
Mac export PATH=$PATH:/usr/local/nodejs/bin
FreeBSD export PATH=$PATH:/usr/local/nodejs/bin

Node Package Manager (NPM) provides two main functionalities −

  • Online repositories for node.js packages/modules which are searchable onnodejs.org
  • Command line utility to install Node.js packages, do version management and dependency management of Node.js packages.
  • $ npm –version

$ sudo npm install npm -g

/usr/bin/npm -> /usr/lib/node_modules/npm/bin/npm-cli.js

npm@2.7.1 /usr/lib/node_modules/npm

Install Express:

  1. https://github.com/petecoop/generator-express , install express generator MVC
  2. express test
  3. open project in editor

Server deployment

npm install pm2 –g

pm2 start app.js

Install mongo admin tool

http://adminmongo.com/docs/

 

check the express.js to customize for APi

app.use(function (req, res, next) {

var err = new Error(‘Not Found’);

err.status = 404;

var chk =req.path;

if( chk.indexOf(‘api’) >= 0){

return res.json( {

message: err.message,

success:false

});
}
});

or

app.use(function (req, res, next) {

var err = new Error(‘Not Found’);

err.status = 404;

var chk =req.path;

if( chk.indexOf(‘api’) >= 0){

return res.json( {

message: err.message,

success:false

});
}else{
res.render(‘error’, {
message: err.message,
error: {},
title: ‘error’
});
}
});

home.jS

please add the following lib for JSON actions with prefixed router

var webrouter = express.Router();
var bodyParser = require(‘body-parser’);
var app = express();
var util = require(‘util’);
var multer = require(‘multer’)
//var http = require(‘http’).Server(app);
var jsonParser = bodyParser.json()
app.use(bodyParser.urlencoded({ extended: true }))
app.use(bodyParser.json());

module.exports = function (app) {
app.use(”,router);
app.use(‘/api/v1’, webrouter);
};

keep the following function for uncaught exception handling

process.on(‘uncaughtException’, function (err) {
console.log(‘uncaughtException: ‘ + err.message);
console.log(err.stack);

});

Simple API with prefix
webrouter.post(‘/add-article’, function (req, res, next) {
user=10;
return res.json({
success: true,
content: {message: ‘artilce added success’,
result: user}
});
});

Common area for authentication purposes.
webrouter.use(function (req, res, next) {

// check header or url parameters or post parameters for token
var token = req.body.token || req.query.token || req.headers[‘token’];
//get last parameter
var requested_url = req.path;

console.log(requested_url);
if (token) {

//some processing
console.log(“common area with token”)
next()

}
else if (req.path == “/signup”)
{
next();
}else{
console.log(“common area with non token”)
var err = new Error(‘Not Found’);
console.log(req.path);
return res.status(403).send({
success: false,
content: {message: ‘No token provided.’}
});

}

});

webrouter.post(‘/signup’, function (req, res, next) {
return res.json({success: false, content: {message:”signup”}});
});

Simple saving function to store student records

create a student model ,student.js
var mongoose = require(‘mongoose’),
Schema = mongoose.Schema;

var StudentSchema = new Schema({
title: String,
url: String,
text: String,
id:Number
});

mongoose.model(‘Student’, StudentSchema);

include this model into home.JS

var Student = mongoose.model(‘Student’)

Make a route

webrouter.post(‘/add-student’, function (req, res) {
title=req.body.title;
id=req.body.id;
Student=new Student();
Student.title=title;
Student.id=id;
Student.save(function (err, row) {
if(err){
return res.json({
success: false,
content: {message: ‘student is not added’,
result: err}
});
}else{
return res.json({
success: true,
content: {message: ‘student added success’,
result: row}
});
}

});

});

Simple get function by ID and updating 
webrouter.post(‘/update-student’, function (req, res) {

console.log(req.body.id);
if (!req.body.id)
return res.json({success: false, content: {message: “Please enter the id.”}});
if (!req.body.text)
return res.json({success: false, content: {message: “Please enter the text.”}});

Student.findOne({“id”: req.body.id}, function (err, student) {
console.log(student);
if (err) {
return res.json({success: false,
content: {message: ‘student is not found’},
result: err});
//check the firstname and last name are exist or not
}

else if (student)
{
student.text = req.body.text;
student.save(function (err, row) {
if (err) {
return res.json({success: false,
content: {message: ‘student is not found’},
result: err});
} else {
return res.json({
success: true,
content: {message: ‘student added success’,
result: row}
});
}
});
return res.json({success: true,
content: {message: ‘student found’},
result: student});
} else {
return res.json({success: false,
content: {message: ‘null’},
result: null});
}

});
});

Example for asyncrinus function

Import the lib (on top of the page),var async = require(‘async’);

webrouter.post(‘/async-example’, function (req, res) {
var async = require(‘async’);
async.parallel({
one: function(callback) {
callback(null, ‘abc\n’);//first parameter is error ,second is returning
},
two: function(callback) {
callback(null, ‘xyz\n’);
}
}, function(err, results) {
// results is now equals to: {one: ‘abc\n’, two: ‘xyz\n’}
console.log(err);
console.log(results.two);
return res.json({success: true,
content: {message: ‘async example ‘},
result: results});
});
});

Simple callback function
webrouter.post(‘/example-callback’, function (req, res) {

a=”test1″;
b=”exam”;
concatenation(a,b,function(results){

if(results){
return res.json({success: true,
content: {message: ‘example-callback’},
result: results});
}else{
return res.json({success: true,
content: {message: ‘example-callback’},
result: results});
}

})

})

function concatenation(a,b,callback){
if(a!=””){
var c=a+b;
console.log(“aftfer c”)
callback(c) ;
}else{
callback(null);
}
}
//async foreach to keep on tracking
var forEach = require(‘async-foreach’).forEach;

//get one stduent and one article set

Topics:

  1. Authentication with web token
  2. Signup and login
  3. Token usage for resources
  4. API with auth and without auth
  5. API access data after checking the updated version
  6. maintain history table to maintain big data
  7. cache for making fast API responses
  8. Installation and deployment
  1. Authentication with web token

There are some situation for open API access(wihtout AUTH ) as well as with AUTH.

with AUTH:

import the following libraries

//start
var bcrypt = require(‘bcrypt’);
var jwt = require(‘jsonwebtoken’);
var User = mongoose.model(‘User’);
var Client = mongoose.model(‘clients’);
var Accesstoken = mongoose.model(‘auth_accesstokens’);
//end

then

//common routing architecture
webrouter.use(function (req, res, next) {

// check header or url parameters or post parameters for token
var token = req.body.token || req.query.token || req.headers[‘token’];
//get last parameter
var requested_url = req.path;
var requested_url_array = requested_url.split(‘/’);
var lastsegment = requested_url_array[requested_url_array.length – 1];
console.log(lastsegment);
if (token) {

//some processing
jwt.verify(token, ‘ilovescotchyscotch’, function (err, decoded) {
if (err) {
return res.json({success: false, content: {message:”auth failed1″}});
} else {
// if everything is good, save to request for use in other routes
var apikey = req.body.apikey || req.query.apikey || req.headers[‘apikey’];
Accesstoken.count({$and: [{‘token’: token}, {‘user_id’: req.headers.user_id}, {‘clientId’: apikey}]}, function (err, clientdata) {
if (clientdata == 0) {
return res.json({success: false, content: {message: “auth failed2”}});
} else {
req.decoded = decoded;
next();
}
}
);
}
});

} else {
console.log(“common area with non token”)
if (req.path == “/authentication” || req.path == “/signup”
|| req.path == “/logout” || req.path ==”/add-apikey”) {

next();
} else {
var err = new Error(‘Not Found’);
console.log(req.path);
return res.status(403).send({
success: false,
content: {message: ‘No token provided.’}
});
return res.status(404).send({
success: false,
content: {message: ‘Page not found.’}
});
}

}
});

Register API auth related models

models:

var User = mongoose.model(‘User’);
var Client = mongoose.model(‘clients’);
var Accesstoken = mongoose.model(‘auth_accesstokens’);

Then add the api key:

webrouter.get(‘/add-apikey’, function (req, res) {

client = Client();
client.name = “webrouter”;
client.clientId = “heRueI9SA”;
client.clientSecret = “heRueI9SAsecretAA”;

client.save(function (err, row) {
if (err) {
return res.json({
success: false,
content: {message: ‘add-apikey is not added’,
result: err}
});
} else {
return res.json({
success: true,
content: {message: ‘add-apikey added success’,
result: row}
});
}

});

});

signup process:
webrouter.post(‘/signup’, function (req, res, next) {

if (!req.headers.fname)
return res.json({success: false, content: {message: “Please enter the name.”}});
if (!req.headers.lname)
return res.json({success: false, content: {message: “Please enter the last name.”}});
if (!req.headers.email)
return res.json({success: false, content: {message: “Please enter the email.”}});
if (!req.headers.password)
return res.json({success: false, content: {message: “Please enter the password.”}});
console.log(“muy test2”)
clientId = req.headers.apikey;
User.findOne({’email’: req.headers.email}, {type: 0}, function (err, user) {
console.log(“inside user”+user)
// if there are any errors, return the error
if (err)
return res.json({success: false, content: {message: err}});
// check to see if theres already a user with that email
if (user) {
console.log(“success: false”);
return res.json({success: false, content: {message: ‘The email is already registered.’}, newuser: 0});
}
else {

Client.findOne({clientId: clientId}, function (err, client) {
if (err)
return res.json({success: false, content: {message: “Wrong API key”}});
console.log(client);
if (!client) {
return res.json({success: false, content: {message: “Wrong API key”}});
}
var user = new User();
var pass;
createHash(req.headers.password,function(pas){
pass=pas;

console.log(“pass”+pass)
user.email = req.headers.email;
user.password = pass
user.firstName = req.headers.fname;
user.lastName = req.headers.lname;

user.save(function (err) {
if (err){
return res.json({success: false, content: {message: err}});

} else {
console.log(“{message: ‘Authentication success'”);
return res.json({
success: true,
content: {message: ‘Authentication success’,

result: user}
});
}

//return res.json({success: true, content: {message: “Signup sucessfully.”}});
});

});

});
} // We’re not logged in, so we’re creating a brand new user.

});
}
);

Note:when u return please avoid to return the encrypted password

auth related functions
function createHash(password,callback){
saltRounds = 10;
console.log(“password”+password);
//return bcrypt.hashSync(password, bcrypt.genSaltSync(8), null);
bcrypt.genSalt(saltRounds, function(err, salt) {
bcrypt.hash(password, salt, function(err, hash) {
// Store hash in your password DB.
console.log(“out”+hash);
callback(hash);
});

});
}

// checking if password is valid
function validPassword(password,hash,callback) {
// return bcrypt.compareSync(password, this.password);
console.log(password);
console.log(hash);
bcrypt.compare(password, hash, function(err, res) {
// res == true
console.log(res);
callback(res);
});
};

authentications
webrouter.post(‘/authentication’, function (req, res) {
var clientId = req.headers.apikey
if (!req.headers.device)
return res.json({success: false, content: {message: “Please send the device token.”}});
if (!req.headers.apikey)
return res.json({success: false, content: {message: “Please send the API key.”}});
if (!req.headers.email)
return res.json({success: false, content: {message: “Please send the user email.”}});
if (!req.headers.password)
return res.json({success: false, content: {message: “Please send the password.”}});

Client.findOne({clientId: clientId}, function (err, client) {
if (err)
return res.json({success: false, content: {message: “Wrong API key”}});
console.log(client);
if (!client) {
return res.json({success: false, content: {message: “Wrong API key”}});
}
User.findOne({’email’: req.headers.email}, function (err, user) {
//console.log(“test”);process.exit();
if (err)
throw err;

if (!user) {
console.log(“three”);
return res.json({success: false, content: {message: ‘Authentication failed. User not found.’}});
} else if (user) {
password = req.headers.password;
username = req.headers.email;
console.log(“password”+password)
console.log(“user.password”+user.password)
try {
validPassword(password, user.password,function(result){
if (!result) {
// if password does not match

return res.json({success: false, content: {message: “Wrong password”}});
} else {
// if everything is OK, return null as the error // and the authenticated user
con_cat = req.headers.device + ” + req.headers.email
var token = jwt.sign(con_cat, ‘ilovescotchyscotch’);
var accesstok = new Accesstoken({
‘clientId’: clientId, “token”: token, “device_token”: req.headers.device, ‘ip’: ‘198.168.1.20’, ‘os’: “ios”, ‘user_id’: user._id
});
Accesstoken.findOne({$and: [{‘device_token’: req.headers.device}, {‘user_id’: user._id}]}, function (err, checkaccesstoken) {
if (err)
throw err;

if (!checkaccesstoken)
{
accesstok.save(function (err, row) {
if (err)
throw err;
// return the information including token as JSON
return res.json({
success: true,
content: {message: ‘Authentication success’,
token: token,
result: user}
});
});
} else {

checkaccesstoken.clientId = clientId;
checkaccesstoken.token = token;
checkaccesstoken.device_token = req.headers.device_token;
checkaccesstoken.ip = ‘198.168.1.20’;
checkaccesstoken.os = “ios”;
checkaccesstoken.user_id = user._id;
checkaccesstoken.save(function (err, row) {
if (err)
throw err;
// return the information including token as JSON
return res.json({
success: true,
content: {message: ‘Authentication success’,
token: token,
result: user}
});
});
}

});

}
});
} catch (e) {
return res.json({success: false, content: {message: “Wrong password”+e}});
}

}

});
});
});

Redis-cache

before access the API check the version number to make sure the data update

//initializing radis connection
var options = {
ttl: -1,
engine: ‘redis’,
port: 6379,
host: ‘localhost’
};

//create an instacne
var Cacheman = require(‘cacheman’);
var cache2 = new Cacheman(‘todo’, options);

webrouter.post(‘/add-top-news’, function (req, res, next) {
user = 10;
console.log(req.body);
var data={
success: true,
content: {message: ‘add-top-news success’,
result: user}
}
cache2.set(‘add-top-news’, data)
.then(function (value) {

return res.json({
success: true,
content: {message: ‘artilce added success’,
result: user}
});

});

});
webrouter.get(‘/get-top-news’, function (req, res, next) {
user = 10;
console.log(req.body);

cache2.get(‘add-top-news’, function (err, value) {
if (err)
throw err;
console.log(value);

return res.json({
success: true,
content: {message: ‘artilce added success’,
result: value}
});

});

});

Redis command:redis 127.0.0.1:6379> keys *

Deployment

Development simple:

Enhanced

For source code :https://github.com/azeemj/nodeJS-API-auth

Node.js and Nginx on an Amazon Linux EC2 Instance

(from:http://thomasmullaly.com/2014/11/02/node-dot-js-and-nginx-on-an-amazon-linux-ec2-instance/)

NOV 2ND, 2014

I’m looking to automate a node.js application on amazon web services. I’m using the amazon linux ami, which is based off of centos 5. Notice that amazon has switched from using para-virtualized (pv) guests to hardware virtualized machine guests (hvm).

Launch the instance and ssh in, and become root (sudo -i). Now install nginx

[ec2-user@ip-172-30-0-164 ~]$ sudo !!
sudo yum install nginx
Loaded plugins: priorities, update-motd, upgrade-helper
amzn-main/latest                                                                     | 2.1 kB     00:00     
amzn-updates/latest                                                                  | 2.3 kB     00:00     
Resolving Dependencies
--> Running transaction check
---> Package nginx.x86_64 1:1.6.2-1.22.amzn1 will be installed
--> Processing Dependency: gd for package: 1:nginx-1.6.2-1.22.amzn1.x86_64
--> Processing Dependency: GeoIP for package: 1:nginx-1.6.2-1.22.amzn1.x86_64
--> Processing Dependency: libprofiler.so.0()(64bit) for package: 1:nginx-1.6.2-1.22.amzn1.x86_64
--> Processing Dependency: libGeoIP.so.1()(64bit) for package: 1:nginx-1.6.2-1.22.amzn1.x86_64
--> Processing Dependency: libgd.so.2()(64bit) for package: 1:nginx-1.6.2-1.22.amzn1.x86_64
--> Running transaction check
---> Package GeoIP.x86_64 0:1.4.8-1.5.amzn1 will be installed
---> Package gd.x86_64 0:2.0.35-11.10.amzn1 will be installed
--> Processing Dependency: libXpm.so.4()(64bit) for package: gd-2.0.35-11.10.amzn1.x86_64
---> Package gperftools-libs.x86_64 0:2.0-11.5.amzn1 will be installed
--> Processing Dependency: libunwind.so.8()(64bit) for package: gperftools-libs-2.0-11.5.amzn1.x86_64
--> Running transaction check
---> Package libXpm.x86_64 0:3.5.10-2.9.amzn1 will be installed
---> Package libunwind.x86_64 0:1.1-2.1.amzn1 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

============================================================================================================
 Package                    Arch              Version                         Repository               Size
============================================================================================================
Installing:
 nginx                      x86_64            1:1.6.2-1.22.amzn1              amzn-updates            523 k
Installing for dependencies:
 GeoIP                      x86_64            1.4.8-1.5.amzn1                 amzn-main               783 k
 gd                         x86_64            2.0.35-11.10.amzn1              amzn-main               155 k
 gperftools-libs            x86_64            2.0-11.5.amzn1                  amzn-main               570 k
 libXpm                     x86_64            3.5.10-2.9.amzn1                amzn-main                54 k
 libunwind                  x86_64            1.1-2.1.amzn1                   amzn-main                69 k

Transaction Summary
============================================================================================================
Install  1 Package (+5 Dependent packages)

Total download size: 2.1 M
Installed size: 4.8 M
Is this ok [y/d/N]: y
Downloading packages:
(1/6): GeoIP-1.4.8-1.5.amzn1.x86_64.rpm                                              | 783 kB     00:00     
(2/6): gd-2.0.35-11.10.amzn1.x86_64.rpm                                              | 155 kB     00:00     
(3/6): gperftools-libs-2.0-11.5.amzn1.x86_64.rpm                                     | 570 kB     00:00     
(4/6): libXpm-3.5.10-2.9.amzn1.x86_64.rpm                                            |  54 kB     00:00     
(5/6): libunwind-1.1-2.1.amzn1.x86_64.rpm                                            |  69 kB     00:00     
(6/6): nginx-1.6.2-1.22.amzn1.x86_64.rpm                                             | 523 kB     00:00     
------------------------------------------------------------------------------------------------------------
Total                                                                        15 MB/s | 2.1 MB  00:00:00     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : libXpm-3.5.10-2.9.amzn1.x86_64                                                           1/6 
  Installing : gd-2.0.35-11.10.amzn1.x86_64                                                             2/6 
  Installing : libunwind-1.1-2.1.amzn1.x86_64                                                           3/6 
  Installing : gperftools-libs-2.0-11.5.amzn1.x86_64                                                    4/6 
  Installing : GeoIP-1.4.8-1.5.amzn1.x86_64                                                             5/6 
  Installing : 1:nginx-1.6.2-1.22.amzn1.x86_64                                                          6/6 
  Verifying  : gperftools-libs-2.0-11.5.amzn1.x86_64                                                    1/6 
  Verifying  : GeoIP-1.4.8-1.5.amzn1.x86_64                                                             2/6 
  Verifying  : 1:nginx-1.6.2-1.22.amzn1.x86_64                                                          3/6 
  Verifying  : gd-2.0.35-11.10.amzn1.x86_64                                                             4/6 
  Verifying  : libunwind-1.1-2.1.amzn1.x86_64                                                           5/6 
  Verifying  : libXpm-3.5.10-2.9.amzn1.x86_64                                                           6/6 

Installed:
  nginx.x86_64 1:1.6.2-1.22.amzn1                                                                           

Dependency Installed:
  GeoIP.x86_64 0:1.4.8-1.5.amzn1   gd.x86_64 0:2.0.35-11.10.amzn1   gperftools-libs.x86_64 0:2.0-11.5.amzn1
  libXpm.x86_64 0:3.5.10-2.9.amzn1 libunwind.x86_64 0:1.1-2.1.amzn1

Complete!

Make sure nginx starts up on a reboot:

sudo chkconfig --level 345 nginx on

Now Let’s install node.js

[root@ip-172-30-0-164 ~]# curl -sL https://rpm.nodesource.com/setup | bash -

## Inspecting system...

+ rpm -q --whatprovides redhat-release
+ uname -m

## Confirming "el7-x86_64" is supported...

+ curl -sLf -o /dev/null 'https://rpm.nodesource.com/pub/el/7/x86_64/nodesource-release-el7-1.noarch.rpm'

## Downloading release setup RPM...

+ mktemp
+ curl -sL -o '/tmp/tmp.tGqnuw1nK9' 'https://rpm.nodesource.com/pub/el/7/x86_64/nodesource-release-el7-1.noarch.rpm'

## Installing release setup RPM...

+ rpm -i --nosignature --force '/tmp/tmp.tGqnuw1nK9'

## Cleaning up...

+ rm -f '/tmp/tmp.tGqnuw1nK9'

## Checking for existing installations...

+ rpm -qa 'node|npm' | grep -v nodesource

## Run `yum install -y nodejs` (as root) to install Node.js and npm.
## You may also need development tools to build native addons:
##   `yum install -y gcc-c++ make`

[root@ip-172-30-0-164 ~]# yum install -y nodejs
Loaded plugins: priorities, update-motd, upgrade-helper
nodesource/x86_64                                                                    | 2.4 kB     00:00     
nodesource/x86_64/primary_db                                                         | 7.9 kB     00:00     
Resolving Dependencies
--> Running transaction check
---> Package nodejs.x86_64 0:0.10.33-1nodesource.el7.centos will be installed
--> Finished Dependency Resolution

Dependencies Resolved

============================================================================================================
 Package           Arch              Version                                    Repository             Size
============================================================================================================
Installing:
 nodejs            x86_64            0.10.33-1nodesource.el7.centos             nodesource            4.5 M

Transaction Summary
============================================================================================================
Install  1 Package

Total download size: 4.5 M
Installed size: 15 M
Downloading packages:
warning: /var/cache/yum/x86_64/latest/nodesource/packages/nodejs-0.10.33-1nodesource.el7.centos.x86_64.rpm: Header V3 RSA/SHA1 Signature, key ID 34fa74dd: NOKEY
Public key for nodejs-0.10.33-1nodesource.el7.centos.x86_64.rpm is not installed
nodejs-0.10.33-1nodesource.el7.centos.x86_64.rpm                                     | 4.5 MB     00:00     
Retrieving key from file:///etc/pki/rpm-gpg/NODESOURCE-GPG-SIGNING-KEY-EL
Importing GPG key 0x34FA74DD:
 Userid     : "NodeSource <gpg-rpm@nodesource.com>"
 Fingerprint: 2e55 207a 95d9 944b 0cc9 3261 5ddb e8d4 34fa 74dd
 Package    : nodesource-release-el7-1.noarch (installed)
 From       : /etc/pki/rpm-gpg/NODESOURCE-GPG-SIGNING-KEY-EL
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Warning: RPMDB altered outside of yum.
  Installing : nodejs-0.10.33-1nodesource.el7.centos.x86_64                                             1/1 
  Verifying  : nodejs-0.10.33-1nodesource.el7.centos.x86_64                                             1/1 

Installed:
  nodejs.x86_64 0:0.10.33-1nodesource.el7.centos                                                            

Complete!

Now install pm2 which is the daemon which runs the node server.

[root@ip-172-30-0-164 ~]# npm install pm2 -g --unsafe-perm
|
> pm2@0.11.1 preinstall /usr/lib/node_modules/pm2
> bash ./scripts/preinstall.sh

npm_config_argv={"remain":["pm2"],"cooked":["install","pm2","--global","--unsafe-perm"],"original":["install","pm2","-g","--unsafe-perm"]}
npm WARN optional dep failed, continuing fsevents@0.3.0
/usr/bin/pm2 -> /usr/lib/node_modules/pm2/bin/pm2

> pm2@0.11.1 postinstall /usr/lib/node_modules/pm2
> bash ./scripts/postinstall.sh

pm2@0.11.1 /usr/lib/node_modules/pm2
├── ikt@0.0.0
├── uid-number@0.0.5
├── json-stringify-safe@5.0.0
├── commander@2.3.0
├── eventemitter2@0.4.14
├── pm2-interface@2.0.4
├── colors@0.6.2
├── cron@1.0.4
├── cli-table@0.3.0
├── pidusage@0.1.0
├── async@0.9.0
├── axm@0.2.13
├── vizionar@0.1.3
├── chokidar@0.9.0 (recursive-readdir@0.0.2)
├── debug@2.0.0 (ms@0.6.2)
├── pm2-multimeter@0.1.2 (charm@0.1.2)
├── pm2-deploy@0.1.2 (tv4@1.0.18)
├── nssocket@0.5.1 (lazy@1.0.11)
├── chalk@0.5.1 (escape-string-regexp@1.0.2, ansi-styles@1.1.0, supports-color@0.2.0, strip-ansi@0.3.0, has-ansi@0.1.0)
├── punt@2.2.0 (amp-message@0.1.2)
├── pm2-axon-rpc@0.3.6 (commander@1.0.5)
├── pm2-axon@2.0.7 (amp-message@0.1.2, escape-regexp@0.0.1, configurable@0.0.1, amp@0.3.1)
├── pm2-rpc-fallback@3.0.9 (commander@1.0.5, axon@1.0.0)
├── coffee-script@1.8.0 (mkdirp@0.3.5)
├── pm2-logs@0.1.1 (chalk@0.4.0, pm2-interface@1.1.0, blessed@0.0.36)
└── moment@2.8.3

create a simple node.js server, create a file called server.js

var http = require('http');

var server = http.createServer(function (request, response) {
  response.writeHead(200, {"Content-Type": "text/plain"});
  response.end("Hello World\n");
});
server.listen(8080);
console.log("Server running at http://127.0.0.1:8080/");

You can try out the server by starting it like this:

[ec2-user@ip-172-30-0-164 ~]$ node server.js
Server running at http://127.0.0.1:8080/

Or you can daemonize it with pm2, like this:

[ec2-user@ip-172-30-0-164 ~]$ pm2 start server.js 
Initializing PM2 configuration (/home/ec2-user/.pm2)
Starting PM2 daemon...
[PM2] Process server.js launched
┌──────────┬────┬──────┬──────┬────────┬───────────┬────────┬────────────┬─────────────┐
│ App name │ id │ mode │ PID  │ status │ restarted │ uptime │     memory │    watching │
├──────────┼────┼──────┼──────┼────────┼───────────┼────────┼────────────┼─────────────┤
│ server   │ 0  │ fork │ 1257 │ online │         0 │ 0s     │ 9.922 MB   │ unactivated │
└──────────┴────┴──────┴──────┴────────┴───────────┴────────┴────────────┴─────────────┘
 Use `pm2 desc[ribe] ` to get more details

Now let’s add the proxy info to nginx, edit /etc/nginx/nginx.conf

server {
        listen       80;
        #server_name  localhost;
        server_name  your-ip-address;
        root         /usr/share/nginx/html;

        #charset koi8-r;

        #access_log  /var/log/nginx/host.access.log  main;

        location / {
		proxy_pass http://localhost:8080;
		proxy_http_version 1.1;
		proxy_set_header Upgrade $http_upgrade;
		proxy_set_header Connection 'upgrade';
		proxy_cache_bypass $http_upgrade;
        }

restart nginx

sudo service nginx restart

hit the ip address with a browser and you should see “hello world”

AWS Polly with Node JS

Hi,

I got a chance to work with AWS Polly and i am willing to share my interesting experiences.

Amazon Polly (Text-to-Speech) implementation with Node JS

What is Amazon Polly?

Amazon Polly is a mechanism that gives text-to-speech cloud service. Amazon Polly supports multiple languages and includes a variety of voices.

Benefits:

  • High quality
  • Low latency
  • Support for different voices
  • Cost effective
  • Cloud based solution

 

Architecture of AWS Polly implementation.

polly2

 

Coding level implementation

 

Node js Function

//initilaization

var fs = require(‘fs’);
var AWS = require(‘aws-sdk’);
require(‘ auth-config-polly.js’);

function PollyImplemetation(req, callback) {
var msg = req.text;
var id = req.id

var polly = new AWS.Polly();
var descParams = {
LanguageCode: ‘en-US’
};

polly.describeVoices(descParams, function (err, data) {
if (err) {
console.log(err);
callback(null)
} else {
var thevoiceId = data.Voices[0].Id;
// synthesizeSpeech
var textMsg = msg;
var speechParams = {
“OutputFormat”: ‘mp3’,
“VoiceId”: thevoiceId,
“Text”: textMsg,
“SampleRate”: ‘22050’,
“TextType”: ‘text’
};
polly.synthesizeSpeech(speechParams, function (err, data) {
if (err) {
console.log(err);
callback(null)
} else {
fs.writeFile(‘./public/ ‘ + id + ‘.mp3’, data.AudioStream, function (err) {
console.log(‘file writing in local ‘);
if (err) {

console.log(err);
callback(null)
} else {
//uploading to S3 bucket process
var fileStream = fs.createReadStream(‘./public/’ + id + “” + ‘.mp3’);
fileStream.on(‘error’, function (err) {
if (err) {
console.log(err);
callback(null)
}
});
fileStream.on(‘open’, function () {
var s3 = new AWS.S3();
s3.putObject({
Bucket: ‘mp3bucketname’,
Key: ‘mp3/’ + id + ‘.mp3’,
ACL: ‘public-read’,
Body: fileStream
}, function (err) {
if (err) {
console.log(err);
callback(null)
} else {
callback(id + ‘.mp3’)
}
});
});

}
});
}
});
}

});
}

 

 

auth-config-polly.js

AWS.config.loadFromPath(‘config_polly.json’);

 

config_polly.json

{
“accessKeyId”: “XXXXXXXXXXX”,
“accessKey”: “XXXXXXXCCYYYYYYYYYY”,
“region”: “XXXXXXXXXXX”,
“version”: “latest”
}

 

How to call 

var reg = [{
“id”: “123”,
“value”: “Test first text to speech”,
}, {
“id”: “1234”,
“value”: “Second text to speech”,
}];

PollyImplemetation(req, function() {

 

});

Uploading images over REST API -Nodejs,expressJS by using multipart

//create a route with prefix

var app = express();

app.use(bodyParser.urlencoded({extended: false}));
app.use(bodyParser.json());
var apiroutes = express.Router();

var multipartMiddleware = multipart();

/**

This is the function we need to call over REST API

**/
apiroutes.post(‘/update-image’, multipartMiddleware, function (req, res) {

if (!req.body.files)
{
profileImage(req, function (result) {

return res.json({message: “updated successfully.”});

});

 

});

/**
* uploading images
* @param {type} req
* @param {type} callback
* @returns {undefined}
*/
function profileImage(req, callback) {
x = req.files.files;
if(x){
fs.readFile(x[‘path’], function (err, data) {
if (err)
return callback(0)
newPath = ‘./public/api-testimages/’ + req.headers.user_id + “”” + x[‘originalFilename’];
fs.writeFile(newPath, data, function (err) {
if (err)
callback(0);
return callback(newPath);
});
//dont forgot the delete the temp files.
});
}else{
callback(0);
}
}

 

 

in ur post man

image-uploading