Saturday, June 21, 2014

Memcached installation on windows and unix based system

Memcached is free and open source, high performance distributed memory object caching system, generic in nature, but intended  for use in speeding up dynamic web applications by alleviating database load.
Memcached is an in-memory key value store for small chunks of arbitrary data storage (strings, objects) from results of database calls, API calls, or for page rendering. Memcached is simple yet powerful. Its simple design promotes quick deployment, ease of deployment and solves many problems facing large data caches. Its API is available for most popular languages.
Memcache installation on windows  
One can download  memcache zip  from here for 64 bit version and  here for 32 bit version .
You can unzip the folder and paste at suitable place in your computer. For installation , you can run following command :
memcached.exe –d install
memcached.exe –d start

When you open your task manager, you can watch this memcached service running in your computer.


Congratulations! J  you have installed memcached in your system.

Memcached installation on Unix based system.

    1. Lets walk through some steps and commands on unix based system
              Download link : http://www.memcached.org/files/memcached-1.4.20.tar.gz

1.                           To start memcache on unix :
shell> memcached
 or sh memcached.sh

2.         By default, memcached uses the following settings:
·         Memory allocation of 64MB
·         Listens for connections on all network interfaces, using port 11211
·         Supports a maximum of 1024 simultaneous connections.

Typically, you would specify the full combination of options that you want when starting memcached, and normally provide a startup script to handle the initialization of memcached.For example, the following line starts memcached with a maximum of 1024MB RAM for the cache,listening on port 11211 on the IP address 192.168.0.110, running as a background daemon:
shell> memcached -d -m 1024 -p 11211 -l 192.168.0.110

       3.       -l interface
Specify a network interface/address to listen for connections. The default is to listen on all available address (INADDR_ANY).
shell> memcached -l 192.168.0.110
Support for IPv6 address support was added in memcached 1.2.5.
-p port
Specify the TCP port to use for connections. Default is 18080.
shell> memcached -p 18080



4. If you start memcached as root, use the -u option to specify the user for executing memcached:

shell> memcached -u memcache

You can use the output of the vmstat command to get the free memory, as shown in free column:

shell> vmstat


5. -c connections

Specify the maximum number of simultaneous connections to the memcached service. The default is 1024.

shell> memcached -c 2048

6. -t threads

Specify the number of threads to use when processing incoming requests.

By default, memcached is configured to use 4 concurrent threads. The threading improves the performance of storing and retrieving data in the cache, using a locking system to prevent different threads overwriting or updating the same values. To increase or decrease the number of threads, use the -t option:

shell> memcached -t 8

7-d

Run memcached as a daemon (background) process:

shell> memcached -d
8 -r

Maximize the size of the core file limit. In the event of a failure, this attempts to dump the entire memory space to disk as a core file, up to any limits imposed by setrlimit.

9.-M

Return an error to the client when the memory has been exhausted. This replaces the normal behavior of removing older items from the cache to make way for new items.

10.-k

Lock down all paged memory. This reserves the memory before use, instead of allocating new slabs of memory as new items are stored in the cache.

Note
There is a user-level limit on how much memory you can lock. Trying to allocate more than the available memory fails. You can set the limit for the user you started the daemon with (not for the -u user user) within the shell by using ulimit -S -l NUM_KB

11.-v

Verbose mode. Prints errors and warnings while executing the main event loop.

12.-vv

Very verbose mode. In addition to information printed by -v, also prints each client command and the response.
13.
-vvv

Extremely verbose mode. In addition to information printed by -vv, also show the internal state transitions.

14.-h

Print the help message and exit.

15.-i

Print the memcached and libevent license.

16.-I mem

Specify the maximum size permitted for storing an object within the memcached instance. The size supports a unit postfix (k for kilobytes, m for megabytes). For example, to increase the maximum supported object size to 32MB:

shell> memcached -I 32m
The maximum object size you can specify is 128MB, the default remains at 1MB.

This option was added in 1.4.2.

17.-b

Set the backlog queue limit. The backlog queue configures how many network connections can be waiting to be processed by memcached. Increasing this limit may reduce errors received by the client that it is not able to connect to the memcached instance, but does not improve the performance of the server. The default is 1024.

18.-P pidfile

Save the process ID of the memcached instance into file.

19.-f

Set the chunk size growth factor. When allocating new memory chunks, the allocated size of new chunks is determined by multiplying the default slab size by this factor.

To see the effects of this option without extensive testing, use the -vv command-line option to show the calculated slab sizes. For more information, see Section 15.6.2.8, “memcached Logs”.

20.-n bytes

The minimum space allocated for the key+value+flags information. The default is 48 bytes.

21.-L

On systems that support large memory pages, enables large memory page use. Using large memory pages enables memcached to allocate the item cache in one large chunk, which can improve the performance by reducing the number misses when accessing memory.

22.-C

Disable the use of compare and swap (CAS) operations.

This option was added in memcached 1.3.x.

23.-D char

Set the default character to be used as a delimiter between the key prefixes and IDs. This is used for the per-prefix statistics reporting (see Section 15.6.4, “Getting memcached Statistics”). The default is the colon (:). If this option is used, statistics collection is turned on automatically. If not used, you can enable stats collection by sending the stats detail on command to the server.

This option was added in memcached 1.3.x.

24.-R num

Sets the maximum number of requests per event process. The default is 20.

25.-B protocol

Set the binding protocol, that is, the default memcached protocol support for client connections. Options are ascii, binary or auto. Automatic (auto) is the default.

This option was added in memcached 1.4.0.

Lets finish configuration related talks here and move to some hard core java programming  for availaing caching.

Thursday, October 10, 2013

Running Hbase with hadoop in standalone mode

Running Hbase with Hadoop  in standalone mode:

I guess the reader of this blog has already set hadoop in on his system and it is in running mode.

For a simple installation you need to follow these quick step.  


 1. If you are using latest version of hadoop use the download latest stable release of hbase .

Here I am using hadoop version 1.1.2 already in running state. I have downloaded latest stable release of hbase so as to work with hadoop.


 2. Configuring the java home path in hbase-env.sh

export JAVA_HOME=/usr/lib/jvm/java-6-openjdk


 3. Configure the region server in hbase-env.sh  

export HBASE_REGIONSERVERS=/opt/hbase/conf/regionservers


 4. Tell HBase whether it should manage it's own instance of Zookeeper or not.

   

export HBASE_MANAGES_ZK=true


 5.  configuration in hbase-site.xml


  

        hbase.rootdir

            hdfs://localhost2:9000/hbase

    

    

     hbase.master

            localhost2:60000

    



     

        dfs.replication

        1   

          hbase.zookeeper.property.clientPort

          2182

          Property from ZooKeeper's config zoo.cfg.

          The port at which the clients will connect.

          

        

    

          hbase.zookeeper.quorum

          localhost2

          Comma separated list of servers in the ZooKeeper Quorum.

          For example, "host1.mydomain.com,host2.mydomain.com,host3.mydomain.com".

          By default this is set to localhost for local and pseudo-distributed modes

          of operation. For a fully-distributed setup, this should be set to a full

          list of ZooKeeper quorum servers. If HBASE_MANAGES_ZK is set in hbase-env.sh

          this is the list of servers which we will start/stop ZooKeeper on.

          

    

    

      hbase.zookeeper.property.dataDir

      /opt/zookeeper

      Property from ZooKeeper's config zoo.cfg.

      The directory where the snapshot is stored.

      

    

     

     hbase.tmp.dir

      /opt/hbase/temp

   

   




 5. change the required setting in host file using command $ sudo gedit /etc/hosts 


 127.0.0.1 localhost

[your-ip]  localhost2


 6 change the required setting in host file using command  $sudo gedit /etc/hostname


 localhost


 7.  Now copy following jars using below command:


 cp ${HADOOP_HOME}/hadoop-core-*.jar   ${HBASE_HOME}/lib/

cp ${HADOOP_HOME}/lib/commons-configuration-*.jar   ${HBASE_HOME}/lib/


 8. now start your hbase using command

$./bin/start-hbase.sh

8. start habse shell using command

 $./bin/hbase shell



Note :


if your master server do not start due to safe mode of dfs problem , try this on hadoop command line:


 hadoop dfsadmin -safemode leave



 For checking if your hbase is running properly, u need to access this URL 

http://[your-ip-add]:60010 or localhost:60010


 you will get below page:



 


Congratulations !!, your set up is ready.


 If you are still facing any  problem , you can leave your comment on this blog.