海运的博客

Kyoto Cabinet和LevelDB实现任务队列

发布时间:January 7, 2014 // 分类:消息队列 // No Comments

根据Tokyo Tyrant自带的Lua队列脚本修改,Kyoto Cabinet和Tokyo Tyrant的API有很大不同。

kt = __kyototycoon__
db = kt.db

-- 记录日志
if kt.thid == 0 then
   kt.log("system", "the Lua script has been loaded")
end

-- 入队列
function enqueue(inmap, outmap)
   local key = inmap.key
   local value = inmap.value
   --队列值自增1,空从0开始
   local id = db:increment_double(key, 1)
   if not id then
      return kt.RVEINTERNAL 
   end 
   key = string.format("%s-%012d", key, id)
   if not db:add(key, value) then
      return kt.RVEINTERNAL
   end
   outmap[key] = "ok"
   return kt.RVSUCCESS
end

-- 出队列
function dequeue(inmap, outmap)
   local key = inmap.key
   local max = inmap.max
   max = tonumber(max)
   if not max or max < 1 then
      max = 1
   end
   key = string.format("%s-", key)
   --匹配队列前缀,返回多个匹配的key
   local keys = db:match_prefix(key, max)
   local res = ""
   for i = 1, #keys do
      local key = keys[i]
      local value = db:get(key)
      if db:remove(key) and value then
         --要返回的结果
         outmap[keys[i]] = value
      end
   end
   return kt.RVSUCCESS
end

-- 查看队列大小
function queuesize(inmap, outmap)
   local key = inmap.key
   key = string.format("%s-", key)
   local keys = db:match_prefix(key)
   outmap.size = #keys
   return kt.RVSUCCESS
end

-- 重置队列ID从0开始 
function queuereset(inmap, outmap)
   local key = inmap.key
   if not key then
      return kt.RVEINVALID
   end
   if not db:remove(key) then
      local err = db:error()
      if err:code() == kt.Error.NOREC then
         return kt.RVELOGIC
      end
      return kt.RVEINTERNAL
   end
   return kt.RVSUCCESS
end

使用:

ktremotemgr script -host 192.168.1.3 -port 1978 enqueue key queue value value1
ktremotemgr script -host 192.168.1.3 -port 1978 dequeue key queue max 10
curl "http://192.168.1.3:1978/rpc/play_script?name=enqueue&_key=queue&_value=value1"
curl "http://192.168.1.3:1978/rpc/play_script?name=dequeue&_key=queue&_max=10"

Tokyo Tyrant和Tokyo Cabinet实现任务队列

发布时间:January 7, 2014 // 分类:消息队列 // No Comments

Tokyo Tyrant官方提供的Lua脚本扩展:

-- 入队列
function enqueue(key, value)
   --通过队列名为key记录入队列的尾部,自加1为当次入队列的起始ID
   local id = _adddouble(key, 1)
   if not id then
      return nil
   end
   --队列key格式为队列名称\t加12位数字
   key = string.format("%s\t%012d", key, id)
   if not _putkeep(key, value) then
      return nil
   end
   return "ok"
end

-- 出队列
function dequeue(key, max)
   max = tonumber(max)
   if not max or max < 1 then
      max = 1
   end
   key = string.format("%s\t", key)
   --匹配队列前缀,返回匹配的多个key
   local keys = _fwmkeys(key, max)
   local res = ""
   for i = 1, #keys do
      local key = keys[i]
      local value = _get(key)
      --删除要出队列的key并返回队列值
      if _out(key) and value then
         res = res .. value .. "\n"
      end
   end
   return res
end

-- 查看队列大小
function queuesize(key)
   key = string.format("%s\t", key)
   local keys = _fwmkeys(key)
   //队列数组大小
   return #keys
end
tcrmgr ext -port 1978 127.0.0.1 enqueue myqueue value1
tcrmgr ext -port 1978 127.0.0.1 dequeue myqueue 10

通过TT入队列:

<?php
   function dequeue(){
      $tt = new TokyoTyrant("localhost", '1978');   
      $content = $tt -> ext('dequeue', '0', 'queue', 10);
      $content = preg_split('/\n/', $content, -1, PREG_SPLIT_NO_EMPTY);
      return $content;
   }
   //队列名称
   $file = 'file.csv';
   $queuename = 'queue';
   $tt = new TokyoTyrant("localhost", '1978');   
   $content = file_get_contents($file);
   $content = preg_split('/\n/', $content, -1, PREG_SPLIT_NO_EMPTY); //分割字符串为数组
   //队列开始ID
   //print_r($content);
   foreach($content as $key => $value)  
   {
      if ($key !== '')  
      {
         $tmpvalue = explode(',',$value);
         preg_match_all("/[0-9]{1}/", $tmpvalue[0], $arrNum, PREG_SET_ORDER);
         if (count($arrNum) > 8)
         {
            $tt->ext('enqueue','0', $queuename, $value );
            //echo $value."\n";
         }
      }
   }
?>

大数据通过TC PHP扩展入队列:

<?php
   //队列名称
   $file = 'file.txt';
   $queuename = 'queue';
   $db = new TCHDB();
   $db->open('/var/queue/db.tch', TCHDB::OWRITER | TCHDB::OCREAT);
   $content = file_get_contents($file);
   $content = preg_split('/\n/', $content, -1, PREG_SPLIT_NO_EMPTY); //分割字符串为数组
   //队列开始ID
   $id = 1;
   //print_r($content);
   foreach($content as $key => $value)  
   {
      if ($key !== '')  
      {
         $id = sprintf('%012d', $id);
         //echo $key.$value."\n";
         $db->put("$queuename-$id", $value);
         $id++;
      }
   }
   $db->out($queuename);
?>

NoSQL持久化存储之Kyoto Cabinet

发布时间:January 6, 2014 // 分类:NoSQL // No Comments

安装Kyoto Cabinet:

yum install gcc gcc-c++ zlib-devel bzip2-devel
wget http://fallabs.com/kyotocabinet/pkg/kyotocabinet-1.2.76.tar.gz
tar zxvf kyotocabinet-1.2.76.tar.gz 
cd kyotocabinet-1.2.76
./configure --prefix=/usr/local/kyotocabinet-1.2.76
make && make install
echo "/usr/local/kyotocabinet-1.2.76/lib/" >> /etc/ld.so.conf.d/kyoto.conf 
ldconfig

安装Lua Kyoto Cabinet扩展:

yum install lua-devel
wget http://fallabs.com/kyotocabinet/luapkg/kyotocabinet-lua-1.28.tar.gz
tar zxvf kyotocabinet-lua-1.28.tar.gz 
cd kyotocabinet-lua-1.28
./configure --with-kc=/usr/local/kyotocabinet-1.2.76/
make && make install

安装Kyoto Tycoon:

wget http://fallabs.com/kyototycoon/pkg/kyototycoon-0.9.56.tar.gz
tar zxvf kyototycoon-0.9.56.tar.gz 
cd kyototycoon-0.9.56
./configure --prefix=/usr/local/kyototycoon-0.9.56 --with-kc=/usr/local/kyotocabinet-1.2.76/ --enable-lua
make && make install
mkdir /etc/ktserver/
cp example/*.lua /etc/ktserver/
echo "/usr/local/kyototycoon-0.9.56/lib/" >> /etc/ld.so.conf.d/kyoto.conf 
ldconfig

使用Kyoto Tycoon为前端管理LevelDB,先安装LevelD存储:然后安装Kyoto Tycoon插件支持LevelDB:

cd lab/leveldb/
export CPLUS_INCLUDE_PATH=/usr/local/kyotocabinet-1.2.76/include/
export LIBRARY_PATH=/usr/local/kyotocabinet-1.2.76/lib
make
cp ktplugdblevel.so /usr/local/kyototycoon-0.9.56/lib/

启动:

ktserver -th 4 -scr /etc/ktserver/ktscrex.lua -pldb /usr/local/kyototycoon-0.9.56/lib/ktplugdblevel.so \
/tmp/leveldb/#ktopts=p

使用HTTP协议管理:

curl "http://192.168.1.3:1978/rpc/set?key=japan&value=tokyo" 
curl "http://192.168.1.3:1978/rpc/get?key=japan" 
curl "http://192.168.1.3:1978/rpc/remove?key=japan" 

使用客户端管理:

ktremotemgr set -host 192.168.1.3 -port 1978 key value
ktremotemgr get -host 192.168.1.3 -port 1978 key
ktremotemgr list -host 192.168.1.3 -port 1978 -pv

使用Kyoto Tycoon管理LevelDB兼容于直接使用LevelDB PHP扩展设置的数据库,启动时添加参数:

#ktopts=p

使用Kyoto Tycoon Lua脚本,API:http://fallabs.com/kyototycoon/luadoc/modules/kyototycoon.html

ktremotemgr script -host 192.168.1.3 -port 1978 set key key1 value value1
ktremotemgr script -host 192.168.1.3 -port 1978 get key key1
curl "http://192.168.1.3:1978/rpc/play_script?name=set&_key=key1&_value=value1"
curl "http://192.168.1.3:1978/rpc/play_script?name=get&_key=key1"

Linux编译/运行软件几个重要的变量参数

发布时间:January 5, 2014 // 分类:Linux基础 // No Comments

C头文件调用路径:

export C_INCLUDE_PATH=/usr/local/include:$C_INCLUDE_PATH
#也可在gcc参数中指定调用路径
-I/usr/local/include  

C++头文件调用路径:

export CPLUS_INCLUDE_PATH=/usr/local/include:$CPLUS_INCLUDE_PATH

动态库调用路径:

export LIBRARY_PATH=/usr/local/lib:$LIBRARY_PATH
#也可在gcc参数中指定调用路径
-L/usr/local/lib  
#指定调用库的名字,如libcurl.so
-lcurl

pkg-config路径:

export PKG_CONFIG_PATH=/usr/local/pkgconfig/

程序运行时动态库调用路径:

export LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH
#也可添加到全局动态库
echo '/usr/local/lib' > /etc/ld.so.conf.d/local.conf
ldconfig

查看gcc默认查找头文件/动态库目录:

`gcc -print-prog-name=cc1` -v 
gcc -print-search-dirs

参考:http://www.cnblogs.com/vamei/archive/2013/04/04/2998850.html

Nginx添加Lua支持

发布时间:January 4, 2014 // 分类:Nginx // No Comments

安装LuaJIT:

wget http://luajit.org/download/LuaJIT-2.0.2.tar.gz
tar zxvf LuaJIT-2.0.2.tar.gz 
make && make install PREFIX=/usr/local/luajit2.02
echo "/usr/local/luajit2.02/lib" > /etc/ld.so.conf.d/luajit.conf
export LUAJIT_LIB=/usr/local/luajit2.02/lib/
export LUAJIT_INC=/usr/local/luajit2.02/include/

安装Nginx及Ngx_lua模块:

wget http://nginx.org/download/nginx-1.4.4.tar.gz
tar zxvf nginx-1.4.4.tar.gz 
cd nginx-1.4.4/
wget -O ngx_devel_kit0.2.19.tar.gz https://github.com/simpl/ngx_devel_kit/archive/v0.2.19.tar.gz
tar zxvf ngx_devel_kit0.2.19.tar.gz
wget -O lua-nginx0.9.3.tar.gz https://github.com/chaoslawful/lua-nginx-module/archive/v0.9.3.tar.gz
tar zxvf lua-nginx0.9.3.tar.gz 
./configure --prefix=/usr/local/nginx1.4.4 --conf-path=/etc/nginx/nginx.conf --with-ipv6 \
--with-http_ssl_module --add-module=./ngx_devel_kit-0.2.19/ --add-module=./lua-nginx-module-0.9.3/ 
make && make install

测试:

location /lua {
    default_type 'text/plain';
    content_by_lua "ngx.print(ngx.var['arg_tag'], '\\n')";
}

访问:

curl -i '192.168.1.2/lua?tag=hello'
分类
最新文章
最近回复
  • opnfense: 谢谢博主!!!解决问题了!!!我之前一直以为内置的odhcp6就是唯一管理ipv6的方式
  • liyk: 这个方法获取的IPv6大概20分钟之后就会失效,默认路由先消失,然后Global IPV6再消失
  • 海运: 不好意思,没有。
  • zongboa: 您好,請問一下有immortalwrt設定guest Wi-Fi的GUI教學嗎?感謝您。
  • 海运: 恩山有很多。
  • swsend: 大佬可以分享一下固件吗,谢谢。
  • Jimmy: 方法一 nghtp3步骤需要改成如下才能编译成功: git clone https://git...
  • 海运: 地址格式和udpxy一样,udpxy和msd_lite能用这个就能用。
  • 1: 怎么用 编译后的程序在家里路由器内任意一台设备上运行就可以吗?比如笔记本电脑 m参数是笔记本的...
  • 孤狼: ups_status_set: seems that UPS [BK650M2-CH] is ...
归档