最近接触到angularjs这个前端框架,也学习到了用restful API的方式进行前后端的交互,在接下来的一段时间中边学习边总结这段时间的学习成果。

要实现登录认证功能,常用的有以下这两种方法:

  1. Basic access authentication
  2. Digest access authentication

这两种方法各有优劣,

  • Basic access authentication 主要是胜在简单,只需要在Requset Header中加入 Authorization: Basic QWxhZGRpbjpvcGVuIHNlc2FtZQ==,其中字符串”Basic “后面的是”username:password”字符串的Base64编码,如果验证不成功,server会在Response Header中返回WWW-Authenticate: Basic realm="domain",状态码为401。但是这个方法不安全,在生产环境中须要在HTTPS下使用。但也因为他够简单,我们可以在开发测试环境中使用。
  • Digest access authentication 顾名思义,主要是通过server端返回的随机码和摘要算法对username和password、realm以及需要发送的请求内容提取摘要,发送给server,该方法相比Basic access authentication来说稍稍复杂,但是却有较强的安全性。

为简单和实用性,本文实现第一种方法,HTTP Basic Auth,主要用到以下插件:

  1. front-end

    AngularJs
    AngularJs-storage
    Bootstrap

  2. back-end

    Flask==0.10.1
    Flask-HTTPAuth==2.5.0
    Flask-SQLAlchemy==2.0
    Flask-Script==2.0.5
    Jinja2==2.8
    MarkupSafe==0.23
    SQLAlchemy==1.0.8
    Werkzeug==0.10.4
    itsdangerous==0.24

angularJs 的 AuthService

AuthService 是我们自己定义的一个service,提供核心的认证功能,包括登录、登出、以及用token认证,关于token,需要强调的是,我们第一次登录的时候使用用户名和密码,server端返回一个token,以后进行认证的时候就使用token,以减少用户名和密码在网上传输的次数。token有使用时间限制,一般是几个小时之内。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
myApp.service('AuthService', function($q, $http){
var self = this;
self.profile = null;
self.isAuthenticated = false;

function request(url, username_or_token, password) {
return {
method: 'GET',
url: url,
headers: {
Accept: 'application/json',
Authorization: 'Basic ' + btoa(username_or_token + ':' + password)
}
};
}

self.signin = function (options, successCallback, errorCallback) {
var req = request('/auth/login', options.username, options.password);
$http(req).
then(function(response) {
var token = response.data.token;
if (token) {
self.isAuthenticated = true;
self.profile = response.data.profile;
successCallback(token);
}

}, function(error) {
console.log(error.message);
errorCallback();
}
);

};

self.signout = function () {
self.isAuthenticated = false;
self.profile = null;
};

self.authenticate = function (token) {
var req = request('/auth/login_with_token', token, '');
var deferred = $q.defer();
$http(req)
.then(function (response) {
self.profile = response.data.profile;
self.isAuthenticated = true;
deferred.resolve();
}, function () {
self.isAuthenticated = false;
deferred.reject();
});
return deferred.promise;
}
});
  1. signin

    首先用requst函数把username:password的base64编码加入http 请求头Authorization中,然后在认证成功后把server端返回的token发送给succesCallback。

  2. authenticate

    与signin一样,首先调用request函数,不过把username替换成token,password置空,用token的方式来认证。与signin提供回调函数不一样,这里使用$q返回promise,可以让调用then方法进一步处理结果。

Angular-Storage 存储token

server返回的token用angular-storage保存,angular-storage使用浏览器的localStorage或者sessionStorage保存,如果浏览器不支持这些功能,就要ngCookies保存。

function onLoginSuccess(token) {
    store.set('token', token);
    $scope.hitMsg = 'login success';
    $location.path('/');
}

function onLoginFailed() {
   $scope.hitMsg = 'username or password incorrect';
}

$scope.submit = function() {
    AuthService.signin({
        username: $scope.user.username,
        password: $scope.user.password
    }, onLoginSuccess, onLoginFailed)
};

上面是LoginCtrl的一段代码,可以看到,当登录成功,用store.set('token', token)保存token。

$rootScope.$on('$routeChangeStart', function (event, next, current) {
      if (!$rootScope.isLogin && next.templateUrl) {
          var token = store.get('token');
          if (token) {
              AuthService.authenticate().then(function() {
              }, function() {
                  store.remove('token');
        $location.path('/login');
              });
          } else if (next.loginRequired) {
              $location.path('/login');
          }
      }
});

上面的代码监听了全局的$routeChangeStart事件,如果用户没有登录,我们提取token进行认证,如果认证失败就删除本地的token(表明token过期了),并提醒用户登录。loginRequired是我们自定义的一个属性,用来标识该route是否需要登录。见下面的route代码:

$routeProvider
.when('/', {
    templateUrl: 'static/partials/welcome.html',
    controller: 'WelcomeCtrl',
    loginRequired: false
})
.when('/login', {
    templateUrl: 'static/partials/login.html',
    controller: 'LoginCtrl',
    loginRequired: false
})
.when('/logout', {
    templateUrl: 'static/partials/welcome.html',
    controller: 'LogoutCtrl',
    loginRequired: true
})
.otherwise({
    rediretTo: '/'
});

拦截 401 错误

如果认证失败,server会返回WWW-Authenticate: Basic realm="domain",状态码为401的错误,此时我们需要拦截这个错误,并要求用户重新登录。

myApp.factory('tokenInvalidInterceptor', ['$q', '$location', '$rootScope', 'store',
    function ($q, $location, $rootScope, store) {
    return {
        'responseError': function (rejection) {
            if (rejection.status == 401) {
                if (store.get('token'))
                    $rootScope.$broadcast('unauth_token');
                else
                    $rootScope.$broadcast('unauth');
            }
            return $q.reject(rejection);
        }
    }
}]);

// 在myApp.config中
$httpProvider.interceptors.push('tokenInvalidInterceptor');

上面这段代码创建了一个Interceptror,如果在responseError中出现401错误码,广播事件给监听者,这里如果store中包含token就广播unauth_token事件,反之,广播unauth事件,下面的代码会根据这两个事件给用户提供相应的提示信息

$rootScope.$on('unauth_token', function () {
    store.remove('token');
    $location.path('/login');
    $scope.hitMsg = 'token expire, please login';
});

$rootScope.$on('unauth', function () {
    $location.path('/login');
    $scope.hitMsg = 'please login';
});

此外,如果server端返回WWW-Authenticate: Basic realm="domain",浏览器会弹出一个对话框
提示用户登录,要避免这种情况,我们可以让服务端返回WWW-Authenticate: xBasic realm="Authentication Required", 如下

http_auth = HTTPBasicAuth()

@http_auth.error_handler
def unauthorized():
    response = make_response()
    response.status_code = 401
    response.headers['WWW-Authenticate'] = 'xBasic realm="{0}"'.format('Authentication Required')
    return response

关于token的生成和认证

  • 生成token

    from itsdangerous import TimedJSONWebSignatureSerializer as Serializer
    def generate_auth_token(self):
      s = Serializer(current_app.config['SECRET_KEY'], expires_in=3600)
      return s.dumps({'id': self.id})
    

把用户id序列化为一个默认一小时expire的字符串

  • 认证token

    @http_auth.verify_password
    def verify_password(username_or_token, password):
        if password == '':
            g.current_user = User.verify_auth_token(username_or_token)
            g.token_used = True
            return g.current_user != None
        user = User.query.filter_by(name=username_or_token).first()
        if not user:
            return False
        g.current_user = user
        g.token_used = False
        return user.verify_password(password)
    

这里是指HTTPAuth的密码验证函数,该函数接收两个参数,如果第二个参数,即 password 为空,则认为是以token认证,否则以username和password认证。

让程序跑一跑

要让程序成功跑起来,首先创建virtualenv,然后安装依赖

pip install -r requirements.txt

然后创建数据库

$ python manager.py create_db

然后创建用户名和密码

$ python manager.py shell

>>> u = User(name='admin')
>>> db.session.add(u)
>>> db.session.commit()

最后运行服务

$ python manager.py runserver
  • 欢迎界面
    welcome.jpg

  • 登录界面
    login.png

  • 登录成功跳转到欢迎界面
    logged.png

附上程序源码

https://github.com/khalily/flask-angular-http-auth

安装 Package Control 的方法:

  • 打开 Sublime Text 2,按下 Control + 调出 Console
  • 将以下代码粘贴进命令行中并回车:
    1
    import urllib2,os; pf='Package Control.sublime-package'; ipp = sublime.installed_packages_path(); os.makedirs( ipp ) if not os.path.exists(ipp) else None; urllib2.install_opener( urllib2.build_opener( urllib2.ProxyHandler( ))); open( os.path.join( ipp, pf), 'wb' ).write( urllib2.urlopen( 'http://sublime.wbond.net/' +pf.replace( ' ','%20' )).read()); print( 'Please restart Sublime Text to finish installation')

SublimeClang 的使用

  • 提供C/C++/object-c等语言的代码提示和浏览功能
  • Control + Shift + P 呼出控制界面 输入 install, 然后 Enter
  • 输入SublimeClang, Enter
  • 打开 SublimeClang 的 user-setting 文件, 添加

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    {
    "additional_language_options":
    {
    "c++" : ["-std=c++11"] // enable C++11
    }
    ,

    "options":
    [
    "-Wno-deprecated-declarations",
    "-isystem", "/usr/include",
    "-isystem", "/usr/include/c++/*",
    "-Wall"
    ]
    }

  • 如果是单独的项目,在项目文件中添加

    1
    2
    3
    4
    5
    6
    7
    8
    "settings":
    {
    "sublimeclang_options":
    [
    "-I/home/wyang/workspace/muduo",
    "-I/home/wyang/workspace/muduo/**"
    ]
    }
  • 记得在build文件中添加–std=c++11使Build&Run功能生效

GoSublime

  • 提供go语言的coding环境, very good!
  • 同上安装就行

SublimeGDB

  • gdb的一个插件,可以用来简单调试
  • 同上安装
  • 在项目文件中添加

    1
    2
    3
    4
    5
    "settings":
    {
    "sublimegdb_commandline": "gdb --interpreter=mi ./contains",
    "sublimegdb_workingdir": "${folder:${project_path:contains.cc}}"
    }
  • 注意在build文件中加入-g(调试)选项

SublimeCodeIntel

  • 多种语言的代码提示功能
  • 打开 SublimeCodeIntel 的 user-setting 文件, 添加
    1
    2
    3
    4
    5
    6
    7
    8
    9
    {
    "codeintel_config": {
    "Python": {
    "env": {
    "PYTHONPATH": "/usr/lib/python2.7/site-packages:/usr/lib/python:$PYTHONPATH"
    }

    }

    }

    }

有用的一些插件

ConvertToUTF8

  • 可以避免在windows中打开包含gbk编码的文件时生成.dump文件
  • 需要卸载 GBK Encoding Support 插件

SublimeLinter

  • 高亮不规范的python代码

Bracket Highlighter

  • 默认配置文件的 "bracket_styles""default""color"改为

    1
    "color": "entity.name.class"  // green

个人设置文件

  • 自己的一些设置

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    {
    "color_scheme": "Packages/Color Scheme - Default/Monokai.tmTheme",
    "default_line_ending": "unix",
    "dictionary": "Packages/Language - English/en_US.dic",
    "ensure_newline_at_eof_on_save": true,
    "fade_fold_buttons": false,
    "font_face": "Consolas",
    "font_size": 12,
    "highlight_line": true,
    "highlight_modified_tabs": true,
    "ignored_packages":
    [
    ],

    "open_files_in_new_window": false,
    "reparse_use_dirty_buffer": true,
    "tab_size": 2,
    "translate_tabs_to_spaces": true,
    "trim_trailing_white_space_on_save": true
    }

1. git subtree

1
2
3
# git subtree 提示不支持该命令时
sudo chmod +x /usr/share/doc/git/contrib/subtree/git-subtree.sh
sudo ln -s /usr/share/doc/git/contrib/subtree/git-subtree.sh /usr/lib/git-core/git-subtree

1
2
3
4
5
6
7
8
9
10
11
# 添加仓库并fetch
git remote add -f <子仓库名> <子仓库地址>

# 加上--squash 表示合并子仓库的所有提交为一次提交(感觉不加好些,可以看日志记录)
git subtree add --prefix=<子目录名> <子仓库名> <分支> --squash

# 从远程子仓库同步
git subtree pull --prefix=<子目录名> <远程分支> <分支> --squash

# 向远程子仓库推送
git subtree push --prefix=<子目录名> <远程分支名> 分支

2. git submodule

注意: submodule 只能在<子目录>下看到log信息,如果需要在web UI 下看log 需要对<子仓库地址>可访问

1
2
3
4
5
6
# 添加子仓库
git submodule add <子仓库地址> <子目录名>
# 删除子仓库
git submodule rm <子目录名>
# 同步子仓库
git submoudle update

pktgen是一个位于linux内核层的高性能网络测试工具。
主要用来测试网络驱动与网卡设备,支持多线程,能够产生随机mac地址、IP地址、UDP端口号的数据包。
pktgen的配置与统计信息查看都使用/proc文件系统完成,/proc文件系统是一种特殊的,有软件创建的文件系统,内核使用/proc文件系统向外界导出信息,外界也可以通过它配置内核态的一些进程的参数,如ps top uptime等linux发行版中的很多工具就是通过/proc实现的。在大多情况下,我们只用/proc读出数据(用于调试内核驱动等),而在pktgen中配置命令就用到了/proc的写入数据功能。

pktgen 相关文件

  • /proc/net/pktgen/pgctrl
    控制pktgen的启动和停止,默认输出当前版信息

    1
    2
    # cat /proc/net/pktgen/pgctrl 
    pktgen 2.72: Packet Generator for packet performance testing.
  • /proc/net/pktgen/kpktgend_X
    pktgen 线程配置文件,个数与CPU核心数一致

  • /proc/net/pktgen/ethX
    pktgen 关联的网卡配置文件
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    # cat /proc/net/pktgen/bond1.32\@0
    Params: count 100000000 min_pkt_size: 64 max_pkt_size: 64
    frags: 0 delay: 0 clone_skb: 0 ifname: bond1.32@0
    flows: 0 flowlen: 0
    queue_map_min: 1 queue_map_max: 1
    dst_min: 32.0.22.1 dst_max:
    src_min: 202.0.0.1 src_max: 202.0.0.32
    src_mac: 90:e2:ba:4d:01:d9 dst_mac: 90:e2:ba:4c:06:51
    udp_src_min: 9 udp_src_max: 9 udp_dst_min: 9 udp_dst_max: 9
    src_mac_count: 0 dst_mac_count: 0
    Flags: IPSRC_RND QUEUE_MAP_CPU
    Current:
    pkts-sofar: 44913913 errors: 0
    started: 414539499019us stopped: 414784402020us idle: 25946us
    seq_num: 44913914 cur_dst_mac_offset: 0 cur_src_mac_offset: 0
    cur_saddr: 0xd0000ca cur_daddr: 0x1160020
    cur_udp_dst: 9 cur_udp_src: 9
    cur_queue_map: 1
    flows: 0

pktgen 基本使用

  1. 加载模块

    1
    modprobe pktgen
  2. 为了方便配置,定义下面的工具函数

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    pgset() {
    local result

    echo $1 > $PGDEV

    result=`cat $PGDEV | fgrep "Result: OK:"`
    if [ "$result" = "" ]; then
    cat $PGDEV | fgrep Result:
    fi
    }

    pg() {
    echo inject > $PGDEV
    cat $PGDEV
    }
  3. 绑定pktgen的0号线程到eth1网卡,添加少量配置并启动发包

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    PGDEV=/proc/net/pktgen/kpktgend_0
    pgset "rem_device_all"
    pgset "add_device eth1"

    PGDEV=/proc/net/pktgen/eth1
    pgset "flag IPSRC_RND"
    pgset "src_min 202.0.0.1"
    pgset "src_max 202.0.0.32"

    PGDEV=/proc/net/pktgen/pgctrl
    pgset "start"

性能优化

  • 为了最大化pktgen的发包性能,增大网卡的TX ring buffer

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    # ethtool -G eth1 tx 4096
    # ethtool -g eth1
    Ring parameters for eth1:
    Pre-set maximums:
    RX: 4096
    RX Mini: 0
    RX Jumbo: 0
    TX: 4096
    Current hardware settings:
    RX: 4096
    RX Mini: 0
    RX Jumbo: 0
    TX: 4096
  • 网卡中断亲和性
    通过一对一绑定网卡中断到cpu可以提高cpu缓存的利用率,并利用到网卡的多队列特性(一个网卡队列对应一个中断号)

  1. 绑定原理

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    # cat /proc/interrupts |grep eth1|awk '{print $1}'  # 获取中断号
    138:
    139:
    140:
    141:
    142:
    143:
    144:
    145:
    146:
    # echo 0x0 > /proc/irq/138/smp_affinity # 绑定0号cpu到irq 138
    # echo 0x2 > /proc/irq/139/smp_affinity # 绑定1号cpu到irq 139
    计算公式:单cpu的 mask 值 = 2 ** cpu号 的16进制值
  2. 绑定脚本 eth2cpu.sh

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    57
    58
    59
    usage()
    {
    echo "Usage: $1 [-f file.conf ] [-h]"
    exit
    }


    cpu_mask() {
    local id=$1
    echo "obase=16;`echo 2^$id|bc`" |bc
    }

    set_eth_cpu()
    {
    conf=$1
    echo $conf set_cpu

    cnt=$(wc -l $conf|awk '{print $1;}')
    for ((i=1;i<=$cnt;i++))
    do
    eth=$(awk -F'=' -v n=$i '{ if ((index($0,"e") || index($0, "p") )&&NR==n) print $1;}' $conf)
    v=$(awk -F'=' -v n=$i '{ if ((index($0,"e") || index($0, "p") )&&NR==n) print strtonum($2);}' $conf)
    if [ $v ]
    then
    interruptnum=$(cat /proc/interrupts | grep $eth | awk -v e=$eth '{if ($NF==e) print $0;}'| awk -F':' '{print strtonum($1);}')
    echo $eth"="$v":"$interruptnum

    targetfile=/proc/irq/$interruptnum/smp_affinity

    mask=$(cpu_mask $v)
    [ -e $targetfile ] && echo $mask > $targetfile
    listfile=/proc/irq/$interruptnum/smp_affinity_list
    printf "$listfile: %d\n" `cat $listfile`
    fi
    done
    }

    while getopts "f:h" flag
    do
    case $flag in
    f)conf=$OPTARG
    ;;
    h)
    usage
    ;;
    *)
    usage
    ;;
    esac
    done

    if [ -s $conf ]
    then
    set_eth_cpu $conf
    fi

    echo "End"

    exit
  3. 绑定配置文件 eth2cpu.conf

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    eth1-TxRx-0=0
    eth1-TxRx-1=1
    eth1-TxRx-2=2
    eth1-TxRx-3=3
    eth1-TxRx-4=4
    eth1-TxRx-5=5
    eth1-TxRx-6=6
    eth1-TxRx-7=7
    eth1-TxRx-8=8
    eth1-TxRx-9=9
    eth1-TxRx-10=10
    eth1-TxRx-11=11
    eth1-TxRx-12=12
    eth1-TxRx-13=13
    eth1-TxRx-14=14
    eth1-TxRx-15=15
  4. 使用方法

    1
    # ./eth2cpu.sh -f eth2cpu.conf

pktgen 发包脚本

一个支持多线程发包的脚本

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
#! /bin/sh

modprobe pktgen

pgset() {
local result

echo $1 > $PGDEV

result=`cat $PGDEV | fgrep "Result: OK:"`
if [ "$result" = "" ]; then
cat $PGDEV | fgrep Result:
fi
}

pg() {
echo inject > $PGDEV
cat $PGDEV
}

for cpu in {1..15}
do
PGDEV=/proc/net/pktgen/kpktgend_${cpu}
echo "Removing all devices"
pgset "rem_device_all"
echo "Adding bond1.32@${cpu}"
pgset "add_device bond1.32@${cpu}"
#echo "Setting max_before_softirq 10000"
#pgset "max_before_softirq 10000"
done

CLONE_SKB="clone_skb 0"
PKT_SIZE="pkt_size 64"
#COUNT="count 0"
COUNT="count 100000000"
DELAY="delay 0"

# 开启15个线程
for cpu in {1..15}
do
PGDEV="/proc/net/pktgen/bond1.32@${cpu}" # 注意每个线程要有不同的标识(`@`前缀)
echo "Configuring $PGDEV"
pgset "delay 0"
pgset "queue_map_min $cpu"
pgset "queue_map_max $cpu"
pgset "flag IPSRC_RND"
pgset "flag QUEUE_MAP_CPU"
pgset "src_min 202.0.0.1"
pgset "src_max 202.0.0.32"

pgset "$COUNT"
pgset "$CLONE_SKB"
pgset "$PKT_SIZE"
pgset "$DELAY"
pgset "dst 32.0.22.1"
pgset "dst_mac 90:e2:ba:4c:06:51"

done

PGDEV=/proc/net/pktgen/pgctrl

echo "Running… ctrl^C to stop"
pgset "start"
echo "Done"

pktgen 配置参考

pgset "clone_skb 1"     sets the number of copies of the same packet
pgset "clone_skb 0"     use single SKB for all transmits
pgset "burst 8"         uses xmit_more API to queue 8 copies of the same
                     packet and update HW tx queue tail pointer once.
                     "burst 1" is the default
pgset "pkt_size 9014"   sets packet size to 9014
pgset "frags 5"         packet will consist of 5 fragments
pgset "count 200000"    sets number of packets to send, set to zero
                     for continuous sends until explicitly stopped.

pgset "delay 5000"      adds delay to hard_start_xmit(). nanoseconds

pgset "dst 10.0.0.1"    sets IP destination address
                     (BEWARE! This generator is very aggressive!)

pgset "dst_min 10.0.0.1"            Same as dst
pgset "dst_max 10.0.0.254"          Set the maximum destination IP.
pgset "src_min 10.0.0.1"            Set the minimum (or only) source IP.
pgset "src_max 10.0.0.254"          Set the maximum source IP.
pgset "dst6 fec0::1"     IPV6 destination address
pgset "src6 fec0::2"     IPV6 source address
pgset "dstmac 00:00:00:00:00:00"    sets MAC destination address
pgset "srcmac 00:00:00:00:00:00"    sets MAC source address

pgset "queue_map_min 0" Sets the min value of tx queue interval
pgset "queue_map_max 7" Sets the max value of tx queue interval, for multiqueue devices
                     To select queue 1 of a given device,
                     use queue_map_min=1 and queue_map_max=1

pgset "src_mac_count 1" Sets the number of MACs we'll range through.
                     The 'minimum' MAC is what you set with srcmac.

pgset "dst_mac_count 1" Sets the number of MACs we'll range through.
                     The 'minimum' MAC is what you set with dstmac.

pgset "flag [name]"     Set a flag to determine behaviour.  Current flags
                     are: IPSRC_RND # IP source is random (between min/max)
                          IPDST_RND # IP destination is random
                          UDPSRC_RND, UDPDST_RND,
                          MACSRC_RND, MACDST_RND
                          TXSIZE_RND, IPV6,
                          MPLS_RND, VID_RND, SVID_RND
                          FLOW_SEQ,
                          QUEUE_MAP_RND # queue map random
                          QUEUE_MAP_CPU # queue map mirrors smp_processor_id()
                          UDPCSUM,
                          IPSEC # IPsec encapsulation (needs CONFIG_XFRM)
                          NODE_ALLOC # node specific memory allocation
                          NO_TIMESTAMP # disable timestamping

pgset spi SPI_VALUE     Set specific SA used to transform packet.

pgset "udp_src_min 9"   set UDP source port min, If < udp_src_max, then
                     cycle through the port range.

pgset "udp_src_max 9"   set UDP source port max.
pgset "udp_dst_min 9"   set UDP destination port min, If < udp_dst_max, then
                     cycle through the port range.
pgset "udp_dst_max 9"   set UDP destination port max.

pgset "mpls 0001000a,0002000a,0000000a" set MPLS labels (in this example
                                     outer label=16,middle label=32,
                 inner label=0 (IPv4 NULL)) Note that
                 there must be no spaces between the
                 arguments. Leading zeros are required.
                 Do not set the bottom of stack bit,
                 that's done automatically. If you do
                 set the bottom of stack bit, that
                 indicates that you want to randomly
                 generate that address and the flag
                 MPLS_RND will be turned on. You
                 can have any mix of random and fixed
                 labels in the label stack.

pgset "mpls 0"          turn off mpls (or any invalid argument works too!)

pgset "vlan_id 77"       set VLAN ID 0-4095
pgset "vlan_p 3"         set priority bit 0-7 (default 0)
pgset "vlan_cfi 0"       set canonical format identifier 0-1 (default 0)

pgset "svlan_id 22"      set SVLAN ID 0-4095
pgset "svlan_p 3"        set priority bit 0-7 (default 0)
pgset "svlan_cfi 0"      set canonical format identifier 0-1 (default 0)

pgset "vlan_id 9999"     > 4095 remove vlan and svlan tags
pgset "svlan 9999"       > 4095 remove svlan tag


pgset "tos XX"           set former IPv4 TOS field (e.g. "tos 28" for AF11 no ECN, default 00)
pgset "traffic_class XX" set former IPv6 TRAFFIC CLASS (e.g. "traffic_class B8" for EF no ECN, default 00)

pgset stop                  aborts injection. Also, ^C aborts generator.

pgset "rate 300M"        set rate to 300 Mb/s
pgset "ratep 1000000"    set rate to 1Mpps

pgset "xmit_mode netif_receive"  RX inject into stack netif_receive_skb()
              Works with "burst" but not with "clone_skb".
              Default xmit_mode is "start_xmit".

参考文档

  1. HOWTO for the linux packet generator
  2. Pktgen Getting Started

记录给ATS做cache预热的方法

磁盘准备

  1. 4 块 SSD 硬盘, 480G
  2. 给4块硬盘分别做 RAID 0 (记住要初始化)
  3. 编辑udev rules 文件, 配置raw磁盘

    # Apache Traffic Server owns disk for RAW access
    
    ACTION=="add", KERNEL=="sdb", RUN+="/bin/raw /dev/raw/raw1 %N"
    ACTION=="add", KERNEL=="sdc", RUN+="/bin/raw /dev/raw/raw2 %N"
    ACTION=="add", KERNEL=="sdd", RUN+="/bin/raw /dev/raw/raw3 %N"
    ACTION=="add", KERNEL=="sde", RUN+="/bin/raw /dev/raw/raw4 %N"
    
    KERNEL=="raw[1-4]", OWNER="nobody", GROUP:="trafficserver", MODE="660
    

    执行命令

    groupadd trafficserver
    start_udev
    

ats配置文件

records.conf

CONFIG proxy.config.http.push_method_enabled INT 1
CONFIG proxy.config.http.cache.when_to_revalidate INT 3
CONFIG proxy.config.http.cache.required_headers INT 0
CONFIG proxy.config.log.logfile_dir STRING /data/var/log/trafficserver

remap.conf

regex_map http://(.*) http://$1
regex_map http://(.*):8080 http://$1

storage.conf

/dev/raw/raw1 
/dev/raw/raw2 
/dev/raw/raw3 
/dev/raw/raw4 

ip_allow.config

src_ip=192.168.5.151                             action=ip_allow method=ALL

开始执行

创建日志目录

mkdir -p /data/var/log/trafficserver

Clear Cache

traffic_server -Cclear

Run Manager

traffic_manager 

日志查看

traffic_logcat -f squid.blog

自动生成URL

sh gen_urls.sh

运行py脚本

sh ats_cache.sh

补充

由于该脚本频繁创建、关闭连接,导致系统出现很多TIME_OUT,用光了系统可用端口,故,
修改两个内核参数,
如下:
    #开启对于TCP时间戳的支持,若该项设置为0,则下面一项设置不起作用
    echo 1 > /proc/sys/net/ipv4/tcp_timestamps
    #表示开启TCP连接中TIME-WAIT sockets的快速回收
    echo 1 > /proc/sys/net/ipv4/ipv4.tcp_tw_recycle

相关shell脚本代码

git clone git@github.com:khalily/ats-cache.git

参考资料

  1. RAID 磁盘阵列详解
  2. raw 磁盘详解
  3. ATS raw磁盘布局
  4. ATS Sample config

运行效果

$ ./expression-tree

Input Expression:     [1 + 23 * 1 - 12 * ( 4 + 2 * 5 )]
Caculate Result:     -144
Prefix Expression:     - + 1 * 23 1 * 12 + 4 * 2 5
Infix Expression:     1 + 23 * 1 - 12 * 4 + 2 * 5
Suffix Expression:     1 23 1 * + 12 4 2 5 * + * -

一个简单的结构图

expression-tree

关于面向对象

  1. interface定义方法

    type BinaryTree interface {
        setLeft(node Node)
        setRight(node Node)
        PrefixExpression() string
        SuffixExpression() string
        NifixExpression() string
    }
    
  2. struct管理数据

    type OperateNode struct {
        operate string
        left    Node
        right   Node
    }
    
  3. 需要改变对象定义指针类型的方法

    func (on *OperateNode) setLeft(node Node) {
        on.left = node
    }
    
  4. 不需要改变对象则定义对象类型的方法

    func (on OperateNode) getName() string {
        return on.operate + " "
    }
    
  5. 赋值给接口用指针类型总是正确的

    func CreateOperateNode(v string) (node BinaryTree) {
        switch v {
        case "+":
            return &Add{OperateNode{"+", nil, nil}}
        case "-":
            return &Subtraction{OperateNode{"-", nil, nil}}
    
        ...
    

代码在这里