Re: ipvsadm: One-packet scheduling with UDP service is unstable
From: Drunkard Zhang
Date: Thu Aug 22 2013 - 07:00:39 EST
2013/8/22 Julian Anastasov <ja@xxxxxx>:
>
> Hello,
>
> On Tue, 20 Aug 2013, Drunkard Zhang wrote:
>
>> Need help here, thank you for replying :-)
>>
>> I'm setting up a syslog cluster based on IPVS, all UDP datagrams sent
>> from firewall with fixed source IP and fixed source port, so
>> pseudo-random balancing based on client IP and port won't working. And
>> it seems that keepalived is not supporting One-packet scheduling
>> option, so I did some hacks on it after keepalived started:
>>
>> 1. dump LVS rules with ipvsadm -S -n > rules-vs3;
>> 2. add --ops option;
>> 3. restore LVS rules with ipvsadm-restore < rules-vs3;
>> 4. dump the running LVS rules with ipvsadm -S -n
>>
>> So, I got two problems here:
>>
>> 1. Dumped rules in step 4 above is not usable anymore, the double-dash
>> in --ops lost, so I can't restore rule with this dump anymore. This
>> must be a bug.
>>
>> 2. The --ops option is not working sometimes you applied the rules,
>> and in most of times the --ops just not working. To make it work, just
>> 'ipvsadm-restore < rules-vs3' for plenty of times until it's working.
>> I haven't find the patterns make it work yet. This is lucky, I can't
>> get it work on second host at all.
>>
>> The "not working" above means the UDP datagrams from one source IP is
>> sticked to one realserver, it doesn't distribute to other realservers
>> which --ops designed for.
>
> Can you try with recent ipvsadm from git:
>
> git clone git://git.kernel.org/pub/scm/utils/kernel/ipvsadm/ipvsadm.git
>
> I see related commit that will print -o for
> the OPS feature:
>
> ===
> commit 6a03100c189d00e3a8235215392465b5b877ba8f
> Author: Krzysztof Gajdemski <songo@xxxxxxxxxxxxx>
> Date: Thu Mar 21 11:40:06 2013 +0100
>
> ipvsadm: Fix wrong format of -o option in FMT_RULE listing
>
> 'ipvsadm -S' listed one-packet scheduling option in wrong format
> ('ops' instead of '--ops' or '-o') preventing any service with OPS
> feature from restoring using 'ipvsadm -R'. Now we use '-o' which
> works well with save/restore commands.
>
> Signed-off-by: Krzysztof Gajdemski <songo@xxxxxxxxxxxxx>
> Signed-off-by: Simon Horman <horms@xxxxxxxxxxxx>
> ===
>
> Let me know if you still have any problems with OPS.
> Sending to lvs-devel@xxxxxxxxxxxxxxx and
> lvs-users@xxxxxxxxxxxxxxxxxxxxxx should be enough for
> ipvsadm related discussions.
Thanks, this resolved my first problem :D
>> So I wondering if there's some CONFIG_* options that ipvs needs, or
>> recent development broke the code?
>
> No kernel options should be related to OPS. I assume
> you are not using the SH scheduler. Make sure the OPS mode
> is properly applied to the virtual service, check for "ops"
> in the configuration:
>
> cat /proc/net/ip_vs
Still no lucky here, ops is set in running config, but it's not like
that in real world.
vs3 ~ # cat /proc/net/ip_vs
IP Virtual Server version 1.2.1 (size=1024)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
UDP 96A46478:0202 wrr ops
-> 96A46459:0202 Route 0 0 0
-> 96A46458:0202 Route 0 0 0
-> 96A46457:0202 Route 0 0 0
-> 96A46456:0202 Route 0 0 0
-> 96A46455:0202 Route 0 0 0
-> 96A46454:0202 Route 0 0 0
-> 96A46453:0202 Route 0 0 0
-> 96A46452:0202 Route 0 0 0
-> 96A46451:0202 Route 0 0 0
-> 96A46450:0202 Route 25 0 1
-> 96A4644F:0202 Route 25 0 1
-> 96A4644E:0202 Route 25 0 1
-> 96A4644D:0202 Route 30 0 2
-> 96A4644C:0202 Route 20 0 1
-> 96A4644B:0202 Route 20 0 1
-> 96A4644A:0202 Route 25 0 1
-> 96A46449:0202 Route 20 0 1
-> 96A46448:0202 Route 25 0 1
-> 96A46447:0202 Route 20 0 1
-> 96A46446:0202 Route 20 0 1
-> 96A46445:0202 Route 20 0 1
-> 96A46444:0202 Route 25 0 1
-> 96A46443:0202 Route 15 0 1
-> 96A46442:0202 Route 20 0 1
-> 96A46441:0202 Route 20 0 1
And the traffic routed to each realserver didn't following weight I
set, it's routed pretty much one to one. I got 17 udp sources sending
to 16 different realservers, the others are bonding to another VIP.
Prot LocalAddress:Port CPS InPPS OutPPS InBPS OutBPS
-> RemoteAddress:Port
UDP x.x.x.120:514 0 67622 0 12339373 0
-> x.x.x.65:514 0 29 0 2895 0
-> x.x.x.66:514 0 225 0 21850 0
-> x.x.x.67:514 0 4003 0 586117 0
-> x.x.x.68:514 0 5049 0 781526 0
-> x.x.x.69:514 0 160 0 16163 0
-> x.x.x.70:514 0 6091 0 914365 0
-> x.x.x.71:514 0 757 0 74428 0
-> x.x.x.72:514 0 4716 0 736039 0
-> x.x.x.73:514 0 4167 0 663728 0
-> x.x.x.74:514 0 3800 0 571342 0
-> x.x.x.75:514 0 192 0 19467 0
-> x.x.x.76:514 0 11309 0 1889147 0
-> x.x.x.77:514 0 3052 0 309840 0
-> x.x.x.78:514 0 8336 0 2004194 0
-> x.x.x.79:514 0 7333 0 1747346 0
-> x.x.x.80:514 0 8403 0 2000929 0
-> x.x.x.81:514 0 0 0 0 0
-> x.x.x.82:514 0 0 0 0 0
-> x.x.x.83:514 0 0 0 0 0
-> x.x.x.84:514 0 0 0 0 0
-> x.x.x.85:514 0 0 0 0 0
-> x.x.x.86:514 0 0 0 0 0
-> x.x.x.87:514 0 0 0 0 0
-> x.x.x.88:514 0 0 0 0 0
-> x.x.x.89:514 0 0 0 0 0
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/