Asterisk 16 Update
11 to 16 - what could possibly go wrong
Jöran Vinzens
@vinzens81
linkedin.com/in/jvinzens/
Why?
Why?
Oh, it is not mandatory? Than do something else.
Why?
It is working, why should we?
Why?
There is a risk that something fails.
Why?
There is nothing new we need.
Why?
Just some years ago we spent a lot of time on it. Why again?
Why?
Not now, no time! If we do it, we have to do it right!
Why?
Doh… We need Opus, security, ARI feature... and stuff….Now!
Why?
OK, now we need to update. ASAP
Why?
It is easy to ignore and always something else is more important.
Where did we start?
Where did we start?
● Asterisk 11
● AGI Scripts
● Patches for lawful Interception
● Chan_sip
● Running in daemon tools
● Debian Lenny / Jessie
Todo
Todo
● Dialplan Macro() -> GoSub()
● Adapt to new channel handling
● PJSIP migration
● Daemon tools to systemd
Test existing featureset
Test existing featureset
SIPp call tests
Test behaviour Asterisk 11 vs. Asterisk 16
https://github.com/sipgate/signaling-test
Test existing featureset
SIPp call tests
Test behaviour Asterisk 11 vs. Asterisk 16
13 Testcases written
Execute via e.g. vREST.io, Jenkins
https://github.com/sipgate/signaling-test
Test existing featureset
SIPp call tests
https://github.com/sipgate/signaling-test
Test Script
SIPp Sender
SIPp Receiver
Test existing featureset
SIPp call tests
https://github.com/sipgate/signaling-test
Test Script
SIPp Sender
SIPp Receiver
Test existing featureset
Test Script
SIPp Sender
SIPp Receiver
Test Script
SIPp Sender
SIPp Receiver
SIPp call tests
https://github.com/sipgate/signaling-test
Rest2Bash
SIPp call tests
https://github.com/sipgate/signaling-test
Test existing featureset
Rest2Bash
Jenkins pipeline
Ansible deploy
Test dev
Ansible deploy
Test production
Test existing featureset
Feature tests:
● Basic Call
● Setting of headers
● Accounting
● Blacklist
● Emergency Calls
● CDR input in DB
● ...some more
Test existing featureset
All feature tests were fine
Let’s do it!
Let's do it!
● Build package for Debian Buster
● Port sip.conf to pjsip.conf
● Check AGI for obsolete code
● Rewrite Dialplan
Let's do it!
Build package for Debian Buster
Let's do it!
Build package for Debian Buster
Using test step to verify our patch.
Let's do it!
Port sip.conf to pjsip.conf
Let's do it!
Port sip.conf to pjsip.conf
Try to use sip2pjsip script… - failed
Build from bottom up
Using Ansible magic to configure production and dev environment
Let's do it!
Check AGI for obsolete code
Let's do it!
Check AGI for obsolete code
Grep for all AGI interaction
Nothing critical found o/
Let's do it!
Set header after Dial() command
Let's do it!
Set header after Dial() command
[someWhere]
exten => _X.,1,SIPSetHeader(Foo: Bar)
exten => _X.,n,Goto(someWhereElse...)
[someWhereElse]
exten => _X.,n,Dial(SIP/${EXTEN}@dest)
Let's do it!
Set header after Dial() command
[someWhere]
exten = _X.,1,Set(_HeaderFoo: Bar)
same = n,Goto(someWhereElse...)
[someWhereElse]
exten = _x.,1,Dial(SIP/${EXTEN}@dest,g(subDialHook^s^1))
[subDialHook] #generic do all Dial()
exten = s,n,ExecIf($[${LEN(${HeaderFoo})} > 0]?
Set(PJSIP_HEADER(add,Foo=${HeaderFoo}))
)
same = n,Return()
Let's do it!
Macro to Gosub
Let's do it!
Macro to Gosub
exten => _X.,1,Macro(doSomething)
[macro_doSomething]
exten => s,1,doSomething(${MACRO_EXTEN})
Let's do it!
Macro to Gosub
exten => _X.,1,GoSub(subDoSomething,${EXTEN},1)
[subDoSomething]
exten => _X.,1,doSomething(${EXTEN})
same = n,Return()
Let's do it!
New formatting
Let's do it!
New formatting
exten => _X.,1,Set(ABC)
exten => _X.,n,Set(DEF)
exten => _X.,n,Set(XYZ)
Let's do it!
New formatting
exten = _X.,1,Set(ABC)
same = n,Set(DEF)
same = n,Set(XYZ)
Let's do it!
● Build package for Debian Buster
● Port sip.conf to pjsip.conf
● Check if AGI do stupid stuff
● Rewrite Dialplan
All needed to be done in order to try it.
SIPp call tests
Test behaviour Asterisk 11 vs. Asterisk 16
There are differences needed to be covered.
Let's do it!
SIPp call tests
Test behaviour Asterisk 11 vs. Asterisk 16
There are differences needed to be covered.
There is no “canreinvite=no”
Let's do it!
Let's do it!
All feature tests were fine
Let’s run it!
Let's run it!
Put some calls on the system (as few as possible) and observe.
Let's run it!
Put some calls on the system (as few as possible) and observe.
CRASH
Let's run it!
Put some calls on the system (as few as possible) and observe.
Sporadic crashes
CRASH
Let's run it!
Put some calls on the system (as few as possible) and observe.
Sporadic crashes
All related to Fax T.38 Invites
CRASH
Let's run it!
Put some calls on the system (as few as possible) and observe.
Asterisk 16.5.1 to the rescue!
Worked fine for several days.
Let's run it!
Put real traffic on the System and observe.
Let's run it!
Put real traffic on the System and observe.
CRASH
Let's run it!
Put real traffic on the System and observe.
Open file limit.
CRASH
Let's run it!
Put real traffic on the System and observe.
SystemD Ulimit was to low.
Let's run it!
Worked with same call load.
Further issues
● Asterisk Updates
● Logging
● Load peaks
● Continuous load
Further issues
Further issues
Asterisk Updates
Further issues
Asterisk Updates
SystemD stops a System while uninstalling.
Install Unitfile from Ansible instead of Debian package
Further issues
Logging
Further issues
Logging
● Log in var/log/syslog
● Log in var/log/daemon.log
● Log in var/log/asterisk/messages
Further issues
Logging
● Log in var/log/syslog
● Log in var/log/daemon.log
● Log in var/log/asterisk/messages
Start asterisk with “-pmgf” and “stdoutput=null” in Unitfile solved it.
Further issues
Load peaks
Further issues
Load peaks
● High disk IO caused by Asterisk
Further issues
Load peaks
● High disk IO caused by Asterisk
Disable logrotate within Asterisk and use Debian logrotate.
Further issues
Continuous load
Further issues
LoadAsterisk 16.5.1
15min avg
2.76
Asterisk 11
15min avg
0.604
Further issues
Load
● Perl AGI scripts cause high load
Move logic from Perl AGI to Java FastAGI
Well..
Load, still ongoing.
Update of asterisk took 4p ~2 months
Questions?
Thanks for listening!
Jöran Vinzens
@vinzens81
linkedin.com/in/jvinzens/

Asterisk 11to16, What could go wrong